Keyword cannibalization audit
Search engines pick one. Usually the wrong one — and the links, the internal anchors and the clicks stay split between two URLs that should have been a single page.
CannibalScan scores every pair of pages in your crawl and shows you which ones are competing, how strongly, and what to do about it. The whole scan happens inside your browser: the CSV never leaves your machine.
The scan
Everything up to the recommendation is arithmetic on your own hardware. That is the whole reason the tool can be free: there is no server doing the expensive part.
wasm-core/parser.rs
The CSV is read and each row becomes a page with url, title, meta, H1, body text and word count. Malformed rows are counted, not silently dropped — the report tells you how many were skipped.
wasm-core/overlaps.rs
Tokens from the title enter the bag three times, the H1 twice, meta and body once. Cannibalization is driven by intent, and intent lives in the title far more than in the body copy.
overlaps::cosine
TF-IDF vectors, cosine similarity, every pair: n*(n-1)/2 comparisons. Above the threshold the pair is reported; the sparse dot product walks the shorter vector, so a few thousand pages stay interactive.
web/src/lib/audit · api/audit
Competing pairs are clustered into groups, and a Mastra agent returns one recommendation per cluster: which URL to keep, which to merge or 301, and a consolidated title and meta for the survivor.
Why it is built this way
The two halves solve different problems and stack instead of competing: WebAssembly makes the heavy, boring part free, and the model is only asked to do the part that needs judgement.
Parsing and pair scoring compile to WebAssembly and run locally. No upload, no queue, no per-scan cost — move the threshold slider and the whole report recomputes.
Trade-offThe WASM package under web/src/wasm/ is build output and is gitignored: clone the repo and you must run wasm-pack build before the app will start.
The audit route is the only thing that talks to a model, so the API key stays on the server and never reaches the client bundle.
Trade-offWithout OPENROUTER_API_KEY the audit is unavailable, and the free half — parsing and pair detection — keeps working. Graceful degradation was a requirement, not an accident.
The Rust logic is plain functions tested with cargo test; wasm-bindgen is a thin shell on top. The audit logic — clustering, schema, prompt — is equally pure TypeScript.
Trade-offTwo toolchains to install. On Windows the Rust GNU toolchain is required: rustup default stable-x86_64-pc-windows-gnu.
Field weights
Two posts about the same subject share a lot of body copy and that is fine. Two posts with near-identical titles are asking the search engine the same question twice. The weights encode exactly that difference.
| Field | Weight | Why |
|---|---|---|
| title | ×3 | The strongest declaration of intent a page makes, and the one the SERP shows. |
| h1 | ×2 | Restates the intent on the page itself; usually agrees with the title, sometimes betrays it. |
| meta description | ×1 | Marketing copy, not ranking copy — useful signal, low weight. |
| body text | ×1 | Topical background. Large and noisy, which is why it does not get to dominate. |
Questions
Keyword cannibalization is when two or more pages on the same site target the same search intent, so a search engine has to choose between them. The result is usually that neither page ranks as well as a single consolidated page would: links, internal anchors and clicks are split across near-duplicate URLs.
It builds a weighted bag of tokens per page — title ×3, H1 ×2, meta and body ×1 — turns each into a TF-IDF vector, and computes the cosine similarity of every page pair, n*(n-1)/2 of them. Pairs at or above the threshold, 0.70 by default, are reported as competing.
No. Parsing and scoring run in your browser in Rust compiled to WebAssembly, so the CSV never leaves your machine. Only the optional audit step sends anything out, and it sends just the URL, title, H1 and meta of the pages already flagged — never the body text, never the full crawl.
Start at 0.70. Lower it toward 0.60 on a small site where loosely related pages still compete; raise it toward 0.80 on a large blog where broad topical overlap is normal and only near-duplicates matter. The scan re-runs instantly when you move the slider, because nothing round-trips to a server.
A crawl exported as CSV with url, title, meta description, H1 and body text columns — Screaming Frog's internal_html export is the reference format. The audit step additionally needs an OpenRouter key; without one, parsing and pair detection still work.
Yes, that is the second mode: compare_sites scores your pages against a competitor's crawl, sharing one IDF across the union of both corpora, and reports both the overlapping pairs and the terms where the competitor is strong and you are absent.
Run it
The engine is a Rust crate, the app is Next.js with the App Router. Step three is the one people forget: the WASM package is build output and is not in the repository.
# 1 · core tests, native cd wasm-core && cargo test # 2 · the same tests against wasm32, via Node wasm-pack test --node # 3 · generate the WASM package inside the web app wasm-pack build wasm-core --target web --out-dir ../web/src/wasm # 4 · agent tests (clustering, schema, prompt) and dev server cd web && npm install && npm run test && npm run devRead the code on GitHub