Link verification

From the RepoWiki wiki

Link verification
Also known as linkViolations, WikipediaCheck, see-also, aliases, LinkValidator
Files 15
Lines of code 2,503
Languages TypeScript, JSON
Entry points packages/engine/src/link/test-wikipedia.ts
packages/engine/src/link/index.ts
First commit 2 October 2026
Last commit 7 October 2026
Main contributors Sean May (100%)
Revision 5b50749

Link verification is the subsystem that validates links in wiki pages and proposals, ensuring id tokens name active features, Wikipedia links are reachable, and feature mentions in See also sections exist. It extracts code identifiers from the repository as feature aliases, checks Wikipedia article titles against the REST API, and rewrites links to point to the correct features or plain text when targets do not exist.

Overview

Link verification ensures that every id token in a page's claims names an active feature in the manifest, preventing broken links in stored revisions.[1] The See also section of a page must list only active features that have pages; retired features and redirects are not valid targets.[2]

Wikipedia links marked with Title are checked against Wikipedia's REST API to confirm articles exist and are not disambiguation pages.[3] Code identifiers—HTTP routes, database table names, environment variables and CLI commands—found in each feature's source files become aliases that readers can link to by name.[4]

The linker rewrites target tokens using the manifest: a feature id is normalized to id form with its title as the label, a Wikipedia title is validated and kept or dropped, and unknown targets become plain text.[5] See also entries are ranked by the combined weight of file-level imports and calls between features, keeping only active features and never listing a page itself.[6] Link verification is called during wiki page generation, rejecting pages with broken links, and during CI to audit stored pages and the project article.[7]

How it works

codeAliases() scans feature files with linear-time patterns to extract routes (Flask and Express decorators), SQL table names, environment variables and Click or Typer commands, deduplicating across features and filtering out identifiers another feature shares or that collide with feature names.[8] createTargetResolver() builds a two-level lookup that first tries feature ids, then normalized names and aliases, and follows redirects to active features only, leaving retired features unreachable.[9] linkTokensIn() finds all target and label tokens in a text by scanning in one left-to-right pass, skipping those inside backtick code spans, using a regex that matches the site's own parser.[10]

createPageLinker() rewrites every link token on a page once only: id tokens become the normalized feature id (or plain text if unknown or the page itself), Title tokens are looked up in Wikipedia and kept only if valid and not already linked, and a page never links the same feature twice.[11] checkWikipediaTitles() fetches article summaries from Wikipedia's REST API for all Title targets, caching results and returning the canonical title for linkable articles or null for missing or disambiguation pages.[12]

featureNeighbours() combines edge weights from the file graph between each pair of features, summing the weights of all file-level imports and calls that cross feature boundaries.[13] seeAlsoFor() ranks a feature's neighbours by combined weight heaviest first, breaks ties by id, filters to active features only, and returns the top 5 or fewer by its limit parameter.[14]

linkViolations() reports every See also id that is not an active feature, every id token that names a retired feature or redirect, every id outside the manifest, and every Title not found by Wikipedia lookup.[7]

Data flow

flowchart LR
  n2["packages/engine/src/link/aliases.ts"]
  n5["packages/engine/src/link/links.ts"]
  n7["packages/engine/src/link/see-also.ts"]
  n11["packages/engine/src/link/violations.ts"]
  n14["packages/engine/src/link/wikipedia.ts"]
  n16[["Core data model"]]
  n20[["Wiki writing"]]
  n2 -->|"imports"| n16
  n5 -->|"imports"| n16
  n7 -->|"imports"| n16
  n11 -->|"calls"| n5
  n11 -->|"imports"| n16
  n11 -->|"imports"| n20
  n14 -->|"imports"| n16
Data flow of Link verification

Repository source code enters codeAliases() as a map of file paths to source text, and code identifiers exit as a record of feature ids to sorted alias lists.[8] A manifest, file graph and Wikipedia cache flow through featureNeighbours() and checkWikipediaTitles() to produce a map of feature names to neighbours and a map of Wikipedia titles to their canonical forms or null.[13][12]

A revision's sections and claims, together with the manifest and Wikipedia link map, enter createPageLinker() which processes claim text and outputs a function that rewrites all link tokens once.[11] Stored revisions older than the current manifest are validated by storedLinkViolations() against the manifest at their own sha and checked that each link still routes in the current manifest.[15]

History

  • Link verification began with feature resolution and See also ranking on 2 October 2026.[16]
  • Wikipedia link checking against the REST summary API through a cache was added on 2 October 2026.[17]
  • Code aliases extracted from routes, table names, environment variables and CLI commands joined the system on 2 October 2026.[18]
  • The linker was hardened against backtracking ReDoS attacks by bounding regex quantifiers on 2 October 2026.[19]
  • Test files and HTTP-client receivers were excluded from code scanning on 2 October 2026.[20]
  • Link tokens are now read the same way the site reads them, names are resolved deterministically, and redirect cycles are dropped on 2 October 2026.[21]
  • Feature titles used as link words are stripped of unsafe characters, and links to retired or redirect ids are flagged on 2 October 2026.[22]
  • Link violation messages now quote model-supplied targets and ids, escaping invisible characters on 2 October 2026.[23]
  • Aliases were checked with the core aliasProblem function and code aliases were added to manifest validation on 2 October 2026.[24]
  • A table is now aliased only when its name shares a word with its feature's own names on 2 October 2026.[25]
  • Code aliases were forbidden from naming another feature's subject on 2 October 2026.[26]
  • Links to active features without a stored page are now counted instead of failing on 2 October 2026.[27]
  • The project article's links are checked in wiki:check on 3 October 2026.[28]
  • The article's links are counted toward features without a page instead of failing on 3 October 2026.[29]
  • Stored pages are judged against their own manifest and checked to route in the current one on 3 October 2026.[30]
  • Never-valid links are reported once instead of repeatedly on 3 October 2026.[31]
  • Greek -sis nouns and their -ses plurals are now matched on 3 October 2026.[32]
  • Short words ending in -sis are left alone and not treated as stems on 3 October 2026.[33]

See also

References

  1. ^ packages/engine/src/link/violations.ts:L5-9@5b50749
  2. ^ packages/engine/src/link/see-also.ts:L26-47@5b50749
  3. ^ packages/engine/src/link/wikipedia.ts:L74-77@5b50749
  4. ^ packages/engine/src/link/aliases.ts:L17-26@5b50749
  5. ^ packages/engine/src/link/links.ts:L115-121@5b50749
  6. ^ packages/engine/src/link/see-also.ts:L4-24@5b50749
  7. ^ a b packages/engine/src/link/violations.ts:L24-31@5b50749
  8. ^ a b packages/engine/src/link/aliases.ts:L249-252@5b50749 (codeAliases)
  9. ^ packages/engine/src/link/links.ts:L52-82@5b50749 (createTargetResolver)
  10. ^ packages/engine/src/link/links.ts:L23-30@5b50749
  11. ^ a b packages/engine/src/link/links.ts:L122-160@5b50749 (createPageLinker)
  12. ^ a b packages/engine/src/link/wikipedia.ts:L119-126@5b50749
  13. ^ a b packages/engine/src/link/see-also.ts:L5-24@5b50749 (featureNeighbours)
  14. ^ packages/engine/src/link/see-also.ts:L34-48@5b50749 (seeAlsoFor)
  15. ^ packages/engine/src/link/violations.ts:L136-173@5b50749
  16. ^ Commit e2fce02: "feat(link): resolve feature links, compute See also, and refuse links to nowhere" (PR #244)
  17. ^ Commit a06337d: "feat(link): check Wikipedia links against the REST summary API through the cache" (PR #246)
  18. ^ Commit 35de3cd: "feat(link): add code identifiers to the manifest's aliases" (PR #247)
  19. ^ Commit e8adf27: "fix(link): count each route once, find Flask and any-receiver routes, read SQL and env forms, skip tests, and stay stable across re-links" (PR #247)
  20. ^ Commit ed2ada4: "fix(link): skip HTTP-client receivers, conftest, tests.py and fixtures; write invisible characters as escapes" (PR #247)
  21. ^ Commit d379186: "fix(link): read tokens as the site does, resolve names deterministically, and drop cycles" (PR #244)
  22. ^ Commit b3ab061: "fix(link): strip a feature title used as link words and flag links to retired or redirect ids" (PR #254)
  23. ^ Commit 33c8831: "fix(link): quote model-supplied targets and ids in link violation messages" (PR #255)
  24. ^ Commit df9879f: "fix(link): check aliases with core's aliasProblem in the manifest step, the linker and the store" (PR #247)
  25. ^ Commit ef85316: "fix(link): alias a table only when its name shares a word with its feature's names" (PR #262)
  26. ^ Commit 9e98e59: "fix(link): never give a feature a code alias that names another feature's subject" (PR #262)
  27. ^ Commit d5095ec: "feat(link): count links to active features without a stored page in wiki:check" (PR #262)
  28. ^ Commit fcc6321: "feat(verify): check the stored project article in wiki:check" (PR #295)
  29. ^ Commit 86a3cd3: "fix(verify): count the article's links to features without a page instead of failing on them" (PR #295)
  30. ^ Commit 9c67728: "fix(link): judge a stored page's links at its own sha and check they still route" (PR #319)
  31. ^ Commit e8de58b: "fix(link): check the About article's links still route and report a never-valid link once" (PR #319)
  32. ^ Commit 8ceabca: "fix(link): match a -sis name with its -ses plural" (PR #320)
  33. ^ Commit 09ab62d: "fix(link): leave a short word ending in sis alone and pin the canceled retry's rows before the rerun" (PR #322)

This page was last edited on 8 October 2026, at commit 5b50749.