feat(doc-links): Markdown, ADR, reST, AsciiDoc and PDF sections link to the code they name

[DeusData]

Summary

Documentation becomes part of the graph, linked by its specific contents to the code it names. Markdown (and MDX), architecture decision records, reStructuredText (Sphinx), AsciiDoc (Antora) and PDF documents get a node per section or page, and each section links to the code files and, where the reference names or covers one, to the exact definition (function, method, class, ...). An agent can go from a section to the code it describes, and from any code node back to the documents that mention it, through the graph alone.

All of it is decided by the indexer, deterministically, from explicit references: links, paths, qualified names in code spans, Sphinx roles and directives, include directives with line ranges and tags, Antora resources and javadoc attributes, and structural mentions in PDF text. Bare names never become edges. A reference that does not resolve with certainty is recorded with a reason instead.

Stacked on #2550, the doc-comment layer (MENTIONS edges, the doc_link_unresolved table, index_status.doc_links), which is stacked on #2543. Upgrade: the semantic index version goes to 6, so an existing index is rebuilt in full once.

What you get

Nodes.

• Section per heading (Markdown, reST, AsciiDoc), with the section's prose as its docstring. Text before the first heading belongs to the document's File node.
• Section per PDF page, with the page's text as its docstring (get_code_snippet returns the page text, never PDF bytes).
• ADR per architecture decision record (Markdown or reST), with id, status, date, deciders and title properties. A record's "supersedes" statement resolves to the other record; a statement (the phrase opens its line or field and names a record) becomes a SUPERSEDES edge; the same words in running prose stay rows.
• AsciiDoc (.adoc, .asciidoc) and PDF files are discovered and indexed; Sphinx projects' extra source_suffix entries (e.g. Django's .txt) are read as reST.

Edges. MENTIONS from a section (or the document) to a File or a definition, with:

{"via":"markdown","syntax":"code_name","tier":"exact","line":42,"count":1,"target_lines":[10,24]}

• via: markdown, rst, asciidoc, pdf. syntax: the family (below). line: where the reference is written. target_lines: present when the reference names a line range (an include with lines=/tag=, literalinclude, #L10-L24).
• A line range binds the innermost definition that holds it, or the definitions it covers, or the file.

Format	Families that ship as edges	Resolved, kept as rows (not yet past the held-out audit)
Markdown	link, path, code_name, code_path, supersedes (an ADR's statement opening its line or field: SUPERSEDES edges)	bare_path (a file or one directory named alone in a code span); supersedes_prose (the same words in running prose)
reStructuredText	role, autodoc, code_name, literalinclude, code_path, object (.. py:class::, .. method::, ...)	include, kernel_doc
AsciiDoc	include	attribute (Antora javadoc/xref resources), code_path, code_name
PDF	pdf_path, pdf_file, pdf_qn	(bare names are never edges)

Unresolved references go to doc_link_unresolved with the layer's reasons (missing, ambiguous, external, test_only_target, not_indexed, graph_gap, unparseable) and are counted in index_status.

Navigation. trace_path inbound with edge_types:["MENTIONS"] from a symbol lists the sections that mention it; query_graph reads any of it. No new tool.

How references are resolved

• Paths: relative to the document's directory, then the repository root (no bare-file-name fallback beyond the document's directory); documentation tools' base paths are honoured (Sphinx: /x is relative to the source dir; Antora: component/module/family resource IDs from antora.yml and the playbook).
• Qualified names: exact definition match first, then a segment-aligned suffix of 2+ pieces; exact case before folded; Python module:attr only in Python; never product documentation to a test target unless the path names it. Five scope rules, each of which can only take a link away (a refused candidate still makes two candidates ambiguous): a name written from std is the standard library's, never a C/C++ definition or header here; a name written from a JVM root package (com., org., ...) matches only where it starts the package, never a relocated copy's tail; a Java/Kotlin/Scala/Groovy/C# type matches in its own case only (debug.log is no member of Debug); a definition in another copy of the document's project (snapshots side by side) never answers the document; a code name never binds a JVM package folder (a dotted config key or cross-reference such as snapshot.mode is not a package; Python packages keep binding).
• Sphinx: the forms T, C.T, M.T, M.C.T under module/currentmodule/class context, Python re-exports through package __init__.py imports, primary_domain, intersphinx and extlinks from conf.py; C domain names by kind. A one-piece name is searched only for classes, exceptions, functions and data in the declared module, never for bare members.
• AsciiDoc: include:: with lines= and tag(s)= (tags read from the included indexed file only), attributes resolved from the document and the Antora component, javadoc attributes to Java/Kotlin types and members.
• PDF: a dependency-free text-layer extractor (zlib only; fonts, encodings, CMaps, Type 3 glyph names, /ActualText, object and cross-reference streams, damaged files repaired as far as the prototype did); structural mentions only.
• Incremental: a section's edges belong to its document. Documents bound by line ranges into a file re-resolve when that file's body changes (a body-only edit leaves line numbers stale otherwise).

Proof on real repositories (development corpora; the held-out list was checked first)

What	Corpus	Result
Markdown links/paths/names	fastapi, cosmos-sdk, dapr, curl	2,899 / 1,286 / 184 / 52 MENTIONS (before: file-to-file REFERENCES_FILE only); link pairs 408/409 equal to the field-test prototype (the one difference a deliberate self-link skip); 44/45 hand-checked name edges correct, the 45th fixed (exact-case precedence)
ADR detection and facts	five corpora	detection 182/182 and titles 182/182 identical to the prototype; status 179/182, date 174/182 (differences explained: archived status, braces in a date field)
reST / Sphinx	django fb113765	6,695 of 7,238 prototype links identical; the rest are repeated-title attribution (same document→target, file instead of section) and deliberate refusals including the prototype's own audited errors
reST ADRs	openedx/course-discovery	34/34 records
AsciiDoc / Antora	junit-framework 6c8ca116	javadoc attributes 362/362 identical; includes 292 identical + 60 bound to the enclosing definition + 7 multi-definition ranges to the file
PDF text layer	749 PDFs from 21 repositories	720/720 comparable documents byte-identical to the prototype's text (1,492/1,492 pages), 0.73 s vs 14.7 s; flattened copies identical; 44,070 mutated files without a sanitizer finding
PDF links	five corpora	identical to the prototype pipeline (53/53)

Extractor fix found by the audits: one Field per class field

The AsciiDoc audit bound a Java field reference to a module-level Variable. Java, C# and Solidity list the same node kind as a field and as a variable, so every class field was emitted twice: as a Field of its class and as a Variable carrying the module's qualified name (demo.query.filters for Query.filters), which collided with package paths and took usages away from the field. A declaration that is already a Field is no longer minted again, and a field with an initializer is named by its declarator's name (it was named PROP = "v"). On junit-framework, Variable nodes drop from 2,899 to 685, and 6,938 of the 9,901 usages that went to the duplicates now reach the field. Swift and GDScript (variables only) and D and Dart (no duplicates) are unchanged.

A field and a method of one name in one class body (executable and executable(Path)) shared one qualified name, so the graph kept only one of them (in junit-framework 107 pairs: the field lost in 90, the method in 17). Such a field is now <Owner>.<name>#field (the base#suffix fence of #macro and the Rust cfg twins; every other field keeps its name), and a read, a write or a plain value reference that resolves to the method goes to its field twin; calls keep the method. junit-framework: Field 2,808 → 2,898 and Method 14,124 → 14,141; WRITES into those methods 66 → 0 and into their fields 1 → 96; USAGE into those methods 97 → 0 and into their fields 2 → 212. Limit: a C# partial class split across files can still collide (the check sees one class body). Both changes ride on this PR's index-format bump (CBM_SEMANTIC_INDEX_VERSION 6 over its base's 5).

Security review

An independent read-only review of the layer found four HIGH, one MEDIUM and one LOW issue, all in the PDF path and all fixed here, each with a test that fails when its fix is reverted, and each re-proven on the PDF corpus (byte-identical text before and after):

• a chain of objects (a /Length naming a stream, object streams inside object streams) cost a stack frame per object: now constant depth (a /Length is read as a value; ISO 32000-1 7.5.7 for object streams);
• forms drawing the next form many times, level under level: the first reading of each form stays whole, re-readings are bounded per document (256 MiB, counted);
• one cross-reference stream named by many sections was decoded each time: now once;
• objects whose value never ended were read to the end of the file each: an object is read up to the next object;
• the mention scanner's line joins compared every candidate of the page: now the two lines' candidates;
• a page's docstring keeps 256 KiB of its text, with a line saying where it was cut.

Tests

New suites doc_links_md (Markdown and ADRs, 10 tests), doc_links_pdf (15), doc_links_rst (7), doc_links_adoc (4); incremental-equals-full steps for every format, including line-bound documents after a body edit. Each PDF security fix has a test that fails when the fix is reverted.

Verification

Local ladder on this PR's tip (the doc-comment layer PR below it included):

• macOS arm64: 8,619 passed, 0 failed, 10 skipped (153 suites).
• Linux arm64 (gcc, ASan + LeakSanitizer): 8,457 passed, 0 failed, 9 skipped, and the gcc -O2 production build.
• Windows: not run on this tip. Hosted CI does not run on this stacked PR (its base is feat/doc-links-core, not main; only DCO runs), so the Windows check comes when the stack is retargeted to main. The local Windows arm64 VM last ran on the previous tip (8,451 passed, 83 skipped); two route tests of edge_types_probe fail there identically on the base of this stack, so they do not come from this PR.
• Lint: cppcheck, clang-format (Homebrew LLVM), no-suppress and the memory-core ratchet.
• Held-out audits: below.

Precision audit

Each family ships as edges only if it passes a held-out audit registered before any output existed on the audit repositories: 100 sampled links per family (census below that, families under 15 links pooled), judged blind against the source, bar Wilson 95 % lower bound ≥ 0.90 and point ≥ 0.95. Held-out repositories: three per format, chosen by an objective rule from code search, disjoint from every development corpus.

PDF (done). The C pipeline ran on the eighteen repositories of the field test's two held-out PDF audits. Its links equal the frozen prototype pipeline's on the same graph in every repository (5,217 of 5,217), and the audited prototype links it makes take their labels (255 of 260):

Family	Correct	Point	Wilson 95 % low	Ships
pdf_path	61 of 61	1.000	0.941	yes
pdf_file	110 of 114	0.965	0.913	yes
pdf_qn	77 of 80	0.963	0.896	no: its references are below_bar_tier rows

Markdown, reStructuredText, AsciiDoc (done). Nine held-out repositories (three per format, the top-starred results of a code search for the format's markers that hold an ADR log, Sphinx autodoc, or AsciiDoc source includes; none used in development): 782 sampled links judged blind, a second reader on every link not judged correct and on a seeded tenth of the rest (102 rechecks, 100 % agreement).

Family	Correct	Point	Wilson 95 % low	Ships
Markdown link	100 of 100	1.000	0.963	yes
Markdown path	50 of 50 (census)	1.000	0.929	yes
Markdown code_name	98 of 100	0.980	0.930	yes
reST role	100 of 100	1.000	0.963	yes
reST autodoc	98 of 98 (census)	1.000	0.962	yes
reST code_name	99 of 100	0.990	0.946	yes
AsciiDoc include	92 of 92 (census)	1.000	0.960	yes
Markdown code_path	93 of 100	0.930	0.863	no: a file name in a code span that means the reader's own file
AsciiDoc code_name	15 of 25 (census)	0.600	0.407	no: Java overloads, a config key and a package read as members
AsciiDoc code_path	12 of 13 (census)	0.923	0.667	no
reST code_path	3 of 3 (census)	1.000	0.439	no: too few held-out links for the bar
ADR supersedes	1 of 1 (census)	1.000	0.207	no: too few held-out links for the bar
reST object, include, literalinclude, kernel_doc; AsciiDoc attribute	none held out			no: not yet audited

Supplementary audits (same method, each registered before its queries; new held-out repositories chosen by the same kind of objective code-search walk, disjoint from all earlier ones). A family that failed was fixed only from what its sample showed, and a fix is judged on a NEW sample, never on the one that found the error:

Family	Correct	Point	Wilson 95 % low	Ships
reST literalinclude	100 of 100	1.000	0.963	yes
reST code_path	99 of 100	0.990	0.946	yes (the one error, a bare ../, is fixed)
PDF pdf_qn, after the owner rule (a Go method whose name lacks its receiver matches only the receiver-qualified form)	97 of 99	0.980	0.929	yes
Markdown code_path, with a directory (src/app/main.go, docs/decisions/)	94 of 100	0.940	0.875	no: all six errors named one directory alone (doc/ of another project)
Markdown code_path without the bare part (two or more path segments)	97 of 100	0.970	0.916	yes
Markdown bare_path (a file or one directory alone)	28 of 41 judged across the samples above	0.683	0.530	no: config.yaml or doc/ in prose is as often the reader's or another project's
reST object	17 of 22 (census)	0.773	0.566	no: a function directive bound an example script's module; fixed, needs a new sample
ADR supersedes	37 of 38 (census, 12 ADR repositories)	0.974	0.865	no: a template's combined label Supersedes / Depends on: read as a statement; fixed
ADR supersedes, after that fix	144 of 163 (census, 8 repositories)	0.883	0.825	no: Supersedes: none (it extends ADR 31), another record as the subject; fixed
ADR supersedes, after those fixes	82 of 87 (census, 40 repositories, at most 25 per repository)	0.943	0.872	no: the remaining errors were in running prose; split into statements (the phrase opens its line or field and names a record: 44 of 45 on this census) and supersedes_prose (38 of 42, held)
ADR supersedes statements	80 of 81 (fresh census, 27 repositories, at most 25 per repository)	0.988	0.933	yes; its one error, a wrapped paragraph whose subject was the line before, is fixed (a statement may not continue a paragraph)
reST kernel_doc	2 of 2 (census)	1.000	0.342	no: too few held-out links
reST object, after the fix (a function or class directive never binds a module)	105 of 105 (census, 6 of 12 new repositories with links, at most 25 per repository)	1.000	0.965	yes
AsciiDoc code_path	66 of 95 (census, 10 of 18 Antora repositories with links)	0.695	0.596	no: a path in product documentation means the reader's own file (.circleci/config.yml in CircleCI's docs: 25 of the 29 errors; ~/.m2/settings.xml, "your pom.xml"), and .. as range syntax
AsciiDoc attribute	none in 48 Antora repositories			no: no held-out links
AsciiDoc code_name, after the field and file-name fixes, overloads judged at method level	43 of 80 (census, 7 of 36 new Antora repositories with links, at most 25 per repository)	0.538	0.429	no: two repositories that carry copies of other projects (Boost vendored into a game, several HBase/Cassandra snapshots in a research repository) gave 17 of 49 correct: a std:: name bound to Boost's specialization, a class bound to another snapshot's copy; the documentation-first Antora repositories gave 26 of 28
AsciiDoc code_name, after four scope rules (std::, rooted JVM names, type case, project copies)	61 of 83 (census, 5 new repositories with links)	0.735	0.631	no: 21 of 22 errors bound a Java package folder to a dotted name that is no code (Debezium's snapshot.mode, OSGi configuration ids, JMX domains, a Maven groupId)
AsciiDoc code_name, after a fifth rule (no JVM package folders)	76 of 82 (census, 4 new repositories with links)	0.927	0.849	no: Ruby predicate methods (sections? bound to sections), a method bound to its file, and name() where a zero-argument overload exists (one node per method name)

A family that does not ship still resolves: its references are doc_link_unresolved rows with reason below_bar_tier, and its tests run it switched on, so a later audit can switch it on without other changes.

Known limits

• Repeated section titles in one document share one node (the existing Markdown rule), so their links attach to the document.
• A changed Python re-export forces a full re-index (as any added definition name already does).
• PDFs: no OCR; encrypted PDFs are not read.
• Links through symlinked directories are not followed (the walk skips symlinks).

Third-party data

The PDF tables carry the Adobe Glyph List subset (BSD-3-Clause), the core-14 font widths of Helvetica and Times-Roman (Adobe), and Unicode Character Database data (Unicode License v3); notices are in THIRD_PARTY.md and the SBOM. scripts/gen-pdf-unicode.py regenerates pdf_unicode.c byte for byte.