Skip to content

fix(extract): C typedefs become nodes; macros and enumerators get their own QNs - #2543

Open
DeusData wants to merge 10 commits into
mainfrom
fix/c-typedefs-macros-enumerators
Open

DeusData wants to merge 10 commits into
mainfrom
fix/c-typedefs-macros-enumerators

Conversation

@DeusData

@DeusData DeusData commented Oct 4, 2026 •

Copy link
Copy Markdown
Owner

Summary

In C-family code several entities competed for one graph node, and some had no node at all. This PR fixes the extraction defects behind that and gives macros and enumerators identities of their own.

Builds on #2459 (merged): the complete doc comments it captures are what the typedef, class and enum nodes below carry as docstrings.

Merged with main after #2458. #2458 also named the anonymous aggregate of typedef struct { ... } Name;, in extract_class_def. Together with this PR's extract_c_typedef that emitted Name twice (two Class defs, every member twice). fix(extract): name an anonymous C typedef aggregate once removes the #2458 case, and extract_c_anonymous_typedef_aggregate_is_one_def pins it (RED with the case restored: 2 Class defs for Tally). Both typedef test sets pass on the merge result.
Main's graph_buffer.c "Definition variants" (read by the test-impact engine) stores variants as [{"file_path","start_line","end_line"}]. This PR's C extractor wrote [{"start","end"}], which the merge dropped, so test_impact_engine_variant_tests_are_one_test failed. fix(extract): write C variants in graph_buffer's schema makes the extractor write the shared schema; the C-only gate and the one-carrier rule are unchanged.

One small commit that is not about extraction (test: checkpoint cross-platform harness fixes) is included here on purpose, so that the complete local test script is green again on a developer machine and on the Windows arm64 VM. It holds three test fixes; each of these steps fails on main in the environment named:

  • Version contract (Step 0x). The scoop manifest was re-pinned for v0.11.0, but its entry in the version-metadata contract stayed pin:0.11.0, and the contract rejects a pin that equals the newest release (Step 0x, for every checkout that sees the release tags). The entry moves to the release rule; both manifest hashes were checked against the v0.11.0 checksums.txt.
  • Watcher kill-switch test (Step 5e). tests/test_watcher_disabled.sh isolated only its cache, so where a released CBM daemon is in use the freshly built binary was refused and the step failed. It now uses scripts/test-runtime.sh like the other process-level tests and is listed in the runtime isolation contract. CI was never affected.
  • Fuzz-harness self-test (Step 0g). In tests/test_security_fuzz_harness.sh the fixture target that must be accepted read each request with the shell's read. For the harness's 1 MB request that took 11.8 s on the Windows arm64 VM, where the shell runs emulated, and the harness kills a target after 10 s, so the step failed there on any tree. The fixture now hands over to one python3 process, which the harness needs anyway (awk is no option: mawk answers nothing while its input stays open, as the interactive case keeps it). Every session the harness sends (32) was replayed through the old loop and the new responder: the answers are byte-identical on macOS, Linux and Windows, and the 1 MB session takes 240 ms on the VM.

Upgrade: node identities change for C-family code, so the semantic index version goes from 3 to 4 and an existing index is rebuilt in full once.

What was wrong

Defect Root cause
C typedef names had no node. With typedef struct/enum {...} X; the fields and enumerators were lost too. type_definition has no name field, so the class path dropped it.
API RetT name(...) was named after its return type, became a Method RetT.name, or vanished. Three tree-sitter error-recovery shapes around an unknown leading macro.
Every bodyless struct X / enum X minted a Class/Enum node. In one file, or in a same-stem .h/.c pair, a reference displaced the definition. No body check; same-QN arbitration is by path and line.
A #define NAME took the node of the function, type or enumerator called NAME. Macros used the same QN formula as definitions, and the later line won.
Enumerators were <scope>.<Enum>.<CONST>. C and unscoped C++ enums put them in the enclosing scope, which is how code refers to them.
A definition repeated across #if branches kept one span. One node per qualified name.
Functions disappeared where #if branches split a brace; a C keyword surfaced as a function named if. Error regions swallow the definitions; any identifier text was accepted as a name.

What changes

Extraction

  • Typedef names are definitions: Type, or Class/Enum with members for an anonymous typedef struct/enum {...} X;. An alias is dropped when the same file defines the tag of that name.
  • Macro-prefixed functions get their real name in all three recovery shapes. Shared recognisers keep the definition, the call scope, the parameter lookup and the C LSP caller name in agreement.
  • A bodyless tag is a reference and creates no node. A definition head left as loose ERROR tokens (struct NAME {) is recovered, spanning to its matching brace.
  • A C keyword is never a function name.
  • First-branch projection: when the raw parse lost structure wholesale (the root is an ERROR, or an error region spans 20 lines or more), the file is parsed once more with one branch kept per #if group and every byte position unchanged. Only functions the raw pass lost are adopted, under the same gates as the existing preprocessed rescue. Sampled on the Linux kernel this re-parses about 1% of C bytes.
  • Macro-wrapped enumerator lists (CURLOPT(CURLOPT_URL, ...)) yield one constant per slot instead of one per identifier.

Identity

  • Macros: a C-preprocessor macro's qualified name is <module>.<NAME>#macro (C, C++, CUDA, GLSL, Objective-C, ISPC). name and the Macro label are unchanged. The # fence is the convention the Rust #[cfg] twins already use. Whatever resolves a name or a plain QN prefers a definition and uses the macro only when no definition of that name is visible.
  • Enumerators: constants of C enums and of unscoped C++/Objective-C enums are <enclosing scope>.<CONST>; enum class / enum struct stay <Enum>.<CONST>. In C and Objective-C an enum declared inside a struct or union puts its constants at file scope, as the language does; in C++ and CUDA the class is the scope. The existing parent_class property names the enum: MATCH (v:Variable) WHERE v.parent_class = '<Enum QN>' RETURN v.name.
  • variants: a node whose file defines its QN more than once under one label carries the line spans of all those definitions, its own included: "variants":[{"start":30,"end":30},{"start":182,"end":227}]. That covers #if twins, macros redefined per platform, and C++ overloads. C-preprocessor languages only; never truncated; definitions in other files are not listed.

Consumers adapted

  • LSP target lookup, trace_path name resolution and get_code_snippet apply the definition-before-macro rule; get_code_snippet still finds a macro by its old QN and by its short name.
  • Full-text search indexes a macro's QN without the fence. Otherwise the word "macro" would be added to every macro row and push real name matches out of the BM25 candidate window (measured on redis: two name matches fell from raw rank 1 and 3 to behind 930 and 4,759 rows).
  • Checked and unchanged: the registry (Macro is no registry label), semantic vectors (macros are not embedded), detect_changes, check_index_coverage.

Upgrade note (for the release notes)

The index is rebuilt in full once after upgrading (semantic index version 4). In C-family code a preprocessor macro's qualified name now ends in #macro (mod.NAME#macro), the constants of C enums and unscoped C++/Objective-C enums are scope.CONST instead of scope.Enum.CONST (their enum is in the parent_class property), and typedef names are nodes. Names are unchanged. An exact macro QN, or a qn_pattern ending in \.NAME$, must become \.NAME#macro$.

Measured effect

redis (4f20cb48) and curl (c345611d), parent commit vs this change.

redis curl
nodes 38,625 → 40,484 28,180 → 29,750
edges 132,474 → 146,883 81,939 → 102,735
Type (typedef names) 0 → 573 0 → 193
Enum 45 → 123 38 → 145
Class 913 → 872 1,429 → 613
phantom Class nodes from references, removed 202 821
functions that a same-named macro had replaced 45 → 0 41 → 0
functions with no node 335 → 296 11 → 0
Macro nodes = distinct #define names with a node 5,513 3,489
macro and definition of one name, both present 107 136
USAGE edges into enumerators 1,355 → 3,363 573 → 14,075
nodes carrying variants 595 414
database size 122.7 → 129.0 MB 61.3 → 66.1 MB
pipeline time, median of 5 alternating runs 2,448 → 2,506 ms 1,833 → 1,872 ms
  • Other languages: nodes of non-C-family files are identical by QN (13,703 in redis, 11,536 in curl; 0 added, 0 removed, 0 changed).
  • Edges: every changed edge key is classified. The growth is USAGE, DEFINES and WRITES edges into nodes this PR adds. Of the 312 CALLS that disappear in redis, 236 were false edges from Tcl, shell and Python callers into C struct fields, and 74 are callers re-attributed from a File or Module node to the function that now exists.
  • Doxygen references: of 174 audited references in redis' vendored xxhash, 174 now reach the right entity (64 before).
  • Docstrings on the new nodes (with fix(extract): capture complete doc comments across languages #2459): typedef names 112/112 and 52/52; classes and enums from anonymous typedefs 85/85 and 39/39.
  • Determinism: two runs give identical nodes, edges and properties. One worker and several workers agree on every node's QN, label, span, variants, parent_class, docstring and signature.
  • Incremental: after editing two files, a closure repair gives the same node set, labels, spans, variants and parent_class as a fresh full index.
  • Upgrade path: an index written by the parent commit is rebuilt in full by this build and equals a fresh index.

Tests

28 new tests in extraction (16), pipeline (7), mcp (2), graph_buffer (2) and store_search (1), each failing without the change on a behaviour assertion. Targeted reverts were checked one cause at a time (version constant, LSP macro retry, MCP macro rank, snippet tier, slot rule, each serialisation path, the language gate, the C struct-scope rule and its C++ counterpart).

Verification

Local ladder on the final tree (both commits):

Leg Result
macOS arm64, complete scripts/test.sh 8,361 passed, 0 failed, 10 skipped (145 suites); every contract and regression step passed, Step 0x with the release tags visible and Step 5e with a released CBM daemon in use on the machine
Linux arm64, ASan + LeakSanitizer, complete scripts/test.sh 8,200 passed, 0 failed, 9 skipped (141 suites); every contract and regression step passed
Windows arm64 (native VM), complete scripts/test.sh every contract step passed, Step 0g and Step 0x included. Suites: 8,132 passed, 0 failed, 79 skipped (145 suites), but one suite did not finish: grammar_probe_d printed its first test name and then nothing until the runner stopped it at 900 s. The same suite passed in this VM on the C1 commit on 2026-10-02, and the other six grammar-probe suites passed in this run in 66 to 205 s, so it is recorded as an unattributed hang of that run, not as a C1 failure; the hosted Windows CI is the gate. (An earlier attempt of this leg stopped on a scheduler infrastructure error before the suites.)
Merge with main (2026-10-04) two textual conflicts with work merged meanwhile (the qn_sig_off field of #2061 next to variants; the macro tier placed after get_code_snippet's new definition tiers, so a definition still wins over a macro of the same name). On the merged tree: extraction, pipeline, mcp, graph_buffer, store_search, registry 1,294 passed, 4 skipped, 0 failed.
make lint-ci (cppcheck, clang-format) passed

Known limits (not changed here)

  • variants lists what extraction finds. Two of curl's 123 groups are short because one variant's head is itself lost to an error region.
  • The node kept for #if twins is still the last by start line; variants now exposes the other spans.
  • A struct tag and a same-named function or variable still share a QN (curl 1, redis 5 cases).
  • #defines inside function, struct and enum bodies have no node, as before.
  • The enumerator rule for macro-wrapped lists takes the first argument of the macro call that opens a slot. That is right for every list in redis and curl (313 constants checked) and remains a heuristic for other X-macro shapes.
  • The projection parse runs without the raw parse's time budget, like the existing preprocessed rescue parse.

Follow-ups (not in this PR)

  • A .h and its same-stem .c share one module QN, which is the root of the remaining cross-file collisions.
  • The unique_name fallback binds calls from other languages to C struct fields.
  • detect_changes could use variants to seed a node from a change inside another variant's span.

…ir own QNs

In C-family code several different entities competed for one node, and
some had no node at all:

- A typedef name was never a definition: `type_definition` has no `name`
  field, so the class path dropped it. With an anonymous
  `typedef struct/enum {...} X;` the fields and enumerators went with it.
- A function behind an unknown leading macro (`API RetT name(...)`) was
  named after its return type, became a Method `RetT.name`, or was
  dropped, depending on the error-recovery shape.
- Every bodyless `struct X` / `enum X` (a parameter, a field type, a
  forward declaration) minted a Class/Enum node. In one file, or in a
  same-stem .h/.c pair, such a reference displaced the real definition.
- A `#define NAME` shared the QN of the function, type or enumerator
  called NAME, and the later line won: a namespace-rename macro replaced
  the typedef it renames, an `#else` stub macro replaced the function.
- Enumerators were named `<scope>.<Enum>.<CONST>`, although C and
  unscoped C++ enums put them in the enclosing scope.
- A definition repeated in several `#if` branches kept one span; the
  others left no trace.
- Braces split across `#if` branches lost whole functions, and a C
  keyword could surface as a function name (`if`).

Extraction:
- typedef names are definitions: `Type`, or `Class`/`Enum` with members
  for an anonymous `typedef struct/enum {...} X;`; an alias is dropped
  when the same file defines the tag of that name
- macro-prefixed functions get their real name in all three recovery
  shapes, consistently for defs, call scopes, parameters and the C LSP
- a bodyless tag is a reference and creates no node; a definition head
  left as loose ERROR tokens (`struct NAME {`) is recovered
- a C keyword is never a function name
- a first-branch projection (one branch per `#if` group, positions
  unchanged) is parsed when the raw parse lost structure wholesale, and
  only functions missing from the raw result are adopted from it
- macro-wrapped enumerator lists yield one constant per slot

Identity (semantic index version 4; an existing index is rebuilt once):
- a C-preprocessor macro's QN is `<module>.<NAME>#macro` (C, C++, CUDA,
  GLSL, Objective-C, ISPC); name and label are unchanged. Resolvers
  prefer a definition and use the macro only when no definition of that
  name is visible (LSP target lookup, trace_path, get_code_snippet)
- enumerators of C enums and of unscoped C++/Objective-C enums are
  `<enclosing scope>.<CONST>`; `enum class` stays nested; `parent_class`
  names the enum
- a node whose file defines its QN more than once under one label
  carries `variants`, the line spans of all those definitions (`#if`
  twins, per-platform macro redefinitions, C++ overloads)

Full-text search indexes a macro's QN without the fence, so macro rows
score as before.

Measured on redis and curl against the parent commit: nodes 38,625 ->
40,484 and 28,180 -> 29,750; Type nodes 0 -> 573 and 0 -> 193; phantom
Class nodes from references 202 and 821 removed; functions lost to a
same-named macro 45 and 41 restored; curl functions with no node 11 ->
0; nodes of non-C-family files unchanged by QN. Of 174 audited Doxygen
references in xxhash, 174 now reach the right entity (64 before). Index
time is unchanged within noise (+2% pipeline time for 5% more nodes).

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Isolate the watcher-disabled runtime, use a deterministic Python responder
for the fuzz harness environment probe, and align the version metadata
contract with the current Scoop manifest.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…-enumerators

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>

# Conflicts:
#	internal/cbm/cbm.h
#	src/mcp/mcp.c
cppcheck 2.20 (the CI linter) reports variableScope for the count taken before the definitions walk, which only the C-family branch uses. The branch now takes the count, walks and drops the shadowed typedefs; the other languages walk as before. Behaviour unchanged: extraction and pipeline suites pass.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Conflicts: internal/cbm/extract_defs.c includes (union of both sides); src/pipeline/registry.c keeps main's lang-aware index_under_name call, and the fence comment combines both facts (Rust cfg twins are variants of one QN; a C macro's #macro QN never reaches the registry).
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Main (#2458) named the anonymous struct/union/enum of `typedef struct { ... } Name;` in extract_class_def from the typedef's declarator. This branch's extract_c_typedef already emits that aggregate as Name from the type_definition, so on the merged tree Name was emitted twice under one label: its members twice and a false variants entry. extract_c_typedef stays the one place that names it; the extract_class_def case is removed. Both typedef test sets pass on the result: #2458's extract_c_anonymous_typedef_aggregate_is_named_by_its_typedef and member-access tests, and this branch's C1 typedef/variant tests (extraction, pipeline, registry, graph_buffer, store_search, mcp: 1,336 passed, 4 skipped).

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Regression test for the merge with #2458: with its extract_class_def case restored, typedef struct { int count; } Tally; yields two Class defs named Tally (count_defs_named == 2, RED at test_extraction.c:1129); with the case removed it is one Class, one count field, one Shade enum, and no variants list (extraction 432 passed).

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Main's graph_buffer.c ("Definition variants") records a definition's
variant spans as [{"file_path","start_line","end_line"}], and the
test-impact engine reads that schema to find every TEST(...) branch of one
test. This branch's extractor wrote [{"start","end"}]. On the merge with
main, graph_buffer's same-QN merge found a non-empty variants list it could
not read and dropped every span, the definition's own included. The engine
then no longer found the second #if branch of TEST(alpha_uses_lib), so
test_impact_engine_variant_tests_are_one_test reported suite alpha
UNMAPPED, in CI on macOS and Windows and locally.

The extractor now writes the shared schema, with the file path
JSON-escaped. The C-preprocessor-only gate and the one-carrier rule are
unchanged, and the variants assertions use the shared format.

test_impact_engine, extraction, pipeline, graph_buffer and
conditional_variants: 857 passed.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…last-line rule

Merging main brought #2340's virtual_newline field in CBMStringInput. cbm_rescue_defs_from_projection still built its input with the two-field initializer, a missing-field-initializers error under -Werror. The projection keeps the raw source's length and line structure, so it now goes through cbm_parse_source like the raw parse and the C++ branch views: the same virtual newline, and the same edit that removes it again.

extraction pipeline graph_buffer store_search mcp registry edge_types_probe parse_coverage c_lsp test_impact_engine conditional_variants on the merge result: 2277 passed, 4 skipped.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant