Skip to content

fix(extraction): parse every file as if its last line were terminated (#2078) - #2340

Merged
DeusData merged 4 commits into
mainfrom
fix/issue-2078
Oct 7, 2026
Merged

DeusData merged 4 commits into
mainfrom
fix/issue-2078

Conversation

@DeusData

Copy link
Copy Markdown
Owner

A file whose last byte is not "\n" leaves the construct on its final line
unterminated for every grammar that treats the line ending as part of
that construct. #1610/#1746 already excused the mild form, a ZERO-WIDTH
MISSING newline at EOF. The severe form is a WIDTH-BEARING error that
cannot be excused. In tree-sitter-markdown an opening fence, a bare list
marker or an ATX heading as the last bytes costs the last line
(text\n + fence -> parse_partial 2-2) or the whole file (a lone fence,
- -> parse_unusable). An unterminated one-line heading was lost
silently: the root is ERROR, there is no Section and no flag. A Makefile
whose last recipe line is unterminated lost the recipe. The same bytes
plus one "\n" parse clean. The reporter confirmed this on 21 real files
(43 -> 23 flagged after appending newlines).

Supply the newline rather than excuse its absence. cbm_parse_source()
feeds the parser one virtual "\n" at offset source_len (through the
TSInput read callback, so the caller's buffer is never copied or
written) and then applies one ts_tree_edit that deletes that byte
again. tree-sitter's edit arithmetic clamps every node that reached into
it back to the real end, bytes and row/column points both. So def line
ranges, node text, call byte spans, parse-coverage ranges and the
retained tree all see only real bytes and real lines. Empty files and
files that already end in "\n" are parsed exactly as before.

Every whole-file parse goes through the helper: the extraction parse,
and the whole-file fallback re-parses in the C/C++, C#, Go, Java,
Kotlin, PHP, Python, Rust and TS LSP passes. That keeps a spilled
result (no retained tree) consistent with a cached one. Sub-range and
synthetic parses (preprocessed C, Rust wrappers, Kotlin patches,
injected Jinja/imports ranges) are unchanged.

The two #1610/#1746 guards that pinned "an unterminated Makefile recipe
is a width-bearing loss" now use a construct that is broken with or
without a newline (an unclosed Python tuple at EOF). The recipe is no
longer lost, so flagging it would be false. The new
makefile_unterminated_recipe_is_parsed_issue2078 test pins that.

Proof (this repo, vendored grammars excluded, before/after binaries):

  • unchanged tree, and the same tree with the final newline stripped from
    all 1,220 files: nodes, edges and parse_partial lists byte-identical
    before vs after;
  • EOF-class corpus (the repo plus each .md with a fence, list marker,
    heading or opening fence appended without a newline, plus an
    unterminated Makefile): parse_partial 122 -> 77, exactly the
    unchanged repo's list. The only graph deltas are the 45 dropped
    ::missed shadow File rows (+7 Folders, 52 CONTAINS edges) and 22
    Section docstrings that now include their real last line.

The other four constructs in #2078 (HTML bare &, bash heredoc +
redirect + pipe, bash $((10#$n)), markdown empty || cell) are
upstream grammar gaps. Reports for them are drafted separately.

Refs #2078 (the EOF class; the remaining triggers are upstream grammar gaps)

…#2078)

A file whose last byte is not "\n" leaves the construct on its final line
unterminated for every grammar that treats the line ending as part of
that construct. #1610/#1746 already excused the mild form, a ZERO-WIDTH
MISSING newline at EOF. The severe form is a WIDTH-BEARING error that
cannot be excused. In tree-sitter-markdown an opening fence, a bare list
marker or an ATX heading as the last bytes costs the last line
(`text\n` + fence -> parse_partial 2-2) or the whole file (a lone fence,
`- ` -> parse_unusable). An unterminated one-line heading was lost
silently: the root is ERROR, there is no Section and no flag. A Makefile
whose last recipe line is unterminated lost the recipe. The same bytes
plus one "\n" parse clean. The reporter confirmed this on 21 real files
(43 -> 23 flagged after appending newlines).

Supply the newline rather than excuse its absence. cbm_parse_source()
feeds the parser one virtual "\n" at offset source_len (through the
TSInput read callback, so the caller's buffer is never copied or
written) and then applies one ts_tree_edit that deletes that byte
again. tree-sitter's edit arithmetic clamps every node that reached into
it back to the real end, bytes and row/column points both. So def line
ranges, node text, call byte spans, parse-coverage ranges and the
retained tree all see only real bytes and real lines. Empty files and
files that already end in "\n" are parsed exactly as before.

Every whole-file parse goes through the helper: the extraction parse,
and the whole-file fallback re-parses in the C/C++, C#, Go, Java,
Kotlin, PHP, Python, Rust and TS LSP passes. That keeps a spilled
result (no retained tree) consistent with a cached one. Sub-range and
synthetic parses (preprocessed C, Rust wrappers, Kotlin patches,
injected Jinja/imports ranges) are unchanged.

The two #1610/#1746 guards that pinned "an unterminated Makefile recipe
is a width-bearing loss" now use a construct that is broken with or
without a newline (an unclosed Python tuple at EOF). The recipe is no
longer lost, so flagging it would be false. The new
makefile_unterminated_recipe_is_parsed_issue2078 test pins that.

Proof (this repo, vendored grammars excluded, before/after binaries):
- unchanged tree, and the same tree with the final newline stripped from
  all 1,220 files: nodes, edges and parse_partial lists byte-identical
  before vs after;
- EOF-class corpus (the repo plus each .md with a fence, list marker,
  heading or opening fence appended without a newline, plus an
  unterminated Makefile): parse_partial 122 -> 77, exactly the
  unchanged repo's list. The only graph deltas are the 45 dropped
  ::missed shadow File rows (+7 Folders, 52 CONTAINS edges) and 22
  Section docstrings that now include their real last line.

The other four constructs in #2078 (HTML bare `&`, bash heredoc +
redirect + pipe, bash `$((10#$n))`, markdown empty `||` cell) are
upstream grammar gaps. Reports for them are drafted separately.

Refs #2078 (the EOF class; the remaining triggers are upstream grammar gaps)

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…e rule

Merging main brought extract_cpp_branch_views (conditional variants), which built its CBMStringInput with the two-field initializer. With this branch's virtual_newline field that is a missing initializer, an error under GCC's -Wextra -Werror (the shadow job's product build). A branch view keeps the raw source's length and line structure, so it now goes through cbm_parse_source like the raw parse: the same virtual newline, and the same edit that removes it again.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Conflict in tests/test_parse_coverage.c: this branch's #2078 tests and main's #1967 line-count test (#2336) were both appended before the suite; both kept, every TEST still registered.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
@DeusData
DeusData merged commit ec47e40 into main Oct 7, 2026
34 of 36 checks passed
DeusData added a commit that referenced this pull request Oct 7, 2026
…last-line rule

Merging main brought #2340's virtual_newline field in CBMStringInput. cbm_rescue_defs_from_projection still built its input with the two-field initializer, a missing-field-initializers error under -Werror. The projection keeps the raw source's length and line structure, so it now goes through cbm_parse_source like the raw parse and the C++ branch views: the same virtual newline, and the same edit that removes it again.

extraction pipeline graph_buffer store_search mcp registry edge_types_probe parse_coverage c_lsp test_impact_engine conditional_variants on the merge result: 2277 passed, 4 skipped.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant