Repository navigation
fix(extraction): parse every file as if its last line were terminated (#2078) - #2340
Merged
Merged
Conversation
This was referenced Sep 25, 2026
DeusData
force-pushed
the
fix/issue-2078
branch
3 times, most recently
from
October 1, 2026 19:00
17692f7 to
0c639f2
Compare
…#2078) A file whose last byte is not "\n" leaves the construct on its final line unterminated for every grammar that treats the line ending as part of that construct. #1610/#1746 already excused the mild form, a ZERO-WIDTH MISSING newline at EOF. The severe form is a WIDTH-BEARING error that cannot be excused. In tree-sitter-markdown an opening fence, a bare list marker or an ATX heading as the last bytes costs the last line (`text\n` + fence -> parse_partial 2-2) or the whole file (a lone fence, `- ` -> parse_unusable). An unterminated one-line heading was lost silently: the root is ERROR, there is no Section and no flag. A Makefile whose last recipe line is unterminated lost the recipe. The same bytes plus one "\n" parse clean. The reporter confirmed this on 21 real files (43 -> 23 flagged after appending newlines). Supply the newline rather than excuse its absence. cbm_parse_source() feeds the parser one virtual "\n" at offset source_len (through the TSInput read callback, so the caller's buffer is never copied or written) and then applies one ts_tree_edit that deletes that byte again. tree-sitter's edit arithmetic clamps every node that reached into it back to the real end, bytes and row/column points both. So def line ranges, node text, call byte spans, parse-coverage ranges and the retained tree all see only real bytes and real lines. Empty files and files that already end in "\n" are parsed exactly as before. Every whole-file parse goes through the helper: the extraction parse, and the whole-file fallback re-parses in the C/C++, C#, Go, Java, Kotlin, PHP, Python, Rust and TS LSP passes. That keeps a spilled result (no retained tree) consistent with a cached one. Sub-range and synthetic parses (preprocessed C, Rust wrappers, Kotlin patches, injected Jinja/imports ranges) are unchanged. The two #1610/#1746 guards that pinned "an unterminated Makefile recipe is a width-bearing loss" now use a construct that is broken with or without a newline (an unclosed Python tuple at EOF). The recipe is no longer lost, so flagging it would be false. The new makefile_unterminated_recipe_is_parsed_issue2078 test pins that. Proof (this repo, vendored grammars excluded, before/after binaries): - unchanged tree, and the same tree with the final newline stripped from all 1,220 files: nodes, edges and parse_partial lists byte-identical before vs after; - EOF-class corpus (the repo plus each .md with a fence, list marker, heading or opening fence appended without a newline, plus an unterminated Makefile): parse_partial 122 -> 77, exactly the unchanged repo's list. The only graph deltas are the 45 dropped ::missed shadow File rows (+7 Folders, 52 CONTAINS edges) and 22 Section docstrings that now include their real last line. The other four constructs in #2078 (HTML bare `&`, bash heredoc + redirect + pipe, bash `$((10#$n))`, markdown empty `||` cell) are upstream grammar gaps. Reports for them are drafted separately. Refs #2078 (the EOF class; the remaining triggers are upstream grammar gaps) Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
DeusData
force-pushed
the
fix/issue-2078
branch
from
October 4, 2026 16:48
0c639f2 to
fa258d3
Compare
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…e rule Merging main brought extract_cpp_branch_views (conditional variants), which built its CBMStringInput with the two-field initializer. With this branch's virtual_newline field that is a missing initializer, an error under GCC's -Wextra -Werror (the shadow job's product build). A branch view keeps the raw source's length and line structure, so it now goes through cbm_parse_source like the raw parse: the same virtual newline, and the same edit that removes it again. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
DeusData
added a commit
that referenced
this pull request
Oct 7, 2026
…last-line rule Merging main brought #2340's virtual_newline field in CBMStringInput. cbm_rescue_defs_from_projection still built its input with the two-field initializer, a missing-field-initializers error under -Werror. The projection keeps the raw source's length and line structure, so it now goes through cbm_parse_source like the raw parse and the C++ branch views: the same virtual newline, and the same edit that removes it again. extraction pipeline graph_buffer store_search mcp registry edge_types_probe parse_coverage c_lsp test_impact_engine conditional_variants on the merge result: 2277 passed, 4 skipped. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A file whose last byte is not "\n" leaves the construct on its final line
unterminated for every grammar that treats the line ending as part of
that construct. #1610/#1746 already excused the mild form, a ZERO-WIDTH
MISSING newline at EOF. The severe form is a WIDTH-BEARING error that
cannot be excused. In tree-sitter-markdown an opening fence, a bare list
marker or an ATX heading as the last bytes costs the last line
(
text\n+ fence -> parse_partial 2-2) or the whole file (a lone fence,--> parse_unusable). An unterminated one-line heading was lostsilently: the root is ERROR, there is no Section and no flag. A Makefile
whose last recipe line is unterminated lost the recipe. The same bytes
plus one "\n" parse clean. The reporter confirmed this on 21 real files
(43 -> 23 flagged after appending newlines).
Supply the newline rather than excuse its absence. cbm_parse_source()
feeds the parser one virtual "\n" at offset source_len (through the
TSInput read callback, so the caller's buffer is never copied or
written) and then applies one ts_tree_edit that deletes that byte
again. tree-sitter's edit arithmetic clamps every node that reached into
it back to the real end, bytes and row/column points both. So def line
ranges, node text, call byte spans, parse-coverage ranges and the
retained tree all see only real bytes and real lines. Empty files and
files that already end in "\n" are parsed exactly as before.
Every whole-file parse goes through the helper: the extraction parse,
and the whole-file fallback re-parses in the C/C++, C#, Go, Java,
Kotlin, PHP, Python, Rust and TS LSP passes. That keeps a spilled
result (no retained tree) consistent with a cached one. Sub-range and
synthetic parses (preprocessed C, Rust wrappers, Kotlin patches,
injected Jinja/imports ranges) are unchanged.
The two #1610/#1746 guards that pinned "an unterminated Makefile recipe
is a width-bearing loss" now use a construct that is broken with or
without a newline (an unclosed Python tuple at EOF). The recipe is no
longer lost, so flagging it would be false. The new
makefile_unterminated_recipe_is_parsed_issue2078 test pins that.
Proof (this repo, vendored grammars excluded, before/after binaries):
all 1,220 files: nodes, edges and parse_partial lists byte-identical
before vs after;
heading or opening fence appended without a newline, plus an
unterminated Makefile): parse_partial 122 -> 77, exactly the
unchanged repo's list. The only graph deltas are the 45 dropped
::missed shadow File rows (+7 Folders, 52 CONTAINS edges) and 22
Section docstrings that now include their real last line.
The other four constructs in #2078 (HTML bare
&, bash heredoc +redirect + pipe, bash
$((10#$n)), markdown empty||cell) areupstream grammar gaps. Reports for them are drafted separately.
Refs #2078 (the EOF class; the remaining triggers are upstream grammar gaps)