Skip to content

perf(lexer): bound repeated failed brace and $[ lookahead scans - #28

Open
jecruz wants to merge 2 commits into
webpro-nl:mainfrom
jecruz:perf/bound-failed-lookahead-scans
Open

jecruz wants to merge 2 commits into
webpro-nl:mainfrom
jecruz:perf/bound-failed-lookahead-scans

Conversation

@jecruz

@jecruz jecruz commented Nov 5, 2026

Copy link
Copy Markdown

Fixes #27

Two commits, each with tests.

Brace expansion. scanBraceExpansion now keeps one cache per source window (keyed on source, window end and nesting depth). Every brace opener visited by a scan gets an entry, including a valid expansion nested inside an unmatched outer brace, so n{{a,b} still yields the inner {a,b} expansion. Cached results are returned before any scanning. The scanner position, error list and unbalanced state are restored exactly as before.

Legacy $[. findClosingArithmeticBracket keeps a per-window set of opener positions whose search is known to have failed. findClosingShellDelimiter takes an optional set and, when a search fails, adds every [ that was still open at that point. Only failures are recorded. Successful inner searches are not cached because they must keep running to retain their existing error and position handling, and readDollar discards the errors of a failed lookahead. The window is re-assigned after the search because the scan can re-enter through nested lexers.

Why failures only for $[: a failed search at an outer opener implies the same text to the same end fails for every opener still open inside it, while a success depends on how errors and positions are handled inside the scan. Caching only failures leaves all successful paths byte-for-byte as before.

Constraints kept: no input length cap, no timeout, no grammar change, a single active cache window per construct, and at most one key per input position. A change of source, window end or nesting depth discards the previous cache. findClosingBracket and subscript recovery are untouched.

Tests (test/failed-lookahead-bounds.test.ts) count String.prototype.charCodeAt reads for "n" + "{".repeat(size - 1), "echo " + "$[".repeat(k) and "echo " + "$[".repeat(k) + "]" at 1, 4, 16 and 64 KiB and assert reads <= 64 * length. Correctness tests cover n{{a,b} (inner expansion kept), n{{{a,b}, echo $[$[1]], echo $[1+$[2]]x and echo $[$[1] (literal $[ followed by $[1]).

Validation measured on current main (61b82c2) and this branch, Node v24.22.2:

  • New tests on main without the change: 12 of 16 fail (all size cases), the 4 correctness tests pass. After the first commit only the 4 brace cases flip to pass; after the second commit all 16 pass.
  • Reads after the change (N = length): n{{{... 3,075 / 12,291 / 49,155 / 196,611 at 1 / 4 / 16 / 64 KiB (3.0 * N); $[ runs 6,120 / 24,552 / 98,280 / 393,192 (about 6 * N); $[ runs with a trailing ] 6,121 / 24,553 / 98,281 / 393,193.
  • Full suite (npm test): 2075 tests, 2075 pass, 0 fail (2059 on main plus the 16 new). npm run test:types passes, oxfmt --check is clean for src and test, oxlint reports the same pre-existing warnings as main.
  • Differential test, old build vs new build, comparing full public AST JSON including node types, errors and positions: 9,379 rows (1,129 hand-written probes, 4,096 seeded fuzz cases, 33 scaling cases up to 16 KiB, 16 legacy $[ controls, 4,096 seeded legacy $[ fuzz cases, 9 extra scaling cases up to 16 KiB). Result: 9,379 identical, 0 differences, 0 parser exceptions. For one 16 KiB deeply nested arithmetic input native JSON.stringify overflows the stack on both builds, so an iterative serializer was used there; it matched native JSON.stringify on all other comparisons.

Not tested: complexity of the rest of the grammar (this bounds the two input families above, it does not prove linear parsing overall; the $( path was only observed, not shown to be uncapped quadratic); inputs above 64 KiB in the committed tests; other runtimes than Node 24 for this branch; wall-clock timing (the tests count reads on purpose); the playground and bench builds.

Overlap: #20 ("keep unclosed subscript recovery linear") also edits findClosingShellDelimiter, adding a trailing optional bracketMatches parameter and bookkeeping next to the same delimiters.push and delimiters.pop sites that the second commit extends with a different trailing optional parameter. The two changes are independent in behaviour (that PR caches inside findClosingBracket, this one inside findClosingArithmeticBracket, which calls the scanner with braces = false) but will conflict textually; whichever lands second needs a small rebase. Neither PR touches scanBraceExpansion.

An unmatched run of '{' in a word made scanBraceExpansion scan to the end of
the window once per opener. Record the result for every opener visited by a
scan, including valid expansions nested in an unmatched outer brace, so each
opener is scanned at most once per source window.
readDollar treats a $[ without a matching ] as a literal and the word reader
retries at the next $[, scanning to the end of the window each time. Record
failed searches only, for every still-open bracket of the failed scan, so a
successful inner search keeps its existing error and position handling.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Quadratic parsing of unmatched { runs and unmatched legacy $[ runs

1 participant