Skip to content

perf: text full of character references parsed at half speed - #125

Merged
jamesconwaydev merged 1 commit into
mainfrom
sync-char-refs
Nov 20, 2026
Merged

jamesconwaydev merged 1 commit into
mainfrom
sync-char-refs

Conversation

@jamesconwaydev

Copy link
Copy Markdown
Collaborator

Problem

In the final-boss batch (#123), three giant pages that are one long text node of character references took Parsec 0.58 s and Lexbor 0.28 s (#124). That one shape gave Lexbor the giants category, and the slowest page.

Change

Syncs Netweir 304eb4a, netweir/netweir#347:

  • Names: reference names are looked up in a hash table of the entity list, built at compile time, instead of a binary search for every prefix length.
  • One run per stretch of text: text and its references reach the tree builder as one run, so a run is no longer split at every reference.
  • Numeric references: read in one pass.
  • Positions: with spans or errors on, the tokenizer keeps one position per reference, so offsets are exact as before.

The README's final-boss tables have the new medians.

Closes #124

Tests

Final boss, medians of 3 runs, each started with the load under 2, Parsec 4d6fe82, M2 Max:

Parsec before Parsec now Lexbor 3.0.1
Seconds per 1,000 pages 1.49 1.02 1.94
Slowest page 0.258 s 0.163 s 0.247 s
Giants 1.317 s 0.854 s 1.221 s
One text node of references (3 pages) 0.58 s 0.211 s 0.280 s
Peak RSS 512 MB 516 MB 952 MB

Parsec is ahead in every category and on every giant shape. No crashes or timeouts, and element counts match on all 1,000 pages.

  • cargo test --workspace: 319 test groups, 0 failed. Tokenizer 7,042 and tree construction 3,898 passed; chunked: 0 of 205,527 cuts differ.
  • Clippy -D warnings is clean; both bindings build.

Syncs Netweir 304eb4a (netweir/netweir#347). Character-reference names
come from a hash table built at compile time instead of a binary
search per prefix length, a stretch of text and its references reaches
the tree builder as one run, and numeric references are read in one
pass. Spans and errors keep their exact positions.

In the final-boss batch the three giant pages that are one text node of
references go from 0.58 s to 0.21 s (Lexbor 0.28), the giants category
from 1.32 s to 0.85 s (Lexbor 1.22), and the whole 1,000 pages from
1.49 s to 1.02 s (Lexbor 1.94). The README's final-boss tables have
the new medians.

@foothillsdigital foothillsdigital left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked 7d8ba9d.

The sync. NETWEIR_COMMIT moves 7de7b48 → 304eb4a, which is just netweir#347. That commit touches three files of netweir-html, and with crate names normalised, Parsec's diff matches netweir's line for line in all three: tokenizer.rs (290 lines), tree.rs (21) and edit.rs (2).

A differential against main, since this rewrites the code path where a parser is easiest to get subtly wrong. I generated 30,000 seeded inputs heavy in references, in every context the tokenizer treats differently:

  • data, quoted and unquoted attribute values, RCDATA (title, textarea), script, and SVG text;
  • named references with and without ;, and prefixes of longer names (&noti/&notin, &ampx);
  • numerics at every edge (0, 0x80–0x9F, surrogates, nonchars, 0x10FFFF, 0x110000, 2^40, leading zeros);
  • bare &, &#, &#x; and the like;
  • one 20,000-reference text node, the #124 shape.

Each was parsed with errors and spans on, by upstream/main and by this PR. For each input I recorded the serialized tree, every error code with its offset, and every node's span/end_span and every attribute's attr_span. The two outputs are byte-identical over all 217,750 inputs. Embedded NULs split the file into more inputs than I generated, which only means more cases.

The rest. cargo clippy --workspace --all-targets -D warnings is clean, and cargo test --workspace passes (320, the html5lib tokenizer and tree-construction suites among them). I didn't re-run the final-boss timings, so the table is yours. Approving.

@jamesconwaydev
jamesconwaydev merged commit e07add7 into main Nov 20, 2026
33 checks passed
@jamesconwaydev
jamesconwaydev deleted the sync-char-refs branch November 20, 2026 07:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A long text node full of character references parses 2.1x slower than in Lexbor

2 participants