Repository navigation
perf: text full of character references parsed at half speed - #125
Conversation
Syncs Netweir 304eb4a (netweir/netweir#347). Character-reference names come from a hash table built at compile time instead of a binary search per prefix length, a stretch of text and its references reaches the tree builder as one run, and numeric references are read in one pass. Spans and errors keep their exact positions. In the final-boss batch the three giant pages that are one text node of references go from 0.58 s to 0.21 s (Lexbor 0.28), the giants category from 1.32 s to 0.85 s (Lexbor 1.22), and the whole 1,000 pages from 1.49 s to 1.02 s (Lexbor 1.94). The README's final-boss tables have the new medians.
foothillsdigital
left a comment
There was a problem hiding this comment.
Checked 7d8ba9d.
The sync. NETWEIR_COMMIT moves 7de7b48 → 304eb4a, which is just netweir#347. That commit touches three files of netweir-html, and with crate names normalised, Parsec's diff matches netweir's line for line in all three: tokenizer.rs (290 lines), tree.rs (21) and edit.rs (2).
A differential against main, since this rewrites the code path where a parser is easiest to get subtly wrong. I generated 30,000 seeded inputs heavy in references, in every context the tokenizer treats differently:
- data, quoted and unquoted attribute values, RCDATA (
title,textarea), script, and SVG text; - named references with and without
;, and prefixes of longer names (¬i/¬in,&x); - numerics at every edge (0, 0x80–0x9F, surrogates, nonchars, 0x10FFFF, 0x110000, 2^40, leading zeros);
- bare
&,&#,&#x;and the like; - one 20,000-reference text node, the #124 shape.
Each was parsed with errors and spans on, by upstream/main and by this PR. For each input I recorded the serialized tree, every error code with its offset, and every node's span/end_span and every attribute's attr_span. The two outputs are byte-identical over all 217,750 inputs. Embedded NULs split the file into more inputs than I generated, which only means more cases.
The rest. cargo clippy --workspace --all-targets -D warnings is clean, and cargo test --workspace passes (320, the html5lib tokenizer and tree-construction suites among them). I didn't re-run the final-boss timings, so the table is yours. Approving.
Problem
In the final-boss batch (#123), three giant pages that are one long text node of character references took Parsec 0.58 s and Lexbor 0.28 s (#124). That one shape gave Lexbor the giants category, and the slowest page.
Change
Syncs Netweir 304eb4a, netweir/netweir#347:
The README's final-boss tables have the new medians.
Closes #124
Tests
Final boss, medians of 3 runs, each started with the load under 2, Parsec 4d6fe82, M2 Max:
Parsec is ahead in every category and on every giant shape. No crashes or timeouts, and element counts match on all 1,000 pages.
cargo test --workspace: 319 test groups, 0 failed. Tokenizer 7,042 and tree construction 3,898 passed; chunked: 0 of 205,527 cuts differ.-D warningsis clean; both bindings build.