Repository navigation
feat(bench): no bench fed the parsers a crawl's mess of bytes at scale - #123
Conversation
Every timing so far was one clean page at a time, already decoded. A crawler gets 1,000 pages of anything: broken fragments, pages in Shift_JIS or UTF-16 with a <meta> that lies, and the odd 30 MB page. `bench -- boss` builds that batch from a seed (900 fragments, 46 real pages, 44 encoding quirks, 10 giants of 10 to 30 MB), hands every parser the same bytes in the same order, each in its own process, and reports seconds per 1,000 pages, the slowest page, peak RSS from getrusage, crashes, 10-second timeouts and pages without a document. Lexbor gets its own BOM sniff, prescan and decoders; html5ever gets encoding_rs with Parsec's sniffing, since it takes text. `boss --smoke` runs 300 pages with nothing over 2 MB through Parsec alone, and the bench job in CI now runs it.
A section, "The final boss", with the batch, how each parser is fed, and the medians of three runs: seconds per 1,000 pages, slowest page, peak RSS, crashes and timeouts, and the split by category, including the giants, where Lexbor is ahead on text full of references.
foothillsdigital
left a comment
There was a problem hiding this comment.
Checked 52cd8fa.
The CI gate works. It's the part that runs on every PR, so I checked it fails when it should. I added if bytes.len() > 1_500_000 { panic!("injected") } before Parsec's parse_bytes, which hits the smoke run's giants. boss --smoke then printed the crashes and exited 1, through failed |= crashes + timeouts + refused + empty > 0. Without the injection it passes here (300 pages, 0 crashes, timeouts, refusals or empty documents, 27 MB peak).
It's deterministic. Two smoke runs gave the same fingerprint, d663124e0d9d8c1d.
The Lexbor FFI reads right:
HtmlEncodingreserves 128 bytes, 16-aligned, for a 64-bytelxb_html_encoding_ton 64-bit. It's initialised before the prescan and destroyed after.- The prescan's name is validated with
lxb_encoding_data_prescan_validatebeforedestroy, whileheadis still alive. - The order follows the HTML Standard's: BOM sniff, transport label through
by_pre_name, prescan, then windows-1252. The BOM length is skipped before decoding. elements()walks withfirst_child,nextandparent, and stops at the document node, so it never leaves the tree.
cargo clippy --manifest-path bench/Cargo.toml --all-targets -D warnings is clean. I didn't run the full three-run comparison, since it waits for the load to drop and takes the giants to 30 MB. So the table's numbers are yours, not mine. Approving.
Problem
Every number in
bench/was one clean page at a time, already decoded into a string. That says how fast a parser is, not whether it holds up when a crawl hands it 1,000 pages of anything: broken scraps, a Shift_JIS page whose<meta>says UTF-8, and the odd 30 MB page. Nothing measured that, and nothing in CI fed Parsec that kind of batch.Change
cargo run --release --manifest-path bench/Cargo.toml -- boss(newbench/src/boss.rs):20261119; pageicomes from the seed andialone, so the same seed gives the same bytes in the same order (the run prints an FNV-1a fingerprint of every byte). Per 1,000: 900 tiny broken fragments, 46 real pages frombench/corpusas served, 44 real pages re-encoded with eleven quirks (Shift_JIS, windows-1251, GBK, EUC-KR, UTF-16LE/BE with a BOM, a lying<meta>, a BOM against the<meta>, an HTTP-style transport hint, undeclared UTF-8, invalid UTF-8 bytes), and 10 giants of 10 to 30 MB (real pages run together, one huge table, one text node of references).parsec_encoding::parse_byteswith the transport hint. Lexbor through its ownlxb_encoding_bom_sniff, transport label,lxb_html_encoding_prescanand decoders, thenlxb_html_document_parse(UTF-8 and plain ASCII go in as they are). html5ever takes text, so encoding_rs decodes for it with Parsec's sniffing.getrusage(RUSAGE_CHILDREN)peak RSS afterwards (bytes on macOS, KB on Linux). A watchdog times a page out at 10 s, and a new worker carries on after a timeout or a crash. The element counts on every page are checked against Parsec's.boss --smoke: 300 pages, nothing over 2 MB, Parsec only. It fails on a crash, a timeout or a page without a document. It isn't a speed gate.Results, median of three, M2 Max, load 1.68 to 1.98:
Lexbor wins the giants, 1.21 s to 1.32, all of it on the three text nodes full of character references, where it's 2.1 times as fast as Parsec. Parsec is ahead in every other category. A second set of three runs after merging main came out within 2%.
Tests
boss --smokepasses locally with the corpus and without it (shop listings stand in for real pages).cargo fmt,cargo clippy -- -D warningsandcargo denypass for the bench and for the workspace.cargo test --workspacepasses after merging main (including build: 0.1.0 had no crates to publish and no path to publish them #122).