Skip to content

feat(bench): no bench fed the parsers a crawl's mess of bytes at scale - #123

Merged
jamesconwaydev merged 3 commits into
mainfrom
bench-final-boss
Nov 20, 2026
Merged

jamesconwaydev merged 3 commits into
mainfrom
bench-final-boss

Conversation

@jamesconwaydev

Copy link
Copy Markdown
Collaborator

Problem

Every number in bench/ was one clean page at a time, already decoded into a string. That says how fast a parser is, not whether it holds up when a crawl hands it 1,000 pages of anything: broken scraps, a Shift_JIS page whose <meta> says UTF-8, and the odd 30 MB page. Nothing measured that, and nothing in CI fed Parsec that kind of batch.

Change

cargo run --release --manifest-path bench/Cargo.toml -- boss (new bench/src/boss.rs):

  • A seeded batch. Seed 20261119; page i comes from the seed and i alone, so the same seed gives the same bytes in the same order (the run prints an FNV-1a fingerprint of every byte). Per 1,000: 900 tiny broken fragments, 46 real pages from bench/corpus as served, 44 real pages re-encoded with eleven quirks (Shift_JIS, windows-1251, GBK, EUC-KR, UTF-16LE/BE with a BOM, a lying <meta>, a BOM against the <meta>, an HTTP-style transport hint, undeclared UTF-8, invalid UTF-8 bytes), and 10 giants of 10 to 30 MB (real pages run together, one huge table, one text node of references).
  • Bytes in, as a crawler gets them. Parsec through parsec_encoding::parse_bytes with the transport hint. Lexbor through its own lxb_encoding_bom_sniff, transport label, lxb_html_encoding_prescan and decoders, then lxb_html_document_parse (UTF-8 and plain ASCII go in as they are). html5ever takes text, so encoding_rs decodes for it with Parsec's sniffing.
  • Each parser in its own process. The bench re-runs itself: a middle process builds the pages and pipes them to that parser's worker, and reads getrusage(RUSAGE_CHILDREN) peak RSS afterwards (bytes on macOS, KB on Linux). A watchdog times a page out at 10 s, and a new worker carries on after a timeout or a crash. The element counts on every page are checked against Parsec's.
  • Medians of three runs, each one waiting for the one-minute load to drop below 2 (for two hours at most), with the parsers going first in turn.
  • CI: the bench job runs boss --smoke: 300 pages, nothing over 2 MB, Parsec only. It fails on a crash, a timeout or a page without a document. It isn't a speed gate.
  • README section "The final boss". The wiki's Benchmarks page gets a section with the method, the seed, every quirk and the raw output (pushed after merge).

Results, median of three, M2 Max, load 1.68 to 1.98:

s per 1,000 pages Slowest page Peak RSS Crashes Timeouts
Parsec 1.49 0.258 s 512 MB 0 0
Lexbor 3.0.1 1.93 0.249 s 1,044 MB 0 0
html5ever 0.40.1 5.99 0.811 s 673 MB 0 0

Lexbor wins the giants, 1.21 s to 1.32, all of it on the three text nodes full of character references, where it's 2.1 times as fast as Parsec. Parsec is ahead in every other category. A second set of three runs after merging main came out within 2%.

Tests

  • boss --smoke passes locally with the corpus and without it (shop listings stand in for real pages).
  • Harness check: I forced a panic, an abort and an 11 s sleep into a test tool. Each showed up as a crash or timeout on the right page, and the run carried on from the next page. That code was then removed.
  • Same pages, same bytes: page hashes match across invocations, and every run includes a check that rebuilds one page.
  • cargo fmt, cargo clippy -- -D warnings and cargo deny pass for the bench and for the workspace. cargo test --workspace passes after merging main (including build: 0.1.0 had no crates to publish and no path to publish them #122).

Every timing so far was one clean page at a time, already decoded. A
crawler gets 1,000 pages of anything: broken fragments, pages in
Shift_JIS or UTF-16 with a <meta> that lies, and the odd 30 MB page.

`bench -- boss` builds that batch from a seed (900 fragments, 46 real
pages, 44 encoding quirks, 10 giants of 10 to 30 MB), hands every
parser the same bytes in the same order, each in its own process, and
reports seconds per 1,000 pages, the slowest page, peak RSS from
getrusage, crashes, 10-second timeouts and pages without a document.
Lexbor gets its own BOM sniff, prescan and decoders; html5ever gets
encoding_rs with Parsec's sniffing, since it takes text.

`boss --smoke` runs 300 pages with nothing over 2 MB through Parsec
alone, and the bench job in CI now runs it.
A section, "The final boss", with the batch, how each parser is fed,
and the medians of three runs: seconds per 1,000 pages, slowest page,
peak RSS, crashes and timeouts, and the split by category, including
the giants, where Lexbor is ahead on text full of references.

@foothillsdigital foothillsdigital left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked 52cd8fa.

The CI gate works. It's the part that runs on every PR, so I checked it fails when it should. I added if bytes.len() > 1_500_000 { panic!("injected") } before Parsec's parse_bytes, which hits the smoke run's giants. boss --smoke then printed the crashes and exited 1, through failed |= crashes + timeouts + refused + empty > 0. Without the injection it passes here (300 pages, 0 crashes, timeouts, refusals or empty documents, 27 MB peak).

It's deterministic. Two smoke runs gave the same fingerprint, d663124e0d9d8c1d.

The Lexbor FFI reads right:

  • HtmlEncoding reserves 128 bytes, 16-aligned, for a 64-byte lxb_html_encoding_t on 64-bit. It's initialised before the prescan and destroyed after.
  • The prescan's name is validated with lxb_encoding_data_prescan_validate before destroy, while head is still alive.
  • The order follows the HTML Standard's: BOM sniff, transport label through by_pre_name, prescan, then windows-1252. The BOM length is skipped before decoding.
  • elements() walks with first_child, next and parent, and stops at the document node, so it never leaves the tree.

cargo clippy --manifest-path bench/Cargo.toml --all-targets -D warnings is clean. I didn't run the full three-run comparison, since it waits for the load to drop and takes the giants to 30 MB. So the table's numbers are yours, not mine. Approving.

@jamesconwaydev
jamesconwaydev merged commit 4d6fe82 into main Nov 20, 2026
33 checks passed
@jamesconwaydev
jamesconwaydev deleted the bench-final-boss branch November 20, 2026 01:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants