Repository navigation
feat(encoding): bytes had to arrive whole before they could be parsed - #115
Conversation
parse_bytes needed the whole page. ByteParser takes it a piece at a time: it holds bytes back only until the HTML Standard's sniffing can decide (a byte order mark, cut over pieces or not; the transport encoding; a whole <meta> from the prescan; or 1,024 bytes), then decodes each piece with encoding_rs's Decoder into the chunked parser. While the encoding is a guess it keeps every byte, and a late <meta> naming another encoding means one more parse at finish, as parse_bytes does. The tests cut every html5lib encoding input, plain and behind UTF-8 and UTF-16 byte order marks, at every byte and at random, and the WPT multi-byte pages at 49,422 places, comparing the serialized tree, the encoding and the confidence with parse_bytes. The encoding bench gets a table of 4 KB pieces against the whole page.
Two runs of the encoding bench at load 2.2 and 2.5: 4 KB pieces with the encoding given run at 91 to 95% of parse_bytes on the whole page, and 83 to 91% when sniffing keeps the bytes for a possible reparse.
foothillsdigital
left a comment
There was a problem hiding this comment.
Reviewed 6d88ec9.
Sniffing. decided() follows the spec's order and never commits sooner than the whole-input sniff would justify:
- a BOM first, then waiting while the bytes so far could still start one (
EF BB, a loneFEorFF); - then the transport encoding;
- at 1,024 bytes or more, the same
sniffparse_bytesuses; - below that,
prescan::scanon the prefix, which only givesSomefor a<meta>read to its>. The scan runs left to right, so a prefix's answer is the whole input's, and a cut construct (an unclosed comment, a half-read attribute) just yields nothing yet.
UTF-16 guesses not being kept, new_decoder_with_bom_removal as parse_bytes decodes, and the decode loop capping each step at 1 MiB and looping while the output is full all read right.
An independent check. I wrote a throwaway property test, not committed, that builds 4,000 random documents from pieces chosen to stress the sniffer: <meta charset> in both forms, decoy metas inside comments, an unclosed <!--, BOMs, a <meta charset= cut before its value, Shift_JIS and UTF-8 multibyte text, CRs, and filler to push metas past byte 1,024. Each was fed in 5 random cuttings: 20,000 cases, 0 differing from parse_bytes in encoding, confidence or text.
Checks. parsec-html-encoding's tests pass (36), as does cargo test --workspace --release (286 tests, 35 groups, 0 failed).
One heads-up, not from this PR: with a local Rust 1.99.1, cargo clippy --workspace --all-targets -- -D warnings fails on main in parsec-css/src/values/convert.rs:922 (needless_range_loop, for i in 0..3 indexing out). CI's toolchain: stable will hit it once it picks that clippy up. I'm happy to send the one-line fix.
Approving.
Problem
parse_bytesneeded the whole page. Now that the core parser takes input in pieces (#6, #114), bytes off a socket still had to be collected before Parsec could find their encoding, decode them and start parsing.Change
ByteParserincrates/parsec-encoding:ByteParser::new(options),feed(&[u8])as the bytes arrive,finish() -> Result<Parsed, ParseError>. It gives whatparse_bytesgives for the same bytes joined, however they were cut.EF,BB,BFin three feeds). With no BOM possible, the transport encoding decides at once. Otherwise the prescan decides once it has read a whole<meta>to its>(so<meta charset=iso-8859-1cut before5>doesn't commit to the wrong one), or at 1,024 bytes, as the spec allows a browser to wait, or at the end of the input.Decoder(new_decoder_with_bom_removal, asparse_bytesdecodes), which carries a cut character over to the next piece, and on intoparsec::Parser.<meta>naming another encoding,finishdecodes and parses once more, asparse_bytesdoes. That's the memory cost, documented on the type: the input once more as bytes while the encoding is tentative; nothing beyond the decoder's state once a BOM or the transport encoding has decided it.feedandfinish, both parses together. The clock is read only with a budget, so WebAssembly still works.prescan's inner scan is nowpub(crate)so the streaming parser can tell a finished<meta>from a cut one.parsec-html-bench encodinggets a table: the transcoded pages throughparse_byteswhole againstByteParserin 4 KB pieces, with the encoding given and sniffed.Not in this PR: streaming from Python and WASM. The bindings don't use
parsec-html-encodingyet at all (Python readsbytesas UTF-8), so byte sniffing has to come to them first; that's more than a small addition here.No new dependencies.
Closes #25
Tests
crates/parsec-encoding/tests/streaming.rscomparesByteParserwithparse_byteson the same bytes: the serialized tree, the encoding and the confidence.encoding/(all four files, the scripted one too), as written and behind UTF-8, UTF-16LE and UTF-16BE byte order marks: cut in two at every byte, plus 100 random cuttings per input (fixed seed) into pieces of 1 to 300 bytes. The inputs without a BOM run again with a transport encoding. 369,396 cuts, 0 differ.<meta>and thousands of multi-byte characters), then the whole page cut at 40 evenly spaced places, fed in 4 KB pieces, and in random pieces of up to 1 KB, 4 KB and 64 KB. 49,422 cuts, 0 differ.<meta>before 1,024 bytes, no<meta>waiting for 1,024, a BOM cut in three, transport waiting only while a BOM is possible, a late<meta>reparsing under a budget, and empty input.parsec-html-bench encoding):ByteParserin 4 KB pieces with the encoding given runs at 91 to 95% ofparse_byteson the whole page (wikipedia-population 502 to 555 MB/s against 533 to 604; nytimes 1,284 to 1,451 against 1,381 to 1,597), and at 83 to 91% when sniffing keeps the bytes. Most of the gap is copying:parse_bytesparses a UTF-8 page where it lies.cargo test --workspace,cargo clippy --workspace --all-targets -- -D warnings,cargo fmt --all --check,cargo deny check: pass.