Skip to content

feat(encoding): bytes had to arrive whole before they could be parsed - #115

Merged
jamesconwaydev merged 3 commits into
mainfrom
encoding-streaming
Nov 19, 2026
Merged

jamesconwaydev merged 3 commits into
mainfrom
encoding-streaming

Conversation

@jamesconwaydev

@jamesconwaydev jamesconwaydev commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

parse_bytes needed the whole page. Now that the core parser takes input in pieces (#6, #114), bytes off a socket still had to be collected before Parsec could find their encoding, decode them and start parsing.

Change

ByteParser in crates/parsec-encoding: ByteParser::new(options), feed(&[u8]) as the bytes arrive, finish() -> Result<Parsed, ParseError>. It gives what parse_bytes gives for the same bytes joined, however they were cut.

  • Sniffing, for a stream. It holds the first bytes back only until the HTML Standard's sniffing can decide what it would decide for the whole input. A byte order mark decides on its last byte, even cut over pieces (EF, BB, BF in three feeds). With no BOM possible, the transport encoding decides at once. Otherwise the prescan decides once it has read a whole <meta> to its > (so <meta charset=iso-8859-1 cut before 5> doesn't commit to the wrong one), or at 1,024 bytes, as the spec allows a browser to wait, or at the end of the input.
  • Decoding. Each piece goes through encoding_rs's Decoder (new_decoder_with_bom_removal, as parse_bytes decodes), which carries a cut character over to the next piece, and on into parsec::Parser.
  • Changing the encoding. While the encoding is a guess, every byte is kept. If the parser meets a <meta> naming another encoding, finish decodes and parses once more, as parse_bytes does. That's the memory cost, documented on the type: the input once more as bytes while the encoding is tentative; nothing beyond the decoder's state once a BOM or the transport encoding has decided it.
  • A budget counts the time in feed and finish, both parses together. The clock is read only with a budget, so WebAssembly still works.
  • prescan's inner scan is now pub(crate) so the streaming parser can tell a finished <meta> from a cut one.
  • parsec-html-bench encoding gets a table: the transcoded pages through parse_bytes whole against ByteParser in 4 KB pieces, with the encoding given and sniffed.
  • README and ROADMAP: The WHATWG Encoding Standard #25 done, so M5 is complete.

Not in this PR: streaming from Python and WASM. The bindings don't use parsec-html-encoding yet at all (Python reads bytes as UTF-8), so byte sniffing has to come to them first; that's more than a small addition here.

No new dependencies.

Closes #25

Tests

crates/parsec-encoding/tests/streaming.rs compares ByteParser with parse_bytes on the same bytes: the serialized tree, the encoding and the confidence.

  • html5lib-tests encoding/ (all four files, the scripted one too), as written and behind UTF-8, UTF-16LE and UTF-16BE byte order marks: cut in two at every byte, plus 100 random cuttings per input (fixed seed) into pieces of 1 to 300 bytes. The inputs without a BOM run again with a transport encoding. 369,396 cuts, 0 differ.
  • web-platform-tests legacy multi-byte pages (Shift_JIS, EUC-JP, ISO-2022-JP, EUC-KR, Big5 and Big5 extra). Cutting each whole page at every byte would be 3.3 million parses of up to 850 KB, hours in a debug build, so it's sampled: each page's first 8 KB as a page of its own cut at every byte (the <meta> and thousands of multi-byte characters), then the whole page cut at 40 evenly spaced places, fed in 4 KB pieces, and in random pieces of up to 1 KB, 4 KB and 64 KB. 49,422 cuts, 0 differ.
  • Unit tests for when it commits: a whole <meta> before 1,024 bytes, no <meta> waiting for 1,024, a BOM cut in three, transport waiting only while a BOM is possible, a late <meta> reparsing under a budget, and empty input.
  • Mutation check: dropping the wait for a cut BOM makes 10,287 cuts differ, so the comparison catches it.
  • Bench, Apple M2 Max, two runs at load 2.2 and 2.5 (parsec-html-bench encoding): ByteParser in 4 KB pieces with the encoding given runs at 91 to 95% of parse_bytes on the whole page (wikipedia-population 502 to 555 MB/s against 533 to 604; nytimes 1,284 to 1,451 against 1,381 to 1,597), and at 83 to 91% when sniffing keeps the bytes. Most of the gap is copying: parse_bytes parses a UTF-8 page where it lies.
  • cargo test --workspace, cargo clippy --workspace --all-targets -- -D warnings, cargo fmt --all --check, cargo deny check: pass.

parse_bytes needed the whole page. ByteParser takes it a piece at a
time: it holds bytes back only until the HTML Standard's sniffing can
decide (a byte order mark, cut over pieces or not; the transport
encoding; a whole <meta> from the prescan; or 1,024 bytes), then
decodes each piece with encoding_rs's Decoder into the chunked
parser. While the encoding is a guess it keeps every byte, and a late
<meta> naming another encoding means one more parse at finish, as
parse_bytes does.

The tests cut every html5lib encoding input, plain and behind UTF-8
and UTF-16 byte order marks, at every byte and at random, and the WPT
multi-byte pages at 49,422 places, comparing the serialized tree, the
encoding and the confidence with parse_bytes. The encoding bench gets
a table of 4 KB pieces against the whole page.
Two runs of the encoding bench at load 2.2 and 2.5: 4 KB pieces with
the encoding given run at 91 to 95% of parse_bytes on the whole page,
and 83 to 91% when sniffing keeps the bytes for a possible reparse.

@foothillsdigital foothillsdigital left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 6d88ec9.

Sniffing. decided() follows the spec's order and never commits sooner than the whole-input sniff would justify:

  • a BOM first, then waiting while the bytes so far could still start one (EF BB, a lone FE or FF);
  • then the transport encoding;
  • at 1,024 bytes or more, the same sniff parse_bytes uses;
  • below that, prescan::scan on the prefix, which only gives Some for a <meta> read to its >. The scan runs left to right, so a prefix's answer is the whole input's, and a cut construct (an unclosed comment, a half-read attribute) just yields nothing yet.

UTF-16 guesses not being kept, new_decoder_with_bom_removal as parse_bytes decodes, and the decode loop capping each step at 1 MiB and looping while the output is full all read right.

An independent check. I wrote a throwaway property test, not committed, that builds 4,000 random documents from pieces chosen to stress the sniffer: <meta charset> in both forms, decoy metas inside comments, an unclosed <!--, BOMs, a <meta charset= cut before its value, Shift_JIS and UTF-8 multibyte text, CRs, and filler to push metas past byte 1,024. Each was fed in 5 random cuttings: 20,000 cases, 0 differing from parse_bytes in encoding, confidence or text.

Checks. parsec-html-encoding's tests pass (36), as does cargo test --workspace --release (286 tests, 35 groups, 0 failed).

One heads-up, not from this PR: with a local Rust 1.99.1, cargo clippy --workspace --all-targets -- -D warnings fails on main in parsec-css/src/values/convert.rs:922 (needless_range_loop, for i in 0..3 indexing out). CI's toolchain: stable will hit it once it picks that clippy up. I'm happy to send the one-line fix.

Approving.

@jamesconwaydev
jamesconwaydev merged commit 840a1c4 into main Nov 19, 2026
33 checks passed
@jamesconwaydev
jamesconwaydev deleted the encoding-streaming branch November 19, 2026 19:18
@jamesconwaydev jamesconwaydev mentioned this pull request Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The WHATWG Encoding Standard

2 participants