Skip to content

feat(bindings): the bindings couldn't sniff a page's encoding from bytes - #118

Merged
jamesconwaydev merged 7 commits into
mainfrom
bindings-bytes
Nov 19, 2026
Merged

jamesconwaydev merged 7 commits into
mainfrom
bindings-bytes

Conversation

@jamesconwaydev

@jamesconwaydev jamesconwaydev commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

parsec-html-encoding sniffs and decodes bytes (parse_bytes, #79) and streams them (ByteParser, #115), but no binding used it. Python read bytes as UTF-8, with anything else turned into U+FFFD, WebAssembly took only strings, and C callers had to decode to UTF-8 themselves. And the offsets that #108 gave each binding only knew UTF-8.

Change

Python. HTMLParser(data) decodes bytes as a browser does: byte order mark, then transport_encoding, then a <meta>, then default_encoding (windows-1252). The names are html5lib's, and so is the TypeError for either one with a str. parser.encoding gives the WHATWG name and parser.encoding_confidence "certain" or "tentative"; encoding_for_locale("ja") gives a language's fallback. StreamParser(...) with feed(data) and close() (lxml's feed-parser and html.parser names) returns the same HTMLParser the whole bytes give. A small behaviour change: undeclared bytes that aren't valid UTF-8 used to get U+FFFD per bad byte and now decode as windows-1252, as in a browser. Undeclared UTF-8 still decodes as UTF-8 (see autodetection below).

WebAssembly. parseBytes(Uint8Array, { transportEncoding, defaultEncoding, ...ParseOptions }) and new ByteParser(options) with feed() / finish(); Document.characterSet (as the DOM's; "UTF-8" for a string) and encodingConfidence; ByteParseOptions in the types.

C. parsec_parse_bytes(bytes, len, transport, transport_len, flags, &doc, &encoding), parsec_document_encoding(doc), and a parsec_byte_parser handle with _new, _feed, _finish and _free. All additions; header regenerated with cbindgen 0.29.4.

Offsets. In all three, spans and error offsets for bytes are byte offsets into those bytes in their own encoding. That's a new parsec_encoding::Offsets, built only when positions or errors are on: it decodes the bytes again noting where each character began (one pass for UTF-8 and single-byte encodings, byte by byte for the others: 10.8 ms against a 4.5 ms parse on a 773 KB Shift_JIS page) and keeps only the points where the two counts stop moving together.

Everywhere, a transport label that names no encoding is passed over, as the HTML Standard says (servers send charset=utf8mb4); an unknown default encoding is an error.

Autodetection. sniff now takes the HTML Standard's autodetection step before the fallback, narrowed to what Firefox's chardetng and Chrome's CED both do: with no BOM, transport encoding or <meta> in the first 1,024 bytes, bytes with something beyond ASCII that are all valid UTF-8 are UTF-8, tentative (a late <meta> still wins). parse_bytes checks the whole input. ByteParser guesses from the bytes it holds back (up to the existing 1,024-byte wait; a character cut off at the end counts as valid), and at finish re-checks the whole input, which it keeps anyway while the guess is tentative; if that disagrees (UTF-8 starting after 1,024 bytes of ASCII, or a bad byte later on) it parses as parse_bytes would. So streaming still equals the whole parse for every cut, without holding bytes back any longer. No html5lib case expected windows-1252 for valid non-ASCII UTF-8; all 82 still pass.

Also fixed: parse_bytes read the clock with no budget, which panics in WebAssembly.

Size. The WebAssembly goes from 870,581 to 1,071,302 bytes, 362,444 to 504,465 gzipped (+142 KB gzipped, encoding_rs's CJK tables). That's under the 150 KB gzipped that would have meant a separate entry point, so it's in the one build.

Tests

  • html5lib encoding/ cases, all 82 that don't need a script engine (1 scripted skipped), through each binding's whole and streamed entry points, the streamed one in pieces of 1 to 7 bytes: Python (pytest), WebAssembly (wasm-bindgen in Node and Firefox) and C (tests/capi.c, built with the folder as HTML5LIB_ENCODING by tests/c.rs and the sanitizer job). 82 of 82 in each.
  • A Shift_JIS page with positions and errors, whole and streamed, its ranges checked against the bytes: pytest, the npm package test and capi.c.
  • crates/parsec-encoding/tests/offsets.rs: 187,952 element and attribute spans on 334 pages (html5lib inputs as written, behind a UTF-8 BOM and widened to UTF-16 LE and BE, plus the WPT legacy multi-byte pages) map back to bytes that decode to what each span covers.
  • Locally: cargo test --workspace, clippy and cargo deny for the workspace and both bindings, pytest (33 passed, 1 skipped as before), wasm-pack test --node and --firefox (10 each), node --test tests/package.test.mjs (11), and capi.c under ASan and UBSan on macOS.

Closes #116

The parser's offsets count UTF-8 bytes into the decoded page, so a
caller holding the Shift_JIS or UTF-16 bytes had no way to find where
a tag was in them. Offsets::new(bytes, encoding) decodes them again,
noting where each character's bytes began, and keeps only the places
where the two stop moving in step; to_source and to_decoded convert.
UTF-8 and single-byte encodings take one pass, the rest go through the
decoder a byte at a time.

tests/offsets.rs maps 187,952 element and attribute spans on 334
pages (the html5lib encoding inputs, behind BOMs and widened to
UTF-16, and the WPT legacy multi-byte pages) back to their bytes.

Also, parse_bytes read the clock even with no budget, which panics in
WebAssembly; it now reads it only for a budget, as ByteParser does.
HTMLParser(data) now decodes bytes as a browser does, through
parsec_encoding: byte order mark, then transport_encoding, then a
<meta>, then default_encoding (windows-1252). The keyword names are
html5lib's; a transport label that names no encoding is passed over,
as the HTML Standard says, and either keyword with a str raises
TypeError, as html5lib's does. parser.encoding gives the WHATWG name
and parser.encoding_confidence "certain" or "tentative";
encoding_for_locale(tag) gives the fallback for a language.

StreamParser takes the same keywords and feed(data) / close(), the
names of lxml's feed parser and html.parser, with close() returning
the HTMLParser. Both let go of the GIL while they work.

Positions and error offsets stay byte offsets into the bytes given:
the units map now uses parsec_encoding::Offsets for any encoding, in
place of the UTF-8-only repair list.

This changes behaviour: a UTF-8 page that doesn't say so now decodes
as windows-1252. The README says so, and how to keep UTF-8.

Tests: the 82 html5lib encoding cases through HTMLParser and through
StreamParser in pieces of 1 to 7 bytes, and a Shift_JIS page whose
ranges, attributes, errors and line_col are checked against its
bytes, whole and streamed. CI checks out html5lib-tests for them.
parseBytes(Uint8Array, { transportEncoding, defaultEncoding, ...})
decodes a page as a browser does and parses it; new ByteParser(options)
with feed() and finish() does the same as the bytes arrive. The
document gains characterSet, as the DOM's (UTF-8 for a string), and
encodingConfidence. ByteParseOptions extends ParseOptions in the
TypeScript types. Spans and error offsets for bytes are byte offsets
into the Uint8Array, in its own encoding.

The WebAssembly grows by 199 KB, 141 KB gzipped, to 1,070 KB and
504 KB gzipped: encoding_rs's decoding tables, mostly CJK. That's
under the 150 KB gzipped that would have meant a separate entry
point, so it's in the one build.

Tests: wasm-bindgen tests (Node and Firefox) run the 82 html5lib
encoding cases whole and in pieces of 1 to 7 bytes; the package test
checks the options, the streaming parser and Shift_JIS spans.
parsec_parse_bytes(bytes, len, transport, transport_len, flags, &doc,
&encoding) sniffs and decodes a page as a browser does, and
parsec_document_encoding(doc) names the encoding (NULL for a document
parsed from text). parsec_byte_parser_new, _feed, _finish and _free do
the same for bytes as they arrive. All additions; the header is
regenerated with cbindgen 0.29.4.

For a document from bytes, the span calls, parsec_line_col and
parsec_errors_get count byte offsets into those bytes, in their own
encoding.

tests/capi.c checks a Shift_JIS page's spans and errors, whole and in
3-byte pieces, and the error paths; given the html5lib-tests
encoding/ folder (tests/c.rs and the sanitizer job pass it) it runs
the 82 cases through both entry points.
README: the C API, Python and WebAssembly bullets and comparison rows,
the Python behaviour change, and the WebAssembly's new size. ROADMAP:
the bindings' bytes under M4.
Comment thread crates/parsec-capi/tests/capi.c Fixed
Comment thread crates/parsec-capi/tests/capi.c Fixed
CodeQL flagged the html5lib folder read from argv and a buffer index
used before its range check. The folder now comes in at build time as
HTML5LIB_ENCODING, defined by tests/c.rs and the sanitizer job, and
the label copy checks its bound first.
With no BOM, transport encoding or <meta>, sniffing went straight to
the fallback, so the most common bytes people pass in, UTF-8 that
doesn't say so, came out garbled. The HTML Standard allows an
autodetection step before the fallback, and Firefox's chardetng and
Chrome's CED both recognise UTF-8. sniff now takes that one narrow,
deterministic step: bytes with something beyond ASCII that are all
valid UTF-8 are UTF-8, tentative, so a late <meta> can still win.

ByteParser guesses from the bytes it holds back (up to the 1,024-byte
wait), where a character cut off at the end still counts. Rather than
wait longer, finish checks the whole input, which a guess keeps
anyway, and when it says otherwise parses as parse_bytes would, so
the streamed result still equals the whole one for every cut.

All 82 html5lib encoding cases still pass, and the streaming
comparisons (418,818 cuts) still match. The bindings' tests and docs
now expect UTF-8 for undeclared UTF-8; the Python behaviour change is
only that undeclared bytes that aren't UTF-8 decode as windows-1252.

@foothillsdigital foothillsdigital left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed d7d63db, including the UTF-8 autodetection that landed while I was testing d94ee84.

C API. The new entry points follow the existing patterns. Each is wrapped in guard, NULL handles give PARSEC_ERR_NULL, and bytes() maps NULL with length 0 to empty. finish takes the parser out, so a second call is PARSEC_ERR_INVALID rather than a double use. free takes NULL and finished parsers, and the encoding name is owned by the document and NUL-terminated. One small thing: parsec_parse_bytes sets *out to NULL on every error but never writes *encoding then, so a C caller who doesn't initialise it reads garbage after a failure. Clearing *encoding to NULL up front, as clear does for *out, would match the header's "on any error". Not blocking.

Offsets. Its unit tests check ASCII positions, so I tested a stronger property with a throwaway test, not committed. For random multilingual text encoded in UTF-8, UTF-16LE/BE, Shift_JIS, EUC-JP, ISO-2022-JP, GBK, GB18030, Big5, EUC-KR and windows-1251, at every character boundary d, decoding bytes[..to_source(d)] must give exactly text[..d]. 209,949 boundaries, 0 wrong, ISO-2022-JP's states, surrogate pairs and GB18030's four-byte sequences included.

UTF-8 autodetection. Taking only the narrow, deterministic step (non-ASCII and all valid UTF-8 means UTF-8, tentative) is the right call, and it fixes the surprise the Python change would otherwise have been. Since ByteParser now guesses from the held-back bytes while parse_bytes judges the whole input, I re-ran my streaming property test against this head with pieces aimed at the gap: valid UTF-8 only after 1,024 bytes of ASCII filler, a late latin-1 byte or Shift_JIS run, sequences cut short, CRLF, metas and a BOM. 24,000 cuttings, 0 differing from parse_bytes in encoding, confidence or text.

Tests. cargo test --workspace --release passes 296 tests in 36 groups, capi.c included. Python built from this head: 34 passed, html5lib encoding: 82 passed, 0 failed, 1 skipped (scripted), whole and streamed. The one other skip is free-threaded. I didn't run the WASM suite (no runner here); CI's is green. Workspace clippy fails only on the pre-existing mix2 lint #117 fixes.

Approving.

@jamesconwaydev
jamesconwaydev merged commit 36f5618 into main Nov 19, 2026
39 checks passed
@jamesconwaydev
jamesconwaydev deleted the bindings-bytes branch November 19, 2026 20:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python and WASM take bytes, sniffing the encoding as browsers do

3 participants