Repository navigation
feat(bindings): the bindings couldn't sniff a page's encoding from bytes - #118
Conversation
The parser's offsets count UTF-8 bytes into the decoded page, so a caller holding the Shift_JIS or UTF-16 bytes had no way to find where a tag was in them. Offsets::new(bytes, encoding) decodes them again, noting where each character's bytes began, and keeps only the places where the two stop moving in step; to_source and to_decoded convert. UTF-8 and single-byte encodings take one pass, the rest go through the decoder a byte at a time. tests/offsets.rs maps 187,952 element and attribute spans on 334 pages (the html5lib encoding inputs, behind BOMs and widened to UTF-16, and the WPT legacy multi-byte pages) back to their bytes. Also, parse_bytes read the clock even with no budget, which panics in WebAssembly; it now reads it only for a budget, as ByteParser does.
HTMLParser(data) now decodes bytes as a browser does, through parsec_encoding: byte order mark, then transport_encoding, then a <meta>, then default_encoding (windows-1252). The keyword names are html5lib's; a transport label that names no encoding is passed over, as the HTML Standard says, and either keyword with a str raises TypeError, as html5lib's does. parser.encoding gives the WHATWG name and parser.encoding_confidence "certain" or "tentative"; encoding_for_locale(tag) gives the fallback for a language. StreamParser takes the same keywords and feed(data) / close(), the names of lxml's feed parser and html.parser, with close() returning the HTMLParser. Both let go of the GIL while they work. Positions and error offsets stay byte offsets into the bytes given: the units map now uses parsec_encoding::Offsets for any encoding, in place of the UTF-8-only repair list. This changes behaviour: a UTF-8 page that doesn't say so now decodes as windows-1252. The README says so, and how to keep UTF-8. Tests: the 82 html5lib encoding cases through HTMLParser and through StreamParser in pieces of 1 to 7 bytes, and a Shift_JIS page whose ranges, attributes, errors and line_col are checked against its bytes, whole and streamed. CI checks out html5lib-tests for them.
parseBytes(Uint8Array, { transportEncoding, defaultEncoding, ...})
decodes a page as a browser does and parses it; new ByteParser(options)
with feed() and finish() does the same as the bytes arrive. The
document gains characterSet, as the DOM's (UTF-8 for a string), and
encodingConfidence. ByteParseOptions extends ParseOptions in the
TypeScript types. Spans and error offsets for bytes are byte offsets
into the Uint8Array, in its own encoding.
The WebAssembly grows by 199 KB, 141 KB gzipped, to 1,070 KB and
504 KB gzipped: encoding_rs's decoding tables, mostly CJK. That's
under the 150 KB gzipped that would have meant a separate entry
point, so it's in the one build.
Tests: wasm-bindgen tests (Node and Firefox) run the 82 html5lib
encoding cases whole and in pieces of 1 to 7 bytes; the package test
checks the options, the streaming parser and Shift_JIS spans.
parsec_parse_bytes(bytes, len, transport, transport_len, flags, &doc, &encoding) sniffs and decodes a page as a browser does, and parsec_document_encoding(doc) names the encoding (NULL for a document parsed from text). parsec_byte_parser_new, _feed, _finish and _free do the same for bytes as they arrive. All additions; the header is regenerated with cbindgen 0.29.4. For a document from bytes, the span calls, parsec_line_col and parsec_errors_get count byte offsets into those bytes, in their own encoding. tests/capi.c checks a Shift_JIS page's spans and errors, whole and in 3-byte pieces, and the error paths; given the html5lib-tests encoding/ folder (tests/c.rs and the sanitizer job pass it) it runs the 82 cases through both entry points.
README: the C API, Python and WebAssembly bullets and comparison rows, the Python behaviour change, and the WebAssembly's new size. ROADMAP: the bindings' bytes under M4.
CodeQL flagged the html5lib folder read from argv and a buffer index used before its range check. The folder now comes in at build time as HTML5LIB_ENCODING, defined by tests/c.rs and the sanitizer job, and the label copy checks its bound first.
With no BOM, transport encoding or <meta>, sniffing went straight to the fallback, so the most common bytes people pass in, UTF-8 that doesn't say so, came out garbled. The HTML Standard allows an autodetection step before the fallback, and Firefox's chardetng and Chrome's CED both recognise UTF-8. sniff now takes that one narrow, deterministic step: bytes with something beyond ASCII that are all valid UTF-8 are UTF-8, tentative, so a late <meta> can still win. ByteParser guesses from the bytes it holds back (up to the 1,024-byte wait), where a character cut off at the end still counts. Rather than wait longer, finish checks the whole input, which a guess keeps anyway, and when it says otherwise parses as parse_bytes would, so the streamed result still equals the whole one for every cut. All 82 html5lib encoding cases still pass, and the streaming comparisons (418,818 cuts) still match. The bindings' tests and docs now expect UTF-8 for undeclared UTF-8; the Python behaviour change is only that undeclared bytes that aren't UTF-8 decode as windows-1252.
foothillsdigital
left a comment
There was a problem hiding this comment.
Reviewed d7d63db, including the UTF-8 autodetection that landed while I was testing d94ee84.
C API. The new entry points follow the existing patterns. Each is wrapped in guard, NULL handles give PARSEC_ERR_NULL, and bytes() maps NULL with length 0 to empty. finish takes the parser out, so a second call is PARSEC_ERR_INVALID rather than a double use. free takes NULL and finished parsers, and the encoding name is owned by the document and NUL-terminated. One small thing: parsec_parse_bytes sets *out to NULL on every error but never writes *encoding then, so a C caller who doesn't initialise it reads garbage after a failure. Clearing *encoding to NULL up front, as clear does for *out, would match the header's "on any error". Not blocking.
Offsets. Its unit tests check ASCII positions, so I tested a stronger property with a throwaway test, not committed. For random multilingual text encoded in UTF-8, UTF-16LE/BE, Shift_JIS, EUC-JP, ISO-2022-JP, GBK, GB18030, Big5, EUC-KR and windows-1251, at every character boundary d, decoding bytes[..to_source(d)] must give exactly text[..d]. 209,949 boundaries, 0 wrong, ISO-2022-JP's states, surrogate pairs and GB18030's four-byte sequences included.
UTF-8 autodetection. Taking only the narrow, deterministic step (non-ASCII and all valid UTF-8 means UTF-8, tentative) is the right call, and it fixes the surprise the Python change would otherwise have been. Since ByteParser now guesses from the held-back bytes while parse_bytes judges the whole input, I re-ran my streaming property test against this head with pieces aimed at the gap: valid UTF-8 only after 1,024 bytes of ASCII filler, a late latin-1 byte or Shift_JIS run, sequences cut short, CRLF, metas and a BOM. 24,000 cuttings, 0 differing from parse_bytes in encoding, confidence or text.
Tests. cargo test --workspace --release passes 296 tests in 36 groups, capi.c included. Python built from this head: 34 passed, html5lib encoding: 82 passed, 0 failed, 1 skipped (scripted), whole and streamed. The one other skip is free-threaded. I didn't run the WASM suite (no runner here); CI's is green. Workspace clippy fails only on the pre-existing mix2 lint #117 fixes.
Approving.
Problem
parsec-html-encodingsniffs and decodes bytes (parse_bytes, #79) and streams them (ByteParser, #115), but no binding used it. Python readbytesas UTF-8, with anything else turned into U+FFFD, WebAssembly took only strings, and C callers had to decode to UTF-8 themselves. And the offsets that #108 gave each binding only knew UTF-8.Change
Python.
HTMLParser(data)decodesbytesas a browser does: byte order mark, thentransport_encoding, then a<meta>, thendefault_encoding(windows-1252). The names are html5lib's, and so is theTypeErrorfor either one with astr.parser.encodinggives the WHATWG name andparser.encoding_confidence"certain"or"tentative";encoding_for_locale("ja")gives a language's fallback.StreamParser(...)withfeed(data)andclose()(lxml's feed-parser andhtml.parsernames) returns the sameHTMLParserthe whole bytes give. A small behaviour change: undeclared bytes that aren't valid UTF-8 used to get U+FFFD per bad byte and now decode as windows-1252, as in a browser. Undeclared UTF-8 still decodes as UTF-8 (see autodetection below).WebAssembly.
parseBytes(Uint8Array, { transportEncoding, defaultEncoding, ...ParseOptions })andnew ByteParser(options)withfeed()/finish();Document.characterSet(as the DOM's;"UTF-8"for a string) andencodingConfidence;ByteParseOptionsin the types.C.
parsec_parse_bytes(bytes, len, transport, transport_len, flags, &doc, &encoding),parsec_document_encoding(doc), and aparsec_byte_parserhandle with_new,_feed,_finishand_free. All additions; header regenerated with cbindgen 0.29.4.Offsets. In all three, spans and error offsets for bytes are byte offsets into those bytes in their own encoding. That's a new
parsec_encoding::Offsets, built only when positions or errors are on: it decodes the bytes again noting where each character began (one pass for UTF-8 and single-byte encodings, byte by byte for the others: 10.8 ms against a 4.5 ms parse on a 773 KB Shift_JIS page) and keeps only the points where the two counts stop moving together.Everywhere, a transport label that names no encoding is passed over, as the HTML Standard says (servers send
charset=utf8mb4); an unknown default encoding is an error.Autodetection.
sniffnow takes the HTML Standard's autodetection step before the fallback, narrowed to what Firefox's chardetng and Chrome's CED both do: with no BOM, transport encoding or<meta>in the first 1,024 bytes, bytes with something beyond ASCII that are all valid UTF-8 are UTF-8, tentative (a late<meta>still wins).parse_byteschecks the whole input.ByteParserguesses from the bytes it holds back (up to the existing 1,024-byte wait; a character cut off at the end counts as valid), and atfinishre-checks the whole input, which it keeps anyway while the guess is tentative; if that disagrees (UTF-8 starting after 1,024 bytes of ASCII, or a bad byte later on) it parses asparse_byteswould. So streaming still equals the whole parse for every cut, without holding bytes back any longer. No html5lib case expected windows-1252 for valid non-ASCII UTF-8; all 82 still pass.Also fixed:
parse_bytesread the clock with no budget, which panics in WebAssembly.Size. The WebAssembly goes from 870,581 to 1,071,302 bytes, 362,444 to 504,465 gzipped (+142 KB gzipped, encoding_rs's CJK tables). That's under the 150 KB gzipped that would have meant a separate entry point, so it's in the one build.
Tests
encoding/cases, all 82 that don't need a script engine (1 scripted skipped), through each binding's whole and streamed entry points, the streamed one in pieces of 1 to 7 bytes: Python (pytest), WebAssembly (wasm-bindgen in Node and Firefox) and C (tests/capi.c, built with the folder asHTML5LIB_ENCODINGbytests/c.rsand the sanitizer job). 82 of 82 in each.capi.c.crates/parsec-encoding/tests/offsets.rs: 187,952 element and attribute spans on 334 pages (html5lib inputs as written, behind a UTF-8 BOM and widened to UTF-16 LE and BE, plus the WPT legacy multi-byte pages) map back to bytes that decode to what each span covers.cargo test --workspace, clippy andcargo denyfor the workspace and both bindings, pytest (33 passed, 1 skipped as before),wasm-pack test --nodeand--firefox(10 each),node --test tests/package.test.mjs(11), andcapi.cunder ASan and UBSan on macOS.Closes #116