Repository navigation
feat: a page had to arrive whole before it could be parsed - #114
Conversation
Syncs Netweir 46ba93b, bringing: - netweir/netweir#334: Parser::new(options), feed(chunk) and finish() build exactly the tree the whole input would, tokenizing what's complete and keeping the unfinished tail. parse_with now runs through the same Parser, fed once. A budget counts only time spent in feed and finish, and the clock is read only when there's a budget. - netweir/netweir#342 by foothillsdigital: ParseOptions::attr_namespaces declares prefixes for attribute selectors, so [xlink|href] matches SVG's xlink:href.
foothillsdigital
left a comment
There was a problem hiding this comment.
Reviewed fde885d.
It's a faithful sync. For tokenizer.rs, tree.rs, css.rs, dom.rs and lib.rs, this PR's change to Parsec is line for line Netweir's change from 57a359d, the last sync, to 46ba93b, with the crate names normalised: 207, 505, 69, 21 and 3 changed lines, 0 differing.
The unsafe in Parser::tokenize. The SAFETY comment rests on nothing growing doc.src while the tokenizer holds the &'static str, so I checked every write to it. They're in push, the constructor, and finish before it calls tokenize, all between runs. The tokenizer only reads it, and detach() blanks its reference right after each run. extend() replaces it before the next one, so even after a panic mid-run a stale reference is never read. The argument holds.
Checks. cargo clippy -p parsec-html --all-targets -D warnings is clean. cargo test --workspace --release passes 277 tests in 34 groups with 0 failures, with html5lib-tests at the pinned commit. tests/chunks.rs reproduces your numbers exactly: 205,527 two-piece cuts and 86,000 random splits, 0 differing from a one-shot parse. The fuzz target builds.
Approving.
Problem
Parsec needed the whole page before it could parse (#6). Lexbor can parse a page as it arrives.
Change
Syncs Netweir 46ba93b:
Parser::new(options), thenfeed(chunk)as each piece arrives andfinish(). Fragments work too, throughParser::fragment.parse_withruns through the sameParser, fed once.feedandfinish. The clock is read only when there's a budget, so WebAssembly still works.unsafeblock in the parser, now inParser::tokenize, with its reasoning beside it.ParseOptions::attr_namespacesdeclares prefixes for attribute selectors, so[xlink|href]matches SVG'sxlink:href.Closes #6
Tests
tests/chunks.rs: every html5lib tokenizer and tree-construction input is cut into two pieces at every character boundary, 205,527 cuts, and split into many pieces at random, 86,000 splits. Nodes, links, attributes, spans and errors are compared with a one-shot parse: 0 differ.cargo test --workspace: 277 test groups, 0 failed. Tokenizer 7,045 and tree construction 3,898 passed, 0 failed. Python and WASM bindings build.