Skip to content

feat: a page had to arrive whole before it could be parsed - #114

Merged
jamesconwaydev merged 1 commit into
mainfrom
sync-chunked
Nov 19, 2026
Merged

jamesconwaydev merged 1 commit into
mainfrom
sync-chunked

Conversation

@jamesconwaydev

Copy link
Copy Markdown
Collaborator

Problem

Parsec needed the whole page before it could parse (#6). Lexbor can parse a page as it arrives.

Change

Syncs Netweir 46ba93b:

Closes #6

Tests

  • tests/chunks.rs: every html5lib tokenizer and tree-construction input is cut into two pieces at every character boundary, 205,527 cuts, and split into many pieces at random, 86,000 splits. Nodes, links, attributes, spans and errors are compared with a one-shot parse: 0 differ.
  • cargo test --workspace: 277 test groups, 0 failed. Tokenizer 7,045 and tree construction 3,898 passed, 0 failed. Python and WASM bindings build.

Syncs Netweir 46ba93b, bringing:

- netweir/netweir#334: Parser::new(options), feed(chunk) and finish()
  build exactly the tree the whole input would, tokenizing what's
  complete and keeping the unfinished tail. parse_with now runs through
  the same Parser, fed once. A budget counts only time spent in feed
  and finish, and the clock is read only when there's a budget.
- netweir/netweir#342 by foothillsdigital: ParseOptions::attr_namespaces
  declares prefixes for attribute selectors, so [xlink|href] matches
  SVG's xlink:href.

@foothillsdigital foothillsdigital left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed fde885d.

It's a faithful sync. For tokenizer.rs, tree.rs, css.rs, dom.rs and lib.rs, this PR's change to Parsec is line for line Netweir's change from 57a359d, the last sync, to 46ba93b, with the crate names normalised: 207, 505, 69, 21 and 3 changed lines, 0 differing.

The unsafe in Parser::tokenize. The SAFETY comment rests on nothing growing doc.src while the tokenizer holds the &'static str, so I checked every write to it. They're in push, the constructor, and finish before it calls tokenize, all between runs. The tokenizer only reads it, and detach() blanks its reference right after each run. extend() replaces it before the next one, so even after a panic mid-run a stale reference is never read. The argument holds.

Checks. cargo clippy -p parsec-html --all-targets -D warnings is clean. cargo test --workspace --release passes 277 tests in 34 groups with 0 failures, with html5lib-tests at the pinned commit. tests/chunks.rs reproduces your numbers exactly: 205,527 two-piece cuts and 86,000 random splits, 0 differing from a one-shot parse. The fuzz target builds.

Approving.

@jamesconwaydev
jamesconwaydev merged commit 1364888 into main Nov 19, 2026
33 checks passed
@jamesconwaydev
jamesconwaydev deleted the sync-chunked branch November 19, 2026 18:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Chunked input that builds exactly the same tree

2 participants