Skip to content

Python and WASM take bytes, sniffing the encoding as browsers do #116

Closed

Description

@jamesconwaydev

parsec-html-encoding sniffs and decodes bytes (parse_bytes, #79) and streams them (ByteParser, #115), but neither binding uses it. Python reads bytes as UTF-8, with bad bytes becoming U+FFFD, and WASM takes only strings.

What we'd like:

  • Python: HTMLParser(b"...", transport_encoding="shift_jis") sniffs like parse_bytes and reports parser.encoding and its confidence. Also a StreamParser with feed(bytes) and finish() over ByteParser. Offsets for bytes input stay byte offsets into the caller's bytes (feat(bindings): Python, C and WASM couldn't see positions or XPath #108).
  • WASM: parseBytes(Uint8Array, { transportEncoding }) and a streaming counterpart, typed.
  • C: parsec_parse_bytes and a streaming handle.

Each with the html5lib encoding cases run through it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    pythonThe Python packagewasmWebAssembly and npm

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions