Skip to content

fix: a lone surrogate in a catalogue name, task.yaml, process input or Context7 query no longer raises UnicodeEncodeError - #863

Merged
antonwolfy merged 6 commits into
intel:mainfrom
antonwolfy:fix/lone-surrogate-encode
Nov 17, 2026
Merged

antonwolfy merged 6 commits into
intel:mainfrom
antonwolfy:fix/lone-surrogate-encode

Conversation

@antonwolfy

@antonwolfy antonwolfy commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A string with a lone surrogate (a \ud800 or \udcff escape in JSON or YAML) reaches a strict UTF-8 encode that is not a hash and raises UnicodeEncodeError. The hash sites were fixed by #761 and #796. On main, four non-hash sites still raise:

Issue Input On main
#755 build_fixture with a catalogue case name: "n\udcff" (or n\ud800), regular or link: symlink UnicodeEncodeError: 'utf-8' codec can't encode character '\udcff' in position 117 at skill_reach.py:302 (:292 for the symlink case)
#759 harbor_view of a terminal-bench task.yaml with author_email: "n\udcff" in the metadata, or the same escape in instruction UnicodeEncodeError from Path.write_text when export writes task.toml (instruction.md for the instruction)
#760 process.run(["cat"], input="n\ud800", ...); FakeRunner(...)(..., input="n\ud800"); reply("n\ud800") UnicodeEncodeError ... position 1 at process.py:190, :777 and :746
#794 Context7Client(cache_dir=d).get_library_docs("org/x", "n\ud800") on a cache miss; a library id with a surrogate UnicodeEncodeError from urllib.parse.quote; a library id with a surrogate fails in httpx when the request is built (the id was put into the path unquoted)

Fix taken for all four: encode with errors="replace" at the write or the send, so the lone surrogate becomes ? (%3F in a URL) and the run goes on. This is what repos/guidance_arms.py already does for its SKILL.md. surrogatepass is ruled out because it writes or sends bytes that are not UTF-8. Refusing the value was not taken because it would turn a catalogue, task or question set that loads today into an error.

A JSON-style escape pair for a non-BMP character (😀) loads as two lone surrogates, so errors="replace" alone would write ?? for it. The pair is joined into its character first (process_io.joined_pairs); only a lone surrogate becomes ?.

llm_cli.call wrote the system prompt file with a strict encode, so system="sys \ud800" raised before the child started (#861); it uses the same replacement now.

The same cause in other places is tracked in #859 (a lone surrogate in an argv entry of process.run), #860 (prompt.txt of the Docker solver) and #862 (the package name in online_checks).

Closes #755
Closes #759
Closes #760
Closes #794
Closes #861

Changes

Risk

A lone surrogate in these places (the four above and the system prompt file of llm_cli.call) was an error before; it is now a ?, and a JSON-style pair (😀) is its character, so a catalogue name, a task field or a prompt that differed only in the surrogate gives the same file or bytes. Output for text that encodes as UTF-8 is unchanged. The one change for ordinary input is the Context7 library id: it is percent-encoded for the URL path, so an id with a space, ?, # or a non-ASCII character is sent encoded where it was sent raw before (letters, digits, / and the path characters above are unchanged).

Testing

On a Linux review host, at a28d84b (on main 08f393c):

  • make ci-local: 34 passed, 0 failed, 14 skipped. tests job: 22414 passed, 19 skipped. Skipped: the jobs that need a CI runner, Docker, micromamba, semgrep or the other Python matrix legs.
  • On main's code the new tests fail: test_skill_reach.py 4 failed, 114 passed (UnicodeEncodeError at skill_reach.py:302 and :292); test_tbench_v1.py 3 failed, 68 passed, 1 skipped (UnicodeEncodeError from write_text); test_process.py 6 failed, 74 passed (UnicodeEncodeError at process.py:190, :746, :777); test_mcp_context7.py 5 failed, 14 passed, 1 skipped (UnicodeEncodeError from quote, and the library-id URL assertions).
  • Mutants, one break each, each failed the test named for it: strict encode in the regular and in the symlink SKILL.md write; surrogatepass there (the read-back as UTF-8 fails); surrogatepass in the harbor write (both harbor tests fail the read-back); strict encode in process.run, in FakeRunner and in reply; strict quote(query) in the GitHub-style and in the website-style URL; the / dropped from the library id's safe characters.
  • Surrogate pairs: with joined_pairs returning its argument, the seven pair tests fail (test_process.py 3, test_skill_reach.py, test_tbench_v1.py, test_mcp_context7.py, and test_llm_cli.py json-pair). With the system prompt file written without errors="replace", the two lone-surrogate cases of test_llm_cli.py fail; with a strict task.toml write, the two harbor metadata tests fail; with a strict encode in process.run, three test_process.py tests fail. On main's code the three test_llm_cli.py system prompt cases fail (UnicodeEncodeError from write_text).
  • The instruction test reads instruction.md with read_text(encoding="utf-8"), so CRLF line ends from a Windows text-mode write still compare equal to "Create n?.txt\n".
  • Not run: Python 3.11, 3.12 and 3.14; Windows (the newline handling of _write_utf8 is write_text's by construction); a real Context7 request or a real httpx call (httpx.get is mocked).

Open questions

None.

Revert

A plain revert restores the UnicodeEncodeError at the five sites, and the Context7 library id goes into the URL path unquoted again. Nothing is stored: files already written with a ? stay as they are.

Checklist

  • Tests added/updated and pytest passes
  • ruff, mypy, and schema checks pass
  • Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
  • Self-reviewed per "Before opening a PR" in AGENTS.md
  • Each new raise/exit and test names the doc, contract or issue it traces to
  • No ad-hoc run artifacts committed
  • New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

…n SKILL.md (intel#755)

build_fixture wrote each SKILL.md with a strict text.encode(), so a catalogue case whose name holds a lone surrogate (a \ud800 or \udcff escape, which load_catalogue accepts) raised UnicodeEncodeError, in the regular branch and in the symlink branch.

Both writes now encode with errors="replace": the lone surrogate becomes ? and the file stays valid UTF-8, as guidance_arms.py already does for its SKILL.md. surrogatepass is not used because it would write bytes that are not UTF-8. Refusing the name was not taken: it would turn a catalogue that loads today into an error. The output of a name that encodes as UTF-8 is unchanged.

The new test builds both names, with and without a symlink case, and reads the written file back as UTF-8.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
…l#759)

harbor_view stopped with UnicodeEncodeError when export wrote task.toml for a task.yaml whose metadata holds a lone surrogate (a \udcff or \ud800 YAML escape): Path.write_text encodes strictly. The content key was already a digest (intel#720), the write was not. The instruction, written by the line above, raised the same way for an instruction with such an escape.

task.toml and instruction.md are now written through open(..., "w", encoding="utf-8", errors="replace"), which keeps write_text's newline handling: the lone surrogate becomes ? and both files stay valid UTF-8. surrogatepass is not used because it would write a task.toml that is not UTF-8. Refusing the value was not taken: it would turn a task that loads today into an error. The files of a task that encodes as UTF-8 are unchanged.

The unreleased fragment that said such an export still stops no longer says so. The new tests read the written task.toml back with tomllib and the instruction as UTF-8.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
…ad of raising (intel#760)

process.run encoded a string input with a strict input.encode("utf-8"), so a prompt built from a question or an artifact text with a \ud800 JSON escape raised UnicodeEncodeError before the child started. FakeRunner.__call__ and reply, the test adapter, raised the same way for a string input and a string stdout or stderr.

All three now encode with errors="replace": the lone surrogate becomes ? and the child gets valid UTF-8, as a surrogatepass encoding would not give. Refusing the input where the prompt is built was not taken: it would turn a question set that loads today into an error. The bytes of a string that encodes as UTF-8 are unchanged.

The new tests run a child that echoes its stdin for the two escapes, and check the bytes FakeRunner records and reply holds.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
… %3F (intel#794)

Context7Client.get_library_docs built the request URL with urllib.parse.quote(query), which encodes strictly, so a question with a \ud800 or \udcff escape raised UnicodeEncodeError on a cache miss, after the cache key (a digest since intel#796) was computed. The library id went into the URL path unquoted, and httpx raised the same error for a lone surrogate there when it built the request.

The query is quoted with errors="replace" and the library id is quoted for the path with errors="replace" and the path characters left as they are, so the lone surrogate is sent as %3F in both. surrogatepass is not used because it would send bytes that are not UTF-8; refusing was not taken because it would turn a question set that loads today into an error. The cache key still hashes the text, so two queries that send the same URL keep separate cache entries. The URL of a query or id that holds only letters, digits and those path characters is unchanged; a library id with another character (a space, ? or #) is percent-encoded now.

The new tests mock httpx.get with an empty cache directory and check the URL requested for both escapes in the query and in a GitHub-style and a website-style library id.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@exai-bot

exai-bot commented Nov 17, 2026 •

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 12 selected item(s).

CI at a28d84b: 29 passing, 3 skipped.

Review details
  • Commit reviewed: a28d84b against merge base 7e0ca3f
  • Coverage: 12 files in 5 groups
  • Engine: OpenCodeReview v1.13.5-exai.4.8f73eada, us.openai.gpt-6.1-sol
  • Usage: 387k tokens (255k from cache), 43s
  • Tool calls: 48 (38 file_read, 5 local_check_results, 3 code_search, 2 file_read_diff)
  • Lone-surrogate encoding and process I/O: 4 files
  • Harbor export and terminal benchmark coverage: 2 files
  • LLM CLI and regression tests: 2 files
  • Context7 MCP integration and tests: 2 files
  • Skill reach repository integration and tests: 2 files

Comment thread dokimi/tests/test_tbench_v1.py Resolved
@codecov-commenter

codecov-commenter commented Nov 17, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.76%. Comparing base (08f393c) to head (a28d84b).
⚠️ Report is 1 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #863   +/-   ##
=======================================
  Coverage   95.76%   95.76%           
=======================================
  Files         424      425    +1     
  Lines       56511    56482   -29     
  Branches     9730     9718   -12     
=======================================
- Hits        54115    54091   -24     
+ Misses       1487     1485    -2     
+ Partials      909      906    -3     
Files with missing lines Coverage Δ
dokimi/dokimi/harnesses/harbor_export.py 91.30% <100.00%> (+0.07%) ⬆️
dokimi/dokimi/llm_cli.py 97.29% <100.00%> (+0.01%) ⬆️
dokimi/dokimi/mcp/context7.py 95.00% <100.00%> (+2.79%) ⬆️
dokimi/dokimi/process.py 90.49% <100.00%> (ø)
dokimi/dokimi/process_io.py 96.77% <100.00%> (+0.22%) ⬆️
dokimi/dokimi/repos/skill_reach.py 99.15% <100.00%> (+<0.01%) ⬆️

... and 8 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…passes on Windows

Settles the exai-bot thread on test_tbench_v1.py:333. _write_utf8 writes in text mode, so on Windows the file has CRLF line ends and the exact-bytes comparison against "\n" failed. read_text(encoding="utf-8") normalizes the line ends and still decodes strictly.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at 6db33d6. The four fixes do what they say for a lone surrogate, and nothing changes for text that encodes as UTF-8. Two gaps are left: a surrogate pair from a YAML escape now turns into ??, and the system prompt of a CLI call still raises; both are in the line comments.

Agreed with @exai-bot on the CRLF read in the instruction.md test.

What I ran, at 6db33d6:

  • make ci-local --offline: 22,352 passed, 0 failed, 72 skipped; every check that runs offline passed.
  • harbor_view on a task.yaml with instruction: "Reply with \ud83d\ude00" (the escape pair json.dumps writes for 😀): instruction.md is b'Reply with ??\n'. Joining the pairs first (UTF-16 with surrogatepass) writes the emoji and still gives ? for a lone \ud800.
  • llm_cli.call with a fake runner: prompt="q \ud800" goes through, system="sys \ud800" raises UnicodeEncodeError.
  • Not run on Windows: the VM cannot build a venv from this lock.

Checked, no issues

  • Tests fail on the base: each new test goes through a strict .encode("utf-8") or quote() on the merge-base, so it raises there.
  • Context7 URL: the library id keeps letters, digits and the path characters, so the URL of an id that worked before is the same; the cache key still encodes with surrogatepass, so two queries that send the same URL keep their own entries.
  • Other process.run callers: the ones that send file names on stdin (hash-object --stdin-paths, check-ignore --stdin, git apply) pass bytes, so the ? does not touch a path.
  • Changelog: the new fragment is under Fixed, and the line in the 20261114 fragment that said the export still stops is removed.

Comment thread dokimi/dokimi/process.py Resolved
Comment thread dokimi/dokimi/process.py Resolved
Settles the review threads on the lone-surrogate encode.

- A JSON escape pair for a non-BMP character (U+1F600 as `😀`) loads as two lone surrogates in PyYAML and json, so `errors="replace"` wrote `??`. `process_io.joined_pairs` joins each pair into its character first (a lone surrogate stays and becomes `?`); `utf8_bytes` is the bytes form. `process.run`, `FakeRunner`, `reply`, `build_fixture`, the Harbor export and the Context7 URL use them.
- `harbor_export._write_utf8` is gone: `write_text(..., errors="replace")` does the same.
- `llm_cli.call` writes the system prompt file with the same replacement, so a lone surrogate in a system message no longer raises (intel#861).
- The `process` module doc says a `str` input goes out as UTF-8 with a lone surrogate as `?`.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

I checked a28d84b. Every point from my first round is fixed: a JSON-style surrogate pair is now joined into its character before a lone surrogate becomes ?, at every site this PR touches, and the system prompt of a CLI call no longer raises.

Merge: safe now: it merges cleanly into current main, 5 commits behind, and the tests of the files it touches pass on the merge (404 passed). Once it is in, #806 (llm_cli.py) and #776 (process.py) conflict with it and need a rebase.

What I ran, at a28d84b:

  • make ci-local --offline: 22,361 passed, 0 failed, 72 skipped. schema_compat fails only because main changed novelty.v1 and repo_event.v1 (#489, #778) after this branch's base; this PR touches no schema.
  • The PR's tests on a merge with current main: 404 passed, 2 skipped.

Checked, no issues

  • joined_pairs: the UTF-16 round trip joins a high+low pair and keeps a lone or reversed surrogate for errors="replace"; text that encodes as UTF-8 comes out the same.
  • Tests: each new pair test goes through the old encode("utf-8", "replace") without the helper and gets ??, so it fails without the fix.
  • Changelog: the fragment names the pair behaviour at each site and adds the llm_cli.call bullet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment