Repository navigation
fix: a lone surrogate in a catalogue name, task.yaml, process input or Context7 query no longer raises UnicodeEncodeError - #863
Conversation
…n SKILL.md (intel#755) build_fixture wrote each SKILL.md with a strict text.encode(), so a catalogue case whose name holds a lone surrogate (a \ud800 or \udcff escape, which load_catalogue accepts) raised UnicodeEncodeError, in the regular branch and in the symlink branch. Both writes now encode with errors="replace": the lone surrogate becomes ? and the file stays valid UTF-8, as guidance_arms.py already does for its SKILL.md. surrogatepass is not used because it would write bytes that are not UTF-8. Refusing the name was not taken: it would turn a catalogue that loads today into an error. The output of a name that encodes as UTF-8 is unchanged. The new test builds both names, with and without a symlink case, and reads the written file back as UTF-8. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
…l#759) harbor_view stopped with UnicodeEncodeError when export wrote task.toml for a task.yaml whose metadata holds a lone surrogate (a \udcff or \ud800 YAML escape): Path.write_text encodes strictly. The content key was already a digest (intel#720), the write was not. The instruction, written by the line above, raised the same way for an instruction with such an escape. task.toml and instruction.md are now written through open(..., "w", encoding="utf-8", errors="replace"), which keeps write_text's newline handling: the lone surrogate becomes ? and both files stay valid UTF-8. surrogatepass is not used because it would write a task.toml that is not UTF-8. Refusing the value was not taken: it would turn a task that loads today into an error. The files of a task that encodes as UTF-8 are unchanged. The unreleased fragment that said such an export still stops no longer says so. The new tests read the written task.toml back with tomllib and the instruction as UTF-8. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
…ad of raising (intel#760) process.run encoded a string input with a strict input.encode("utf-8"), so a prompt built from a question or an artifact text with a \ud800 JSON escape raised UnicodeEncodeError before the child started. FakeRunner.__call__ and reply, the test adapter, raised the same way for a string input and a string stdout or stderr. All three now encode with errors="replace": the lone surrogate becomes ? and the child gets valid UTF-8, as a surrogatepass encoding would not give. Refusing the input where the prompt is built was not taken: it would turn a question set that loads today into an error. The bytes of a string that encodes as UTF-8 are unchanged. The new tests run a child that echoes its stdin for the two escapes, and check the bytes FakeRunner records and reply holds. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
… %3F (intel#794) Context7Client.get_library_docs built the request URL with urllib.parse.quote(query), which encodes strictly, so a question with a \ud800 or \udcff escape raised UnicodeEncodeError on a cache miss, after the cache key (a digest since intel#796) was computed. The library id went into the URL path unquoted, and httpx raised the same error for a lone surrogate there when it built the request. The query is quoted with errors="replace" and the library id is quoted for the path with errors="replace" and the path characters left as they are, so the lone surrogate is sent as %3F in both. surrogatepass is not used because it would send bytes that are not UTF-8; refusing was not taken because it would turn a question set that loads today into an error. The cache key still hashes the text, so two queries that send the same URL keep separate cache entries. The URL of a query or id that holds only letters, digits and those path characters is unchanged; a library id with another character (a space, ? or #) is percent-encoded now. The new tests mock httpx.get with an empty cache directory and check the URL requested for both escapes in the query and in a GitHub-style and a website-style library id. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 12 selected item(s). CI at Review details
|
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #863 +/- ##
=======================================
Coverage 95.76% 95.76%
=======================================
Files 424 425 +1
Lines 56511 56482 -29
Branches 9730 9718 -12
=======================================
- Hits 54115 54091 -24
+ Misses 1487 1485 -2
+ Partials 909 906 -3
🚀 New features to boost your workflow:
|
…passes on Windows Settles the exai-bot thread on test_tbench_v1.py:333. _write_utf8 writes in text mode, so on Windows the file has CRLF line ends and the exact-bytes comparison against "\n" failed. read_text(encoding="utf-8") normalizes the line ends and still decodes strictly. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
vlad-perevezentsev
left a comment
There was a problem hiding this comment.
Review at 6db33d6. The four fixes do what they say for a lone surrogate, and nothing changes for text that encodes as UTF-8. Two gaps are left: a surrogate pair from a YAML escape now turns into ??, and the system prompt of a CLI call still raises; both are in the line comments.
Agreed with @exai-bot on the CRLF read in the instruction.md test.
What I ran, at 6db33d6:
make ci-local --offline: 22,352 passed, 0 failed, 72 skipped; every check that runs offline passed.harbor_viewon atask.yamlwithinstruction: "Reply with \ud83d\ude00"(the escape pairjson.dumpswrites for 😀):instruction.mdisb'Reply with ??\n'. Joining the pairs first (UTF-16 withsurrogatepass) writes the emoji and still gives?for a lone\ud800.llm_cli.callwith a fake runner:prompt="q \ud800"goes through,system="sys \ud800"raisesUnicodeEncodeError.- Not run on Windows: the VM cannot build a venv from this lock.
Checked, no issues
- Tests fail on the base: each new test goes through a strict
.encode("utf-8")orquote()on the merge-base, so it raises there. - Context7 URL: the library id keeps letters, digits and the path characters, so the URL of an id that worked before is the same; the cache key still encodes with
surrogatepass, so two queries that send the same URL keep their own entries. - Other
process.runcallers: the ones that send file names on stdin (hash-object --stdin-paths,check-ignore --stdin,git apply) pass bytes, so the?does not touch a path. - Changelog: the new fragment is under Fixed, and the line in the 20261114 fragment that said the export still stops is removed.
Settles the review threads on the lone-surrogate encode. - A JSON escape pair for a non-BMP character (U+1F600 as `😀`) loads as two lone surrogates in PyYAML and json, so `errors="replace"` wrote `??`. `process_io.joined_pairs` joins each pair into its character first (a lone surrogate stays and becomes `?`); `utf8_bytes` is the bytes form. `process.run`, `FakeRunner`, `reply`, `build_fixture`, the Harbor export and the Context7 URL use them. - `harbor_export._write_utf8` is gone: `write_text(..., errors="replace")` does the same. - `llm_cli.call` writes the system prompt file with the same replacement, so a lone surrogate in a system message no longer raises (intel#861). - The `process` module doc says a `str` input goes out as UTF-8 with a lone surrogate as `?`. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
vlad-perevezentsev
left a comment
There was a problem hiding this comment.
LGTM
I checked a28d84b. Every point from my first round is fixed: a JSON-style surrogate pair is now joined into its character before a lone surrogate becomes ?, at every site this PR touches, and the system prompt of a CLI call no longer raises.
Merge: safe now: it merges cleanly into current main, 5 commits behind, and the tests of the files it touches pass on the merge (404 passed). Once it is in, #806 (llm_cli.py) and #776 (process.py) conflict with it and need a rebase.
What I ran, at a28d84b:
make ci-local --offline: 22,361 passed, 0 failed, 72 skipped.schema_compatfails only becausemainchangednovelty.v1andrepo_event.v1(#489, #778) after this branch's base; this PR touches no schema.- The PR's tests on a merge with current
main: 404 passed, 2 skipped.
Checked, no issues
joined_pairs: the UTF-16 round trip joins a high+low pair and keeps a lone or reversed surrogate forerrors="replace"; text that encodes as UTF-8 comes out the same.- Tests: each new pair test goes through the old
encode("utf-8", "replace")without the helper and gets??, so it fails without the fix. - Changelog: the fragment names the pair behaviour at each site and adds the
llm_cli.callbullet.
Summary
A string with a lone surrogate (a
\ud800or\udcffescape in JSON or YAML) reaches a strict UTF-8 encode that is not a hash and raisesUnicodeEncodeError. The hash sites were fixed by #761 and #796. Onmain, four non-hash sites still raise:mainbuild_fixturewith a catalogue casename: "n\udcff"(orn\ud800), regular orlink: symlinkUnicodeEncodeError: 'utf-8' codec can't encode character '\udcff' in position 117atskill_reach.py:302(:292for the symlink case)harbor_viewof a terminal-benchtask.yamlwithauthor_email: "n\udcff"in the metadata, or the same escape ininstructionUnicodeEncodeErrorfromPath.write_textwhenexportwritestask.toml(instruction.mdfor the instruction)process.run(["cat"], input="n\ud800", ...);FakeRunner(...)(..., input="n\ud800");reply("n\ud800")UnicodeEncodeError ... position 1atprocess.py:190,:777and:746Context7Client(cache_dir=d).get_library_docs("org/x", "n\ud800")on a cache miss; a library id with a surrogateUnicodeEncodeErrorfromurllib.parse.quote; a library id with a surrogate fails inhttpxwhen the request is built (the id was put into the path unquoted)Fix taken for all four: encode with
errors="replace"at the write or the send, so the lone surrogate becomes?(%3Fin a URL) and the run goes on. This is whatrepos/guidance_arms.pyalready does for itsSKILL.md.surrogatepassis ruled out because it writes or sends bytes that are not UTF-8. Refusing the value was not taken because it would turn a catalogue, task or question set that loads today into an error.A JSON-style escape pair for a non-BMP character (
😀) loads as two lone surrogates, soerrors="replace"alone would write??for it. The pair is joined into its character first (process_io.joined_pairs); only a lone surrogate becomes?.llm_cli.callwrote the system prompt file with a strict encode, sosystem="sys \ud800"raised before the child started (#861); it uses the same replacement now.The same cause in other places is tracked in #859 (a lone surrogate in an argv entry of
process.run), #860 (prompt.txtof the Docker solver) and #862 (the package name inonline_checks).Closes #755
Closes #759
Closes #760
Closes #794
Closes #861
Changes
process_io.py:joined_pairs(text)joins each high surrogate followed by a low one into its character and leaves a lone surrogate;utf8_bytes(text)is that andencode("utf-8", "replace").repos/skill_reach.pybuild_fixture: the text of theSKILL.mdgoes throughjoined_pairs, and both writes (regular and symlink case) encode witherrors="replace"(skill-reach build_fixture raises UnicodeEncodeError for a catalogue case name with a lone surrogate #755).harnesses/harbor_export.pyexport:task.tomlandinstruction.mdare written withwrite_text(joined_pairs(text), encoding="utf-8", errors="replace"). The issue namestask.toml;instruction.mdraised the same way on the line above._content_keyis untouched (harbor_view stops with UnicodeEncodeError when export writes task.toml for a task.yaml whose metadata holds a lone surrogate #759).process.py:run,FakeRunner.__call__andreplyencode a string withutf8_bytes; the module doc says astrinput goes out as UTF-8 with a lone surrogate as?, and that bytes are the way to pass a name fromos.fsdecode(process.run raises UnicodeEncodeError for an input string with a lone surrogate #760).llm_cli.pycall: the system prompt file is written withjoined_pairsanderrors="replace"(llm_cli.call raises UnicodeEncodeError for a system prompt with a lone surrogate #861).mcp/context7.pyget_library_docs: the query and the library id go throughjoined_pairsand are quoted witherrors="replace"; the library id is quoted for the path with the path characters/%:@!$&'()*+,;=-._~left as they are. A library id with another character (a space,?,#) is percent-encoded now. The cache key still hashes the text, so two queries that send the same%3Fkeep separate cache entries (Context7Client.get_library_docs raises UnicodeEncodeError on a cache miss for a query with a lone surrogate #794).20261114_120000_antonwolfy_surrogate-digests.mdsaid a lone surrogate intask.yamlmetadata "still stops the Harbor export"; that sentence is removed.tests/test_skill_reach.py(both names, with and without a symlink case),tests/test_tbench_v1.py(metadata and instruction throughharbor_view),tests/test_process.py(a child that echoes stdin,FakeRunner,reply, each with a lone surrogate and with a pair),tests/test_mcp_context7.py(query and library id, GitHub-style and website-style, and a pair),tests/test_llm_cli.py(the system prompt file: a lone surrogate and a pair).Risk
A lone surrogate in these places (the four above and the system prompt file of
llm_cli.call) was an error before; it is now a?, and a JSON-style pair (😀) is its character, so a catalogue name, a task field or a prompt that differed only in the surrogate gives the same file or bytes. Output for text that encodes as UTF-8 is unchanged. The one change for ordinary input is the Context7 library id: it is percent-encoded for the URL path, so an id with a space,?,#or a non-ASCII character is sent encoded where it was sent raw before (letters, digits,/and the path characters above are unchanged).Testing
On a Linux review host, at
a28d84b(onmain08f393c):make ci-local: 34 passed, 0 failed, 14 skipped.testsjob: 22414 passed, 19 skipped. Skipped: the jobs that need a CI runner, Docker,micromamba,semgrepor the other Python matrix legs.main's code the new tests fail:test_skill_reach.py4 failed, 114 passed (UnicodeEncodeErroratskill_reach.py:302and:292);test_tbench_v1.py3 failed, 68 passed, 1 skipped (UnicodeEncodeErrorfromwrite_text);test_process.py6 failed, 74 passed (UnicodeEncodeErroratprocess.py:190,:746,:777);test_mcp_context7.py5 failed, 14 passed, 1 skipped (UnicodeEncodeErrorfromquote, and the library-id URL assertions).encodein the regular and in the symlinkSKILL.mdwrite;surrogatepassthere (the read-back as UTF-8 fails);surrogatepassin the harbor write (both harbor tests fail the read-back); strict encode inprocess.run, inFakeRunnerand inreply; strictquote(query)in the GitHub-style and in the website-style URL; the/dropped from the library id's safe characters.joined_pairsreturning its argument, the seven pair tests fail (test_process.py3,test_skill_reach.py,test_tbench_v1.py,test_mcp_context7.py, andtest_llm_cli.pyjson-pair). With the system prompt file written withouterrors="replace", the two lone-surrogate cases oftest_llm_cli.pyfail; with a stricttask.tomlwrite, the two harbor metadata tests fail; with a strictencodeinprocess.run, threetest_process.pytests fail. Onmain's code the threetest_llm_cli.pysystem prompt cases fail (UnicodeEncodeErrorfromwrite_text).instruction.mdwithread_text(encoding="utf-8"), so CRLF line ends from a Windows text-mode write still compare equal to"Create n?.txt\n"._write_utf8iswrite_text's by construction); a real Context7 request or a realhttpxcall (httpx.getis mocked).Open questions
None.
Revert
A plain revert restores the
UnicodeEncodeErrorat the five sites, and the Context7 library id goes into the URL path unquoted again. Nothing is stored: files already written with a?stay as they are.Checklist
pytestpassesruff,mypy, and schema checks passAGENTS.mdraise/exit and test names the doc, contract or issue it traces to.github/.licenserc.yaml)