Repository navigation
fix: a lone surrogate in an argv entry, a task instruction or a package name no longer raises UnicodeEncodeError - #886
Open
antonwolfy wants to merge 4 commits into
Open
antonwolfy wants to merge 4 commits into
antonwolfy wants to merge 4 commits into
Conversation
…s ? instead of raising (intel#859) process.run passed argv entries to subprocess unchanged, whose os.fsencode raises UnicodeEncodeError for a lone high surrogate (a JSON/YAML \ud800 escape in a prompt given as `claude -p <prompt>`), before the child starts. _check now sends each entry through process_io.argv_text: surrogate pairs are joined as on stdin, and each character os.fsencode cannot encode becomes ?. The surrogateescape range U+DC80..U+DCFF stays on POSIX, so a non-UTF-8 file name from os.fsdecode still names the same file. An entry os.fsencode accepts is unchanged. Completed.argv, and the argv FakeRunner records, hold the text sent. Closes intel#859 Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
…with a lone surrogate as ? instead of raising (intel#860) DockerSolverHarness._claude_args wrote prompt.txt with a strict UTF-8 encode, so a task instruction with a \ud800 or \udcff escape raised UnicodeEncodeError. It now writes joined_pairs(prompt) with errors="replace", as the Harbor export does: an escape pair becomes its character and a lone surrogate becomes ?, which is what the child gets from the argv entry (intel#859). Closes intel#860 Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
…instead of raising (intel#862) online_checks._package_finding built the registry URL with a strict quote(), so a package name with a \ud800 or \udcff escape raised UnicodeEncodeError, outside the try that turns probe errors into outcomes. It now quotes joined_pairs(name) with errors="replace", as Context7Client.get_library_docs does: a lone surrogate becomes %3F, the registry answers 404 and the usual package-not-found warning follows. The check is not skipped. Closes intel#862 Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 8 selected item(s). CI at Review details
|
exai-bot
reviewed
Nov 17, 2026
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #886 +/- ##
==========================================
- Coverage 95.79% 95.79% -0.01%
==========================================
Files 429 432 +3
Lines 56842 57220 +378
Branches 9787 9858 +71
==========================================
+ Hits 54452 54812 +360
- Misses 1487 1497 +10
- Partials 903 911 +8
🚀 New features to boost your workflow:
|
…rompt.txt Review point: with the instruction "n\udcff", prompt.txt held "n?" while `-p` carried the original text, which argv_text keeps as byte 0xff. The prompt is now normalised once (pairs joined, a lone surrogate a "?") and that text is both written and sent. The harness test asserts the sent -p argument equals the saved text. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three leftovers of #863, which fixed a lone surrogate (from a
\ud800JSON/YAML escape) raisingUnicodeEncodeErrorat other sites. Onmain:mainprocess.run(['echo', 'n\ud800'], timeout=5, env=process.Env.inherit())UnicodeEncodeError: 'utf-8' codec can't encode character '\ud800' in position 1: surrogates not allowed, fromsubprocess, before the child starts.'n\udcff'does not raise (the child gets byte 0xff) and is kept that wayDockerSolverHarness._claude_argswith the instruction'n\ud800'or'n\udcff'UnicodeEncodeError, fromwrite_textofprompt.txtonline_checks._package_finding(None, 'pypi', 'n\ud800')(also'n\udcff', andnpm)UnicodeEncodeError, fromquote(name), outside thetrythat turns probe errors into outcomesFix taken for each, as #863 did it (
joined_pairs, then a replacement of what cannot be encoded), so the user's output is kept and nothing is refused or skipped:errors="replace"was not taken: it would turn'n\udcff'(a surrogateescape byte, which is how a non-UTF-8 file name fromos.fsdecodereaches argv) inton?and change the file the child opens. Only the charactersos.fsencodecannot encode become?.?, as the Harbor export writesinstruction.md), and that one text is both written toprompt.txtand sent as the-pargument, so the saved prompt is the evaluated one. Without it'n\udcff'was saved asn?while-pcarried the original, which reaches the child as byte 0xff.errors="replace", asContext7Client.get_library_docsdoes. The issue's other option (skip the lookup) was not taken: a name that is not valid text would then get no finding at all, where the replaced URL (%3F) gets a 404 and the usualpackage-not-foundwarning, which is right for a name no registry has.Closes #859
Closes #860
Closes #862
The same cause at two more sites, found by the search for other strict encodes, is tracked in #884 (
clip_outputof a trace's command output) and #885 (the mail subject of a replayed pull request).Changes
process_io.argv_text(next toutf8_bytes): joins surrogate pairs, then replaces each characteros.fsencodecannot encode with?; U+DC80..U+DCFF stay on POSIX. An entryos.fsencodeaccepts is returned as it was.process._checkapplies it to every entry, sorunandFakeRunnerboth send and record the same text, andCompleted.argvholds the text sent. The module doc ofprocess.pystates the argv rule next to the stdin rule (two paragraphs of the doc were joined to keep the file at its 800-line cap). Windows:os.fsencodeencodes withsurrogatepassthere and does not raise, so only the pair joining applies.DockerSolverHarness._claude_argsnormalises the prompt withprocess_io.utf8_bytes(...).decode()and uses that text for bothprompt.txtand-p; the harness test assertsargs[:2] == ["-p", saved].online_checks._package_findingquotesjoined_pairs(name)witherrors="replace"for both registries. The finding's message uses{name!r}, which escapes a surrogate, so it stays printable.tests/test_process.pyandtests/test_catalog_online.pyare at the 1000-line cap:tests/test_process_argv.py,tests/test_online_package_surrogate.py; the docker test is intests/test_docker_solver_harness.py. No existing test is changed.Risk
An argv entry with a lone surrogate or a
😀escape pair reaches the child changed (?or the character), where it raised before;Completed.argvfor such a call holds the changed text. Every entryos.fsencodealready accepted, including a surrogateescape byte, is passed unchanged.prompt.txtof a run whose instruction has such an escape is written where the run stopped before. A package name with a lone surrogate now gets apackage-not-foundwarning wherecatalog check --onlinestopped.Open PRs on the same files merge apart: #776 moves code out of
process.pyand edits itsprocess_ioimport block (this PR adds one import line below the typing import and changes_checkand the module doc); #848 and #873 changedocker_solver.pyinrun,_build_image,_start_containerand the imports from line 75 on (this PR changes thedokimiimport at line 60 and theprompt.txtwrite in_claude_args).Testing
On a Linux review host, at
c4bd8fa(onmaina98a0e7):make ci-local: 36 passed, 0 failed, 14 skipped (the Docker external-catalog smoke, the pinned-client fidelity install,test-latest, the other-Python matrix legs, conda-forge, fuzz, semgrep, docs deploy, pip-audit report). 22739 tests passed, 19 skipped; changed-line coverage 100% of 20 lines; mutation of changed lines: 3 of 3 killed.prompt_txtcases oftests/test_docker_solver_harness.py, which now also assert the-pargument equals the saved text, fail on the previous head9131571(3 failed) and pass here.main's code the new tests fail:tests/test_process_argv.py5 failed, 1 passed (the\udcffcase passes onmainby design);tests/test_online_package_surrogate.py6 failed; the threeprompt_txtcases oftests/test_docker_solver_harness.py3 failed. All withUnicodeEncodeErroron the\ud800or\udcffinput, or the pair left unjoined.argv_textreplacing with""(the\ud800case), replacing every surrogate (the\udcff-beside-\ud800case), not joining pairs (the pair case and the fake runner),Completed.argvholding the caller's text (test_completed_argv_holds_the_text_sent);prompt.txtwithoutjoined_pairs(pair) and withouterrors="replace"(\ud800,\udcff);_package_findingquoting npm or pypi withouterrors="replace", withoutjoined_pairs, and with the message built from{name}instead of{name!r}.os.fsencodedoes not raise), the Docker-backed smoke (--with-docker), semgrep.Open questions
None.
Revert
A plain revert restores the old behaviour (the three
UnicodeEncodeErrors). Nothing is stored.Checklist
pytestpassesruff,mypy, and schema checks passAGENTS.mdraise/exit and test names the doc, contract or issue it traces to.github/.licenserc.yaml)