Skip to content

fix: a lone surrogate in an argv entry, a task instruction or a package name no longer raises UnicodeEncodeError - #886

Open
antonwolfy wants to merge 4 commits into
intel:mainfrom
antonwolfy:fix/lone-surrogate-leftovers
Open

antonwolfy wants to merge 4 commits into
intel:mainfrom
antonwolfy:fix/lone-surrogate-leftovers

Conversation

@antonwolfy

@antonwolfy antonwolfy commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Three leftovers of #863, which fixed a lone surrogate (from a \ud800 JSON/YAML escape) raising UnicodeEncodeError at other sites. On main:

Issue Input Output on main
#859 process.run(['echo', 'n\ud800'], timeout=5, env=process.Env.inherit()) UnicodeEncodeError: 'utf-8' codec can't encode character '\ud800' in position 1: surrogates not allowed, from subprocess, before the child starts. 'n\udcff' does not raise (the child gets byte 0xff) and is kept that way
#860 DockerSolverHarness._claude_args with the instruction 'n\ud800' or 'n\udcff' the same UnicodeEncodeError, from write_text of prompt.txt
#862 online_checks._package_finding(None, 'pypi', 'n\ud800') (also 'n\udcff', and npm) the same UnicodeEncodeError, from quote(name), outside the try that turns probe errors into outcomes

Fix taken for each, as #863 did it (joined_pairs, then a replacement of what cannot be encoded), so the user's output is kept and nothing is refused or skipped:

Closes #859
Closes #860
Closes #862

The same cause at two more sites, found by the search for other strict encodes, is tracked in #884 (clip_output of a trace's command output) and #885 (the mail subject of a replayed pull request).

Changes

  • process_io.argv_text (next to utf8_bytes): joins surrogate pairs, then replaces each character os.fsencode cannot encode with ?; U+DC80..U+DCFF stay on POSIX. An entry os.fsencode accepts is returned as it was. process._check applies it to every entry, so run and FakeRunner both send and record the same text, and Completed.argv holds the text sent. The module doc of process.py states the argv rule next to the stdin rule (two paragraphs of the doc were joined to keep the file at its 800-line cap). Windows: os.fsencode encodes with surrogatepass there and does not raise, so only the pair joining applies.
  • DockerSolverHarness._claude_args normalises the prompt with process_io.utf8_bytes(...).decode() and uses that text for both prompt.txt and -p; the harness test asserts args[:2] == ["-p", saved].
  • online_checks._package_finding quotes joined_pairs(name) with errors="replace" for both registries. The finding's message uses {name!r}, which escapes a surrogate, so it stays printable.
  • Tests in new files, since tests/test_process.py and tests/test_catalog_online.py are at the 1000-line cap: tests/test_process_argv.py, tests/test_online_package_surrogate.py; the docker test is in tests/test_docker_solver_harness.py. No existing test is changed.
  • Changelog fragment (Fixed).

Risk

An argv entry with a lone surrogate or a 😀 escape pair reaches the child changed (? or the character), where it raised before; Completed.argv for such a call holds the changed text. Every entry os.fsencode already accepted, including a surrogateescape byte, is passed unchanged. prompt.txt of a run whose instruction has such an escape is written where the run stopped before. A package name with a lone surrogate now gets a package-not-found warning where catalog check --online stopped.

Open PRs on the same files merge apart: #776 moves code out of process.py and edits its process_io import block (this PR adds one import line below the typing import and changes _check and the module doc); #848 and #873 change docker_solver.py in run, _build_image, _start_container and the imports from line 75 on (this PR changes the dokimi import at line 60 and the prompt.txt write in _claude_args).

Testing

On a Linux review host, at c4bd8fa (on main a98a0e7):

  • make ci-local: 36 passed, 0 failed, 14 skipped (the Docker external-catalog smoke, the pinned-client fidelity install, test-latest, the other-Python matrix legs, conda-forge, fuzz, semgrep, docs deploy, pip-audit report). 22739 tests passed, 19 skipped; changed-line coverage 100% of 20 lines; mutation of changed lines: 3 of 3 killed.
  • The three prompt_txt cases of tests/test_docker_solver_harness.py, which now also assert the -p argument equals the saved text, fail on the previous head 9131571 (3 failed) and pass here.
  • On main's code the new tests fail: tests/test_process_argv.py 5 failed, 1 passed (the \udcff case passes on main by design); tests/test_online_package_surrogate.py 6 failed; the three prompt_txt cases of tests/test_docker_solver_harness.py 3 failed. All with UnicodeEncodeError on the \ud800 or \udcff input, or the pair left unjoined.
  • Mutants, each failing the test that pins it: argv_text replacing with "" (the \ud800 case), replacing every surrogate (the \udcff-beside-\ud800 case), not joining pairs (the pair case and the fake runner), Completed.argv holding the caller's text (test_completed_argv_holds_the_text_sent); prompt.txt without joined_pairs (pair) and without errors="replace" (\ud800, \udcff); _package_finding quoting npm or pypi without errors="replace", without joined_pairs, and with the message built from {name} instead of {name!r}.
  • Not run: Python 3.11, 3.12 and 3.14, Windows (the argv tests are skipped there, where os.fsencode does not raise), the Docker-backed smoke (--with-docker), semgrep.

Open questions

None.

Revert

A plain revert restores the old behaviour (the three UnicodeEncodeErrors). Nothing is stored.

Checklist

  • Tests added/updated and pytest passes
  • ruff, mypy, and schema checks pass
  • Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
  • Self-reviewed per "Before opening a PR" in AGENTS.md
  • Each new raise/exit and test names the doc, contract or issue it traces to
  • No ad-hoc run artifacts committed
  • New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

…s ? instead of raising (intel#859)

process.run passed argv entries to subprocess unchanged, whose os.fsencode raises UnicodeEncodeError for a lone high surrogate (a JSON/YAML \ud800 escape in a prompt given as `claude -p <prompt>`), before the child starts.

_check now sends each entry through process_io.argv_text: surrogate pairs are joined as on stdin, and each character os.fsencode cannot encode becomes ?. The surrogateescape range U+DC80..U+DCFF stays on POSIX, so a non-UTF-8 file name from os.fsdecode still names the same file. An entry os.fsencode accepts is unchanged. Completed.argv, and the argv FakeRunner records, hold the text sent.

Closes intel#859

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
…with a lone surrogate as ? instead of raising (intel#860)

DockerSolverHarness._claude_args wrote prompt.txt with a strict UTF-8 encode, so a task instruction with a \ud800 or \udcff escape raised UnicodeEncodeError.

It now writes joined_pairs(prompt) with errors="replace", as the Harbor export does: an escape pair becomes its character and a lone surrogate becomes ?, which is what the child gets from the argv entry (intel#859).

Closes intel#860

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
…instead of raising (intel#862)

online_checks._package_finding built the registry URL with a strict quote(), so a package name with a \ud800 or \udcff escape raised UnicodeEncodeError, outside the try that turns probe errors into outcomes.

It now quotes joined_pairs(name) with errors="replace", as Context7Client.get_library_docs does: a lone surrogate becomes %3F, the registry answers 404 and the usual package-not-found warning follows. The check is not skipped.

Closes intel#862

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@exai-bot

exai-bot commented Nov 17, 2026 •

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 8 selected item(s).

CI at c4bd8fa: 26 passing, 3 skipped.

Review details
  • Commit reviewed: c4bd8fa against merge base a98a0e7
  • Coverage: 8 files in 1 groups
  • Engine: OpenCodeReview v1.13.5-exai.4.8f73eada, us.openai.gpt-6.1-sol
  • Usage: 159k tokens (104k from cache), 44s
  • Tool calls: 19 (12 file_read, 5 code_search, 1 ci_status, 1 local_check_results)
  • Lone-surrogate handling in online checks and process execution: 8 files

@codecov-commenter

codecov-commenter commented Nov 17, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.79%. Comparing base (a98a0e7) to head (c4bd8fa).
⚠️ Report is 4 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #886      +/-   ##
==========================================
- Coverage   95.79%   95.79%   -0.01%     
==========================================
  Files         429      432       +3     
  Lines       56842    57220     +378     
  Branches     9787     9858      +71     
==========================================
+ Hits        54452    54812     +360     
- Misses       1487     1497      +10     
- Partials      903      911       +8     
Files with missing lines Coverage Δ
dokimi/dokimi/external/online_checks.py 96.38% <100.00%> (+0.01%) ⬆️
dokimi/dokimi/harnesses/docker_solver.py 97.97% <100.00%> (+<0.01%) ⬆️
dokimi/dokimi/process.py 90.51% <100.00%> (+0.02%) ⬆️
dokimi/dokimi/process_io.py 97.33% <100.00%> (+0.55%) ⬆️

... and 23 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…rompt.txt

Review point: with the instruction "n\udcff", prompt.txt held "n?" while `-p` carried the original text, which argv_text keeps as byte 0xff. The prompt is now normalised once (pairs joined, a lone surrogate a "?") and that text is both written and sent. The harness test asserts the sent -p argument equals the saved text.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants