Skip to content
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,4 @@
- **A matrix cell name with a lone surrogate no longer stops the plugin-delta pairing** — the pair suffix of the delta artifact in `eval/matrix_rollup.py` is hashed with `surrogatepass`, so a JSON escape such as `"n\ud800"` or `"n\udcff"` in the matrix config gets its own suffix, and the suffix of a name that encodes as UTF-8 is unchanged.
- **A symbol with a non-UTF-8 byte no longer stops `task_id` of the graph tasks** — the hash of the symbols in `repos/graph_tasks.py` uses their UTF-8 bytes (`surrogateescape`); the id of a UTF-8 symbol set is unchanged.
- **A bundle file whose name is not UTF-8 no longer stops `Bundle.measured_digest`** — the digest in `novelty/bundle.py` is taken over the UTF-8 bytes (`surrogateescape`) of the path rows; the digest of a UTF-8 path is unchanged.
- **A `\udXXX` escape in a loaded JSON or YAML file no longer stops the digests of the parsed data** — `item_digest` and `assignment_digest` (leak gate), `graph_digest` (eval coverage), `expectations_digest` (conformance), `response_cache_key` and the Harbor export's content key encode with `surrogatepass`, so a lone surrogate gets a digest of its own and every digest that encoded before is unchanged. A terminal-bench task whose `task.yaml` metadata holds such an escape still stops the Harbor export when it writes `task.toml`.
- **A `\udXXX` escape in a loaded JSON or YAML file no longer stops the digests of the parsed data** — `item_digest` and `assignment_digest` (leak gate), `graph_digest` (eval coverage), `expectations_digest` (conformance), `response_cache_key` and the Harbor export's content key encode with `surrogatepass`, so a lone surrogate gets a digest of its own and every digest that encoded before is unchanged.
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
### Fixed

- **A skill-reach catalogue case whose name holds a lone surrogate no longer stops `build_fixture`** — a `\ud800` or `\udcff` escape in a case `name` raised `UnicodeEncodeError` when the case's `SKILL.md` was written. The file is written with the lone surrogate replaced by `?`, so it stays valid UTF-8 and the run goes on; a JSON-style pair such as `\ud83d\ude00` is joined into its character first; the file of a name that encodes as UTF-8 is unchanged.
- **A terminal-bench task with a lone surrogate in its `task.yaml` no longer stops the Harbor export** — a `\udcff` or `\ud800` escape in the metadata or the instruction raised `UnicodeEncodeError` when `harbor_view` wrote `task.toml` or `instruction.md`. Both files are written with the lone surrogate replaced by `?`, so they stay valid UTF-8; a JSON-style pair such as `\ud83d\ude00` is joined into its character first; the files of a task that encodes as UTF-8 are unchanged.
- **An input string with a lone surrogate no longer stops `process.run`** — a prompt built from a question or an artifact text with a `\ud800` JSON escape raised `UnicodeEncodeError` before the child started. The string is sent to the child's stdin with the lone surrogate replaced by `?`, so the child gets valid UTF-8; a JSON-style pair such as `\ud83d\ude00` is sent as its character; `FakeRunner` and `reply` of the test adapter encode the same way. The bytes of a string that encodes as UTF-8 are unchanged.
- **A system prompt with a lone surrogate no longer stops `llm_cli.call`** — the system prompt file of a CLI call was written with a strict UTF-8 encode, so a `\ud800` escape in an imported question's system message raised `UnicodeEncodeError` before the child started. It is written with the lone surrogate replaced by `?`; a JSON-style pair such as `\ud83d\ude00` is joined into its character.
- **A query or library id with a lone surrogate no longer stops `Context7Client.get_library_docs` on a cache miss** — `urllib.parse.quote(query)` raised `UnicodeEncodeError` for a `\ud800` or `\udcff` escape in a question, and a library id with one failed when the request was built. Both now go into the URL with the lone surrogate as `%3F` (a JSON-style pair such as `\ud83d\ude00` as its UTF-8 percent-encoding); the library id is percent-encoded for the URL path (letters, digits, `/` and the path characters `%:@!$&'()*+,;=-._~` stay as they are), and the cache key still tells two such queries apart. The URL of a query or id that encodes as UTF-8 and holds only those characters is unchanged.
7 changes: 5 additions & 2 deletions dokimi/dokimi/harnesses/harbor_export.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@

import yaml

from dokimi.process_io import joined_pairs

from . import tbench_v1
from .tasks import FORMAT_HARBOR, FORMAT_TBENCH_V1, TaskSpec

Expand Down Expand Up @@ -137,8 +139,9 @@ def export(task: TaskSpec, dest: Path) -> Path:
if script.is_file():
script.chmod(0o755)
instruction = task.instruction if task.instruction.endswith("\n") else task.instruction + "\n"
(dest / "instruction.md").write_text(instruction, encoding="utf-8")
(dest / "task.toml").write_text(_task_toml(task), encoding="utf-8")
# A `😀` escape is one character; a lone surrogate is written as `?`, so the file stays valid UTF-8.
(dest / "instruction.md").write_text(joined_pairs(instruction), encoding="utf-8", errors="replace")
(dest / "task.toml").write_text(joined_pairs(_task_toml(task)), encoding="utf-8", errors="replace")
return dest


Expand Down
3 changes: 2 additions & 1 deletion dokimi/dokimi/llm_cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@

from dokimi import process
from dokimi.metrics.usage import UsageRecord
from dokimi.process_io import joined_pairs

#: Variables a Claude Code session sets for its own children, plus the API key. Dropped, so
#: a call made from inside a session is a session of its own and runs on the CLI's login.
Expand Down Expand Up @@ -431,7 +432,7 @@ def call(provider: str, *, model: str, prompt: str, system: Optional[str] = None
# A private temp dir (mkdtemp, mode 0700) for the one call: the system prompt goes by file.
with tempfile.TemporaryDirectory(prefix="dokimi-cli-") as tmp:
system_file = Path(tmp) / "system-prompt.txt"
system_file.write_text(system or DEFAULT_SYSTEM, encoding="utf-8")
system_file.write_text(joined_pairs(system or DEFAULT_SYSTEM), encoding="utf-8", errors="replace") # lone surrogate: `?`
try:
done = (runner or process.run)(cli.build_argv(model, str(system_file)), timeout=timeout,
env=_call_env(cli, cwd), cwd=cwd, input=prompt, capture_limit=None)
Expand Down
13 changes: 9 additions & 4 deletions dokimi/dokimi/mcp/context7.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@
import hashlib
import logging

from dokimi.process_io import joined_pairs

from . import MCPClient, MCPConnectionError, MCPLibraryNotFoundError
from .utils import read_cache_text, write_cache_text

Expand Down Expand Up @@ -153,20 +155,23 @@ def get_library_docs(
# Website IDs: no slash + (underscore OR 'github_io' pattern)
# This avoids misclassifying GitHub repos with underscores (e.g., some_org/repo)
is_website = "/" not in library_id and ("_" in library_id or "github_io" in library_id)
# A surrogate pair is one character, a lone surrogate is %3F.
path_id = urllib.parse.quote(joined_pairs(library_id), safe="/%:@!$&'()*+,;=-._~", errors="replace")
topic = urllib.parse.quote(joined_pairs(query), errors="replace")

if is_website:
# Website format: websites/intelpython_github_io_dpnp
url = (
f"{self.endpoint}/websites/{library_id}/llms.txt"
f"{self.endpoint}/websites/{path_id}/llms.txt"
f"?tokens={max_tokens}"
f"&topic={urllib.parse.quote(query)}"
f"&topic={topic}"
)
else:
# GitHub repo format: uxlfoundation/oneTBB (or some_org/repo)
url = (
f"{self.endpoint}/{library_id}/llms.txt"
f"{self.endpoint}/{path_id}/llms.txt"
f"?tokens={max_tokens}"
f"&topic={urllib.parse.quote(query)}"
f"&topic={topic}"
)

logger.info(f"Fetching from Context7: {library_id} query={query[:50]}...")
Expand Down
12 changes: 6 additions & 6 deletions dokimi/dokimi/process.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,8 @@
* **The environment is a stated policy.** There is no default ``env``: a call
site passes :meth:`Env.inherit` (the parent's environment plus overrides) or
:meth:`Env.only` (an allow-list plus overrides).
* **stdin is closed** (``/dev/null``) unless ``input`` is given, so a child that
reads stdin gets EOF instead of the caller's terminal.
* **stdin is closed** (``/dev/null``) unless ``input`` is given (a child gets EOF, not the caller's terminal).
A ``str`` input is sent as UTF-8, a lone surrogate as ``?``: pass bytes for a name from ``os.fsdecode``.

Errors: a command that cannot be started (missing binary, missing ``cwd``, no
permission) raises :class:`OSError`; a non-zero exit never raises. After
Expand All @@ -55,7 +55,7 @@

from __future__ import annotations

from .process_io import PipeReader, PipeReaderOwner, path_text
from .process_io import PipeReader, PipeReaderOwner, path_text, utf8_bytes

import dataclasses
import os
Expand Down Expand Up @@ -187,7 +187,7 @@ def run(
"""Run ``argv`` (no shell) to completion or ``timeout`` seconds; see the module doc."""
args = _check(argv, timeout, capture_limit, term_grace)
preexec = _limits_preexec(limits) if limits else None
data = input.encode("utf-8") if isinstance(input, str) else input
data = utf8_bytes(input) if isinstance(input, str) else input
with _Spool(stdout_to) as out_file, _Spool(stderr_to) as err_file:
start = time.monotonic()
try:
Expand Down Expand Up @@ -743,7 +743,7 @@ class Call:
def reply(stdout: str | bytes = "", stderr: str | bytes = "", returncode: int = 0, *,
timed_out: bool = False) -> Completed:
"""A :class:`FakeRunner` answer (``argv`` and timing are filled in from the call)."""
enc = (lambda s: s.encode("utf-8") if isinstance(s, str) else s)
enc = (lambda s: utf8_bytes(s) if isinstance(s, str) else s)
return Completed((), TIMEOUT_RC if timed_out else returncode, enc(stdout), enc(stderr), timed_out)


Expand Down Expand Up @@ -774,7 +774,7 @@ def __call__(
term_grace: float = 0.0,
) -> Completed:
args = _check(argv, timeout, capture_limit, term_grace)
data = input.encode("utf-8") if isinstance(input, str) else input
data = utf8_bytes(input) if isinstance(input, str) else input
path = (lambda p: Path(p) if p is not None else None)
call = Call(tuple(args), path(cwd), env.resolve(), timeout, data, capture_stdout, limits,
capture_limit, path(stdout_to), path(stderr_to), term_grace)
Expand Down
11 changes: 11 additions & 0 deletions dokimi/dokimi/process_io.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,17 @@
_POSIX = sys.platform != "win32"


def joined_pairs(text: str) -> str:
"""*text* with each high surrogate followed by a low one (a JSON ``\\ud83d\\ude00`` escape, which PyYAML and ``json``
load as two code points) joined into its character; a lone surrogate stays, for ``errors="replace"`` to write as ``?``."""
return text.encode("utf-16-le", "surrogatepass").decode("utf-16-le", "surrogatepass")


def utf8_bytes(text: str) -> bytes:
"""*text* as UTF-8 with its surrogate pairs joined (:func:`joined_pairs`) and a lone surrogate sent as ``?``."""
return joined_pairs(text).encode("utf-8", "replace")


def path_text(raw: bytes) -> str:
"""A file name git printed, as the path the file system opens: on POSIX a non-UTF-8 byte stays one
(``os.fsdecode``), on Windows it becomes U+FFFD. Every reader of a ``-z`` listing decodes a path this way."""
Expand Down
7 changes: 4 additions & 3 deletions dokimi/dokimi/repos/skill_reach.py
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@
import yaml

from dokimi import process
from dokimi.process_io import joined_pairs
from dokimi.utils import key_bytes, retry_writable

from .events import convert
Expand Down Expand Up @@ -284,12 +285,12 @@ def build_fixture(cases: list[Case], profile: HarnessProfile, roots: dict[str, P
continue
taken[dest] = case.id
tok, body_tok = token(salt, case.id, "description"), token(salt, case.id, "body")
text = skill_text(case, tok, body_tok)
text = joined_pairs(skill_text(case, tok, body_tok)) # a `😀` escape is one character; a lone surrogate is `?`
link_kind = None
if case.link == "symlink":
target = outside / f"case-{n}" / case.dirname # the index, not the id: an id is not a path
target.mkdir(parents=True)
(target / "SKILL.md").write_bytes(text.encode())
(target / "SKILL.md").write_bytes(text.encode("utf-8", "replace"))
dest.parent.mkdir(parents=True, exist_ok=True)
try:
os.symlink(target, dest, target_is_directory=True)
Expand All @@ -299,7 +300,7 @@ def build_fixture(cases: list[Case], profile: HarnessProfile, roots: dict[str, P
link_kind = "symlink"
else:
dest.mkdir(parents=True, exist_ok=True)
(dest / "SKILL.md").write_bytes(text.encode())
(dest / "SKILL.md").write_bytes(text.encode("utf-8", "replace"))
fixture.placed.append(Placed(case, scope, rel, tok, body_tok, link_kind))
return fixture

Expand Down
11 changes: 11 additions & 0 deletions dokimi/tests/test_llm_cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,17 @@ def test_the_call_is_isolated_and_the_prompt_goes_on_stdin(runner, monkeypatch):
assert "--system-prompt-file" in second.argv and runner.system_texts[1] == llm_cli.DEFAULT_SYSTEM


@pytest.mark.parametrize(("system", "written"), [
pytest.param("sys \ud800", "sys ?", id="lone-high"),
pytest.param("sys \udcff", "sys ?", id="lone-low"),
pytest.param("sys \ud83d\ude00", "sys \U0001f600", id="json-pair"),
])
def test_a_surrogate_in_the_system_prompt_is_written_as_utf8_not_raised(runner, system, written):
runner.answers.append(reply(_ok()))
llm_cli.call("claude-cli", model="sonnet", prompt="q", system=system)
assert runner.system_texts == [written]


def test_a_long_system_prompt_goes_in_a_file_not_an_argument(runner):
# One argument is limited to 128 KiB on Linux; a system prompt of this size must not reach argv.
system = "s" * 150_000
Expand Down
28 changes: 28 additions & 0 deletions dokimi/tests/test_mcp_context7.py
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,34 @@ def test_get_library_docs_with_cache(self, mock_get, tmp_path):
assert docs2[0]["content"] == content
assert mock_get.call_count == 1 # No additional HTTP call

@patch('httpx.get')
def test_get_library_docs_sends_a_lone_surrogate_in_the_query_as_a_question_mark(self, mock_get, tmp_path):
mock_get.return_value = Mock(text="Documentation about parallel_for. " * 4)
client = Context7Client(cache_dir=tmp_path / "cache")
for query in ("n\ud800", "n\udcff"):
assert not client.get_library_docs("org/x", query)[0]["cached"]
assert mock_get.call_args[0][0].endswith("&topic=n%3F")
assert mock_get.call_count == 2 # two queries that send the same URL keep their own cache entry

@patch('httpx.get')
@pytest.mark.parametrize(("library_id", "path"), [
("org/n\ud800", "org/n%3F/llms.txt"),
("n\udcff_x", "websites/n%3F_x/llms.txt"),
("org/a-b.c~d", "org/a-b.c~d/llms.txt"),
("org/x%20y:z", "org/x%20y:z/llms.txt"),
])
def test_get_library_docs_sends_a_lone_surrogate_in_the_library_id_as_a_question_mark(
self, mock_get, tmp_path, library_id, path):
mock_get.return_value = Mock(text="Documentation about parallel_for. " * 4)
Context7Client(cache_dir=tmp_path / "cache").get_library_docs(library_id, "n\ud800")
assert mock_get.call_args[0][0].endswith("/" + path + "?tokens=8000&topic=n%3F")

@patch('httpx.get')
def test_get_library_docs_sends_a_json_surrogate_pair_as_one_character(self, mock_get, tmp_path):
mock_get.return_value = Mock(text="Documentation about parallel_for. " * 4)
Context7Client(cache_dir=tmp_path / "cache").get_library_docs("org/n\ud83d\ude00", "q\ud83d\ude00")
assert mock_get.call_args[0][0].endswith("/org/n%F0%9F%98%80/llms.txt?tokens=8000&topic=q%F0%9F%98%80")

@patch('httpx.get')
def test_get_library_docs_404(self, mock_get):
"""Test handling of library not found (404)."""
Expand Down
37 changes: 37 additions & 0 deletions dokimi/tests/test_process.py
Original file line number Diff line number Diff line change
Expand Up @@ -754,6 +754,43 @@ def handler(call):
assert fake.argvs() == [("ok", str(tmp_path)), ("slow",), ("missing",)]


@pytest.mark.parametrize("text", [pytest.param("n\ud800", id="d800"), pytest.param("n\udcff", id="dcff")])
def test_a_lone_surrogate_in_the_input_reaches_the_child_as_a_question_mark(text):
echo = "import sys; sys.stdout.buffer.write(sys.stdin.buffer.read())"
done = _py(echo, input=text)
assert done.returncode == 0 and done.stdout_bytes == b"n?"


@pytest.mark.parametrize("text", [pytest.param("n\ud800", id="d800"), pytest.param("n\udcff", id="dcff")])
def test_fake_runner_records_a_lone_surrogate_in_the_input_as_a_question_mark(text):
fake = FakeRunner()
assert fake(["x"], timeout=1, env=INHERIT, input=text).ok
assert fake.calls[0].input == b"n?"


@pytest.mark.parametrize("text", [pytest.param("n\ud800", id="d800"), pytest.param("n\udcff", id="dcff")])
def test_reply_encodes_a_lone_surrogate_in_the_output_as_a_question_mark(text):
assert reply(text, text).stdout_bytes == b"n?" and reply(text, text).stderr_bytes == b"n?"


_PAIR = "a\ud83d\ude00" # what PyYAML and json load for a JSON-style escape of U+1F600


def test_a_surrogate_pair_in_the_input_reaches_the_child_as_one_character():
echo = "import sys; sys.stdout.buffer.write(sys.stdin.buffer.read())"
assert _py(echo, input=_PAIR).stdout_bytes == "a\U0001f600".encode()


def test_fake_runner_records_a_surrogate_pair_in_the_input_as_one_character():
fake = FakeRunner()
assert fake(["x"], timeout=1, env=INHERIT, input=_PAIR).ok
assert fake.calls[0].input == "a\U0001f600".encode()


def test_reply_encodes_a_surrogate_pair_as_one_character():
assert reply(_PAIR, _PAIR).stdout_bytes == reply(_PAIR, _PAIR).stderr_bytes == "a\U0001f600".encode()


def test_input_larger_than_the_pipe_reaches_a_child_that_reads_late():
data = b"x" * 300_000
done = _py("import sys, time; time.sleep(0.6); sys.stdout.buffer.write(sys.stdin.buffer.read())",
Expand Down
25 changes: 25 additions & 0 deletions dokimi/tests/test_skill_reach.py
Original file line number Diff line number Diff line change
Expand Up @@ -784,6 +784,31 @@ def test_the_skill_name_loads_back_as_the_same_string(name):
assert yaml.safe_load(text.split("---")[1])["name"] == name


@pytest.mark.parametrize("link", ["", "symlink"])
@pytest.mark.parametrize("name", [pytest.param("n\udcff", id="dcff"), pytest.param("n\ud800", id="d800")])
def test_a_case_name_with_a_lone_surrogate_is_written_as_a_question_mark(tmp_path, name, link):
if link:
try:
os.symlink(tmp_path, tmp_path / "probe-link", target_is_directory=True)
except (OSError, NotImplementedError) as exc:
pytest.skip(f"cannot create a symlink here: {exc}")
profile = load_profile(str(_profile(tmp_path)))
roots = _roots(tmp_path)
case = Case(id="c", at="project:x", name=name, dirname="d1", link=link or None)
fixture = build_fixture([case], profile, roots, tmp_path / "outside", "s")
assert [p.case.id for p in fixture.placed] == ["c"]
text = (roots["project"] / "x/d1/SKILL.md").read_bytes().decode("utf-8")
assert "\n# n?\n" in text


def test_a_case_name_written_as_a_json_surrogate_pair_is_one_character(tmp_path):
profile = load_profile(str(_profile(tmp_path)))
roots = _roots(tmp_path)
case = Case(id="c", at="project:x", name="n\ud83d\ude00", dirname="d1")
build_fixture([case], profile, roots, tmp_path / "outside", "s")
assert "\n# n\U0001f600\n" in (roots["project"] / "x/d1/SKILL.md").read_bytes().decode("utf-8")


def test_a_skill_dir_above_the_workspace_is_refused(tmp_path):
(tmp_path / ".claude" / "skills").mkdir(parents=True)
workspace = tmp_path / "ws"
Expand Down
Loading
Loading