Repository navigation
fix(codex-app): read the Codex App CLI's stdout as UTF-8 - #8485
Conversation
The Codex App automation helper shells out to the shipped `loopx` CLI and parses its JSON. The CLI pins its own stdio to UTF-8 (loopx-project#6119), so its stdout bytes are UTF-8 whatever the host locale is, but these three `subprocess.run(text=True)` calls left the codec to the host code page. On a non-UTF-8 host the outcome is not a mojibake warning but a dead automation path. The host codec raises UnicodeDecodeError inside CPython's reader thread, which CPython swallows: `subprocess.run` returns normally with `stdout=None` instead of raising, so the `except json.JSONDecodeError` guards never run and `json.loads(None)` surfaces a bare TypeError. The same swallowed exception also reaches `_subprocess_failure_detail`, whose `value or ""` turned a stream that had actually carried the CLI's reason into "no diagnostic output" - the operator is told the CLI was silent when it was not. Pin UTF-8 on all three reads, which matches what the CLI pins on its side, and report an undecoded stream as undecoded rather than as empty. The two `json.loads` sites also grow an explicit `stdout is None` guard, since a decoder failure arrives as `None` rather than as a JSON decode error. The rows in tests/test_codex_app_apply_rrule_utf8_io.py drive the real helper functions against a real child process, because a faked CompletedProcess cannot reproduce a decoder failure in CPython's reader thread. Signed-off-by: GZY-SUPER-HACKER <162807803+GZY-SUPER-HACKER@users.noreply.github.com>
71dd39c to
a0bae06
Compare
loopx-agent
left a comment
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent · gpt-6.1-sol · OpenAI · runtime_reported · reasoning_effort=max
本头未发现 P0–P2 阻塞问题，结论 APPROVE。有一项 P3 测试建议：新 fixture 没有真正强制父进程使用非 UTF-8 解码器；本次已另用真实进程对照验证当前产品修复。
动机 / Motivation
使用 Codex App 离线自动化辅助脚本、且父进程采用非 UTF-8 编码的操作者。
同一份含中文的 CLI JSON，旧版按 GBK 解码失败，修复版读到原内容并继续；CLI 拒绝时仍停止并显示原原因。
独立真实子进程及隔离 SQLite/TOML 对照验证了解码、规则写入、拒绝和 ACK 次序；正常 UTF-8 路径的七类结果与基线一致。
不修改实际 App 自动化，不改变 scheduler/ACK 权限、等待、租约或配额；不认证原生 Windows 读线程、安装态或长期模型收益。
这是已接受的 CLI UTF-8 输出合同在父进程读取端的遗漏。child 自己固定输出编码，不能替 parent 选择解码器；保留旧行为会让中文结果在解析前失败，并掩盖原本有用的拒绝原因。修复无需操作者再设一次编码或修改返回命令，原有的一次脚本调用保持可用。
改动思路 / Approach
三个读取属于同一个本地 IO 边界，调度和写入权限仍由原 owner 持有，没有证据需要新的通用状态或路由机制。
本 PR 只修三个 CLI 读取及未解码诊断，实际进程/存储行为已验证；真实 App 采用和原生 Windows 另行验收。
具体做法是在三个已有 subprocess.run 调用固定 encoding="utf-8"，并在原有 _subprocess_failure_detail 内区分未解码的 None 与实际空串。两个 JSON consumer 在 json.loads 前拒绝 None，避免次生 TypeError。没有新增进程、配置、状态、通用 wrapper 或另一份调度决策。
不改代码仍保留可复现的解码失败；只改 child 的 locale 无法控制 parent；把整个 scripts/ 纳入所有编码扫描则是更大的维护任务，不是本修复的前置。这里保留直接 IO 调用的清晰边界，沿用既有错误、写入和 ACK owner。重构判断为 unnecessary；测试可在现有 fixture 中复用已存在的强制父 codec 模式。
具体改动 / Changes
精确 head a0bae06f93d1fdd9697ecbd8dedd4307b9ff828c，差异基线 9508f4de93628c97bce4e97f24000151e9188d36。完整两文件 +265/-1，其中生产脚本 +20/-1、新测试 +245/-0；未修改作者代码。
接受规格是修改前的 docs/reference/source-cli-entrypoint.md，spec_revision 9508f4de93628c97bce4e97f24000151e9188d36 的 User-visible compatibility；同时读回已合入的 #6119 及现有编码 guard。以下是本次审查条款标签：
- UTF8-output：implemented。原
pin_utf8_stdio()producer 保持 UTF-8，不把本地代码页传给下游。 - UTF8-reader：implemented。三个 consumer 现在显式使用同一编码；独立反例在不可变旧版实际失败、当前头通过。
- strict-json：implemented。非法 JSON 和 CLI 拒绝仍是失败，不静默修复为有效结果，也不提前写规则。
关键代码讲解
_heartbeat_task_body:103读出原task_body；错误状态、缺失流、非法 JSON 仍走退出边界。_load_scheduler_hint:387保留父 Turn、能力参数及 raw-hint 环境，只改变解码；main后续仍决定 apply/no-apply/dry-run。_run_ack:492继续执行原 argv，非零状态仍拒绝；中文原因不再因本地解码而丢失。
独立对照使用实际 child 输出 UTF-8、父 _text_encoding 分别为 UTF-8/GBK，在临时真实 SQLite/TOML 上从脚本 main 执行，并单独穿过 prompt helper。共28条观察：七类 UTF-8 base/head 结果逐项一致；GBK 下旧版六类在解码处失败，当前头取得正确 prompt、hint、规则读回或原拒绝。ASCII 非法 JSON 在两端仍拒绝。干跑及 hint 拒绝不写规则，no-apply 保留原 ACK 行为，ACK 失败仍保留原先已写规则的语义，没有虚构事务回滚。
对主干的风险 / Risks
P3：让回归 fixture 真正固定父解码器
_host_codec_is_the_locale_one 只删除环境变量；这不会把已经运行的 UTF-8 pytest 进程变成 GBK。独立 AST mutation 仅从三个 subprocess.run 删除 encoding、保留全部新诊断和测试，五条新测试仍全部通过。child 的编码及 codec.decode 的单独断言不能证明 parent 实际用了该 codec。
建议复用现有 tests/test_runtime_subprocess_utf8.py::test_goal_mode_cli_probe_survives_gbk_host_locale 的 subprocess._text_encoding monkeypatch，要求移除任一编码 pin 就让对应真实进程用例失败。这是持久测试覆盖的非阻塞改进；当前头已经由独立强制 codec 的真实读取、错误和存储对照证明，不把测试弱点写成已观察到的产品失败。
| 本轮独立检查 | 结果与范围 |
|---|---|
| 七个 affected/adjacent Python 模块 | 46 PASS，零 skip；包含同父 Turn、scheduler fallback 和身份断言。 |
| 实际进程与隔离存储对照 | 28条观察；七类 UTF-8 对照同义，GBK 下六个旧版缺陷可区分，新头 oracle 全通过。 |
| 编码移除 mutation | 新测试5 PASS，暴露上述灵敏度缺口；这不是“成功拦截 mutation”。 |
| Ruff / mypy / diff | 两个变更文件 Ruff PASS；配置17根 mypy PASS；准确差异检查 PASS。 |
| 原生 standard premerge | 4项直接检查加 Python compile 均 PASS，语义 smoke PASS，gate exit0；维护性 ratchet 原始 FAIL 被 native owner 列为 advisory，保留失败，未独立编造历史归因。 |
| 精确质量回执 | cqr_a9b1df584cf08c775454 record/verify valid；一项上述测试 warning，无 blocker。 |
初始私人 mutation driver 把另一处文本写入的编码行也计入，断言在修改前停止；改为只选择 AST 的三个 subprocess 调用后取得上述结果。首次质量记录的引用类型被拒绝，修正为当前 scope 引用后才取得有效回执。这些是审查器错误，不归因于 PR。自有临时目录均已移除，日志保留。没有读取、轮询或等待 GitHub CI。
我的整体评价 / Verdict
当前修复在原 IO owner 内完成了有用且可逆的结果，完整范围与问题相称；无需新的持久兼容分支、能力开关或 TypeScript 决策 owner。long_horizon=preserved，受测的原 Turn 参数、规则写入、拒绝、干跑及 ACK 次序不变；user_experience=improved，中文结果和失败原因可读，无新增用户输入或确认轮次。仓库经验用于判断真实 caller 和边界，没有把旧案例结论当作本头验收或 memory 收益。
已检查脚本及现有入口；差异只含进程解码、退出诊断和测试，没有渲染页面或布局变化。视觉未验证范围：none。实际 App/安装态、原生 Windows pipe-reader thread、跨平台进程启动和长期模型效果均未在本轮认证；这不是实际自动化已更新的声明。
无阻塞 finding；保留 P3 测试建议、ratchet 原始失败及上述资格边界。本次是 review 和本地检查结论，不授予合并、安装或新的调度权限。
English verdict: APPROVE at a0bae06. Independent real-process/store counterfactuals establish UTF-8 reading and preserved refusal/effect ordering;46focusedtests and native local gate pass. A nonblocking parent-codec fixture weakness and the raw advisory ratchet failure remain disclosed. Native Windows and actual App adoption are unqualified.
| gbk host gbk, it only stops UTF-8 from masking a missing pin. | ||
| """ | ||
|
|
||
| monkeypatch.delenv("PYTHONIOENCODING", raising=False) |
There was a problem hiding this comment.
[P3] 让 fixture 真正固定父进程解码器。这里删除环境变量不会改变已经启动的 UTF-8 pytest 父进程；仅在内存删除三个 subprocess 的 encoding 参数后，本模块仍为5 PASS。建议复用现有 test_goal_mode_cli_probe_survives_gbk_host_locale 的 subprocess._text_encoding monkeypatch，要求漏掉任一 pin 时对应真实子进程测试失败。当前产品已另经强制GBK的真实子进程/隔离存储对照验证，因此这是非阻塞覆盖建议。
Goal And Delivered Outcome
Outcome basis / optional anchor: no public issue, roadmap card or RFC id is invented for this repair. It is a self-contained bug fix in the Codex App automation helper, and its child-side contract is already merged: fix(cli): pin the shipped CLI's own stdio to UTF-8 #6119 (2026-10-10)
fix(cli): pin the shipped CLI's own stdio to UTF-8. This PR is the read-side counterpart of that contract.Goal/source and gap:
scripts/codex_app_apply_rrule.pyshells out to the shippedloopxCLI and parses its JSON. The CLI pins its own stdio to UTF-8 (loopx/entrypoint.py→pin_utf8_stdio()), so the bytes it emits are UTF-8 whatever the host locale is. Threesubprocess.run(..., text=True)calls in that helper pass noencoding, so the parent decodes with the host code page —cp936on a zh-CN Windows host. On such a host the outcome is not mojibake: the host codec raisesUnicodeDecodeErrorinside CPython's reader thread, CPython swallows a reader-thread exception, andsubprocess.runreturns normally withstdout=None. Two consequences follow.json.loads(None)raisesTypeError, notJSONDecodeError, so the existingexcept json.JSONDecodeErrorguards never run and the operator sees a bare traceback. And_subprocess_failure_detail'svalue or ""renders an undecoded stream exactly like a silent one, so a CLI that did print its refusal is reported as having printed nothing.Observable before → after, with the validation row that proves it: before,
_heartbeat_task_bodyand_load_scheduler_hintraiseTypeError: the JSON object must be str, bytes or bytearray, not NoneType, and_run_ackreportsloopx scheduler ACK failed (3): no diagnostic outputeven though the CLI wrote its reason to stdout. After, both payload helpers return their decoded values and a failing CLI keeps its reason (validation rows 1–2).Issue/task and intended base: no linked issue; intended base
main.Author Declaration
Implemented against
Specification and revision: the merged child-side contract fix(cli): pin the shipped CLI's own stdio to UTF-8 #6119 (
loopx/entrypoint.py→pin_utf8_stdio()), plus the invarianttests/test_runtime_subprocess_utf8.pyalready states in its docstring — "a single non-ASCII byte kills the pipe reader thread,result.stdoutbecomesNonewithreturncode == 0, and the caller reports a misleading secondary error." The scan roots of the two shipped guards areREPO_ROOT / "loopx"(tests/test_loopx_text_io_utf8.py,tests/test_runtime_subprocess_utf8.py), which is why a site underscripts/can survive them.Criteria:
loopx/entrypoint.py→pin_utf8_stdio()tests/test_cli_stdio_utf8.pyscripts/codex_app_apply_rrule.py:129,420,498tests/test_codex_app_apply_rrule_utf8_io.pyscripts/codex_app_apply_rrule.py→_subprocess_failure_detailtests/test_codex_app_apply_rrule_utf8_io.py::test_failure_detail_separates_undecoded_from_silenttests/test_codex_app_apply_rrule.pybefore changing anything. I reproduced the failure with a real child process (the parent keeps the host code page; the child reconfigure its stdout to UTF-8, which is the same contractpin_utf8_stdio()implements) rather than a fakedCompletedProcess, because a fake cannot raise inside the reader thread. I deliberately did not expand the guards' scan roots: doing that would immediately surface a dozen other undecided sites underscripts/and would turn this into "widen the guard plus fix everything", so it stays a separate change. I did not touch the CLI side, the automation install semantics, or the multi-host selection policy.Scope And Continuation
Completed scope and remaining work: complete within this scope — three CLI stdout reads and the diagnostic rendering that depends on them.
Unrelated but visible:
scripts/qualify-native-goal-release.pyreads the same CLI chain the same way, but its launcher is POSIX-only (#!/bin/sh+set -euo pipefail), and its CI locale is typically UTF-8, so the failure cannot be reproduced on this host. By the same discipline I used elsewhere, I do not claim it.Slice boundary / successor: independently testable and reversible. A succeeding guard expansion (adding
scripts/to the two scan roots) is the natural successor, but it needs its own census of the remaining sites first. This PR changes no product behavior beyond the decode boundary.Validation
a0bae06f9(rebased ontoorigin/main@461799597)unitpassedtests/test_codex_app_apply_rrule_utf8_io.py— 5 passed. The rows drive the real helper functions against a real child process that pins its stdout to UTF-8 under a locale codec the parent cannot decode.regression_paritypassedtests/test_codex_app_apply_rrule.py— 15 passed; one further test in that module fails on this host withOSError [WinError 193]and fails identically with the file reverted to upstream, so it is environmental and not introduced here.staticpassedruff checkclean on both changed files;git diff --checkcleanunitpassedtests/test_loopx_text_io_utf8.py,tests/test_runtime_subprocess_utf8.py,tests/test_cli_stdio_utf8.py— 9 passedreal_entrypointpassedTypeError ... NoneTypefrom the undecoded stream and_run_ackprintsno diagnostic outputWinError 193failure noted above is a host limitation, not a product signal. A guard expansion that would prevent the same shape from returning is deliberately out of scope.Frontend / Visual Evidence
Type of Change
LoopX Area
Technical Direction
Shared-authority RFC fixture impact
Boundary Checklist
.loopx/,.codex/goals/, and liveACTIVE_GOAL_STATE.md).python3 scripts/check_rfc_bilingual.py --base origin/main --head HEADpasses, or no paired RFC changed.none.Signed-off-bytrailer (git commit -s).