Repository navigation
Recover exact returned native Host failures - #8509
Conversation
Bind an explicit scheduler retry to the original failed attempt, process return, session, and matching native error. Observe the retained native store in isolation and continue through the existing typed retry and admission rules. Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Require one failed cursor from the provider-selected rollout inside the observed return window. Confirm the app-server tail names that cursor before accepting the original CLI error proof. Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
loopx-agent
left a comment
There was a problem hiding this comment.
Approval conclusion (author-owned PR; GitHub blocks formal self-approval)
Reviewer: model_agent · gpt-6.1-sol · OpenAI · runtime_reported · max
动机
操作者要在一次 benchmark 的原 Codex 会话上继续工作:Host 已返回明确的失败,外部原因已经处理,剩余阶段预算也已经获准。
旧调度器只允许显式续跑 quota 失败;即使留有原会话、已退出的进程和完整失败回执,类型为 unknown 的失败也会卡住。新路径可以在同一会话继续,但先验证原进程、原 Turn 和原错误是一件事。
在相同 SQLite 用例中,主干拒绝这次显式续跑;本 head 能在原会话继续,后来又失败时保存新失败,重放旧请求不会清掉它。操作者要处理后来那次失败,必须用它自己的摘要。
这次不新增自动重试,不改变模型、账号、Stop 或预算授权,也不把失败 Turn 当成已完成 checkpoint。 验证覆盖源码调度路径和本机 Codex 的真实回执读取;正式 worker 镜像与 benchmark 的科学验收另行确认。
改动思路
只有错误摘要不能区分同一会话里两次相同失败;把原进程返回窗口与 provider 选出的 Turn 对上,才能保留原会话恢复并拒绝替代。资格仍由现有调度器、共享重试规则和当前准入决定。 本 PR 只扩展现有 benchmark 调度命令的显式失败续跑,保留原 quota 路径、失败状态、attempt 和时间上限。
用合入前 docs/reference/protocols/loopx-turn-v0.md @ d6f3dcba2594be71128c6cf70bd8ac0ee8d2263c 的 --retry-failed-turn、loopx_turn_host_failure_v0、host_outcome_unknown 核对:必须显式请求并受原 Session Binding 和重试预算约束;未知是否返回的执行不得重放。原 benchmark/README.md 只支持已返回的 quota 失败,本次有意扩展该操作,保留其原身份、原 phase 记录和有限预算条件。
具体改动
公开补丁与实际目标的六文件差异一致(+444/−43);本地组合树保留主干后续改动。改动跨了三个边界:调度器确认调用者选择的原失败和原 phase;worker 检查原 Wake 的完整退出和 terminal error;Codex provider 读取保留历史并对上精确 Turn。附加回执只存摘要与指针,不复制错误正文。
关键代码讲解
resume_state_file和read_failed_host:只在传入原失败摘要时进入续跑;复用共享重试资格和当前 Goal/会话/账号准入。精确重放沿用原 phase,保留该 phase 后来的失败。_returned_failure_witness:绑定原 Wake、stdout/stderr、同一会话和退出前最后的失败;用首次观测到进程已返回的时间为上界,避免把清理期间的新 Turn 纳入。native_failed_turn与read_failed_turn_cursor:在隔离副本上读取原生历史,并以原物理 rollout 的精确起止记录绑定 ID,再对照错误和未完成效果。重复读取不会改写原失败,后来的同错误 Turn 也不能补位。
对主干的风险
未传摘要时,两边都会保留失败且不会自动启动下一次 Host。旧 quota 与普通入口 9 项在主干通过,本 head 的 78 项覆盖同一组;组合树另过 6 项,含原退出边界和 Stop 在准备后生效。SQLite 测试使用脚本 Host 走真实 worker、进程与持久状态;我另用本机 Codex 0.167.0-alpha.3.2 和回环响应实际生成失败,原失败通过,错误或时间窗不匹配被拒绝。同一会话后一次完全相同错误也被拒绝;读回只访问原生历史,没有新增模型请求或改动源文件。Ruff、DCO、6 项直接和 19 项选定自动检查通过;benchmark 维护者门仍待独立满足。
我的整体评价
批准本次扩展。原会话在有确切证据时可以继续,证据不齐时保留失败;未把回执摘要当成新的执行权限。变更只涉及 benchmark 调度 CLI、Codex 回执读取、测试和 benchmark/README.md,没有变更 App、Lark 或其他渲染界面。 没有验证付费 benchmark、正式 worker 镜像或 Windows 端到端;benchmark 维护者检查仍是独立合入门。
English verdict: APPROVE — Explicit recovery preserves the original session, failure and phase budget. Matched scheduler runs and a real loopback Codex check reject a later Turn with the same error. Benchmark maintainer review remains separate.
Problem and outcome
The benchmark scheduler's explicit failed-state retry could recover a returned quota failure. A native Host can also return a complete failed Turn whose typed kind is
unknown. The quota-only guard stranded that session even after an operator resolved the underlying issue and supplied the exact failed-state digest.The existing
--retry-failed-state-sha256entrypoint binds a non-quota retry to the original failed attempt, Wake receipt, process return, session and profile. It reads the provider-selected retained rollout and requires one failed native Turn whose start and completion records fit inside the original process window through its first observed return. The app-server tail must name that exact Turn and agree with the CLI's terminal error. Another Turn with the same error, an ambiguous timestamp, or a pending effect holds the retry.The patch reuses the existing native-store isolation path. The shared typed retry owner and ordinary phase, Goal and receiver allowance checks govern admission. The legacy quota fixture exercises the typed failed-tail proof.
Verification
4357fa360d, 31 focused tests passed across the native witness, explicit SQLite recovery, return-boundary rejection and legacy quota cases. The independent foreign-tail oracle rejected both a later-second and a same-second substitute after accepting the original Turn.d78064e0c8and the candidate fixture. The candidate recovered the same session and preserved a subsequent failure across replay. A broader run before the final timestamp-precision guard passed 111 tests with one local native fixture skip.