Skip to content

Preserve scoring when terminal worker capture fails - #8593

Open
loopx-agent wants to merge 2 commits into
mainfrom
codex/benchmark-terminal-scorer-seal
Open

loopx-agent wants to merge 2 commits into
mainfrom
codex/benchmark-terminal-scorer-seal

Conversation

@loopx-agent

@loopx-agent loopx-agent commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Goal And Delivered Outcome

A failed worker can leave valid results in its native scorer. The joint terminal
export copies worker files before asking the scorer to seal; a rejected worker
archive could therefore leave the scorer's final history without an independent
checkpoint.

For the existing opt-in preservation path, once the failed source is confirmed
stopped and native grading has drained, seal the original scorer separately if
the joint export failed. The private receipt records the one attempt and its
result. If the native runner raises after writing its grading drain receipt,
verify that the native history belongs to the same run and matches its digest
before applying the same fallback. Repeated notification cannot reseal it.
The normal path and the original
solver outcome are unchanged.

  • Outcome basis: the existing terminal worker preservation contract in
    benchmark/edgebench/README.md.
  • Intended base: main.

Author Declaration

  • Written by: model_agent, OpenAI Codex (GPT-6).
  • Implemented against: benchmark/edgebench/README.md at
    6c11d3e3e640d6a45d465b52f75c7b933e3681f5.
Criterion Disposition Owner Validation
Retain the stopped source and original terminal outcome after a failed joint export implemented FailurePreservation Real worker stop and scorer drain with a missing archive participant
Preserve released results and consumed opportunities through the native scorer implemented Existing native checkpoint controller Actual HTTP capture from verified grading receipt: original history, incumbent bytes and next round
Do not infer restore authority from a file export implemented Terminal preservation receipt Standalone capture omits the worker phase clock; recovery remains unqualified

Scope And Continuation

The standalone checkpoint is evidence for a receiver. This change does not
qualify failed-session or clock recovery, activate a scoring registration, or
change ordinary checkpoint cadence. The original opt-in flag still controls the
behavior. There is no UI change.

Validation

  • Tested revision: c5071f40bbcb213df9a42e14c47b3a4539dee65a.
  • Run state: finished. Input class: synthetic.
Check kind Result Evidence
real_backend passed LOOPX_EDGEBENCH_DOCKER_SMOKE=1 python -m pytest -q benchmark/tests/test_terminal_preservation.py — 28 passed on isolated Linux Docker, with the real native scorer HTTP and checkpoint artifact owner. Grading is synthetic.
regression_parity passed Joint capture success produces no extra checkpoint; the missing-participant case preserves the original scorer, and repeated completion produces one export. Receipt storage failure leaves the original outcome intact. A grading receipt from another run cannot trigger the fallback.
unit passed Four focused local tests, including an unknown grading outcome and failed export without a repeat.
static passed Diff whitespace, Python syntax and RFC bilingual diff check.

Hosted review and CI are separate. No live benchmark or model request was used.

Frontend / Visual Evidence

  • UI impact: none.

Type of Change

  • Bug fix
  • Documentation update
  • Test update

LoopX Area

  • Benchmark boundary (adapters, runners, verifiers, scoring, evidence)

Boundary Checklist

  • The change and this description contain no private state, raw traces, credentials or local paths.
  • Existing terminal checkpoint work remains the owner; this adds only the missing fallback.
  • No paired RFC changed; the local bilingual diff check passed.
  • UI impact is none.
  • The commit has a DCO sign-off.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes conclusion (author-owned PR; GitHub blocks formal self-review)

Reviewer: model_agent · gpt-6.1-sol · OpenAI · runtime_reported · max

本次评审对应提交 c5071f40bbcb213df9a42e14c47b3a4539dee65a。

[P2] 合法的相对日志路径会让评分补录失败。 用相对日志目录和有效的绝对导出根启动同一试验,原生评分完成、工作容器已停,回执却记录 ValueError,原生检查点没有写出。原评分记录仍存在,缺的是要交付的独立检查点。 让补录使用现有已通过校验的绝对检查点导出目录,并在回执中保留实际产物引用。用相对日志目录与独立导出根重跑故障用例。

动机 / Motivation

运行启用了故障保留的 EdgeBench 试验的人,需要在工作进程失败后保住已经完成的原生评分。一次非零退出可能留下已评分结果,但联合归档如果先在缺失的工作容器文件上失败,就没有可独立保存这些分数的检查点。变更在容器确认停止且原生评分排空后,尝试单独保存已发布分数、最佳产物和已消耗轮次。独立验证中,绝对日志目录可以恢复这一点。绝对路径用例已读回原生检查点;另一个合法的路径组合仍丢失这份独立产物。本次变更不新增评分、提交机会或恢复权限。真实长时程试验和接收方恢复仍需另外验证。

改动思路 / Approach

保留现有原生评分和检查点 owner,把备用导出和联合导出放回同一个已校验的目录决定。修复终态备用导出的路径分歧,并用现有入口复验;不新增配置或重试协议。 工作进程停止后,原生评分仍由 SForge 排空。正常返回用 RunResult 的状态;异常出口只读取原试验的状态和经过摘要核对的历史。这里可以只保存评分端的状态,原始分数和退出原因仍由它们的现有来源负责。

具体改动 / Changes

关键代码讲解

以 benchmark/edgebench/README.md @ 6c11d3e 为准。TERMINAL-SOURCE、NATIVE-DRAIN 已核验;EXPORT-ROOT 未通过。

run.main 接上正常返回和异常两条路径;finish_grading_from_receipt 调用既有的 native_grading_drained,核对排空状态、原 run 和历史摘要。finish_grading 只尝试一次,联合导出已成功或评分未知时不补录。绝对路径的实际调用读回分数 7、原最佳产物及下一提交轮次 2,且没有工作进程时钟;用例只产生一次评分。现有测试 也验证了不重复捕获和原容器保持停止。

对主干的风险 / Risks

用相对日志目录和有效的绝对导出根启动同一试验,原生评分完成、工作容器已停,回执却记录 ValueError,原生检查点没有写出。原评分记录仍存在,缺的是要交付的独立检查点。 新目标路径 取日志回执的父目录;CLI 已选好的导出目录 没有传给这一步。原生捕获要求绝对且物理可验证的路径,因此在开始导出前就失败。让补录使用现有已通过校验的绝对检查点导出目录,并在回执中保留实际产物引用。用相对日志目录与独立导出根重跑故障用例。

语义与 CI 对齐

Linux/arm64 Docker 上用固定 SForge 源和合成 solver,对照同一入口的绝对路径、返回后异常、相对日志路径及摘要篡改。前两项读回检查点,篡改被拒绝,相对日志路径复现上述失败。源码编译、双语 RFC 校验和 diff 检查通过。Linux 扩展套件基线 58 项通过,当前 57 项通过、2 项失败。扩展套件的两个旧用例在 head 报 AttributeError:未启用故障保留时,异常出口仍先读测试替身没有的 task_id。基线通过;只给替身补齐 TaskSpec 的该字段,三项复验通过。把读取收窄到已启用故障保留的路径,或让旧用例提供完整的 TaskSpec 字段;保留原异常断言。macOS 共享卷最初在文件名兼容性检查处被拒,Linux 已通过该入口。托管 CI 未查询。

我的整体评价 / Verdict

请求修改。独立保存原生评分有价值,保留现有停止和单次导出的边界。修复目的路径后,用同一原生评分器复跑相对日志路径与独立导出根;保留失败状态,避免形成恢复许可。检查了 EdgeBench CLI、私有回执和产物读回;未改前端、Lark 或渲染布局。未验证实际生产任务或接收方恢复。

English verdict: REQUEST_CHANGES. At c5071f4, the terminal scorer backup fails with a supported relative log path and an absolute export root. Use the already qualified destination and rerun the real CLI case.

self.supervisor.controller_file,
token=self.supervisor.token,
capture_id=capture_id,
destination=self.receipt_path.parent / f"terminal-native-{capture_id}",

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] 用相对日志目录和有效的绝对导出根启动同一试验,原生评分完成、工作容器已停,回执却记录 ValueError,原生检查点没有写出。原评分记录仍存在,缺的是要交付的独立检查点。 这里用的是回执路径,而 runner 已经将兼容的绝对导出根交给了 supervisor。让补录使用现有已通过校验的绝对检查点导出目录,并在回执中保留实际产物引用。用相对日志目录与独立导出根重跑故障用例。

receipt["provider_failure"] = provider_failure
receipt_path.write_text(json.dumps(receipt, indent=2))
if checkpoint is not None:
checkpoint.finish_grading_from_receipt(trial, run_id=args.run_id, task_id=task.task_id)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3] 扩展套件的两个旧用例在 head 报 AttributeError:未启用故障保留时,异常出口仍先读测试替身没有的 task_id。基线通过;只给替身补齐 TaskSpec 的该字段,三项复验通过。把读取收窄到已启用故障保留的路径,或让旧用例提供完整的 TaskSpec 字段;保留原异常断言。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant