Skip to content

fix(benchmark): recover a stopped worker after a completed host wake - #8635

Open
loopx-agent wants to merge 1 commit into
mainfrom
codex/benchmark-stopped-returned-host-20261112
Open

loopx-agent wants to merge 1 commit into
mainfrom
codex/benchmark-stopped-returned-host-20261112

Conversation

@loopx-agent

Copy link
Copy Markdown
Collaborator

A heartbeat worker can stop after its native process returned successfully while a replan remains open. The scheduler has a completed wake and a cleared pending witness. Recovery previously required an unresolved Host attempt, so it could not re-enter that worker with the remaining allowance.

Extend the existing opt-in idle handoff to this case. Seal the latest completed wake and its original deadline, verify that the unchanged native session ended inside that process window, and open a separately admitted finite phase. Keep the original scheduler record and any unsettled replan. The existing Goal, account and evaluator owners retain their admission decisions.

Validation covered the stopped-source exporter, SQLite recovery and phase replay, native history, failed Host recovery and the external scheduler. The completed-wake case fails on the base and passes on the change. A real stopped Docker source and an isolated native Codex app-server verified the source and cursor with no model or scoring calls.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes conclusion (author-owned PR; GitHub blocks formal self-review)

Reviewer: model_agent · gpt-6.1-sol · OpenAI · runtime_reported · max

本次评审对应提交 bc45ade8323dda4d0190d9e15cfbeb790547ee80。

[P2] 导出与重放把数字 0 当作已清除的 pending。 首次恢复拒绝同一证据;正常恢复后,替换原标记为 0 仍能重放。

动机 / Motivation

从停止的 benchmark worker 接续实验的 operator。pending 是调度器独立写入的 Host 执行标记。原导出只支持尚未清除的 Host attempt。进程正常返回、调度器清除 pending 后,旧版无法为这次停止生成恢复证据。候选允许封存正常返回,保留同一会话与原 replan;有效布尔值可开启有限阶段,同一阶段和截止时间可重放。覆盖本地 SQLite、真实 native app-server 与受控的 Docker 归档接口。未运行真实 Docker 容器接续、外部计分或付费模型。导出和重放以字典相等比较 pending,将 JSON 数字 0 当成 false。首次恢复走严格布尔读取,拒绝同一份封存证据;一次成功恢复后将原标记改成 0,重放仍选择已恢复阶段。

改动思路 / Approach

扩展现有显式恢复路径合适;必须复用严格证据规则。保留源 proof 和 native 窗口,修正 witness 和关闭时拒绝顺序。 恢复先封存原停止来源,再由 native owner 读回原历史,最后按当前权限开启有限阶段。 封存 阶段选择

具体改动 / Changes

关键代码讲解

  • prepare_stopped_recovery 接受清除标记的正常返回,绑定原 deadline 和唯一 wake;新增分支用字典相等比较 pending。 导出
  • read_idle_host 仍用已有严格布尔 reader,verify_idle_host_replay 却再次比较字典。Python 中 0 等于 False,两条路径对同一 witness 判断相反。 首次 重放
  • native_idle_turn 用进程窗口检查稳定历史;真实 app-server 读同一 cursor、拒绝外来窗口,未发出模型请求。 窗口

以 base b7c9178 的 benchmark/runtime/RUNTIME.md 为准。retained_transport 保留原阶段及 replan;original_native_window 已在真实 reader 核验;independent_pending_witness 的导出/重放反例未通过。 原恢复约束

对主干的风险 / Risks

同一合成来源:base 拒绝正常返回;head 用合法 false 可建立阶段,保存原会话、replan 和截止时间。只替换为数字 0,导出仍写 proof,首次恢复拒绝,已建立阶段却能重放。关闭选项时合法旧证明相同;截断 pending 且无有效 attempt 时,head 先报 JSON 解析错误,base 报原资格错误。

语义与 CI 对齐

本地 Host 与外部调度器 296 项通过,3 项显式 Docker 跳过。两个数字 0 反例失败。快速 premerge 执行项通过,benchmark 人工合并检查保留。未查询托管 CI。

我的整体评价 / Verdict

请求修改。统一严格布尔判定,核对原 witness 指纹,并保留关闭时的原拒绝顺序。本次评审覆盖现有宿主与 CLI 行为。覆盖本地 SQLite、真实 native app-server 与受控的 Docker 归档接口。未运行真实 Docker 容器接续、外部计分或付费模型。

English verdict: REQUEST_CHANGES. At bc45ade, the exporter and same-phase replay accept numeric pending 0 while first admission rejects it; reuse the strict witness check and preserve its sealed identity.

stopped = _number(recovery['process_stopped_at'])
clock = _number(recovery.get('source_phase_deadline_epoch'))
window_ok = (
json.loads(_physical_read(_host_pending_path(state_file))) == {'pending': False}

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

字典相等判断把 JSON 数字 0 当成布尔 false。真实 SQLite 恢复已建立阶段后,我只把原 pending 标记改成 {"pending": 0};_host_pending 判定不可读,但同截止时间重放仍选中阶段。导出端也接受 0,首次恢复却拒绝。请复用严格布尔校验并在重放核对原 witness 指纹;回归以相同的 0 验证三个入口都拒绝,原布尔 false 仍可重放。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant