Skip to content

Allow independent native evaluators in new EdgeBench cohorts - #8308

Merged
huangruiteng merged 4 commits into
mainfrom
codex/edgebench-native-judge-lanes-main
Nov 10, 2026
Merged

huangruiteng merged 4 commits into
mainfrom
codex/edgebench-native-judge-lanes-main

Conversation

@loopx-agent

@loopx-agent loopx-agent commented Nov 7, 2026 •

Copy link
Copy Markdown
Collaborator

What changes

Pin the default provider for new EdgeBench CLI cohorts to the reviewed
476b34d17e75 head of EdgeBench #27. The candidate is now durably reachable
both from the merged codex/edgebench-integration branch and the permanent
loopx-8308-edgebench-20261110 tag. The existing checkpoint codec patch stays
in the frozen source pair.

A matched native or blind archive-submission cohort can now give
python -m benchmark.edgebench.serve one --judge-isolated-run per worker run
ID. Each named run gets its own FIFO evaluator. A saturated lane cannot occupy
another arm's queue, and a capacity rejection leaves the submission round and
cooldown available. The runner guide shows the four-arm command and host
capacity requirements. With the option omitted, the judge uses its shared pool.
Previously frozen cohorts reopen on their original source.

Verification

  • EdgeBench Collapse source registry shadows in status queue #27 received an exact-head approval review and passed its focused
    run-capacity, aggregate-capacity, fair-admission, host-capacity and
    nonfinite-feedback suite: 67 tests.
  • The production source CLI fetched the tagged commit into a fresh directory,
    verified and applied the compatibility patch, produced a clean source
    checkout, and recorded the expected upstream and effective revisions.
  • 65 relevant LoopX source, registration, runtime and checkpoint tests passed;
    349 environment-dependent tests were skipped.
  • compileall and git diff --check origin/main...HEAD passed.

The only conflict with current main was in the EdgeBench runner guide; both the
new evaluator-lane section and main's cached-image inventory section are
preserved. Actual Docker grading throughput remains an operator provisioning
boundary documented by the guide.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent, gpt-6.1-sol, OpenAI, runtime_reported, reasoning_effort=max

Request changes conclusion (author-owned PR; GitHub blocks formal self-review)

评审提交 233e2fa,PR 目标基线 e2371c4,实际共同祖先 90ffbe2。

[P2] 默认来源需要已审且可持续获取的提交。 共享集成分支与候选提交分叉,上游 PR 尚无评审。仅同步发布分支时,默认源码准备从成功变成 fetch 失败,judge 与 worker 都拿不到来源回执。 默认 pin 上游 #27

动机 / Motivation

运行新一组 EdgeBench 对照实验的人,需要让两个实验组各自得到评估结果,也要能从默认来源启动 judge 和 worker。 旧版让各组共用评估池,一组排满时会挤占另一组的容量。新提交支持按运行 ID 预留评估器,并把所有新批次的默认来源提前切到上游 PR 的提交。

同一慢组排队时,另一组仍能完成,超额提交保留原轮次。GitHub 当前可取到新提交,旧批次离线读回原回执;只同步共享集成分支的新批次取不到这个默认提交。 本轮验证覆盖来源冻结、注册和评估队列。真实容器评分吞吐和实验结果仍需独立验收。

改动思路 / Approach

独立 FIFO 仍由原生 SForge judge 管理,LoopX 继续用现有来源冻结层。当前风险来自默认指针推进早于提供方发布,应把两步接在同一条可验证的交付链上。 保留现有默认,直至上游已审提交可从持久 ref 取回。之后只改现有默认来源配对,复跑同一批次和注册对照。

SForge 为各运行 ID 管理队列和容量,仍用原生提交轮次与冷却的记账边界。LoopX 的 source、worker 和 judge 在进入这个逻辑之前,先使用同一份已验证的提供方检出。该检出失败时,两种调度模式都无法启动。

具体改动 / Changes

三个文件共 +31/−13。README 增加四组运行的启动示例,默认文件只推进提供方提交,测试更新了预期字面值。规范取自 benchmark/edgebench/README.md @ 90ffbe2。变更前来源冻结契约

  • Frozen cohort:旧 v2 回执在断网时仍逐字节读回。
  • Reviewed provider pair:缺上游独立评审与正式来源的可达性。集成分支 efdfd9c3b41fac53cb9b0e4101f9217d0a905ff8 和候选 476b34d17e759a7dbcf9ec36a12ff42b757b462a 分叉。
  • Patch integrity:同一补丁通过 preimage 检查并生成新的有效提交,旧记录继续验原来的回执和补丁。

关键代码讲解

  • source-default.ref 将新批次的默认从 118f9257d39c71d0805bb5b9cd4eb85d13a1e6d2 推进到 476b34d17e759a7dbcf9ec36a12ff42b757b462a。此选择发生在解析是否预留评估器之前。来源 pin
  • prepare_source 对空目录先精确 fetch,再校验补丁和发布来源回执。只同步正式集成分支的镜像中,基线退出 0,候选退出 2,底层 fetch 退出 128,未发布回执。已冻结目录仍可离线打开。取源与冻结
  • serve.main 从冻结检出导入 SForge。两版共享模式五个运行 ID 都注册成功且返回字段相同。开启列表后四个指定 ID 成功,额外 ID 收到 404,容量不足时拒绝启动。真实 CLI 入口 配置说明

对主干的风险 / Risks

PASS:基线和候选各 48 个来源测试通过,配套原生提供方分别 114 和 118 个测试通过。运行真实 serve CLI 与 HTTP 注册,Docker 只替代 version 接口。慢组的控制实验验证另一组能完成,超额提交没有花掉下一轮。标准 premerge 5 项直接、19 项选中检查通过,基准相关人工核验仍在。远端 CI 未查。

FAIL:用仅同步共享集成分支的镜像复跑 python -m benchmark.edgebench.source --log-dir FRESH_DIR,候选默认取源失败。直接 GitHub 取到 PR 提交的成功结果也已保留。等上游独立评审完成,把已审提交保留在正式发布分支或可长期获取的 tag 上。用该提交和现有兼容补丁在空目录重新冻结,并复跑默认模式、独立模式和旧回执读回。

语义与 CI 对齐

变更的是现有默认契约的外部输入。运行列表是显式启用,取源却是所有新批次的必经步骤。默认字面值测试可以校验 SHA 的格式和配置,无法发现正式分支缺少该对象。修复后应在正式 ref 的干净镜像上重复来源和启动验证。

我的整体评价 / Verdict

独立队列的并发行为已有证据。重复开展实验还依赖可重建的默认提供方来源,普通共享模式的首次启动也受这条依赖影响。旧批次离线读回已确认,正式 pin 和真实容器评分留待验证。此次涉及 README 和 CLI,视觉评审不适用。

English verdict: REQUEST_CHANGES 233e2fa. The FIFO and frozen-cohort checks pass. The new default pin fails a fresh published-branch-only bootstrap and still needs a reviewed, durable upstream ref.

@mergify

mergify Bot commented Nov 9, 2026

Copy link
Copy Markdown

This pull request has merge conflicts with main and cannot be merged
until they are resolved. Please rebase or merge the base branch, @loopx-agent.

Choose the remote for the base repository, not an out-of-date fork.
For a fork clone, first inspect git remote -v; upstream must point
to https://github.com/loopx-project/loopx.git. If it is absent, add it
with git remote add upstream https://github.com/loopx-project/loopx.git.
Then run:

git fetch upstream
git rebase upstream/main
# Resolve each conflict, git add the resolved files, then git rebase --continue.
git push --force-with-lease origin HEAD

For a same-repository clone whose origin points to
https://github.com/loopx-project/loopx.git, use origin instead of
upstream for fetch/rebase. If you prefer merging the base, use
git merge <base-remote>/main and push normally.

Keep the DCO Signed-off-by trailer on every commit when you rebase.
https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase Mergify: the pull request has merge conflicts with its base branch label Nov 9, 2026
…0261110

Signed-off-by: huangruiteng <huangrt01@163.com>

# Conflicts:
#	benchmark/edgebench/README.md
Signed-off-by: huangruiteng <huangrt01@163.com>
@mergify mergify Bot removed the needs-rebase Mergify: the pull request has merge conflicts with its base branch label Nov 10, 2026

@huangruiteng huangruiteng left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent, GPT-5, OpenAI, self_reported

动机 / Motivation

运行 matched EdgeBench 对照实验并通过 LoopX 默认来源启动 judge/worker 的实验操作者,需要既拿到各组反馈,也能在新目录可靠重建来源。旧默认与一个只存在于 PR ref 的候选分叉,而共享 judge 又让慢 arm 占用 peer arm 的队列。现在 pin 指向已审且同时存在于持久 integration branch 与长期 tag 的提交;显式 run 列表可为每条 arm 预留 FIFO,省略时继续使用共享池。

旧默认来源只指向共享分支,且一个慢 arm 可占用共同 judge 队列;现在默认 pin 指向持久可达的已审提交,显式 run 列表可为每个 arm 分配独立 FIFO。全新目录可从默认 pin 拉取代码、验证补丁并生成 clean source receipt;上游 67 项相关测试和 LoopX 65 项相关测试通过。本 PR 不声称真实容器吞吐、模型实验等价或 PostgreSQL benchmark 支持。

改动思路 / Approach

默认来源继续由现有 source-default 与 freeze-once 层拥有,EdgeBench 继续拥有 judge admission。这样不需要在 LoopX 再写一个 scheduler 或 provider selector。三个文件只推进不可变来源配对、补上操作者说明并更新字面值测试;历史 cohort 的 receipt 不随默认移动。

具体边界是:新 cohort 使用已审 pin;显式 --judge-isolated-run 才启用每条 arm 的 FIFO;两条路径都复用原 registration、round、cooldown、spool 与 cleanup owner。

具体改动 / Changes

评审提交 be7e0c3;diff 为 3 个文件、+31/−13。规范基线 benchmark/edgebench/README.md 中 source-default 是:source-default(已验证)、freeze-once(已验证)、reviewed-provider(已验证)。

关键代码讲解

  • source-default.json 把 ref 从 118f9257... 推进到 476b34d1...,repository 与 nonfinite-checkpoint-v2 补丁摘要保持不变。 来源 pin
  • prepare_source 对空目录做精确 fetch、补丁 preimage/摘要校验并原子发布 v2 receipt;旧 receipt 仍离线复用。 取源与冻结
  • README 新增四 arm 的重复 run ID 示例,明确 FIFO、pending、容量拒绝和 host 预配边界;当前 main 的 cached-image 说明也完整保留。 操作指南

对主干的风险 / Risks

PASS:在全新目录运行生产 source CLI,成功从长期 tag 获取 476b34d1...,应用同一补丁并生成 clean checkout;receipt 记录 upstream 476b34d1...、effective 3aa6eb4... 和原 patch SHA。LoopX 相关 65 项通过、349 项因环境条件跳过;EdgeBench 的 run-capacity、fair-admission、host-capacity 与 nonfinite-feedback 67 项通过;compileall 与 diff hygiene 通过。

旧 P2 已解决:候选同时进入持久 codex/edgebench-integration 分支与 loopx-8308-edgebench-20261110 tag,不再依赖 PR-only ref。冲突只出现在 README,已同时保留新的 evaluator-lane 与当前 main 的 cached-image inventory 内容。

UNVERIFIED:没有运行真实 Docker 评分吞吐、真实模型实验或长程 benchmark 等价性;按用户要求未等待远端 CI。

语义与 CI 对齐

默认 ref 变化是真实行为变化:它适用于所有没有显式 source 的新 cohort;历史 cohort 不迁移。per-arm 调度仍是显式 opt-in,没有改变默认 judge 行为。

我的整体评价 / Verdict

未发现阻断问题。此次变更把 provider 修复、默认 pin 和操作者说明放在同一条可验证交付链上;旧 cohort 兼容性、补丁身份和持久 fetch 都有独立证据。剩余风险是实际评分吞吐与研究效果,而不是来源或 admission 契约的正确性。CLI 与 Markdown 变化不需要浏览器首屏截图。

English verdict: APPROVE at be7e0c3. The reviewed provider is now durably tagged, the default source freezes successfully in a fresh directory, and per-arm evaluator admission passes its focused provider suite.

@mergify

mergify Bot commented Nov 10, 2026

Copy link
Copy Markdown

Hi @loopx-agent, the DCO Sign-off check did not pass. Please inspect
its details first: checkout, fetch, timeout or infrastructure errors
need their own recovery, not a rewrite of otherwise signed commits.

If the log confirms a missing Signed-off-by trailer, amend the
affected commit with git commit --amend -s; for multiple commits,
use an interactive rebase against the current base from the correct
base-repository remote and sign off each affected commit. Push the
rewritten PR branch with git push --force-with-lease origin HEAD.

@huangruiteng
huangruiteng merged commit 117cc05 into main Nov 10, 2026
19 of 25 checks passed
@huangruiteng
huangruiteng deleted the codex/edgebench-native-judge-lanes-main branch November 10, 2026 05:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants