Repository navigation
feat(eval): add Harbor trial environment, experiment runner, and canonical reporting - #6850
Closed
Colafornia wants to merge 2 commits into
Closed
Colafornia wants to merge 2 commits into
Colafornia wants to merge 2 commits into
Conversation
…tore Add the maka-eval foundation modules: - strict maka.eval.v1 projection and native maka.eval.v2 orthogonal attempt contracts, condition-only RunFingerprint, and semantic difference registry - exact 6-counter token accounting, non-mutating local metering gateway, admission cutoffs, HMAC-authenticated checkpoints, and receipt-vs-proxy reconciliation - single-writer append-only attempt journal, SQLite runtime artifact capture, critical artifact verification, and deterministic state reconstruction Generated-by: Codex Generated-by: Google Antigravity
…nical reporting Add execution orchestration, Harbor isolation, and deterministic projections: - version-pinned Harbor / Pier trial environment bridge, loopback TCP + HMAC relay agent, and DNS/IP egress filtering proxy - task-group admission concurrency, shared preparation cache lock, phase-scoped deadlines, bounded retries, and crash recovery - external CLI subject adapter (exit-code and protocol-v1 result frames) and canonical v1/v2 journal projections with 3-verdict differential oracle evaluation Generated-by: Codex Generated-by: Google Antigravity
This was referenced Nov 13, 2026
Member
Author
|
Parked until the preceding layer merges into rust-preview so each PR shows a single-layer diff; will reopen sequentially. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
English
Summary
Multi-arm evaluation must isolate trial execution inside Harbor while keeping task-group scheduling, deadline enforcement, crash recovery, and result projection under
maka-evalauthority. This PR adds the Harbor trial environment bridge, egress proxy policy, external CLI subject adapter, experiment runner, and canonical reporting modules.Refs #6667. Builds on #6849 (until it merges, review commit
9d32e6037for this layer alone).environment&harbor: Bridge per-attempt trials to a version-pinned Harbor Python runner via an authenticated loopback TCP relay, serialize shared preparation locks, propagate cancellation, and enforce DNS/IP egress filtering.subject::external_cli: Run external CLI harnesses with explicit credential mappings and decodeexit-codeorprotocol-v1(MAKA-EVAL-RESULT-V1) frames while rejecting duplicate or ambiguous frames.runner: EnforcemaxConcurrentTaskGroups, boundedinfra_failedretries, and pre-dispatch crash recovery that blocks automatic re-runs when prior cleanup remains unconfirmed.report& license check: Project committed journals deterministically tov1/v2, compute paired cohort summaries, evaluate the 3-verdict differential oracle, and registernetwork-policyinscripts/asf-license-headers.mjs.Verification
Validated at
9d32e6037:cargo test --locked -p maka-eval— 15 passed (2 unit tests and 13 integration tests, adding coverage for all 5 semantic difference scenarios and the 3-verdict oracle, crash and post-hoc recovery, preparation cache locking, result-frame ambiguity rejection, Harbor loopback handshake, unconfirmed cleanup blocking, and cancellation).cargo clippy --locked -p maka-eval --all-targets -- -D warnings— passed.RUSTDOCFLAGS="-D warnings" cargo doc --locked -p maka-eval --no-deps— passed.node scripts/asf-license-headers.mjs checkandgit diff --check— passed.Live Docker container trials, egress proxy container builds, frozen legacy oracle cohorts, and full
just checkwere not run.AI use
Tool(s) and scope: OpenAI Codex authored the Harbor bridge, egress filter, runner, external CLI adapter, and reporting modules. Google Antigravity split the review stack and ran the verification checks above. Both
Generated-bytrailers are preserved.Checklist
Does this PR entail a change in behavior?
中文
概述
多臂评测需要把单次试验隔离在 Harbor 中运行，同时由
maka-eval统一掌握任务组调度、截止时间、崩溃恢复与结果投影权威。本 PR 新增 Harbor 试验环境桥接、出站代理策略、外部 CLI 被测适配器、实验编排器与规范报告模块。关联 #6667。基于 #6849（合入前可直接查看提交
9d32e6037以仅审阅本层改动）。environment与harbor：通过鉴权的回环 TCP 中继桥接锁定版本的 Harbor Python 执行器，串行化共享预备锁，传递取消信号并实施 DNS/IP 出站过滤。subject::external_cli：按显式凭据映射运行外部 CLI，解析exit-code或protocol-v1（MAKA-EVAL-RESULT-V1）结果帧并拒绝重复歧义帧。runner：实施maxConcurrentTaskGroups并发准入、有界infra_failed重试及派发前崩溃恢复，在历史清理未确认时跨运行阻断重跑。report与许可证检查：从已提交日志确定性导出v1/v2结果与成对队列汇总，实现三项裁决差分比对，并将network-policy纳入scripts/asf-license-headers.mjs。验证
在提交
9d32e6037上验证：cargo test --locked -p maka-eval— 15 项通过（2 项单测与 13 项集成测试，新增五类语义差异与差分裁决、日志崩溃与事后恢复、预备缓存锁、结果帧歧义拒绝、Harbor 回环握手、未确认清理阻断及取消信号测试）。cargo clippy --locked -p maka-eval --all-targets -- -D warnings— 通过。RUSTDOCFLAGS="-D warnings" cargo doc --locked -p maka-eval --no-deps— 通过。node scripts/asf-license-headers.mjs check与git diff --check— 通过。未运行真实 Docker 容器试验、出站代理镜像构建、历史参考实现队列比对及完整
just check。AI 使用
工具及范围：OpenAI Codex 编写 Harbor 桥接、出站过滤、编排器、外部 CLI 适配器与报告模块；Google Antigravity 切分评审栈并执行上述验证。提交保留两个工具的
Generated-by标记。检查清单
本 PR 是否改变行为？