Skip to content

feat(eval): enforce product parity and runtime binary identity gates - #6853

Closed
Colafornia wants to merge 4 commits into
apache:rust-previewfrom
Colafornia:feat/eval-4-parity-gates
Closed

Colafornia wants to merge 4 commits into
apache:rust-previewfrom
Colafornia:feat/eval-4-parity-gates

Conversation

@Colafornia

@Colafornia Colafornia commented Nov 13, 2026 •

Copy link
Copy Markdown
Member
English

Summary

Condition fingerprints alone cannot prove that maka-eval drives Maka identically to normal user execution or that settled results came from the parity-verified maka binary. This final PR adds end-to-end Product Parity verification (maka-eval parity) and enforces parity and runtime binary identity in M3 and M5 milestone gates.

Refs #6667. Builds on #6852 (until it merges, review commit 08b1ca4a2 for this layer alone).

# Layer Links
1 Contracts, metering gateway, and attempt store #6849
2 Harbor trial environment, runner, and reporting #6850 · diff
3 Native Maka Host subject and CLI entrypoint #6852 · diff
4 Product parity and binary identity gates #6853 (this PR) · diff
  • parity & maka-eval parity: Preserve declared endpoint semantics on the reference path while driving maka-eval against a local probe under identical inputs, verifying committed RunSpec/RequestSpec facts, resolved upstream routes, effective instructions, tool exposure (Code Mode, skills, tool schemas), and execution policies.
  • report & M3/M5 milestone gates: Require passing parity reports matching each Maka subject's public configuration digest, verify confirmed runtime maka binary SHA-256 identity from settled execution evidence across local and container paths (failing closed when missing or mismatched), and require complete non-indeterminate settlement for m5.

Verification

Validated at 08b1ca4a2 (git diff against the full feat/maka-eval implementation at 4eddc488c is empty):

  • cargo test --locked -p maka-eval — 25 passed (2 unit tests and 23 integration tests covering live M3/M5 product parity, instruction/tool/policy drift detection, end-to-end Zenmux route parity, and CLI M3 binary-identity enforcement across local and container paths).
  • node --test scripts/ci.test.mjs — 18 passed.
  • cargo clippy --locked -p maka-eval --no-deps --all-targets -- -D warnings — passed.
  • RUSTDOCFLAGS="-D warnings" cargo doc --locked -p maka-eval --no-deps — passed.
  • node scripts/asf-license-headers.mjs check, node scripts/ci.mjs check format, and git diff --check — passed.

Dependency-inclusive Clippy hits the existing clippy::needless_update in plugins/agent/provider-protocol/src/chat_completions/response.rs:87 outside this PR's diff. Live Docker Harbor cohorts, frozen legacy oracle cohorts (#6667), paid model endpoints, and full just check remain before marking ready for final acceptance.

AI use

  • No generative tool made a substantive contribution
  • Generative tooling made a substantive contribution

Tool(s) and scope: Google Antigravity authored the Product Parity verifier, M3/M5 milestone and binary-identity gates, CLI wiring, documentation updates, and integration tests, and verified the stack. Commit carries Generated-by: Google Antigravity.

Checklist

  • Tests cover the change and fail without it
  • Format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

  • Yes — described under Summary above
  • No
中文

概述

仅靠运行条件指纹无法证明 maka-eval 与常规用户路径以完全一致的语义驱动 Maka,也无法确保已结算结果来自通过等价性验证的同一份 maka 二进制。本 PR 新增端到端产品等价性验证(maka-eval parity),并在 M3 与 M5 里程碑门禁中强制校验等价性报告与运行时二进制身份。

关联 #6667。基于 #6852(合入前可直接查看提交 08b1ca4a2 以仅审阅本层改动)。

# 分层职责 链接
1 实验契约、计量网关与尝试存储 #6849
2 Harbor 试验环境、编排器与规范报告 #6850 · 增量对比
3 原生 Maka Host 适配器与 CLI 入口 #6852 · 增量对比
4 产品等价性校验与二进制身份门禁 #6853(本 PR) · 增量对比
  • parity 与 maka-eval parity:保留参考路径的原始端点语义并在相同输入下驱动 maka-eval 连接本地探针,比对落盘的 RunSpec/RequestSpec 事实、解析后的上游路由、生效指令、工具暴露(Code Mode、内置技能、工具模式定义)及执行策略。
  • report 与 M3/M5 里程碑门禁:强制校验匹配公开配置摘要的等价性通过报告,依据已结算执行产物核验实际 maka 二进制 SHA-256 身份(支持本地与容器路径,缺失或摘要不匹配时显式失败),并在 m5 下要求全量完成确定性结算。

验证

在提交 08b1ca4a2 上验证(与完整分支 feat/maka-eval 的 4eddc488c 对比差异为空):

  • cargo test --locked -p maka-eval — 25 项通过(2 项单测与 23 项集成测试,新增 M3/M5 产品等价性校验、指令/工具/策略漂移检测、Zenmux 路由端到端等价性及本地与容器路径的 M3 二进制身份门禁测试)。
  • node --test scripts/ci.test.mjs — 18 项通过。
  • cargo clippy --locked -p maka-eval --no-deps --all-targets -- -D warnings — 通过。
  • RUSTDOCFLAGS="-D warnings" cargo doc --locked -p maka-eval --no-deps — 通过。
  • node scripts/asf-license-headers.mjs check、node scripts/ci.mjs check format 与 git diff --check — 通过。

包含依赖的 Clippy 受基线已有的 plugins/agent/provider-protocol/src/chat_completions/response.rs:87(clippy::needless_update,不在本 PR 改动范围内)阻断。真实 Docker Harbor 队列、#6667 历史参考实现队列比对、付费模型端点及完整 just check 仍需在转为正式验收前完成。

AI 使用

  • 没有生成式工具作出实质性贡献
  • 生成式工具作出了实质性贡献

工具及范围:Google Antigravity 编写产品等价性校验器、M3/M5 里程碑与二进制身份门禁、CLI 接线、文档更新及集成测试,并完成整组分支栈验证。提交包含 Generated-by: Google Antigravity。

检查清单

  • 测试覆盖改动,且在缺少改动时会失败
  • 格式、类型检查及受影响的测试套件均在本地通过

本 PR 是否改变行为?

  • 是 — 已在上方概述中说明
  • 否

…tore

Add the maka-eval foundation modules:
- strict maka.eval.v1 projection and native maka.eval.v2 orthogonal attempt contracts, condition-only RunFingerprint, and semantic difference registry
- exact 6-counter token accounting, non-mutating local metering gateway, admission cutoffs, HMAC-authenticated checkpoints, and receipt-vs-proxy reconciliation
- single-writer append-only attempt journal, SQLite runtime artifact capture, critical artifact verification, and deterministic state reconstruction

Generated-by: Codex
Generated-by: Google Antigravity
…nical reporting

Add execution orchestration, Harbor isolation, and deterministic projections:
- version-pinned Harbor / Pier trial environment bridge, loopback TCP + HMAC relay agent, and DNS/IP egress filtering proxy
- task-group admission concurrency, shared preparation cache lock, phase-scoped deadlines, bounded retries, and crash recovery
- external CLI subject adapter (exit-code and protocol-v1 result frames) and canonical v1/v2 journal projections with 3-verdict differential oracle evaluation

Generated-by: Codex
Generated-by: Google Antigravity
Add real Maka Host subject execution and CLI commands:
- resolve outbound model routes through Maka provider policy before attaching the non-mutating local metering gateway
- initialize isolated Host roots via maka host init --settings-stdin without secret leakage, execute through public Settings/Session and composer draft/receipt APIs, and record executed Maka binary digests
- preserve unattended approvals and questions as explicit settlement outcomes and report environment limitations without silent bypass
- add maka-eval run/report/host-subject CLI commands and wire CI runtime helper consumers

Generated-by: Codex
Generated-by: Google Antigravity
Add end-to-end Product Parity verification and M3/M5 milestone acceptance gates:
- add maka-eval parity to verify product composition, effective instructions, tool exposure, execution policies, and resolved outbound routes between normal Maka execution and maka-eval
- enforce passing parity reports and confirmed runtime Maka binary SHA-256 identity from settled execution artifacts in M3 acceptance and M5 production delivery gates

Generated-by: Google Antigravity
@Colafornia

Copy link
Copy Markdown
Member Author

Parked until the preceding layer merges into rust-preview so each PR shows a single-layer diff; will reopen sequentially.

@Colafornia Colafornia closed this Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant