[Colafornia]

English

Summary

Benchmark runs need strict experiment contracts and exact token semantics before any execution or persistence layer touches them. This PR introduces the products/eval crate with only the core contract layer and pure 6-counter NormalizedUsage / DEEPSEEK_V4_FLASH_COST pricing semantics, keeping persistence (store) and HTTP metering (gateway) in parallel follow-ups built on top of it.

Refs #6667.

#	Layer	Links
1a	Experiment contracts and usage pricing semantics	#6849 (this PR)
1b	Single-writer append-only attempt store and artifact capture	diff (parallel on 1a)
1c	Local metering gateway, HMAC checkpoints, and usage reconciliation	diff (parallel on 1a)
2	Harbor trial environment, runner, and reporting	#6850 (closed draft, pending rebase on 1b+1c)
3	Native Maka Host subject and CLI entrypoint	#6852 (closed draft, pending rebase)
4	Product parity and binary identity gates	#6853 (closed draft, pending rebase)

• contract: Validate maka.eval.v1 specs via strict serde::Deserialize, expand cells deterministically, model closed orthogonal maka.eval.v2 enums (AttemptMetering, AttemptCancellation, EgressAuditState), compute normalized condition-only RunFingerprint, and project v2 -> v1 across all 5 adjudicated semantic difference scenarios.
• measurement (pure value & pricing slice): Preserve all six token counters (inputTokens, outputTokens, cacheReadTokens, cacheWriteTokens, reasoningTokens, totalTokens) without synthesizing totalTokens, and compute standardized PricingRates::cost_usd (DEEPSEEK_V4_FLASH_COST).
• Scope boundary: Minimal pure-value library crate in the root workspace (hex, serde, serde_json, sha2 only); no filesystem store, HTTP server/client, CLI binary, Harbor subprocesses, or internal maka-* dependencies.

Verification

Validated at 44441d27b on rust-preview (b86d78606):

• cargo test --locked -p maka-eval — 5 passed (tests/contract.rs: spec validation and cell expansion, normalized fingerprint key/float invariance, 6-counter usage + DEEPSEEK_V4_FLASH_COST pricing, earliest-valid canonical attempt selection with terminal 0.0 score, strict v1/v2 deserialization rejection + lossless v1 round-trip, and v2 -> v1 projections across all 5 adjudicated difference scenarios).
• cargo clippy --locked -p maka-eval --all-targets -- -D warnings — passed.
• RUSTDOCFLAGS="-D warnings" cargo doc --locked -p maka-eval --no-deps — passed.
• cargo fmt -p maka-eval -- --check, node scripts/asf-license-headers.mjs check, and git diff --check — passed.

Full just check, filesystem store / SQLite tests (1b), HTTP gateway tests (1c), Docker Harbor trials, and live upstream model requests are out of scope for 1a and were not run here.

AI use

• No generative tool made a substantive contribution
• Generative tooling made a substantive contribution

Tool(s) and scope: OpenAI Codex authored the initial foundation modules and integration tests. Google Antigravity converged the contract/state enums, split feat/eval-1-foundation into the 1a -> {1b, 1c} diamond DAG (store and metering + gateway), fixed the v1/v2 contract decoding discrepancies, ablated redundant abstractions and dual decoders (44441d27b), and ran the verification checks above. Both Generated-by trailers are preserved.

Checklist

• Tests cover the change and fail without it
• Format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

• Yes — described under Summary above
• No

中文

概述

在接入持久化存储与运行时代理之前，多臂评测需要先确立严格的实验契约与精确的 Token 语义。本 PR 仅向工作区引入最小化的 products/eval 库，包含 contract 核心契约与纯值的 6 计数器 NormalizedUsage / DEEPSEEK_V4_FLASH_COST 计价语义；尝试存储（store）与 HTTP 计量网关（gateway）拆出为基于本 PR 并行的两层增量分支。

关联 #6667。

#	分层职责	链接
1a	实验契约与 Token 计价语义	#6849（本 PR）
1b	单写者只追加尝试存储与产物捕获	增量对比 （基于 1a 并行）
1c	本地计量网关、HMAC 检查点与用量对账	增量对比 （基于 1a 并行）
2	Harbor 试验环境、编排器与规范报告	#6850 （已关闭草稿，待基于 1b+1c 重基）
3	原生 Maka Host 适配器与 CLI 入口	#6852 （已关闭草稿，待重基）
4	产品等价性校验与二进制身份门禁	#6853 （已关闭草稿，待重基）

• contract：通过严格的 serde::Deserialize 校验 maka.eval.v1 实验规格并确定性展开单元，使用闭合正交枚举建模 maka.eval.v2 状态（AttemptMetering、AttemptCancellation、EgressAuditState），计算归一化运行条件 RunFingerprint，并实现覆盖全部 5 类裁定场景的 v2 -> v1 投影。
• measurement（纯值与定价子集）：独立保留六项 Token 计数器（inputTokens、outputTokens、cacheReadTokens、cacheWriteTokens、reasoningTokens、totalTokens，不求和覆盖），并通过 PricingRates::cost_usd 实现标准 DEEPSEEK_V4_FLASH_COST 计价公式。
• 范围边界：作为最小纯库接入根工作区（仅依赖 hex、serde、serde_json、sha2）；不含文件存储、HTTP 网关、CLI 二进制、Harbor 子进程及内部 maka-* 依赖。

验证

基于 rust-preview（b86d78606）在提交 44441d27b 上验证：

• cargo test --locked -p maka-eval — 5 项通过（tests/contract.rs：规格校验与单元展开、归一化 fingerprint 键序与浮点等价不变性、6 计数器核算与 DEEPSEEK_V4_FLASH_COST 计价、最早有效规范结果选样及 0.0 终态语义、v1/v2 非法反序列化拒绝与无损往返、以及全部 5 类语义差异裁定场景的 v2 -> v1 投影）。
• cargo clippy --locked -p maka-eval --all-targets -- -D warnings — 通过。
• RUSTDOCFLAGS=\"-D warnings\" cargo doc --locked -p maka-eval --no-deps — 通过。
• cargo fmt -p maka-eval -- --check、node scripts/asf-license-headers.mjs check 与 git diff --check — 通过。

未在本层运行完整 just check、文件存储 / SQLite 测试（1b）、HTTP 网关测试（1c）、Docker Harbor 容器试验及真实上游模型请求。

AI 使用

• 没有生成式工具作出实质性贡献
• 生成式工具作出了实质性贡献

工具及范围：OpenAI Codex 编写初始底座模块与集成测试；Google Antigravity 收敛契约状态枚举，将 feat/eval-1-foundation 拆分为 1a -> {1b, 1c} 菱形 DAG（store 与 metering + gateway），修复 v1/v2 解码一致性问题，消融移除冗余抽象与双解码器（44441d27b）并执行上述验证。提交保留两个工具的 Generated-by 标记。

检查清单

• 测试覆盖改动，且在缺少改动时会失败
• 格式、类型检查及受影响的测试套件均在本地通过

本 PR 是否改变行为？

• 是 — 已在上方概述中说明
• 否