Skip to content

feat(subjects): skill_evidence results say when they may be compared (protocol digest, verdict_rule, conformance fixtures) - #804

Open
napetrov wants to merge 20 commits into
intel:mainfrom
napetrov:evidence-protocol
Open

napetrov wants to merge 20 commits into
intel:mainfrom
napetrov:evidence-protocol

Conversation

@napetrov

Copy link
Copy Markdown
Contributor

Follow-up to #344. A skill_evidence.v1 subject digest pins a skill's bytes, not how a number was measured, so two results about one digest can disagree on judge, sampling, prompt or thresholds, and a reader can't recompute a verdict. This PR implements what agentskills/agentskills#643 replies promised.

What changes

  • Every result carries:
    • metric_spec (unit, range, higher_is_better, judged);
    • verdict_rule (basis: p_adjusted, alpha, test, correction, p_decimals);
    • protocol {id, version, digest, resolved, unresolved[, aliases]}.
  • protocol.digest is sha256 of the canonical JSON of the resolved inputs only: sorted keys, no whitespace, UTF-8, integral floats written as ints. The inputs are:
    • eval-set digest and metric;
    • judge model, provider and context;
    • judge sampling and prompt sha256;
    • repeats;
    • test, correction, alpha and CI confidence.
  • An input that is missing, or that names an alias, goes to unresolved and stays out of the digest. Aliases are -latest/:default/auto, bare family names like opus/sonnet/gpt-4o, and an alias nested inside a value.
  • comparable / non_comparable give the reasons a result cannot be compared:
    • a verdict without its rule, unit or direction;
    • a verdict that does not follow its rule;
    • a judged result without a judge;
    • a digest that does not match resolved.
  • New dokimi.subjects.evidence_check with the consumer checks: comparability, compare, mark_against and recompute_verdict.
  • Three conformance fixtures in tests/fixtures/evidence_conformance:
    • a verdict without its rule is non-comparable;
    • with the rule, the verdict recomputes;
    • two judge snapshots give two digests, and the newer result is non-comparable to the older one.
  • Matrix-rollup and scorecard cells now pass through the judge fields arms.v1 already records (judge_model, judge_provider, judge_context, judge_temperature, judge_prompt_sha256).

Behaviour to know

  • Older results are non-comparable. Already-published v1 files still validate, since every new key is optional. A consumer using comparability() will mark their verdicts non-comparable (verdict without verdict_rule). That is intended: no rule means no recompute.
  • Today's results stay unresolved on some inputs. Runs record no repeat count, and dokimi's claude-cli judges default to the bare opus/sonnet aliases. Their results list those inputs as unresolved, so compare reports them as not comparable across runs until runs record a snapshot id and repeats. That is a follow-up on the run side.
  • p_value near alpha. p_value is p_adjusted rounded to 4 places. Within that rounding of alpha, the verdict cannot be recomputed, and the result gets a reason saying so; it is not reported as a contradiction.

Testing

  • tests/test_evidence_protocol.py: 33 tests, red then green, including an independent hashlib check of the digest.
  • Evidence, rollup, scorecard and schema suites: 200 passed, 23 skipped.
  • Full suite on Windows: failures in test_repo_receipt.py / test_catalog_eval_cli.py, which fail the same way on unmodified main on this box.

Self-review before push: local pr-review, 2 passes. Pass 1 found 3 blocking issues:

  • rounding near alpha;
  • judge temperature and prompt hash dropped in the rollup;
  • bare judge names hashed as snapshots.

It also found 2 nits: int vs float in the digest, and aliases nested in a value. All are fixed with tests.

🤖 Generated with Claude Code

@napetrov
napetrov requested a review from a team as a code owner November 16, 2026 05:38
Each result carries metric_spec, the verdict_rule its verdict follows and
protocol {id, version, digest}: sha256 over the canonical JSON of the
resolved measurement inputs only. A missing input or an alias such as
'latest' goes to protocol.unresolved, outside the digest. comparable and
non_comparable give the reasons a result cannot be compared.
dokimi.subjects.evidence_check holds the consumer checks; three conformance
fixtures pin them. Rollup and scorecard cells now pass through the judge.

Signed-off-by: Petrov, Nikolay A <nikolay.a.petrov@intel.com>
…e inputs, bare judge names

A p_value within its rounding of alpha no longer reads as a verdict that
contradicts its rule. Rollup and scorecard cells carry judge_temperature and
judge_prompt_sha256, so the protocol resolves them. Bare family names (opus,
sonnet, gpt-4o) and aliases nested in a value count as unresolved. Integral
floats hash as ints.

Signed-off-by: Petrov, Nikolay A <nikolay.a.petrov@intel.com>
@exai-bot

exai-bot commented Nov 16, 2026 •

Copy link
Copy Markdown

OpenCodeReview: Commit 46f07a2 was not reviewed. This PR has reached its review limit (6 reviews).

CI at `46f07a2`: 12 still running, 4 passing, 1 skipped
Check Status
Build docs (strict) still running
CLI interface diff still running
CodeQL (actions) still running
CodeQL (python) still running
External catalog smoke test (uxlfoundation/skills) still running
Linux Python 3.11 still running
Linux Python 3.14 still running
Linux Wheels Python 3.11 still running
Linux Wheels Python 3.14 still running
Static checks (ruff, schema, zizmor, licenses, pre-commit, bandit, secrets, changelog, schema com... still running

2 more not listed.

To review this commit: re-request review from exai-bot in the Reviewers list.

<details><summary>Last review (1cd1ebb): no findings</summary>

Open threads from the bot: 1

</details>

Comment thread dokimi/dokimi/subjects/evidence.py Resolved
Comment thread dokimi/dokimi/subjects/evidence.py Resolved
Signed-off-by: Petrov, Nikolay A <nikolay.a.petrov@intel.com>

@vchamarthi vchamarthi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Changes needed: 1 must-fix, 1 nit · reviewed 791d106

Next: @napetrov: fix the 1 must-fix finding and answer 7 open threads from exai-bot; then push for a re-review. 1 nit optional.

Blocking

  • Still holds on this head: thread by exai-bot – compare(copy, original) with only judge_model changed to claude-sonnet-5-5-20261101 returns []; verdict_rule.alpha is not cross-checked against protocol.resolved either.
  • Still holds on this head: thread by exai-bot – compare with only metric_spec.unit changed to percent returns [].
  • Still holds on this head: thread by exai-bot – compare of a result whose protocol has repeats unresolved against a copy with no protocol returns [].
  • Still holds on this head: thread by exai-bot – _result sets rule["test"] and the protocol test to the constant METHOD whatever ci_method the scorecard recorded.
  • Still holds on this head: thread by exai-bot – eval_set_digest is suite.question_set_hash, which scorecard.py:102 computes over all questions of the suite, not the cell's own set.
  • Still holds on this head: thread by exai-bot – compare(r, r) with a protocol built from only judge_model returns [], since build_protocol visits only supplied keys.
  • Still holds on this head: thread by exai-bot – comparability with verdict_rule.p_decimals of -1000 raises OverflowError at _near_alpha.

Verified on 791d106: ✅ changed tests: 33 passed, fail on main · ❌ full suite with main merged in fails at tests/test_growth_ratchet.py::test_no_file_or_function_grew_past_its_baseline, tests/test_model_defaults.py::test_no_source_file_still_names_gpt_4o_mini, ci.yml:typecheck / mypy · ✅ merges onto main · 2 mutations tried · 1 edge input tried

3 checks behind it, each with its evidence
Area What holds How
Verdict recompute Flipping the direction in recompute_verdict fails 9 of 60 tests in test_evidence_protocol.py and test_subjects_evidence.py broke the code, tests caught it
Changed tests The changed tests pass on the head tests on the head
Merge The branch merges onto main without conflict merge onto main

Comment thread dokimi/tests/test_evidence_protocol.py Resolved
…protocol

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

# Conflicts:
#	dokimi/dokimi/subjects/evidence.py
…protocol

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The consumer checks now refuse a result whose stated judge, provider,
metric specification or verdict rule differs from what its protocol
resolved, a protocol on one side of a pair only, and an omitted
dokimi.awareness v1 input (unresolved, not resolved). An unsupported
rounding precision is a reason, not an overflow. The exporter takes
the test from the recorded ci_method, and each cell carries the digest
of its own question file, so a multi-set suite no longer shares one
eval-set digest. The fixture README takes the repo's license header.

Thread: PRRT_kwDOVBg7g863y74Z
Thread: PRRT_kwDOVBg7g863y74c
Thread: PRRT_kwDOVBg7g863y74g
Thread: PRRT_kwDOVBg7g863y74k
Thread: PRRT_kwDOVBg7g863y74r
Thread: PRRT_kwDOVBg7g863zDgM
Thread: PRRT_kwDOVBg7g863zDgP
Thread: PRRT_kwDOVBg7g863zjbt
Thread: PRRT_kwDOVBg7g863zjb1
Thread: PRRT_kwDOVBg7g863zjb6
Thread: PRRT_kwDOVBg7g863z_n6
Thread: PRRT_kwDOVBg7g863z_oG
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRR_kwDOVBg7g88AAAABYKB_6w: already fixed in 76c09e9
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov

Copy link
Copy Markdown
Contributor Author

Re: review by vchamarthi

Must fix: judge_provider ignored without protocols; nit: judge fields untested
Both fixed in the threads: 76c09e9 and 76c09e9.

Comment thread dokimi/dokimi/subjects/evidence.py Resolved
Comment thread dokimi/tests/test_evidence_protocol.py Resolved
@codecov-commenter

codecov-commenter commented Nov 16, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 94.47005% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.78%. Comparing base (b62613a) to head (46f07a2).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
dokimi/dokimi/subjects/evidence_check.py 93.37% 5 Missing and 7 partials ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #804      +/-   ##
==========================================
- Coverage   95.79%   95.78%   -0.01%     
==========================================
  Files         432      433       +1     
  Lines       57204    57419     +215     
  Branches     9858     9907      +49     
==========================================
+ Hits        54796    54999     +203     
- Misses       1497     1502       +5     
- Partials      911      918       +7     
Files with missing lines Coverage Δ
dokimi/dokimi/eval/matrix_rollup.py 90.00% <ø> (ø)
dokimi/dokimi/subjects/evidence.py 100.00% <100.00%> (ø)
dokimi/dokimi/subjects/scorecard.py 92.04% <100.00%> (+0.09%) ⬆️
dokimi/dokimi/subjects/evidence_check.py 93.37% <93.37%> (ø)

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…-protocol

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…historical scorecard

Thread: PRRT_kwDOVBg7g864FRtF
Thread: PRRT_kwDOVBg7g864FRtM
Thread: PRRT_kwDOVBg7g864FRtR
Thread: PRRT_kwDOVBg7g864FRta
Thread: PRRT_kwDOVBg7g864FRtj
Thread: IC_kwDOVBg7g88AAAABm8bdwg#6
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov

Copy link
Copy Markdown
Contributor Author

Re: comment by exai-bot

unusable protocol content aborts consumer batches
Fixed in 674aa80: a NaN is a reason now.

A negative p-value was below alpha, so recompute_verdict read it as
"improved" and comparability passed it with no reason. Outside [0, 1]
(p) or (0, 1) (alpha) the rule now cannot be applied.

Thread: PRRT_kwDOVBg7g864Gw0t
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Comment on lines +100 to +101
if cell.get("questions_digest") is not None:
return cell["questions_digest"]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix — reconstructed question provenance is treated as a resolved measurement input. Trigger: retain an awareness rollup, edit its questions file, then rebuild the scorecard and evidence. Evidence: scorecard.py creates questions_digest by reading the file during scorecard construction, not from recorded run metadata; test_each_cell_of_a_two_question_set_suite_carries_the_digest_of_its_own_set even verifies that rebuilding with unchanged runs after editing a.json changes this digest. This branch then hashes that new value into the old statistics' protocol as resolved. Impact: an old measurement can acquire the identity of a different evaluation set and, when the remaining inputs match, be authorized for comparison with measurements of that new set. Workaround: preserve the original question files and avoid reconstructing historical scorecards. Capture the question digest when the run loads its questions and propagate that recorded value; leave eval_set_digest unresolved when that provenance is unavailable, including the suite fallback. Severity justification: medium because this is a bounded but material false measurement identity, not security compromise or data loss.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping question paths cwd-relative for now, as decided. The gap is real: a rebuilt scorecard records an edited questions file as resolved. Filed as #854; this thread stays open until that lands.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and #854 is right for the rebuild case. One part can be fixed in this PR: when the questions file is not found from the cwd, _digest_values hashes only its name, and that value goes into protocol.resolved.eval_set_digest. Two subjects that both name questions/q.json and are built where the file is missing get the same digest whatever the files hold. The path string is hashed too, so q.json and ./q.json differ. Suggestion: give None (unresolved) when the file is missing.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a: Rebuilt scorecards still read the questions file from the cwd. Tracked in #854; this thread stays open until it lands. b: A missing file now leaves eval_set_digest unresolved. Test added.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a: Rebuilt scorecards still read the questions file from the cwd. Tracked in #854; this thread stays open until it lands. b: A missing file now leaves eval_set_digest unresolved. Test added.

…protocol

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…a protocol

The protocol digest does not cover harness, and the fallback check that
compared it only ran without protocols, so two results from different
harnesses compared as equal.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at 1cd1ebb. The consumer checks are much stronger after the earlier rounds, but some inputs that change a result still do not reach the protocol: a mixed judge context, the correction family, and a missing questions file. Also, the stored comparable flag ignores unresolved inputs. Details are in the line comments.

Agreed with @exai-bot on the questions digest taken when the scorecard is rebuilt; I added a case to its thread that this PR can fix.

What I ran, at 1cd1ebb:

  • make ci-local --offline: 0 failed, 23 passed, 20 skipped (network-only jobs).
  • Repros through _result / compare(): a cell with every judge field gives comparable: true with unresolved: ["repeats"], while compare(r, copy) rejects it; two treatment_material.v1+snippet_fallback cells compare clean; family_size 2 vs 6 compares clean; is_alias() is False for claude-sonnet-4-5, gpt-4.1, gpt-5-mini; _digest_values of a missing file is the same value from any cwd.

Checked, no issues

  • Earlier threads: the fixes for the stated-vs-resolved fields, metric_spec, protocol on one side, omitted and null/alias inputs, p_decimals bounds, p and alpha ranges, recorded ci_method and opus[1m] hold on this head.
  • Schema: skill_evidence.v1 and matrix_rollup.v1 are only widened; schema_compat passes, and the export of a scorecard without ci_method validates.
  • Changelog: the fragment says what a result now carries and that repeats and bare family names leave results not comparable across runs.

Comment on lines +167 to +169
reasons = comparability(result)
result["comparable"] = not reasons
result["non_comparable"] = reasons

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A result says comparable: true while its protocol lists unresolved inputs, so the stored flag and compare() disagree.

comparable comes from comparability(), which never looks at protocol.unresolved. No run records repeats, so every judged result dokimi writes today has unresolved: ["repeats"] and comparable: true, yet compare(r, copy_of_r) returns ['protocol inputs not resolved: repeats']. A reader who trusts the field merges results the protocol cannot pin down.

Suggestion: add a reason to comparability() when the protocol has unresolved inputs (the _unresolved() set), or rename the field so it does not claim comparability.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comparable stays per-result: it is the result's own verdict, and compare() is the pair check that reports the unresolved input it names. Making it false for every result without repeats would change all current results. Open to a rename if you prefer.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comparable stays per-result: it is the result's own verdict, and compare() is the pair check that reports the unresolved input it names. Making it false for every result without repeats would change all current results. Open to a rename if you prefer.

Comment thread dokimi/dokimi/subjects/evidence.py Resolved
Comment thread dokimi/dokimi/subjects/evidence.py Resolved
Comment on lines +43 to +52
def _normal(value: Any) -> Any:
# An integral float is written as an int, so 0 and 0.0 (or 3 and 3.0) are one measurement.
if isinstance(value, float) and value.is_integer():
return int(value)
if isinstance(value, Mapping):
return {str(k): _normal(v) for k, v in value.items()}
if isinstance(value, (list, tuple)):
return [_normal(v) for v in value]
return value

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: _normal repeats commands/headroom._canonical (integral floats to ints, recursively) for the same purpose, a digest that is one for 95 and 95.0.

Two copies of one digest rule can drift apart: headroom guards bool explicitly, this one relies on isinstance(True, float) being False.

Suggestion: move it into one shared helper and use it in both places.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Declined here: both copies give the same digest for every input we have (bool is never a float), so this is a refactor, not a bug. Happy to take it as its own change.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Declined here: both copies give the same digest for every input we have (bool is never a float), so this is a refactor, not a bug. Happy to take it as its own change.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…gest unresolved

Thread: PRRT_kwDOVBg7g864IfKs
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g864Z2pL
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The change is in a904610 (Thread: PRRT_kwDOVBg7g864IfKs); this commit names item b of that thread.

Thread: PRRT_kwDOVBg7g864IfKs.b
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…file contract

A question file that is not found now has no digest, so the test that expected a hash of the name is replaced.

Test-narrowed: test_a_question_value_with_a_lone_surrogate_is_digested: a missing file is unresolved (None), not hashed by name; the name-hash contract is gone.
Thread: PRRT_kwDOVBg7g864IfKs.b
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Test-narrowed: test_a_question_value_with_a_lone_surrogate_is_digested now asserts None: a missing file has no digest, the name is not hashed.
Thread: PRRT_kwDOVBg7g864IfKs.b
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov

Copy link
Copy Markdown
Contributor Author

Re: review by vlad-perevezentsev

Mixed context, correction family, missing file, comparable flag
Each has its own thread: context and missing file fixed here; family and alias held for you; comparable declined; questions digest in #854.

…napshot resolves an alias

Protocol version 2: family_tested joins the resolved inputs, so a result corrected over a
different number of tests is not comparable with one corrected over another number. Version
1 digests are not comparable with version 2.

A claude-cli judge records the snapshot it ran (judge_resolved_models). When the stated judge
is an alias and exactly one snapshot was recorded, the protocol resolves the judge to it;
two or none leave it unresolved. The rollup and scorecard cells carry judge_resolved_models
when the run recorded it, and the skill_evidence and cell schemas describe it.

Thread: PRRT_kwDOVBg7g864Z2pR
Thread: PRRT_kwDOVBg7g864Z2pW
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov

Copy link
Copy Markdown
Contributor Author

Mixed context, correction family, missing file, comparable flag
Each has its own thread: context and missing file fixed here; family and alias held for you; comparable declined; questions digest in #854.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants