Repository navigation
Conversation
Each result carries metric_spec, the verdict_rule its verdict follows and
protocol {id, version, digest}: sha256 over the canonical JSON of the
resolved measurement inputs only. A missing input or an alias such as
'latest' goes to protocol.unresolved, outside the digest. comparable and
non_comparable give the reasons a result cannot be compared.
dokimi.subjects.evidence_check holds the consumer checks; three conformance
fixtures pin them. Rollup and scorecard cells now pass through the judge.
Signed-off-by: Petrov, Nikolay A <nikolay.a.petrov@intel.com>
…e inputs, bare judge names A p_value within its rounding of alpha no longer reads as a verdict that contradicts its rule. Rollup and scorecard cells carry judge_temperature and judge_prompt_sha256, so the protocol resolves them. Bare family names (opus, sonnet, gpt-4o) and aliases nested in a value count as unresolved. Integral floats hash as ints. Signed-off-by: Petrov, Nikolay A <nikolay.a.petrov@intel.com>
a602451 to
896f332
Compare
|
OpenCodeReview: Commit 46f07a2 was not reviewed. This PR has reached its review limit (6 reviews). CI at `46f07a2`: 12 still running, 4 passing, 1 skipped
2 more not listed. To review this commit: re-request review from exai-bot in the Reviewers list. <details><summary>Last review (1cd1ebb): no findings</summary> Open threads from the bot: 1 </details> |
Signed-off-by: Petrov, Nikolay A <nikolay.a.petrov@intel.com>
vchamarthi
left a comment
There was a problem hiding this comment.
🔴 Changes needed: 1 must-fix, 1 nit · reviewed 791d106
Next: @napetrov: fix the 1 must-fix finding and answer 7 open threads from exai-bot; then push for a re-review. 1 nit optional.
Blocking
- Still holds on this head: thread by exai-bot –
compare(copy, original)with onlyjudge_modelchanged toclaude-sonnet-5-5-20261101returns[];verdict_rule.alphais not cross-checked againstprotocol.resolvedeither. - Still holds on this head: thread by exai-bot –
comparewith onlymetric_spec.unitchanged topercentreturns[]. - Still holds on this head: thread by exai-bot –
compareof a result whose protocol hasrepeatsunresolved against a copy with noprotocolreturns[]. - Still holds on this head: thread by exai-bot –
_resultsetsrule["test"]and the protocoltestto the constantMETHODwhateverci_methodthe scorecard recorded. - Still holds on this head: thread by exai-bot –
eval_set_digestissuite.question_set_hash, whichscorecard.py:102computes over all questions of the suite, not the cell's own set. - Still holds on this head: thread by exai-bot –
compare(r, r)with a protocol built from onlyjudge_modelreturns[], sincebuild_protocolvisits only supplied keys. - Still holds on this head: thread by exai-bot –
comparabilitywithverdict_rule.p_decimalsof-1000raisesOverflowErrorat_near_alpha.
Verified on 791d106: ✅ changed tests: 33 passed, fail on main · ❌ full suite with main merged in fails at tests/test_growth_ratchet.py::test_no_file_or_function_grew_past_its_baseline, tests/test_model_defaults.py::test_no_source_file_still_names_gpt_4o_mini, ci.yml:typecheck / mypy · ✅ merges onto main · 2 mutations tried · 1 edge input tried
3 checks behind it, each with its evidence
| Area | What holds | How |
|---|---|---|
| Verdict recompute | Flipping the direction in recompute_verdict fails 9 of 60 tests in test_evidence_protocol.py and test_subjects_evidence.py |
broke the code, tests caught it |
| Changed tests | The changed tests pass on the head | tests on the head |
| Merge | The branch merges onto main without conflict | merge onto main |
…protocol Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> # Conflicts: # dokimi/dokimi/subjects/evidence.py
…protocol Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The consumer checks now refuse a result whose stated judge, provider, metric specification or verdict rule differs from what its protocol resolved, a protocol on one side of a pair only, and an omitted dokimi.awareness v1 input (unresolved, not resolved). An unsupported rounding precision is a reason, not an overflow. The exporter takes the test from the recorded ci_method, and each cell carries the digest of its own question file, so a multi-set suite no longer shares one eval-set digest. The fixture README takes the repo's license header. Thread: PRRT_kwDOVBg7g863y74Z Thread: PRRT_kwDOVBg7g863y74c Thread: PRRT_kwDOVBg7g863y74g Thread: PRRT_kwDOVBg7g863y74k Thread: PRRT_kwDOVBg7g863y74r Thread: PRRT_kwDOVBg7g863zDgM Thread: PRRT_kwDOVBg7g863zDgP Thread: PRRT_kwDOVBg7g863zjbt Thread: PRRT_kwDOVBg7g863zjb1 Thread: PRRT_kwDOVBg7g863zjb6 Thread: PRRT_kwDOVBg7g863z_n6 Thread: PRRT_kwDOVBg7g863z_oG Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRR_kwDOVBg7g88AAAABYKB_6w: already fixed in 76c09e9 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #804 +/- ##
==========================================
- Coverage 95.79% 95.78% -0.01%
==========================================
Files 432 433 +1
Lines 57204 57419 +215
Branches 9858 9907 +49
==========================================
+ Hits 54796 54999 +203
- Misses 1497 1502 +5
- Partials 911 918 +7
🚀 New features to boost your workflow:
|
…-protocol Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…historical scorecard Thread: PRRT_kwDOVBg7g864FRtF Thread: PRRT_kwDOVBg7g864FRtM Thread: PRRT_kwDOVBg7g864FRtR Thread: PRRT_kwDOVBg7g864FRta Thread: PRRT_kwDOVBg7g864FRtj Thread: IC_kwDOVBg7g88AAAABm8bdwg#6 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
A negative p-value was below alpha, so recompute_verdict read it as "improved" and comparability passed it with no reason. Outside [0, 1] (p) or (0, 1) (alpha) the rule now cannot be applied. Thread: PRRT_kwDOVBg7g864Gw0t Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
| if cell.get("questions_digest") is not None: | ||
| return cell["questions_digest"] |
There was a problem hiding this comment.
Must-fix — reconstructed question provenance is treated as a resolved measurement input. Trigger: retain an awareness rollup, edit its questions file, then rebuild the scorecard and evidence. Evidence: scorecard.py creates questions_digest by reading the file during scorecard construction, not from recorded run metadata; test_each_cell_of_a_two_question_set_suite_carries_the_digest_of_its_own_set even verifies that rebuilding with unchanged runs after editing a.json changes this digest. This branch then hashes that new value into the old statistics' protocol as resolved. Impact: an old measurement can acquire the identity of a different evaluation set and, when the remaining inputs match, be authorized for comparison with measurements of that new set. Workaround: preserve the original question files and avoid reconstructing historical scorecards. Capture the question digest when the run loads its questions and propagate that recorded value; leave eval_set_digest unresolved when that provenance is unavailable, including the suite fallback. Severity justification: medium because this is a bounded but material false measurement identity, not security compromise or data loss.
There was a problem hiding this comment.
Keeping question paths cwd-relative for now, as decided. The gap is real: a rebuilt scorecard records an edited questions file as resolved. Filed as #854; this thread stays open until that lands.
There was a problem hiding this comment.
Agreed, and #854 is right for the rebuild case. One part can be fixed in this PR: when the questions file is not found from the cwd, _digest_values hashes only its name, and that value goes into protocol.resolved.eval_set_digest. Two subjects that both name questions/q.json and are built where the file is missing get the same digest whatever the files hold. The path string is hashed too, so q.json and ./q.json differ. Suggestion: give None (unresolved) when the file is missing.
There was a problem hiding this comment.
a: Rebuilt scorecards still read the questions file from the cwd. Tracked in #854; this thread stays open until it lands. b: A missing file now leaves eval_set_digest unresolved. Test added.
There was a problem hiding this comment.
a: Rebuilt scorecards still read the questions file from the cwd. Tracked in #854; this thread stays open until it lands. b: A missing file now leaves eval_set_digest unresolved. Test added.
…protocol Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…a protocol The protocol digest does not cover harness, and the fallback check that compared it only ran without protocols, so two results from different harnesses compared as equal. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
vlad-perevezentsev
left a comment
There was a problem hiding this comment.
Review at 1cd1ebb. The consumer checks are much stronger after the earlier rounds, but some inputs that change a result still do not reach the protocol: a mixed judge context, the correction family, and a missing questions file. Also, the stored comparable flag ignores unresolved inputs. Details are in the line comments.
Agreed with @exai-bot on the questions digest taken when the scorecard is rebuilt; I added a case to its thread that this PR can fix.
What I ran, at 1cd1ebb:
make ci-local --offline: 0 failed, 23 passed, 20 skipped (network-only jobs).- Repros through
_result/compare(): a cell with every judge field givescomparable: truewithunresolved: ["repeats"], whilecompare(r, copy)rejects it; twotreatment_material.v1+snippet_fallbackcells compare clean;family_size2 vs 6 compares clean;is_alias()is False forclaude-sonnet-4-5,gpt-4.1,gpt-5-mini;_digest_valuesof a missing file is the same value from any cwd.
Checked, no issues
- Earlier threads: the fixes for the stated-vs-resolved fields,
metric_spec, protocol on one side, omitted and null/alias inputs,p_decimalsbounds, p and alpha ranges, recordedci_methodandopus[1m]hold on this head. - Schema:
skill_evidence.v1andmatrix_rollup.v1are only widened; schema_compat passes, and the export of a scorecard withoutci_methodvalidates. - Changelog: the fragment says what a result now carries and that
repeatsand bare family names leave results not comparable across runs.
| reasons = comparability(result) | ||
| result["comparable"] = not reasons | ||
| result["non_comparable"] = reasons |
There was a problem hiding this comment.
A result says comparable: true while its protocol lists unresolved inputs, so the stored flag and compare() disagree.
comparable comes from comparability(), which never looks at protocol.unresolved. No run records repeats, so every judged result dokimi writes today has unresolved: ["repeats"] and comparable: true, yet compare(r, copy_of_r) returns ['protocol inputs not resolved: repeats']. A reader who trusts the field merges results the protocol cannot pin down.
Suggestion: add a reason to comparability() when the protocol has unresolved inputs (the _unresolved() set), or rename the field so it does not claim comparability.
There was a problem hiding this comment.
comparable stays per-result: it is the result's own verdict, and compare() is the pair check that reports the unresolved input it names. Making it false for every result without repeats would change all current results. Open to a rename if you prefer.
There was a problem hiding this comment.
comparable stays per-result: it is the result's own verdict, and compare() is the pair check that reports the unresolved input it names. Making it false for every result without repeats would change all current results. Open to a rename if you prefer.
| def _normal(value: Any) -> Any: | ||
| # An integral float is written as an int, so 0 and 0.0 (or 3 and 3.0) are one measurement. | ||
| if isinstance(value, float) and value.is_integer(): | ||
| return int(value) | ||
| if isinstance(value, Mapping): | ||
| return {str(k): _normal(v) for k, v in value.items()} | ||
| if isinstance(value, (list, tuple)): | ||
| return [_normal(v) for v in value] | ||
| return value | ||
|
|
There was a problem hiding this comment.
Nit: _normal repeats commands/headroom._canonical (integral floats to ints, recursively) for the same purpose, a digest that is one for 95 and 95.0.
Two copies of one digest rule can drift apart: headroom guards bool explicitly, this one relies on isinstance(True, float) being False.
Suggestion: move it into one shared helper and use it in both places.
There was a problem hiding this comment.
Declined here: both copies give the same digest for every input we have (bool is never a float), so this is a refactor, not a bug. Happy to take it as its own change.
There was a problem hiding this comment.
Declined here: both copies give the same digest for every input we have (bool is never a float), so this is a refactor, not a bug. Happy to take it as its own change.
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…gest unresolved Thread: PRRT_kwDOVBg7g864IfKs Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g864Z2pL Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The change is in a904610 (Thread: PRRT_kwDOVBg7g864IfKs); this commit names item b of that thread. Thread: PRRT_kwDOVBg7g864IfKs.b Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…file contract A question file that is not found now has no digest, so the test that expected a hash of the name is replaced. Test-narrowed: test_a_question_value_with_a_lone_surrogate_is_digested: a missing file is unresolved (None), not hashed by name; the name-hash contract is gone. Thread: PRRT_kwDOVBg7g864IfKs.b Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Test-narrowed: test_a_question_value_with_a_lone_surrogate_is_digested now asserts None: a missing file has no digest, the name is not hashed. Thread: PRRT_kwDOVBg7g864IfKs.b Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
Re: review by vlad-perevezentsev
|
…napshot resolves an alias Protocol version 2: family_tested joins the resolved inputs, so a result corrected over a different number of tests is not comparable with one corrected over another number. Version 1 digests are not comparable with version 2. A claude-cli judge records the snapshot it ran (judge_resolved_models). When the stated judge is an alias and exactly one snapshot was recorded, the protocol resolves the judge to it; two or none leave it unresolved. The rollup and scorecard cells carry judge_resolved_models when the run recorded it, and the skill_evidence and cell schemas describe it. Thread: PRRT_kwDOVBg7g864Z2pR Thread: PRRT_kwDOVBg7g864Z2pW Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
Follow-up to #344. A
skill_evidence.v1subject digest pins a skill's bytes, not how a number was measured, so two results about one digest can disagree on judge, sampling, prompt or thresholds, and a reader can't recompute a verdict. This PR implements what agentskills/agentskills#643 replies promised.What changes
metric_spec(unit,range,higher_is_better,judged);verdict_rule(basis: p_adjusted,alpha,test,correction,p_decimals);protocol {id, version, digest, resolved, unresolved[, aliases]}.protocol.digestissha256of the canonical JSON of the resolved inputs only: sorted keys, no whitespace, UTF-8, integral floats written as ints. The inputs are:unresolvedand stays out of the digest. Aliases are-latest/:default/auto, bare family names likeopus/sonnet/gpt-4o, and an alias nested inside a value.comparable/non_comparablegive the reasons a result cannot be compared:resolved.dokimi.subjects.evidence_checkwith the consumer checks:comparability,compare,mark_againstandrecompute_verdict.tests/fixtures/evidence_conformance:arms.v1already records (judge_model,judge_provider,judge_context,judge_temperature,judge_prompt_sha256).Behaviour to know
comparability()will mark their verdicts non-comparable (verdict without verdict_rule). That is intended: no rule means no recompute.opus/sonnetaliases. Their results list those inputs as unresolved, socomparereports them as not comparable across runs until runs record a snapshot id and repeats. That is a follow-up on the run side.p_valueisp_adjustedrounded to 4 places. Within that rounding of alpha, the verdict cannot be recomputed, and the result gets a reason saying so; it is not reported as a contradiction.Testing
tests/test_evidence_protocol.py: 33 tests, red then green, including an independenthashlibcheck of the digest.test_repo_receipt.py/test_catalog_eval_cli.py, which fail the same way on unmodifiedmainon this box.Self-review before push: local
pr-review, 2 passes. Pass 1 found 3 blocking issues:It also found 2 nits: int vs float in the digest, and aliases nested in a value. All are fixed with tests.
🤖 Generated with Claude Code