Repository navigation
feat(subjects): skill_evidence results say when they may be compared (protocol digest, verdict_rule, conformance fixtures) #804
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
135432b
896f332
791d106
14d9b32
4392402
76c09e9
eeb71e8
38225c2
674aa80
52d4945
ca7769b
6553155
1cd1ebb
633a852
a904610
dfd982e
0da9b1b
fb83e28
14af859
46f07a2
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| <!-- | ||
| Copyright (C) 2026 Intel Corporation | ||
| SPDX-License-Identifier: Apache-2.0 | ||
| --> | ||
|
|
||
| ### Added | ||
|
|
||
| - **`subjects evidence`: each result says when it may be compared.** A result now carries `metric_spec` (unit, range, direction, judged), the `verdict_rule` its verdict follows (`p_adjusted` below `alpha`), and `protocol {id, version, digest}`: the digest is `sha256` over the canonical JSON of the resolved measurement inputs (judge snapshot, sampling, judge prompt hash, eval-set digest, repeats, test, correction, thresholds). A missing input or an alias such as `latest` is listed in `protocol.unresolved` and left out of the digest. `comparable`/`non_comparable` say why a result cannot be compared. `dokimi.subjects.evidence_check` holds the consumer checks, with conformance fixtures in `tests/fixtures/evidence_conformance`. Matrix-rollup and scorecard cells now carry the judge the run recorded (`judge_model`, `judge_provider`, `judge_context`, `judge_temperature`, `judge_prompt_sha256`). Runs do not record a repeat count, and a judge named by a bare family name (`opus`, `sonnet`) is an alias, so such results list those inputs as unresolved and are not comparable across runs. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| <!-- | ||
| Copyright (C) 2026 Intel Corporation | ||
| SPDX-License-Identifier: Apache-2.0 | ||
| --> | ||
|
|
||
| ### Changed | ||
|
|
||
| - **`dokimi.awareness` protocol is version 2**: `family_tested` (the multiplier the correction ran over) | ||
| is a protocol input, so a result corrected over a different number of tests is no longer comparable | ||
| with one that was not. Version 1 digests cannot be compared with version 2. A judge that | ||
| recorded exactly one snapshot resolves an alias judge (`opus`) to it; the rollup and scorecard cells | ||
| carry `judge_resolved_models`. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -19,6 +19,8 @@ | |
|
|
||
| from dokimi import __version__ | ||
| from dokimi.eval.comparison_verdict import decide | ||
| from dokimi.eval.sign_flip import ALPHA | ||
| from dokimi.subjects.evidence_check import build_protocol, comparability, is_alias | ||
|
|
||
| #: The arm metric the awareness summary carries per arm (``summary.per_arm[arm]``). | ||
| METRIC = "avg_aggregate" | ||
|
|
@@ -40,7 +42,7 @@ | |
| if baseline is None: | ||
| warnings.append(f"matrix cell {cell.get('matrix_cell')!r}: no single baseline arm, skipped") | ||
| continue | ||
| results.extend(_result(cell, arm, row) for arm, row in per_arm.items() if arm != baseline) | ||
| results.extend(_result(cell, arm, row, suite) for arm, row in per_arm.items() if arm != baseline) | ||
| if subject.get("distribution_digest") is None: | ||
| warnings.append("subject has no distribution_digest (not a SKILL.md, a skill with resources, " | ||
| "or a non-local ref): a client cannot match this document by digest") | ||
|
|
@@ -85,22 +87,77 @@ | |
| return candidates[0] if len(candidates) == 1 else None | ||
|
|
||
|
|
||
| def _result(cell: dict[str, Any], arm: str, row: dict[str, Any]) -> dict[str, Any]: | ||
| #: The places ``stats._round_p`` rounds ``p_adjusted`` to before it reaches the scorecard. | ||
| P_DECIMALS = 4 | ||
|
|
||
| #: What ``METRIC`` is: a 0-100 judge score per answer. | ||
| METRIC_SPEC = {"id": METRIC, "unit": "score_points", "range": [0, 100], | ||
| "higher_is_better": True, "judged": True} | ||
|
|
||
|
|
||
| def _eval_set_digest(cell: dict[str, Any], suite: dict[str, Any] | None) -> str | None: | ||
| """The digest of the question set this cell measured; the suite's only when one set serves every cell.""" | ||
| if cell.get("questions_digest") is not None: | ||
| return cell["questions_digest"] | ||
| questions = (suite or {}).get("questions") or [] | ||
| if len(questions) == 1: | ||
| return (suite or {}).get("question_set_hash") | ||
| return None | ||
|
|
||
|
|
||
| def _result(cell: dict[str, Any], arm: str, row: dict[str, Any], | ||
| suite: dict[str, Any] | None = None) -> dict[str, Any]: | ||
| # Every statistic is the paired test's, so n, delta, CI and p describe the same questions. | ||
| # The per-arm means are left out: their difference is not the delta once pairs drop out. | ||
| sig = row.get("significance") or {} | ||
| delta = sig.get("delta_mean") if sig else None | ||
| return { | ||
| correction = sig.get("correction") | ||
| judge = cell.get("judge_model") | ||
| # A claude-cli judge records the snapshot it ran; one recorded id resolves an alias judge (``opus``). | ||
| # Two ids, or none, leave the alias unresolved: which one scored is not known. | ||
| snapshots = cell.get("judge_resolved_models") | ||
| if is_alias(judge) and isinstance(snapshots, list) and len(snapshots) == 1 and isinstance(snapshots[0], str): | ||
| judge = snapshots[0] | ||
| sampling = cell.get("judge_sampling") | ||
| if sampling is None and cell.get("judge_temperature") is not None: | ||
| sampling = {"temperature": cell.get("judge_temperature")} | ||
| # The recorded ci_method, not the current implementation's constant: a historical scorecard keeps | ||
| # the method it ran. Unrecorded, it is None and the protocol lists it as unresolved. | ||
| test = sig.get("ci_method") | ||
| # The rule the verdict below applies, so a consumer can recompute it from delta and p_value. | ||
| # p_value is p_adjusted rounded to P_DECIMALS; within that rounding of alpha it cannot be recomputed. | ||
| rule = {"basis": "p_adjusted", "alpha": ALPHA, "correction": correction, "p_decimals": P_DECIMALS} | ||
| if test is not None: # the schema has no null test: an unrecorded one is left out, not written as null | ||
| rule["test"] = test | ||
| # Only resolved values enter the digest; what the scorecard does not record stays unresolved. | ||
| protocol = build_protocol({ | ||
| "eval_set_digest": _eval_set_digest(cell, suite), | ||
| "metric": METRIC, | ||
| "judge_model": judge, | ||
| "judge_provider": cell.get("judge_provider"), | ||
| "judge_context": cell.get("judge_context"), | ||
|
napetrov marked this conversation as resolved.
Resolved
|
||
| "judge_sampling": sampling, | ||
| "judge_prompt_sha256": cell.get("judge_prompt_sha256"), | ||
| "repeats": cell.get("repeats"), | ||
| "test": test, | ||
| "correction": correction, | ||
|
napetrov marked this conversation as resolved.
Resolved
|
||
| "family_tested": sig.get("family_tested"), | ||
| "alpha": ALPHA, | ||
| "ci_confidence": sig.get("ci_confidence"), | ||
| }) | ||
| result = { | ||
| "product": cell.get("product"), | ||
| "matrix_cell": cell.get("matrix_cell"), | ||
| "model": cell.get("model"), | ||
| "provider": cell.get("provider"), | ||
| "harness": cell.get("harness"), | ||
| # Absent until a run records the effort it applied; null means "not recorded", not "default". | ||
| "effort": cell.get("effort"), | ||
| "judge_model": cell.get("judge_model"), | ||
| "judge_model": judge, | ||
| "judge_provider": cell.get("judge_provider"), | ||
| "arm": arm, | ||
| "metric": METRIC, | ||
| "metric_spec": dict(METRIC_SPEC), | ||
| "n": sig.get("n"), | ||
| "delta": delta, | ||
| "ci": None if sig.get("ci_lo") is None else { | ||
|
|
@@ -109,8 +166,14 @@ | |
| }, | ||
| "p_value": sig.get("p_adjusted"), | ||
| "p_correction": sig.get("correction"), | ||
| "verdict_rule": rule, | ||
| "verdict": _verdict(delta, sig), | ||
| "protocol": protocol, | ||
| } | ||
| reasons = comparability(result) | ||
| result["comparable"] = not reasons | ||
| result["non_comparable"] = reasons | ||
|
Comment on lines
+173
to
+175
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. A result says
Suggestion: add a reason to
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
| return result | ||
|
|
||
|
|
||
| def _verdict(delta: float | None, sig: dict[str, Any]) -> str: | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Must-fix — reconstructed question provenance is treated as a resolved measurement input. Trigger: retain an awareness rollup, edit its questions file, then rebuild the scorecard and evidence. Evidence: scorecard.py creates questions_digest by reading the file during scorecard construction, not from recorded run metadata; test_each_cell_of_a_two_question_set_suite_carries_the_digest_of_its_own_set even verifies that rebuilding with unchanged runs after editing a.json changes this digest. This branch then hashes that new value into the old statistics' protocol as resolved. Impact: an old measurement can acquire the identity of a different evaluation set and, when the remaining inputs match, be authorized for comparison with measurements of that new set. Workaround: preserve the original question files and avoid reconstructing historical scorecards. Capture the question digest when the run loads its questions and propagate that recorded value; leave eval_set_digest unresolved when that provenance is unavailable, including the suite fallback. Severity justification: medium because this is a bounded but material false measurement identity, not security compromise or data loss.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Keeping question paths cwd-relative for now, as decided. The gap is real: a rebuilt scorecard records an edited questions file as resolved. Filed as #854; this thread stays open until that lands.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed, and #854 is right for the rebuild case. One part can be fixed in this PR: when the questions file is not found from the cwd,
_digest_valueshashes only its name, and that value goes intoprotocol.resolved.eval_set_digest. Two subjects that both namequestions/q.jsonand are built where the file is missing get the same digest whatever the files hold. The path string is hashed too, soq.jsonand./q.jsondiffer. Suggestion: giveNone(unresolved) when the file is missing.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
a: Rebuilt scorecards still read the questions file from the cwd. Tracked in #854; this thread stays open until it lands. b: A missing file now leaves
eval_set_digestunresolved. Test added.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
a: Rebuilt scorecards still read the questions file from the cwd. Tracked in #854; this thread stays open until it lands. b: A missing file now leaves
eval_set_digestunresolved. Test added.