Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
135432b
feat(subjects): skill_evidence results say when they may be compared
napetrov Nov 16, 2026
896f332
fix(subjects): answer self-review: rounding near alpha, recorded judg…
napetrov Nov 16, 2026
791d106
test(subjects): the fixture test checks the fixture set and its digests
napetrov Nov 16, 2026
14d9b32
Merge branch 'main' of https://github.com/intel/dokimi into evidence-…
napetrov Nov 16, 2026
4392402
Merge branch 'main' of https://github.com/intel/dokimi into evidence-…
napetrov Nov 16, 2026
76c09e9
Address review: check what a result states against its protocol
napetrov Nov 16, 2026
eeb71e8
Address review: answer the review body of vchamarthi
napetrov Nov 16, 2026
38225c2
Merge commit '9aa10c7101429cbabfa61ae77eaf3fa5f124a16f' into evidence…
napetrov Nov 16, 2026
674aa80
Address review: judge aliases, unresolved values and the export of a …
napetrov Nov 16, 2026
52d4945
Address review: a p-value or alpha outside [0, 1] gives no verdict
napetrov Nov 16, 2026
ca7769b
Merge intel/main into evidence-protocol
napetrov Nov 16, 2026
6553155
Merge branch 'main' of https://github.com/intel/dokimi into evidence-…
napetrov Nov 17, 2026
1cd1ebb
subjects: a harness change is not comparable when both results carry …
napetrov Nov 17, 2026
633a852
Merge remote-tracking branch 'intel/main' into evidence-protocol
napetrov Nov 17, 2026
a904610
Address review: a questions file not found from the cwd leaves its di…
napetrov Nov 17, 2026
dfd982e
Address review: a mixed judge context is an unresolved protocol input
napetrov Nov 17, 2026
0da9b1b
Address review: the missing-file digest is unresolved (item b)
napetrov Nov 17, 2026
fb83e28
Address review: narrow the lone-surrogate digest test to the missing-…
napetrov Nov 17, 2026
14af859
Address review: say why the lone-surrogate digest test was narrowed
napetrov Nov 17, 2026
46f07a2
Address review: family_tested is a protocol input; a recorded judge s…
napetrov Nov 18, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
<!--
Copyright (C) 2026 Intel Corporation
SPDX-License-Identifier: Apache-2.0
-->

### Added

- **`subjects evidence`: each result says when it may be compared.** A result now carries `metric_spec` (unit, range, direction, judged), the `verdict_rule` its verdict follows (`p_adjusted` below `alpha`), and `protocol {id, version, digest}`: the digest is `sha256` over the canonical JSON of the resolved measurement inputs (judge snapshot, sampling, judge prompt hash, eval-set digest, repeats, test, correction, thresholds). A missing input or an alias such as `latest` is listed in `protocol.unresolved` and left out of the digest. `comparable`/`non_comparable` say why a result cannot be compared. `dokimi.subjects.evidence_check` holds the consumer checks, with conformance fixtures in `tests/fixtures/evidence_conformance`. Matrix-rollup and scorecard cells now carry the judge the run recorded (`judge_model`, `judge_provider`, `judge_context`, `judge_temperature`, `judge_prompt_sha256`). Runs do not record a repeat count, and a judge named by a bare family name (`opus`, `sonnet`) is an alias, so such results list those inputs as unresolved and are not comparable across runs.
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
<!--
Copyright (C) 2026 Intel Corporation
SPDX-License-Identifier: Apache-2.0
-->

### Changed

- **`dokimi.awareness` protocol is version 2**: `family_tested` (the multiplier the correction ran over)
is a protocol input, so a result corrected over a different number of tests is no longer comparable
with one that was not. Version 1 digests cannot be compared with version 2. A judge that
recorded exactly one snapshot resolves an alias judge (`opus`) to it; the rollup and scorecard cells
carry `judge_resolved_models`.
7 changes: 7 additions & 0 deletions dokimi/dokimi/eval/matrix_rollup.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,13 @@ def build_matrix_rollup(
"plugin_set": data.get("plugin_set", "none"),
"plugin_set_id": data.get("plugin_set_id"),
"baseline_arm": data.get("baseline_arm"),
"judge_model": data.get("judge_model"),
"judge_provider": data.get("judge_provider"),
"judge_context": data.get("judge_context"),
"judge_temperature": data.get("judge_temperature"),
"judge_prompt_sha256": data.get("judge_prompt_sha256"),
**({"judge_resolved_models": data["judge_resolved_models"]}
if "judge_resolved_models" in data else {}),
"summary": data.get("summary", {}),
"cost_summary": data.get("cost_summary", {}),
}
Expand Down
7 changes: 7 additions & 0 deletions dokimi/dokimi/schemas/arms.v1.json
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,13 @@
"null"
]
},
"judge_resolved_models": {
"type": "array",
"items": {
"type": "string"
},
"description": "the snapshot ids the judge reported, sorted; present when the run recorded them"
},
"judge_context": {
"description": "Protocol for what the judge was shown about each arm's treatment material. treatment_material.v1 when every score saw the stored material; treatment_material.v1+snippet_fallback when only some did (pools with nothing). Absent when every score used the 200-char report snippets — the protocol of runs judged before the stamp existed.",
"type": "string"
Expand Down
36 changes: 36 additions & 0 deletions dokimi/dokimi/schemas/matrix_rollup.v1.json
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,42 @@
"string",
"null"
]
},
"judge_model": {
"type": [
"string",
"null"
]
},
"judge_resolved_models": {
"type": "array",
"items": {
"type": "string"
}
},
"judge_provider": {
"type": [
"string",
"null"
]
},
"judge_context": {
"type": [
"string",
"null"
]
},
"judge_temperature": {
"type": [
"number",
"null"
]
},
"judge_prompt_sha256": {
"type": [
"string",
"null"
]
}
},
"additionalProperties": true
Expand Down
104 changes: 104 additions & 0 deletions dokimi/dokimi/schemas/skill_evidence.v1.json
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,13 @@
]
},
"judge_model": {
"type": [
"string",
"null"
],
"description": "judge snapshot id when the run recorded exactly one for an alias judge, else the configured judge model"
},
"judge_provider": {
"type": [
"string",
"null"
Expand All @@ -189,6 +196,30 @@
"type": "string",
"description": "dokimi's mean judge aggregate score per answer (higher is better); delta is arm minus baseline"
},
"metric_spec": {
"type": "object",
"description": "what the metric is; a verdict needs unit and higher_is_better to be comparable",
"properties": {
"id": {
"type": "string"
},
"unit": {
"type": "string"
},
"range": {
"type": "array",
"items": {
"type": "number"
}
},
"higher_is_better": {
"type": "boolean"
},
"judged": {
"type": "boolean"
}
}
},
"n": {
"type": [
"integer",
Expand Down Expand Up @@ -245,12 +276,85 @@
"null"
]
},
"verdict_rule": {
"type": "object",
"description": "how verdict follows from delta and p_value; a consumer recomputes it",
"properties": {
"basis": {
"enum": [
"p_adjusted"
]
},
"alpha": {
"type": "number"
},
"test": {
"type": "string"
},
"correction": {
"type": [
"string",
"null"
]
},
"p_decimals": {
"type": "integer",
"description": "places p_value is rounded to; within that of alpha a verdict cannot be recomputed"
}
}
},
"verdict": {
"enum": [
"improved",
"regressed",
"inconclusive"
]
},
"protocol": {
"type": "object",
"description": "the measurement: digest is sha256 of the canonical JSON (sorted keys, no whitespace, UTF-8) of resolved; missing or alias inputs are listed in unresolved, outside the digest",
"required": [
"id",
"version",
"digest"
],
"properties": {
"id": {
"type": "string"
},
"version": {
"type": "string"
},
"digest": {
"type": "string",
"pattern": "^sha256:[0-9a-f]{64}$"
},
"resolved": {
"type": "object"
},
"unresolved": {
"type": "array",
"items": {
"type": "string"
}
},
"aliases": {
"type": "object",
"additionalProperties": {
"type": "string"
}
}
}
},
"comparable": {
"type": "boolean"
},
"non_comparable": {
"type": "array",
"items": {
"type": "string"
},
"description": "why this result cannot be compared or shown with its verdict"
}
}
}
Expand Down
8 changes: 7 additions & 1 deletion dokimi/dokimi/schemas/subject_scorecard.v2.json
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@
}
},
"question_set_hash": {
"type": "string"
"type": ["string", "null"]
},
"tasks": {
"type": "array",
Expand Down Expand Up @@ -236,6 +236,12 @@
"null"
]
},
"judge_resolved_models": {
"type": "array",
"items": {
"type": "string"
}
},
"atoms": {
"type": "object"
},
Expand Down
71 changes: 67 additions & 4 deletions dokimi/dokimi/subjects/evidence.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@

from dokimi import __version__
from dokimi.eval.comparison_verdict import decide
from dokimi.eval.sign_flip import ALPHA
from dokimi.subjects.evidence_check import build_protocol, comparability, is_alias

#: The arm metric the awareness summary carries per arm (``summary.per_arm[arm]``).
METRIC = "avg_aggregate"
Expand All @@ -40,7 +42,7 @@
if baseline is None:
warnings.append(f"matrix cell {cell.get('matrix_cell')!r}: no single baseline arm, skipped")
continue
results.extend(_result(cell, arm, row) for arm, row in per_arm.items() if arm != baseline)
results.extend(_result(cell, arm, row, suite) for arm, row in per_arm.items() if arm != baseline)
if subject.get("distribution_digest") is None:
warnings.append("subject has no distribution_digest (not a SKILL.md, a skill with resources, "
"or a non-local ref): a client cannot match this document by digest")
Expand Down Expand Up @@ -85,22 +87,77 @@
return candidates[0] if len(candidates) == 1 else None


def _result(cell: dict[str, Any], arm: str, row: dict[str, Any]) -> dict[str, Any]:
#: The places ``stats._round_p`` rounds ``p_adjusted`` to before it reaches the scorecard.
P_DECIMALS = 4

#: What ``METRIC`` is: a 0-100 judge score per answer.
METRIC_SPEC = {"id": METRIC, "unit": "score_points", "range": [0, 100],

Check warning on line 94 in dokimi/dokimi/subjects/evidence.py

View workflow job for this annotation

GitHub Actions / Mutation of changed lines

mutant survived: bump-int `0` -> `1`
"higher_is_better": True, "judged": True}


def _eval_set_digest(cell: dict[str, Any], suite: dict[str, Any] | None) -> str | None:
"""The digest of the question set this cell measured; the suite's only when one set serves every cell."""
if cell.get("questions_digest") is not None:
return cell["questions_digest"]
Comment on lines +100 to +101

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix — reconstructed question provenance is treated as a resolved measurement input. Trigger: retain an awareness rollup, edit its questions file, then rebuild the scorecard and evidence. Evidence: scorecard.py creates questions_digest by reading the file during scorecard construction, not from recorded run metadata; test_each_cell_of_a_two_question_set_suite_carries_the_digest_of_its_own_set even verifies that rebuilding with unchanged runs after editing a.json changes this digest. This branch then hashes that new value into the old statistics' protocol as resolved. Impact: an old measurement can acquire the identity of a different evaluation set and, when the remaining inputs match, be authorized for comparison with measurements of that new set. Workaround: preserve the original question files and avoid reconstructing historical scorecards. Capture the question digest when the run loads its questions and propagate that recorded value; leave eval_set_digest unresolved when that provenance is unavailable, including the suite fallback. Severity justification: medium because this is a bounded but material false measurement identity, not security compromise or data loss.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping question paths cwd-relative for now, as decided. The gap is real: a rebuilt scorecard records an edited questions file as resolved. Filed as #854; this thread stays open until that lands.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, and #854 is right for the rebuild case. One part can be fixed in this PR: when the questions file is not found from the cwd, _digest_values hashes only its name, and that value goes into protocol.resolved.eval_set_digest. Two subjects that both name questions/q.json and are built where the file is missing get the same digest whatever the files hold. The path string is hashed too, so q.json and ./q.json differ. Suggestion: give None (unresolved) when the file is missing.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a: Rebuilt scorecards still read the questions file from the cwd. Tracked in #854; this thread stays open until it lands. b: A missing file now leaves eval_set_digest unresolved. Test added.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a: Rebuilt scorecards still read the questions file from the cwd. Tracked in #854; this thread stays open until it lands. b: A missing file now leaves eval_set_digest unresolved. Test added.

questions = (suite or {}).get("questions") or []
if len(questions) == 1:
return (suite or {}).get("question_set_hash")
return None


def _result(cell: dict[str, Any], arm: str, row: dict[str, Any],
suite: dict[str, Any] | None = None) -> dict[str, Any]:
# Every statistic is the paired test's, so n, delta, CI and p describe the same questions.
# The per-arm means are left out: their difference is not the delta once pairs drop out.
sig = row.get("significance") or {}
delta = sig.get("delta_mean") if sig else None
return {
correction = sig.get("correction")
judge = cell.get("judge_model")
# A claude-cli judge records the snapshot it ran; one recorded id resolves an alias judge (``opus``).
# Two ids, or none, leave the alias unresolved: which one scored is not known.
snapshots = cell.get("judge_resolved_models")
if is_alias(judge) and isinstance(snapshots, list) and len(snapshots) == 1 and isinstance(snapshots[0], str):
judge = snapshots[0]
sampling = cell.get("judge_sampling")
if sampling is None and cell.get("judge_temperature") is not None:
sampling = {"temperature": cell.get("judge_temperature")}
# The recorded ci_method, not the current implementation's constant: a historical scorecard keeps
# the method it ran. Unrecorded, it is None and the protocol lists it as unresolved.
test = sig.get("ci_method")
# The rule the verdict below applies, so a consumer can recompute it from delta and p_value.
# p_value is p_adjusted rounded to P_DECIMALS; within that rounding of alpha it cannot be recomputed.
rule = {"basis": "p_adjusted", "alpha": ALPHA, "correction": correction, "p_decimals": P_DECIMALS}
if test is not None: # the schema has no null test: an unrecorded one is left out, not written as null
rule["test"] = test
# Only resolved values enter the digest; what the scorecard does not record stays unresolved.
protocol = build_protocol({
"eval_set_digest": _eval_set_digest(cell, suite),
"metric": METRIC,
"judge_model": judge,
"judge_provider": cell.get("judge_provider"),
"judge_context": cell.get("judge_context"),
Comment thread
napetrov marked this conversation as resolved.
Resolved
"judge_sampling": sampling,
"judge_prompt_sha256": cell.get("judge_prompt_sha256"),
"repeats": cell.get("repeats"),
"test": test,
"correction": correction,
Comment thread
napetrov marked this conversation as resolved.
Resolved
"family_tested": sig.get("family_tested"),
"alpha": ALPHA,
"ci_confidence": sig.get("ci_confidence"),
})
result = {
"product": cell.get("product"),
"matrix_cell": cell.get("matrix_cell"),
"model": cell.get("model"),
"provider": cell.get("provider"),
"harness": cell.get("harness"),
# Absent until a run records the effort it applied; null means "not recorded", not "default".
"effort": cell.get("effort"),
"judge_model": cell.get("judge_model"),
"judge_model": judge,
"judge_provider": cell.get("judge_provider"),
"arm": arm,
"metric": METRIC,
"metric_spec": dict(METRIC_SPEC),
"n": sig.get("n"),
"delta": delta,
"ci": None if sig.get("ci_lo") is None else {
Expand All @@ -109,8 +166,14 @@
},
"p_value": sig.get("p_adjusted"),
"p_correction": sig.get("correction"),
"verdict_rule": rule,
"verdict": _verdict(delta, sig),
"protocol": protocol,
}
reasons = comparability(result)
result["comparable"] = not reasons
result["non_comparable"] = reasons
Comment on lines +173 to +175

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A result says comparable: true while its protocol lists unresolved inputs, so the stored flag and compare() disagree.

comparable comes from comparability(), which never looks at protocol.unresolved. No run records repeats, so every judged result dokimi writes today has unresolved: ["repeats"] and comparable: true, yet compare(r, copy_of_r) returns ['protocol inputs not resolved: repeats']. A reader who trusts the field merges results the protocol cannot pin down.

Suggestion: add a reason to comparability() when the protocol has unresolved inputs (the _unresolved() set), or rename the field so it does not claim comparability.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comparable stays per-result: it is the result's own verdict, and compare() is the pair check that reports the unresolved input it names. Making it false for every result without repeats would change all current results. Open to a rename if you prefer.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comparable stays per-result: it is the result's own verdict, and compare() is the pair check that reports the unresolved input it names. Making it false for every result without repeats would change all current results. Open to a rename if you prefer.

return result


def _verdict(delta: float | None, sig: dict[str, Any]) -> str:
Expand Down
Loading
Loading