Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
a75c564
docs: judge-panel study of the skill-novelty headline numbers
napetrov Nov 8, 2026
6fe07ce
docs: fix judge-panel claims found in self-review
napetrov Nov 8, 2026
18d532d
feat(novelty): judge panel for skill novelty analyze and probe
napetrov Nov 8, 2026
838d465
Merge branch 'research/judge-panel' into feat/judge-panel-cli
napetrov Nov 8, 2026
585e340
fix(novelty): refuse a panel judge model given twice; self-judge warn…
napetrov Nov 8, 2026
3955f03
docs(research): judge generations, old vs new, on the judge-panel sample
napetrov Nov 8, 2026
11694ae
Merge branch 'research/judge-panel' into feat/judge-panel-cli
napetrov Nov 8, 2026
0d00e9c
Merge branch 'main' of https://github.com/intel/dokimi into feat/judg…
napetrov Nov 10, 2026
386856a
novelty: break the judge panel import cycle and shrink novelty.py
napetrov Nov 10, 2026
6f75fad
research/judge-panel: pass the static checks
napetrov Nov 10, 2026
93dd94b
docs: judge-panel study renders under strict mkdocs and is linked fro…
napetrov Nov 10, 2026
ce40b8a
research/judge-panel: keep the section 5 comment to two lines
napetrov Nov 10, 2026
d4d8464
docs: join the split sentence in the three-family panel bullet
napetrov Nov 10, 2026
6db47b0
deptry: exclude the research study scripts instead of ignoring llm by…
napetrov Nov 10, 2026
35337db
docs: blank line between two items of the panel report list
napetrov Nov 10, 2026
1dbb56b
ci: exclude the judge-panel study's generated tables and figures from…
napetrov Nov 10, 2026
ccdc44e
Merge branch 'main' into feat/judge-panel-cli
napetrov Nov 10, 2026
691b281
Merge remote-tracking branch 'origin/main' into HEAD
vchamarthi Nov 14, 2026
9dabb8b
fix(novelty): judge panel validation, shared-pair range, estimate cou…
vchamarthi Nov 14, 2026
840d418
Merge branch 'main' of https://github.com/intel/dokimi into pr489-judge
napetrov Nov 14, 2026
1fef17e
fix(novelty): judge panel verdicts, warnings and estimate
napetrov Nov 14, 2026
9329068
research/judge-panel: pin the study to its panel and undefined kappa
napetrov Nov 14, 2026
eace298
docs(novelty): say which vendor families the same-vendor warning knows
napetrov Nov 14, 2026
4919509
docs and tests: record the review points already fixed at 9dabb8b
napetrov Nov 14, 2026
fbacfe7
docs(novelty): name the grok and xai family in the vendor list
antonwolfy Nov 14, 2026
937c77f
fix(research): kappa is undefined when chance agreement is 1
antonwolfy Nov 14, 2026
644e3e4
fix(research): skip undefined shares in the panel-size spreads
antonwolfy Nov 14, 2026
3ef9e97
fix(research): leave undefined shares out of the judge dot plot
antonwolfy Nov 14, 2026
9e3c003
fix(research): measure old and new panels on the same cells
antonwolfy Nov 14, 2026
c6b0fb9
test(novelty): pin that probe and analyze pass the answerer to the pa…
antonwolfy Nov 14, 2026
ef1710b
docs(judge-panel): mark the generation rates not re-run
napetrov Nov 15, 2026
f69dc15
fix(novelty): errored panel votes, partial panel record, ProbeRun order
vchamarthi Nov 15, 2026
4fa2e8e
Merge origin/main into feat/judge-panel-cli
vchamarthi Nov 15, 2026
d1eabe2
fix(novelty): errored panel votes, partial panel record, ProbeRun order
vchamarthi Nov 15, 2026
d91b591
fix(novelty): panel regrade vs first judge, split majority test
vchamarthi Nov 15, 2026
418dac7
Merge branch 'main' of https://github.com/intel/dokimi into pr489-r3
napetrov Nov 16, 2026
3ff38d0
Address review: record items already fixed on this branch
napetrov Nov 16, 2026
202e80b
Merge branch 'main' of https://github.com/intel/dokimi into pr489-r3
napetrov Nov 16, 2026
030b840
fix(novelty): a panel stopped before every judge voted keeps its judges
napetrov Nov 16, 2026
ab3a4e8
docs: take main's README; its documentation index already links the j…
napetrov Nov 16, 2026
026dbde
docs(novelty): declare the judge_panel block in novelty.v1
napetrov Nov 16, 2026
3fdb501
Keep measure.py within the 800-line cap
napetrov Nov 16, 2026
23407a7
Merge branch 'main' of https://github.com/intel/dokimi into pr489-r3
napetrov Nov 17, 2026
9d0a302
Restore the pathlib import that the merge dropped
napetrov Nov 17, 2026
10a5fdd
docs(novelty): a permanent judge error stops the run, not the vote
antonwolfy Nov 17, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/.licenserc.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,9 @@ header:
# written verbatim as the agent's answer (.agent-eval/run.py).
- '.agent-eval/tasks/**/*.patch'
- '.agent-eval/tasks/**/oracle_answer.md'
# Generated study tables and figures (the judge-panel study's CSV and SVG, written by
# research/judge-panel): a header would break the CSV readers and is lost on regeneration.
- 'dokimi/docs/assets/judge-panel/**'
# Changelog fragments (<timestamp>_<user>_<branch>.md) and the template
# `scriv create` copies into each: `scriv collect` moves their text into
# CHANGELOG.md, which carries the header itself.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
### Added

- **Judge panel for skill novelty**: `skill novelty analyze` and `probe` now take `--judge PROVIDER/MODEL`. Repeat the flag to form a panel. `--judge-aggregate {median,majority,unanimous}` (default `median`) turns the panel's verdicts into one per answer. The record gets a `judge_panel` block: each judge's own paired 2×2, the judge range of the added share, and a `judge_sensitive` flag set when that range exceeds 0.10. Each run keeps every judge's verdict (`analyze` writes them to `<record>.judge_verdicts.json`), and the report names the panel. A warning is printed when two judges share a vendor family. The `--estimate-only` dollar figure and call counts include the extra judges. A model given twice, under any spelling, is refused. Background: `docs/judge-panel-study.md`.
4 changes: 4 additions & 0 deletions dokimi/docs/reference/cli-interface.json
Original file line number Diff line number Diff line change
Expand Up @@ -1611,6 +1611,8 @@
{"key": "--extract-provider", "action": "_StoreAction", "choices": ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"], "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--help/-h", "action": "_HelpAction", "choices": null, "const": null, "default": "==SUPPRESS==", "group_key": null, "group_required": null, "hidden": false, "is_flag": true, "kind": "option", "nargs": 0, "position": null, "required": false, "type": null},
{"key": "--include-resources", "action": "_StoreTrueAction", "choices": null, "const": true, "default": false, "group_key": null, "group_required": null, "hidden": false, "is_flag": true, "kind": "option", "nargs": 0, "position": null, "required": false, "type": null},
{"key": "--judge", "action": "_AppendAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--judge-aggregate", "action": "_StoreAction", "choices": ["'majority'", "'median'", "'unanimous'"], "const": null, "default": "median", "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--judge-model", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--judge-provider", "action": "_StoreAction", "choices": ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"], "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--judge2", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
Expand Down Expand Up @@ -1707,6 +1709,8 @@
{"key": "--extract-model", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--extract-provider", "action": "_StoreAction", "choices": ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"], "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--help/-h", "action": "_HelpAction", "choices": null, "const": null, "default": "==SUPPRESS==", "group_key": null, "group_required": null, "hidden": false, "is_flag": true, "kind": "option", "nargs": 0, "position": null, "required": false, "type": null},
{"key": "--judge", "action": "_AppendAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--judge-aggregate", "action": "_StoreAction", "choices": ["'majority'", "'median'", "'unanimous'"], "const": null, "default": "median", "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--judge-model", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--judge-provider", "action": "_StoreAction", "choices": ["'amazon-bedrock'", "'anthropic'", "'claude-cli'", "'google-vertex'", "'openai'", "'openai-codex'", "'openrouter'"], "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
{"key": "--judge2", "action": "_StoreAction", "choices": null, "const": null, "default": null, "group_key": null, "group_required": null, "hidden": false, "is_flag": false, "kind": "option", "nargs": null, "position": null, "required": false, "type": null},
Expand Down
Loading
Loading