Repository navigation
feat(novelty): judge panel (--judge repeatable, --judge-aggregate) - #489
Conversation
Re-grade 2,932 study answers with eight judges from seven families and report per-judge added/known/gated shares, noise floor, pairwise kappa, self-preference, six aggregation rules and panel size. Review how Dokimi configures judge panels today and propose --judges, --aggregate and a judge family policy for arms run. Register entries X030, F020, F021. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Petrov, Nikolay A <nikolay.a.petrov@intel.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Petrov, Nikolay A <nikolay.a.petrov@intel.com>
--judge PROVIDER/MODEL, repeatable, is the panel; --judge-aggregate
{median,majority,unanimous} turns the verdicts into one. Records carry a
judge_panel block (per-judge 2x2, added-share range, judge_sensitive) and
every run keeps each judge's verdict. Warns on same-vendor judges; the
estimate counts the extra judges.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Petrov, Nikolay A <nikolay.a.petrov@intel.com>
…ing covers the panel Verdicts are keyed by model name, so the same model on two providers collapsed into one vote. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Petrov, Nikolay A <nikolay.a.petrov@intel.com>
CLI interface change detectedThis PR changes dokimi's user-facing CLI surface (commands and/or options): ~ `dokimi skill novelty analyze` parameters changed:
+ added option `--judge`
+ added option `--judge-aggregate`
~ `dokimi skill novelty probe` parameters changed:
+ added option `--judge`
+ added option `--judge-aggregate`Updated on every push; the |
Seven more judges (kimi-k2.5, haiku-4.5, deepseek-r1, gpt-6-sol, opus-5.5, nemotron-super-3, minimax-m2.5) re-grade the same 2,932 rows. Within a family, generations agree at kappa 0.61-0.72, about like two vendors; one added share moves up to 0.15; a three-family panel keeps it within 0.07. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Petrov, Nikolay A <nikolay.a.petrov@intel.com>
…e-panel-cli Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> # Conflicts: # dokimi/docs/reference/cli-interface.json # dokimi/dokimi/commands/novelty.py # dokimi/dokimi/novelty/analyze.py # dokimi/research/records/experiment/X030.md # dokimi/research/records/finding/F020.md
Move the panel helpers the command used (panel clients, judge refs, role models, the judge2-style stamp of the panel block) into novelty_run.py, and move panel_record into measure.py, so panel.py no longer imports measure. measure keeps the one lazy edge it needs out of panel at module level. Lowers the novelty.py growth baseline to the actual 881 lines. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Make the three shebang scripts executable (EXE001), mark the one exec of this directory's own analyze.py for bandit (B102), shorten three comment blocks to the two-line limit, and allow the sibling llm and run_probes imports in deptry (DEP001), as the audit_mocks precedent does. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…m the index Indent the nested lists so mkdocs reads them the same as GitHub, and mark the proposed arms run CLI as proposed text, not a command the doc checker must resolve. Link judge-panel-study.md from docs/README.md. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Comment blocks of three lines or more fail the comment check (AGENTS.md). Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The list reflow put a paragraph break inside the sentence, so the site rendered two fragments. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… name The DEP001 ignore is by module name for the whole scan, so a bare llm would hide a real missing dependency elsewhere. The study scripts are not shipped, so the path exclude is the narrower fix. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… the license check The CSV and SVG files under docs/assets/judge-panel are written by research/judge-panel. A header would break the CSV readers, and regeneration would drop it, so they are ignored like the other data files. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #489 +/- ##
==========================================
+ Coverage 95.75% 95.76% +0.01%
==========================================
Files 424 428 +4
Lines 56428 56703 +275
Branches 9720 9766 +46
==========================================
+ Hits 54032 54303 +271
- Misses 1487 1491 +4
Partials 909 909
🚀 New features to boost your workflow:
|
|
OpenCodeReview: Commit 10a5fdd was not reviewed. This PR has reached its review limit (6 reviews). CI at `10a5fdd`: 7 still running, 7 passing, 2 skipped
To review this commit: re-request review from exai-bot in the Reviewers list. <details><summary>Last review (d1eabe2): 2 medium</summary> Open threads from the bot: 0 </details> |
| "lenient_rate_delta": round(out["per_judge"][new]["lenient_rate"] | ||
| - out["per_judge"][old]["lenient_rate"], 4), |
There was a problem hiding this comment.
Must-fix — Impact: the reported generation leniency delta mixes a judge change with a change in answer composition. Trigger: run gen.py on the documented study containing fresh no-skill repeats. Evidence: judge_leniency.csv reports 2,664 grades for gpt-5.6-luna versus 2,932 for gpt-6-luna/sol; generation_kappa.csv nevertheless labels their unmatched rate differences (-0.0422/-0.0428) with n=2,664. The extra 268 rows are exclusively no-skill repeats from three answer models, not a comparable sample of all six arms. Compute both rates on the same intersection of keys used for each pair's kappa. Workaround: exclude repeats from both judges before calculating generation rate deltas. Severity justification: medium because this produces a confounded research statistic for the actual supported dataset, but the defect is bounded to the generation comparison.
There was a problem hiding this comment.
Partly fixed in 91ad8bd: gen.py computes the lenient-rate delta of each pair on the answers both judges graded. Not changed: generation_kappa.csv and the section 6 numbers are not regenerated, since the study data is not in the repository.
There was a problem hiding this comment.
Same as the population thread: a is fixed in 9329068; b (regenerating the tables) waits for the author.
Waiting on a decision before I change this: regenerate generation_kappa.csv and section 6 (A), or drop the rate-delta column until a re-run (B)? My recommendation: B for this PR, A as a follow-up, since the study data is not in the repository. The thread stays open until then.
There was a problem hiding this comment.
Part (b) is not re-run in this PR: the regeneration needs the study data, which is not in the repo.
ef1710b drops lenient_rate_delta from generation_kappa.csv, and section 6 marks its rates and deltas as not re-run.
The matched-key regeneration (gen.py on the study data, then section 6) is tracked in #775.
Leaving this thread open until that re-run lands, or until you agree to the deferral.
There was a problem hiding this comment.
🟡 Changes recommended
Critical defects allow duplicate-model voting, stale-grade reuse, and mismatched analysis populations, making panel results unreliable and causing analysis crashes.
7 open issues
Detect duplicate judges by model identity, not raw spelling · New Keep analysis population pinned to the declared judge panel · New Compare judge rates on shared answer keys · New Prevent stale grades when regenerating repeat answers · New Include additional judges in estimated call counts · New Render panel members and aggregation rule in reports · New Handle undefined kappa for empty or unanimous grade sets · New
What changed in this PR
10 findings identified.
This PR adds multi-model judging to Dokimi’s skill-novelty analysis, building on the judge-panel study in #482 to expose how judge choice affects measured results.
Changes:
- Adds repeatable
--judgeoptions and median, majority, or unanimous aggregation. - Records individual verdicts, per-judge outcomes, sensitivity warnings, and additional judge costs.
- Includes the supporting study, research scripts, data tables, figures, documentation, and tests.
| File | Description |
|---|---|
dokimi/tests/test_novelty_tiers.py |
Updates role-model helper imports. |
dokimi/tests/test_novelty_panel.py |
Tests aggregation, recording, validation, and failures. |
dokimi/scripts/growth_baseline.json |
Lowers the novelty command’s size baseline. |
dokimi/research/records/finding/F023.md |
Records judge-dependent headline sensitivity. |
dokimi/research/records/finding/F022.md |
Records judge-generation effects. |
dokimi/research/records/finding/F021.md |
Records aggregation-rule effects. |
dokimi/research/records/experiment/X031.md |
Summarizes the judge-panel experiment. |
dokimi/research/judge-panel/judge_panel.py |
Samples answers, generates repeats, and grades them. |
dokimi/research/judge-panel/gen.py |
Compares judge generations. |
dokimi/research/judge-panel/fig.py |
Generates judge-comparison figures. |
dokimi/research/judge-panel/analyze.py |
Computes study metrics and tables. |
dokimi/pyproject.toml |
Excludes external-checkout study scripts from dependency checks. |
dokimi/mkdocs.yml |
Adds study navigation. |
dokimi/dokimi/novelty/panel.py |
Implements aggregation and panel warnings. |
dokimi/dokimi/novelty/measure.py |
Grades with multiple judges and summarizes outcomes. |
dokimi/dokimi/novelty/analyze.py |
Integrates panels into the analysis pipeline. |
dokimi/dokimi/commands/novelty.py |
Wires panel options into analyze and probe. |
dokimi/dokimi/commands/novelty_run.py |
Adds panel parsing, clients, pricing, and metadata. |
dokimi/dokimi/cli_flags.py |
Registers the aggregation flag. |
dokimi/docs/skill-novelty.md |
Documents panel usage and output. |
dokimi/docs/reference/cli-interface.json |
Updates the CLI interface snapshot. |
dokimi/docs/README.md |
Links the study. |
dokimi/docs/judge-panel-study.md |
Presents study methods and conclusions. |
dokimi/docs/assets/judge-panel/self_preference.csv |
Stores judge self-preference measurements. |
dokimi/docs/assets/judge-panel/per_judge_headline.csv |
Stores per-judge headline outcomes. |
dokimi/docs/assets/judge-panel/panel_size.csv |
Compares panel sizes. |
dokimi/docs/assets/judge-panel/pairwise_kappa.csv |
Stores pairwise judge agreement. |
dokimi/docs/assets/judge-panel/headline_range.csv |
Summarizes headline ranges across judges. |
dokimi/docs/assets/judge-panel/generations/judge_mean_kappa.csv |
Stores mean agreement by judge. |
dokimi/docs/assets/judge-panel/generations/judge_leniency.csv |
Stores judge acceptance rates. |
dokimi/docs/assets/judge-panel/generations/generation_panels.csv |
Compares old and new panels. |
dokimi/docs/assets/judge-panel/generations/generation_kappa.csv |
Stores generation-pair agreement and rate shifts. |
dokimi/docs/assets/judge-panel/generations/generation_added_share.csv |
Stores generation-dependent added-share changes. |
dokimi/docs/assets/judge-panel/gated_added_by_judge.svg |
Visualizes lenient gated-added shares. |
dokimi/docs/assets/judge-panel/gated_added_by_judge_strict.svg |
Visualizes strict gated-added shares. |
dokimi/docs/assets/judge-panel/disagreement_by_group.csv |
Groups disagreement by arm and answer characteristics. |
dokimi/docs/assets/judge-panel/agreement_with_ds.csv |
Compares judges and rules with inferred consensus. |
dokimi/docs/assets/judge-panel/aggregation_rules.csv |
Stores outcomes for each aggregation rule. |
dokimi/changelog.d/20261108_120000_napetrov_judge-panel-cli.md |
Announces novelty judge-panel support. |
.github/.licenserc.yaml |
Exempts generated study assets from headers. |
Review effort: Balanced
| "lenient_rate_delta": round(out["per_judge"][new]["lenient_rate"] | ||
| - out["per_judge"][old]["lenient_rate"], 4), |
There was a problem hiding this comment.
Partly fixed in 91ad8bd: gen.py computes the rate delta on the answers both judges graded. Not changed: generation_kappa.csv and the section 6 numbers are not regenerated, since the study data is not in the repository.
There was a problem hiding this comment.
a: gen.py computes the rate delta on the answers both judges graded. b: generation_kappa.csv and section 6 are not regenerated here; the study data is not in the repository.
Waiting on a decision before I change this: regenerate generation_kappa.csv and section 6 (A), or drop the rate-delta column until a re-run (B)? My recommendation: B for this PR, A as a follow-up, since the study data is not in the repository. The thread stays open until then.
There was a problem hiding this comment.
Part (b) is not re-run in this PR: the regeneration needs the study data, which is not in the repo.
ef1710b drops lenient_rate_delta from generation_kappa.csv, and section 6 marks its rates and deltas as not re-run.
The matched-key regeneration (gen.py on the study data, then section 6) is tracked in #775.
Leaving this thread open until that re-run lands, or until you agree to the deferral.
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g863iax3: already fixed in 021762f Thread: PRRT_kwDOVBg7g863iax4: already fixed in d1eabe2 Thread: PRRT_kwDOVBg7g863iax7: already fixed in 937c77f Thread: PRRT_kwDOVBg7g863iayE: already fixed in 021762f Thread: PRRT_kwDOVBg7g863jNyw: already fixed in d1eabe2 Thread: PRRT_kwDOVBg7g863jNyy: already fixed in 021762f Thread: PRRT_kwDOVBg7g863jNy1: already fixed in 021762f Thread: PRRT_kwDOVBg7g863mqmA: already fixed in d1eabe2 Thread: PRRT_kwDOVBg7g863mqmC: already fixed in d1eabe2 Thread: PRR_kwDOVBg7g88AAAABYBomrA: already fixed in d1eabe2 Thread: PRR_kwDOVBg7g88AAAABYB2f5A: already fixed in c6b0fb9 Thread: IC_kwDOVBg7g88AAAABmcxO4A: already fixed in 9dabb8b Thread: IC_kwDOVBg7g88AAAABm3nDIA: already fixed in d91b591 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
|
|
Round at 3ff38d0: merged intel/main (47f89ed). The merge was clean, with no conflicts, and the judge-panel study files are identical on main and on this branch.
|
vchamarthi
left a comment
There was a problem hiding this comment.
🔴 Changes needed: 1 must-fix, 2 nits · reviewed 3ff38d0
Next: @napetrov: fix the 1 must-fix finding; then push for a re-review. 2 nits optional.
Nits
-
dokimi/dokimi/schemas/novelty.v1.json:518– Thejudge_panelblock written to anovelty.v1record is not declared innovelty.v1.json, whilejudge2is.Input: a record from
analyze --judge a --judge b. It carriesjudge_panel, andpropertiesinnovelty.v1.jsonhas no such key. Expected: the key declared with a description, asjudge2is.Also at:
dokimi/dokimi/novelty/analyze.py:501.Suggestion: Add a
judge_panelproperty to the schema describing judges, rule, per_judge_paired, added_share_range and judge_sensitive.Patch, run on
3ff38d0: the failing case passes; tests are no worse.Patch
--- a/dokimi/dokimi/schemas/novelty.v1.json +++ b/dokimi/dokimi/schemas/novelty.v1.json @@ -519,6 +519,10 @@ "type": "object", "description": "With --judge2: a seeded --judge2-frac subset of the graded runs regraded by a second judge; lenient_agreement, kappa_lenient, exact_agreement over n_compared, and that judge's cost." }, + "judge_panel": { + "type": "object", + "description": "With two or more --judge: judges, rule, shared_vendors, same_vendor_as_answerer, failed_grades, each judge's own paired 2x2 (per_judge_paired), the spread of the added share across judges (added_share_range) and judge_sensitive." + }, "status": { "const": "partial", "description": "Present only on a partial record: `skill novelty analyze` stopped by --max-usd (or a call with no known price) keeps what it paid for in <out stem>.partial.json. Every reader of a finished novelty.json refuses it."
Verified on 3ff38d0: ✅ changed tests: 84 passed, fail on main · ❌ full suite with main merged in fails at the whole run (exit 130, no failing test named) · ✅ merges onto main · 1 mutation tried · 5 edge inputs tried
6 checks behind it, each with its evidence
| Area | What holds | How |
|---|---|---|
| Panel flags | judge_refs refuses one model given twice under another provider or spelling (openai/gpt-5 and openrouter/gpt-5, GPT-5), a bad ref, an empty ref and --judge with --judge-model, and returns [] without the flag |
ran it, edge input (strings) |
| Aggregation rules | aggregate([], 'median') is None, one wrong_reversed under unanimous stays wrong_reversed, and a two-judge split rounds toward wrong |
ran it, edge input (collections) |
| Aggregation rules | Changing the median index to the upper side fails test_aggregate_rules |
broke the code, tests caught it |
| Judge2 regrade | first_verdict returns the first judge's own vote when it is correct_complete and the aggregate is wrong_other, so --judge2 is compared with the first judge |
ran it |
| Changed tests | test_novelty_panel.py and test_novelty_provenance.py pass (84 tests) and do not import on main |
tests on the head |
| Merge | The head merges onto main with no conflict | merge onto main |
| stops.append(exc) | ||
| named = _reply_verdict(exc.reply) | ||
| panel_votes = getattr(exc, "panel_votes", None) | ||
| named = None if exc.truncated or panel_votes is not None else _reply_verdict(exc.reply) |
There was a problem hiding this comment.
🟡 Nit: A single-judge run stopped by the cap on a cut judge reply naming correct_complete now keeps error; main keeps the named verdict.
Input: a SpendCapReached with reply='{"verdict": "correct_complete"}' and truncated=True, no panel. On the head _capped_grade_run gives verdict error. Main gives correct_complete. Neither the description nor the changelog asks for a change to the one-judge path.
Suggestion: Apply only the panel_votes is not None condition, or say in the description why a cut reply no longer counts.
There was a problem hiding this comment.
Not changed. Main's unpriced path already stores a reply cut at the ceiling as an error (measure.py:315), and test_a_truncated_judge_reply_under_a_cap_is_a_failed_grade pins the same rule for the priced path. Say if you want the named verdict kept there.
|
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com> # Conflicts: # dokimi/docs/README.md
The run's panel is read from the votes it recorded, so a cap that stops the panel after the first judge voted saved no judge_panel block. The measurement now carries the configured judges, and panel_record reads them first. Thread: PRRT_kwDOVBg7g8630QBs Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…udge-panel study Main restructured the README (62 lines) and moved its guide list to documentation-index.md, which links judge-panel-study.md. The merge had kept the old README body and the PR's three-line link, so the merged file repeated main's content. This takes main's README as it is. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The record writes a judge_panel block with two or more judges, but the schema declared only judge2. The block is added with its fields named. Thread: PRR_kwDOVBg7g88AAAABYKKyew#1 Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The judges field of MeasureResult added a line; the field carries its comment on the same line, and one two-line comment is one line. check_growth passes at 800 lines. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
save_panel_verdicts annotates with Path; main dropped the import and the clean merge took that deletion, leaving F821 on head. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
antonwolfy
left a comment
There was a problem hiding this comment.
LGTM. The delta since c6b0fb9 holds: a stopped panel keeps its judges, an errored vote is no datum, --judge2 is compared with the first judge's own vote, judge_panel is declared in novelty.v1, and the run-wide stop (#708) now covers the panel grading. One nit on the docs.
What I ran, on a Linux review host at 9d0a302:
make ci-local: 34 passed, 0 failed, 14 skipped; 22,369 tests; changed-line coverage 96%.- Reproduction for the nit, with the LLM transport mocked.
- A mutation run on
test_novelty_panel.pyandtest_novelty_provenance.py: 9 mutants of the new code (first_verdict, the errored-vote and quorum guards inpanel_record,MeasureResult.judges, thejudge_panelof a stoppedanalyze, the keyword-onlyjudge_verdicts, the run-wide stop in the answer handler, the errored-vote filter ingrade_panel), all 9 killed.
I agree with @vchamarthi that a reply cut at the output ceiling under a spend cap now keeps error on the one-judge path too, where the base keeps the named verdict. The author's reason (the unpriced path already stores a cut reply as an error) holds, but the changelog fragment does not mention the change.
Checked, no issues
- Repo rules: every non-merge commit has a
Signed-off-by,cli-interface.jsonandcli_flags.pymatch the flags,judge_panelis an additive key declared innovelty.v1.json, the changelog fragment is present, and the new tests write nothing outsidetmp_path. - Judge2:
first_verdicttakes the first entry ofjudge_verdicts, and removing it fails a test.
Bot points I checked that no longer apply at this head (2)
- exai-bot's point on the generation rate delta does not apply: the study scripts and tables are not in this PR's diff, they are on its base, and the open follow-up is tracked in #775.
- Copilot's point on the generation rate delta does not apply: the study scripts and tables are not in this PR's diff, they are on its base, and the open follow-up is tracked in #775.
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
antonwolfy
left a comment
There was a problem hiding this comment.
LGTM. I pushed a commit per nit, up to 10a5fdd, which fix the nit from my review:
10a5fdd:dokimi/docs/skill-novelty.md:333: the sentence now says a failed judge is left out of the vote, except for a permanent provider error, which stops the run and keeps the partial record.
make ci-local at 10a5fdd: 34 passed, 0 failed, 14 skipped.


Implements the judge panel the study in #482 recommends, for
skill novelty analyzeandskill novelty probe. Stacked on #482: this branch mergesresearch/judge-panel, so the diff shows the study too until #482 lands.--judge PROVIDER/MODELis repeatable, and two or more judges form a panel. It is the canonical spelling from CLI API: the evaluation loop at the top level, subject and eval groups below, one enforced flag vocabulary (proposal v2.1) #94. Mixing it with--judge-model/--judge-provideris refused, as is a model given twice. The first judge is the primary judge and the default extractor.--judge-aggregate {median,majority,unanimous}, defaultmedian, registered incli_flags.pyand thecli-interface.jsonsnapshot.dokimi/novelty/panel.py:judge_panelrecord block, which has each judge's own paired 2×2,added_share_rangeandjudge_sensitive(lenient range > 0.10).run_measurementgrades each answer with every judge and storesjudge_verdicts. A failed judge is left out of the vote. Every panel judge gets the self-judge check, and cost and estimate include every judge.Tests:
tests/test_novelty_panel.py(21). Ruff is clean. The parity, cli-surface and symlink failures on Windows also fail on origin/main.Follow-ups:
eval score --judgepanels, a Dawid-Skene rule, and a wider vendor map.🤖 Generated with Claude Code