Skip to content

feat(novelty): judge panel (--judge repeatable, --judge-aggregate) - #489

Merged
antonwolfy merged 45 commits into
intel:mainfrom
napetrov:feat/judge-panel-cli
Nov 17, 2026
Merged

antonwolfy merged 45 commits into
intel:mainfrom
napetrov:feat/judge-panel-cli

Conversation

@napetrov

@napetrov napetrov commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Implements the judge panel the study in #482 recommends, for skill novelty analyze and skill novelty probe. Stacked on #482: this branch merges research/judge-panel, so the diff shows the study too until #482 lands.

  • --judge PROVIDER/MODEL is repeatable, and two or more judges form a panel. It is the canonical spelling from CLI API: the evaluation loop at the top level, subject and eval groups below, one enforced flag vocabulary (proposal v2.1) #94. Mixing it with --judge-model/--judge-provider is refused, as is a model given twice. The first judge is the primary judge and the default extractor.
  • New --judge-aggregate {median,majority,unanimous}, default median, registered in cli_flags.py and the cli-interface.json snapshot.
  • New dokimi/novelty/panel.py:
    • The aggregation rules.
    • The judge_panel record block, which has each judge's own paired 2×2, added_share_range and judge_sensitive (lenient range > 0.10).
    • Warnings for same-vendor judges.
  • run_measurement grades each answer with every judge and stores judge_verdicts. A failed judge is left out of the vote. Every panel judge gets the self-judge check, and cost and estimate include every judge.

Tests: tests/test_novelty_panel.py (21). Ruff is clean. The parity, cli-surface and symlink failures on Windows also fail on origin/main.

Follow-ups: eval score --judge panels, a Dawid-Skene rule, and a wider vendor map.

🤖 Generated with Claude Code

napetrov and others added 5 commits November 8, 2026 13:16
Re-grade 2,932 study answers with eight judges from seven families and
report per-judge added/known/gated shares, noise floor, pairwise kappa,
self-preference, six aggregation rules and panel size. Review how Dokimi
configures judge panels today and propose --judges, --aggregate and a
judge family policy for arms run. Register entries X030, F020, F021.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Petrov, Nikolay A <nikolay.a.petrov@intel.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Petrov, Nikolay A <nikolay.a.petrov@intel.com>
--judge PROVIDER/MODEL, repeatable, is the panel; --judge-aggregate
{median,majority,unanimous} turns the verdicts into one. Records carry a
judge_panel block (per-judge 2x2, added-share range, judge_sensitive) and
every run keeps each judge's verdict. Warns on same-vendor judges; the
estimate counts the extra judges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Petrov, Nikolay A <nikolay.a.petrov@intel.com>
…ing covers the panel

Verdicts are keyed by model name, so the same model on two providers
collapsed into one vote.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Petrov, Nikolay A <nikolay.a.petrov@intel.com>
@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown

CLI interface change detected

This PR changes dokimi's user-facing CLI surface (commands and/or options):

~ `dokimi skill novelty analyze` parameters changed:
    + added option `--judge`
    + added option `--judge-aggregate`
~ `dokimi skill novelty probe` parameters changed:
    + added option `--judge`
    + added option `--judge-aggregate`

Updated on every push; the cli-interface-change label is on while the diff is non-empty.

napetrov and others added 12 commits November 8, 2026 15:20
Seven more judges (kimi-k2.5, haiku-4.5, deepseek-r1, gpt-6-sol, opus-5.5,
nemotron-super-3, minimax-m2.5) re-grade the same 2,932 rows. Within a
family, generations agree at kappa 0.61-0.72, about like two vendors; one
added share moves up to 0.15; a three-family panel keeps it within 0.07.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Petrov, Nikolay A <nikolay.a.petrov@intel.com>
…e-panel-cli

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

# Conflicts:
#	dokimi/docs/reference/cli-interface.json
#	dokimi/dokimi/commands/novelty.py
#	dokimi/dokimi/novelty/analyze.py
#	dokimi/research/records/experiment/X030.md
#	dokimi/research/records/finding/F020.md
Move the panel helpers the command used (panel clients, judge refs, role models,
the judge2-style stamp of the panel block) into novelty_run.py, and move
panel_record into measure.py, so panel.py no longer imports measure. measure
keeps the one lazy edge it needs out of panel at module level. Lowers the
novelty.py growth baseline to the actual 881 lines.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Make the three shebang scripts executable (EXE001), mark the one exec of this
directory's own analyze.py for bandit (B102), shorten three comment blocks to
the two-line limit, and allow the sibling llm and run_probes imports in deptry
(DEP001), as the audit_mocks precedent does.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…m the index

Indent the nested lists so mkdocs reads them the same as GitHub, and mark the
proposed arms run CLI as proposed text, not a command the doc checker must
resolve. Link judge-panel-study.md from docs/README.md.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Comment blocks of three lines or more fail the comment check (AGENTS.md).

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The list reflow put a paragraph break inside the sentence, so the site rendered
two fragments.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… name

The DEP001 ignore is by module name for the whole scan, so a bare llm would
hide a real missing dependency elsewhere. The study scripts are not shipped,
so the path exclude is the narrower fix.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
… the license check

The CSV and SVG files under docs/assets/judge-panel are written by
research/judge-panel. A header would break the CSV readers, and regeneration
would drop it, so they are ignored like the other data files.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@codecov-commenter

codecov-commenter commented Nov 10, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 94.33198% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 95.76%. Comparing base (01c83db) to head (10a5fdd).
⚠️ Report is 4 commits behind head on main.

Files with missing lines Patch % Lines
dokimi/dokimi/novelty/cost.py 91.11% 3 Missing and 1 partial ⚠️
dokimi/dokimi/novelty/panel.py 94.64% 2 Missing and 1 partial ⚠️
dokimi/dokimi/commands/novelty_run.py 93.54% 2 Missing ⚠️
dokimi/dokimi/novelty/panel_grade.py 91.66% 1 Missing and 1 partial ⚠️
dokimi/dokimi/novelty/panel_record.py 95.34% 1 Missing and 1 partial ⚠️
dokimi/dokimi/novelty/analyze.py 88.88% 0 Missing and 1 partial ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #489      +/-   ##
==========================================
+ Coverage   95.75%   95.76%   +0.01%     
==========================================
  Files         424      428       +4     
  Lines       56428    56703     +275     
  Branches     9720     9766      +46     
==========================================
+ Hits        54032    54303     +271     
- Misses       1487     1491       +4     
  Partials      909      909              
Files with missing lines Coverage Δ
dokimi/dokimi/cli_flags.py 100.00% <ø> (ø)
dokimi/dokimi/commands/novelty.py 99.74% <100.00%> (+<0.01%) ⬆️
dokimi/dokimi/commands/novelty_budget.py 96.42% <100.00%> (+0.16%) ⬆️
dokimi/dokimi/novelty/analyze_partial.py 100.00% <ø> (ø)
dokimi/dokimi/novelty/client.py 98.57% <100.00%> (+0.01%) ⬆️
dokimi/dokimi/novelty/judge2.py 100.00% <100.00%> (ø)
dokimi/dokimi/novelty/measure.py 98.80% <100.00%> (+0.02%) ⬆️
dokimi/dokimi/novelty/report.py 98.80% <100.00%> (+0.02%) ⬆️
dokimi/dokimi/subjects/novelty_track.py 97.50% <100.00%> (+0.13%) ⬆️
dokimi/dokimi/novelty/analyze.py 97.79% <88.88%> (+0.96%) ⬆️
... and 5 more

... and 17 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@exai-bot

exai-bot commented Nov 13, 2026 •

Copy link
Copy Markdown

OpenCodeReview: Commit 10a5fdd was not reviewed. This PR has reached its review limit (6 reviews).

CI at `10a5fdd`: 7 still running, 7 passing, 2 skipped
Check Status
CodeQL (python) still running
Platforms and Python versions still running
Static checks (ruff, schema, zizmor, licenses, pre-commit, bandit, secrets, changelog, schema com... still running
Windows smoke test (py3.13) still running
atheris (parsers) still running
benchmark still running
mypy still running

To review this commit: re-request review from exai-bot in the Reviewers list.

<details><summary>Last review (d1eabe2): 2 medium</summary>

Open threads from the bot: 0

</details>

Comment thread dokimi/docs/judge-panel-study.md Resolved
Comment thread dokimi/docs/judge-panel-study.md Resolved
Comment thread dokimi/dokimi/commands/novelty_run.py Resolved
Comment thread dokimi/dokimi/commands/novelty.py Resolved
Comment thread dokimi/dokimi/commands/novelty.py Resolved
Comment thread dokimi/dokimi/novelty/measure.py Resolved
Comment thread dokimi/research/judge-panel/analyze.py Resolved
Comment thread dokimi/research/judge-panel/gen.py Outdated
Comment on lines +77 to +78
"lenient_rate_delta": round(out["per_judge"][new]["lenient_rate"]
- out["per_judge"][old]["lenient_rate"], 4),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix — Impact: the reported generation leniency delta mixes a judge change with a change in answer composition. Trigger: run gen.py on the documented study containing fresh no-skill repeats. Evidence: judge_leniency.csv reports 2,664 grades for gpt-5.6-luna versus 2,932 for gpt-6-luna/sol; generation_kappa.csv nevertheless labels their unmatched rate differences (-0.0422/-0.0428) with n=2,664. The extra 268 rows are exclusively no-skill repeats from three answer models, not a comparable sample of all six arms. Compute both rates on the same intersection of keys used for each pair's kappa. Workaround: exclude repeats from both judges before calculating generation rate deltas. Severity justification: medium because this produces a confounded research statistic for the actual supported dataset, but the defect is bounded to the generation comparison.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partly fixed in 91ad8bd: gen.py computes the lenient-rate delta of each pair on the answers both judges graded. Not changed: generation_kappa.csv and the section 6 numbers are not regenerated, since the study data is not in the repository.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as the population thread: a is fixed in 9329068; b (regenerating the tables) waits for the author.
Waiting on a decision before I change this: regenerate generation_kappa.csv and section 6 (A), or drop the rate-delta column until a re-run (B)? My recommendation: B for this PR, A as a follow-up, since the study data is not in the repository. The thread stays open until then.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Part (b) is not re-run in this PR: the regeneration needs the study data, which is not in the repo.
ef1710b drops lenient_rate_delta from generation_kappa.csv, and section 6 marks its rates and deltas as not re-run.
The matched-key regeneration (gen.py on the study data, then section 6) is tracked in #775.
Leaving this thread open until that re-run lands, or until you agree to the deferral.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical defects allow duplicate-model voting, stale-grade reuse, and mismatched analysis populations, making panel results unreliable and causing analysis crashes.

7 open issues
What changed in this PR

10 findings identified.

This PR adds multi-model judging to Dokimi’s skill-novelty analysis, building on the judge-panel study in #482 to expose how judge choice affects measured results.

Changes:

  • Adds repeatable --judge options and median, majority, or unanimous aggregation.
  • Records individual verdicts, per-judge outcomes, sensitivity warnings, and additional judge costs.
  • Includes the supporting study, research scripts, data tables, figures, documentation, and tests.
File Description
dokimi/​tests/​test_novelty_tiers.py Updates role-model helper imports.
dokimi/​tests/​test_novelty_panel.py Tests aggregation, recording, validation, and failures.
dokimi/​scripts/​growth_baseline.json Lowers the novelty command’s size baseline.
dokimi/​research/​records/​finding/​F023.md Records judge-dependent headline sensitivity.
dokimi/​research/​records/​finding/​F022.md Records judge-generation effects.
dokimi/​research/​records/​finding/​F021.md Records aggregation-rule effects.
dokimi/​research/​records/​experiment/​X031.md Summarizes the judge-panel experiment.
dokimi/​research/​judge-panel/​judge_panel.py Samples answers, generates repeats, and grades them.
dokimi/​research/​judge-panel/​gen.py Compares judge generations.
dokimi/​research/​judge-panel/​fig.py Generates judge-comparison figures.
dokimi/​research/​judge-panel/​analyze.py Computes study metrics and tables.
dokimi/​pyproject.toml Excludes external-checkout study scripts from dependency checks.
dokimi/​mkdocs.yml Adds study navigation.
dokimi/​dokimi/​novelty/​panel.py Implements aggregation and panel warnings.
dokimi/​dokimi/​novelty/​measure.py Grades with multiple judges and summarizes outcomes.
dokimi/​dokimi/​novelty/​analyze.py Integrates panels into the analysis pipeline.
dokimi/​dokimi/​commands/​novelty.py Wires panel options into analyze and probe.
dokimi/​dokimi/​commands/​novelty_run.py Adds panel parsing, clients, pricing, and metadata.
dokimi/​dokimi/​cli_flags.py Registers the aggregation flag.
dokimi/​docs/​skill-novelty.md Documents panel usage and output.
dokimi/​docs/​reference/​cli-interface.json Updates the CLI interface snapshot.
dokimi/​docs/​README.md Links the study.
dokimi/​docs/​judge-panel-study.md Presents study methods and conclusions.
dokimi/​docs/​assets/​judge-panel/​self_preference.csv Stores judge self-preference measurements.
dokimi/​docs/​assets/​judge-panel/​per_judge_headline.csv Stores per-judge headline outcomes.
dokimi/​docs/​assets/​judge-panel/​panel_size.csv Compares panel sizes.
dokimi/​docs/​assets/​judge-panel/​pairwise_kappa.csv Stores pairwise judge agreement.
dokimi/​docs/​assets/​judge-panel/​headline_range.csv Summarizes headline ranges across judges.
dokimi/​docs/​assets/​judge-panel/​generations/​judge_mean_kappa.csv Stores mean agreement by judge.
dokimi/​docs/​assets/​judge-panel/​generations/​judge_leniency.csv Stores judge acceptance rates.
dokimi/​docs/​assets/​judge-panel/​generations/​generation_panels.csv Compares old and new panels.
dokimi/​docs/​assets/​judge-panel/​generations/​generation_kappa.csv Stores generation-pair agreement and rate shifts.
dokimi/​docs/​assets/​judge-panel/​generations/​generation_added_share.csv Stores generation-dependent added-share changes.
dokimi/​docs/​assets/​judge-panel/​gated_added_by_judge.svg Visualizes lenient gated-added shares.
dokimi/​docs/​assets/​judge-panel/​gated_added_by_judge_strict.svg Visualizes strict gated-added shares.
dokimi/​docs/​assets/​judge-panel/​disagreement_by_group.csv Groups disagreement by arm and answer characteristics.
dokimi/​docs/​assets/​judge-panel/​agreement_with_ds.csv Compares judges and rules with inferred consensus.
dokimi/​docs/​assets/​judge-panel/​aggregation_rules.csv Stores outcomes for each aggregation rule.
dokimi/​changelog.d/​20261108_120000_napetrov_judge-panel-cli.md Announces novelty judge-panel support.
.github/​.licenserc.yaml Exempts generated study assets from headers.

Review effort: Balanced

Comment thread dokimi/dokimi/commands/novelty_run.py Resolved
Comment thread dokimi/research/judge-panel/analyze.py Resolved
Comment thread dokimi/research/judge-panel/gen.py Outdated
Comment on lines +77 to +78
"lenient_rate_delta": round(out["per_judge"][new]["lenient_rate"]
- out["per_judge"][old]["lenient_rate"], 4),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partly fixed in 91ad8bd: gen.py computes the rate delta on the answers both judges graded. Not changed: generation_kappa.csv and the section 6 numbers are not regenerated, since the study data is not in the repository.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a: gen.py computes the rate delta on the answers both judges graded. b: generation_kappa.csv and section 6 are not regenerated here; the study data is not in the repository.
Waiting on a decision before I change this: regenerate generation_kappa.csv and section 6 (A), or drop the rate-delta column until a re-run (B)? My recommendation: B for this PR, A as a follow-up, since the study data is not in the repository. The thread stays open until then.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Part (b) is not re-run in this PR: the regeneration needs the study data, which is not in the repo.
ef1710b drops lenient_rate_delta from generation_kappa.csv, and section 6 marks its rates and deltas as not re-run.
The matched-key regeneration (gen.py on the study data, then section 6) is tracked in #775.
Leaving this thread open until that re-run lands, or until you agree to the deferral.

Comment thread dokimi/dokimi/commands/novelty.py Resolved
Comment thread dokimi/dokimi/novelty/analyze.py Resolved
Comment thread dokimi/research/judge-panel/gen.py Resolved
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g863iax3: already fixed in 021762f
Thread: PRRT_kwDOVBg7g863iax4: already fixed in d1eabe2
Thread: PRRT_kwDOVBg7g863iax7: already fixed in 937c77f
Thread: PRRT_kwDOVBg7g863iayE: already fixed in 021762f
Thread: PRRT_kwDOVBg7g863jNyw: already fixed in d1eabe2
Thread: PRRT_kwDOVBg7g863jNyy: already fixed in 021762f
Thread: PRRT_kwDOVBg7g863jNy1: already fixed in 021762f
Thread: PRRT_kwDOVBg7g863mqmA: already fixed in d1eabe2
Thread: PRRT_kwDOVBg7g863mqmC: already fixed in d1eabe2
Thread: PRR_kwDOVBg7g88AAAABYBomrA: already fixed in d1eabe2
Thread: PRR_kwDOVBg7g88AAAABYB2f5A: already fixed in c6b0fb9
Thread: IC_kwDOVBg7g88AAAABmcxO4A: already fixed in 9dabb8b
Thread: IC_kwDOVBg7g88AAAABm3nDIA: already fixed in d91b591
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov

Copy link
Copy Markdown
Contributor Author

Re: review by antonwolfy

the three risks you list (unanimous wording, stale verdicts file, unpinned estimate)
Fixed in d1eabe2, and the other two were resolved threads eJk3 and eJk4. See 3ff38d0.

@napetrov

Copy link
Copy Markdown
Contributor Author

Re: review by antonwolfy

the six nit commits fbacfe7..c6b0fb9
They are on the branch. See 3ff38d0.

@napetrov

Copy link
Copy Markdown
Contributor Author

Re: comment by xaleryb

xaleryb's points 1-5 (veto, same-vendor, failing provider, measure.py cap, command-level tests)
Waiting on a decision before I change this: points 1-4 look addressed in d1eabe2 and c6b0fb9, not re-run probe by probe. Point 5 is in #775. My recommendation: #775 for the study points. The thread stays open until then.

@napetrov

Copy link
Copy Markdown
Contributor Author

Re: comment by vchamarthi

the status of ccdc44e..9dabb8b
Those commits are on the branch. See 3ff38d0.

@napetrov

Copy link
Copy Markdown
Contributor Author

Re: comment by vchamarthi

the question: --judge2 with a panel, the aggregate or the primary judge's raw verdict?
Waiting on a decision before I change this: the same as the mql4 thread. Head does B; my recommendation is B. Open until you confirm.

@napetrov

Copy link
Copy Markdown
Contributor Author

Re: comment by vchamarthi

d1eabe2..d91b591 is ready for re-review
It is on the branch. The --judge2 question is open in the mql4 thread. See 3ff38d0.

@napetrov

Copy link
Copy Markdown
Contributor Author

Round at 3ff38d0: merged intel/main (47f89ed). The merge was clean, with no conflicts, and the judge-panel study files are identical on main and on this branch.

@vchamarthi vchamarthi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Changes needed: 1 must-fix, 2 nits · reviewed 3ff38d0

Next: @napetrov: fix the 1 must-fix finding; then push for a re-review. 2 nits optional.

Nits

  • dokimi/dokimi/schemas/novelty.v1.json:518 – The judge_panel block written to a novelty.v1 record is not declared in novelty.v1.json, while judge2 is.

    Input: a record from analyze --judge a --judge b. It carries judge_panel, and properties in novelty.v1.json has no such key. Expected: the key declared with a description, as judge2 is.

    Also at: dokimi/dokimi/novelty/analyze.py:501.

    Suggestion: Add a judge_panel property to the schema describing judges, rule, per_judge_paired, added_share_range and judge_sensitive.

    Patch, run on 3ff38d0: the failing case passes; tests are no worse.

    Patch
    --- a/dokimi/dokimi/schemas/novelty.v1.json
    +++ b/dokimi/dokimi/schemas/novelty.v1.json
    @@ -519,6 +519,10 @@
           "type": "object",
           "description": "With --judge2: a seeded --judge2-frac subset of the graded runs regraded by a second judge; lenient_agreement, kappa_lenient, exact_agreement over n_compared, and that judge's cost."
         },
    +    "judge_panel": {
    +      "type": "object",
    +      "description": "With two or more --judge: judges, rule, shared_vendors, same_vendor_as_answerer, failed_grades, each judge's own paired 2x2 (per_judge_paired), the spread of the added share across judges (added_share_range) and judge_sensitive."
    +    },
         "status": {
           "const": "partial",
           "description": "Present only on a partial record: `skill novelty analyze` stopped by --max-usd (or a call with no known price) keeps what it paid for in <out stem>.partial.json. Every reader of a finished novelty.json refuses it."

Verified on 3ff38d0: ✅ changed tests: 84 passed, fail on main · ❌ full suite with main merged in fails at the whole run (exit 130, no failing test named) · ✅ merges onto main · 1 mutation tried · 5 edge inputs tried

6 checks behind it, each with its evidence
Area What holds How
Panel flags judge_refs refuses one model given twice under another provider or spelling (openai/gpt-5 and openrouter/gpt-5, GPT-5), a bad ref, an empty ref and --judge with --judge-model, and returns [] without the flag ran it, edge input (strings)
Aggregation rules aggregate([], 'median') is None, one wrong_reversed under unanimous stays wrong_reversed, and a two-judge split rounds toward wrong ran it, edge input (collections)
Aggregation rules Changing the median index to the upper side fails test_aggregate_rules broke the code, tests caught it
Judge2 regrade first_verdict returns the first judge's own vote when it is correct_complete and the aggregate is wrong_other, so --judge2 is compared with the first judge ran it
Changed tests test_novelty_panel.py and test_novelty_provenance.py pass (84 tests) and do not import on main tests on the head
Merge The head merges onto main with no conflict merge onto main

Comment thread dokimi/dokimi/novelty/panel_record.py Resolved
stops.append(exc)
named = _reply_verdict(exc.reply)
panel_votes = getattr(exc, "panel_votes", None)
named = None if exc.truncated or panel_votes is not None else _reply_verdict(exc.reply)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Nit: A single-judge run stopped by the cap on a cut judge reply naming correct_complete now keeps error; main keeps the named verdict.

Input: a SpendCapReached with reply='{"verdict": "correct_complete"}' and truncated=True, no panel. On the head _capped_grade_run gives verdict error. Main gives correct_complete. Neither the description nor the changelog asks for a change to the one-judge path.

Suggestion: Apply only the panel_votes is not None condition, or say in the description why a cut reply no longer counts.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changed. Main's unpriced path already stores a reply cut at the ceiling as an error (measure.py:315), and test_a_truncated_judge_reply_under_a_cap_is_a_failed_grade pins the same rule for the priced path. Say if you want the named verdict kept there.

@napetrov

Copy link
Copy Markdown
Contributor Author

Re: comment by vchamarthi

the question: --judge2 with a panel, the aggregate or the primary judge's raw verdict?
Option B: the primary judge's raw verdict from judge_verdicts, as the regrade() docstring says. It is on the branch (3ff38d0), so this question is closed.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

# Conflicts:
#	dokimi/docs/README.md
The run's panel is read from the votes it recorded, so a cap that stops
the panel after the first judge voted saved no judge_panel block. The
measurement now carries the configured judges, and panel_record reads
them first.

Thread: PRRT_kwDOVBg7g8630QBs
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…udge-panel study

Main restructured the README (62 lines) and moved its guide list to
documentation-index.md, which links judge-panel-study.md. The merge had
kept the old README body and the PR's three-line link, so the merged file
repeated main's content. This takes main's README as it is.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The record writes a judge_panel block with two or more judges, but the
schema declared only judge2. The block is added with its fields named.

Thread: PRR_kwDOVBg7g88AAAABYKKyew#1
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The judges field of MeasureResult added a line; the field carries its
comment on the same line, and one two-line comment is one line.
check_growth passes at 800 lines.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov

Copy link
Copy Markdown
Contributor Author

Re: review by vchamarthi

Fixed in 026dbde: novelty.v1 declares judge_panel.

Changes needed: 1 must-fix, 2 nits
Answered item by item. The first nit is fixed in 026dbde, the must-fix in 030b840. The other nit is pushed back in its thread, with the reason.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
save_panel_verdicts annotates with Path; main dropped the import and the
clean merge took that deletion, leaving F821 on head.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov

Copy link
Copy Markdown
Contributor Author

Re: comment by xaleryb

xaleryb's points 1-5
Points 1-4 are on the branch with tests: the veto (test_a_judge_sees_the_veto_its_own_verdict_meets...), the same-vendor and failing-judge warnings (test_panel_warns_for_each_judge...), and measure.py under the cap. Point 5 is tracked in #852.

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The delta since c6b0fb9 holds: a stopped panel keeps its judges, an errored vote is no datum, --judge2 is compared with the first judge's own vote, judge_panel is declared in novelty.v1, and the run-wide stop (#708) now covers the panel grading. One nit on the docs.

What I ran, on a Linux review host at 9d0a302:

  • make ci-local: 34 passed, 0 failed, 14 skipped; 22,369 tests; changed-line coverage 96%.
  • Reproduction for the nit, with the LLM transport mocked.
  • A mutation run on test_novelty_panel.py and test_novelty_provenance.py: 9 mutants of the new code (first_verdict, the errored-vote and quorum guards in panel_record, MeasureResult.judges, the judge_panel of a stopped analyze, the keyword-only judge_verdicts, the run-wide stop in the answer handler, the errored-vote filter in grade_panel), all 9 killed.

I agree with @vchamarthi that a reply cut at the output ceiling under a spend cap now keeps error on the one-judge path too, where the base keeps the named verdict. The author's reason (the unpriced path already stores a cut reply as an error) holds, but the changelog fragment does not mention the change.

Checked, no issues

  • Repo rules: every non-merge commit has a Signed-off-by, cli-interface.json and cli_flags.py match the flags, judge_panel is an additive key declared in novelty.v1.json, the changelog fragment is present, and the new tests write nothing outside tmp_path.
  • Judge2: first_verdict takes the first entry of judge_verdicts, and removing it fails a test.
Bot points I checked that no longer apply at this head (2)
  • exai-bot's point on the generation rate delta does not apply: the study scripts and tables are not in this PR's diff, they are on its base, and the open follow-up is tracked in #775.
  • Copilot's point on the generation rate delta does not apply: the study scripts and tables are not in this PR's diff, they are on its base, and the open follow-up is tracked in #775.

Comment thread dokimi/docs/skill-novelty.md Resolved
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

@antonwolfy antonwolfy left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I pushed a commit per nit, up to 10a5fdd, which fix the nit from my review:

  • 10a5fdd: dokimi/docs/skill-novelty.md:333: the sentence now says a failed judge is left out of the vote, except for a permanent provider error, which stops the run and keeps the partial record.

make ci-local at 10a5fdd: 34 passed, 0 failed, 14 skipped.

@antonwolfy
antonwolfy merged commit 96dfccf into intel:main Nov 17, 2026
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cli-interface-change Changes dokimi's user-facing CLI surface

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants