Skip to content

fix(eval): paired_summary returns no delta where a paired difference leaves the float range (#830) - #869

Merged
antonwolfy merged 5 commits into
intel:mainfrom
antonwolfy:fix/paired-summary-overflow
Nov 17, 2026
Merged

antonwolfy merged 5 commits into
intel:mainfrom
antonwolfy:fix/paired-summary-overflow

Conversation

@antonwolfy

@antonwolfy antonwolfy commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

paired_summary raises OverflowError for two finite scores whose difference leaves the float range, and two callers did not catch it. On main, paired_summary([{"question_id": "q1", "with_docs": {"aggregate": 1.7e308}, "without_docs": {"aggregate": -1.7e308}}, {"question_id": "q2", "with_docs": {"aggregate": 0.0}, "without_docs": {"aggregate": 0.0}}]) raises OverflowError: the difference of 1.7e+308 and -1.7e+308 leaves the float range.

Caller On main
Step 5 of the evaluate pipeline (orchestrator/pipeline.py) stops after the answers and evaluations are saved, before Step 6, so there is no summary and no report
report generate (report/generator.py, _compute_stats, _arm_means) raises, so there is no HTML or Markdown report
report eval (report/eval_report.py) raises in the Summary table (stats), and again in significance_test, which calls paired_test

Options the issue lists: catch OverflowError in the callers and print a stated refusal, or have paired_summary return a summary with no delta. This PR takes the second: the arm averages, ranges and counts of such scores are finite and stay, and the reports already print an em dash for a statistic with no value (docs/knowledge-runs.md:128: a summary without complete pairs "shows —"). A refusal in the callers would have dropped those averages. repos/compare.py and report/model_compare.py refuse with a stated error because their delta is the thing they report; paired_summary's delta is one column next to the averages.

Closes #830

Changes

  • eval/stats.py: PairedSummary.delta_mean, delta_min and delta_max are None when any complete pair's difference leaves the float range, and the new delta_overflow counts those pairs. The arm means, ranges, total, paired and unpaired are unchanged. improvements, degradations and neutral take the sign of a pair without a delta from comparing its two scores (exact for finite scores), and from the difference otherwise, so every summary whose differences fit the range is unchanged. checked_delta, paired_test, paired_delta and the arms path still raise.
  • report/generator.py: _stat_block and _arm_means give None for a statistic with no value, which the page prints as an em dash through _shown. _unpaired_note adds the line _No delta: N complete pairs differ by more than the float range._, and _overall_advice says there is no delta and no conclusion instead of comparing None. The JSON stats carry delta_overflow_questions.
  • orchestrator/pipeline.py (Step 5 only): the evaluation summary has delta_avg: None and the printed line reads delta=none (a paired difference leaves the float range). commands/orchestrate.py prints an em dash for it.
  • report/eval_report.py: stats and the Summary, Overall, agreement and delta lines read a None delta (em dash or "none — a paired difference leaves the float range"). Its significance section, which calls paired_test, prints _Not computed: <the refusal>._ instead of stopping.
  • tests/test_stats_float_range.py: the test that pinned paired_summary raising now asserts it returns no delta, and keeps the two raises that stay (paired_delta, arm_significance).
  • New tests/test_paired_summary_overflow.py, including one test for the Delta: line of cmd_evaluate (pipeline mocked) and one for the Dynamic vs Static agreement line of report eval.
  • gate/trust_gate.py: the min_delta check, which subtracted the arm means directly, is measured=False with value null and a "Not measured" message when that difference is not finite, so report generate --format json no longer writes Infinity for it and the verdict lists min_delta under not_measured. Test: test_report_json_trust_reports_min_delta_not_measured_and_is_strict_json (three pairs of 1.7e308 and -1.7e308, parsed with non-finite constants rejected).
  • Changelog fragments (Fixed): one for the summary, one for the trust check.

Risk

PairedSummary.delta_mean, delta_min and delta_max are now Optional[float]; every reader in the package is updated (git grep for paired_summary and the fields finds generator.py, eval_report.py, pipeline.py and commands/orchestrate.py). Output for scores whose differences fit the range is unchanged. For scores that do not, a run that failed now completes with an em dash. The unpaired-note line is new text in the Markdown report only for that case. The cluster and source tables show an em dash for a group with such a pair, without the note.

This touches the same files as open PR #667 (report/generator.py; this PR leaves _shown and _format_markdown alone) and PR #820 (orchestrator/pipeline.py; only the Step 5 lines that read the delta change) and #841 (eval/stats.py, which appends pearson) and #806 (report/eval_report.py, where it adds an answers_from_items helper and a diagnosis label, away from the stats, summary and significance lines changed here). The edits are on separate lines.

Testing

On a Linux review host, at 4b9e0c6 (on main 08f393c):

  • make ci-local: 34 passed, 0 failed, 14 skipped (docker, other Python versions, conda-forge, fuzz, semgrep and the jobs that only run on merge). 22402 tests passed, 19 skipped.
  • tests/test_paired_summary_overflow.py alone now takes both branches of the delta.avg is None check in _overall_advice (--cov-branch lists neither 525->526 nor 525->527 as missing); a new test draws the "hurting" and "minimal impact" conclusions from a delta that exists. Two mutants (the check inverted, the check made == 0.0) each fail it.
  • The note under the report table reads "1 complete pair differs" for one overflowing pair and "2 complete pairs differ" for two; two mutants (the singular chosen never, the singular chosen for every count) each fail one of the two tests that pin the wording.
  • Three pairs of 1.7e308 and -1.7e308 through ReportGenerator.generate_report(..., "json"): before the trust-gate change the min_delta check had value: Infinity and passed: true; now value: null, measured: false. The new test fails with the guard removed.
  • The issue's input on main's code: 9 of the 10 new tests fail (the tenth is the regression guard for pairs inside the range), the first with OverflowError: the difference of 1.7e+308 and -1.7e+308 leaves the float range; the changed tests/test_stats_float_range.py test fails there with the same OverflowError. All pass here.
  • 19 mutants, each failing at least one new test: restoring the raise; keeping delta_mean or delta_min/delta_max; counting an overflowing pair as neither improvement nor degradation; dropping the arm means or ranges; not counting delta_overflow; dropping the note, the _overall_advice branch or the None guards of _stat_block and _arm_means; rounding or formatting a None delta in the pipeline and in report eval; re-raising in the report eval refusal; flipping the None test of the Delta: print of cmd_evaluate and of the agreement line.
  • Not run: the Python 3.11, 3.12 and 3.14 legs, and Docker steps.
  • Merged together with fix/exact-sign-flip-sums, fix/interaction-tie-width on main 08f393c: pytest -n auto 22418 passed, 19 skipped, mypy clean, ruff check clean.

Open questions

None.

Revert

A plain revert restores the OverflowError from paired_summary and the pipeline and report failures for such scores. Nothing is stored.

Checklist

  • Tests added/updated and pytest passes
  • ruff, mypy, and schema checks pass
  • Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
  • Self-reviewed per "Before opening a PR" in AGENTS.md
  • Each new raise/exit and test names the doc, contract or issue it traces to
  • No ad-hoc run artifacts committed
  • New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

…leaves the float range (intel#830)

paired_summary formed its deltas through checked_delta, which raises OverflowError for two finite scores such as 1.7e308 and -1.7e308. The evaluate pipeline stopped after saving the evaluations and before the report, and `report generate` and `report eval` failed.

paired_summary now returns the arm means, ranges and counts with delta_mean, delta_min and delta_max set to None, and delta_overflow counts the pairs without a delta. A pair without a delta is counted as an improvement or a degradation by comparing its two scores. The pipeline prints delta=none, `orchestrate` prints an em dash, the reports print an em dash with a one-line note, and `report eval` states that its significance test was refused instead of stopping. paired_test, paired_delta and the arms path still raise as before.

Test-narrowed: test_paired_summary_paired_delta_and_arm_significance_refuse_the_same_deltas pinned the raise of paired_summary that this change removes on purpose; it is replaced by test_paired_delta_and_arm_significance_refuse_deltas_beyond_the_range_while_paired_summary_has_none, which keeps the two raises that stay and asserts paired_summary returns no delta.
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
… of the eval report for a summary with no delta (intel#830)

The `Delta:` print of `cmd_evaluate` for a summary whose `delta_avg` is None and the `Dynamic vs Static agreement` bullet when one set has no delta had no test. One test each now: `cmd_evaluate` with a mocked pipeline result, and `generate_report` on an evaluations file with a golden set and a generated set where the generated one overflows.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@antonwolfy
antonwolfy requested a review from a team as a code owner November 17, 2026 11:45
@antonwolfy antonwolfy self-assigned this Nov 17, 2026
@exai-bot

exai-bot commented Nov 17, 2026 •

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 1 selected item(s).

CI at 4b9e0c6: 25 passing, 3 skipped.

Review details
  • Commit reviewed: 4b9e0c6 against merge base 6462ca4
  • Coverage: 1 files in 1 groups
  • Engine: OpenCodeReview v1.13.5-exai.4.8f73eada, us.openai.gpt-6.1-sol
  • Usage: 58k tokens (39k from cache), 24s
  • Tool calls: 6 (4 file_read, 2 code_search)
  • dokimi/tests/test_paired_summary_overflow.py: 1 file

Comment thread dokimi/dokimi/report/generator.py Resolved
@codecov-commenter

codecov-commenter commented Nov 17, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.78%. Comparing base (08f393c) to head (4b9e0c6).
⚠️ Report is 1 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #869      +/-   ##
==========================================
+ Coverage   95.76%   95.78%   +0.02%     
==========================================
  Files         424      425       +1     
  Lines       56511    56503       -8     
  Branches     9730     9724       -6     
==========================================
+ Hits        54115    54120       +5     
+ Misses       1487     1479       -8     
+ Partials      909      904       -5     
Files with missing lines Coverage Δ
dokimi/dokimi/commands/orchestrate.py 100.00% <100.00%> (+10.00%) ⬆️
dokimi/dokimi/eval/stats.py 97.89% <100.00%> (+0.06%) ⬆️
dokimi/dokimi/gate/trust_gate.py 100.00% <100.00%> (ø)
dokimi/dokimi/orchestrator/pipeline.py 92.15% <100.00%> (+0.02%) ⬆️
dokimi/dokimi/report/eval_report.py 93.14% <100.00%> (+0.41%) ⬆️
dokimi/dokimi/report/generator.py 95.29% <100.00%> (+1.24%) ⬆️

... and 7 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at 72eff4a. paired_summary, the pipeline line, cmd_evaluate and both reports now handle a pair whose difference leaves the float range. But the trust gate that report generate runs on the same data still computes its own delta and spread, so the fix of #830 is not complete yet.

Agreed with @exai-bot on the trust checks; I added a crash case in its thread.

What I ran, at 72eff4a:

  • make ci-local --offline: 22,346 passed, 0 failed, 72 skipped.
  • The new test file on the merge-base: 9 of its 10 tests fail there. The one that passes, ..._pairs_inside_the_range_still_has_a_delta, guards behaviour that did not change.
  • ReportGenerator.generate_report with [(1.7e308, -1.7e308), (-1.7e308, 1.7e308)]: OverflowError from statistics.stdev in evaluate_trust. With three (1.7e308, -1.7e308) pairs: min_delta passes on inf, and the JSON report has Infinity.
  • This branch merged with #867 (same stats.py and test_stats_float_range.py): both test files pass, 41 tests.

Checked, no issues

  • Counts: an overflowing pair is counted as an improvement or a degradation by comparing its scores, and never as neutral. That is right, since its scores differ.
  • Callers: every reader of delta_mean / delta_avg (pipeline, cmd_evaluate, eval_report Summary/Overall/agreement lines, generator stats, clusters, sources, advice) handles None. The grouped tables sort a None delta last.
  • report eval significance: an OverflowError from significance_test becomes a "Not computed: " line, not a number.
  • Changelog: the fragment is under Fixed and says what changed for the user.

Comment thread dokimi/dokimi/report/generator.py Resolved
… not measured

Settles the review point that evaluate_trust still subtracted the arm means directly, so a report whose paired delta is unavailable (three pairs of 1.7e308 and -1.7e308) wrote Infinity as the min_delta value and marked it passed.

The check now has value None, measured False and a "Not measured" message when the difference of the means is not finite, so the JSON stays standard and the verdict lists min_delta under not_measured. A regression test parses the report JSON with non-finite constants rejected.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Settles review thread 4599606890: the report note read "1 complete pairs differ" for the issue's own input. It now reads "1 complete pair differs" for one pair and "N complete pairs differ" otherwise, and the tests pin both.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
…t to the one that has none (intel#830)

The codecov concern on generator.py:526 stays open because the PR's own test file only ever takes the no-delta branch of `_overall_advice`; the other branch was reached by unrelated test files only. The new test draws the hurting and minimal-impact conclusions from a delta that exists and asserts the float-range line is absent, so both branches of the `delta.avg is None` check are pinned by tests/test_paired_summary_overflow.py itself.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@antonwolfy

Copy link
Copy Markdown
Contributor Author

Ready for another look at 4b9e0c6; CI is green.

Declined, with the evidence in the reply:

  • codecov: No changed line of generator.py is uncovered or partial here: both branches of the line 525 test run (the new return at 526 in test_paired_summary_overflow.py, the fall-through in the golden/arm-agreement tests).

Follow-up issues opened:

@antonwolfy

Copy link
Copy Markdown
Contributor Author

Correction to the summary above: the codecov point is not declined. It was fixed in 4b9e0c6, which adds a test that takes the other branch of the delta.avg is None check in _overall_advice; codecov is satisfied at that head and CI is green.

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

I checked 4b9e0c6. c97bcee makes the trust gate report min_delta as not measured when the arm means differ by more than the float range, so the JSON report is standard again. The score_cv crash on opposite-signed scores is tracked in #870, which is fine with me. 6462ca4 fixes the singular in the note.

Merge: safe now. It merges cleanly into current main, 5 commits behind, and its tests pass on the merge (main has changed generator.py since the base). No open PR conflicts with it.

What I ran, at 4b9e0c6:

  • make ci-local --offline: 22,349 passed, 0 failed, 72 skipped. The only FAIL row is schema_compat, which compares against the newer main (repo_event.v1.json, judge_panel in novelty.v1.json). This branch does not touch the schemas, so it is not from this PR.
  • On the merge with current main: test_paired_summary_overflow.py, test_stats_float_range.py, both trust gate tests and test_report_generator_labels.py: 108 passed.
  • evaluate_trust on three (1.7e308, -1.7e308) pairs: min_delta has value null and measured false, and the JSON parses with non-finite constants rejected.

Checked, no issues

  • Not measured, not a pass: the new branch uses measured=False, the same pattern as retrieval_hit_rate, and the verdict lists it under not_measured. The difference of two finite means can only be infinite, never NaN, so isfinite covers every case.
  • Tests: the strict-JSON test fails without c97bcee (the old value was inf). The two new advice tests cover the branch of _overall_advice that does have a delta.
  • Changelog: the second fragment is under Fixed and names the user-visible change (value null, measured false, not_measured).

@antonwolfy
antonwolfy merged commit ed9221e into intel:main Nov 17, 2026
28 checks passed
@antonwolfy
antonwolfy deleted the fix/paired-summary-overflow branch November 17, 2026 17:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

paired_summary raises OverflowError for scores whose paired difference leaves the float range, ending the evaluate pipeline and the HTML report

4 participants