Skip to content

fix(eval): keep an arms run whose scores overflow the significance tests - #831

Merged
antonwolfy merged 5 commits into
intel:mainfrom
antonwolfy:fix/arms-overflow-save
Nov 17, 2026
Merged

antonwolfy merged 5 commits into
intel:mainfrom
antonwolfy:fix/arms-overflow-save

Conversation

@antonwolfy

@antonwolfy antonwolfy commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

An arms run is lost when the statistics refuse its scores. ArmRunner._summarize computes the per-arm paired delta, arm_significance and interaction_significance after the arms have run and before ArmRunner.save. Scores whose paired difference or interaction contrast leave the float range raise OverflowError there (#599, #638, #703), the exception leaves ArmRunner.run, and the answers and evaluations, which cost model calls, are never written.

Input (two questions) On main Now
q1 a=1.7e308, baseline=-1.7e308, q2 zeros OverflowError: the difference of 1.7e+308 and -1.7e+308 leaves the float range (raised by the paired delta, before arm_significance runs); nothing saved saved; summary.significance_error holds that text, delta_vs_baseline is null, no per-arm significance
q1 ab=1.7e308, a=-1.7e308, b=0, base=0, q2 zeros OverflowError: deltas too large: a contrast of the four scores leaves the float range; nothing saved saved; summary.interactions_error holds that text, no interactions, the arm significance is kept

The issue offers two options: validate the score range before the run, or save the run and report the refusal. This takes the second. The scores come from the judge after the arms ran, so a pre-run check cannot see them, and a check as each arm's scores arrive would abort the run just as the overflow does. It also matches repos/compare.py and report/model_compare.py, which turn the same OverflowError into a stated refusal.

The same cause in paired_summary (eval/stats.py), which the evaluate pipeline and the HTML report call, is tracked in #830.

Closes #750

Changes

  • eval/arm_summary.py: refusal_kept(summary, key, compute) returns compute(), or on OverflowError logs a warning, records the exception text under summary[key] (the first one stays) and returns None.
  • eval/arm_runner.py _summarize: the per-arm paired_delta, arm_significance and interaction_significance go through refusal_kept. The arm family and the interaction family are guarded separately, so one refusal keeps the other family. A refused family is left out (no per-arm significance, no interactions) and its text is in summary.significance_error / summary.interactions_error. A refused paired_delta is scoped to its arm: delta_vs_baseline is null and the text is in per_arm.<arm>.delta_error, so an arm the family skips (one pair) does not hide another arm's significance table.
  • eval/arm_summary.py mean_in_range, used for the per-arm avg_aggregate: when the plain mean of finite scores leaves the float range (-1.7e308, -1.7e308 and 1.7e308, 1.7e308 summed to -inf / inf, saved as Infinity), it averages exactly with statistics.mean, which also holds for three sys.float_info.max scores (dividing each first still summed to inf). eval/stats.py paired_delta averages its differences with the same mean_in_range (+1e308, -1e308, 0 gives 0.0, 1e308 twice gives 1e308, where it returned inf); a single difference that leaves the range is still refused (a null delta with delta_error).
  • commands/arms.py: the console summary of arms run prints (delta not computed: <the refusal>) for an arm with a delta_error.
  • report/arms_report.py: where the significance table or the interaction table would be, the report prints Not computed: <the refusal>. An arm's delta_error is printed as a `<arm>` not computed: <the refusal> note under the significance table (under its heading when no arm has a row), so another arm's table stays.
  • schemas/arms.v1.json: the three keys (significance_error, interactions_error, per_arm.<arm>.delta_error) are described, a widening in place (the schema already accepted unknown keys in summary).
  • docs/benchmarking-and-comparison.md: a paragraph on the refusal keys.
  • tests/test_arm_runner_float_range.py: both repros through ArmRunner.run and build_output, saved and read back; the report text; no new key for an input without overflow; finite averages of ±1.7e308 scores, and of three sys.float_info.max scores, in a strictly parsed file; the refused delta of a thin arm is rendered as a note beside another arm's table; a delta whose sum leaves the range, and deltas that cancel, are averaged exactly; the console line of a refused delta; the logged warning; a refused delta of a one-pair arm keeps another arm's significance table.
  • eval/arm_summary.py question_deltas, called by ArmRunner.judge: a per-question delta whose scores differ by more than the float range (-1.7e308 against 1.7e308 gave inf in evaluations[].deltas_vs_baseline, saved as Infinity) is left out of deltas_vs_baseline and named in that row's delta_errors; rows without an overflow have no such key.
  • Changelog fragment (Fixed).

eval/interactions.py is unchanged and still raises, and so do arm_significance and the single-difference check of paired_delta in eval/stats.py. The other readers of summary.per_arm.*.significance and summary.interactions (subjects/evidence.py, report/arms_report.py) already treat their absence as a legal state (too few pairs).

Risk

None for an input without overflow: the summary is the same, with no new key. For an overflowing input the run now finishes and saves with the section missing and the refusal recorded, where it used to raise. A reader that needs significance for every arm must now also handle an arm whose delta_vs_baseline is null together with summary.significance_error or its own delta_error. paired_delta now returns the exact mean where it returned inf for differences whose sum leaves the float range; its only caller outside tests is the arms summary.

Open PRs on the same files: #667 moves the number formatters of report/arms_report.py to report/fmt.py; this change adds an early return at the top of _significance_lines and _interaction_lines, and the lines #667 rewrites in those functions are further down, so they merge apart (the new Not computed lines use no formatter). #818 adds a paragraph to docs/benchmarking-and-comparison.md about the sign-flip tie width, in a different section from this change's paragraph on the refusal keys. #784 and #806 change eval/arms_run.py, eval/answerer.py and docs of the knowledge runs, none of the lines this change edits.

Testing

On a Linux review host, at daaaa62b (on main b3225e1):

  • make ci-local: 34 passed, 0 failed, 14 skipped (the other Python legs, docker, the semgrep org check, conda-forge, fuzz, the Claude Code install and jobs that do not run on a PR). 22069 tests passed, 19 skipped.
  • The repros: on main's code tests/test_arm_runner_float_range.py gives 3 failed, 1 passed, the first failure OverflowError: the difference of 1.7e+308 and -1.7e+308 leaves the float range from build_output; with this change all 4 pass.
  • Mutants, each failing the test named: interaction guard removed (test_overflowing_interaction_contrast_keeps_the_arm_significance); arm-significance guard removed (test_overflowing_paired_difference_keeps_the_run, test_report_states_why_the_sections_are_missing); delta guard removed (the same two); report refusal text removed for the interaction table, and for the significance table (test_report_states_why_the_sections_are_missing); refusal key written on every call (test_no_refusal_key_without_overflow and the two repro tests).
  • Review repros: ±1.7e308 on two questions saved avg_aggregate: -Infinity / Infinity, now -1.7e308 / 1.7e308 (strict parse of the saved file); eight questions with q1 baseline=-1.7e308, a=1.7e308, b=-1.7e308 and the rest baseline=0, b=1 with a unscored gave significance_error and a b table in the same summary, now a.delta_error only and b's table is rendered; a=1e308 on two questions with baseline=0 gave delta_vs_baseline: Infinity, now null with delta_error.
  • Review repros: mean_in_range([sys.float_info.max] * 3) returned inf with the divide-first fallback, now sys.float_info.max. Mutants: the fallback back to divide-first fails test_saved_average_of_three_maximum_scores_is_the_maximum; the arm notes removed from the report fails test_refused_delta_of_a_thin_arm_keeps_another_arms_significance.
  • Mutants of these tests: mean_in_range always the plain mean (test_saved_averages_of_extreme_scores_are_finite_json, test_saved_average_of_three_maximum_scores_is_the_maximum, test_delta_whose_sum_leaves_the_range_is_averaged_exactly); a delta refusal written to summary.significance_error (test_overflowing_paired_difference_keeps_the_run, test_refused_delta_of_a_thin_arm_keeps_another_arms_significance).
  • Review repros: deltas +1e308, -1e308, 0 were refused with delta_error and are now 0.0, 1e308 twice is 1e308 (strict parse of the saved file); a refused delta is printed by arms run. Mutants: paired_delta back to the sum-of-magnitudes check fails test_delta_whose_sum_leaves_the_range_is_averaged_exactly and test_cancelling_deltas_average_to_zero_like_the_scores; the console line removed fails test_console_summary_prints_an_arms_delta_refusal; the logger.warning removed fails test_overflowing_paired_difference_keeps_the_run.
  • Review repro: a custom judge scoring the baseline -1.7e308 and the arm 1.7e308 through ArmRunner.judge, build_output and save stored inf in deltas_vs_baseline and wrote Infinity; now the row has deltas_vs_baseline: {} and delta_errors, and the whole file parses strictly (test_judged_overflowing_delta_is_refused_in_the_saved_artifact). Mutant: the finite check in question_deltas removed fails that test with Infinity is not JSON.
  • Not run: Python 3.11, 3.12 and 3.14 legs, and CI's merge commit.

Open questions

None.

Revert

A plain revert restores the old behaviour: the run raises before saving. Artifacts already saved with significance_error or interactions_error stay valid under arms.v1 after a revert, since the schema accepted unknown summary keys before.

Checklist

  • Tests added/updated and pytest passes
  • ruff, mypy, and schema checks pass
  • Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
  • Self-reviewed per "Before opening a PR" in AGENTS.md
  • Each new raise/exit and test names the doc, contract or issue it traces to
  • No ad-hoc run artifacts committed
  • New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

…sts (intel#750)

ArmRunner._summarize computed the per-arm paired delta, arm_significance and interaction_significance after the arms had run and before the save. Scores whose paired difference or interaction contrast leave the float range raise OverflowError there (intel#599, intel#638), the exception left ArmRunner.run, and the run's answers and evaluations, which cost model calls, were never written.

Each of the three computations is now guarded (refusal_kept in eval/arm_summary.py): on OverflowError it logs a warning, leaves the result out (delta_vs_baseline null, no per-arm significance, no interactions) and records the exception text in summary.significance_error or summary.interactions_error. The two families are guarded separately, so an interaction refusal keeps the arm significance. The arms report prints the recorded refusal where the table would be, and arms.v1.json describes the two keys (a widening). An input without overflow gives the same summary as before. stats.py and interactions.py still raise.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@exai-bot

exai-bot commented Nov 16, 2026 •

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 5 selected item(s).

CI at daaaa62: 25 passing, 3 skipped.

Review details
  • Commit reviewed: daaaa62 against merge base 99683e4
  • Coverage: 5 files in 1 groups
  • Engine: OpenCodeReview v1.13.5-exai.4.8f73eada, us.openai.gpt-6.1-sol
  • Usage: 158k tokens (105k from cache), 28s
  • Tool calls: 17 (11 file_read, 3 code_search, 1 ci_status, 1 local_check_results, 1 pr_threads)
  • ARM overflow handling, summaries, and regression coverage: 5 files

Comment thread dokimi/dokimi/eval/arm_runner.py Resolved
Comment thread dokimi/dokimi/report/arms_report.py Resolved
@codecov-commenter

codecov-commenter commented Nov 16, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.76%. Comparing base (b3225e1) to head (daaaa62).
⚠️ Report is 9 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #831      +/-   ##
==========================================
- Coverage   95.79%   95.76%   -0.03%     
==========================================
  Files         419      424       +5     
  Lines       55879    56463     +584     
  Branches     9604     9726     +122     
==========================================
+ Hits        53527    54071     +544     
- Misses       1452     1484      +32     
- Partials      900      908       +8     
Files with missing lines Coverage Δ
dokimi/dokimi/commands/arms.py 97.80% <100.00%> (+0.01%) ⬆️
dokimi/dokimi/eval/arm_runner.py 95.84% <100.00%> (-0.03%) ⬇️
dokimi/dokimi/eval/arm_summary.py 98.18% <100.00%> (+0.42%) ⬆️
dokimi/dokimi/eval/stats.py 97.83% <100.00%> (+<0.01%) ⬆️
dokimi/dokimi/report/arms_report.py 98.92% <100.00%> (+0.03%) ⬆️

... and 28 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

… arm

Settles the two review threads on intel#831 about the kept overflow run.

The per-arm average of finite extreme scores (-1.7e308, 1.7e308) summed to inf and was saved as Infinity, which strict JSON readers reject. mean_in_range divides first when the plain mean leaves the float range. paired_delta refuses a delta whose sum leaves the range, so delta_vs_baseline is null instead of Infinity.

A paired delta that overflows no longer sets summary.significance_error, which means the whole arm family was refused. It is recorded as per_arm.<arm>.delta_error, so an arm that the family skips (one pair) does not hide another arm's valid significance table. significance_error stays for a refused family.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Comment thread dokimi/dokimi/eval/arm_runner.py Resolved
Comment thread dokimi/dokimi/eval/arm_summary.py Resolved
…aximum scores exactly

Settles two review points on intel#831. The arms report never read `per_arm.<arm>.delta_error`, so an arm whose delta was refused showed a dash and no reason when another arm's family test succeeded; the significance section now prints an arm-labelled `not computed` note beside the table, or under its heading when no arm has a row.

`mean_in_range` divided each score first, and three `sys.float_info.max` scores still summed to `inf` after rounding; it now falls back to `statistics.mean`, as `stats._mean` does. Tests: the thin-arm test asserts the rendered note, and a three-question maximum-score run is saved and parsed strictly.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

@vchamarthi vchamarthi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Not approved yet · reviewed 6ff3ab5

Next: @antonwolfy: settle 3 open questions (below). 1 nit optional.

Blocking

  • Open question for @antonwolfy: paired_delta refuses by the sum of magnitudes (dokimi/dokimi/eval/stats.py:444), so +1e308, -1e308 and 0 (mean 0.0), and 1e308 twice (mean 1e308), get a null delta_vs_baseline with delta_error, while the same scores' avg_aggregate is exact. Compute the delta with statistics.mean, as mean_in_range does, or state in the PR description and dokimi/docs/benchmarking-and-comparison.md that a delta is refused by the sum of magnitudes.
  • Open question for @antonwolfy: The console summary (dokimi/dokimi/commands/arms.py:93-95) prints no delta and no reason for a refused arm, where the report prints delta_error. Print delta_error in the console summary, or say in the PR that the console summary omits refusals.
  • Open question for @antonwolfy: The PR description says eval/stats.py is unchanged and still raises, but the diff changes paired_delta (dokimi/dokimi/eval/stats.py:444). Correct that sentence in the PR description.

Verified on 6ff3ab5: ✅ changed tests: 8 passed, fail on main · ✅ full suite with main merged in: 22226 passed · ✅ merges onto main · ⚠️ 12 broken variants of the changed code: 10 caught by the tests, 2 not · 5 mutations tried · 5 edge inputs tried

15 checks behind it, each with its evidence
Area What holds How
Overflow run saved The eight changed tests pass on the head. They pin both #750 repros: the run is saved with its refusal key, and an interaction refusal keeps the arm significance tests on the head
Main refuses On main seven of the eight changed tests fail with OverflowError for the paired difference, so the run is not saved same tests on main
Merge The PR merges onto current main with no conflict merge onto main
Local ci make ci-local with main merged in exits 0, including the changed-line coverage check full suite, main merged in
No-overflow summary For three questions and four arms with no overflow, the saved summary serialises to the same JSON as on main ran it
Averages and empty delta mean_in_range returns exact means for ±1.7e308 and for three sys.float_info.max scores, keeps inf and nan, and paired_delta([]) is None ran it, edge input (numbers)
One item and order mean_in_range gives the same mean for one item, and for the same values in any order ran it, edge input (collections)
Report refusal keys Null, empty or missing refusal keys print no note, a refusal for an arm outside the arms list prints none, and notes follow the arms list order ran it, edge input (structured)
Cancelling deltas Deltas of +1e308, -1e308 and 0 are refused with delta_error, the arm average saved is 0.0, and the saved file parses strictly ran it, edge input (cancelling_signs)
Saved file Two scores of 1e308 save as an average of 1e308 with a null delta, and the file parses with no Infinity ran it, edge input (numbers)
Callers Only dokimi/dokimi/eval/arm_runner.py calls paired_delta outside the tests, so the refusal reaches no other reader ran it
Mean_in_range tests Negating the finite-mean test at dokimi/dokimi/eval/arm_summary.py:61 fails the changed tests broke the code, tests caught it
Paired_delta tests Dropping the check_sum_range call at dokimi/dokimi/eval/stats.py:444 fails the changed tests broke the code, tests caught it
Report tests Negating the significance_error check at dokimi/dokimi/report/arms_report.py:98 fails the changed tests broke the code, tests caught it
Refusal_kept re-raise The raise a broken variant adds after return None at dokimi/dokimi/eval/arm_summary.py:55 is unreachable, so that variant behaves as the head does broke the code, tests caught it

Comment thread dokimi/dokimi/eval/arm_summary.py Resolved
…ole, test the warning

Settles the review of 6ff3ab5. `paired_delta` averaged the differences with `mean_in_range`, as the per-arm `avg_aggregate` does, instead of refusing by the sum of magnitudes: `+1e308, -1e308, 0` is `0.0` and `1e308` twice is `1e308`; only a single difference that leaves the range is refused. `arms run` prints `(delta not computed: <the refusal>)` for an arm with a `delta_error`. `test_overflowing_paired_difference_keeps_the_run` checks the logged warning. The docs and the changelog fragment say so.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@antonwolfy

Copy link
Copy Markdown
Contributor Author

(commented) 🟡 Not approved yet · reviewed 6ff3ab5

@vchamarthi (#831 (review)): All three open questions are settled in 99683e4. (1) paired_delta no longer refuses by the sum of magnitudes: it averages with mean_in_range, like avg_aggregate, so +1e308, -1e308, 0 is 0.0 and 1e308 twice is 1e308; only a single difference that leaves the range is refused. Tests: test_delta_whose_sum_leaves_the_range_is_averaged_exactly, test_cancelling_deltas_average_to_zero_like_the_scores; the docs paragraph in benchmarking-and-comparison.md says so. (2) arms run prints (delta not computed: <the refusal>) in its console summary for an arm with a delta_error (test_console_summary_prints_an_arms_delta_refusal). (3) The description no longer says eval/stats.py is unchanged: it states the paired_delta change, and that arm_significance and the single-difference check still raise.

… the judge

ArmRunner.judge subtracted the baseline score unchecked, so scores of -1.7e308 and 1.7e308 stored inf in evaluations[].deltas_vs_baseline and the saved artifact held Infinity (review thread on test_arm_runner_float_range.py). The delta is now left out and named in the row's delta_errors; a test judges, builds and saves, then parses the whole file strictly.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

@napetrov napetrov left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at daaaa62. The vchamarthi blockers from 6ff3ab5 are fixed on head: paired_delta is mean-exact (+1e308, -1e308, 0 gives 0.0), the console prints delta_error, and the PR text is corrected.
Probed the overflow cases, mean_in_range on float max, and the saved JSON; 183 arms and sign-flip tests pass.
Non-blocking question: one arm with two overflowing pairs drops the significance of the whole family, even for arms whose pairs are finite. Documented as 'a refused family is left out'; intended?

@antonwolfy

Copy link
Copy Markdown
Contributor Author

(approved) Approving at daaaa62. The vchamarthi blockers from 6ff3ab5 are fixed on head: paired_delta is mean-exact (+1e308, -1e308, 0 gives 0.0), the console prints delta_error, and the PR text is c

@napetrov (#831 (review)): Yes, intended; no change. arm_significance (eval/stats.py:668) corrects each arm's sign-flip p across all arms tested in the call (_apply_correction, Holm by default, stats.py:709). Leaving one arm out would change the adjusted p of the other arms, so a table for the finite arms would carry p-values from a different family than the one asked for. The family is therefore refused as a unit and summary.significance_error names the cause. Each arm's paired_delta is guarded separately, so a refused delta stays scoped to its arm (delta_error) and keeps the other arms' averages and deltas. I read the code; no code change this round.

@antonwolfy
antonwolfy merged commit 08f393c into intel:main Nov 17, 2026
28 checks passed
@antonwolfy
antonwolfy deleted the fix/arms-overflow-save branch November 17, 2026 09:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Arms run is lost when a score-range OverflowError is raised after the arms have run

5 participants