Repository navigation
fix(eval): keep an arms run whose scores overflow the significance tests - #831
Conversation
…sts (intel#750) ArmRunner._summarize computed the per-arm paired delta, arm_significance and interaction_significance after the arms had run and before the save. Scores whose paired difference or interaction contrast leave the float range raise OverflowError there (intel#599, intel#638), the exception left ArmRunner.run, and the run's answers and evaluations, which cost model calls, were never written. Each of the three computations is now guarded (refusal_kept in eval/arm_summary.py): on OverflowError it logs a warning, leaves the result out (delta_vs_baseline null, no per-arm significance, no interactions) and records the exception text in summary.significance_error or summary.interactions_error. The two families are guarded separately, so an interaction refusal keeps the arm significance. The arms report prints the recorded refusal where the table would be, and arms.v1.json describes the two keys (a widening). An input without overflow gives the same summary as before. stats.py and interactions.py still raise. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 5 selected item(s). CI at Review details
|
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #831 +/- ##
==========================================
- Coverage 95.79% 95.76% -0.03%
==========================================
Files 419 424 +5
Lines 55879 56463 +584
Branches 9604 9726 +122
==========================================
+ Hits 53527 54071 +544
- Misses 1452 1484 +32
- Partials 900 908 +8
🚀 New features to boost your workflow:
|
… arm Settles the two review threads on intel#831 about the kept overflow run. The per-arm average of finite extreme scores (-1.7e308, 1.7e308) summed to inf and was saved as Infinity, which strict JSON readers reject. mean_in_range divides first when the plain mean leaves the float range. paired_delta refuses a delta whose sum leaves the range, so delta_vs_baseline is null instead of Infinity. A paired delta that overflows no longer sets summary.significance_error, which means the whole arm family was refused. It is recorded as per_arm.<arm>.delta_error, so an arm that the family skips (one pair) does not hide another arm's valid significance table. significance_error stays for a refused family. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
…aximum scores exactly Settles two review points on intel#831. The arms report never read `per_arm.<arm>.delta_error`, so an arm whose delta was refused showed a dash and no reason when another arm's family test succeeded; the significance section now prints an arm-labelled `not computed` note beside the table, or under its heading when no arm has a row. `mean_in_range` divided each score first, and three `sys.float_info.max` scores still summed to `inf` after rounding; it now falls back to `statistics.mean`, as `stats._mean` does. Tests: the thin-arm test asserts the rendered note, and a three-question maximum-score run is saved and parsed strictly. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
vchamarthi
left a comment
There was a problem hiding this comment.
🟡 Not approved yet · reviewed 6ff3ab5
Next: @antonwolfy: settle 3 open questions (below). 1 nit optional.
Blocking
- Open question for @antonwolfy:
paired_deltarefuses by the sum of magnitudes (dokimi/dokimi/eval/stats.py:444), so+1e308,-1e308and0(mean0.0), and1e308twice (mean1e308), get a nulldelta_vs_baselinewithdelta_error, while the same scores'avg_aggregateis exact. Compute the delta withstatistics.mean, asmean_in_rangedoes, or state in the PR description anddokimi/docs/benchmarking-and-comparison.mdthat a delta is refused by the sum of magnitudes. - Open question for @antonwolfy: The console summary (
dokimi/dokimi/commands/arms.py:93-95) prints no delta and no reason for a refused arm, where the report printsdelta_error. Printdelta_errorin the console summary, or say in the PR that the console summary omits refusals. - Open question for @antonwolfy: The PR description says
eval/stats.pyis unchanged and still raises, but the diff changespaired_delta(dokimi/dokimi/eval/stats.py:444). Correct that sentence in the PR description.
Verified on 6ff3ab5: ✅ changed tests: 8 passed, fail on main · ✅ full suite with main merged in: 22226 passed · ✅ merges onto main ·
15 checks behind it, each with its evidence
| Area | What holds | How |
|---|---|---|
| Overflow run saved | The eight changed tests pass on the head. They pin both #750 repros: the run is saved with its refusal key, and an interaction refusal keeps the arm significance | tests on the head |
| Main refuses | On main seven of the eight changed tests fail with OverflowError for the paired difference, so the run is not saved |
same tests on main |
| Merge | The PR merges onto current main with no conflict | merge onto main |
| Local ci | make ci-local with main merged in exits 0, including the changed-line coverage check |
full suite, main merged in |
| No-overflow summary | For three questions and four arms with no overflow, the saved summary serialises to the same JSON as on main | ran it |
| Averages and empty delta | mean_in_range returns exact means for ±1.7e308 and for three sys.float_info.max scores, keeps inf and nan, and paired_delta([]) is None |
ran it, edge input (numbers) |
| One item and order | mean_in_range gives the same mean for one item, and for the same values in any order |
ran it, edge input (collections) |
| Report refusal keys | Null, empty or missing refusal keys print no note, a refusal for an arm outside the arms list prints none, and notes follow the arms list order | ran it, edge input (structured) |
| Cancelling deltas | Deltas of +1e308, -1e308 and 0 are refused with delta_error, the arm average saved is 0.0, and the saved file parses strictly |
ran it, edge input (cancelling_signs) |
| Saved file | Two scores of 1e308 save as an average of 1e308 with a null delta, and the file parses with no Infinity |
ran it, edge input (numbers) |
| Callers | Only dokimi/dokimi/eval/arm_runner.py calls paired_delta outside the tests, so the refusal reaches no other reader |
ran it |
| Mean_in_range tests | Negating the finite-mean test at dokimi/dokimi/eval/arm_summary.py:61 fails the changed tests |
broke the code, tests caught it |
| Paired_delta tests | Dropping the check_sum_range call at dokimi/dokimi/eval/stats.py:444 fails the changed tests |
broke the code, tests caught it |
| Report tests | Negating the significance_error check at dokimi/dokimi/report/arms_report.py:98 fails the changed tests |
broke the code, tests caught it |
| Refusal_kept re-raise | The raise a broken variant adds after return None at dokimi/dokimi/eval/arm_summary.py:55 is unreachable, so that variant behaves as the head does |
broke the code, tests caught it |
…ole, test the warning Settles the review of 6ff3ab5. `paired_delta` averaged the differences with `mean_in_range`, as the per-arm `avg_aggregate` does, instead of refusing by the sum of magnitudes: `+1e308, -1e308, 0` is `0.0` and `1e308` twice is `1e308`; only a single difference that leaves the range is refused. `arms run` prints `(delta not computed: <the refusal>)` for an arm with a `delta_error`. `test_overflowing_paired_difference_keeps_the_run` checks the logged warning. The docs and the changelog fragment say so. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@vchamarthi (#831 (review)): All three open questions are settled in 99683e4. (1) |
… the judge ArmRunner.judge subtracted the baseline score unchecked, so scores of -1.7e308 and 1.7e308 stored inf in evaluations[].deltas_vs_baseline and the saved artifact held Infinity (review thread on test_arm_runner_float_range.py). The delta is now left out and named in the row's delta_errors; a test judges, builds and saves, then parses the whole file strictly. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
napetrov
left a comment
There was a problem hiding this comment.
Approving at daaaa62. The vchamarthi blockers from 6ff3ab5 are fixed on head: paired_delta is mean-exact (+1e308, -1e308, 0 gives 0.0), the console prints delta_error, and the PR text is corrected.
Probed the overflow cases, mean_in_range on float max, and the saved JSON; 183 arms and sign-flip tests pass.
Non-blocking question: one arm with two overflowing pairs drops the significance of the whole family, even for arms whose pairs are finite. Documented as 'a refused family is left out'; intended?
@napetrov (#831 (review)): Yes, intended; no change. |
Summary
An arms run is lost when the statistics refuse its scores.
ArmRunner._summarizecomputes the per-arm paired delta,arm_significanceandinteraction_significanceafter the arms have run and beforeArmRunner.save. Scores whose paired difference or interaction contrast leave the float range raiseOverflowErrorthere (#599, #638, #703), the exception leavesArmRunner.run, and the answers and evaluations, which cost model calls, are never written.maina=1.7e308, baseline=-1.7e308, q2 zerosOverflowError: the difference of 1.7e+308 and -1.7e+308 leaves the float range(raised by the paired delta, beforearm_significanceruns); nothing savedsummary.significance_errorholds that text,delta_vs_baselineis null, no per-armsignificanceab=1.7e308, a=-1.7e308, b=0, base=0, q2 zerosOverflowError: deltas too large: a contrast of the four scores leaves the float range; nothing savedsummary.interactions_errorholds that text, nointeractions, the armsignificanceis keptThe issue offers two options: validate the score range before the run, or save the run and report the refusal. This takes the second. The scores come from the judge after the arms ran, so a pre-run check cannot see them, and a check as each arm's scores arrive would abort the run just as the overflow does. It also matches
repos/compare.pyandreport/model_compare.py, which turn the sameOverflowErrorinto a stated refusal.The same cause in
paired_summary(eval/stats.py), which theevaluatepipeline and the HTML report call, is tracked in #830.Closes #750
Changes
eval/arm_summary.py:refusal_kept(summary, key, compute)returnscompute(), or onOverflowErrorlogs a warning, records the exception text undersummary[key](the first one stays) and returns None.eval/arm_runner.py_summarize: the per-armpaired_delta,arm_significanceandinteraction_significancego throughrefusal_kept. The arm family and the interaction family are guarded separately, so one refusal keeps the other family. A refused family is left out (no per-armsignificance, nointeractions) and its text is insummary.significance_error/summary.interactions_error. A refusedpaired_deltais scoped to its arm:delta_vs_baselineis null and the text is inper_arm.<arm>.delta_error, so an arm the family skips (one pair) does not hide another arm's significance table.eval/arm_summary.pymean_in_range, used for the per-armavg_aggregate: when the plain mean of finite scores leaves the float range (-1.7e308, -1.7e308and1.7e308, 1.7e308summed to-inf/inf, saved asInfinity), it averages exactly withstatistics.mean, which also holds for threesys.float_info.maxscores (dividing each first still summed toinf).eval/stats.pypaired_deltaaverages its differences with the samemean_in_range(+1e308, -1e308, 0gives0.0,1e308twice gives1e308, where it returnedinf); a single difference that leaves the range is still refused (a null delta withdelta_error).commands/arms.py: the console summary ofarms runprints(delta not computed: <the refusal>)for an arm with adelta_error.report/arms_report.py: where the significance table or the interaction table would be, the report printsNot computed: <the refusal>. An arm'sdelta_erroris printed as a`<arm>` not computed: <the refusal>note under the significance table (under its heading when no arm has a row), so another arm's table stays.schemas/arms.v1.json: the three keys (significance_error,interactions_error,per_arm.<arm>.delta_error) are described, a widening in place (the schema already accepted unknown keys insummary).docs/benchmarking-and-comparison.md: a paragraph on the refusal keys.tests/test_arm_runner_float_range.py: both repros throughArmRunner.runandbuild_output, saved and read back; the report text; no new key for an input without overflow; finite averages of±1.7e308scores, and of threesys.float_info.maxscores, in a strictly parsed file; the refused delta of a thin arm is rendered as a note beside another arm's table; a delta whose sum leaves the range, and deltas that cancel, are averaged exactly; the console line of a refused delta; the logged warning; a refused delta of a one-pair arm keeps another arm's significance table.eval/arm_summary.pyquestion_deltas, called byArmRunner.judge: a per-question delta whose scores differ by more than the float range (-1.7e308against1.7e308gaveinfinevaluations[].deltas_vs_baseline, saved asInfinity) is left out ofdeltas_vs_baselineand named in that row'sdelta_errors; rows without an overflow have no such key.eval/interactions.pyis unchanged and still raises, and so doarm_significanceand the single-difference check ofpaired_deltaineval/stats.py. The other readers ofsummary.per_arm.*.significanceandsummary.interactions(subjects/evidence.py,report/arms_report.py) already treat their absence as a legal state (too few pairs).Risk
None for an input without overflow: the summary is the same, with no new key. For an overflowing input the run now finishes and saves with the section missing and the refusal recorded, where it used to raise. A reader that needs
significancefor every arm must now also handle an arm whosedelta_vs_baselineis null together withsummary.significance_erroror its owndelta_error.paired_deltanow returns the exact mean where it returnedinffor differences whose sum leaves the float range; its only caller outside tests is the arms summary.Open PRs on the same files: #667 moves the number formatters of
report/arms_report.pytoreport/fmt.py; this change adds an early return at the top of_significance_linesand_interaction_lines, and the lines #667 rewrites in those functions are further down, so they merge apart (the newNot computedlines use no formatter). #818 adds a paragraph todocs/benchmarking-and-comparison.mdabout the sign-flip tie width, in a different section from this change's paragraph on the refusal keys. #784 and #806 changeeval/arms_run.py,eval/answerer.pyand docs of the knowledge runs, none of the lines this change edits.Testing
On a Linux review host, at
daaaa62b(onmainb3225e1):make ci-local: 34 passed, 0 failed, 14 skipped (the other Python legs, docker, the semgrep org check, conda-forge, fuzz, the Claude Code install and jobs that do not run on a PR). 22069 tests passed, 19 skipped.main's codetests/test_arm_runner_float_range.pygives3 failed, 1 passed, the first failureOverflowError: the difference of 1.7e+308 and -1.7e+308 leaves the float rangefrombuild_output; with this change all 4 pass.test_overflowing_interaction_contrast_keeps_the_arm_significance); arm-significance guard removed (test_overflowing_paired_difference_keeps_the_run,test_report_states_why_the_sections_are_missing); delta guard removed (the same two); report refusal text removed for the interaction table, and for the significance table (test_report_states_why_the_sections_are_missing); refusal key written on every call (test_no_refusal_key_without_overflowand the two repro tests).±1.7e308on two questions savedavg_aggregate: -Infinity/Infinity, now-1.7e308/1.7e308(strict parse of the saved file); eight questions with q1baseline=-1.7e308, a=1.7e308, b=-1.7e308and the restbaseline=0, b=1withaunscored gavesignificance_errorand abtable in the same summary, nowa.delta_erroronly andb's table is rendered;a=1e308on two questions withbaseline=0gavedelta_vs_baseline: Infinity, now null withdelta_error.mean_in_range([sys.float_info.max] * 3)returnedinfwith the divide-first fallback, nowsys.float_info.max. Mutants: the fallback back to divide-first failstest_saved_average_of_three_maximum_scores_is_the_maximum; the arm notes removed from the report failstest_refused_delta_of_a_thin_arm_keeps_another_arms_significance.mean_in_rangealways the plain mean (test_saved_averages_of_extreme_scores_are_finite_json,test_saved_average_of_three_maximum_scores_is_the_maximum,test_delta_whose_sum_leaves_the_range_is_averaged_exactly); a delta refusal written tosummary.significance_error(test_overflowing_paired_difference_keeps_the_run,test_refused_delta_of_a_thin_arm_keeps_another_arms_significance).+1e308, -1e308, 0were refused withdelta_errorand are now0.0,1e308twice is1e308(strict parse of the saved file); a refused delta is printed byarms run. Mutants:paired_deltaback to the sum-of-magnitudes check failstest_delta_whose_sum_leaves_the_range_is_averaged_exactlyandtest_cancelling_deltas_average_to_zero_like_the_scores; the console line removed failstest_console_summary_prints_an_arms_delta_refusal; thelogger.warningremoved failstest_overflowing_paired_difference_keeps_the_run.-1.7e308and the arm1.7e308throughArmRunner.judge,build_outputandsavestoredinfindeltas_vs_baselineand wroteInfinity; now the row hasdeltas_vs_baseline: {}anddelta_errors, and the whole file parses strictly (test_judged_overflowing_delta_is_refused_in_the_saved_artifact). Mutant: the finite check inquestion_deltasremoved fails that test withInfinity is not JSON.Open questions
None.
Revert
A plain revert restores the old behaviour: the run raises before saving. Artifacts already saved with
significance_errororinteractions_errorstay valid underarms.v1after a revert, since the schema accepted unknownsummarykeys before.Checklist
pytestpassesruff,mypy, and schema checks passAGENTS.mdraise/exit and test names the doc, contract or issue it traces to.github/.licenserc.yaml)