Repository navigation
fix(eval): checked_mean sums floats exactly, not by the version's sum() - #823
Conversation
…() (intel#772) checked_mean added its values with sum(), which compensates float rounding only from Python 3.12: before that, deltas that cancel (`[1e307] * 6 + [-1e307] * 6`) left a residue of -2.079e290 in the repo compare summary mean while the interval used math.fsum. Even on 3.12+, sum() loses its compensation when the first value is an int: `checked_mean([1, 1e16, 1.0, -1e16])` was 0.25, the exact mean is 0.5. The finite branch now returns math.fsum(values) / len(values) after check_sum_range, so the mean is the same on every version. Integer-only inputs keep the exact integer sum, because math.fsum would first round each integer to a float (`[2**60 + 1, 1, -2**60]` has the mean 2/3, fsum gives 1/3). The non-finite branch and the refusals are unchanged. test_checked_mean_is_the_plain_mean_where_it_is_finite now compares with math.fsum(...) / n, since sum() can differ from it in the last bit before 3.12. New tests pin the cancelling case, the int-first case, the exact integer mean and the sign_flip_summary mean. Changelog fragment (Fixed). Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 3 selected item(s). CI at Review details
|
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #823 +/- ##
==========================================
- Coverage 95.76% 95.74% -0.02%
==========================================
Files 416 423 +7
Lines 55624 56426 +802
Branches 9553 9718 +165
==========================================
+ Hits 53267 54025 +758
- Misses 1456 1492 +36
- Partials 901 909 +8
🚀 New features to boost your workflow:
|
Settles the review point that `checked_mean([2**53 + 1, -(2**53), 0.0])` returned 0.0: `math.fsum` rounds each integer to a float first. Floats only still use `fsum` and integers only still sum as integers; a mix is now summed as exact fractions, so the result is 1 / 3. The test pins that case. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
…d the float range Settles the review point that checked_mean could leak a raw OverflowError on Python 3.11: the magnitude precheck used sum(), which rounds small terms away left to right and stays finite, then math.fsum raised. check_sum_range now sums the magnitudes with math.fsum, so the precheck and the mean agree on every version and the requested error, with its context, is raised. The test emulates the pre-3.12 sum() and checks the CompareError and the context prefix. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Settles the review point that the range check refused integer-only and mixed inputs whose exact total fits the float range: math.fsum rounds each integer to a float first, so [m - 2**970 + 1, 2**970 - 1] (m the largest float) raised although the exact total is m. With an integer among the values the check now sums the magnitudes as exact fractions and refuses only when that total does not fit a float; float-only inputs keep math.fsum. A test pins both cases from the review (with and without an appended 0.0). Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
napetrov
left a comment
There was a problem hiding this comment.
Change class: not computed (no receipt profile for this repo, or receipt.py failed)
Verdict: approve. No blocking findings; one nit, one question, one nit on the test. Head cbafa13, merge base 41c2ba9, lane L1. Ran: probes/edge2.py on base and head (scratch venv; numpy present); the pass ran the three checked_mean tests on head.
CI: 0 failing Actions jobs on head cbafa13; merge base is behind base tip (not a red).
| # | Finding | Label | Provenance | Scope | Probe | Expected |
|---|---|---|---|---|---|---|
| 1 | [M, 2**969, 2.0**969] is refused with OverflowError; main returned the finite mean (C3) |
nit | changed | contract | probes/edge2.py | the Risk section's "no input is refused that was taken before", or the refusal named there |
| 2 | numpy float32 deltas raise TypeError; main returned a mean | question | changed | new surface | probes/edge2.py | a float32 caller is out of the Sequence[float] contract; question whether one can reach it |
| 3 | The test named for the integer total passes on main | nit | pre-existing | local | probes/edge2.py | a test that fails on main for the input it names |
Raised before
@exai-botT1finite_delta.py(earlier line): mixed int/float inputs lose the nonzero mean — fixed on head.@exai-botT2finite_delta.py(earlier line): raw OverflowError leak inrepo compare— answered (resolved; the error type comes from the caller'serrorargument).@exai-botT3finite_delta.py(earlier line): precheck rejects supported integer inputs — fixed on head (probes/edge2.pyhead: the integer pair gives8.988e307).
Design
None.
Questions
- Pre-existing, outside the PR's lines:
checked_mean([])raises a bare ZeroDivisionError fromlen(values)rather than the caller'serrortype. Same on both trees; not this PR's change.
Coverage
Checked: whole-PR and edge passes on both trees; probes/edge2.py on base and head (integer, mixed, float32 and the overflow cases); CI summary. Not checked: the full test suite, ruff, mypy, Python 3.11/3.13 (the version-dependent sum() results are the PR's own report), the cost of the mixed path (20k values, 0.19 s on head vs 0.01 s on main, not counted as a defect), _ends in sign_flip.py still using fsum.
Follow-up issues
- The changelog fragment
20261116_130000_checked-mean-fsum.mdhas no author segment; the other fragments use<timestamp>_<author>_<branch>.md.
… main takes or refuses Settles the review points on the checked_mean tests and wording. The integer test took an input main already accepted (the integers summed as integers first); it now puts a float ahead of them, which main refused, so it fails there. A new test pins the refusal of `[m, 2**969, 2.0**969]`, where main returned the mean of the rounded total, and the changelog fragment names that refusal. The docstring says that only floats and integers are accepted, since `numpy.float32` was never part of the `Sequence[float]` contract. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@napetrov (#823 (review)): The three findings are the threads 4591740362, 4591740382 and 4591740395, answered there. The pre-existing |
|
Review points answered at Declined, with the evidence in the reply:
|
Summary
checked_mean(dokimi/dokimi/eval/finite_delta.py), which gives therepo comparesummary mean (sign_flip_summary) and the per-question means of the model comparison, added its values withsum().sum()compensates float rounding only from Python 3.12, so the mean depended on the Python version while the interval usedmath.fsum. Onmain:main[1e307] * 6 + [-1e307] * 6-2.079e290on Python 3.11 (plain left-to-right addition),0.0on 3.12+0.0[1, 1e16, 1.0, -1e16]0.25on every version, including 3.13 (an int first loses the compensation)0.5The finite branch now returns
math.fsum(values) / len(values)aftercheck_sum_range, so the mean is the same on every version. Integer-only inputs keep the exact integer sum:math.fsumrounds each integer to a float first, which would turn the mean of[2**60 + 1, 1, -2**60]from2/3into1/3. Integers mixed with floats are summed as exact fractions for the same reason:checked_mean([2**53 + 1, -(2**53), 0.0])is1/3, wherefsumgives0.0. The non-finite branch (inf and NaN keep their meaning) and the refusals ofcheck_sum_rangeare unchanged.check_sum_rangesums the magnitudes withmath.fsumtoo: a left-to-rightsum()(before 3.12) rounds small terms away and stays finite for[0x1.fffffffffffffp+1023, 2**969, 2**969], whichfsumthen refuses with a bareOverflowError; the precheck now raises the requestederrorwith itswherecontext for that input on every version. With an integer among the values the precheck sums the magnitudes as exact fractions instead (fsumrounds each integer to a float first):[m - 2**970 + 1, 2**970 - 1], withmthe largest float and with or without an appended0.0, has the exact totalmand now gives its mean.maintook that pair, but refused the same integers when a float came first ([0.0, m - 2**970 + 1, 2**970 - 1],[m - 2**970 + 1, 0.0, 2**970 - 1]: its left-to-rightsum()rounded the partial total and overflowed); those are taken now.Closes #772
Changes
checked_meanindokimi/dokimi/eval/finite_delta.py:math.fsumfor floats only, the exact integer sum for integers only, exact fractions for a mix; docstring says the sum is exactly rounded.check_sum_rangesums the magnitudes withmath.fsumfor floats only, and as exact fractions when an integer is among the values.test_checked_mean_is_the_plain_mean_where_it_is_finitecompares withmath.fsum(...) / n, sincesum()can differ from it in the last bit before 3.12. New tests: the cancelling case, the int-first case, the exact integer mean, the integer-beside-float mean, thesign_flip_summarymean, and the refusal of a total only the exact sum sees beyond the range (withsum()emulated as on 3.11),test_checked_mean_refuses_a_mixed_total_that_only_the_exact_sum_sees_beyond_the_range([m, 2**969, 2.0**969]), andtest_checked_mean_takes_integers_whose_exact_total_fits_the_float_range(the integer pair above with a0.0ahead of it or between them). The docstring ofchecked_meansays that it takes floats and integers:numpy.float32raisedTypeErrorthere, as it does now, and no caller passes one (the scores come from JSON records).Risk
Means of float deltas can change in the last bit (now the correctly rounded sum), and the cancelling inputs above change from a residue to the exact value; ordinary inputs give the same result up to the last bit. Integer-only inputs are unchanged. One kind of input is refused that was taken before: values whose exact sum of magnitudes passes the float range but whose left-to-right sum rounds back under it, such as
[m, 2**969, 2.0**969]withmthe largest float (mainreturned5.992e307; it now raises the requested error). Apart from that, no input is refused that was taken before; integer inputs whose exact total fits the float range, refused before by the range check when a float came first, are taken now. Open #822 editsdokimi/tests/test_repo_compare_overflow.pytoo (imports and the tests abovetest_checked_mean_is_the_plain_mean_where_it_is_finite): the two touch neighbouring lines of that file, so whichever merges second may need a trivial rebase; no source file is shared.Testing
On a Linux review host, at
5dcaf03(onmain41c2ba97):make ci-local: 33 passed, 0 failed, 14 skipped (steps that need other Pythons, micromamba, semgrep, or a real runner). 21742 tests passed, 19 skipped; changed-line coverage 100%.main's code the new tests fail:test_checked_mean_sums_exactly_whatever_the_python_versionandtest_sign_flip_summary_mean_is_the_exact_mean(assert 0.25 == 0.5); 2 failed, 20 passed.fsum(-2.079e290 == 0.0and0.0 == 0.5),sum()instead offsum(0.25 == 0.5), no exact-integer branch (0.333… == 2 / 3),fsumfor a mix of integers and floats (test_checked_mean_keeps_integer_contributions_beside_floatsfails), andsum()instead offsumincheck_sum_range(test_checked_mean_refuses_a_total_that_only_the_exact_sum_sees_beyond_the_rangefails withOverflowError: intermediate overflow in fsum), andfsumfor every input incheck_sum_range(both parametrisations oftest_checked_mean_takes_integers_whose_exact_total_fits_the_float_rangefail), and no check at all in the integer branch ofcheck_sum_range(test_checked_mean_refuses_a_mixed_total_that_only_the_exact_sum_sees_beyond_the_rangefails).main's code the mixed-refusal test and both parametrisations of the integer test fail (3 failed):mainreturns5.992e307for[m, 2**969, 2.0**969]and refuses[0.0, m - 2**970 + 1, 2**970 - 1]withOverflowError.[1e307] * 6 + [-1e307] * 6already gives0.0; the int-first input stands for themainfailure there).Open questions
None.
Revert
A plain revert restores the
sum()mean. Nothing is stored.Checklist
pytestpassesruff,mypy, and schema checks passAGENTS.mdraise/exit and test names the doc, contract or issue it traces to.github/.licenserc.yaml)