Skip to content

fix(eval): checked_mean sums floats exactly, not by the version's sum() - #823

Merged
antonwolfy merged 5 commits into
intel:mainfrom
antonwolfy:fix/checked-mean-fsum
Nov 16, 2026
Merged

antonwolfy merged 5 commits into
intel:mainfrom
antonwolfy:fix/checked-mean-fsum

Conversation

@antonwolfy

@antonwolfy antonwolfy commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

checked_mean (dokimi/dokimi/eval/finite_delta.py), which gives the repo compare summary mean (sign_flip_summary) and the per-question means of the model comparison, added its values with sum(). sum() compensates float rounding only from Python 3.12, so the mean depended on the Python version while the interval used math.fsum. On main:

Input main Exact mean
[1e307] * 6 + [-1e307] * 6 -2.079e290 on Python 3.11 (plain left-to-right addition), 0.0 on 3.12+ 0.0
[1, 1e16, 1.0, -1e16] 0.25 on every version, including 3.13 (an int first loses the compensation) 0.5

The finite branch now returns math.fsum(values) / len(values) after check_sum_range, so the mean is the same on every version. Integer-only inputs keep the exact integer sum: math.fsum rounds each integer to a float first, which would turn the mean of [2**60 + 1, 1, -2**60] from 2/3 into 1/3. Integers mixed with floats are summed as exact fractions for the same reason: checked_mean([2**53 + 1, -(2**53), 0.0]) is 1/3, where fsum gives 0.0. The non-finite branch (inf and NaN keep their meaning) and the refusals of check_sum_range are unchanged. check_sum_range sums the magnitudes with math.fsum too: a left-to-right sum() (before 3.12) rounds small terms away and stays finite for [0x1.fffffffffffffp+1023, 2**969, 2**969], which fsum then refuses with a bare OverflowError; the precheck now raises the requested error with its where context for that input on every version. With an integer among the values the precheck sums the magnitudes as exact fractions instead (fsum rounds each integer to a float first): [m - 2**970 + 1, 2**970 - 1], with m the largest float and with or without an appended 0.0, has the exact total m and now gives its mean. main took that pair, but refused the same integers when a float came first ([0.0, m - 2**970 + 1, 2**970 - 1], [m - 2**970 + 1, 0.0, 2**970 - 1]: its left-to-right sum() rounded the partial total and overflowed); those are taken now.

Closes #772

Changes

  • checked_mean in dokimi/dokimi/eval/finite_delta.py: math.fsum for floats only, the exact integer sum for integers only, exact fractions for a mix; docstring says the sum is exactly rounded. check_sum_range sums the magnitudes with math.fsum for floats only, and as exact fractions when an integer is among the values.
  • test_checked_mean_is_the_plain_mean_where_it_is_finite compares with math.fsum(...) / n, since sum() can differ from it in the last bit before 3.12. New tests: the cancelling case, the int-first case, the exact integer mean, the integer-beside-float mean, the sign_flip_summary mean, and the refusal of a total only the exact sum sees beyond the range (with sum() emulated as on 3.11), test_checked_mean_refuses_a_mixed_total_that_only_the_exact_sum_sees_beyond_the_range ([m, 2**969, 2.0**969]), and test_checked_mean_takes_integers_whose_exact_total_fits_the_float_range (the integer pair above with a 0.0 ahead of it or between them). The docstring of checked_mean says that it takes floats and integers: numpy.float32 raised TypeError there, as it does now, and no caller passes one (the scores come from JSON records).
  • Changelog fragment (Fixed).

Risk

Means of float deltas can change in the last bit (now the correctly rounded sum), and the cancelling inputs above change from a residue to the exact value; ordinary inputs give the same result up to the last bit. Integer-only inputs are unchanged. One kind of input is refused that was taken before: values whose exact sum of magnitudes passes the float range but whose left-to-right sum rounds back under it, such as [m, 2**969, 2.0**969] with m the largest float (main returned 5.992e307; it now raises the requested error). Apart from that, no input is refused that was taken before; integer inputs whose exact total fits the float range, refused before by the range check when a float came first, are taken now. Open #822 edits dokimi/tests/test_repo_compare_overflow.py too (imports and the tests above test_checked_mean_is_the_plain_mean_where_it_is_finite): the two touch neighbouring lines of that file, so whichever merges second may need a trivial rebase; no source file is shared.

Testing

On a Linux review host, at 5dcaf03 (on main 41c2ba97):

  • make ci-local: 33 passed, 0 failed, 14 skipped (steps that need other Pythons, micromamba, semgrep, or a real runner). 21742 tests passed, 19 skipped; changed-line coverage 100%.
  • On main's code the new tests fail: test_checked_mean_sums_exactly_whatever_the_python_version and test_sign_flip_summary_mean_is_the_exact_mean (assert 0.25 == 0.5); 2 failed, 20 passed.
  • Mutants, each failing the new tests: left-to-right addition instead of fsum (-2.079e290 == 0.0 and 0.0 == 0.5), sum() instead of fsum (0.25 == 0.5), no exact-integer branch (0.333… == 2 / 3), fsum for a mix of integers and floats (test_checked_mean_keeps_integer_contributions_beside_floats fails), and sum() instead of fsum in check_sum_range (test_checked_mean_refuses_a_total_that_only_the_exact_sum_sees_beyond_the_range fails with OverflowError: intermediate overflow in fsum), and fsum for every input in check_sum_range (both parametrisations of test_checked_mean_takes_integers_whose_exact_total_fits_the_float_range fail), and no check at all in the integer branch of check_sum_range (test_checked_mean_refuses_a_mixed_total_that_only_the_exact_sum_sees_beyond_the_range fails).
  • On main's code the mixed-refusal test and both parametrisations of the integer test fail (3 failed): main returns 5.992e307 for [m, 2**969, 2.0**969] and refuses [0.0, m - 2**970 + 1, 2**970 - 1] with OverflowError.
  • Not run: Python 3.11, 3.12 and 3.14 (the host runs 3.13, where [1e307] * 6 + [-1e307] * 6 already gives 0.0; the int-first input stands for the main failure there).

Open questions

None.

Revert

A plain revert restores the sum() mean. Nothing is stored.

Checklist

  • Tests added/updated and pytest passes
  • ruff, mypy, and schema checks pass
  • Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
  • Self-reviewed per "Before opening a PR" in AGENTS.md
  • Each new raise/exit and test names the doc, contract or issue it traces to
  • No ad-hoc run artifacts committed
  • New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

…() (intel#772)

checked_mean added its values with sum(), which compensates float rounding only from Python 3.12: before that, deltas that cancel (`[1e307] * 6 + [-1e307] * 6`) left a residue of -2.079e290 in the repo compare summary mean while the interval used math.fsum. Even on 3.12+, sum() loses its compensation when the first value is an int: `checked_mean([1, 1e16, 1.0, -1e16])` was 0.25, the exact mean is 0.5.

The finite branch now returns math.fsum(values) / len(values) after check_sum_range, so the mean is the same on every version. Integer-only inputs keep the exact integer sum, because math.fsum would first round each integer to a float (`[2**60 + 1, 1, -2**60]` has the mean 2/3, fsum gives 1/3). The non-finite branch and the refusals are unchanged.

test_checked_mean_is_the_plain_mean_where_it_is_finite now compares with math.fsum(...) / n, since sum() can differ from it in the last bit before 3.12. New tests pin the cancelling case, the int-first case, the exact integer mean and the sign_flip_summary mean. Changelog fragment (Fixed).

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@antonwolfy
antonwolfy requested a review from a team as a code owner November 16, 2026 13:51
@antonwolfy antonwolfy self-assigned this Nov 16, 2026
@exai-bot

exai-bot commented Nov 16, 2026 •

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 3 selected item(s).

CI at 5dcaf03: 25 passing, 3 skipped.

Review details
  • Commit reviewed: 5dcaf03 against merge base cbafa13
  • Coverage: 3 files in 1 groups
  • Engine: OpenCodeReview v1.13.5-exai.4.8f73eada, us.openai.gpt-6.1-sol
  • Usage: 93k tokens (63k from cache), 25s
  • Tool calls: 12 (6 file_read, 3 pr_threads, 1 ci_status, 1 code_search, 1 local_check_results)
  • small change set: 3 files

Comment thread dokimi/dokimi/eval/finite_delta.py Resolved
@codecov-commenter

codecov-commenter commented Nov 16, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.74%. Comparing base (41c2ba9) to head (5dcaf03).
⚠️ Report is 13 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #823      +/-   ##
==========================================
- Coverage   95.76%   95.74%   -0.02%     
==========================================
  Files         416      423       +7     
  Lines       55624    56426     +802     
  Branches     9553     9718     +165     
==========================================
+ Hits        53267    54025     +758     
- Misses       1456     1492      +36     
- Partials      901      909       +8     
Files with missing lines Coverage Δ
dokimi/dokimi/eval/finite_delta.py 100.00% <100.00%> (ø)

... and 35 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Settles the review point that `checked_mean([2**53 + 1, -(2**53), 0.0])` returned 0.0: `math.fsum` rounds each integer to a float first. Floats only still use `fsum` and integers only still sum as integers; a mix is now summed as exact fractions, so the result is 1 / 3. The test pins that case.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Comment thread dokimi/dokimi/eval/finite_delta.py Resolved
…d the float range

Settles the review point that checked_mean could leak a raw OverflowError on Python 3.11: the magnitude precheck used sum(), which rounds small terms away left to right and stays finite, then math.fsum raised.

check_sum_range now sums the magnitudes with math.fsum, so the precheck and the mean agree on every version and the requested error, with its context, is raised. The test emulates the pre-3.12 sum() and checks the CompareError and the context prefix.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Comment thread dokimi/dokimi/eval/finite_delta.py Resolved
Settles the review point that the range check refused integer-only and mixed inputs whose exact total fits the float range: math.fsum rounds each integer to a float first, so [m - 2**970 + 1, 2**970 - 1] (m the largest float) raised although the exact total is m.

With an integer among the values the check now sums the magnitudes as exact fractions and refuses only when that total does not fit a float; float-only inputs keep math.fsum. A test pins both cases from the review (with and without an appended 0.0).

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

@napetrov napetrov left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change class: not computed (no receipt profile for this repo, or receipt.py failed)

Verdict: approve. No blocking findings; one nit, one question, one nit on the test. Head cbafa13, merge base 41c2ba9, lane L1. Ran: probes/edge2.py on base and head (scratch venv; numpy present); the pass ran the three checked_mean tests on head.

CI: 0 failing Actions jobs on head cbafa13; merge base is behind base tip (not a red).

# Finding Label Provenance Scope Probe Expected
1 [M, 2**969, 2.0**969] is refused with OverflowError; main returned the finite mean (C3) nit changed contract probes/edge2.py the Risk section's "no input is refused that was taken before", or the refusal named there
2 numpy float32 deltas raise TypeError; main returned a mean question changed new surface probes/edge2.py a float32 caller is out of the Sequence[float] contract; question whether one can reach it
3 The test named for the integer total passes on main nit pre-existing local probes/edge2.py a test that fails on main for the input it names

Raised before

  • @exai-bot T1 finite_delta.py (earlier line): mixed int/float inputs lose the nonzero mean — fixed on head.
  • @exai-bot T2 finite_delta.py (earlier line): raw OverflowError leak in repo compare — answered (resolved; the error type comes from the caller's error argument).
  • @exai-bot T3 finite_delta.py (earlier line): precheck rejects supported integer inputs — fixed on head (probes/edge2.py head: the integer pair gives 8.988e307).

Design

None.

Questions

  • Pre-existing, outside the PR's lines: checked_mean([]) raises a bare ZeroDivisionError from len(values) rather than the caller's error type. Same on both trees; not this PR's change.

Coverage

Checked: whole-PR and edge passes on both trees; probes/edge2.py on base and head (integer, mixed, float32 and the overflow cases); CI summary. Not checked: the full test suite, ruff, mypy, Python 3.11/3.13 (the version-dependent sum() results are the PR's own report), the cost of the mixed path (20k values, 0.19 s on head vs 0.01 s on main, not counted as a defect), _ends in sign_flip.py still using fsum.

Follow-up issues

  • The changelog fragment 20261116_130000_checked-mean-fsum.md has no author segment; the other fragments use <timestamp>_<author>_<branch>.md.

Comment thread dokimi/dokimi/eval/finite_delta.py Resolved
Comment thread dokimi/dokimi/eval/finite_delta.py Resolved
… main takes or refuses

Settles the review points on the checked_mean tests and wording. The integer test took an input main already accepted (the integers summed as integers first); it now puts a float ahead of them, which main refused, so it fails there. A new test pins the refusal of `[m, 2**969, 2.0**969]`, where main returned the mean of the rounded total, and the changelog fragment names that refusal. The docstring says that only floats and integers are accepted, since `numpy.float32` was never part of the `Sequence[float]` contract.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@antonwolfy

Copy link
Copy Markdown
Contributor Author

(approved) Change class: not computed (no receipt profile for this repo, or receipt.py failed)

@napetrov (#823 (review)): The three findings are the threads 4591740362, 4591740382 and 4591740395, answered there. The pre-existing checked_mean([]) ZeroDivisionError is the same on both trees and outside this PR. On the fragment name: the changelog README sets no author segment and many fragments in changelog.d/ have none (e.g. 20261114_130000_cache-dir-unusable.md), so it stays.

@antonwolfy

Copy link
Copy Markdown
Contributor Author

Review points answered at 5dcaf03; CI is green.

Declined, with the evidence in the reply:

@antonwolfy
antonwolfy merged commit 8f6e8e3 into intel:main Nov 16, 2026
28 checks passed
@antonwolfy
antonwolfy deleted the fix/checked-mean-fsum branch November 16, 2026 23:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

checked_mean returns a rounding residue for deltas that cancel on Python < 3.12 because it uses plain sum()

4 participants