Skip to content

fix(eval): the sign-flip p adds its deltas exactly, and paired_test runs on finite deltas whose sum leaves the float range - #867

Merged
antonwolfy merged 4 commits into
intel:mainfrom
antonwolfy:fix/exact-sign-flip-sums
Nov 17, 2026
Merged

antonwolfy merged 4 commits into
intel:mainfrom
antonwolfy:fix/exact-sign-flip-sums

Conversation

@antonwolfy

@antonwolfy antonwolfy commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Two sign-flip results are wrong on main when a sum leaves what float addition resolves.

Input On main Now
sign_flip_p([1e15, 1, 1, 1, 1, 1]) 0.03125 0.03125
sign_flip_p([1e16, 1, 1, 1, 1, 1]) (also 1e17, 1e20, 1e300) 1.0 0.03125
sign_flip_p([1e17] + [1.0] * 17) (Monte Carlo) 0.0255987 1 / 20001
paired_test([1e308 * (g / 9) for g in [2, 4, 3, 5, 6, 9, 7, 8]], [0.0] * 8) OverflowError: the sum of the magnitudes of the values leaves the float range delta_mean 6.11e307, delta_std 2.72e307, p 0.0078, significant

#816: the five deltas of 1 share a sign, so only the all-same-sign pattern and its mirror reach the observed sum and p is 2 / 2**6. From 1e16 on, every one of the 64 patterns summed to the same float (1e17), so all of them reached the observed sum and p read 1.0. The rounding slack is not the cause (it is capped at EPS, 1e-9, so it is 1e-9 there, not the 133 that n * 2**-52 * sum|x| gives). The fix takes the first option of the issue and computes the sums exactly: each delta is an integer multiple of one power of two (float.as_integer_ratio) and Python adds the integers exactly, for the exact enumeration (each pattern's sum built by doubling) and for the Monte Carlo draws, which are the same draws for a seed. The issue's second option, a refusal with a stated reason, would turn a computable p into no p. math.fsum is not enough: at 1e17 every pattern's sum rounds to the same float again.

#832: the issue's literal input 1e308 * g / 9 is not finite (1e308 * 2 is inf before the division), so its delta_mean inf and missing p are what the module gives a non-finite delta and stay as they are. The deltas the issue means, 1e308 * (g / 9), are finite and their magnitudes sum to 4.9e308; main refused them with an OverflowError from check_sum_range in _describe_deltas, though the mean, the sd and the sign-flip p all fit (_mean and _sample_sd already average and scale exactly, and the exact sums make the p and the interval do too). The refusal now applies only where the sd of the finite deltas is itself beyond the float range ([1.7e308, -1.7e308] * 3): an OverflowError saying the sd of the deltas leaves the float range.

Closes #816

Closes #832

The interval ends of a row such as [1e17, 1, 1, 1, 1, 1] came from the float search of ShiftTest and read 0.0001 for the lower end from 1e16 on (0.9375 at 1e15; the true end is 1.0, where the five deltas of 1 tie). For up to 16 changed deltas that span more than 40 binary orders of magnitude, ShiftTest.p now sums exactly too and the lower end is 1.0; a row of more than 16 such deltas keeps the float search: tracked in #865. repos/compare.py shows baseline_mean / treatment_mean as None for finite scores whose plain sum overflows: tracked in #866.

Changes

  • eval/sign_flip.py: sign_flip_p sums the deltas as exact integers (_as_integers) in both paths, and compares the exact sums with the observed one less the rounding slack of _sum_slack (its formula, capped at EPS), so ordinary-scale rows keep their p. The Monte Carlo draws are unchanged.
  • eval/sign_flip.py: ShiftTest.p calls the same exact enumeration (_enumerated_p) for up to 16 shifted deltas that span more than 40 binary orders of magnitude, so the interval search does not lose the small deltas; the _int_sum alias is gone and the module's sum adds the integers directly.
  • eval/stats.py: _describe_deltas no longer refuses finite deltas for a sum of magnitudes past the range; it refuses only when delta_std is not finite, with the sd of the deltas leaves the float range. arm_significance and interaction_test go through it.
  • eval/interactions.py: interaction_test raises OverflowError when the two finite arm deltas add past the float range (the additive expectation would be inf next to a computed p).
  • report/model_compare.py: a section keeps refusing a sum past the range, naming the run, now by calling check_sum_range where it calls the test (the section's averages are float sums, +inf for these deltas). One blank line is dropped to stay within the file's growth baseline.
  • Tests: test_sign_flip_float_range.py (three tests for sign_flip_p returns 1.0 when one delta is 1e17 or more times the others #816 and one for the lower end of [m, 1, 1, 1, 1, 1]), test_stats_float_range.py (one for paired_test: eight finite deltas whose sum leaves the float range give delta_mean inf, no p and no verdict #832), test_interactions_float_range.py (one for the additive expectation). The tests that swapped sign_flip.sum for float sums lose the swap (the p and the ends add integers), and test_the_ends_are_the_same_whether_python_sums_compensated_or_left_to_right is removed (Test-narrowed). Changed pins: test_the_row_p_places_zero_where_large_deltas_round_p_at_zero_apart has one p, 0.0625, on both ways of summing (sum() left to right gave 0.046875 before); test_the_p_and_ends_of_a_row_near_the_maximum_are_those_of_the_row_scaled still compares both ends of [max, 1, 1, 1, 1, 1]; test_paired_test_raises_when_the_deltas_leave_the_float_range keeps the 1.7e308 row, as [9e307, -9e307] * 3 (sd 9.9e307) now runs (Test-narrowed).
  • docs/benchmarking-and-comparison.md: the p's sums are exact; a significance row is refused only when its sd leaves the range.
  • Changelog fragments (Fixed), one per issue.

Risk

sign_flip_p changes where float sums used to merge patterns: rows with a delta 1e16 or more times the others give the exact p, and the 1e9 + 1/3 row whose sums only tie in exact arithmetic is 0.0625 on every Python (0.046875 on Python 3.11 before). Ordinary-scale rows, the Monte Carlo draws for a seed and the pinned repo compare rows give the same p. The Monte Carlo path does integer sums per resample: 40 deltas take about 0.09 s. Rows whose finite deltas have a sum of magnitudes past the range, which raised OverflowError in paired_test, arm_significance and interaction_test, now return a row; interaction_test still refuses a row whose two arm deltas add past the range. The one new raise is that refusal (docs/benchmarking-and-comparison.md names it). The lower end of a row with a dwarfing delta is 1.0 where it read 0.0001. Open PRs on the same files: #841 only appends pearson to the end of eval/stats.py, #705 rewrites test_the_row_p_places_zero_where_large_deltas_round_p_at_zero_apart in test_repo_compare_interval.py, which this PR also changes (it pins the p of the 1e9 + 1/3 row, 0.0625 on every Python here): whichever merges second resolves that test, and the p it must expect is 0.0625 for both ways of summing; and #667 renames the formatters of report/model_compare.py away from the lines changed here (the call to check_sum_range in _adjusted_significance); apart from that test, none changes the same lines.

Testing

On a Linux review host, at 6778b36 (on main ea458d2):

  • make ci-local: 34 passed, 0 failed, 14 skipped (docker, the pinned Claude Code install, the test-latest and matrix legs, micromamba, the fuzz run, semgrep, the docs deploy). 22657 tests passed, 19 skipped.
  • Merge of main: interactions.py conflicted, where main passes per-row magnitudes to _describe_deltas and _arm_delta and this PR refuses an additive expectation past the float range; both are kept. The float-range, stats, interval, interaction and guard test files pass on the merge (1533 passed), and with the refusal removed from the merged interactions.py test_two_arm_deltas_whose_sum_leaves_the_float_range_raise_instead_of_an_inf_additive_expectation fails.
  • On main's code the new and changed tests fail: 7 failed, 157 passed over test_sign_flip_float_range.py, test_stats_float_range.py, test_repo_compare_interval.py and test_stats.py, for example assert 1.0 == (2 / (2 ** 6)) for [1e16, 1, 1, 1, 1, 1] and the OverflowError above for paired_test: eight finite deltas whose sum leaves the float range give delta_mean inf, no p and no verdict #832.
  • On main's code the tests of the interval's lower end and of the interaction refusal fail too (9 failed, 30 passed over test_sign_flip_float_range.py and test_interactions_float_range.py).
  • Seven mutants, each failing the tests that pin it: float sums in the exact path, a float comparison in the Monte Carlo path, the slack uncapped, no slack, the refusal always on, a plain-sum mean, and the interval search unscaled.
  • Three more mutants: the interaction guard off (test_two_arm_deltas_whose_sum_leaves_the_float_range_raise_instead_of_an_inf_additive_expectation fails), the exact sums of ShiftTest.p off (test_the_lower_end_of_a_row_with_a_dwarfing_delta_is_where_the_small_deltas_stop_rejecting and the max and ones case fail), and the sd refusal worded as the sum of magnitudes (test_paired_test_raises_when_the_deltas_leave_the_float_range fails).
  • Not run: the other Python versions (3.11, 3.12, 3.14), docker steps.

Open questions

None.

Revert

A plain revert restores the float sums (sign_flip_p of a row with a huge delta reads 1.0 again) and the refusal of paired_test for a sum past the range. Nothing is stored.

Checklist

  • Tests added/updated and pytest passes
  • ruff, mypy, and schema checks pass
  • Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
  • Self-reviewed per "Before opening a PR" in AGENTS.md
  • Each new raise/exit and test names the doc, contract or issue it traces to
  • No ad-hoc run artifacts committed
  • New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

…oes not hide the small ones (intel#816)

sign_flip_p added the signed deltas as floats. From a delta of about 1e16 on, the small deltas were absorbed by the large one: all 64 sign patterns of [1e17, 1, 1, 1, 1, 1] summed to the same 1e17, so every pattern reached the observed sum and sign_flip_p was 1.0 instead of 2 / 2**6, the p of the same row at 1e15. The rounding slack is not the cause: it is capped at EPS, 1e-9 there.

The deltas are now integer multiples of one power of two and their sums are exact, for the exact enumeration (each pattern's sum built by doubling) and for the Monte Carlo draws, which are the same draws for a seed. Compensated float sums (math.fsum) are not enough: at 1e17 every pattern's sum rounds to the same float. The slack that lets a pattern tie the observed sum up to float noise (0.1 + 0.2 against 0.3) is kept as it was, in the same units.

Pinned values that change: test_the_row_p_places_zero_where_large_deltas_round_p_at_zero_apart no longer has a p that depends on the Python's sum() (0.0625 on both, 0.046875 before on a left-to-right sum); test_the_p_and_ends_of_a_row_near_the_maximum_are_those_of_the_row_scaled no longer compares the lower end of [max, 1, 1, 1, 1, 1], which the float search of ShiftTest does not resolve once the p stops reading no evidence (Test-narrowed: the end is the product's noise at either scale).

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
…at range (intel#832)

The issue's input, stats.paired_test([1e308 * g / 9 for g in [2, 4, 3, 5, 6, 9, 7, 8]], [0.0] * 8), is not finite: 1e308 * 2 is inf before the division, so all eight deltas are inf and the inf mean with no p is what the module gives a non-finite delta (test_paired_test_keeps_the_meaning_of_a_non_finite_input). With the finite deltas the issue means, 1e308 * (g / 9), main raised OverflowError from check_sum_range in _describe_deltas, because their magnitudes sum to 4.9e308, though the mean (6.1e307), the sd (2.7e307) and the sign-flip p all fit: _mean and _sample_sd already scale or average exactly, and sign_flip_p and the interval search scale (and with the previous commit add exactly).

_describe_deltas now refuses only when the sd of finite deltas is itself past the float range, with the same OverflowError as before, since an inf delta_std next to a finite mean would be the only other result. arm_significance and interaction_test go through _describe_deltas and follow. The model compare section keeps its refusal of a sum past the range, now checked where it calls the test, so its behaviour and message are unchanged.

test_paired_test_raises_when_the_deltas_leave_the_float_range pinned the refusal for [9e307, -9e307] * 3; that row has an sd of 9.9e307 and now runs, so the test keeps the 1.7e308 row, whose sd leaves the range (Test-narrowed: 9e307 is no longer past the range). model_compare.py loses a blank line to stay within its growth baseline.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@exai-bot

exai-bot commented Nov 17, 2026 •

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 12 selected item(s).

CI at 6778b36: 25 passing, 3 skipped.

Review details
  • Commit reviewed: 6778b36 against merge base ea458d2
  • Coverage: 12 files in 3 groups
  • Engine: OpenCodeReview v1.13.5-exai.4.8f73eada, us.openai.gpt-6.1-sol
  • Usage: 354k tokens (224k from cache), 42s
  • Tool calls: 32 (21 file_read, 5 code_search, 2 file_read_diff, 2 local_check_results, 1 ci_status, 1 pr_threads)
  • Sign-flip exact sums and float-range tests: 3 files
  • Paired-test and interaction sum range handling: 6 files
  • Model comparison intervals and documentation: 3 files

@codecov-commenter

codecov-commenter commented Nov 17, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.79%. Comparing base (ea458d2) to head (6778b36).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #867      +/-   ##
==========================================
- Coverage   95.79%   95.79%   -0.01%     
==========================================
  Files         429      429              
  Lines       56821    56841      +20     
  Branches     9783     9787       +4     
==========================================
+ Hits        54433    54451      +18     
- Misses       1485     1487       +2     
  Partials      903      903              
Files with missing lines Coverage Δ
dokimi/dokimi/eval/interactions.py 98.55% <100.00%> (+0.04%) ⬆️
dokimi/dokimi/eval/sign_flip.py 98.88% <100.00%> (-1.12%) ⬇️
dokimi/dokimi/eval/stats.py 97.90% <100.00%> (+<0.01%) ⬆️
dokimi/dokimi/report/model_compare.py 95.22% <100.00%> (+0.01%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at a0ba034. The exact integer sums fix the p of a delta that dwarfs the others, and the sum check is gone from paired_test as promised. Two gaps are left: interaction_test can now return an inf additive expectation, and the lower end of a row whose p this fixes is bisection noise. Both are in the line comments.

What I ran, at a0ba034:

  • make ci-local --offline: all checks pass except the fidelity golden. Its tests passed, and then pytest crashed cleaning up a shared /tmp/pytest-of-* directory, so that failure is local and not from this PR.
  • The new tests on the merge-base: the five [m, 1, 1, 1, 1, 1] / Monte Carlo p tests and the 4.9e308 paired_test row fail there, so they cover the fix. The cancelling ±1e17 test passes on both, as a guard.
  • sign_flip_p on 6,000 random rows (6–12 deltas, from 1 to 1e9, with decimal fractions): the same p as the base on every one.
  • Timing: the exact enumeration got faster (16 deltas: 0.08 s → 0.01 s). Monte Carlo got slower on long rows (400 deltas: 0.46 s → 0.73 s).

Checked, no issues

  • Exact sums: _as_integers puts every delta on one denominator, and the slack (n * sum|ints|) >> 52, capped by the tie width, is _sum_slack in the same units. So a pattern still ties 0.1 + 0.2 against 0.3, and the slack can never swallow a delta of 1.
  • sd refusal: a finite row's sd is at most the sum of its magnitudes (from n = 2 on), so every row with a non-finite sd still raises.
  • model compare: keeping the refusal there is right for now: without it, the section's delta_avg and with_avg (float sums) print +inf for the changelog's row.
  • Changelog and docs: both fragments are under Fixed and name the p values that change. The docs paragraphs match the code.

Comment thread dokimi/dokimi/eval/stats.py Resolved
Comment thread dokimi/dokimi/eval/sign_flip.py Resolved
Comment thread dokimi/dokimi/eval/sign_flip.py Resolved
Comment thread dokimi/dokimi/eval/stats.py Resolved
Comment thread dokimi/dokimi/report/model_compare.py Resolved
Settles the review of intel#867.

interaction_test now raises OverflowError when the two arm deltas are finite but their sum leaves the float range, instead of returning additive_expectation inf next to a computed p.

ShiftTest.p sums up to 16 shifted deltas that span more than 40 binary orders of magnitude exactly (the integer enumeration of sign_flip_p, now shared as _enumerated_p), so the lower end of [1e17, 1, 1, 1, 1, 1] is 1.0 and no longer the 0.0001 a float search read as measured. Rows of more than 16 such deltas keep the float search (intel#865).

sign_flip_p uses the builtin sum on integers directly: the _int_sum alias and the tests' monkeypatches of sign_flip.sum, which no longer change any float sum, are dropped, with the test that compared the ends under three sums.

_describe_deltas names the sd in its refusal, and the comment in model_compare says why the sum check stays.

Test-narrowed: the monkeypatched sum variants of test_an_end_is_bounded_at_deltas_too_large_to_move_by_one and test_the_row_p_places_zero_where_large_deltas_round_p_at_zero_apart, and test_the_ends_are_the_same_whether_python_sums_compensated_or_left_to_right, no longer have a float sum to swap: the p and the ends add integers, exact on every Python.
Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@antonwolfy

Copy link
Copy Markdown
Contributor Author

(commented) Review at a0ba034. The exact integer sums fix the p of a delta that dwarfs the others, and the

@vlad-perevezentsev (#867 (review)): The two gaps of the review are the threads 4599615294 (interaction_test additive expectation) and 4599615310 (lower end of the dwarfing-delta row); both are fixed in d465cf1. The rest of the review lists checks with no issue found.

@antonwolfy

Copy link
Copy Markdown
Contributor Author

Ready for another look at d465cf1; CI is green.

Declined, with the evidence in the reply:

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

I checked d465cf1. Every point of my first round is fixed. interaction_test refuses an additive expectation past the float range, and the lower end of [1e17, 1, 1, 1, 1, 1] is now 1.0, which is right: the five deltas of 1 tie there, so my 0.94 was the float value. Rows of more than 16 such deltas keep the float search, as #865 tracks.

Merge: safe now: it merges cleanly into current main, 5 commits behind, and its tests pass on the merge (main changed model_compare.py since the base). Once it is in, #868 (interactions.py) and #705 (test_repo_compare_interval.py) conflict with it and need a rebase.

What I ran, at d465cf1:

  • make ci-local --offline: 22,343 tests passed, 0 failed. The one FAIL row is schema_compat, for repo_event.v1.json. This PR does not touch schemas: main added that file after the PR's base.
  • On the merge with current main: the float-range, stats, interval, interaction and model compare tests, 324 passed.
  • The new lower-end test and the new interaction test fail on the previous head a0ba034, so both cover their fix.
  • sign_flip_row on 121 random rows of 6–16 deltas, at scales from 1 to 1e6, gives the same p and ends as at a0ba034, in the same time. So the exact path in ShiftTest only changes the rows it is for.

Checked, no issues

  • Exact path in ShiftTest.p: it reuses _enumerated_p with the same slack cap as the float path (the largest tie width).
  • Dropped tests: the removed sum monkeypatches had no float sum left to swap. The integer sums are exact on every Python.

dokimi/dokimi/eval/interactions.py conflicted: main passes per-row magnitudes to _describe_deltas and _arm_delta, the branch refuses an additive expectation past the float range. Both are kept: the calls take main's magnitudes arguments and the branch's OverflowError check follows them.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@antonwolfy

Copy link
Copy Markdown
Contributor Author

(approved) LGTM

@vlad-perevezentsev (#867 (review)): Nothing to change in the code for this approval. main now conflicts with the branch in eval/interactions.py (main passes per-row magnitudes to _describe_deltas and _arm_delta, the PR refuses an additive expectation past the float range); the merge commit 6778b36 keeps both. On the merge the float-range, stats, interval, interaction and guard tests pass (1533 passed), make ci-local gives 0 failed, 34 passed, 14 skipped, and removing the refusal from the merged file fails the interaction test.

@antonwolfy
antonwolfy merged commit a98a0e7 into intel:main Nov 17, 2026
37 of 38 checks passed
@antonwolfy
antonwolfy deleted the fix/exact-sign-flip-sums branch November 17, 2026 19:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

4 participants