Repository navigation
fix(eval): the sign-flip p adds its deltas exactly, and paired_test runs on finite deltas whose sum leaves the float range - #867
Conversation
…oes not hide the small ones (intel#816) sign_flip_p added the signed deltas as floats. From a delta of about 1e16 on, the small deltas were absorbed by the large one: all 64 sign patterns of [1e17, 1, 1, 1, 1, 1] summed to the same 1e17, so every pattern reached the observed sum and sign_flip_p was 1.0 instead of 2 / 2**6, the p of the same row at 1e15. The rounding slack is not the cause: it is capped at EPS, 1e-9 there. The deltas are now integer multiples of one power of two and their sums are exact, for the exact enumeration (each pattern's sum built by doubling) and for the Monte Carlo draws, which are the same draws for a seed. Compensated float sums (math.fsum) are not enough: at 1e17 every pattern's sum rounds to the same float. The slack that lets a pattern tie the observed sum up to float noise (0.1 + 0.2 against 0.3) is kept as it was, in the same units. Pinned values that change: test_the_row_p_places_zero_where_large_deltas_round_p_at_zero_apart no longer has a p that depends on the Python's sum() (0.0625 on both, 0.046875 before on a left-to-right sum); test_the_p_and_ends_of_a_row_near_the_maximum_are_those_of_the_row_scaled no longer compares the lower end of [max, 1, 1, 1, 1, 1], which the float search of ShiftTest does not resolve once the p stops reading no evidence (Test-narrowed: the end is the product's noise at either scale). Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
…at range (intel#832) The issue's input, stats.paired_test([1e308 * g / 9 for g in [2, 4, 3, 5, 6, 9, 7, 8]], [0.0] * 8), is not finite: 1e308 * 2 is inf before the division, so all eight deltas are inf and the inf mean with no p is what the module gives a non-finite delta (test_paired_test_keeps_the_meaning_of_a_non_finite_input). With the finite deltas the issue means, 1e308 * (g / 9), main raised OverflowError from check_sum_range in _describe_deltas, because their magnitudes sum to 4.9e308, though the mean (6.1e307), the sd (2.7e307) and the sign-flip p all fit: _mean and _sample_sd already scale or average exactly, and sign_flip_p and the interval search scale (and with the previous commit add exactly). _describe_deltas now refuses only when the sd of finite deltas is itself past the float range, with the same OverflowError as before, since an inf delta_std next to a finite mean would be the only other result. arm_significance and interaction_test go through _describe_deltas and follow. The model compare section keeps its refusal of a sum past the range, now checked where it calls the test, so its behaviour and message are unchanged. test_paired_test_raises_when_the_deltas_leave_the_float_range pinned the refusal for [9e307, -9e307] * 3; that row has an sd of 9.9e307 and now runs, so the test keeps the 1.7e308 row, whose sd leaves the range (Test-narrowed: 9e307 is no longer past the range). model_compare.py loses a blank line to stay within its growth baseline. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 12 selected item(s). CI at Review details
|
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #867 +/- ##
==========================================
- Coverage 95.79% 95.79% -0.01%
==========================================
Files 429 429
Lines 56821 56841 +20
Branches 9783 9787 +4
==========================================
+ Hits 54433 54451 +18
- Misses 1485 1487 +2
Partials 903 903
🚀 New features to boost your workflow:
|
vlad-perevezentsev
left a comment
There was a problem hiding this comment.
Review at a0ba034. The exact integer sums fix the p of a delta that dwarfs the others, and the sum check is gone from paired_test as promised. Two gaps are left: interaction_test can now return an inf additive expectation, and the lower end of a row whose p this fixes is bisection noise. Both are in the line comments.
What I ran, at a0ba034:
make ci-local --offline: all checks pass except the fidelity golden. Its tests passed, and then pytest crashed cleaning up a shared/tmp/pytest-of-*directory, so that failure is local and not from this PR.- The new tests on the merge-base: the five
[m, 1, 1, 1, 1, 1]/ Monte Carlo p tests and the 4.9e308paired_testrow fail there, so they cover the fix. The cancelling±1e17test passes on both, as a guard. sign_flip_pon 6,000 random rows (6–12 deltas, from 1 to 1e9, with decimal fractions): the same p as the base on every one.- Timing: the exact enumeration got faster (16 deltas: 0.08 s → 0.01 s). Monte Carlo got slower on long rows (400 deltas: 0.46 s → 0.73 s).
Checked, no issues
- Exact sums:
_as_integersputs every delta on one denominator, and the slack(n * sum|ints|) >> 52, capped by the tie width, is_sum_slackin the same units. So a pattern still ties 0.1 + 0.2 against 0.3, and the slack can never swallow a delta of 1. - sd refusal: a finite row's sd is at most the sum of its magnitudes (from n = 2 on), so every row with a non-finite sd still raises.
- model compare: keeping the refusal there is right for now: without it, the section's
delta_avgandwith_avg(float sums) print+inffor the changelog's row. - Changelog and docs: both fragments are under Fixed and name the p values that change. The docs paragraphs match the code.
Settles the review of intel#867. interaction_test now raises OverflowError when the two arm deltas are finite but their sum leaves the float range, instead of returning additive_expectation inf next to a computed p. ShiftTest.p sums up to 16 shifted deltas that span more than 40 binary orders of magnitude exactly (the integer enumeration of sign_flip_p, now shared as _enumerated_p), so the lower end of [1e17, 1, 1, 1, 1, 1] is 1.0 and no longer the 0.0001 a float search read as measured. Rows of more than 16 such deltas keep the float search (intel#865). sign_flip_p uses the builtin sum on integers directly: the _int_sum alias and the tests' monkeypatches of sign_flip.sum, which no longer change any float sum, are dropped, with the test that compared the ends under three sums. _describe_deltas names the sd in its refusal, and the comment in model_compare says why the sum check stays. Test-narrowed: the monkeypatched sum variants of test_an_end_is_bounded_at_deltas_too_large_to_move_by_one and test_the_row_p_places_zero_where_large_deltas_round_p_at_zero_apart, and test_the_ends_are_the_same_whether_python_sums_compensated_or_left_to_right, no longer have a float sum to swap: the p and the ends add integers, exact on every Python. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@vlad-perevezentsev (#867 (review)): The two gaps of the review are the threads 4599615294 ( |
|
Ready for another look at Declined, with the evidence in the reply:
|
vlad-perevezentsev
left a comment
There was a problem hiding this comment.
LGTM
I checked d465cf1. Every point of my first round is fixed. interaction_test refuses an additive expectation past the float range, and the lower end of [1e17, 1, 1, 1, 1, 1] is now 1.0, which is right: the five deltas of 1 tie there, so my 0.94 was the float value. Rows of more than 16 such deltas keep the float search, as #865 tracks.
Merge: safe now: it merges cleanly into current main, 5 commits behind, and its tests pass on the merge (main changed model_compare.py since the base). Once it is in, #868 (interactions.py) and #705 (test_repo_compare_interval.py) conflict with it and need a rebase.
What I ran, at d465cf1:
make ci-local --offline: 22,343 tests passed, 0 failed. The one FAIL row isschema_compat, forrepo_event.v1.json. This PR does not touch schemas:mainadded that file after the PR's base.- On the merge with current
main: the float-range, stats, interval, interaction and model compare tests, 324 passed. - The new lower-end test and the new interaction test fail on the previous head a0ba034, so both cover their fix.
sign_flip_rowon 121 random rows of 6–16 deltas, at scales from 1 to 1e6, gives the same p and ends as at a0ba034, in the same time. So the exact path inShiftTestonly changes the rows it is for.
Checked, no issues
- Exact path in
ShiftTest.p: it reuses_enumerated_pwith the same slack cap as the float path (the largest tie width). - Dropped tests: the removed
summonkeypatches had no float sum left to swap. The integer sums are exact on every Python.
dokimi/dokimi/eval/interactions.py conflicted: main passes per-row magnitudes to _describe_deltas and _arm_delta, the branch refuses an additive expectation past the float range. Both are kept: the calls take main's magnitudes arguments and the branch's OverflowError check follows them. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@vlad-perevezentsev (#867 (review)): Nothing to change in the code for this approval. |
Summary
Two sign-flip results are wrong on
mainwhen a sum leaves what float addition resolves.mainsign_flip_p([1e15, 1, 1, 1, 1, 1])sign_flip_p([1e16, 1, 1, 1, 1, 1])(also1e17,1e20,1e300)sign_flip_p([1e17] + [1.0] * 17)(Monte Carlo)paired_test([1e308 * (g / 9) for g in [2, 4, 3, 5, 6, 9, 7, 8]], [0.0] * 8)OverflowError: the sum of the magnitudes of the values leaves the float rangedelta_mean6.11e307,delta_std2.72e307,p0.0078, significant#816: the five deltas of 1 share a sign, so only the all-same-sign pattern and its mirror reach the observed sum and p is
2 / 2**6. From 1e16 on, every one of the 64 patterns summed to the same float (1e17), so all of them reached the observed sum and p read 1.0. The rounding slack is not the cause (it is capped atEPS, 1e-9, so it is 1e-9 there, not the 133 thatn * 2**-52 * sum|x|gives). The fix takes the first option of the issue and computes the sums exactly: each delta is an integer multiple of one power of two (float.as_integer_ratio) and Python adds the integers exactly, for the exact enumeration (each pattern's sum built by doubling) and for the Monte Carlo draws, which are the same draws for a seed. The issue's second option, a refusal with a stated reason, would turn a computable p into no p.math.fsumis not enough: at 1e17 every pattern's sum rounds to the same float again.#832: the issue's literal input
1e308 * g / 9is not finite (1e308 * 2isinfbefore the division), so itsdelta_meaninf and missing p are what the module gives a non-finite delta and stay as they are. The deltas the issue means,1e308 * (g / 9), are finite and their magnitudes sum to 4.9e308;mainrefused them with anOverflowErrorfromcheck_sum_rangein_describe_deltas, though the mean, the sd and the sign-flip p all fit (_meanand_sample_sdalready average and scale exactly, and the exact sums make the p and the interval do too). The refusal now applies only where the sd of the finite deltas is itself beyond the float range ([1.7e308, -1.7e308] * 3): anOverflowErrorsayingthe sd of the deltas leaves the float range.Closes #816
Closes #832
The interval ends of a row such as
[1e17, 1, 1, 1, 1, 1]came from the float search ofShiftTestand read 0.0001 for the lower end from 1e16 on (0.9375 at 1e15; the true end is 1.0, where the five deltas of 1 tie). For up to 16 changed deltas that span more than 40 binary orders of magnitude,ShiftTest.pnow sums exactly too and the lower end is 1.0; a row of more than 16 such deltas keeps the float search: tracked in #865.repos/compare.pyshowsbaseline_mean/treatment_meanasNonefor finite scores whose plain sum overflows: tracked in #866.Changes
eval/sign_flip.py:sign_flip_psums the deltas as exact integers (_as_integers) in both paths, and compares the exact sums with the observed one less the rounding slack of_sum_slack(its formula, capped atEPS), so ordinary-scale rows keep their p. The Monte Carlo draws are unchanged.eval/sign_flip.py:ShiftTest.pcalls the same exact enumeration (_enumerated_p) for up to 16 shifted deltas that span more than 40 binary orders of magnitude, so the interval search does not lose the small deltas; the_int_sumalias is gone and the module'ssumadds the integers directly.eval/stats.py:_describe_deltasno longer refuses finite deltas for a sum of magnitudes past the range; it refuses only whendelta_stdis not finite, withthe sd of the deltas leaves the float range.arm_significanceandinteraction_testgo through it.eval/interactions.py:interaction_testraisesOverflowErrorwhen the two finite arm deltas add past the float range (the additive expectation would beinfnext to a computed p).report/model_compare.py: a section keeps refusing a sum past the range, naming the run, now by callingcheck_sum_rangewhere it calls the test (the section's averages are float sums,+inffor these deltas). One blank line is dropped to stay within the file's growth baseline.test_sign_flip_float_range.py(three tests for sign_flip_p returns 1.0 when one delta is 1e17 or more times the others #816 and one for the lower end of[m, 1, 1, 1, 1, 1]),test_stats_float_range.py(one for paired_test: eight finite deltas whose sum leaves the float range give delta_mean inf, no p and no verdict #832),test_interactions_float_range.py(one for the additive expectation). The tests that swappedsign_flip.sumfor float sums lose the swap (the p and the ends add integers), andtest_the_ends_are_the_same_whether_python_sums_compensated_or_left_to_rightis removed (Test-narrowed). Changed pins:test_the_row_p_places_zero_where_large_deltas_round_p_at_zero_aparthas one p, 0.0625, on both ways of summing (sum()left to right gave 0.046875 before);test_the_p_and_ends_of_a_row_near_the_maximum_are_those_of_the_row_scaledstill compares both ends of[max, 1, 1, 1, 1, 1];test_paired_test_raises_when_the_deltas_leave_the_float_rangekeeps the 1.7e308 row, as[9e307, -9e307] * 3(sd 9.9e307) now runs (Test-narrowed).docs/benchmarking-and-comparison.md: the p's sums are exact; a significance row is refused only when its sd leaves the range.Risk
sign_flip_pchanges where float sums used to merge patterns: rows with a delta 1e16 or more times the others give the exact p, and the1e9 + 1/3row whose sums only tie in exact arithmetic is 0.0625 on every Python (0.046875 on Python 3.11 before). Ordinary-scale rows, the Monte Carlo draws for a seed and the pinnedrepo comparerows give the same p. The Monte Carlo path does integer sums per resample: 40 deltas take about 0.09 s. Rows whose finite deltas have a sum of magnitudes past the range, which raisedOverflowErrorinpaired_test,arm_significanceandinteraction_test, now return a row;interaction_teststill refuses a row whose two arm deltas add past the range. The one newraiseis that refusal (docs/benchmarking-and-comparison.md names it). The lower end of a row with a dwarfing delta is 1.0 where it read 0.0001. Open PRs on the same files: #841 only appendspearsonto the end ofeval/stats.py, #705 rewritestest_the_row_p_places_zero_where_large_deltas_round_p_at_zero_apartintest_repo_compare_interval.py, which this PR also changes (it pins the p of the1e9 + 1/3row, 0.0625 on every Python here): whichever merges second resolves that test, and the p it must expect is 0.0625 for both ways of summing; and #667 renames the formatters ofreport/model_compare.pyaway from the lines changed here (the call tocheck_sum_rangein_adjusted_significance); apart from that test, none changes the same lines.Testing
On a Linux review host, at
6778b36(onmainea458d2):make ci-local: 34 passed, 0 failed, 14 skipped (docker, the pinned Claude Code install, thetest-latestand matrix legs, micromamba, the fuzz run, semgrep, the docs deploy). 22657 tests passed, 19 skipped.main:interactions.pyconflicted, wheremainpasses per-row magnitudes to_describe_deltasand_arm_deltaand this PR refuses an additive expectation past the float range; both are kept. The float-range, stats, interval, interaction and guard test files pass on the merge (1533 passed), and with the refusal removed from the mergedinteractions.pytest_two_arm_deltas_whose_sum_leaves_the_float_range_raise_instead_of_an_inf_additive_expectationfails.main's code the new and changed tests fail: 7 failed, 157 passed overtest_sign_flip_float_range.py,test_stats_float_range.py,test_repo_compare_interval.pyandtest_stats.py, for exampleassert 1.0 == (2 / (2 ** 6))for[1e16, 1, 1, 1, 1, 1]and theOverflowErrorabove for paired_test: eight finite deltas whose sum leaves the float range give delta_mean inf, no p and no verdict #832.main's code the tests of the interval's lower end and of the interaction refusal fail too (9 failed, 30 passed overtest_sign_flip_float_range.pyandtest_interactions_float_range.py).test_two_arm_deltas_whose_sum_leaves_the_float_range_raise_instead_of_an_inf_additive_expectationfails), the exact sums ofShiftTest.poff (test_the_lower_end_of_a_row_with_a_dwarfing_delta_is_where_the_small_deltas_stop_rejectingand themax and onescase fail), and the sd refusal worded as the sum of magnitudes (test_paired_test_raises_when_the_deltas_leave_the_float_rangefails).Open questions
None.
Revert
A plain revert restores the float sums (
sign_flip_pof a row with a huge delta reads 1.0 again) and the refusal ofpaired_testfor a sum past the range. Nothing is stored.Checklist
pytestpassesruff,mypy, and schema checks passAGENTS.mdraise/exit and test names the doc, contract or issue it traces to.github/.licenserc.yaml)