Repository navigation
fix(eval): the sign-flip tie width follows the values of each pair - #818
Conversation
…ntel#762) The tie tolerance EPS = 1e-9 was absolute, so a delta vector that varies at a tiny scale (largest delta 1e-300) was all ties: paired_test gave it n_changed 0 and no p next to a large effect size, and repo compare counted every pair as tied. A new helper tie_tolerance(deltas) in eval/sign_flip.py gives the width of a tie for a vector: EPS when its largest finite delta is EPS or more, so every result at score scale is unchanged, and EPS times that largest delta below it. The scale is the vector's, not a pair's, and an all-zero vector keeps EPS, so it has no p. sign_flip_p, ShiftTest, sign_flip_interval and sign_flip_row, the n_changed of _describe_deltas, the up, down, tied and mde of repo compare, its headroom checks and the non-inferiority slack of paired_cluster_difference all take it from that helper. The interval shifts the deltas by theta but keeps the tolerance (and the bisection step and far bound) of the original vector, so p(theta) stays one step function. The ledger's abs(mean) > EPS guard and checklist.TIE_EPSILON compare scores in [0, 1] and keep the absolute width. Two tests that pinned the absolute tie on a vector whose largest delta is 1e-12 now pin the scaled one. EPS stays importable from eval.sign_flip, eval.stats and repos.compare. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 4 selected item(s). CI at Review details
|
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #818 +/- ##
==========================================
- Coverage 95.76% 95.74% -0.02%
==========================================
Files 416 423 +7
Lines 55624 56457 +833
Branches 9553 9717 +164
==========================================
+ Hits 53267 54056 +789
- Misses 1456 1492 +36
- Partials 901 909 +8
🚀 New features to boost your workflow:
|
vlad-perevezentsev
left a comment
There was a problem hiding this comment.
Review at 2228387. The 1e-300 case from #762 is fixed and score-scale results are unchanged, but the tie width now comes from the delta vector itself, so a vector of pure float noise is no longer tied and gets a significant p. Details are in the line comments.
Agreed with @exai-bot: the interval of subnormal deltas is [0, 0] next to a significant p; the pytest.approx asserts accept a zero tie width.
What I ran, at 2228387:
make ci-local --offline: 0 failed, 22 passed, 20 skipped (network, matrix legs); 21,689 tests passed, 72 skipped.- Repros on this head and on the merge-base d92be0c:
paired_testandsign_flip_summaryon0.1 + 0.2vs0.3(8 pairs),headroomon 1.0 vs1 - 1.1e-16,paired_cluster_differenceon noise bounds with margin 0,[5e-10] * 8with and without one1.1e-9, subnormal deltas at 1e-300 to 1e-320 and at5e-324. - A mutant where
tie_tolerancereturns 0.0 belowEPS: the four test files of the change still pass (176 passed).
Checked, no issues
- Score-scale results: a vector whose largest delta is 1e-9 or more keeps
EPSat every site, so existing rows, p values and verdicts do not move. - Interval and p: the shifted deltas keep the original vector's tolerance and the bisection step scales with it; the interval excludes 0 exactly when p < 0.05 down to about 1e-320.
- Changelog and docs: a fragment under Fixed, and the docs page states the new rule.
Settles the review of intel#818: a tie width taken from the delta vector made a vector of float noise significant (`paired_test([0.1 + 0.2] * 8, [0.3] * 8)` gave p 0.0078 and 8 changed), made more evidence in one direction remove a significant result (`[5e-10] * 8` is p 0.0078, with one more delta of 1.1e-9 it was 1.0), and gave deltas of the smallest floats an interval of `[0, 0]` beside a significant p. A pair now ties within `EPS * min(1, max(|treatment|, |baseline|))` (`pair_magnitudes`, `tie_widths`), since float noise scales with the values subtracted. `sign_flip_p`, `ShiftTest` (a width per delta), `sign_flip_interval`, `sign_flip_row`, `paired_test`, `arm_significance` and `sign_flip_summary` take the pairs' magnitudes; deltas alone keep the absolute `EPS`. Values all below 2**-664 are searched multiplied by a power of two, and an end that comes back below the smallest float is that float on its side of 0, so the interval excludes 0 with the p. The `headroom`, `format_compare` and `paired_cluster` changes of the first commit are taken back: they read tolerances off deltas, which are float noise next to the values they come from. The unused `EPS` imports go with them. `format_compare` keeps `EPS` and says its line follows the printed, rounded deltas. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@vlad-perevezentsev (#818 (review)): The review recaps what was run and points to the line comments. Its threads are answered above (4585134951, 4585134956, 4585310754, 4585310766, 4585310782, 4585310789); the "Checked, no issues" items still hold at 464c159 (score-scale results with values of 1 or more are unchanged, the interval excludes 0 exactly when p < 0.05, changelog and docs updated). |
… tied pairs Settles the review point that an unchanged pair of large values loosened the sums of small changed pairs: paired_test([1e-12] * 8 + [1.0], [0.0] * 8 + [1.0]) gave p 1.0, where the eight small pairs alone give 0.0078125. sign_flip_p takes the scale of the slack from the magnitudes of the deltas it sums, and ShiftTest takes it from the tie widths of the deltas that are not ties at that theta, instead of the widest width of every pair. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
|
Ready for another look at Declined, with the evidence in the reply:
Follow-up issues opened:
|
vchamarthi
left a comment
There was a problem hiding this comment.
🔴 Changes needed: 1 must-fix, 1 nit · reviewed aae990b
Next: @antonwolfy: fix the 1 must-fix finding; then push for a re-review. 1 nit optional.
Must-fix
-
dokimi/dokimi/eval/sign_flip.py:286– Eight pairs of 1e-12 beside one tied pair of 1.0 give an interval at8.9e-13, a shift the test rejects.Input:
[1e-12] * 8 + [1.0]against[0.0] * 8 + [1.0]. Head givesci_loandci_hiboth8.888888888888889e-13, the mean of the nine deltas. At that shift the eight changed deltas still give p0.0078, so the end is rejected. Expected:1e-12at both ends, as the eight pairs alone give. Cause:_endreturns its starting point, the mean, without checking that the test accepts it; the tied zero delta pulls the mean off the one accepted shift, which is narrower than the search step.Suggestion: Search from a shift the test accepts, such as the mean of the changed deltas, instead of returning the mean unchecked.
Follow-ups, not blocking
dokimi/dokimi/eval/stats.py:618– Input: eight pairs of1e308 * g / 9(g in 2, 4, 3, 5, 6, 9, 7, 8) against zeros (main has it too).
Verified on aae990b: ✅ changed tests: 9 passed, fail on main · ✅ full suite with main merged in: 22043 passed · ✅ merges onto main · ✅ 12 broken variants of the changed code: 12 caught by the tests · 1 mutation tried · 5 edge inputs tried
9 checks behind it, each with its evidence
| Area | What holds | How |
|---|---|---|
| Float-noise and 1e-300 ties | The nine tests in tests/test_sign_flip_relative_tie.py pass on the head. They pin 0.1 + 0.2 against 0.3 as a tie with no p, and the 1e-300 vector as p 0.0078 with 8 changed and significant True, as its 1e-3 copy gives |
tests on the head |
| Smallest float deltas | Six pairs of 5e-324 against zeros (p 0.03125) give an interval that excludes 0, pinned by test_deltas_of_the_smallest_floats_have_an_interval_that_excludes_zero_with_their_significant_p |
tests on the head, edge input (subnormal) |
| More evidence in one direction | Eight pairs of 5e-10 give p 0.0078, and a ninth delta of 1.1e-9 gives 0.0039, pinned by test_more_evidence_in_the_same_direction_does_not_remove_a_significant_result |
tests on the head |
| Tie scale cap | Dropping the cap at 1 in tie_scale (sign_flip.py:78) fails the changed tests, so they pin the cap |
broke the code, tests caught it |
| Merge | The branch merges onto current main with no conflict | merge onto main |
| Full suite | The full suite and the static checks pass with main merged in | full suite, main merged in |
| Numbers | Int inputs give the same p as floats (0.0078), and a NaN delta gives no p and no verdict | ran it, edge input (numbers) |
| Collections | Empty and single-pair inputs give no test, pair order changes neither p nor the interval, and 200 pairs at score scale get a Monte Carlo p with 171 changed | ran it, edge input (collections) |
| No values given | With no values given a tie is the absolute EPS, so [1e-12] * 8 has no p, and an all-zero vector has no p |
ran it, edge input (none) |
Settles the review of intel#818: eight pairs of 1e-12 beside one tied pair of 1.0 gave an interval at 8.9e-13 at both ends, the mean of the nine deltas, a shift the test rejects. The search returned its starting point unchecked; it now starts from the mean of the changed deltas when the mean of all is rejected, so the ends are 1e-12 as for the eight pairs alone. The docs sentence on tie widths now says a pair of zeros takes the width of the largest value among the pairs. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@vchamarthi (#818 (review)): Fixed in eb1a924. Reproduced: |
Settles the three open threads of the review. The slack on a sum of deltas was the widest tie width among them, so a changed pair just past its width (1.0 + 1.1e-9 against 1.0) made every sign pattern count and turned eight significant pairs of 1e-12 into p 1.0. The slack is now the rounding error of adding the deltas (n * 2**-52 times their absolute sum), capped by the tie width it replaces, in sign_flip_p and ShiftTest.p. The interval search started from the mean of the deltas, or of the changed ones, without checking either was accepted: with every delta changed, a mean that ties the largest delta returned two rejected ends. It now tries the mean, the mean of the changed deltas, then up to 16 of the deltas nearest the median, and keeps the first the test accepts. The mde of a repo compare row read the spread against 1e-9 times the largest value of any pair, so an unchanged pair of large values removed it. It now reads the scale of the changed pairs only. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
|
Ready for another look at Follow-up issues opened:
|
napetrov
left a comment
There was a problem hiding this comment.
Approving at a4ad627. Every earlier must-fix (interval ends, unchanged-pair p, 9th pair, 5e-324 interval, test abs tolerance) holds on head.
Probed: body repros, 3000 untied vectors match main, interval/p agree on 5500 fuzzed vectors; 168 sign-flip tests pass.
Non-blocking: mde prints 0.0 at 1e-12 scale (#817 family); four changelog fragments for one fix; 1 ulp noise above 1 is still a change (pre-existing).
@napetrov (#818 (review)): No change this round. The mde printing 0.0 at a 1e-12 scale is the rounding of sign-flip rows already tracked in #817. The four changelog fragments each describe one commit's change to the same tie-width rule and are published; they are left as they are. The 1 ulp noise above 1 being treated as a change is how main behaved before this PR and is outside it. |
…heir values (#868) * fix(eval): interaction_test reads contrasts of tiny-scale scores by their values (#821) interaction_test and interaction_significance passed the per-question contrasts AB - A - B + baseline, and the three arm deltas of the row, to the sign-flip description without the values they come from, so every tie was within the absolute 1e-9. Eight evaluations with a+b scores g * 1e-300 and 0 elsewhere gave p None, n_changed 0 and significant None, where the same scores at 1e-3 give p 0.0078, n_changed 8 and significant True. The contrast of a row now passes the largest magnitude of its four scores, and each arm delta the larger magnitude of its pair against the baseline, as paired_test does since #818. A tie is within 1e-9 times that value when it is below 1, and 1e-9 otherwise, so rows at ordinary score scales are unchanged. The 1e-300 rows read as their 1e-3 copies. Signed-off-by: Anton Volkov <antonwolfy@gmail.com> * fix(eval): interaction tie-width docs and a test of the contrast's scale (#868) Settles the review of #868: the tie-rule example in benchmarking-and-comparison.md named a contrast of 1e-12 that scores of 1e-300 cannot produce, and now uses AB = 1e-300 with A, B and baseline zero. A new test row where `a` is the largest score (contrast -7e-13, a tie at 1e-12) fails when the contrast's scale is taken from `ab` and `baseline` only. The `interaction_test` docstring states the tie width of the contrast and of each arm delta, and `_interaction_contrast` takes the ab, a and b columns once for both `_deltas` and `pair_magnitudes`. Signed-off-by: Anton Volkov <antonwolfy@gmail.com> --------- Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Summary
#762: the sign-flip tie tolerance
EPS = 1e-9is absolute, so a delta vector that varies at a tiny scale is all ties. Onmain,stats.paired_test([2e-300, 4e-300, 3e-300, 5e-300, 6e-300, 9e-300, 7e-300, 8e-300], [1e-300] * 8)gives:pn_changedsignificantcohens_dNoneNonelarge)Truerepo comparecounts the same pairs as0/0/8up/down/tied and gives nopormde. The[0.0, 1e-12, -1e-12]and[0.0, 0.0, 1e-12]vectors likewise have noponmain.A pair is now a tie when its delta is within
EPS * min(1, m)of 0, wheremis the larger of the pair's two values in magnitude (tie_widthsineval/sign_flip.py):EPS(1e-9) for values of 1 or more, and less below that, because float noise int - bscales with the values subtracted and not with the delta. Each pair keeps its own width whatever the other deltas are, so more evidence in one direction cannot widen the tie of the rest. A pair whose values are both 0 takes the scale of the vector's largest value, an all-zero vector still has nop, and a caller that passes deltas alone (no values) keeps the absoluteEPS. After the change the 1e-300 vector getsp0.0078,n_changed8 andsignificantTrue, as its 1e-3 copy does, andpaired_test([0.1 + 0.2] * 8, [0.3] * 8)stays a tie (pNone,n_changed0).Closes #762
The tolerance of
eval/checklist.py(TIE_EPSILON) is left as it is, see Changes. The printed mean and interval of a row below 5e-5 are still rounded to 0.0001, which these rows now show; that is tracked in #817.interaction_testpasses contrasts without values and so keeps the absolute width; that is tracked in #821.Changes
eval/sign_flip.py: newpair_magnitudes(treatment, baseline),tie_scale(magnitudes)andtie_widths(magnitudes, n).sign_flip_p,sign_flip_intervalandsign_flip_row(n_changed) takemagnitudes=;ShiftTest'sepsis a width per delta (a float still works). The interval shifts the deltas by theta but keeps the widths of the original pairs, and scales the bisection step (TOL) and the far bound withtie_scale, sop(theta)is one step function and the interval still excludes 0 exactly whenp < ALPHA. When all the values are below 2**-664 the search runs on the deltas multiplied by a power of two (as huge ones are already divided), and an end that comes back below the smallest float is that float on its side of 0, so the interval of[5e-324] * 6 + [0.0] * 7is no longer[0, 0]besidep0.03125.eval/sign_flip.py: the slack on a sum of deltas follows the pairs summed, not the largest value of any pair.sign_flip_ptakes the scale of its slack from the magnitudes of the changed pairs, andShiftTest.ptakes it from the widest tie width among the deltas that are not ties at that shift. That width is only the cap: the slack is the rounding error of adding the deltas summed (_sum_slack,n * 2**-52times their absolute sum), so a changed pair just past its own tie width does not make every sign pattern count.paired_test([1e-12] * 8 + [1.0 + 1.1e-9], [0.0] * 8 + [1.0])gavep1.0 and now gives 2/2**9.paired_test([1e-12] * 8 + [1.0], [0.0] * 8 + [1.0])gavep1.0 (the unchanged 1.0 pair made the slack 1e-9, so every sign pattern counted) and now gives 0.0078125, as the eight small pairs alone do; its interval excludes 0 (it was[0, 8.9e-13]).eval/sign_flip.py: the interval search starts from a shift the test accepts. When the mean of all the deltas is a shift the test rejects (a tied delta pulls it off the one accepted shift, narrower than the search step),_endsstarts from the mean of the changed deltas, and when that is rejected too (every delta changed, the mean ties the largest) from the first of up to 16 of the deltas, nearest the median first, that the test accepts (the mean again if none is).[1e-12] * 8 + [0.0017 + 1.8e-12]against[0.0] * 8 + [0.0017]returned two shifts of p 0.0078 and now gives1e-12at both ends, whereShiftTest.pis None. The ends of[1e-12] * 8 + [1.0]against[0.0] * 8 + [1.0]are1e-12at both ends, as for the eight pairs alone (they were8.9e-13, rejected at p 0.0078).eval/stats.py:paired_testandarm_significancepass the pairs' magnitudes to_describe_deltas, which uses them forp, the interval andn_changed;interactionspasses none and keeps the absolute width. The unusedEPSimport is dropped.repos/compare.py: theup,down,tiedcounts and thesdtest ofmdeuse each pair's width, from the pairs' baseline and treatment; thesdtest ofmdereads the scale of the changed pairs only, so an unchanged pair of large values (1.0against1.0) beside eight changed pairs of about 1e-12 no longer removes themde.headroomand the "changed" line offormat_compareare as onmain:headroomcompares values withtopand a delta of a pair's own values (1.0 against1 - 1.1e-16is stillceiling), and the "changed" line reads the deltas the row prints, rounded to 6 places, so it keepsEPS(a comment says so). The.reportimport line and therefuse_mixed_executionsline are not touched.eval/paired_cluster.pyis as onmain: its slack is float noise in a bound next to-margin, not a delta of a pair.repos/ledger.py(abs(mean) > EPS) andeval/checklist.py(TIE_EPSILON) keep the absolute width: the ledger judges gate and score rows, and the checklist compares coverage fractions of whole points (one point of three is 0.33), both bounded scores where 1e-9 is float noise.tests/test_sign_flip_relative_tie.py(with a test that an unchanged pair of large values leaves the p, the significance and the interval of eight small changed pairs as they are alone);test_repo_compare.pyandtest_repo_compare_interval.pyare as onmain, since a delta vector alone keeps the absolute width.docs/benchmarking-and-comparison.mdstates the tie rule with the0.1 + 0.2and 1e-12 examples.Risk
A pair whose larger value is below 1 has a tie width of 1e-9 times that value, narrower than the 1e-9 it had. A delta of such a pair that lies between the two widths (for example 0.5 against 0.5 + 7e-10, or 0 against 1e-12) was a tie and is now a change, so
paired_test,arm_significanceandrepo comparerows with such deltas getn_changed,up/downcounts, apand an interval where they had none, and a row with some of them can move itsp. Float noise in the values (0.1 + 0.2against0.3, a delta of 5.5e-17) stays a tie. Rows whose pairs all have a value of 1 or more, the JSON shapes and the schemas are unchanged;headroom, the "changed" line, the paired-cluster check, the ledger and the checklist keep the 1e-9 they had. The roundedci_lo/ci_hiof a 1e-300 row are still printed at 4 places, as±0.0001, which is the existing display rule and is not changed here. Open PR #628 edits two other lines ofrepos/compare.py(the.reportimport andrefuse_mixed_executions), and open PR #705 edits other tests oftests/test_repo_compare_interval.py;git merge-treeof this branch with #705 is clean, and #628's lines are not touched here.Open questions
The width could follow the vector (
EPS * min(1, largest delta)) or each pair's larger value in magnitude. The reviewer's per-pair rule is taken: a delta is a tie when it is withinEPS * min(1, m)of 0, withmthe larger of the pair's two values. The discontinuity of the vector rule no longer applies: the width of a pair does not depend on the other deltas, sopaired_test([5e-10] * 8, [0.0] * 8)keepsp0.0078 and a ninth delta of 1.1e-9 does not turn the eight into ties (p0.0039).Testing
On a Linux review host, at
a4ad627(onmaind92be0c):make ci-local: 33 passed, 0 failed, 14 skipped (docker smoke, the matrix legs on other Pythons, deploy and PR-only jobs, conda-forge, fuzz, semgrep, the pinned Claude Code install). 21748 tests passed, 19 skipped; changed-line coverage 100% (68 lines).paired_test([1e-12] * 8 + [1.0 + 1.1e-9], [0.0] * 8 + [1.0], rounded=False)givesp0.00390625 (2/2**9) andsignificantTrue here, andp1.0 before the slack was rounding error; the interval of[1e-12] * 8 + [0.0017 + 1.8e-12]against[0.0] * 8 + [0.0017]is1e-12at both ends withShiftTest.pNone there, and was1.089e-12at both with p 0.0078125;sign_flip_summaryof eight pairs of 2e-12 to 9e-12 against zeros plus a1.0/1.0pair gives anmde(it was None).test_a_changed_pair_the_size_of_its_tie_width_does_not_make_every_sign_pattern_count;_startreturning the mean failstest_the_ends_of_the_interval_are_shifts_the_test_accepts_when_the_mean_is_not_oneandtest_an_unchanged_pair_of_large_values_does_not_pull_the_interval_off_the_shift_the_small_pairs_accept; themdespread read at the scale of every pair failstest_the_mde_of_changed_pairs_does_not_depend_on_an_unchanged_pair_of_large_values. The three new tests also fail on the code ofeb1a924.paired_test([1e-12] * 8 + [1.0], [0.0] * 8 + [1.0])givesp0.0078125 andsignificantTrue here, andp1.0 before the slack followed the pairs summed; the interval of those deltas is(1e-12, 1e-12), which excludes 0 and is the interval of the eight small pairs alone; it was(8.9e-13, 8.9e-13), a shift the test rejects, before the search started from an accepted shift._endsdisabled failstest_an_unchanged_pair_of_large_values_does_not_pull_the_interval_off_the_shift_the_small_pairs_accept; the width of a pair of zeros taken as 1.0 failstest_a_pair_of_zeros_takes_the_width_of_the_largest_value_among_the_pairs, and two other tests of the file.sign_flip_ptaken from every pair again, andShiftTest's slack taken from every width again, each fail the new test.paired_teston the 1e-300 vector givesp0.0078,n_changed8,significantTrue here andNone, 0,Noneonmain.paired_test([0.1 + 0.2] * 8, [0.3] * 8)givespNone andn_changed0, andsign_flip_summaryon those pairs gives 0/0/8 up/down/tied.paired_test([5e-10] * 8, [0.0] * 8)givesp0.0078 and with a ninth delta of 1.1e-9 it gives 0.0039.sign_flip_rowon[5e-324] * 6 + [0.0] * 7against zeros givesp0.03125 and an interval of5e-324to5e-324that excludes 0. The interval excludes 0 exactly whenp < 0.05on a 1e-300 vector, a 1e-12 vector and a vector of six 1e-3 deltas and one 1e-12 delta against 0 (n_changed7: the 1e-12 is a change at its own scale).test_sign_flip_relative_tie.py: the width absolute for every pair, the width 0,ShiftTestwith one width for all deltas, the end below the smallest float left at 0, the power-of-two search for tiny values removed, themdetest absolute,paired_testwithout the magnitudes,tiedabsolute,n_changedabsolute in_describe_deltasand insign_flip_row.Revert
A plain revert restores the absolute 1e-9 tie. Nothing is stored.
Checklist
pytestpassesruff,mypy, and schema checks passAGENTS.mdraise/exit and test names the doc, contract or issue it traces to.github/.licenserc.yaml)