Skip to content

fix(eval): the sign-flip tie width follows the values of each pair - #818

Merged
antonwolfy merged 5 commits into
intel:mainfrom
antonwolfy:fix/sign-flip-relative-tie
Nov 17, 2026
Merged

antonwolfy merged 5 commits into
intel:mainfrom
antonwolfy:fix/sign-flip-relative-tie

Conversation

@antonwolfy

@antonwolfy antonwolfy commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

#762: the sign-flip tie tolerance EPS = 1e-9 is absolute, so a delta vector that varies at a tiny scale is all ties. On main, stats.paired_test([2e-300, 4e-300, 3e-300, 5e-300, 6e-300, 9e-300, 7e-300, 8e-300], [1e-300] * 8) gives:

deltas p n_changed significant cohens_d
1e-300 scale None 0 None 1.837 (large)
the same at 1e-3 scale 0.0078 8 True 1.837

repo compare counts the same pairs as 0/0/8 up/down/tied and gives no p or mde. The [0.0, 1e-12, -1e-12] and [0.0, 0.0, 1e-12] vectors likewise have no p on main.

A pair is now a tie when its delta is within EPS * min(1, m) of 0, where m is the larger of the pair's two values in magnitude (tie_widths in eval/sign_flip.py): EPS (1e-9) for values of 1 or more, and less below that, because float noise in t - b scales with the values subtracted and not with the delta. Each pair keeps its own width whatever the other deltas are, so more evidence in one direction cannot widen the tie of the rest. A pair whose values are both 0 takes the scale of the vector's largest value, an all-zero vector still has no p, and a caller that passes deltas alone (no values) keeps the absolute EPS. After the change the 1e-300 vector gets p 0.0078, n_changed 8 and significant True, as its 1e-3 copy does, and paired_test([0.1 + 0.2] * 8, [0.3] * 8) stays a tie (p None, n_changed 0).

Closes #762

The tolerance of eval/checklist.py (TIE_EPSILON) is left as it is, see Changes. The printed mean and interval of a row below 5e-5 are still rounded to 0.0001, which these rows now show; that is tracked in #817. interaction_test passes contrasts without values and so keeps the absolute width; that is tracked in #821.

Changes

  • eval/sign_flip.py: new pair_magnitudes(treatment, baseline), tie_scale(magnitudes) and tie_widths(magnitudes, n). sign_flip_p, sign_flip_interval and sign_flip_row (n_changed) take magnitudes=; ShiftTest's eps is a width per delta (a float still works). The interval shifts the deltas by theta but keeps the widths of the original pairs, and scales the bisection step (TOL) and the far bound with tie_scale, so p(theta) is one step function and the interval still excludes 0 exactly when p < ALPHA. When all the values are below 2**-664 the search runs on the deltas multiplied by a power of two (as huge ones are already divided), and an end that comes back below the smallest float is that float on its side of 0, so the interval of [5e-324] * 6 + [0.0] * 7 is no longer [0, 0] beside p 0.03125.
  • eval/sign_flip.py: the slack on a sum of deltas follows the pairs summed, not the largest value of any pair. sign_flip_p takes the scale of its slack from the magnitudes of the changed pairs, and ShiftTest.p takes it from the widest tie width among the deltas that are not ties at that shift. That width is only the cap: the slack is the rounding error of adding the deltas summed (_sum_slack, n * 2**-52 times their absolute sum), so a changed pair just past its own tie width does not make every sign pattern count. paired_test([1e-12] * 8 + [1.0 + 1.1e-9], [0.0] * 8 + [1.0]) gave p 1.0 and now gives 2/2**9. paired_test([1e-12] * 8 + [1.0], [0.0] * 8 + [1.0]) gave p 1.0 (the unchanged 1.0 pair made the slack 1e-9, so every sign pattern counted) and now gives 0.0078125, as the eight small pairs alone do; its interval excludes 0 (it was [0, 8.9e-13]).
  • eval/sign_flip.py: the interval search starts from a shift the test accepts. When the mean of all the deltas is a shift the test rejects (a tied delta pulls it off the one accepted shift, narrower than the search step), _ends starts from the mean of the changed deltas, and when that is rejected too (every delta changed, the mean ties the largest) from the first of up to 16 of the deltas, nearest the median first, that the test accepts (the mean again if none is). [1e-12] * 8 + [0.0017 + 1.8e-12] against [0.0] * 8 + [0.0017] returned two shifts of p 0.0078 and now gives 1e-12 at both ends, where ShiftTest.p is None. The ends of [1e-12] * 8 + [1.0] against [0.0] * 8 + [1.0] are 1e-12 at both ends, as for the eight pairs alone (they were 8.9e-13, rejected at p 0.0078).
  • Docs: the tie-width sentence says a pair of zeros takes the width of the largest value among the pairs (at most 1), the one exception to each pair keeping its own width.
  • eval/stats.py: paired_test and arm_significance pass the pairs' magnitudes to _describe_deltas, which uses them for p, the interval and n_changed; interactions passes none and keeps the absolute width. The unused EPS import is dropped.
  • repos/compare.py: the up, down, tied counts and the sd test of mde use each pair's width, from the pairs' baseline and treatment; the sd test of mde reads the scale of the changed pairs only, so an unchanged pair of large values (1.0 against 1.0) beside eight changed pairs of about 1e-12 no longer removes the mde. headroom and the "changed" line of format_compare are as on main: headroom compares values with top and a delta of a pair's own values (1.0 against 1 - 1.1e-16 is still ceiling), and the "changed" line reads the deltas the row prints, rounded to 6 places, so it keeps EPS (a comment says so). The .report import line and the refuse_mixed_executions line are not touched.
  • eval/paired_cluster.py is as on main: its slack is float noise in a bound next to -margin, not a delta of a pair.
  • repos/ledger.py (abs(mean) > EPS) and eval/checklist.py (TIE_EPSILON) keep the absolute width: the ledger judges gate and score rows, and the checklist compares coverage fractions of whole points (one point of three is 0.33), both bounded scores where 1e-9 is float noise.
  • Tests: new tests/test_sign_flip_relative_tie.py (with a test that an unchanged pair of large values leaves the p, the significance and the interval of eight small changed pairs as they are alone); test_repo_compare.py and test_repo_compare_interval.py are as on main, since a delta vector alone keeps the absolute width.
  • Docs: docs/benchmarking-and-comparison.md states the tie rule with the 0.1 + 0.2 and 1e-12 examples.
  • Changelog fragment (Fixed).

Risk

A pair whose larger value is below 1 has a tie width of 1e-9 times that value, narrower than the 1e-9 it had. A delta of such a pair that lies between the two widths (for example 0.5 against 0.5 + 7e-10, or 0 against 1e-12) was a tie and is now a change, so paired_test, arm_significance and repo compare rows with such deltas get n_changed, up/down counts, a p and an interval where they had none, and a row with some of them can move its p. Float noise in the values (0.1 + 0.2 against 0.3, a delta of 5.5e-17) stays a tie. Rows whose pairs all have a value of 1 or more, the JSON shapes and the schemas are unchanged; headroom, the "changed" line, the paired-cluster check, the ledger and the checklist keep the 1e-9 they had. The rounded ci_lo/ci_hi of a 1e-300 row are still printed at 4 places, as ±0.0001, which is the existing display rule and is not changed here. Open PR #628 edits two other lines of repos/compare.py (the .report import and refuse_mixed_executions), and open PR #705 edits other tests of tests/test_repo_compare_interval.py; git merge-tree of this branch with #705 is clean, and #628's lines are not touched here.

Open questions

The width could follow the vector (EPS * min(1, largest delta)) or each pair's larger value in magnitude. The reviewer's per-pair rule is taken: a delta is a tie when it is within EPS * min(1, m) of 0, with m the larger of the pair's two values. The discontinuity of the vector rule no longer applies: the width of a pair does not depend on the other deltas, so paired_test([5e-10] * 8, [0.0] * 8) keeps p 0.0078 and a ninth delta of 1.1e-9 does not turn the eight into ties (p 0.0039).

Testing

On a Linux review host, at a4ad627 (on main d92be0c):

  • make ci-local: 33 passed, 0 failed, 14 skipped (docker smoke, the matrix legs on other Pythons, deploy and PR-only jobs, conda-forge, fuzz, semgrep, the pinned Claude Code install). 21748 tests passed, 19 skipped; changed-line coverage 100% (68 lines).
  • Repro: paired_test([1e-12] * 8 + [1.0 + 1.1e-9], [0.0] * 8 + [1.0], rounded=False) gives p 0.00390625 (2/2**9) and significant True here, and p 1.0 before the slack was rounding error; the interval of [1e-12] * 8 + [0.0017 + 1.8e-12] against [0.0] * 8 + [0.0017] is 1e-12 at both ends with ShiftTest.p None there, and was 1.089e-12 at both with p 0.0078125; sign_flip_summary of eight pairs of 2e-12 to 9e-12 against zeros plus a 1.0/1.0 pair gives an mde (it was None).
  • Mutants: the slack of a sum taken as the tie-width cap again fails test_a_changed_pair_the_size_of_its_tie_width_does_not_make_every_sign_pattern_count; _start returning the mean fails test_the_ends_of_the_interval_are_shifts_the_test_accepts_when_the_mean_is_not_one and test_an_unchanged_pair_of_large_values_does_not_pull_the_interval_off_the_shift_the_small_pairs_accept; the mde spread read at the scale of every pair fails test_the_mde_of_changed_pairs_does_not_depend_on_an_unchanged_pair_of_large_values. The three new tests also fail on the code of eb1a924.
  • Repro: paired_test([1e-12] * 8 + [1.0], [0.0] * 8 + [1.0]) gives p 0.0078125 and significant True here, and p 1.0 before the slack followed the pairs summed; the interval of those deltas is (1e-12, 1e-12), which excludes 0 and is the interval of the eight small pairs alone; it was (8.9e-13, 8.9e-13), a shift the test rejects, before the search started from an accepted shift.
  • Mutants: the fallback start of _ends disabled fails test_an_unchanged_pair_of_large_values_does_not_pull_the_interval_off_the_shift_the_small_pairs_accept; the width of a pair of zeros taken as 1.0 fails test_a_pair_of_zeros_takes_the_width_of_the_largest_value_among_the_pairs, and two other tests of the file.
  • Mutants: the slack scale of sign_flip_p taken from every pair again, and ShiftTest's slack taken from every width again, each fail the new test.
  • Repro: paired_test on the 1e-300 vector gives p 0.0078, n_changed 8, significant True here and None, 0, None on main. paired_test([0.1 + 0.2] * 8, [0.3] * 8) gives p None and n_changed 0, and sign_flip_summary on those pairs gives 0/0/8 up/down/tied. paired_test([5e-10] * 8, [0.0] * 8) gives p 0.0078 and with a ninth delta of 1.1e-9 it gives 0.0039. sign_flip_row on [5e-324] * 6 + [0.0] * 7 against zeros gives p 0.03125 and an interval of 5e-324 to 5e-324 that excludes 0. The interval excludes 0 exactly when p < 0.05 on a 1e-300 vector, a 1e-12 vector and a vector of six 1e-3 deltas and one 1e-12 delta against 0 (n_changed 7: the 1e-12 is a change at its own scale).
  • Mutants: each of 10 breaks of the change, one per copy, fails test_sign_flip_relative_tie.py: the width absolute for every pair, the width 0, ShiftTest with one width for all deltas, the end below the smallest float left at 0, the power-of-two search for tiny values removed, the mde test absolute, paired_test without the magnitudes, tied absolute, n_changed absolute in _describe_deltas and in sign_flip_row.
  • Not run: the matrix legs on Python 3.11, 3.12 and 3.14, and the docker smoke.

Revert

A plain revert restores the absolute 1e-9 tie. Nothing is stored.

Checklist

  • Tests added/updated and pytest passes
  • ruff, mypy, and schema checks pass
  • Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
  • Self-reviewed per "Before opening a PR" in AGENTS.md
  • Each new raise/exit and test names the doc, contract or issue it traces to
  • No ad-hoc run artifacts committed
  • New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

…ntel#762)

The tie tolerance EPS = 1e-9 was absolute, so a delta vector that varies at a tiny scale (largest delta 1e-300) was all ties: paired_test gave it n_changed 0 and no p next to a large effect size, and repo compare counted every pair as tied.

A new helper tie_tolerance(deltas) in eval/sign_flip.py gives the width of a tie for a vector: EPS when its largest finite delta is EPS or more, so every result at score scale is unchanged, and EPS times that largest delta below it. The scale is the vector's, not a pair's, and an all-zero vector keeps EPS, so it has no p. sign_flip_p, ShiftTest, sign_flip_interval and sign_flip_row, the n_changed of _describe_deltas, the up, down, tied and mde of repo compare, its headroom checks and the non-inferiority slack of paired_cluster_difference all take it from that helper. The interval shifts the deltas by theta but keeps the tolerance (and the bisection step and far bound) of the original vector, so p(theta) stays one step function.

The ledger's abs(mean) > EPS guard and checklist.TIE_EPSILON compare scores in [0, 1] and keep the absolute width. Two tests that pinned the absolute tie on a vector whose largest delta is 1e-12 now pin the scaled one. EPS stays importable from eval.sign_flip, eval.stats and repos.compare.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@exai-bot

exai-bot commented Nov 16, 2026 •

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 4 selected item(s).

CI at a4ad627: 29 passing, 3 skipped.

Review details
  • Commit reviewed: a4ad627 against merge base eb1a924
  • Coverage: 4 files in 1 groups
  • Engine: OpenCodeReview v1.13.5-exai.4.8f73eada, us.openai.gpt-6.1-sol
  • Usage: 96k tokens (51k from cache), 31s
  • Tool calls: 9 (6 file_read, 1 ci_status, 1 local_check_results, 1 pr_threads)
  • Sign-flip sum slack rounding and comparison behavior: 4 files

Comment thread dokimi/dokimi/eval/sign_flip.py Resolved
@codecov-commenter

codecov-commenter commented Nov 16, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.74%. Comparing base (d92be0c) to head (a4ad627).
⚠️ Report is 15 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #818      +/-   ##
==========================================
- Coverage   95.76%   95.74%   -0.02%     
==========================================
  Files         416      423       +7     
  Lines       55624    56457     +833     
  Branches     9553     9717     +164     
==========================================
+ Hits        53267    54056     +789     
- Misses       1456     1492      +36     
- Partials      901      909       +8     
Files with missing lines Coverage Δ
dokimi/dokimi/eval/sign_flip.py 100.00% <100.00%> (ø)
dokimi/dokimi/eval/stats.py 97.82% <100.00%> (ø)
dokimi/dokimi/repos/compare.py 99.52% <100.00%> (+<0.01%) ⬆️

... and 34 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at 2228387. The 1e-300 case from #762 is fixed and score-scale results are unchanged, but the tie width now comes from the delta vector itself, so a vector of pure float noise is no longer tied and gets a significant p. Details are in the line comments.

Agreed with @exai-bot: the interval of subnormal deltas is [0, 0] next to a significant p; the pytest.approx asserts accept a zero tie width.

What I ran, at 2228387:

  • make ci-local --offline: 0 failed, 22 passed, 20 skipped (network, matrix legs); 21,689 tests passed, 72 skipped.
  • Repros on this head and on the merge-base d92be0c: paired_test and sign_flip_summary on 0.1 + 0.2 vs 0.3 (8 pairs), headroom on 1.0 vs 1 - 1.1e-16, paired_cluster_difference on noise bounds with margin 0, [5e-10] * 8 with and without one 1.1e-9, subnormal deltas at 1e-300 to 1e-320 and at 5e-324.
  • A mutant where tie_tolerance returns 0.0 below EPS: the four test files of the change still pass (176 passed).

Checked, no issues

  • Score-scale results: a vector whose largest delta is 1e-9 or more keeps EPS at every site, so existing rows, p values and verdicts do not move.
  • Interval and p: the shifted deltas keep the original vector's tolerance and the bisection step scales with it; the interval excludes 0 exactly when p < 0.05 down to about 1e-320.
  • Changelog and docs: a fragment under Fixed, and the docs page states the new rule.

Comment thread dokimi/dokimi/eval/sign_flip.py Resolved
Comment thread dokimi/dokimi/eval/sign_flip.py Resolved
Comment thread dokimi/dokimi/repos/compare.py Resolved
Comment thread dokimi/dokimi/eval/paired_cluster.py Resolved
Settles the review of intel#818: a tie width taken from the delta vector made a vector of float noise significant (`paired_test([0.1 + 0.2] * 8, [0.3] * 8)` gave p 0.0078 and 8 changed), made more evidence in one direction remove a significant result (`[5e-10] * 8` is p 0.0078, with one more delta of 1.1e-9 it was 1.0), and gave deltas of the smallest floats an interval of `[0, 0]` beside a significant p.

A pair now ties within `EPS * min(1, max(|treatment|, |baseline|))` (`pair_magnitudes`, `tie_widths`), since float noise scales with the values subtracted. `sign_flip_p`, `ShiftTest` (a width per delta), `sign_flip_interval`, `sign_flip_row`, `paired_test`, `arm_significance` and `sign_flip_summary` take the pairs' magnitudes; deltas alone keep the absolute `EPS`.

Values all below 2**-664 are searched multiplied by a power of two, and an end that comes back below the smallest float is that float on its side of 0, so the interval excludes 0 with the p.

The `headroom`, `format_compare` and `paired_cluster` changes of the first commit are taken back: they read tolerances off deltas, which are float noise next to the values they come from. The unused `EPS` imports go with them. `format_compare` keeps `EPS` and says its line follows the printed, rounded deltas.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@antonwolfy antonwolfy changed the title fix(eval): the sign-flip tie width follows the scale of the deltas fix(eval): the sign-flip tie width follows the values of each pair Nov 16, 2026
@antonwolfy

Copy link
Copy Markdown
Contributor Author

(commented) Review at 2228387. The 1e-300 case from #762 is fixed and score-scale results are unchanged, bu

@vlad-perevezentsev (#818 (review)): The review recaps what was run and points to the line comments. Its threads are answered above (4585134951, 4585134956, 4585310754, 4585310766, 4585310782, 4585310789); the "Checked, no issues" items still hold at 464c159 (score-scale results with values of 1 or more are unchanged, the interval excludes 0 exactly when p < 0.05, changelog and docs updated).

Comment thread dokimi/dokimi/eval/sign_flip.py Resolved
… tied pairs

Settles the review point that an unchanged pair of large values loosened the sums of small changed pairs: paired_test([1e-12] * 8 + [1.0], [0.0] * 8 + [1.0]) gave p 1.0, where the eight small pairs alone give 0.0078125.

sign_flip_p takes the scale of the slack from the magnitudes of the deltas it sums, and ShiftTest takes it from the tie widths of the deltas that are not ties at that theta, instead of the widest width of every pair.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@antonwolfy

Copy link
Copy Markdown
Contributor Author

Ready for another look at aae990b; CI is green.

Declined, with the evidence in the reply:

  • dokimi/tests/test_sign_flip_relative_tie.py:29 (exai-bot) (fixed in part): The [1e-12]*6+[1e-22] case with n_changed == 6 is not added: under the per-pair width the 1e-22 against 0 is a change at its own scale (n_changed 7 here).

Follow-up issues opened:

@vchamarthi vchamarthi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Changes needed: 1 must-fix, 1 nit · reviewed aae990b

Next: @antonwolfy: fix the 1 must-fix finding; then push for a re-review. 1 nit optional.

Must-fix

  • dokimi/dokimi/eval/sign_flip.py:286 – Eight pairs of 1e-12 beside one tied pair of 1.0 give an interval at 8.9e-13, a shift the test rejects.

    Input: [1e-12] * 8 + [1.0] against [0.0] * 8 + [1.0]. Head gives ci_lo and ci_hi both 8.888888888888889e-13, the mean of the nine deltas. At that shift the eight changed deltas still give p 0.0078, so the end is rejected. Expected: 1e-12 at both ends, as the eight pairs alone give. Cause: _end returns its starting point, the mean, without checking that the test accepts it; the tied zero delta pulls the mean off the one accepted shift, which is narrower than the search step.

    Suggestion: Search from a shift the test accepts, such as the mean of the changed deltas, instead of returning the mean unchecked.

Follow-ups, not blocking

  • dokimi/dokimi/eval/stats.py:618 – Input: eight pairs of 1e308 * g / 9 (g in 2, 4, 3, 5, 6, 9, 7, 8) against zeros (main has it too).

Verified on aae990b: ✅ changed tests: 9 passed, fail on main · ✅ full suite with main merged in: 22043 passed · ✅ merges onto main · ✅ 12 broken variants of the changed code: 12 caught by the tests · 1 mutation tried · 5 edge inputs tried

9 checks behind it, each with its evidence
Area What holds How
Float-noise and 1e-300 ties The nine tests in tests/test_sign_flip_relative_tie.py pass on the head. They pin 0.1 + 0.2 against 0.3 as a tie with no p, and the 1e-300 vector as p 0.0078 with 8 changed and significant True, as its 1e-3 copy gives tests on the head
Smallest float deltas Six pairs of 5e-324 against zeros (p 0.03125) give an interval that excludes 0, pinned by test_deltas_of_the_smallest_floats_have_an_interval_that_excludes_zero_with_their_significant_p tests on the head, edge input (subnormal)
More evidence in one direction Eight pairs of 5e-10 give p 0.0078, and a ninth delta of 1.1e-9 gives 0.0039, pinned by test_more_evidence_in_the_same_direction_does_not_remove_a_significant_result tests on the head
Tie scale cap Dropping the cap at 1 in tie_scale (sign_flip.py:78) fails the changed tests, so they pin the cap broke the code, tests caught it
Merge The branch merges onto current main with no conflict merge onto main
Full suite The full suite and the static checks pass with main merged in full suite, main merged in
Numbers Int inputs give the same p as floats (0.0078), and a NaN delta gives no p and no verdict ran it, edge input (numbers)
Collections Empty and single-pair inputs give no test, pair order changes neither p nor the interval, and 200 pairs at score scale get a Monte Carlo p with 171 changed ran it, edge input (collections)
No values given With no values given a tie is the absolute EPS, so [1e-12] * 8 has no p, and an all-zero vector has no p ran it, edge input (none)

Settles the review of intel#818: eight pairs of 1e-12 beside one tied pair of 1.0 gave an interval at 8.9e-13 at both ends, the mean of the nine deltas, a shift the test rejects. The search returned its starting point unchecked; it now starts from the mean of the changed deltas when the mean of all is rejected, so the ends are 1e-12 as for the eight pairs alone.

The docs sentence on tie widths now says a pair of zeros takes the width of the largest value among the pairs.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@antonwolfy

Copy link
Copy Markdown
Contributor Author

(commented) 🔴 Changes needed: 1 must-fix, 1 nit · reviewed aae990b

@vchamarthi (#818 (review)): Fixed in eb1a924. Reproduced: [1e-12] * 8 + [1.0] against [0.0] * 8 + [1.0] gave (8.888888888888889e-13, 8.888888888888889e-13), the mean of the nine deltas, which the test rejects. _ends now checks the test at the mean and, when it rejects it, starts the search from the mean of the changed deltas; the ends are (1e-12, 1e-12), as the eight pairs alone give. test_an_unchanged_pair_of_large_values_does_not_pull_the_interval_off_the_shift_the_small_pairs_accept fails with that check disabled. The follow-up at stats.py:618 holds and is not this PR's: on main and here paired_test on eight pairs of 1e308 * g / 9 against zeros gives delta_mean inf, p None, n_changed 8 and no verdict. It is tracked in #832.

Comment thread dokimi/dokimi/eval/sign_flip.py Resolved
Comment thread dokimi/dokimi/eval/sign_flip.py Resolved
Comment thread dokimi/dokimi/repos/compare.py Resolved
Settles the three open threads of the review. The slack on a sum of deltas was the widest tie width among them, so a changed pair just past its width (1.0 + 1.1e-9 against 1.0) made every sign pattern count and turned eight significant pairs of 1e-12 into p 1.0. The slack is now the rounding error of adding the deltas (n * 2**-52 times their absolute sum), capped by the tie width it replaces, in sign_flip_p and ShiftTest.p.

The interval search started from the mean of the deltas, or of the changed ones, without checking either was accepted: with every delta changed, a mean that ties the largest delta returned two rejected ends. It now tries the mean, the mean of the changed deltas, then up to 16 of the deltas nearest the median, and keeps the first the test accepts.

The mde of a repo compare row read the spread against 1e-9 times the largest value of any pair, so an unchanged pair of large values removed it. It now reads the scale of the changed pairs only.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@antonwolfy

Copy link
Copy Markdown
Contributor Author

Ready for another look at a4ad627; CI is green.

Follow-up issues opened:

@napetrov napetrov left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at a4ad627. Every earlier must-fix (interval ends, unchanged-pair p, 9th pair, 5e-324 interval, test abs tolerance) holds on head.
Probed: body repros, 3000 untied vectors match main, interval/p agree on 5500 fuzzed vectors; 168 sign-flip tests pass.
Non-blocking: mde prints 0.0 at 1e-12 scale (#817 family); four changelog fragments for one fix; 1 ulp noise above 1 is still a change (pre-existing).

@antonwolfy

Copy link
Copy Markdown
Contributor Author

(approved) Approving at a4ad627. Every earlier must-fix (interval ends, unchanged-pair p, 9th pair, 5e-324 interval, test abs tolerance) holds on head.

@napetrov (#818 (review)): No change this round. The mde printing 0.0 at a 1e-12 scale is the rounding of sign-flip rows already tracked in #817. The four changelog fragments each describe one commit's change to the same tie-width rule and are published; they are left as they are. The 1 ulp noise above 1 being treated as a change is how main behaved before this PR and is outside it.

@antonwolfy
antonwolfy merged commit 4828265 into intel:main Nov 17, 2026
32 checks passed
@antonwolfy
antonwolfy deleted the fix/sign-flip-relative-tie branch November 17, 2026 09:44
antonwolfy added a commit that referenced this pull request Nov 17, 2026
…heir values (#868)

* fix(eval): interaction_test reads contrasts of tiny-scale scores by their values (#821)

interaction_test and interaction_significance passed the per-question contrasts AB - A - B + baseline, and the three arm deltas of the row, to the sign-flip description without the values they come from, so every tie was within the absolute 1e-9. Eight evaluations with a+b scores g * 1e-300 and 0 elsewhere gave p None, n_changed 0 and significant None, where the same scores at 1e-3 give p 0.0078, n_changed 8 and significant True.

The contrast of a row now passes the largest magnitude of its four scores, and each arm delta the larger magnitude of its pair against the baseline, as paired_test does since #818. A tie is within 1e-9 times that value when it is below 1, and 1e-9 otherwise, so rows at ordinary score scales are unchanged. The 1e-300 rows read as their 1e-3 copies.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

* fix(eval): interaction tie-width docs and a test of the contrast's scale (#868)

Settles the review of #868: the tie-rule example in benchmarking-and-comparison.md named a contrast of 1e-12 that scores of 1e-300 cannot produce, and now uses AB = 1e-300 with A, B and baseline zero.

A new test row where `a` is the largest score (contrast -7e-13, a tie at 1e-12) fails when the contrast's scale is taken from `ab` and `baseline` only.

The `interaction_test` docstring states the tie width of the contrast and of each arm delta, and `_interaction_contrast` takes the ab, a and b columns once for both `_deltas` and `pair_magnitudes`.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

---------

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sign-flip tie tolerance EPS = 1e-9 is absolute: deltas that vary at a tiny scale are ties and get no p

6 participants