Repository navigation
fix(eval): interaction_test reads contrasts of tiny-scale scores by their values - #868
Conversation
…heir values (intel#821) interaction_test and interaction_significance passed the per-question contrasts AB - A - B + baseline, and the three arm deltas of the row, to the sign-flip description without the values they come from, so every tie was within the absolute 1e-9. Eight evaluations with a+b scores g * 1e-300 and 0 elsewhere gave p None, n_changed 0 and significant None, where the same scores at 1e-3 give p 0.0078, n_changed 8 and significant True. The contrast of a row now passes the largest magnitude of its four scores, and each arm delta the larger magnitude of its pair against the baseline, as paired_test does since intel#818. A tie is within 1e-9 times that value when it is below 1, and 1e-9 otherwise, so rows at ordinary score scales are unchanged. The 1e-300 rows read as their 1e-3 copies. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 4 selected item(s). CI at Review details
|
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #868 +/- ##
=======================================
Coverage 95.76% 95.76%
=======================================
Files 424 425 +1
Lines 56511 56477 -34
Branches 9730 9718 -12
=======================================
- Hits 54115 54084 -31
+ Misses 1487 1486 -1
+ Partials 909 907 -2
🚀 New features to boost your workflow:
|
vlad-perevezentsev
left a comment
There was a problem hiding this comment.
Review at 8b00389. The fix is right: the contrast now ties within 1e-9 times the largest of its four scores, and each arm delta uses the same pair rule as paired_test, so the 1e-300 rows read like their 1e-3 copies. One test gap is in the line comments.
Agreed with @exai-bot on the impossible 1e-12 example in the docs.
What I ran, at 8b00389:
make ci-local --offline: 0 failed, 23 passed, 20 skipped; the test suite 22,345 passed, 72 skipped.- A mutation that takes the contrast's scale from
abandbaselineonly: all 9 new tests still pass; the row in the line comment catches it. - The new test file on the merge-base: the 1e-300 cases fail, so they cover the fix.
Checked, no issues
- Call sites:
interaction_testandinteraction_significanceboth go through_interaction_contrast, so the corrected family gets the same tie widths. - Rows at ordinary scales: scores of 1 or more keep the absolute 1e-9 width, as the changelog says; below 1 the widths follow #818's rule.
- Changelog: a
Fixedfragment that names the case and the new rule.
…ale (intel#868) Settles the review of intel#868: the tie-rule example in benchmarking-and-comparison.md named a contrast of 1e-12 that scores of 1e-300 cannot produce, and now uses AB = 1e-300 with A, B and baseline zero. A new test row where `a` is the largest score (contrast -7e-13, a tie at 1e-12) fails when the contrast's scale is taken from `ab` and `baseline` only. The `interaction_test` docstring states the tie width of the contrast and of each arm delta, and `_interaction_contrast` takes the ab, a and b columns once for both `_deltas` and `pair_magnitudes`. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@vlad-perevezentsev (#868 (review)): The review body recaps its checks and points at its two threads: the impossible docs example (4599152107) and the missing |
vlad-perevezentsev
left a comment
There was a problem hiding this comment.
LGTM
I checked 42b9f4b after the update. The new test pins the four-score scale of the contrast, the docstring states the rule, and the docs example is now possible. The fix itself holds as in the first round.
Merge: safe now: it merges cleanly into current main, 5 commits behind, and main has not touched its files since the base. Once it is in, #867 conflicts with it in interactions.py and needs a rebase.
What I ran, at 42b9f4b:
make ci-local --offline: 22,346 passed, 72 skipped.schema_compatfails because the branch is behindmain, which addedrepo_event.v1.jsonafter this base. It is not from this PR.- The new row fails with a contrast scale of
max(abs(ab), abs(base))(n_changed 9 instead of 8).
Summary
interaction_testandinteraction_significancegave_describe_deltasthe per-question contrastsAB - A - B + baseline, and the three arm deltas of the row, without the scores they come from, so the sign-flip tie width was the absoluteEPS = 1e-9(the same cause as #762, which #818 settled forpaired_test). Eight evaluations witha+bscoresg * scalefor g in 2, 4, 3, 5, 6, 9, 7, 8 and 0 for the other three arms:pn_changedsignificantmainThe arm deltas had the same cause:
delta_combinedof the 1e-300 row printed5.5e-300onmain(read as a tie, so unrounded) and prints whatpaired_testprints for the same pairs now.The fix is the one the issue names: the contrast passes the largest of the four scores' magnitudes of its row, since the float noise of
ab - a - b + basescales with the largest of the four values (the pair rule, larger of two, extended to four). Each arm delta passes the pair magnitudes of its scores against the baseline.Closes #821
Changes
dokimi/eval/interactions.py:_interaction_contrastpassesmagnitudes=(max(abs(ab), abs(a), abs(b), abs(base))per row, in the order ofcontrasts) to_describe_deltas;_arm_deltatakes the pair magnitudes (pair_magnitudesof thea,borabscores against the baseline) fordelta_a,delta_banddelta_combined.dokimi/eval/interactions.py: theinteraction_testdocstring states the tie width of the contrast and of each arm delta, and_interaction_contrasttakes theab,aandbcolumns once for_deltasandpair_magnitudes.docs/benchmarking-and-comparison.md: the interaction paragraph says the tie rule applies to the contrast with the largest of its four scores and to each arm delta with its pair's larger score; its example is AB = 1e-300 with A, B and baseline all zero, a change.tests/test_interactions_tie_width.py: the 1e-300 and 1e-3 rows, a tie and a change row at different scales, a row whereais the largest score (contrast -7e-13 is a tie), and the arm deltas againstpaired_test.Risk
The tie width is
EPS * min(1, m), which isEPSfor m of 1 or more: rows at ordinary score scales (0 to 100) give the samep,n_changed, ends and verdict, and every existing test ofinteractions.pypasses unchanged. Rows whose four scores are all below 1 can now count a contrast under 1e-9 as a change, where it was a tie. At scores near 1e-300 the printedinteractionand arm deltas are now rounded to 4 decimals (0.0001), as any row with a p is; that rounding is #817, not this change.Testing
On a Linux review host, at
42b9f4b(onmain08f393c):make ci-local: 34 passed, 0 failed, 14 skipped (steps needing docker and network, the matrix legs, jobs that do not run on a PR, micromamba, semgrep and the fuzz run). 22399 tests passed, 19 skipped.tests/test_interactions_tie_width.pyonmain's code: 4 failed, 5 passed (before thea-largest row was added; that row passes onmain, whose width is absolute); the first failure isassert 5.5e-300 == 0.0001(arm delta), and the contrast row fails onassert 0 == 8(n_changed). Here: 9 passed.magnitudesfrom the contrast fails the 1e-300 row (assert 0 == 8); dropping it from_arm_deltafails the three 1e-300 arm-delta cases (assert 5.5e-300 == 0.0001); passingabs(contrast)as the magnitudes fails the tie row (assert 9 == 8); taking the contrast's scale frommax(abs(ab), abs(base))fails thea-largest row (assert 9 == 8).fix/exact-sign-flip-sums,fix/paired-summary-overflowonmain08f393c:pytest -n auto22418 passed, 19 skipped,mypyclean,ruff checkclean.Open questions
None.
Revert
A plain revert restores the absolute 1e-9 tie width of the interaction rows. Nothing is stored.
Checklist
pytestpassesruff,mypy, and schema checks passAGENTS.mdraise/exit and test names the doc, contract or issue it traces to.github/.licenserc.yaml)