Repository navigation
fix(eval): the interval of a Monte Carlo row of widely spread deltas counts its resamples on exact sums - #893
Open
antonwolfy wants to merge 1 commit into
Open
antonwolfy wants to merge 1 commit into
antonwolfy wants to merge 1 commit into
Conversation
…counts its resamples on exact sums (intel#865) For more than 16 changed deltas, ShiftTest.p counted the resamples with a float matrix product, so the small deltas of [1e16] + [1.0] * 19 were absorbed by the large one and the lower end came out as 0.9474 (0.0001 for 1e20 and 1e300) where the exact p puts it at 1.0. The upper end of that row was already right (about 1e16 / 7). When the shifted deltas span more than 40 binary orders, the resamples are now counted on exact integer sums (limbs summed as int64 products, joined as Python ints), with the same draws and the same observed-sum slack as sign_flip_p, so p(0) still equals sign_flip_p for the seed. Rows below that spread keep the numpy product. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
#865:
sign_flip_intervalof a row with a delta far larger than the others took its ends from a float search that loses the small deltas. The rows of the issue's text ([m, 1, 1, 1, 1, 1], 6 deltas) are already fixed atmainby #867 (ShiftTest.psums up to 16 widely spread shifted deltas exactly,sign_flip.py). What still goes wrong is the Monte Carlo branch, more than 16 shifted deltas, which counted resamples withsigns @ xin floats. Onmain, withp = sign_flip_p(row)of 5e-05:main[1e16] + [1.0] * 19[1e20] + [1.0] * 19[1e300] + [1.0] * 19The upper end of these rows was already right (about
m / 7; the exact p at 1.4e15 is 0.081 and at 1.43e15 it is 0.030), so only the lower end changes. A mixed-sign row of 19 deltas from -1e30 to 1e20 also hadShiftTest(row).p(0.0) == 1.0onmain, against 0.8906 fromsign_flip_p.The issue offers two fixes: sum exactly in
ShiftTest.p, or bisect on the exactsign_flip_p. This takes the first for the Monte Carlo branch only where it is needed: when the shifted deltas span more than 40 binary orders (the test the exact branch uses), the resamples are counted on exact integer sums with the same draws. Rows below that spread keep the numpy product and are unchanged.Closes #865
Changes
dokimi/dokimi/eval/sign_flip.py:ShiftTest.pcalls the new_exact_hitsin its Monte Carlo branch whenspread > 40. It reuses_as_integersand_observed(the slack ofsign_flip_p) and the_drawssigns, sop(0)equalssign_flip_pfor the seed. The integers are split into limbs that cannot overflow int64 over all the deltas, each limb is summed as an int64 product, and the limbs are joined as Python ints. No float sum is used. TheShiftTestand module docstrings say so in place of "more of them are not (sign_flip_interval: the ends of a row with one delta far larger than the others come from a float search that loses the small deltas #865)".dokimi/tests/test_sign_flip_mc_spread.py: the 20-delta rows with 1e16 and 1e300 (lower end 1.0,p(0)equal tosign_flip_p), and a mixed-sign row with several limbs.Risk
Rows of more than 16 shifted deltas that span more than 40 binary orders get new lower ends (the exact ones); every other row gives the same ends. Those rows are slower to search:
sign_flip_intervalof[1e16] + [1.0] * 19takes about 0.2 s, and of[1e300] + [1.0] * 19about 15 s (it was under 0.1 s), since the limb count grows with the spread. The ends are the same from run to run. The fragment of #867 says a row of more than 16 such deltas keeps the float search; it is unreleased text that this change makes stale, and is left as it is.Testing
On Linux, at
c79ba9e(onmain7125813):make ci-local: 36 passed, 0 failed, 14 skipped (docker/network catalog smoke, the pinned Claude Code, test-latest, deploy jobs, conda-forge, fuzz, the intel-ci matrix, semgrep). 22781 tests passed, 19 skipped; changed-line coverage 100%.main's code the new test file fails 3 of 3: the lower end is 7.1e-08 for1e300and 0.9474 for1e16where 1.0 is exact, and the mixed-sign row'sp(0)is 1.0 against 0.8906.spread > 4000) fails the lower-end assertion; dropping the+ 1of the exact p fails thep(0)assertion; a raised threshold in_exact_hitsfails the lower-end assertion; ignoring the sign of a delta in the limbs fails the mixed-sign test. The mutation job ofci-localleaves 8 survivors on changed lines, equivalent or not worth an assertion: negating every limb sum (absfollows), the limb width 62 to 63, thespread > 40threshold to 41, and the first-limb index.fix/score-cv-overflowonmainb62613a:pytest -n auto22866 passed, 19 skipped,mypyclean,ruff checkclean.Open questions
None.
Revert
A plain revert restores the float search for Monte Carlo rows of widely spread deltas. Nothing is stored.
Checklist
pytestpassesruff,mypy, and schema checks passAGENTS.mdraise/exit and test names the doc, contract or issue it traces to.github/.licenserc.yaml)