Skip to content

fix(eval): the interval of a Monte Carlo row of widely spread deltas counts its resamples on exact sums - #893

Open
antonwolfy wants to merge 1 commit into
intel:mainfrom
antonwolfy:fix/sign-flip-mc-spread
Open

antonwolfy wants to merge 1 commit into
intel:mainfrom
antonwolfy:fix/sign-flip-mc-spread

Conversation

@antonwolfy

Copy link
Copy Markdown
Contributor

Summary

#865: sign_flip_interval of a row with a delta far larger than the others took its ends from a float search that loses the small deltas. The rows of the issue's text ([m, 1, 1, 1, 1, 1], 6 deltas) are already fixed at main by #867 (ShiftTest.p sums up to 16 widely spread shifted deltas exactly, sign_flip.py). What still goes wrong is the Monte Carlo branch, more than 16 shifted deltas, which counted resamples with signs @ x in floats. On main, with p = sign_flip_p(row) of 5e-05:

row lower end on main exact
[1e16] + [1.0] * 19 0.9474 1.0
[1e20] + [1.0] * 19 0.0001 1.0
[1e300] + [1.0] * 19 0.0001 1.0

The upper end of these rows was already right (about m / 7; the exact p at 1.4e15 is 0.081 and at 1.43e15 it is 0.030), so only the lower end changes. A mixed-sign row of 19 deltas from -1e30 to 1e20 also had ShiftTest(row).p(0.0) == 1.0 on main, against 0.8906 from sign_flip_p.

The issue offers two fixes: sum exactly in ShiftTest.p, or bisect on the exact sign_flip_p. This takes the first for the Monte Carlo branch only where it is needed: when the shifted deltas span more than 40 binary orders (the test the exact branch uses), the resamples are counted on exact integer sums with the same draws. Rows below that spread keep the numpy product and are unchanged.

Closes #865

Changes

  • dokimi/dokimi/eval/sign_flip.py: ShiftTest.p calls the new _exact_hits in its Monte Carlo branch when spread > 40. It reuses _as_integers and _observed (the slack of sign_flip_p) and the _draws signs, so p(0) equals sign_flip_p for the seed. The integers are split into limbs that cannot overflow int64 over all the deltas, each limb is summed as an int64 product, and the limbs are joined as Python ints. No float sum is used. The ShiftTest and module docstrings say so in place of "more of them are not (sign_flip_interval: the ends of a row with one delta far larger than the others come from a float search that loses the small deltas #865)".
  • dokimi/tests/test_sign_flip_mc_spread.py: the 20-delta rows with 1e16 and 1e300 (lower end 1.0, p(0) equal to sign_flip_p), and a mixed-sign row with several limbs.
  • Changelog fragment (Fixed).

Risk

Rows of more than 16 shifted deltas that span more than 40 binary orders get new lower ends (the exact ones); every other row gives the same ends. Those rows are slower to search: sign_flip_interval of [1e16] + [1.0] * 19 takes about 0.2 s, and of [1e300] + [1.0] * 19 about 15 s (it was under 0.1 s), since the limb count grows with the spread. The ends are the same from run to run. The fragment of #867 says a row of more than 16 such deltas keeps the float search; it is unreleased text that this change makes stale, and is left as it is.

Testing

On Linux, at c79ba9e (on main 7125813):

  • make ci-local: 36 passed, 0 failed, 14 skipped (docker/network catalog smoke, the pinned Claude Code, test-latest, deploy jobs, conda-forge, fuzz, the intel-ci matrix, semgrep). 22781 tests passed, 19 skipped; changed-line coverage 100%.
  • On main's code the new test file fails 3 of 3: the lower end is 7.1e-08 for 1e300 and 0.9474 for 1e16 where 1.0 is exact, and the mixed-sign row's p(0) is 1.0 against 0.8906.
  • Mutants, each failing the test named: the exact branch disabled (spread > 4000) fails the lower-end assertion; dropping the + 1 of the exact p fails the p(0) assertion; a raised threshold in _exact_hits fails the lower-end assertion; ignoring the sign of a delta in the limbs fails the mixed-sign test. The mutation job of ci-local leaves 8 survivors on changed lines, equivalent or not worth an assertion: negating every limb sum (abs follows), the limb width 62 to 63, the spread > 40 threshold to 41, and the first-limb index.
  • Not run: the other Python versions (3.11, 3.12, 3.14).
  • Merged together with fix/score-cv-overflow on main b62613a: pytest -n auto 22866 passed, 19 skipped, mypy clean, ruff check clean.

Open questions

None.

Revert

A plain revert restores the float search for Monte Carlo rows of widely spread deltas. Nothing is stored.

Checklist

  • Tests added/updated and pytest passes
  • ruff, mypy, and schema checks pass
  • Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
  • Self-reviewed per "Before opening a PR" in AGENTS.md
  • Each new raise/exit and test names the doc, contract or issue it traces to
  • No ad-hoc run artifacts committed
  • New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

…counts its resamples on exact sums (intel#865)

For more than 16 changed deltas, ShiftTest.p counted the resamples with a float matrix product, so the small deltas of [1e16] + [1.0] * 19 were absorbed by the large one and the lower end came out as 0.9474 (0.0001 for 1e20 and 1e300) where the exact p puts it at 1.0. The upper end of that row was already right (about 1e16 / 7).

When the shifted deltas span more than 40 binary orders, the resamples are now counted on exact integer sums (limbs summed as int64 products, joined as Python ints), with the same draws and the same observed-sum slack as sign_flip_p, so p(0) still equals sign_flip_p for the seed. Rows below that spread keep the numpy product.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@antonwolfy
antonwolfy requested a review from a team as a code owner November 17, 2026 21:19
@antonwolfy antonwolfy self-assigned this Nov 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sign_flip_interval: the ends of a row with one delta far larger than the others come from a float search that loses the small deltas

1 participant