Repository navigation
fix(eval): the interval of a Monte Carlo row of widely spread deltas counts its resamples on exact sums #893
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| ### Fixed | ||
|
|
||
| - **Sign-flip interval of a Monte Carlo row whose deltas span more than a float sum resolves** — for a row of more than 16 changed deltas, `sign_flip_interval` counted each shift's resamples with float sums, so the small deltas of `[1e16] + [1.0] * 19` were absorbed by the large one and its lower end came out as 0.9474 (0.0001 for `1e20` or `1e300`) while the exact p puts it at 1.0. When the shifted deltas span more than 40 binary orders of magnitude, the resamples are now counted on exact integer sums, with the same draws, so `p(0)` still equals `sign_flip_p` for the seed. A row below that spread, and any row of 16 deltas or fewer, gives the same ends as before. The exact sums make such a row slower to search: about 0.2 s for `1e16` and about 15 s for `1e300`. |
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -8,7 +8,7 @@ | |||||
| :func:`min_sign_flip_p` is the floor it cannot go below over k changed deltas, ``2 / 2**k`` exact. | ||||||
|
|
||||||
| ``p(theta)`` is the two-sided sign-flip p of the shifted deltas ``d - theta`` over every delta, as | ||||||
| :func:`sign_flip_p` computes it on them (that one adds the deltas as exact integers; this one, below, in floats, so its p can differ at the ends of a row whose deltas floats cannot resolve): a shifted delta within the tie width of its pair (:func:`tie_widths`, kept | ||||||
| :func:`sign_flip_p` computes it on them (that one adds the deltas as exact integers; this one, below, in floats unless the shifted deltas span more than 40 binary orders, so its p can differ at the ends of a row whose deltas floats cannot resolve in fewer): a shifted delta within the tie width of its pair (:func:`tie_widths`, kept | ||||||
| from the original deltas) of 0 is a tie, so a delta tied at 0 counts again once ``theta`` moves off 0. Exact, a sign | ||||||
| pattern counts when its sum falls short of the observed one by at most the rounding error of adding the deltas summed (:func:`_sum_slack`; :data:`EPS` at most, and no more than the | ||||||
| largest tie width among them), | ||||||
|
|
@@ -167,6 +167,24 @@ | |||||
| return [num * (denominator // den) for num, den in ratios], math.floor(Fraction(cap) * denominator) | ||||||
|
|
||||||
|
|
||||||
| def _exact_hits(signs: np.ndarray, values: list[float], cap_eps: float) -> int: | ||||||
| """The resamples (rows of ``signs``) whose exact ``|sum|`` reaches the observed one less its slack: the integers of | ||||||
| ``values`` are split into limbs that cannot overflow int64 over all of them, each limb summed as an int64 | ||||||
| product, and the limbs joined as Python ints.""" | ||||||
| import numpy as np | ||||||
|
|
||||||
| ints, cap = _as_integers(values, cap_eps) | ||||||
| observed = _observed(ints, cap) | ||||||
| bits = max((abs(v).bit_length() for v in ints), default=0) | ||||||
| s64 = signs.astype(np.int64) | ||||||
| total = np.zeros(signs.shape[0], dtype=object) | ||||||
| width = 62 - len(ints).bit_length() # n limbs of this many bits sum within int64 | ||||||
| for shift in range(0, bits, width): | ||||||
| limbs = np.array([(1 if v >= 0 else -1) * ((abs(v) >> shift) & ((1 << width) - 1)) for v in ints], dtype=np.int64) | ||||||
|
Check warning on line 183 in dokimi/dokimi/eval/sign_flip.py
|
||||||
| total = total + (s64 @ limbs).astype(object) * (1 << shift) | ||||||
| return sum(abs(t) >= observed for t in total.tolist()) | ||||||
|
|
||||||
|
|
||||||
| @functools.cache | ||||||
| def _patterns(k: int) -> np.ndarray: | ||||||
| """Every sign pattern of k deltas, one per row: ``2**k x k``, as floats for the product.""" | ||||||
|
|
@@ -199,7 +217,8 @@ | |||||
| on, a sign pattern whose sum ties the observed one in exact arithmetic (magnitudes that repeat, such as 1e9 + 0.1) | ||||||
| can round to one side of it in the product and to the other in the exact sums of :func:`sign_flip_p`, so ``p(0)`` can differ there: the | ||||||
| row's own p, not ``p(0)``, places 0 (:func:`sign_flip_interval`). Up to :data:`EXACT_MAX` shifted deltas that span more than | ||||||
| 40 binary orders of magnitude are summed exactly too, as floats lose the small ones there; more of them are not (#865).""" | ||||||
| 40 binary orders of magnitude are summed exactly too, as floats lose the small ones there; so are the Monte Carlo | ||||||
| resamples of more of them (:func:`_exact_hits`), whose p then follows :func:`sign_flip_p`'s integer sums.""" | ||||||
|
|
||||||
| def __init__(self, deltas: list[float], seed: int = 0, n_resamples: int = N_RESAMPLES, *, | ||||||
| eps: float | Sequence[float] = EPS) -> None: | ||||||
|
|
@@ -232,6 +251,8 @@ | |||||
| least = min(observed, float(sums[0]), float(sums[-1])) # the product rounds them otherwise from 1e6 on | ||||||
| return int(np.count_nonzero(sums >= least - eps)) / 2 ** moved.size | ||||||
| signs = _draws(self.seed, self.changed, self.d.size, self.n_resamples) | ||||||
| if spread > 40: # floats lose the small deltas: count the resamples on exact integer sums | ||||||
| return (_exact_hits(signs, x.tolist(), float(self.widths[~tied].max())) + 1) / (self.n_resamples + 1) | ||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Check run: inconclusive, the result is not shown as a pass. Suggestion:
Suggested change
|
||||||
| return (int(np.count_nonzero(np.abs(signs @ x) >= observed - eps)) + 1) / (self.n_resamples + 1) | ||||||
|
|
||||||
|
|
||||||
|
|
||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| # Copyright (C) 2026 Intel Corporation | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| """The Monte Carlo ends of a row whose deltas span more than a float sum resolves (#865).""" | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import pytest | ||
|
|
||
| from dokimi.eval.sign_flip import ShiftTest, sign_flip_interval, sign_flip_p | ||
|
|
||
|
|
||
| @pytest.mark.parametrize("big", [1e16, 1e300]) | ||
| def test_a_large_delta_does_not_hide_the_small_ones_at_the_ends(big): | ||
| row = [big] + [1.0] * 19 # 20 deltas: Monte Carlo, and 1.0 vanishes next to big in a float sum | ||
| p = sign_flip_p(row) | ||
| lo, _ = sign_flip_interval(row, p, rounded=False) | ||
| # the small deltas are the sign-flip evidence: shifting by 1 ties them all, so the lower end is 1, not ~0 | ||
| assert (p, lo) == (pytest.approx(5e-5, abs=1e-6), pytest.approx(1.0, abs=1e-6)) | ||
| test = ShiftTest(row) | ||
| assert test.p(0.0) == sign_flip_p(row, exact=True) | ||
|
|
||
|
|
||
| def test_exact_resamples_of_deltas_of_both_signs_and_several_limbs_match_sign_flip_p(): | ||
| row = [-1e30, 7.5, -3.25, 2.0, -1.0, 4.5, 1e-3, -6.0, 8.0, 0.5, -2.5, 3.0, 1e20, -9.0, 5.0, 1.5, -4.0, 6.5, 2.5] | ||
| for seed in (0, 3): | ||
| assert ShiftTest(row, seed).p(0.0) == sign_flip_p(row, seed=seed, exact=True) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
CI check
Test Linux Wheels Python 3.11fails: The added 1e300 interval test repeatedly enters the new exact-sum loop during bisection and exceeds the 60-second test timeout. Sparse integer limbs still trigger full matrix products and object-array arithmetic, including limbs that contribute zero. Proposed fix, not run by the bot.CI check
Test Linux Wheels Python 3.14fails for the same reason.CI check
Tests (py3.11)fails for the same reason.CI check
Tests (py3.14)fails for the same reason.