Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions dokimi/changelog.d/20261117_180000_sign-flip-mc-spread.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
### Fixed

- **Sign-flip interval of a Monte Carlo row whose deltas span more than a float sum resolves** — for a row of more than 16 changed deltas, `sign_flip_interval` counted each shift's resamples with float sums, so the small deltas of `[1e16] + [1.0] * 19` were absorbed by the large one and its lower end came out as 0.9474 (0.0001 for `1e20` or `1e300`) while the exact p puts it at 1.0. When the shifted deltas span more than 40 binary orders of magnitude, the resamples are now counted on exact integer sums, with the same draws, so `p(0)` still equals `sign_flip_p` for the seed. A row below that spread, and any row of 16 deltas or fewer, gives the same ends as before. The exact sums make such a row slower to search: about 0.2 s for `1e16` and about 15 s for `1e300`.
25 changes: 23 additions & 2 deletions dokimi/dokimi/eval/sign_flip.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
:func:`min_sign_flip_p` is the floor it cannot go below over k changed deltas, ``2 / 2**k`` exact.

``p(theta)`` is the two-sided sign-flip p of the shifted deltas ``d - theta`` over every delta, as
:func:`sign_flip_p` computes it on them (that one adds the deltas as exact integers; this one, below, in floats, so its p can differ at the ends of a row whose deltas floats cannot resolve): a shifted delta within the tie width of its pair (:func:`tie_widths`, kept
:func:`sign_flip_p` computes it on them (that one adds the deltas as exact integers; this one, below, in floats unless the shifted deltas span more than 40 binary orders, so its p can differ at the ends of a row whose deltas floats cannot resolve in fewer): a shifted delta within the tie width of its pair (:func:`tie_widths`, kept
from the original deltas) of 0 is a tie, so a delta tied at 0 counts again once ``theta`` moves off 0. Exact, a sign
pattern counts when its sum falls short of the observed one by at most the rounding error of adding the deltas summed (:func:`_sum_slack`; :data:`EPS` at most, and no more than the
largest tie width among them),
Expand Down Expand Up @@ -167,6 +167,24 @@
return [num * (denominator // den) for num, den in ratios], math.floor(Fraction(cap) * denominator)


def _exact_hits(signs: np.ndarray, values: list[float], cap_eps: float) -> int:
"""The resamples (rows of ``signs``) whose exact ``|sum|`` reaches the observed one less its slack: the integers of
``values`` are split into limbs that cannot overflow int64 over all of them, each limb summed as an int64
product, and the limbs joined as Python ints."""
import numpy as np

ints, cap = _as_integers(values, cap_eps)
observed = _observed(ints, cap)
bits = max((abs(v).bit_length() for v in ints), default=0)

Check warning on line 178 in dokimi/dokimi/eval/sign_flip.py

View workflow job for this annotation

GitHub Actions / Mutation of changed lines

mutant survived: bump-int `0` -> `1`
s64 = signs.astype(np.int64)
total = np.zeros(signs.shape[0], dtype=object)
width = 62 - len(ints).bit_length() # n limbs of this many bits sum within int64

Check warning on line 181 in dokimi/dokimi/eval/sign_flip.py

View workflow job for this annotation

GitHub Actions / Mutation of changed lines

mutant survived: bump-int `62` -> `63`
for shift in range(0, bits, width):

Check warning on line 182 in dokimi/dokimi/eval/sign_flip.py

View workflow job for this annotation

GitHub Actions / Mutation of changed lines

mutant survived: bump-int `0` -> `1`
limbs = np.array([(1 if v >= 0 else -1) * ((abs(v) >> shift) & ((1 << width) - 1)) for v in ints], dtype=np.int64)

Check warning on line 183 in dokimi/dokimi/eval/sign_flip.py

View workflow job for this annotation

GitHub Actions / Mutation of changed lines

mutant survived: bump-int `0` -> `1`

Check warning on line 183 in dokimi/dokimi/eval/sign_flip.py

View workflow job for this annotation

GitHub Actions / Mutation of changed lines

mutant survived: negate-condition `v >= 0` -> `not (v >= 0)`

Check warning on line 183 in dokimi/dokimi/eval/sign_flip.py

View workflow job for this annotation

GitHub Actions / Mutation of changed lines

mutant survived: flip-compare `v >= 0` -> `v < 0`
total = total + (s64 @ limbs).astype(object) * (1 << shift)

Check warning on line 184 in dokimi/dokimi/eval/sign_flip.py

View workflow job for this annotation

GitHub Actions / Mutation of changed lines

mutant survived: swap-addsub `total + (s64 @ limbs).astype(object) * (1 << shift)` -> `total - (s64 @ limbs).astype(object) * (1 << shift)`

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI check Test Linux Wheels Python 3.11 fails: The added 1e300 interval test repeatedly enters the new exact-sum loop during bisection and exceeds the 60-second test timeout. Sparse integer limbs still trigger full matrix products and object-array arithmetic, including limbs that contribute zero. Proposed fix, not run by the bot.

CI check Test Linux Wheels Python 3.14 fails for the same reason.

CI check Tests (py3.11) fails for the same reason.

CI check Tests (py3.14) fails for the same reason.

Suggested change
total = total + (s64 @ limbs).astype(object) * (1 << shift)
if np.any(limbs):
total = total + (s64 @ limbs).astype(object) * (1 << shift)

return sum(abs(t) >= observed for t in total.tolist())


@functools.cache
def _patterns(k: int) -> np.ndarray:
"""Every sign pattern of k deltas, one per row: ``2**k x k``, as floats for the product."""
Expand Down Expand Up @@ -199,7 +217,8 @@
on, a sign pattern whose sum ties the observed one in exact arithmetic (magnitudes that repeat, such as 1e9 + 0.1)
can round to one side of it in the product and to the other in the exact sums of :func:`sign_flip_p`, so ``p(0)`` can differ there: the
row's own p, not ``p(0)``, places 0 (:func:`sign_flip_interval`). Up to :data:`EXACT_MAX` shifted deltas that span more than
40 binary orders of magnitude are summed exactly too, as floats lose the small ones there; more of them are not (#865)."""
40 binary orders of magnitude are summed exactly too, as floats lose the small ones there; so are the Monte Carlo
resamples of more of them (:func:`_exact_hits`), whose p then follows :func:`sign_flip_p`'s integer sums."""

def __init__(self, deltas: list[float], seed: int = 0, n_resamples: int = N_RESAMPLES, *,
eps: float | Sequence[float] = EPS) -> None:
Expand Down Expand Up @@ -232,6 +251,8 @@
least = min(observed, float(sums[0]), float(sums[-1])) # the product rounds them otherwise from 1e6 on
return int(np.count_nonzero(sums >= least - eps)) / 2 ** moved.size
signs = _draws(self.seed, self.changed, self.d.size, self.n_resamples)
if spread > 40: # floats lose the small deltas: count the resamples on exact integer sums

Check warning on line 254 in dokimi/dokimi/eval/sign_flip.py

View workflow job for this annotation

GitHub Actions / Mutation of changed lines

mutant survived: bump-int `40` -> `41`
return (_exact_hits(signs, x.tolist(), float(self.widths[~tied].max())) + 1) / (self.n_resamples + 1)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bug · medium
Must-fix — Impact: tied deltas inflate the rounding slack, so this branch can count extra resamples and disagree with sign_flip_p even at theta=0. Trigger: a widely spread Monte Carlo row with ties, e.g. row = [4096.0] + [2e-9] * 7 + [2.1e-9] + [-2e-9] * 8 + [0.0] * 300. Evidence: _exact_hits passes all 317 entries to _observed, whose slack multiplies by len(ints), whereas sign_flip_p removes ties and uses 17 entries. The slack becomes about 2.88e-10 instead of 1.55e-11; resamples with a balanced small-delta sum of -1e-10 now count against the observed +1e-10. This adds roughly 0.098 to the p-value distribution and breaks the promised same-seed equality. Workaround/fix: retain the existing draws but exclude tied columns and values before exact counting; add a regression with tied deltas. Severity: medium, because this is a bounded but material statistical-result error.

Check run: inconclusive, the result is not shown as a pass.

Suggestion:

Suggested change
return (_exact_hits(signs, x.tolist(), float(self.widths[~tied].max())) + 1) / (self.n_resamples + 1)
return (_exact_hits(signs[:, ~tied], x[~tied].tolist(), float(self.widths[~tied].max())) + 1) / (self.n_resamples + 1)

return (int(np.count_nonzero(np.abs(signs @ x) >= observed - eps)) + 1) / (self.n_resamples + 1)


Expand Down
27 changes: 27 additions & 0 deletions dokimi/tests/test_sign_flip_mc_spread.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Copyright (C) 2026 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

"""The Monte Carlo ends of a row whose deltas span more than a float sum resolves (#865)."""

from __future__ import annotations

import pytest

from dokimi.eval.sign_flip import ShiftTest, sign_flip_interval, sign_flip_p


@pytest.mark.parametrize("big", [1e16, 1e300])
def test_a_large_delta_does_not_hide_the_small_ones_at_the_ends(big):
row = [big] + [1.0] * 19 # 20 deltas: Monte Carlo, and 1.0 vanishes next to big in a float sum
p = sign_flip_p(row)
lo, _ = sign_flip_interval(row, p, rounded=False)
# the small deltas are the sign-flip evidence: shifting by 1 ties them all, so the lower end is 1, not ~0
assert (p, lo) == (pytest.approx(5e-5, abs=1e-6), pytest.approx(1.0, abs=1e-6))
test = ShiftTest(row)
assert test.p(0.0) == sign_flip_p(row, exact=True)


def test_exact_resamples_of_deltas_of_both_signs_and_several_limbs_match_sign_flip_p():
row = [-1e30, 7.5, -3.25, 2.0, -1.0, 4.5, 1e-3, -6.0, 8.0, 0.5, -2.5, 3.0, 1e20, -9.0, 5.0, 1.5, -4.0, 6.5, 2.5]
for seed in (0, 3):
assert ShiftTest(row, seed).p(0.0) == sign_flip_p(row, seed=seed, exact=True)
Loading