Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
### Fixed

- **The sign-flip test no longer reads deltas between values of a tiny scale as ties** — `paired_test([2e-300, 4e-300, ...], [1e-300] * 8)` had `n_changed` 0 and no `p` next to a `large` effect size, because every delta was within the absolute 1e-9 tie width. A pair whose larger value is below 1 now ties within 1e-9 times that value (1e-9 for values of 1 or more, as before), since float noise in `t - b` scales with the values and not with the delta: that vector gets a `p`, an interval and the verdict of its 1e-3 scale copy, and `0.1 + 0.2` against `0.3` stays a tie. Each pair keeps its own width whatever the other deltas are. `paired_test` and `repo compare` (`n_changed`, `up`, `down`, `tied`, `mde`) use it; a caller that passes deltas alone keeps the absolute 1e-9. An interval of deltas near the smallest floats excludes 0 with its `p`, where it was `[0, 0]`.
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
### Fixed

- **An unchanged pair of large values no longer widens the sign-flip slack on the sums of small changed pairs** — `paired_test([1e-12] * 8 + [1.0], [0.0] * 8 + [1.0])` gave `p` 1.0 and no significance, where the eight small pairs alone give 0.0078125, because the slack on a pattern's sum followed the largest value of any pair, tied ones included. The slack now follows the pairs that are summed (the changed ones, and at a shift of the interval search the deltas that are not ties there), in the test and in the interval that inverts it.
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
### Fixed

- **The sign-flip interval starts from a shift the test accepts** — eight pairs of `1e-12` beside one tied pair of `1.0` gave an interval of `8.9e-13` at both ends, a shift the test rejects, where the eight pairs alone give `1e-12`. A tied delta pulled the mean off the one accepted shift, so the search returned its starting point unchecked; it now starts from the mean of the changed deltas when the mean of all is rejected.
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
### Fixed

- **The sign-flip slack on a sum is rounding error, not a tie width** — a ninth changed pair just past its tie width (`1.0 + 1.1e-9` against `1.0`) beside eight pairs of `1e-12` gave `p` 1.0, because its width of `1e-9` let every sign pattern count; it now gives `2/2**9`. The interval starts from an accepted shift when the mean of the deltas is rejected (it could return two rejected ends), and the `mde` of `repo compare` reads the spread at the scale of the changed pairs, so an unchanged pair of large values no longer removes it.
5 changes: 4 additions & 1 deletion dokimi/docs/benchmarking-and-comparison.md
Original file line number Diff line number Diff line change
Expand Up @@ -369,7 +369,10 @@ Each row's `p` is one two-sided sign-flip test on the per-question deltas: under
the null each delta's sign is a fair coin, and p is the share of sign patterns
whose |sum| reaches the observed one (exact up to 16 changed pairs, Monte Carlo
with a fixed seed above). A delta within 1e-9 of 0 is a tie and carries no sign,
so over k changed pairs (`n_changed`) p cannot go below 2/2^k: at 5 or fewer no
unless the values of its pair are below 1: float noise in a delta scales with the
values subtracted, so a pair whose larger value is m below 1 ties within 1e-9 times m
(a score of 0.3 against 0.1 + 0.2 is a tie; 1e-12 against 0 is a change, as 1e-3 against 0
is). Each pair keeps its own width whatever the other deltas are, except a pair of zeros, which takes the width of the largest value among all the pairs (at most 1). So over k changed pairs (`n_changed`) p cannot go below 2/2^k: at 5 or fewer no
data whatsoever reaches 0.05. The test runs from 2 pairs and its p is printed.
An arm under that floor can never be significant, so it stays out of the
multiplier, as a `repo compare` row does: it counts in `family_size` but not in
Expand Down
163 changes: 126 additions & 37 deletions dokimi/dokimi/eval/sign_flip.py

Large diffs are not rendered by default.

24 changes: 18 additions & 6 deletions dokimi/dokimi/eval/stats.py
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,16 @@
from typing import Any, Callable, Dict, Iterable, List, Optional, Sequence, Tuple

from dokimi.eval.finite_delta import check_sum_range, checked_delta
from dokimi.eval.sign_flip import ALPHA, EPS, METHOD, min_sign_flip_p, round_mean, sign_flip_row, within
from dokimi.eval.sign_flip import (
ALPHA,
METHOD,
min_sign_flip_p,
pair_magnitudes,
round_mean,
sign_flip_row,
tie_widths,
within,
)
from dokimi.records import BASELINE, CONTEXT, RecordLike, records

#: Checked without importing scipy, whose OpenBLAS starts a thread per core
Expand Down Expand Up @@ -565,13 +574,15 @@ def _cohens_d(d_mean: float, d_sd: float) -> Optional[float]:


def _describe_deltas(
deltas: Sequence[float], *, confidence: float, seed: int
deltas: Sequence[float], *, confidence: float, seed: int, magnitudes: Optional[Sequence[float]] = None
) -> Dict[str, Any]:
"""Effect size, the sign-flip p, and the interval that inverts it, for a delta vector.

``p`` is the two-sided sign-flip p over the deltas not within ``EPS`` of 0
(:func:`dokimi.eval.sign_flip.sign_flip_p`, exact up to 16 of them, Monte
Carlo from ``seed`` above), left unrounded for the caller to correct and
Carlo from ``seed`` above), or not within ``EPS`` times the larger value of
their pair when ``magnitudes`` (:func:`dokimi.eval.sign_flip.pair_magnitudes`)
are given and below 1, left unrounded for the caller to correct and
then :func:`_finalize`. Correcting an already-rounded p multiplies its
quantization error by the family size: a true p of 0.016666 in a family of
three gives an exact Holm p of 0.049998 (significant), but
Expand Down Expand Up @@ -604,11 +615,11 @@ def _describe_deltas(
# The sign-flip sums would overflow and read as a measured zero effect.
# A finite sum of magnitudes bounds the spread (sd**2 <= sum(x**2) <= that sum squared), so it is finite too.
check_sum_range(d)
row = sign_flip_row(d, seed=seed, exact=True)
row = sign_flip_row(d, seed=seed, exact=True, magnitudes=magnitudes)
p, lo, hi, shown_lo, shown_hi = row["p"], row["lo"], row["hi"], row["ci_lo"], row["ci_hi"]
else: # a NaN delta is no evidence either way: no p, no verdict, open ends
p, lo, hi, shown_lo, shown_hi = None, None, None, None, None
n_changed = sum(1 for x in d if abs(x) > EPS)
n_changed = sum(1 for x, w in zip(d, tie_widths(magnitudes, len(d)), strict=True) if abs(x) > w)

return {
"n": len(d),
Expand Down Expand Up @@ -660,7 +671,7 @@ def paired_test(
if len(treatment) != len(control) or len(treatment) < 2:
return None
deltas = _deltas(treatment, control)
result = _describe_deltas(deltas, confidence=confidence, seed=seed)
result = _describe_deltas(deltas, confidence=confidence, seed=seed, magnitudes=pair_magnitudes(treatment, control))
return _finalize(result) if rounded else result


Expand Down Expand Up @@ -702,6 +713,7 @@ def arm_significance(
_deltas(treat, base),
confidence=confidence,
seed=seed,
magnitudes=pair_magnitudes(treat, base),
)

ordered = [a for a in targets if a in out]
Expand Down
26 changes: 20 additions & 6 deletions dokimi/dokimi/repos/compare.py
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,17 @@
from typing import Any

from dokimi.context.consistency import Run, run_from_events
from dokimi.eval.sign_flip import ALPHA, EPS, METHOD, min_sign_flip_p, sign_flip_row, within
from dokimi.eval.sign_flip import (
ALPHA,
EPS,
METHOD,
min_sign_flip_p,
pair_magnitudes,
sign_flip_row,
tie_scale,
tie_widths,
within,
)
from dokimi.eval.stats import correct_family
from dokimi.eval.finite_delta import checked_delta, checked_mean
from dokimi.utils import same_model
Expand Down Expand Up @@ -580,9 +590,12 @@ def sign_flip_summary(pairs: list[dict[str, Any]], measure: str = "gate", seed:
Holm's p is known. ``measure`` is not read: every measure gets the same statistics.
"""
deltas = [p["delta"] for p in pairs]
magnitudes = pair_magnitudes([p["treatment"] for p in pairs], [p["baseline"] for p in pairs])
widths = tie_widths(magnitudes, len(deltas)) # a tie is within float noise of the pair's own values
changed = [m for m, d, w in zip(magnitudes, deltas, widths, strict=True) if abs(d) > w] # the spread is read at their scale
mean = checked_mean(deltas, CompareError)
try:
row = sign_flip_row(deltas, seed=seed, exact=exact)
row = sign_flip_row(deltas, seed=seed, exact=exact, magnitudes=magnitudes)
sd = statistics.stdev(deltas) if len(deltas) > 1 else 0.0
except OverflowError as exc: # finite deltas whose sum or spread leaves the float range: no test to run
raise CompareError("the deltas of the tasks are too large to test: a sum or a spread of them leaves the float range") from exc
Expand All @@ -597,11 +610,11 @@ def sign_flip_summary(pairs: list[dict[str, Any]], measure: str = "gate", seed:
"p": p,
"p_holm": None,
"p_holm_floor": None,
"mde": _round(min_detectable(sd, len(deltas))) if sd > EPS and reachable else None,
"mde": _round(min_detectable(sd, len(deltas))) if sd > EPS * tie_scale(changed) and reachable else None,
"interval_p_disagree": p is not None and includes_zero(lo, hi) == (p < ALPHA),
"up": sum(d > EPS for d in deltas),
"down": sum(d < -EPS for d in deltas),
"tied": sum(abs(d) <= EPS for d in deltas),
"up": sum(d > w for d, w in zip(deltas, widths, strict=True)),
"down": sum(d < -w for d, w in zip(deltas, widths, strict=True)),
"tied": sum(abs(d) <= w for d, w in zip(deltas, widths, strict=True)),
"tasks": [{**p, "baseline": _round(p["baseline"]), "treatment": _round(p["treatment"]),
"delta": _round(p["delta"])} for p in pairs],
}
Expand All @@ -626,6 +639,7 @@ def format_compare(summary: dict[str, Any]) -> str:
f"{r['up']}/{r['down']}/{r['tied']}")
for r in rows:
if r["measure"] in PRIMARY:
# the tasks hold rounded values: this line follows the printed deltas, which are 0 or 1e-6 and more
moved = [t for t in r["tasks"] if abs(t["delta"]) > EPS]
if moved:
lines.append(f" {r['measure']} changed: " + ", ".join(
Expand Down
Loading
Loading