Skip to content

fix(repo): mde of deltas near the float maximum is None, not an overflow warning - #822

Merged
antonwolfy merged 2 commits into
intel:mainfrom
antonwolfy:fix/mde-overflow
Nov 17, 2026
Merged

antonwolfy merged 2 commits into
intel:mainfrom
antonwolfy:fix/mde-overflow

Conversation

@antonwolfy

@antonwolfy antonwolfy commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

sign_flip_summary (repo compare) warns when the deltas are near the float maximum, under warnings.simplefilter("error"):

deltas on main now
[sys.float_info.max, 1, 1, 1, 1, 1] RuntimeWarning: overflow encountered in scalar multiply in min_detectable (compare_reach.py:26) mde is None
twelve deltas of ±1e307 RuntimeWarning: overflow encountered in multiply in _round (compare.py:756), for an mde of 9.3e306 mde is 9.276042159221303e+306, no warning

Without -W error the first gave mde inf, the second the unrounded value. The interval search no longer warns for these rows (fixed by the merged change that scales it); only the mde product and its rounding were left. repo compare drops task values past 1e150, so the CLI does not reach this; a caller of sign_flip_summary does.

Of the issue's two fix options this takes the second: mde is None when the planning figure is not finite, and the arithmetic cannot warn. Scaling the sd is not taken: mde is a planning figure, and a value above the float range has no number to show, so a scaled-back result would overflow again on the way back.

Closes #769

Changes

  • min_detectable (dokimi/dokimi/repos/compare_reach.py) adds the t quantiles and multiplies by the spread as Python floats, so a product past the range is inf with no numpy warning. It still returns a float; min_detectable(0.222, 26) and min_detectable(0.456435, 4) are unchanged.
  • _round (dokimi/dokimi/repos/compare.py) converts to a float first and returns None for an inf or nan, which is no figure to round. Its other callers (the baseline and treatment means, the per-task baseline, treatment and delta) pass finite values or None: the task values come through as_measure, which drops inf and nan.
  • The mde line of sign_flip_summary is not edited.
  • min_detectable of no task (n == 0) is nan, as on main; the Python-float division would otherwise raise ZeroDivisionError.
  • _inconclusive_note (compare.py) prints an mde of 1e6 or more in scientific notation (about ±9.276e+306 for the twelve ±1e307 deltas, not 300 digits); an mde below 1e6 prints with three decimals, unchanged.
  • Module docstring of compare.py, the contract (docs/repo-evaluation-contract.md) and the comment of the silent branch of _inconclusive_note name the float-range case among those where mde is None.
  • Tests in dokimi/tests/test_repo_compare_overflow.py: one per repro under warnings as errors, _round of inf, -inf and nan, _round of a numpy scalar near the range, min_detectable past the range and at n == 0, and the note for a large and a small mde. The existing test of the ±1e307 case no longer needs its filterwarnings("ignore") for the mde overflow, so it loses it.
  • Changelog fragment (Fixed), named with the author segment like the other fragments.

Risk

A row whose mde is beyond the float range now has mde: None instead of inf; its inconclusive note then falls to the "no mde" branch. That silence is intended and documented in the contract: with 5 or fewer changed tasks, or a Holm floor of 0.05 or more, the note names that floor (run: [sys.float_info.max, 1, 1, 1, 1, 1] with a Holm floor of 0.5 gives the Holm note); it says nothing only when the changed tasks and Holm floor could reach p < 0.05 (floor 0.0312 in the same run), where main printed ±inf. A note whose mde is 1e6 or more prints it in scientific notation. Rows with an in-range mde are unchanged, as are None for sd <= EPS and unreachable rows.

Open PRs on compare.py: #818 rewrites the mde, up, down and tied lines of sign_flip_summary (sd > tol) and #628 changes the imports of compare(); this change edits neither those lines nor compare(), only _round, the mde text of _inconclusive_note, an import math line and the module docstring, so a merge with either meets at most a textual conflict in the import block.

Testing

On a Linux review host, at 74fdba5 (on main 41c2ba9):

  • make ci-local: 31 passed, 2 failed, 14 skipped (docker steps, the matrix legs of intel-ci.yml, conda-forge, fuzz, semgrep, jobs that do not run on a PR). The 2 failures are the host's: Install gitleaks (download, exit 22) and so Scan commits (exit 127). 21744 tests passed, 19 skipped; changed-line coverage 100% (11 lines).
  • The two repros of the issue fail on main (the warnings above) and pass here. test_repo_compare_overflow.py on main's code: 7 failed, 18 passed.
  • The note and n == 0 tests on the previous head 2c4d4cb: test_inconclusive_note_prints_a_large_mde_in_scientific_notation and test_min_detectable_of_no_task_is_nan fail (a 300-digit figure, ZeroDivisionError), the other 27 pass.
  • Mutants of this round, each failing its test: the note always formatted with three decimals (the scientific-notation test); min_detectable without the n == 0 return (the nan test); _round without float() (the numpy-scalar test, with the overflow warning).
  • Mutants, each failing the tests that pin it: _round without the finite check (the inf, -inf, nan cases and the float_info.max case fail); min_detectable on numpy arithmetic again (the float_info.max case and the inf case fail with the overflow warning); _round returning None above 1e300 (the ±1e307 case fails).
  • Same-class: mean_ci and paired_test on [float max, 1, 1, 1, 1, 1] and on ±1e307 under warnings as errors give no warning (an OverflowError for mean_ci where its interval leaves the range, by design).
  • Not run: the other Python versions (3.11, 3.12, 3.14), docker steps, the gitleaks scan (its download failed).

Open questions

None.

Revert

A plain revert restores the old behaviour. Nothing is stored.

Checklist

  • Tests added/updated and pytest passes
  • ruff, mypy, and schema checks pass
  • Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
  • Self-reviewed per "Before opening a PR" in AGENTS.md
  • Each new raise/exit and test names the doc, contract or issue it traces to
  • No ad-hoc run artifacts committed
  • New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

…low warning (intel#769)

sign_flip_summary computed mde as a product of numpy float64 t quantiles and the spread, so a spread near the float maximum emitted RuntimeWarning: overflow encountered in scalar multiply (deltas [float max, 1, 1, 1, 1, 1]), and _round on a numpy float64 emitted overflow encountered in multiply for an mde of 9.3e306 (twelve deltas of plus and minus 1e307). Without -W error the mde was inf or the unrounded value.

min_detectable now does its arithmetic on Python floats, so a product past the range is inf with no warning. _round converts to float first and returns None for an inf or nan, since the planning figure has no number to show beyond the float range. Every other caller of _round passes a finite mean, delta or task value, or None, so they are unchanged. The scaled-sd option is not taken: a scaled-back mde would overflow again.

The existing test that ignored the overflow warning for the 1e307 case no longer needs the filter and loses it.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@exai-bot

exai-bot commented Nov 16, 2026 •

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 5 selected item(s).

CI at 74fdba5: 29 passing, 3 skipped.

Review details
  • Commit reviewed: 74fdba5 against merge base 41c2ba9
  • Coverage: 5 files in 1 groups
  • Engine: OpenCodeReview v1.13.5-exai.4.8f73eada, us.openai.gpt-6.1-sol
  • Usage: 70k tokens (41k from cache), 20s
  • Tool calls: 8 (4 file_read, 1 ci_status, 1 code_search, 1 linked_items, 1 local_check_results)
  • Repository comparison MDE float-range handling and overflow coverage: 5 files

@codecov-commenter

codecov-commenter commented Nov 16, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.75%. Comparing base (41c2ba9) to head (74fdba5).
⚠️ Report is 14 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #822      +/-   ##
==========================================
- Coverage   95.76%   95.75%   -0.02%     
==========================================
  Files         416      423       +7     
  Lines       55624    56424     +800     
  Branches     9553     9716     +163     
==========================================
+ Hits        53267    54027     +760     
- Misses       1456     1489      +33     
- Partials      901      908       +7     
Files with missing lines Coverage Δ
dokimi/dokimi/repos/compare.py 99.52% <100.00%> (+<0.01%) ⬆️
dokimi/dokimi/repos/compare_reach.py 100.00% <100.00%> (ø)

... and 35 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@napetrov napetrov left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change class: not computed (no receipt profile for this repo, or receipt.py failed)

Verdict: approve. No blocking findings; two nits and one question. Head 2c4d4cb, merge base 41c2ba9, lane L1. Ran: probes probes/probe_note.py, probes/probe_n0.py on base and head (scratch venv with the package deps); the passes ran four overflow and compare test files on head (180 passed, reported by the pass, not re-run by me).

CI: 0 failing Actions jobs on head 2c4d4cb; merge base is behind base tip (not a red).

# Finding Label Provenance Scope Probe Expected
1 Inconclusive note prints a 300-digit mde for a ±1e307 spread nit changed new surface probes/probe_note.py a note a reader can use; base printed ±inf
2 min_detectable(sd, 0) raises ZeroDivisionError where main returned nan nit changed contract probes/probe_n0.py nan, or the change documented; C3 claims the in-range values only

Raised before

No prior row makes a claim (the bot summary and the coverage notice are dropped).

Design

None.

Questions

  • question: the silence for mde None (the PR's own Risk section). A row whose mde leaves the float range now gets no note even when a finding could reach p < 0.05 (p_holm_floor 0.0312 in the probe); base printed ±inf, equally uninformative. Is the silence intended, or should the branch say the power figure is beyond range? The code comment in that branch also reads as if mde were always finite.

Coverage

Checked: whole-PR pass and edge pass on both trees; probes of the note output and min_detectable at n=0 on base and head; CI summary. Not checked: the full test suite, ruff, mypy, Python 3.11/3.14 legs.

Follow-up issues

  • Changelog fragment 20261116_130000_mde-float-range.md has no author segment; the other fragments in changelog.d/ use <timestamp>_<author>_<branch>.md. Mechanical rename; the other antonwolfy fragments carry the segment.

Comment thread dokimi/dokimi/repos/compare.py Resolved
Comment thread dokimi/dokimi/repos/compare_reach.py Resolved

@vchamarthi vchamarthi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Not approved yet · reviewed 2c4d4cb

Next: @antonwolfy: answer 2 open threads from napetrov and settle 1 open question (below). 1 nit optional.

Blocking

  • Still holds on this head: thread by @napetrov – On the head, _inconclusive_note prints a 300-digit figure for twelve deltas of plus and minus 1e307 (mde 9.276042159221303e+306); main printed ±inf.
  • Still holds on this head: thread by @napetrov – min_detectable(0.5, 0) raises ZeroDivisionError on the head where main returns nan, and only a direct call reaches it.
  • Open question for @antonwolfy: A row whose mde is None past the float range, with its interval including 0 and a Holm floor below 0.05, prints no inconclusive note (compare.py:678-679). State in the PR description that such rows are silent, or make the branch say that the figure is beyond the float range.

Nits

  • dokimi/docs/repo-evaluation-contract.md:2507 – [sys.float_info.max, 1, 1, 1, 1, 1] gives mde None, which the contract's list of null cases does not name.

    Input [sys.float_info.max, 1, 1, 1, 1, 1] gives mde None on the head, where main gave inf. The contract says null is for one task or deltas with no spread, and then no note. For that input the inconclusive note is not silent when the Holm floor is 0.5 (compare.py:680-683). Expected: the list names the float-range case, and "no note" is said only of the cases that have none.

    Suggestion: Name the float-range case in that list, and say what the note does for it.

    Patch, run on 2c4d4cb: the quoted text is gone; tests are no worse.

    Patch
    --- a/dokimi/docs/repo-evaluation-contract.md
    +++ b/dokimi/docs/repo-evaluation-contract.md
    @@ -2504,8 +2504,8 @@
       `(t(0.975, n-1) + t(0.80, n-1)) * sd / sqrt(n)` over the per-task deltas, of
       the mean delta a one-sample t test at the uncorrected two-sided 5% level
       detects with 80% power at that spread, the t quantiles at `n-1` degrees of
    -  freedom (every row has `mde` under `data` in `--json`; `null` for one task or deltas with
    -  no spread, and then no note). It is not the power of the sign-flip test nor
    +  freedom (every row has `mde` under `data` in `--json`; `null` for one task, for deltas with no spread
    +  (then no note), or for a figure that leaves the float range (the note below, if any). It is not the power of the sign-flip test nor
       of the Holm-corrected decision; a Holm-calibrated figure would depend on the
       other rows' observed p, which a figure for planning the next run's task count
       cannot know. The sign-flip p over k changed (non-tied)

Follow-ups, not blocking

  • dokimi/dokimi/repos/compare.py:759 – Removing the float() in _round (compare.py:759) leaves all 25 cases green, since no test passes a numpy scalar.

Verified on 2c4d4cb: ✅ changed tests: 25 passed, fail on main · ✅ full suite with main merged in: 22222 passed · ✅ merges onto main · ✅ 3 broken variants of the changed code: 3 caught by the tests · 5 mutations tried · 16 edge inputs tried

20 checks behind it, each with its evidence
Area What holds How
Float-range mde sign_flip_summary on [sys.float_info.max, 1, 1, 1, 1, 1] gives mde None with warnings as errors, pinned by test_mde_is_none_when_the_planning_figure_leaves_the_float_range tests on the head, edge input (numbers)
Non-finite rounding _round gives None for inf, -inf and nan, pinned by test_round_gives_no_figure_for_a_non_finite_value; main gives the value itself tests on the head, edge input (numbers)
Large spread mde Twelve deltas of plus and minus 1e307 give mde about 9.276e306 with no warning under warnings as errors, pinned to 0.1% by test_mde_of_a_large_spread_is_rounded_without_overflowing tests on the head, edge input (numbers)
Min_detectable past range min_detectable(sys.float_info.max, 3) is inf with no warning, pinned by test_min_detectable_past_the_range_is_inf_without_a_warning tests on the head, edge input (numbers)
In-range mde min_detectable(0.222, 26) and min_detectable(0.456435, 4) keep the main values, which the reading tests pin to 1e-6, and the full test run with main merged in has 0 failures full suite, main merged in
Static checks The static checks, mypy and changed-line coverage steps pass with main merged in full suite, main merged in
None input _round(None) gives None, the value main gives ran it, edge input (none)
Negative input _round(-2.5) gives -2.5 on both the head and main ran it, edge input (numbers)
Empty mean _round(_mean([])) gives None, the value main gives ran it, edge input (numbers)
Int input _round(3) gives 3.0, a float; main gives the int 3 ran it, edge input (numbers)
Numpy scalar input _round(np.float64(1.0e307)) gives 1e307 with no warning under warnings as errors; main raises RuntimeWarning ran it, edge input (numpy_scalar)
As_measure bound as_measure gives None for inf, -inf, nan and 1e151, so the values _mean and _round get from the comparison are finite and within 1e150 ran it, edge input (numbers)
One task sign_flip_summary on one task gives mde None, so the one-task path does not reach min_detectable ran it, edge input (numbers)
Negated none check Negating the None check in _round fails the changed tests broke the code, tests caught it, edge input (none)
Rounding returns none Making _round return None for every value fails the changed tests broke the code, tests caught it, edge input (numbers)
Quantile float Dropping the float() around the t quantiles at compare_reach.py:27 fails 2 of the 25 changed cases broke the code, tests caught it, edge input (numbers)
Sd float Dropping float(sd) at compare_reach.py:28 leaves the changed tests green, since sd is a Python float from statistics.stdev at compare.py:587 broke the code, tests caught it, edge input (numbers)
Merge The PR merges onto main with no conflict merge onto main
Changed tests on head All 25 cases of tests/test_repo_compare_overflow.py pass on the head tests on the head
Changed tests on main 7 of the 25 cases fail on main: the float-range, large-spread, min_detectable and non-finite cases, so they pin the change same tests on main

…of no task is nan

Settles the review of intel#822: the inconclusive note printed every digit of an mde near the float range (twelve deltas of ±1e307 gave a 300-digit figure); it now prints an mde of 1e6 or more as `9.276e+306`, and one below 1e6 as before.

min_detectable(sd, 0) is nan again, as on main, instead of a ZeroDivisionError from the Python-float arithmetic.

The contract names the float-range case among those where mde is null and says what the note does for it, the comment of the silent branch of _inconclusive_note names it, and the changelog fragment carries the author segment of its name.

Tests pin the note for a large and a small mde, min_detectable at n=0, and _round of a numpy scalar near the range.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@antonwolfy

Copy link
Copy Markdown
Contributor Author

(approved) Change class: not computed (no receipt profile for this repo, or receipt.py failed)

@napetrov (#822 (review)): Threads 4591728280 and 4591728292 are answered above. The question on the silence for an mde of None: it is intended where the changed tasks and the Holm floor could reach p < 0.05 (the base printed ±inf, no more useful); the Risk section of the description now says so, and says that with a Holm floor of 0.5 the same input gets the Holm note. The comment of that branch now names the float-range case. The changelog fragment is renamed with the author segment, 20261116_130000_antonwolfy_mde-float-range.md. Commit 74fdba5.

@antonwolfy

Copy link
Copy Markdown
Contributor Author

(commented) 🟡 Not approved yet · reviewed 2c4d4cb

@vchamarthi (#822 (review)): The two threads are answered above. The open question is settled in the description's Risk section (the silence is intended; where a floor keeps p above 0.05 the note names it, shown with a Holm floor of 0.5). The contract now names the float-range case among those where mde is null and says what the note does for it (repo-evaluation-contract.md), in commit 74fdba5. The follow-up on _round's float() is covered by test_round_of_a_numpy_scalar_near_the_range_does_not_warn, which fails with the overflow warning when the float() is dropped.

@antonwolfy
antonwolfy merged commit b524769 into intel:main Nov 17, 2026
32 checks passed
@antonwolfy
antonwolfy deleted the fix/mde-overflow branch November 17, 2026 00:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

repo compare: sign_flip_summary warns (overflow) computing mde for deltas near the float maximum

5 participants