Skip to content

fix(eval): interaction_test reads contrasts of tiny-scale scores by their values - #868

Merged
antonwolfy merged 2 commits into
intel:mainfrom
antonwolfy:fix/interaction-tie-width
Nov 17, 2026
Merged

antonwolfy merged 2 commits into
intel:mainfrom
antonwolfy:fix/interaction-tie-width

Conversation

@antonwolfy

@antonwolfy antonwolfy commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

interaction_test and interaction_significance gave _describe_deltas the per-question contrasts AB - A - B + baseline, and the three arm deltas of the row, without the scores they come from, so the sign-flip tie width was the absolute EPS = 1e-9 (the same cause as #762, which #818 settled for paired_test). Eight evaluations with a+b scores g * scale for g in 2, 4, 3, 5, 6, 9, 7, 8 and 0 for the other three arms:

scale p n_changed significant
1e-3 0.0078 8 True
1e-300 on main None 0 None
1e-300 here 0.0078 8 True

The arm deltas had the same cause: delta_combined of the 1e-300 row printed 5.5e-300 on main (read as a tie, so unrounded) and prints what paired_test prints for the same pairs now.

The fix is the one the issue names: the contrast passes the largest of the four scores' magnitudes of its row, since the float noise of ab - a - b + base scales with the largest of the four values (the pair rule, larger of two, extended to four). Each arm delta passes the pair magnitudes of its scores against the baseline.

Closes #821

Changes

  • dokimi/eval/interactions.py: _interaction_contrast passes magnitudes= (max(abs(ab), abs(a), abs(b), abs(base)) per row, in the order of contrasts) to _describe_deltas; _arm_delta takes the pair magnitudes (pair_magnitudes of the a, b or ab scores against the baseline) for delta_a, delta_b and delta_combined.
  • dokimi/eval/interactions.py: the interaction_test docstring states the tie width of the contrast and of each arm delta, and _interaction_contrast takes the ab, a and b columns once for _deltas and pair_magnitudes.
  • docs/benchmarking-and-comparison.md: the interaction paragraph says the tie rule applies to the contrast with the largest of its four scores and to each arm delta with its pair's larger score; its example is AB = 1e-300 with A, B and baseline all zero, a change.
  • tests/test_interactions_tie_width.py: the 1e-300 and 1e-3 rows, a tie and a change row at different scales, a row where a is the largest score (contrast -7e-13 is a tie), and the arm deltas against paired_test.
  • Changelog fragment (Fixed).

Risk

The tie width is EPS * min(1, m), which is EPS for m of 1 or more: rows at ordinary score scales (0 to 100) give the same p, n_changed, ends and verdict, and every existing test of interactions.py passes unchanged. Rows whose four scores are all below 1 can now count a contrast under 1e-9 as a change, where it was a tie. At scores near 1e-300 the printed interaction and arm deltas are now rounded to 4 decimals (0.0001), as any row with a p is; that rounding is #817, not this change.

Testing

On a Linux review host, at 42b9f4b (on main 08f393c):

  • make ci-local: 34 passed, 0 failed, 14 skipped (steps needing docker and network, the matrix legs, jobs that do not run on a PR, micromamba, semgrep and the fuzz run). 22399 tests passed, 19 skipped.
  • tests/test_interactions_tie_width.py on main's code: 4 failed, 5 passed (before the a-largest row was added; that row passes on main, whose width is absolute); the first failure is assert 5.5e-300 == 0.0001 (arm delta), and the contrast row fails on assert 0 == 8 (n_changed). Here: 9 passed.
  • Mutants, one break per copy: dropping magnitudes from the contrast fails the 1e-300 row (assert 0 == 8); dropping it from _arm_delta fails the three 1e-300 arm-delta cases (assert 5.5e-300 == 0.0001); passing abs(contrast) as the magnitudes fails the tie row (assert 9 == 8); taking the contrast's scale from max(abs(ab), abs(base)) fails the a-largest row (assert 9 == 8).
  • Merged together with fix/exact-sign-flip-sums, fix/paired-summary-overflow on main 08f393c: pytest -n auto 22418 passed, 19 skipped, mypy clean, ruff check clean.
  • Not run: Python 3.11, 3.12 and 3.14, and the CI merge commit.

Open questions

None.

Revert

A plain revert restores the absolute 1e-9 tie width of the interaction rows. Nothing is stored.

Checklist

  • Tests added/updated and pytest passes
  • ruff, mypy, and schema checks pass
  • Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
  • Self-reviewed per "Before opening a PR" in AGENTS.md
  • Each new raise/exit and test names the doc, contract or issue it traces to
  • No ad-hoc run artifacts committed
  • New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

…heir values (intel#821)

interaction_test and interaction_significance passed the per-question contrasts AB - A - B + baseline, and the three arm deltas of the row, to the sign-flip description without the values they come from, so every tie was within the absolute 1e-9. Eight evaluations with a+b scores g * 1e-300 and 0 elsewhere gave p None, n_changed 0 and significant None, where the same scores at 1e-3 give p 0.0078, n_changed 8 and significant True.

The contrast of a row now passes the largest magnitude of its four scores, and each arm delta the larger magnitude of its pair against the baseline, as paired_test does since intel#818. A tie is within 1e-9 times that value when it is below 1, and 1e-9 otherwise, so rows at ordinary score scales are unchanged. The 1e-300 rows read as their 1e-3 copies.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@antonwolfy
antonwolfy requested a review from a team as a code owner November 17, 2026 11:45
@antonwolfy antonwolfy self-assigned this Nov 17, 2026
@exai-bot

exai-bot commented Nov 17, 2026 •

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 4 selected item(s).

CI at 42b9f4b: 25 passing, 3 skipped.

Review details
  • Commit reviewed: 42b9f4b against merge base 08f393c
  • Coverage: 4 files in 1 groups
  • Engine: OpenCodeReview v1.13.5-exai.4.8f73eada, us.openai.gpt-6.1-sol
  • Usage: 197k tokens (132k from cache), 39s
  • Tool calls: 20 (12 file_read, 5 code_search, 1 ci_status, 1 file_find, 1 local_check_results)
  • Interaction tie-width behavior, tests, and documentation: 4 files

@codecov-commenter

codecov-commenter commented Nov 17, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.76%. Comparing base (08f393c) to head (42b9f4b).
⚠️ Report is 1 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #868   +/-   ##
=======================================
  Coverage   95.76%   95.76%           
=======================================
  Files         424      425    +1     
  Lines       56511    56477   -34     
  Branches     9730     9718   -12     
=======================================
- Hits        54115    54084   -31     
+ Misses       1487     1486    -1     
+ Partials      909      907    -2     
Files with missing lines Coverage Δ
dokimi/dokimi/eval/interactions.py 98.50% <100.00%> (+0.12%) ⬆️

... and 8 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at 8b00389. The fix is right: the contrast now ties within 1e-9 times the largest of its four scores, and each arm delta uses the same pair rule as paired_test, so the 1e-300 rows read like their 1e-3 copies. One test gap is in the line comments.

Agreed with @exai-bot on the impossible 1e-12 example in the docs.

What I ran, at 8b00389:

  • make ci-local --offline: 0 failed, 23 passed, 20 skipped; the test suite 22,345 passed, 72 skipped.
  • A mutation that takes the contrast's scale from ab and baseline only: all 9 new tests still pass; the row in the line comment catches it.
  • The new test file on the merge-base: the 1e-300 cases fail, so they cover the fix.

Checked, no issues

  • Call sites: interaction_test and interaction_significance both go through _interaction_contrast, so the corrected family gets the same tie widths.
  • Rows at ordinary scales: scores of 1 or more keep the absolute 1e-9 width, as the changelog says; below 1 the widths follow #818's rule.
  • Changelog: a Fixed fragment that names the case and the new rule.

Comment thread dokimi/dokimi/eval/interactions.py Resolved
…ale (intel#868)

Settles the review of intel#868: the tie-rule example in benchmarking-and-comparison.md named a contrast of 1e-12 that scores of 1e-300 cannot produce, and now uses AB = 1e-300 with A, B and baseline zero.

A new test row where `a` is the largest score (contrast -7e-13, a tie at 1e-12) fails when the contrast's scale is taken from `ab` and `baseline` only.

The `interaction_test` docstring states the tie width of the contrast and of each arm delta, and `_interaction_contrast` takes the ab, a and b columns once for both `_deltas` and `pair_magnitudes`.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@antonwolfy

Copy link
Copy Markdown
Contributor Author

(commented) Review at 8b00389. The fix is right: the contrast now ties within 1e-9 times the largest of its

@vlad-perevezentsev (#868 (review)): The review body recaps its checks and points at its two threads: the impossible docs example (4599152107) and the missing a-largest row (4599541296), both addressed in 42b9f4b.

@vlad-perevezentsev vlad-perevezentsev left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

I checked 42b9f4b after the update. The new test pins the four-score scale of the contrast, the docstring states the rule, and the docs example is now possible. The fix itself holds as in the first round.

Merge: safe now: it merges cleanly into current main, 5 commits behind, and main has not touched its files since the base. Once it is in, #867 conflicts with it in interactions.py and needs a rebase.

What I ran, at 42b9f4b:

  • make ci-local --offline: 22,346 passed, 72 skipped. schema_compat fails because the branch is behind main, which added repo_event.v1.json after this base. It is not from this PR.
  • The new row fails with a contrast scale of max(abs(ab), abs(base)) (n_changed 9 instead of 8).

@antonwolfy
antonwolfy merged commit 2b1a5bf into intel:main Nov 17, 2026
28 checks passed
@antonwolfy
antonwolfy deleted the fix/interaction-tie-width branch November 17, 2026 17:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

interaction_test reads contrasts of tiny-scale scores as ties: no p, n_changed 0

4 participants