Repository navigation
fix(eval): interaction_test reads contrasts of tiny-scale scores by their values #868
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| ### Fixed | ||
|
|
||
| - **`interaction_test` no longer reads contrasts of tiny-scale scores as ties** — eight evaluations whose `a+b` scores were `g * 1e-300` (g in 2, 4, 3, 5, 6, 9, 7, 8) and 0 for the other three arms gave `p` null, `n_changed` 0 and `significant` null, where the same scores at `1e-3` give `p` 0.0078, `n_changed` 8 and `significant` true, because every contrast was within the absolute 1e-9 tie width. The contrast `AB − A − B + baseline` of a question now ties within 1e-9 times the largest of its four scores when that is below 1 (1e-9 for 1 or more, as before), and each arm delta of the row (`delta_a`, `delta_b`, `delta_combined`) uses the larger score of its pair, as `paired_test` does. Scores of 1 or more give the same rows as before. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| # Copyright (C) 2026 Intel Corporation | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| """The sign-flip tie width of an interaction row follows the scores of the row (issue #821). | ||
|
|
||
| The rule is the one of docs/benchmarking-and-comparison.md ("A delta within 1e-9 of 0 is a tie ... unless the values | ||
| of its pair are below 1"), applied to the contrast ``AB - A - B + baseline`` with the largest of the four scores as | ||
| the scale of its float noise, and to each arm delta with the larger score of its pair. | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| import pytest | ||
|
|
||
| from dokimi.eval.interactions import interaction_test | ||
| from dokimi.eval.stats import paired_test | ||
|
|
||
| G = [2, 4, 3, 5, 6, 9, 7, 8] | ||
| ARMS = {"arm_a": "a", "arm_b": "b", "combined_arm": "a+b", "baseline_arm": "baseline"} | ||
|
|
||
|
|
||
| def _ev(qid: str, ab: float, a: float, b: float, base: float) -> dict: | ||
| scores = {"a+b": ab, "a": a, "b": b, "baseline": base} | ||
| return {"question_id": qid, "scores": {k: {"aggregate": v} for k, v in scores.items()}} | ||
|
|
||
|
|
||
| def _gains(scale: float, arm: str) -> list[dict]: | ||
| rows = [] | ||
| for i, g in enumerate(G): | ||
| vals = {"a+b": 0.0, "a": 0.0, "b": 0.0, "baseline": 0.0} | ||
| vals[arm] = g * scale | ||
| rows.append(_ev(f"q{i}", vals["a+b"], vals["a"], vals["b"], vals["baseline"])) | ||
| return rows | ||
|
|
||
|
|
||
| @pytest.mark.parametrize("scale", [1e-3, 1e-300]) | ||
| def test_a_contrast_of_tiny_scores_is_a_change_with_a_p_at_any_scale(scale): | ||
| res = interaction_test(_gains(scale, "a+b"), **ARMS) | ||
| assert res is not None | ||
| assert res["n_changed"] == 8 | ||
| assert res["p"] == 0.0078 | ||
| assert res["significant"] is True | ||
|
|
||
|
|
||
| def test_the_tie_width_of_a_row_follows_its_largest_score(): | ||
| # 1e-14 on scores of 1e-3 is float noise (a tie), 1e-6 on scores of 100 is a change | ||
| tie_row = _ev("t", 1e-3 + 1e-14, 1e-3, 1e-3, 1e-3) | ||
| change_row = _ev("c", 100 + 1e-6, 100.0, 100.0, 100.0) | ||
| big = [_ev(f"q{i}", g * 1e-3, 0.0, 0.0, 0.0) for i, g in enumerate(G)] | ||
| tie = interaction_test([*big, tie_row], **ARMS) | ||
| change = interaction_test([*big, change_row], **ARMS) | ||
| assert tie is not None and change is not None | ||
| assert tie["n_changed"] == 8 | ||
| assert change["n_changed"] == 9 | ||
|
|
||
|
antonwolfy marked this conversation as resolved.
Resolved
|
||
|
|
||
| def test_the_scale_of_a_contrast_is_the_largest_of_all_four_scores_not_of_ab_and_baseline(): | ||
| # the contrast is -7e-13 and `a` is the largest score: a tie at 1e-12 (EPS times a); EPS times max(ab, base) = 5e-13 would count a change | ||
| row = _ev("x", 5e-4, 1e-3 + 7e-13, 0.0, 5e-4) | ||
| big = [_ev(f"q{i}", g * 1e-3, 0.0, 0.0, 0.0) for i, g in enumerate(G)] | ||
| res = interaction_test([*big, row], **ARMS) | ||
| assert res is not None | ||
| assert res["n_changed"] == 8 | ||
|
|
||
|
|
||
| @pytest.mark.parametrize("scale", [1e-3, 1e-300]) | ||
| @pytest.mark.parametrize("arm", ["a", "b", "a+b"]) | ||
| def test_an_arm_delta_of_tiny_scores_is_read_like_the_arms_table_reads_it(arm, scale): | ||
| res = interaction_test(_gains(scale, arm), **ARMS) | ||
| assert res is not None | ||
| treatment = [g * scale for g in G] | ||
| expected = paired_test(treatment, [0.0] * len(G))["delta_mean"] | ||
| key = {"a": "delta_a", "b": "delta_b", "a+b": "delta_combined"}[arm] | ||
| assert res[key] == expected | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.