Repository navigation
fix(eval): paired_summary returns no delta where a paired difference leaves the float range (#830) - #869
Conversation
…leaves the float range (intel#830) paired_summary formed its deltas through checked_delta, which raises OverflowError for two finite scores such as 1.7e308 and -1.7e308. The evaluate pipeline stopped after saving the evaluations and before the report, and `report generate` and `report eval` failed. paired_summary now returns the arm means, ranges and counts with delta_mean, delta_min and delta_max set to None, and delta_overflow counts the pairs without a delta. A pair without a delta is counted as an improvement or a degradation by comparing its two scores. The pipeline prints delta=none, `orchestrate` prints an em dash, the reports print an em dash with a one-line note, and `report eval` states that its significance test was refused instead of stopping. paired_test, paired_delta and the arms path still raise as before. Test-narrowed: test_paired_summary_paired_delta_and_arm_significance_refuse_the_same_deltas pinned the raise of paired_summary that this change removes on purpose; it is replaced by test_paired_delta_and_arm_significance_refuse_deltas_beyond_the_range_while_paired_summary_has_none, which keeps the two raises that stay and asserts paired_summary returns no delta. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
… of the eval report for a summary with no delta (intel#830) The `Delta:` print of `cmd_evaluate` for a summary whose `delta_avg` is None and the `Dynamic vs Static agreement` bullet when one set has no delta had no test. One test each now: `cmd_evaluate` with a mocked pipeline result, and `generate_report` on an evaluations file with a golden set and a generated set where the generated one overflows. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 1 selected item(s). CI at Review details
|
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #869 +/- ##
==========================================
+ Coverage 95.76% 95.78% +0.02%
==========================================
Files 424 425 +1
Lines 56511 56503 -8
Branches 9730 9724 -6
==========================================
+ Hits 54115 54120 +5
+ Misses 1487 1479 -8
+ Partials 909 904 -5
🚀 New features to boost your workflow:
|
vlad-perevezentsev
left a comment
There was a problem hiding this comment.
Review at 72eff4a. paired_summary, the pipeline line, cmd_evaluate and both reports now handle a pair whose difference leaves the float range. But the trust gate that report generate runs on the same data still computes its own delta and spread, so the fix of #830 is not complete yet.
Agreed with @exai-bot on the trust checks; I added a crash case in its thread.
What I ran, at 72eff4a:
make ci-local --offline: 22,346 passed, 0 failed, 72 skipped.- The new test file on the merge-base: 9 of its 10 tests fail there. The one that passes,
..._pairs_inside_the_range_still_has_a_delta, guards behaviour that did not change. ReportGenerator.generate_reportwith[(1.7e308, -1.7e308), (-1.7e308, 1.7e308)]:OverflowErrorfromstatistics.stdevinevaluate_trust. With three(1.7e308, -1.7e308)pairs:min_deltapasses oninf, and the JSON report hasInfinity.- This branch merged with #867 (same
stats.pyandtest_stats_float_range.py): both test files pass, 41 tests.
Checked, no issues
- Counts: an overflowing pair is counted as an improvement or a degradation by comparing its scores, and never as neutral. That is right, since its scores differ.
- Callers: every reader of
delta_mean/delta_avg(pipeline,cmd_evaluate,eval_reportSummary/Overall/agreement lines, generator stats, clusters, sources, advice) handlesNone. The grouped tables sort aNonedelta last. report evalsignificance: anOverflowErrorfromsignificance_testbecomes a "Not computed: " line, not a number.- Changelog: the fragment is under Fixed and says what changed for the user.
… not measured Settles the review point that evaluate_trust still subtracted the arm means directly, so a report whose paired delta is unavailable (three pairs of 1.7e308 and -1.7e308) wrote Infinity as the min_delta value and marked it passed. The check now has value None, measured False and a "Not measured" message when the difference of the means is not finite, so the JSON stays standard and the verdict lists min_delta under not_measured. A regression test parses the report JSON with non-finite constants rejected. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
Settles review thread 4599606890: the report note read "1 complete pairs differ" for the issue's own input. It now reads "1 complete pair differs" for one pair and "N complete pairs differ" otherwise, and the tests pin both. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
…t to the one that has none (intel#830) The codecov concern on generator.py:526 stays open because the PR's own test file only ever takes the no-delta branch of `_overall_advice`; the other branch was reached by unrelated test files only. The new test draws the hurting and minimal-impact conclusions from a delta that exists and asserts the float-range line is absent, so both branches of the `delta.avg is None` check are pinned by tests/test_paired_summary_overflow.py itself. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
|
Ready for another look at Declined, with the evidence in the reply:
Follow-up issues opened:
|
|
Correction to the summary above: the codecov point is not declined. It was fixed in 4b9e0c6, which adds a test that takes the other branch of the |
vlad-perevezentsev
left a comment
There was a problem hiding this comment.
LGTM
I checked 4b9e0c6. c97bcee makes the trust gate report min_delta as not measured when the arm means differ by more than the float range, so the JSON report is standard again. The score_cv crash on opposite-signed scores is tracked in #870, which is fine with me. 6462ca4 fixes the singular in the note.
Merge: safe now. It merges cleanly into current main, 5 commits behind, and its tests pass on the merge (main has changed generator.py since the base). No open PR conflicts with it.
What I ran, at 4b9e0c6:
make ci-local --offline: 22,349 passed, 0 failed, 72 skipped. The only FAIL row isschema_compat, which compares against the newermain(repo_event.v1.json,judge_panelinnovelty.v1.json). This branch does not touch the schemas, so it is not from this PR.- On the merge with current
main:test_paired_summary_overflow.py,test_stats_float_range.py, both trust gate tests andtest_report_generator_labels.py: 108 passed. evaluate_truston three(1.7e308, -1.7e308)pairs:min_deltahasvaluenull andmeasuredfalse, and the JSON parses with non-finite constants rejected.
Checked, no issues
- Not measured, not a pass: the new branch uses
measured=False, the same pattern asretrieval_hit_rate, and the verdict lists it undernot_measured. The difference of two finite means can only be infinite, never NaN, soisfinitecovers every case. - Tests: the strict-JSON test fails without c97bcee (the old value was
inf). The two new advice tests cover the branch of_overall_advicethat does have a delta. - Changelog: the second fragment is under Fixed and names the user-visible change (
valuenull,measuredfalse,not_measured).
Summary
paired_summaryraisesOverflowErrorfor two finite scores whose difference leaves the float range, and two callers did not catch it. Onmain,paired_summary([{"question_id": "q1", "with_docs": {"aggregate": 1.7e308}, "without_docs": {"aggregate": -1.7e308}}, {"question_id": "q2", "with_docs": {"aggregate": 0.0}, "without_docs": {"aggregate": 0.0}}])raisesOverflowError: the difference of 1.7e+308 and -1.7e+308 leaves the float range.mainorchestrator/pipeline.py)report generate(report/generator.py,_compute_stats,_arm_means)report eval(report/eval_report.py)stats), and again insignificance_test, which callspaired_testOptions the issue lists: catch
OverflowErrorin the callers and print a stated refusal, or havepaired_summaryreturn a summary with no delta. This PR takes the second: the arm averages, ranges and counts of such scores are finite and stay, and the reports already print an em dash for a statistic with no value (docs/knowledge-runs.md:128: a summary without complete pairs "shows—"). A refusal in the callers would have dropped those averages.repos/compare.pyandreport/model_compare.pyrefuse with a stated error because their delta is the thing they report;paired_summary's delta is one column next to the averages.Closes #830
Changes
eval/stats.py:PairedSummary.delta_mean,delta_minanddelta_maxareNonewhen any complete pair's difference leaves the float range, and the newdelta_overflowcounts those pairs. The arm means, ranges,total,pairedandunpairedare unchanged.improvements,degradationsandneutraltake the sign of a pair without a delta from comparing its two scores (exact for finite scores), and from the difference otherwise, so every summary whose differences fit the range is unchanged.checked_delta,paired_test,paired_deltaand the arms path still raise.report/generator.py:_stat_blockand_arm_meansgiveNonefor a statistic with no value, which the page prints as an em dash through_shown._unpaired_noteadds the line_No delta: N complete pairs differ by more than the float range._, and_overall_advicesays there is no delta and no conclusion instead of comparingNone. The JSON stats carrydelta_overflow_questions.orchestrator/pipeline.py(Step 5 only): the evaluation summary hasdelta_avg: Noneand the printed line readsdelta=none (a paired difference leaves the float range).commands/orchestrate.pyprints an em dash for it.report/eval_report.py:statsand the Summary, Overall, agreement and delta lines read aNonedelta (em dash or "none — a paired difference leaves the float range"). Its significance section, which callspaired_test, prints_Not computed: <the refusal>._instead of stopping.tests/test_stats_float_range.py: the test that pinnedpaired_summaryraising now asserts it returns no delta, and keeps the two raises that stay (paired_delta,arm_significance).tests/test_paired_summary_overflow.py, including one test for theDelta:line ofcmd_evaluate(pipeline mocked) and one for the Dynamic vs Static agreement line ofreport eval.gate/trust_gate.py: themin_deltacheck, which subtracted the arm means directly, ismeasured=Falsewithvaluenull and a "Not measured" message when that difference is not finite, soreport generate --format jsonno longer writesInfinityfor it and the verdict listsmin_deltaundernot_measured. Test:test_report_json_trust_reports_min_delta_not_measured_and_is_strict_json(three pairs of 1.7e308 and -1.7e308, parsed with non-finite constants rejected).Risk
PairedSummary.delta_mean,delta_minanddelta_maxare nowOptional[float]; every reader in the package is updated (git grepforpaired_summaryand the fields findsgenerator.py,eval_report.py,pipeline.pyandcommands/orchestrate.py). Output for scores whose differences fit the range is unchanged. For scores that do not, a run that failed now completes with an em dash. The unpaired-note line is new text in the Markdown report only for that case. The cluster and source tables show an em dash for a group with such a pair, without the note.This touches the same files as open PR #667 (
report/generator.py; this PR leaves_shownand_format_markdownalone) and PR #820 (orchestrator/pipeline.py; only the Step 5 lines that read the delta change) and #841 (eval/stats.py, which appendspearson) and #806 (report/eval_report.py, where it adds ananswers_from_itemshelper and a diagnosis label, away from thestats, summary and significance lines changed here). The edits are on separate lines.Testing
On a Linux review host, at
4b9e0c6(onmain08f393c):make ci-local: 34 passed, 0 failed, 14 skipped (docker, other Python versions, conda-forge, fuzz, semgrep and the jobs that only run on merge). 22402 tests passed, 19 skipped.tests/test_paired_summary_overflow.pyalone now takes both branches of thedelta.avg is Nonecheck in_overall_advice(--cov-branchlists neither 525->526 nor 525->527 as missing); a new test draws the "hurting" and "minimal impact" conclusions from a delta that exists. Two mutants (the check inverted, the check made== 0.0) each fail it.ReportGenerator.generate_report(..., "json"): before the trust-gate change themin_deltacheck hadvalue: Infinityandpassed: true; nowvalue: null,measured: false. The new test fails with the guard removed.main's code: 9 of the 10 new tests fail (the tenth is the regression guard for pairs inside the range), the first withOverflowError: the difference of 1.7e+308 and -1.7e+308 leaves the float range; the changedtests/test_stats_float_range.pytest fails there with the sameOverflowError. All pass here.delta_meanordelta_min/delta_max; counting an overflowing pair as neither improvement nor degradation; dropping the arm means or ranges; not countingdelta_overflow; dropping the note, the_overall_advicebranch or theNoneguards of_stat_blockand_arm_means; rounding or formatting aNonedelta in the pipeline and inreport eval; re-raising in thereport evalrefusal; flipping theNonetest of theDelta:print ofcmd_evaluateand of the agreement line.fix/exact-sign-flip-sums,fix/interaction-tie-widthonmain08f393c:pytest -n auto22418 passed, 19 skipped,mypyclean,ruff checkclean.Open questions
None.
Revert
A plain revert restores the
OverflowErrorfrompaired_summaryand the pipeline and report failures for such scores. Nothing is stored.Checklist
pytestpassesruff,mypy, and schema checks passAGENTS.mdraise/exit and test names the doc, contract or issue it traces to.github/.licenserc.yaml)