You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
paired_summary raises OverflowError for scores whose paired difference leaves the float range, ending the evaluate pipeline and the HTML report #830
dokimi.eval.stats.paired_summary forms the paired deltas through _deltas -> checked_delta (#599), which raises OverflowError for a difference beyond the float range. Two callers do not catch it: orchestrator/pipeline.py:420 (Step 5 of the evaluate pipeline, after judge.save_evaluations at pipeline.py:752, so the answers and evaluations are on disk but the pipeline stops before Step 6 and the summary) and report/generator.py:156 and :538 (the HTML report).
Repro (offline): paired_summary([{"question_id": "q1", "with_docs": {"aggregate": 1.7e308}, "without_docs": {"aggregate": -1.7e308}}, {"question_id": "q2", "with_docs": {"aggregate": 0.0}, "without_docs": {"aggregate": 0.0}}]) raises OverflowError: the difference of 1.7e+308 and -1.7e+308 leaves the float range.
The arms run had the same crash before the save (#750, fixed by keeping the run and recording the refusal). Options here: catch OverflowError in the two callers and print a stated refusal in place of the summary, as repos/compare.py and report/model_compare.py do, or have paired_summary return a summary with no delta. Judge scores are normally 0-100, so the trigger needs a hand-written or corrupted evaluations file.
Activity
test(eval): pin the branch of the Overall advice that has a delta nex…
fix(eval): paired_summary returns no delta where a paired difference …