Skip to content

paired_summary raises OverflowError for scores whose paired difference leaves the float range, ending the evaluate pipeline and the HTML report #830

Closed

Description

@antonwolfy

dokimi.eval.stats.paired_summary forms the paired deltas through _deltas -> checked_delta (#599), which raises OverflowError for a difference beyond the float range. Two callers do not catch it: orchestrator/pipeline.py:420 (Step 5 of the evaluate pipeline, after judge.save_evaluations at pipeline.py:752, so the answers and evaluations are on disk but the pipeline stops before Step 6 and the summary) and report/generator.py:156 and :538 (the HTML report).

Repro (offline): paired_summary([{"question_id": "q1", "with_docs": {"aggregate": 1.7e308}, "without_docs": {"aggregate": -1.7e308}}, {"question_id": "q2", "with_docs": {"aggregate": 0.0}, "without_docs": {"aggregate": 0.0}}]) raises OverflowError: the difference of 1.7e+308 and -1.7e+308 leaves the float range.

The arms run had the same crash before the save (#750, fixed by keeping the run and recording the refusal). Options here: catch OverflowError in the two callers and print a stated refusal in place of the summary, as repos/compare.py and report/model_compare.py do, or have paired_summary return a summary with no delta. Judge scores are normally 0-100, so the trigger needs a hand-written or corrupted evaluations file.

Found while fixing #750. The fix's PR is #831.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions