Skip to content

fix(gate): trust gate reads a spread beyond the float range as not measured - #894

Open
antonwolfy wants to merge 1 commit into
intel:mainfrom
antonwolfy:fix/score-cv-overflow
Open

antonwolfy wants to merge 1 commit into
intel:mainfrom
antonwolfy:fix/score-cv-overflow

Conversation

@antonwolfy

Copy link
Copy Markdown
Contributor

Summary

#870: evaluate_trust (and so ReportGenerator.generate_report) raises OverflowError where the variance of finite scores leaves the float range. On main, context-arm scores [1.7e308, -1.7e308, 1.7e308] (check 5, score_cv) and per-run averages [1.7e308, -1.7e308, 1.7e308] (check 8, multirun_variance) both stop the gate with OverflowError: integer division result too large for a float, from statistics.stdev.

The fix takes the first option of the issue: the check is reported as not measured (value null, passed true, measured false, a "Not measured: " message), exactly as min_delta is since #869. It follows the repo's own fix of the same cause and keeps the report generating; a range-safe stdev would still give a CV that no threshold reads sensibly. multirun_variance is the same function and the same reader, so it is fixed here too.

The other statistics.stdev sites are not reachable with such values: eval/panel.py and questions/panel_reviewer.py clamp every score to 0-100 before the spread, commands/benchmark.py takes the spread of means of those judge scores, and repos/compare.py already catches OverflowError.

Closes #870

Changes

  • dokimi/gate/trust_gate.py: checks 5 (score_cv) and 8 (multirun_variance) catch OverflowError from statistics.stdev and append a not-measured Check; finite spreads keep their value, message and order. The verdict line, the markdown table and the report JSON already read measured=False (min_delta and retrieval_hit_rate use it), so they show the new checks the same way.
  • dokimi/docs/methodology.md: one sentence on when the delta, CV and multi-run checks are not measured.
  • Tests in dokimi/tests/test_trust_gate.py (TestASpreadBeyondTheFloatRange): score_cv and the report JSON, the rendered block, multirun_variance.
  • Changelog fragment (Fixed).

Risk

Only inputs that raised before change: a context arm or a multi-run list whose variance leaves the float range now yields a not-measured check, so the verdict can read PASS (1 of 7 not measured) where evaluate_trust raised. Every finite spread gives the same value and message as before, including tests/fixtures/gate_golden.json.

Testing

On Linux, at cc571a6 (on main 7125813):

  • make ci-local: 36 passed, 0 failed, 14 skipped (the Docker catalog smoke, other-Python matrix legs, conda-forge, fuzz, semgrep, jobs conditioned off a PR). 22779 tests passed, 19 skipped; changed-line coverage 100%.
  • On main's code the 3 new tests fail with OverflowError: integer division result too large for a float (3 failed, 54 passed); here the file passes (57 passed).
  • Mutants: score_cv branch passed=False fails the score_cv test; multirun_variance branch passed=False and a changed message fail the multirun test; measured=True on score_cv fails the report/JSON and rendered-block tests.
  • Merged together with fix/sign-flip-mc-spread on main b62613a: pytest -n auto 22866 passed, 19 skipped, mypy clean, ruff check clean.
  • Not run: the other Python versions (3.11, 3.12, 3.14), the Docker smoke.

Open questions

None.

Revert

A plain revert restores the OverflowError. Nothing is stored.

Checklist

  • Tests added/updated and pytest passes
  • ruff, mypy, and schema checks pass
  • Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
  • Self-reviewed per "Before opening a PR" in AGENTS.md
  • Each new raise/exit and test names the doc, contract or issue it traces to
  • No ad-hoc run artifacts committed
  • New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

…asured (intel#870)

evaluate_trust called statistics.stdev on the context-arm scores (score_cv) and on the per-run averages (multirun_variance). For finite values such as 1.7e308, -1.7e308 and 1.7e308 the variance leaves the float range and stdev raised OverflowError, which stopped ReportGenerator.generate_report.

Both checks now catch OverflowError and report a Check with value None, passed True and measured False and a "Not measured" message, as min_delta does for an overflowing delta. The verdict lists them under not_measured; every finite spread keeps its value and message.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@antonwolfy
antonwolfy requested a review from a team as a code owner November 17, 2026 21:19
@antonwolfy antonwolfy self-assigned this Nov 17, 2026
@exai-bot

exai-bot commented Nov 17, 2026 •

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 4 selected item(s).

CI at `cc571a6`: 6 still running, 20 passing, 2 skipped
Check Status
Test Linux Python 3.11 still running
Test Linux Python 3.14 still running
Test Linux Wheels Python 3.11 still running
Test Linux Wheels Python 3.14 still running
Tests (py3.11) still running
Tests (py3.14) still running
Review details
  • Commit reviewed: cc571a6 against merge base 7125813
  • Coverage: 4 files in 1 groups
  • Engine: OpenCodeReview v1.13.5-exai.4.8f73eada, us.openai.gpt-6.1-sol
  • Usage: 86k tokens (40k from cache), 20s
  • Tool calls: 8 (6 file_read, 1 linked_items, 1 local_check_results)
  • Trust gate float range update, tests, and documentation: 4 files

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

trust gate: score_cv raises OverflowError for context scores whose variance leaves the float range

2 participants