Repository navigation
fix(gate): trust gate reads a spread beyond the float range as not measured - #894
Open
antonwolfy wants to merge 1 commit into
Open
antonwolfy wants to merge 1 commit into
antonwolfy wants to merge 1 commit into
Conversation
…asured (intel#870) evaluate_trust called statistics.stdev on the context-arm scores (score_cv) and on the per-run averages (multirun_variance). For finite values such as 1.7e308, -1.7e308 and 1.7e308 the variance leaves the float range and stdev raised OverflowError, which stopped ReportGenerator.generate_report. Both checks now catch OverflowError and report a Check with value None, passed True and measured False and a "Not measured" message, as min_delta does for an overflowing delta. The verdict lists them under not_measured; every finite spread keeps its value and message. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 4 selected item(s). CI at `cc571a6`: 6 still running, 20 passing, 2 skipped
Review details
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
#870:
evaluate_trust(and soReportGenerator.generate_report) raisesOverflowErrorwhere the variance of finite scores leaves the float range. Onmain, context-arm scores[1.7e308, -1.7e308, 1.7e308](check 5,score_cv) and per-run averages[1.7e308, -1.7e308, 1.7e308](check 8,multirun_variance) both stop the gate withOverflowError: integer division result too large for a float, fromstatistics.stdev.The fix takes the first option of the issue: the check is reported as not measured (
valuenull,passedtrue,measuredfalse, a "Not measured: " message), exactly asmin_deltais since #869. It follows the repo's own fix of the same cause and keeps the report generating; a range-safe stdev would still give a CV that no threshold reads sensibly.multirun_varianceis the same function and the same reader, so it is fixed here too.The other
statistics.stdevsites are not reachable with such values:eval/panel.pyandquestions/panel_reviewer.pyclamp every score to 0-100 before the spread,commands/benchmark.pytakes the spread of means of those judge scores, andrepos/compare.pyalready catchesOverflowError.Closes #870
Changes
dokimi/gate/trust_gate.py: checks 5 (score_cv) and 8 (multirun_variance) catchOverflowErrorfromstatistics.stdevand append a not-measuredCheck; finite spreads keep their value, message and order. The verdict line, the markdown table and the report JSON already readmeasured=False(min_deltaandretrieval_hit_rateuse it), so they show the new checks the same way.dokimi/docs/methodology.md: one sentence on when the delta, CV and multi-run checks are not measured.dokimi/tests/test_trust_gate.py(TestASpreadBeyondTheFloatRange):score_cvand the report JSON, the rendered block,multirun_variance.Risk
Only inputs that raised before change: a context arm or a multi-run list whose variance leaves the float range now yields a not-measured check, so the verdict can read
PASS (1 of 7 not measured)whereevaluate_trustraised. Every finite spread gives the same value and message as before, includingtests/fixtures/gate_golden.json.Testing
On Linux, at
cc571a6(onmain7125813):make ci-local: 36 passed, 0 failed, 14 skipped (the Docker catalog smoke, other-Python matrix legs, conda-forge, fuzz, semgrep, jobs conditioned off a PR). 22779 tests passed, 19 skipped; changed-line coverage 100%.main's code the 3 new tests fail withOverflowError: integer division result too large for a float(3 failed, 54 passed); here the file passes (57 passed).score_cvbranchpassed=Falsefails thescore_cvtest;multirun_variancebranchpassed=Falseand a changed message fail the multirun test;measured=Trueonscore_cvfails the report/JSON and rendered-block tests.fix/sign-flip-mc-spreadonmainb62613a:pytest -n auto22866 passed, 19 skipped,mypyclean,ruff checkclean.Open questions
None.
Revert
A plain revert restores the
OverflowError. Nothing is stored.Checklist
pytestpassesruff,mypy, and schema checks passAGENTS.mdraise/exit and test names the doc, contract or issue it traces to.github/.licenserc.yaml)