Repository navigation
fix(eval): paired_summary returns no delta where a paired difference leaves the float range (#830) #869
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
antonwolfy
merged 5 commits into
intel:main
from
antonwolfy:fix/paired-summary-overflow
Nov 17, 2026
Merged
fix(eval): paired_summary returns no delta where a paired difference leaves the float range (#830) #869
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
4f29307
fix(eval): paired_summary returns no delta where a paired difference …
antonwolfy 72eff4a
test(eval): pin the Delta line of cmd_evaluate and the agreement line…
antonwolfy c97bcee
fix(eval): the trust gate reads a min_delta beyond the float range as…
antonwolfy 6462ca4
fix(eval): the no-delta note agrees in number with the pairs it counts
antonwolfy 4b9e0c6
test(eval): pin the branch of the Overall advice that has a delta nex…
antonwolfy File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
8 changes: 8 additions & 0 deletions
8
dokimi/changelog.d/20261117_120000_paired-summary-overflow.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| <!-- | ||
| Copyright (C) 2026 Intel Corporation | ||
| SPDX-License-Identifier: Apache-2.0 | ||
| --> | ||
|
|
||
| ### Fixed | ||
|
|
||
| - **The evaluate pipeline and the reports finish when a paired score difference leaves the float range.** `paired_summary` raised `OverflowError` for scores such as 1.7e308 and -1.7e308 in one pair, which ended `dokimi orchestrate` after the evaluations were saved and failed `report generate` and `report eval`. It now returns the arm means, ranges and counts with no delta (`delta_mean`, `delta_min` and `delta_max` are `None`, `delta_overflow` counts the pairs); improvements and degradations of such a pair are counted by comparing its two scores. The pipeline prints `delta=none`, the reports print an em dash with a one-line note, and `report eval` names the refused significance test instead of stopping (#830). |
8 changes: 8 additions & 0 deletions
8
dokimi/changelog.d/20261117_130000_trust-min-delta-float-range.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| <!-- | ||
| Copyright (C) 2026 Intel Corporation | ||
| SPDX-License-Identifier: Apache-2.0 | ||
| --> | ||
|
|
||
| ### Fixed | ||
|
|
||
| - **The trust gate reads a `min_delta` whose arm means differ by more than the float range as not measured.** `report generate --format json` wrote `Infinity` (not standard JSON) as that check's value and marked it passed for scores such as 1.7e308 and -1.7e308. The check now has `value` null, `measured` false and a "Not measured" message, and the verdict lists `min_delta` under `not_measured` (#830). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.