Skip to content

docs(repo): state the 1e150 measure bound of repo report and repo compare, and pin it at sign_flip_summary - #880

Open
antonwolfy wants to merge 2 commits into
intel:mainfrom
antonwolfy:docs/repo-score-bound
Open

antonwolfy wants to merge 2 commits into
intel:mainfrom
antonwolfy:docs/repo-score-bound

Conversation

@antonwolfy

@antonwolfy antonwolfy commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The issue reports that sign_flip_summary gives baseline_mean and treatment_mean None for finite scores whose plain sum overflows. Its premise does not hold on main: no sum overflows. _mean reads its values through finite_values, which keeps only numbers within SCORE_BOUND (1e150, dokimi/measure.py: squared deltas and sums stay finite), so a value past the bound is no measurement and is dropped before any sum. The mean of the empty list is None.

input (3 pairs, delta 0.0) baseline_mean, treatment_mean on main
baseline = treatment = 1.7e308 None, None (all three values are past the bound, so none is a measurement)
baseline = treatment = 1e150 1e+150, 1e+150

What was missing is that no page under docs/ states the bound; a reader meets it only in refusal messages. This PR documents it and pins the compare-level consequence with a test. No code behaviour changes. The issue's proposed fix (average with statistics.mean when the plain sum is not finite) is not taken: it would change a deliberate bound, and calling it on the filtered values would divide by zero on the empty list. The eval-side overflow is the subject of #832 and #867, which this PR does not touch.

Closes #866 as a documentation fix (not a code fix).

Changes

  • docs/repo-evaluation-contract.md, §9: a bullet "The measure bound" before the repo compare bullet. repo report and repo compare read a score, --metric value, cost, turn count, token count or process count as a measurement only when it is finite and within 1e150. A score metric past it makes the grade a grader failure and a stored grade.score past it is read as one (repos/grading.py, repos/report.py); a cost_usd, num_turns, token count or score_max past it makes the result.json refused (repos/report.py); any other value past it (a --metric, a process count, first_prompt_tokens, first_useful_*) is left out of the mean and of n. In repo compare such a task is unpaired (no metric:x in baseline or treatment), so a metric past the bound in every task gives no row at all, not an overflow.
  • tests/test_repo_compare_overflow.py: test_sign_flip_summary_leaves_values_past_the_measure_bound_out_of_the_means runs the issue's own input (1.7e308: both means None), 2 * SCORE_BOUND (a sum that stays finite: both means None), a mix of valid and out-of-bound rows (the mean of the valid ones) and the same rows at SCORE_BOUND (both means equal it). test_compare_gives_no_row_for_a_metric_past_the_measure_bound_and_lists_its_tasks_unpaired runs task_values and _compare_measure on three tasks at 1.7e308: no row, three unpaired tasks; at SCORE_BOUND a row with that mean.
  • No changelog fragment: only docs/ and tests/ change.

Risk

None: no code changes. The test passes on main as well, since it documents existing behaviour.

Testing

On a Linux review host, at a1c134e (on main 725417b):

  • make ci-local: 34 passed, 0 failed, 14 skipped (steps that need a tool or job condition this run lacks: micromamba, semgrep, fuzz, the other Pythons' matrix legs, deploy). 22572 tests passed, 19 skipped.
  • The new test on main's code: 38 passed (test file copied onto main); it passes there because the PR documents existing behaviour.
  • Mutants, each failing the new test: replacing the SCORE_BOUND filter in as_measure with an unbounded one and _mean with statistics.mean fails the (None, None) assertion ((1.7e+308, 1.7e+308) == (None, None)); making the bound exclusive (abs(num) < bound) fails the at-the-bound assertion ((None, None) == (1e+150, 1e+150)).
  • Mutant for the review round: raising as_measure's bound to 1.79e308 fails both test_sign_flip_summary_leaves_values_past_the_measure_bound_out_of_the_means (the 2 * SCORE_BOUND case does not overflow, so it is what catches a missing bound) and test_compare_gives_no_row_for_a_metric_past_the_measure_bound_and_lists_its_tasks_unpaired.
  • A plain float filter alone (no bound) does not fail the test: the plain sum of three 1.7e308 overflows to inf and _round gives None anyway, which is the issue's observation.
  • Not run: the other Python versions (3.11, 3.12, 3.14), as CI's matrix does.

Open questions

None.

Revert

A plain revert removes the doc bullet and the test. Nothing is stored.

Checklist

  • Tests added/updated and pytest passes
  • ruff, mypy, and schema checks pass
  • Docs updated (or N/A) — see the docs-drift checklist in CONTRIBUTING.md
  • Self-reviewed per "Before opening a PR" in AGENTS.md
  • Each new raise/exit and test names the doc, contract or issue it traces to
  • No ad-hoc run artifacts committed
  • New files carry the SPDX license header (or are exempt in .github/.licenserc.yaml)

…pare, and pin it at sign_flip_summary (intel#866)

The issue reports baseline_mean and treatment_mean None for three pairs of 1.7e308 and blames an overflowing plain sum. The sum never runs: _mean reads its values through finite_values, which keeps only numbers within SCORE_BOUND (1e150, so that sums and squared deltas stay finite), so the three values are left out and the mean of nothing is None. The same rows at 1e150 give 1e+150. The bound is deliberate and no page of docs/ stated it.

repo-evaluation-contract.md now says which values count as a measurement and what happens to one past the bound: a score past it is a grader failure, a cost, turn, token or score_max past it makes the result.json refused, and any other measure past it is left out of the mean. test_repo_compare_overflow.py pins that consequence at sign_flip_summary. No code changes.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
@antonwolfy
antonwolfy requested a review from a team as a code owner November 17, 2026 17:54
@antonwolfy antonwolfy self-assigned this Nov 17, 2026
@exai-bot

exai-bot commented Nov 17, 2026 •

Copy link
Copy Markdown

✅ OpenCodeReview: Review complete: 0 finding(s) across 2 selected item(s).

CI at `a1c134e`: 1 failing, 24 passing, 3 skipped
Check Status
Test Linux Wheels Python 3.11 failing at step Install the wheel and run ci/test.py
Review details
  • Commit reviewed: a1c134e against merge base 725417b
  • Coverage: 2 files in 1 groups
  • Engine: OpenCodeReview v1.13.5-exai.4.8f73eada, us.openai.gpt-6.1-sol
  • Usage: 174k tokens (137k from cache), 39s
  • Tool calls: 17 (13 file_read, 3 code_search, 1 local_check_results)
  • small change set: 2 files

Comment thread dokimi/docs/repo-evaluation-contract.md Resolved
@codecov-commenter

codecov-commenter commented Nov 17, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.79%. Comparing base (b768023) to head (a1c134e).
⚠️ Report is 10 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #880      +/-   ##
==========================================
+ Coverage   95.76%   95.79%   +0.03%     
==========================================
  Files         429      429              
  Lines       56714    56842     +128     
  Branches     9766     9787      +21     
==========================================
+ Hits        54310    54454     +144     
+ Misses       1494     1486       -8     
+ Partials      910      902       -8     

see 20 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

… row; pin out-of-bound means

Settles the two threads of the review of intel#880. The contract promised a JSON mean of null for three tasks at 1.7e308, but repo compare drops such a metric to absent, lists the tasks as unpaired and emits no row; the example now says that, and a test runs task_values and _compare_measure to pin it.

The sign_flip_summary test also passed with the bound removed, because three 1.7e308 values sum to inf and round to None; it now uses 2 * SCORE_BOUND (a sum that stays finite) and a mix of a valid and an out-of-bound arm.

Signed-off-by: Anton Volkov <antonwolfy@gmail.com>

@vchamarthi vchamarthi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Not approved yet · reviewed a1c134e

Next: @antonwolfy: settle 1 open question (below). 1 nit optional.

Blocking

  • Open question for @antonwolfy: A score past the bound counts as 0 in the mean of repo compare (a grader failure in task_values), while a metric or cost past it is left out. State in the bullet that a score past the bound counts as 0 in the compare mean, or leave it out like the other values.

Verified on a1c134e: ✅ changed tests: 39 passed, pass on main too · ❌ full suite with main merged in fails at tests/test_import_architecture.py::test_only_the_frontend_ends_the_process, ci.yml:test / Run tests · ✅ merges onto main · 2 mutations tried · 5 edge inputs tried

8 checks behind it, each with its evidence
Area What holds How
Bound value SCORE_BOUND is 1e150 and inclusive: 1e150 and -1e150 are measurements; the next float past it, a 10**400 int, NaN and True are not ran it, edge input (measure bound)
Cost refusal A cost_usd past the bound makes _check_cell raise CellFileError; one at the bound is read ran it, edge input (measure bound)
Score past bound A score past the bound is no measure, and the grade gets the grader-failure reason the contract quotes ran it, edge input (measure bound)
Compare row The changed tests pass on the head (39 passed). test_compare_gives_no_row_for_a_metric_past_the_measure_bound_and_lists_its_tasks_unpaired pins no row and three unpaired tasks at 1.7e308, and a row with means of 1e150 at the bound tests on the head
Issue premise sign_flip_summary gives None means for three pairs at 1.7e308 and for three at 2 * SCORE_BOUND; test_sign_flip_summary_leaves_values_past_the_measure_bound_out_of_the_means pins both tests on the head
Bound pinned in compare Removing the bound from as_measure fails two changed tests broke the code, tests caught it, edge input (measure bound)
Bound pinned at source Raising SCORE_BOUND to 1.79e308 fails two changed tests broke the code, tests caught it, edge input (measure bound)
Merge The PR merges onto main with no conflict merge onto main

Comment on lines +2518 to +2519
- **The measure bound.** `repo report` and `repo compare` read a score, `--metric`
value, cost, turn count, token count or `process` count as a measurement only

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Nit: The bound bullet names repo report as reading a --metric value, but only repo compare takes --metric.

--metric is added to compare_p only (dokimi/dokimi/commands/repo_run.py:578), so repo report reads no --metric value. Expected: the bullet names repo compare for --metric and repo report for the other measures.

Fix, run on a1c134e: the quoted text is gone; tests are no worse.

Suggested change
- **The measure bound.** `repo report` and `repo compare` read a score, `--metric`
value, cost, turn count, token count or `process` count as a measurement only
- **The measure bound.** `repo report` reads a score, cost, turn count, token count
or `process` count, and `repo compare` also reads a `--metric` value, as a measurement only

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

repo compare: baseline_mean and treatment_mean are None for finite scores whose plain sum overflows

4 participants