Repository navigation
docs(repo): state the 1e150 measure bound of repo report and repo compare, and pin it at sign_flip_summary - #880
antonwolfy wants to merge 2 commits into
Conversation
…pare, and pin it at sign_flip_summary (intel#866) The issue reports baseline_mean and treatment_mean None for three pairs of 1.7e308 and blames an overflowing plain sum. The sum never runs: _mean reads its values through finite_values, which keeps only numbers within SCORE_BOUND (1e150, so that sums and squared deltas stay finite), so the three values are left out and the mean of nothing is None. The same rows at 1e150 give 1e+150. The bound is deliberate and no page of docs/ stated it. repo-evaluation-contract.md now says which values count as a measurement and what happens to one past the bound: a score past it is a grader failure, a cost, turn, token or score_max past it makes the result.json refused, and any other measure past it is left out of the mean. test_repo_compare_overflow.py pins that consequence at sign_flip_summary. No code changes. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
|
✅ OpenCodeReview: Review complete: 0 finding(s) across 2 selected item(s). CI at `a1c134e`: 1 failing, 24 passing, 3 skipped
Review details
|
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #880 +/- ##
==========================================
+ Coverage 95.76% 95.79% +0.03%
==========================================
Files 429 429
Lines 56714 56842 +128
Branches 9766 9787 +21
==========================================
+ Hits 54310 54454 +144
+ Misses 1494 1486 -8
+ Partials 910 902 -8 🚀 New features to boost your workflow:
|
… row; pin out-of-bound means Settles the two threads of the review of intel#880. The contract promised a JSON mean of null for three tasks at 1.7e308, but repo compare drops such a metric to absent, lists the tasks as unpaired and emits no row; the example now says that, and a test runs task_values and _compare_measure to pin it. The sign_flip_summary test also passed with the bound removed, because three 1.7e308 values sum to inf and round to None; it now uses 2 * SCORE_BOUND (a sum that stays finite) and a mix of a valid and an out-of-bound arm. Signed-off-by: Anton Volkov <antonwolfy@gmail.com>
vchamarthi
left a comment
There was a problem hiding this comment.
🟡 Not approved yet · reviewed a1c134e
Next: @antonwolfy: settle 1 open question (below). 1 nit optional.
Blocking
- Open question for @antonwolfy: A
scorepast the bound counts as 0 in the mean ofrepo compare(a grader failure intask_values), while a metric or cost past it is left out. State in the bullet that a score past the bound counts as 0 in the compare mean, or leave it out like the other values.
Verified on a1c134e: ✅ changed tests: 39 passed, pass on main too · ❌ full suite with main merged in fails at tests/test_import_architecture.py::test_only_the_frontend_ends_the_process, ci.yml:test / Run tests · ✅ merges onto main · 2 mutations tried · 5 edge inputs tried
8 checks behind it, each with its evidence
| Area | What holds | How |
|---|---|---|
| Bound value | SCORE_BOUND is 1e150 and inclusive: 1e150 and -1e150 are measurements; the next float past it, a 10**400 int, NaN and True are not |
ran it, edge input (measure bound) |
| Cost refusal | A cost_usd past the bound makes _check_cell raise CellFileError; one at the bound is read |
ran it, edge input (measure bound) |
| Score past bound | A score past the bound is no measure, and the grade gets the grader-failure reason the contract quotes |
ran it, edge input (measure bound) |
| Compare row | The changed tests pass on the head (39 passed). test_compare_gives_no_row_for_a_metric_past_the_measure_bound_and_lists_its_tasks_unpaired pins no row and three unpaired tasks at 1.7e308, and a row with means of 1e150 at the bound |
tests on the head |
| Issue premise | sign_flip_summary gives None means for three pairs at 1.7e308 and for three at 2 * SCORE_BOUND; test_sign_flip_summary_leaves_values_past_the_measure_bound_out_of_the_means pins both |
tests on the head |
| Bound pinned in compare | Removing the bound from as_measure fails two changed tests |
broke the code, tests caught it, edge input (measure bound) |
| Bound pinned at source | Raising SCORE_BOUND to 1.79e308 fails two changed tests |
broke the code, tests caught it, edge input (measure bound) |
| Merge | The PR merges onto main with no conflict | merge onto main |
| - **The measure bound.** `repo report` and `repo compare` read a score, `--metric` | ||
| value, cost, turn count, token count or `process` count as a measurement only |
There was a problem hiding this comment.
🟡 Nit: The bound bullet names repo report as reading a --metric value, but only repo compare takes --metric.
--metric is added to compare_p only (dokimi/dokimi/commands/repo_run.py:578), so repo report reads no --metric value. Expected: the bullet names repo compare for --metric and repo report for the other measures.
Fix, run on a1c134e: the quoted text is gone; tests are no worse.
| - **The measure bound.** `repo report` and `repo compare` read a score, `--metric` | |
| value, cost, turn count, token count or `process` count as a measurement only | |
| - **The measure bound.** `repo report` reads a score, cost, turn count, token count | |
| or `process` count, and `repo compare` also reads a `--metric` value, as a measurement only |
Summary
The issue reports that
sign_flip_summarygivesbaseline_meanandtreatment_meanNone for finite scores whose plain sum overflows. Its premise does not hold onmain: no sum overflows._meanreads its values throughfinite_values, which keeps only numbers withinSCORE_BOUND(1e150,dokimi/measure.py: squared deltas and sums stay finite), so a value past the bound is no measurement and is dropped before any sum. The mean of the empty list is None.baseline_mean,treatment_meanonmainbaseline=treatment= 1.7e308None,None(all three values are past the bound, so none is a measurement)baseline=treatment= 1e1501e+150,1e+150What was missing is that no page under
docs/states the bound; a reader meets it only in refusal messages. This PR documents it and pins the compare-level consequence with a test. No code behaviour changes. The issue's proposed fix (average withstatistics.meanwhen the plain sum is not finite) is not taken: it would change a deliberate bound, and calling it on the filtered values would divide by zero on the empty list. The eval-side overflow is the subject of #832 and #867, which this PR does not touch.Closes #866 as a documentation fix (not a code fix).
Changes
docs/repo-evaluation-contract.md, §9: a bullet "The measure bound" before therepo comparebullet.repo reportandrepo compareread a score,--metricvalue, cost, turn count, token count orprocesscount as a measurement only when it is finite and within 1e150. Ascoremetric past it makes the grade a grader failure and a storedgrade.scorepast it is read as one (repos/grading.py,repos/report.py); acost_usd,num_turns, token count orscore_maxpast it makes theresult.jsonrefused (repos/report.py); any other value past it (a--metric, aprocesscount,first_prompt_tokens,first_useful_*) is left out of the mean and ofn. Inrepo comparesuch a task is unpaired (no metric:x in baseline or treatment), so a metric past the bound in every task gives no row at all, not an overflow.tests/test_repo_compare_overflow.py:test_sign_flip_summary_leaves_values_past_the_measure_bound_out_of_the_meansruns the issue's own input (1.7e308: both meansNone),2 * SCORE_BOUND(a sum that stays finite: both meansNone), a mix of valid and out-of-bound rows (the mean of the valid ones) and the same rows atSCORE_BOUND(both means equal it).test_compare_gives_no_row_for_a_metric_past_the_measure_bound_and_lists_its_tasks_unpairedrunstask_valuesand_compare_measureon three tasks at 1.7e308: no row, three unpaired tasks; atSCORE_BOUNDa row with that mean.docs/andtests/change.Risk
None: no code changes. The test passes on
mainas well, since it documents existing behaviour.Testing
On a Linux review host, at
a1c134e(onmain725417b):make ci-local: 34 passed, 0 failed, 14 skipped (steps that need a tool or job condition this run lacks: micromamba, semgrep, fuzz, the other Pythons' matrix legs, deploy). 22572 tests passed, 19 skipped.main's code:38 passed(test file copied ontomain); it passes there because the PR documents existing behaviour.SCORE_BOUNDfilter inas_measurewith an unbounded one and_meanwithstatistics.meanfails the(None, None)assertion ((1.7e+308, 1.7e+308) == (None, None)); making the bound exclusive (abs(num) < bound) fails the at-the-bound assertion ((None, None) == (1e+150, 1e+150)).as_measure's bound to 1.79e308 fails bothtest_sign_flip_summary_leaves_values_past_the_measure_bound_out_of_the_means(the2 * SCORE_BOUNDcase does not overflow, so it is what catches a missing bound) andtest_compare_gives_no_row_for_a_metric_past_the_measure_bound_and_lists_its_tasks_unpaired.infand_roundgivesNoneanyway, which is the issue's observation.Open questions
None.
Revert
A plain revert removes the doc bullet and the test. Nothing is stored.
Checklist
pytestpassesruff,mypy, and schema checks passAGENTS.mdraise/exit and test names the doc, contract or issue it traces to.github/.licenserc.yaml)