Skip to content

feat(cli): repeat evaluate runs with --runs and summarize them with report - #25

Merged
kottinov merged 3 commits into
mainfrom
feat/cli-evaluate-runs
Nov 18, 2026
Merged

kottinov merged 3 commits into
mainfrom
feat/cli-evaluate-runs

Conversation

@kottinov

Copy link
Copy Markdown
Collaborator

Why

Claude Sonnet 5 has no temperature setting, so one evorca evaluate run is one sample of an agent's accuracy. A thesis result needs a mean and a spread over several runs. Before this PR, you started each run by hand, tracked the seeds yourself, and computed the statistics outside evorca. A stopped run also under-reported its cost. One react run showed $9.80 in run.json and $10.34 in its requests.jsonl files.

What changed

The three commits each pass pnpm check on their own.

  1. Stopped tasks. When ctrl+c or SIGTERM stops a run, each running task gets the outcome stopped with what its requests.jsonl says it spent. run.json now matches the request logs at the stop. A resume runs a stopped task again and moves its spend to abandoned, as it already did for other tasks it runs again. endedAt is now written after scores.json, so a run with an endedAt is always scored.
  2. --runs=N. evorca evaluate makes N complete runs, one after another, with the same settings. --jobs still sets how many tasks of a run go at once. Run k of series <id> has the run ID <id>-<k> and the seed of run 1 plus k minus 1. Each run.json records series: { id, run, runs }, and each task line starts with run k/N. Resuming any run of a series finishes the whole series. The command prints that resume command when the series starts and again when ctrl+c stops it.
  3. evorca report. After the last run, --runs prints a summary in two Markdown tables. The first gives strict and partial accuracy per difficulty and in total, as mean, sample SD, minimum, and maximum. The second gives each run's seed, accuracy, requests, cost, and wall time, with the mean and the total. evorca report --benchmark=openrca [--scenario=...] <run ID>... prints the same summary from run.json and scores.json and never scores again. It refuses runs whose settings differ and names the field. It leaves out unfinished runs and runs that lack the scenario, and names them.

The Repeat a run section of the CLI README has the commands and a real sample summary.

Scope

All changes are in packages/cli. The PR adds no dependency and does not touch the shell UI or agents/react. A single run keeps the old run ID, writes no series field, and prints the same lines. Its predictions are byte for byte the same as on main. An old run.json resumes unchanged.

Tradeoffs

  • Series state lives in each run's run.json. A separate series file would be a second record that can disagree with the run folders. The resume derives every step from the folders. It skips finished runs, resumes the unfinished one, and starts missing ones, so the same command is safe to repeat.
  • One command finishes the series. A resume of just the stopped run, followed by a printed command for the remaining runs, needs two commands and does not end with the summary.
  • Stopped spend stays on the task until a resume. If the stop wrote that spend to abandoned, the resume would add it again when it moves the task's spend there.
  • Seeds are consecutive. Run k can be repeated alone with EVORCA_SEED set to its seed. Run 1 of a series at seed 7 equals a single run at seed 7. EVORCA_SEED=12 --runs=5 extends a series of seeds 7 to 11, and the report pools all ten runs.
  • The report derives wall time. A scenario's wall time starts at the end of the scenario that ran before it in the same run, or else at the run's start. This needs no new field in run.json.

Blast Radius

  • Outcome gains stopped. Only packages/cli reads outcomes.
  • A crash during scoring now leaves endedAt null. The run counts as unfinished, and a resume scores it.
  • The command line takes positional run IDs for report only. Other commands still refuse extra arguments.

Known gaps, left as they are:

  • A series resumed with a narrower --scenario can start later runs without the other scenarios. The report names such runs. A resume does not fill them in.
  • On main as well, a kill between writing run.json and deleting task folders during a resume's setup counts that spend twice.
  • If a series stops before run 1 writes its run.json, for example during the image build, the printed finish command finds no run. Nothing is lost. Start the series again.
  • As for single runs on main, two series that start in the same second with the same agent get the same run IDs.
  • The report does not compare pinnedInputs. A resume does not compare them either.

Verification

No test made a real API call. Model runs used a stub agent and a mock upstream through EVORCA_UPSTREAM_URL.

  • pnpm check passes at each commit, with 115, 117, and 128 CLI tests. fetch.test.ts in packages/openrca failed 2 of 5 full checks and passed 6 of 6 runs alone. That package is the same as on main.
  • --runs=3 with random, telecom, and EVORCA_SEED=7 writes three different predictions.csv files. Run 1 matches main's single run at seed 7. Every later series with that seed, including the interrupted ones, reproduced all three byte for byte.
  • After ctrl+c in run 2, the process exits with 130, and run 2 holds 12 answered and 2 stopped tasks. The printed command left run 1's run.json unchanged, ran the other 39 tasks of run 2 and all 51 of run 3, and printed a summary of 3 runs.
  • After SIGKILL at the same point, the process exits with 137. The command printed at the start finished the series with the same files and removed the containers that the kill left.
  • evorca report prints the same text as the series summary. It refuses mixed rows with rows: the run has [0,7,21], not null, a missing run, and a bad run ID.
  • With the mock upstream, ctrl+c in run 2 of 2 left 54 requests and $0.399918 in both run.json and requests.jsonl. After the resume, run.json shows 102 requests and $0.755934. The logs hold 96 requests and $0.711456, and abandoned holds 6 and $0.044478.
  • A copy of the example react run, with only its upstream URL changed to the mock, resumed. It kept 26 tasks and moved rows 26 and 27 to abandoned. Its $10.340867 total equals the remaining logs plus abandoned.
  • A series whose runs ended with budget-exhausted tasks is not done. The report refused it, and a resume with --budget-usd=5 finished both runs.
  • Unit tests cover the statistics, unfinished runs, settings checks, wall time across scenarios, and runs that lack a scenario. Switching to the population SD, or timing from the run's start, fails them.

…s the request logs

A stopped run left its running tasks out of run.json until a resume read
their requests.jsonl, so its totals fell short of the logs ($9.80 against
$10.34 in one run). Each such task now gets the outcome `stopped` with
what it spent, and a resume runs it again and moves that spend to
`abandoned`. `endedAt` is now set after scores.json is written, so a run
with `endedAt` is scored.
Claude Sonnet 5 has no temperature setting, so one run is one sample of
an agent's accuracy. `--runs=N` makes N complete runs one after another
with the same settings. Run k of series <id> has the run ID <id>-<k> and
the seed of run 1 plus k - 1, and its run.json records its place in the
series. Resuming any run of a series finishes the series. It skips each
finished run, resumes the unfinished one, and starts each missing one,
so the same command is safe to repeat after ctrl+c, a crash, or a kill.
After its last run, `evaluate --runs` prints each accuracy per
difficulty as mean, sample standard deviation, minimum, and maximum,
then each run's seed, accuracy, requests, cost, and wall time with their
mean and total. `evorca report <run ID>...` prints the same summary from
run.json and scores.json and never scores again. It refuses runs whose
settings differ and names the field, and it names each run it leaves
out, such as an unfinished run. The tables are Markdown.
@kottinov
kottinov merged commit bc31df9 into main Nov 18, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant