Repository navigation
feat(cli): repeat evaluate runs with --runs and summarize them with report - #25
Merged
Merged
Conversation
…s the request logs A stopped run left its running tasks out of run.json until a resume read their requests.jsonl, so its totals fell short of the logs ($9.80 against $10.34 in one run). Each such task now gets the outcome `stopped` with what it spent, and a resume runs it again and moves that spend to `abandoned`. `endedAt` is now set after scores.json is written, so a run with `endedAt` is scored.
Claude Sonnet 5 has no temperature setting, so one run is one sample of an agent's accuracy. `--runs=N` makes N complete runs one after another with the same settings. Run k of series <id> has the run ID <id>-<k> and the seed of run 1 plus k - 1, and its run.json records its place in the series. Resuming any run of a series finishes the series. It skips each finished run, resumes the unfinished one, and starts each missing one, so the same command is safe to repeat after ctrl+c, a crash, or a kill.
After its last run, `evaluate --runs` prints each accuracy per difficulty as mean, sample standard deviation, minimum, and maximum, then each run's seed, accuracy, requests, cost, and wall time with their mean and total. `evorca report <run ID>...` prints the same summary from run.json and scores.json and never scores again. It refuses runs whose settings differ and names the field, and it names each run it leaves out, such as an unfinished run. The tables are Markdown.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Claude Sonnet 5 has no temperature setting, so one
evorca evaluaterun is one sample of an agent's accuracy. A thesis result needs a mean and a spread over several runs. Before this PR, you started each run by hand, tracked the seeds yourself, and computed the statistics outside evorca. A stopped run also under-reported its cost. One react run showed $9.80 inrun.jsonand $10.34 in itsrequests.jsonlfiles.What changed
The three commits each pass
pnpm checkon their own.stoppedwith what itsrequests.jsonlsays it spent.run.jsonnow matches the request logs at the stop. A resume runs astoppedtask again and moves its spend toabandoned, as it already did for other tasks it runs again.endedAtis now written afterscores.json, so a run with anendedAtis always scored.--runs=N.evorca evaluatemakes N complete runs, one after another, with the same settings.--jobsstill sets how many tasks of a run go at once. Run k of series<id>has the run ID<id>-<k>and the seed of run 1 plus k minus 1. Eachrun.jsonrecordsseries: { id, run, runs }, and each task line starts withrun k/N. Resuming any run of a series finishes the whole series. The command prints that resume command when the series starts and again when ctrl+c stops it.evorca report. After the last run,--runsprints a summary in two Markdown tables. The first gives strict and partial accuracy per difficulty and in total, as mean, sample SD, minimum, and maximum. The second gives each run's seed, accuracy, requests, cost, and wall time, with the mean and the total.evorca report --benchmark=openrca [--scenario=...] <run ID>...prints the same summary fromrun.jsonandscores.jsonand never scores again. It refuses runs whose settings differ and names the field. It leaves out unfinished runs and runs that lack the scenario, and names them.The Repeat a run section of the CLI README has the commands and a real sample summary.
Scope
All changes are in
packages/cli. The PR adds no dependency and does not touch the shell UI oragents/react. A single run keeps the old run ID, writes noseriesfield, and prints the same lines. Its predictions are byte for byte the same as onmain. An oldrun.jsonresumes unchanged.Tradeoffs
run.json. A separate series file would be a second record that can disagree with the run folders. The resume derives every step from the folders. It skips finished runs, resumes the unfinished one, and starts missing ones, so the same command is safe to repeat.abandoned, the resume would add it again when it moves the task's spend there.EVORCA_SEEDset to its seed. Run 1 of a series at seed 7 equals a single run at seed 7.EVORCA_SEED=12 --runs=5extends a series of seeds 7 to 11, and the report pools all ten runs.run.json.Blast Radius
Outcomegainsstopped. Onlypackages/clireads outcomes.endedAtnull. The run counts as unfinished, and a resume scores it.reportonly. Other commands still refuse extra arguments.Known gaps, left as they are:
--scenariocan start later runs without the other scenarios. The report names such runs. A resume does not fill them in.mainas well, a kill between writingrun.jsonand deleting task folders during a resume's setup counts that spend twice.run.json, for example during the image build, the printed finish command finds no run. Nothing is lost. Start the series again.main, two series that start in the same second with the same agent get the same run IDs.pinnedInputs. A resume does not compare them either.Verification
No test made a real API call. Model runs used a stub agent and a mock upstream through
EVORCA_UPSTREAM_URL.pnpm checkpasses at each commit, with 115, 117, and 128 CLI tests.fetch.test.tsinpackages/openrcafailed 2 of 5 full checks and passed 6 of 6 runs alone. That package is the same as onmain.--runs=3with random, telecom, andEVORCA_SEED=7writes three differentpredictions.csvfiles. Run 1 matchesmain's single run at seed 7. Every later series with that seed, including the interrupted ones, reproduced all three byte for byte.run.jsonunchanged, ran the other 39 tasks of run 2 and all 51 of run 3, and printed a summary of 3 runs.evorca reportprints the same text as the series summary. It refuses mixed rows withrows: the run has [0,7,21], not null, a missing run, and a bad run ID.run.jsonandrequests.jsonl. After the resume,run.jsonshows 102 requests and $0.755934. The logs hold 96 requests and $0.711456, andabandonedholds 6 and $0.044478.abandoned. Its $10.340867 total equals the remaining logs plusabandoned.budget-exhaustedtasks is not done. The report refused it, and a resume with--budget-usd=5finished both runs.