Repository navigation
Conversation
The shell's benchmark picker groups tasks by difficulty the way the scorer does, so difficultyOf and Difficulty move from the scoring summary to tasks.ts and the package exports them.
--rows can repeat, and a value that starts with a scenario, such as --rows=bank:3,4, gives the rows of that scenario only. A bare --rows=0,7,21 still gives the rows of every scenario that no other value names. run.json keeps its format, so a resume checks the rows of each scenario against its own record, and earlier runs resume and report as before. The scenario list moves to runs.ts so the parser and report share it.
/benchmark opens a picker of four steps: the scenarios, the tasks of each with their difficulty, the agent and its settings, and a confirmation with the evorca evaluate command that repeats the run and, for an agent that calls a model, a cost estimate from finished runs. The picker is a pure state machine in benchmark.ts, and benchmark-view.tsx only draws it. It never shows the scoring points. The run goes in the shell's process, in the view of evorca watch, with the input line under the feed. ctrl+c stops it and returns to the transcript with the run's lines and the command that finishes it. A second ctrl+c stops at once. /exit asks first, then stops the run and leaves. The shell's palette and chrome move to parts.tsx because the shell now imports the watch view, which imported them from the shell.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Picking what to evaluate meant composing
pnpm evorca evaluateflags by hand, and--rowscould only give the same rows to every scenario./benchmarkin the shell picks the scenarios, the tasks in each, and the agent, shows the exact command, and runs it in the shell. It never shows the scoring points.What changed
Three commits, each of which typechecks and passes the tests on its own:
feat(openrca): export the difficulty of each task index.difficultyOfandDifficultymove from the scoring summary totasks.ts, and the package exports them so the picker groups tasks the way the scorer does.feat(cli): give --rows per scenario.--rowscan repeat, and--rows=<scenario>:<rows>gives the rows of one scenario. A bare--rows=0,7,21still covers every scenario that no other value names.run.jsonkeeps its format:rowswas already stored per scenario, so a resume checks each scenario's rows against its own record.parseRowsmoves toruns.tswith the scenario list thatreport.tsandopenrca.tsshare.feat(cli): pick and run a benchmark with /benchmark in the shell.Catalogholds what the picker offers:ScenarioData[](tasks, telemetry on disk) and theRunRecords that the estimate learns from.Pickeris the state (step, cursor, chosen scenarios, chosen rows per scenario, settings, seed), andPlanis the evaluate request that a pick describes.benchmark.tsis a pure state machine:update(picker, action, catalog),itemsOf,blockerOf,planOf,commandOf,estimateOf. OnlyreadCatalogtouches the disk.benchmark-view.tsxmaps keys to actions and draws lines.evaluateOpenRcawith anAbortSignal, in theWatchview with the input line under the feed.Watchtakeskeysandchildrenso the prompt can own the keys while it holds text./exitduring a run asks first, then stops the run and leaves once it has stopped. The run's lines go into the transcript, or to stdout when evorca leaves.data/runswith the same agent, model, and effort, times the tasks and runs. It is labelled an estimate, it is absent without such a run, and it uses no price table.EVORCA_SEED=<seed> pnpm evorca evaluate ...), uses--rows=<scenario>:<rows>only for scenarios with tasks left out, and wraps at 72 columns with\so a copy runs as it is.palette,Mark,Hints,Footer, and the rest of the shell's chrome move verbatim toparts.tsx. The shell now imports the watch view, which imported them from the shell, and that cycle hits the temporal dead zone.Scope
/benchmarkforopenrca(the benchmark step is skipped while it is the only benchmark), per-scenario--rows,difficultyOfexport, README and changesets.--jobsin the picker (runs use the default), resuming from the picker (the stop line gives the command), more benchmarks, mouse input./deslopandcontrol-cliare not installed, so neither ran./no-commentsran. It deleted one JSDoc on an internal type and flaggedRun.lines, which heldEntry[]. That field is nowRun.entries. No inline comments are added.Blast Radius
evorca evaluate --rows. The bare form is unchanged, and the flag now accepts<scenario>:values and repeats. New refusals: a value that names a scenario outside the run, and two values for the same scenario.run.jsonformat is unchanged. A series started onfeat/cli-evaluate-viewresumes and reports on this branch (below)./helplists/benchmark, and ctrl+c and/exitkeep their meaning outside a run.openShellnow takes the data and runs folders and anevaluatefunction.@evorca/openrcagains two exports. The scorer's behaviour is unchanged.Watchgains two optional props, and their defaults keepevorca watchandevorca evaluateas they were.Verification
pnpm checkpasses: Prettier, typecheck, tests (cli 194, openrca 46 plus 1 skipped, react 53, random 10, core 4), and build. The skipped openrca test needsmarket-cloudbed-2/record.csv, which I left out of the worktree to show a scenario without data.Mutation checks: each of these mutations, applied alone, fails the tests:
--rowsoverrides a scenario's rows--rows/exitleaves without asking11 of 11 were caught.
Every run used the
randomagent. The estimate frame reads a copy of one finishedreactrun.json, and nothing was run with it.Frames come from a real pty (pyte, 120 columns). In them,
◦marks a dot drawn dim (not chosen). The terminal draws both as•.Scenarios. market-cloudbed-2 has no tasks on disk, so it cannot be chosen:
Tasks, after
eandmdropped the easy and middle tasks of telecom (9 hard left):An agent that reads telemetry that a chosen scenario lacks:
Confirm for random, 2 runs, telecom with hard tasks plus two market-cloudbed-1 tasks:
Confirm with the estimate (react, telecom), from the copied finished run. Esc walks back and closes the picker:
The
--runs=2per-scenario run in the shell:The run recorded exactly the picked rows: telecom
[4,7,12,15,16,21,27,28,34]and market-cloudbed-1[0,2]in both runs. The summary lands in the log, and the shell keeps working:One ctrl+c gives a graceful stop. The log holds the command that started the run and the finish command, and the shell keeps running:
The finish command completes the series from the CLI:
/exitduring a run asks first:A second
/exitstops the run, and evorca prints the run's lines as it leaves. The process exited 0:Two ctrl+c in a row stop at once. The process exited 130, and no
randomcontainer or network was left:--rowson the CLI, all with the random agent:--rows=telecom:0,7 --rows=bank:3,4,10 --runs=2recorded telecom[0,7]and bank[3,4,10].--rows=0,7over telecom and market-cloudbed-1 recorded[0,7]for both.telecom rows: the run has [0,7], not [0]--rows=bank:0 names bank, but the run covers telecom.telecom has no row 99. Its tasks are rows 0 to 50.An old run resumes. I checked out
feat/cli-evaluate-view'spackages/cli/srcand started--rows=0,7 --runs=2on telecom and bank, then stopped it with SIGINT. Back on this branch,--resumefinished it: "bank: 2 tasks, 1 done", then run 2. Thenpnpm evorca reportprinted both scenarios as2 runs of random, rows 0,7.Scoring points: every frame file, the scrollbacks included (25 files), was checked against every scoring-point line of the real
query.csvof bank, telecom, and market-cloudbed-1 (287 lines). I also searched for "scoring" and "root cause component/reason/occurrence". Both checks found nothing. The shell tests also assert that thescoring_pointsof their fixture never reach a frame.