Skip to content

feat(cli): pick and run a benchmark with /benchmark in the shell - #29

Open
kottinov wants to merge 3 commits into
feat/cli-evaluate-viewfrom
feat/cli-benchmark
Open

kottinov wants to merge 3 commits into
feat/cli-evaluate-viewfrom
feat/cli-benchmark

Conversation

@kottinov

Copy link
Copy Markdown
Collaborator

Why

Picking what to evaluate meant composing pnpm evorca evaluate flags by hand, and --rows could only give the same rows to every scenario. /benchmark in the shell picks the scenarios, the tasks in each, and the agent, shows the exact command, and runs it in the shell. It never shows the scoring points.

What changed

Three commits, each of which typechecks and passes the tests on its own:

  1. feat(openrca): export the difficulty of each task index. difficultyOf and Difficulty move from the scoring summary to tasks.ts, and the package exports them so the picker groups tasks the way the scorer does.
  2. feat(cli): give --rows per scenario. --rows can repeat, and --rows=<scenario>:<rows> gives the rows of one scenario. A bare --rows=0,7,21 still covers every scenario that no other value names. run.json keeps its format: rows was already stored per scenario, so a resume checks each scenario's rows against its own record. parseRows moves to runs.ts with the scenario list that report.ts and openrca.ts share.
  3. feat(cli): pick and run a benchmark with /benchmark in the shell.
    • The data shape comes first. Catalog holds what the picker offers: ScenarioData[] (tasks, telemetry on disk) and the RunRecords that the estimate learns from. Picker is the state (step, cursor, chosen scenarios, chosen rows per scenario, settings, seed), and Plan is the evaluate request that a pick describes.
    • benchmark.ts is a pure state machine: update(picker, action, catalog), itemsOf, blockerOf, planOf, commandOf, estimateOf. Only readCatalog touches the disk. benchmark-view.tsx maps keys to actions and draws lines.
    • The run goes in the shell's process through evaluateOpenRca with an AbortSignal, in the Watch view with the input line under the feed. Watch takes keys and children so the prompt can own the keys while it holds text.
    • ctrl+c stops gracefully, and a second stops at once and exits with 130. /exit during a run asks first, then stops the run and leaves once it has stopped. The run's lines go into the transcript, or to stdout when evorca leaves.
    • The estimate is the mean cost per task of finished runs in data/runs with the same agent, model, and effort, times the tasks and runs. It is labelled an estimate, it is absent without such a run, and it uses no price table.
    • The command pins the seed (EVORCA_SEED=<seed> pnpm evorca evaluate ...), uses --rows=<scenario>:<rows> only for scenarios with tasks left out, and wraps at 72 columns with \ so a copy runs as it is.
    • palette, Mark, Hints, Footer, and the rest of the shell's chrome move verbatim to parts.tsx. The shell now imports the watch view, which imported them from the shell, and that cycle hits the temporal dead zone.

Scope

  • In scope: /benchmark for openrca (the benchmark step is skipped while it is the only benchmark), per-scenario --rows, difficultyOf export, README and changesets.
  • Out of scope: --jobs in the picker (runs use the default), resuming from the picker (the stop line gives the command), more benchmarks, mouse input.
  • Tooling: /deslop and control-cli are not installed, so neither ran. /no-comments ran. It deleted one JSDoc on an internal type and flagged Run.lines, which held Entry[]. That field is now Run.entries. No inline comments are added.

Blast Radius

  • evorca evaluate --rows. The bare form is unchanged, and the flag now accepts <scenario>: values and repeats. New refusals: a value that names a scenario outside the run, and two values for the same scenario.
  • run.json format is unchanged. A series started on feat/cli-evaluate-view resumes and reports on this branch (below).
  • The shell: /help lists /benchmark, and ctrl+c and /exit keep their meaning outside a run. openShell now takes the data and runs folders and an evaluate function.
  • @evorca/openrca gains two exports. The scorer's behaviour is unchanged.
  • Watch gains two optional props, and their defaults keep evorca watch and evorca evaluate as they were.

Verification

  • pnpm check passes: Prettier, typecheck, tests (cli 194, openrca 46 plus 1 skipped, react 53, random 10, core 4), and build. The skipped openrca test needs market-cloudbed-2/record.csv, which I left out of the worktree to show a scenario without data.

  • Mutation checks: each of these mutations, applied alone, fails the tests:

    • a bare --rows overrides a scenario's rows
    • a named scenario outside the run is accepted
    • whole scenarios get --rows
    • a toggle always chooses
    • a scenario without tasks can be chosen
    • the estimate ignores the effort
    • the estimate ignores the runs
    • the command drops the seed
    • a second ctrl+c waits
    • /exit leaves without asking
    • the run's lines stay out of the log

    11 of 11 were caught.

  • Every run used the random agent. The estimate frame reads a copy of one finished react run.json, and nothing was run with it.

  • Frames come from a real pty (pyte, 120 columns). In them, ◦ marks a dot drawn dim (not chosen). The terminal draws both as •.

Scenarios. market-cloudbed-2 has no tasks on disk, so it cannot be chosen:

  Scenarios   openrca, step 1 of 4
    ◦ bank                136 tasks, no telemetry
    • telecom             51 tasks, with telemetry
    • market-cloudbed-1   70 tasks, no telemetry
  › × market-cloudbed-2   Fetch its tasks with pnpm evorca fetch --benchmark=openrca --scenario=market-cloudbed-2
  ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  ↑↓ move   space choose   enter next   esc close

Tasks, after e and m dropped the easy and middle tasks of telecom (9 hard left):

  Tasks   openrca, step 2 of 4
    • telecom   9 of 51 tasks
  › ◦   000  task_2  easy     During the specified time range of April 11, 2020, from 00:00 to 00:30, there was one fa
    ◦   001  task_4  middle   Within the time range of April 11, 2020, from 01:30 to 02:00, there was one failure in t
    ◦   002  task_2  easy     On April 11, 2020, between 02:00 and 02:30, a failure occurred in the system. The number
    ◦   003  task_2  easy     A single failure was detected during the time range of April 11, 2020, from 02:30 to 03:
    •   004  task_7  hard     On April 11, 2020, between 04:30 and 05:00, there was a reported failure in the system.
    ◦   005  task_1  easy     Within the specified time range of April 11, 2020, from 05:30 to 06:00, a failure was de
    ◦   006  task_4  middle   There was a single failure in the system within the time range of May 22, 2020, from 00:
    •   007  task_7  hard     During the specified time range of May 22, 2020, from 00:30 to 01:00, the system experie
    ◦   008  task_5  middle   During the time range of May 22, 2020, from 01:30 to 02:00, there was one failure report
    ◦   009  task_1  easy     On May 22, 2020, between 02:00 and 02:30, the system encountered a failure. The exact ti

An agent that reads telemetry that a chosen scenario lacks:

  Agent and runs   openrca, step 3 of 4
  › • agent        react
    • runs         1
    • model        claude-sonnet-5
    • effort       high
    • run budget   $50
    • task budget  $2
  Fetch the telemetry that react reads with pnpm evorca fetch --benchmark=openrca --scenario=market-cloudbed-1
  ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  ↑↓ move   ←→ change   enter next   esc back

Confirm for random, 2 runs, telecom with hard tasks plus two market-cloudbed-1 tasks:

  Run   openrca, step 4 of 4
  tasks     11 per run: telecom 9 of 51, market-cloudbed-1 2 of 70
  agent     random, 2 runs, seed 954091097
  command   EVORCA_SEED=954091097 pnpm evorca evaluate --benchmark=openrca \
              --agent=random --scenario=telecom --scenario=market-cloudbed-1 \
              --rows=telecom:4,7,12,15,16,21,27,28,34 --rows=market-cloudbed-1:0,2 \
              --runs=2
  ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  enter run   esc back

Confirm with the estimate (react, telecom), from the copied finished run. Esc walks back and closes the picker:

  Run   openrca, step 4 of 4
  tasks     51 per run: telecom 51 of 51
  agent     react, 1 run, seed 2759723334
  model     claude-sonnet-5 high
  budget    $50 per run, $2 per task
  estimate  about $22.61, at $0.4434 per task over 51 tasks of finished runs
  command   EVORCA_SEED=2759723334 pnpm evorca evaluate --benchmark=openrca \
              --agent=react --scenario=telecom --model=claude-sonnet-5 \
              --effort=high --budget-usd=50 --task-budget-usd=2
  ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  enter run   esc back
  13:44:12  you     /benchmark
  13:44:17  evorca  No run started.
  › What changed before node-6 started
  ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  enter send   /help commands   ctrl+c exit

The --runs=2 per-scenario run in the shell:

  evorca   telecom   run 1 of 2   20261118T115427Z-random-1
  5/9 tasks   5 answered   $0.0000 spent   1s elapsed   about 1s left, estimated
  • • • • • ◦ ◦ ◦ ◦
  task 021   task_7   running   $0.0000                                                                 live   tailing
  run 1/2 telecom 2/9 task_7 answered 0.21 s
  run 1/2 telecom 3/9 task_7 answered 0.22 s
  run 1/2 telecom 4/9 task_7 answered 0.20 s
  run 1/2 telecom 5/9 task_7 answered 0.21 s
  › /exit to stop the run and leave
  ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  ↑↓ scroll   end tail   o output   ←→ task   tab running   f live   ctrl+c stop
  data/runs/openrca/telecom/20261118T115427Z-random-1                                                    evorca v0.0.0

The run recorded exactly the picked rows: telecom [4,7,12,15,16,21,27,28,34] and market-cloudbed-1 [0,2] in both runs. The summary lands in the log, and the shell keeps working:

  13:54:38  evorca  | ------------------------- | --------: | ---------: | ----------: | -------: | ---------: | -----
                    ---: |
  13:54:38  evorca  | 20261118T115427Z-random-1 | 954091097 |       0.00 |       25.00 |        - |          - |   0:
                    00:01 |
  13:54:38  evorca  | 20261118T115427Z-random-2 | 954091098 |       0.00 |        0.00 |        - |          - |   0:
                    00:01 |
  13:54:38  evorca  | Mean                      |           |            |             |        - |          - |   0:
                    00:01 |
  13:54:38  evorca  | Total                     |           |            |             |        - |          - |   0:
                    00:02 |
  13:54:38  evorca
                    Print this summary again with: pnpm evorca report --benchmark=openrca --scenario=telecom --
                    scenario=market-cloudbed-1 20261118T115427Z-random-1 20261118T115427Z-random-2
  ›
  ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  enter send   /help commands   ctrl+c exit
                    scenario=market-cloudbed-1 20261118T115427Z-random-1 20261118T115427Z-random-2
  13:54:39  you     /help
  13:54:39  evorca  /help       List the commands
                    /benchmark  Pick scenarios, tasks, and an agent, and run them
                    /exit       Exit evorca
  › Why did checkout p99 latency spik
  ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  enter send   /help commands   ctrl+c exit

One ctrl+c gives a graceful stop. The log holds the command that started the run and the finish command, and the shell keeps running:

  13:55:02  you     /benchmark
  13:55:05  evorca  Running the evaluation that this command repeats:
                    EVORCA_SEED=1024454525 pnpm evorca evaluate --benchmark=openrca \
                      --agent=random --scenario=telecom --runs=2
  13:55:05  evorca  2 runs, 20261118T115505Z-random-1 to 20261118T115505Z-random-2, seeds 1024454525 to 1024454526.
  13:55:05  evorca  If they stop, finish them with: pnpm evorca evaluate --benchmark=openrca --
                    resume=20261118T115505Z-random-1
  13:55:05  evorca
  13:55:05  evorca  Agent random, image evorca-agent-random:360560e3bc6a, sha256:
                    7794c85693d8640838d4843d464c93648e8f00ebcbb8595aac5519f3c7f64653
  13:55:05  evorca  Seed 1024454525.
  13:55:05  evorca  git 5f28ec8998b2fb81f999ba788c314dbd14d5c04a
  13:55:05  evorca  node v26.13.1, docker 28.2.2, daemon 28.2.2
  13:55:05  evorca  Run 1 of 2, 20261118T115505Z-random-1, 1 task at a time.
  13:55:06  evorca
                    telecom: 51 tasks, run folder data/runs/openrca/telecom/20261118T115505Z-random-1
  13:55:06  evorca  run 1/2 telecom 1/51 task_2 answered 0.30 s
  13:55:06  evorca  run 1/2 telecom 2/51 task_4 answered 0.19 s
  13:55:06  evorca  run 1/2 telecom 3/51 task_2 answered 0.20 s
  13:55:07  evorca  run 1/2 telecom 4/51 task_2 answered 0.20 s
  13:55:07  evorca  run 1/2 telecom 5/51 task_7 answered 0.18 s
  13:55:07  evorca  run 1/2 telecom 6/51 task_1 answered 0.20 s
  13:55:07  evorca  run 1/2 telecom 7/51 task_4 answered 0.21 s
  13:55:07  evorca  Stopping. Press ctrl+c again to stop at once.
  13:55:08  evorca  Stopped. Finish the 2 runs with: pnpm evorca evaluate --benchmark=openrca --
                    resume=20261118T115505Z-random-1
  › What changed before node-6 started OOM-killing pods?
  ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  enter send   /help commands   ctrl+c exit
                    resume=20261118T115505Z-random-1
  13:55:09  you     /help
  13:55:09  evorca  /help       List the commands
                    /benchmark  Pick scenarios, tasks, and an agent, and run them
                    /exit       Exit evorca
  › Why did checkout p99 latency spi
  ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  enter send   /help commands   ctrl+c exit

The finish command completes the series from the CLI:

$ pnpm evorca evaluate --benchmark=openrca --resume=20261118T115505Z-random-1 --plain
Resuming run 1 of 2, 20261118T115505Z-random-1, 1 task at a time.
telecom: 51 tasks, 7 done, run folder data/runs/openrca/telecom/20261118T115505Z-random-1
Run 2 of 2, 20261118T115505Z-random-2, 1 task at a time.
| 20261118T115505Z-random-1 | 1024454525 |       3.92 |        9.14 |        - |          - |  0:00:25* |
| 20261118T115505Z-random-2 | 1024454526 |       3.92 |        9.76 |        - |          - |   0:00:13 |
| Mean                      |            |            |             |        - |          - |   0:00:19 |

/exit during a run asks first:

  evorca   telecom   20261118T114701Z-random
  10/51 tasks   10 answered   $0.0000 spent   3s elapsed   about 10s left, estimated
  • • • • • • • • • •   • ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦   ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦   ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦   ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦ ◦   ◦
  task 010   task_1   running   $0.0000                                                                 live   tailing
  A run is going. Type /exit again to stop it and leave evorca.
  telecom 8/51 task_7 answered 0.20 s
  telecom 9/51 task_5 answered 0.24 s
  telecom 10/51 task_1 answered 0.21 s
  › /exit to stop the run and leave
  ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  ↑↓ scroll   end tail   o output   ←→ task   tab running   f live   ctrl+c stop
  data/runs/openrca/telecom/20261118T114701Z-random                                                      evorca v0.0.0

A second /exit stops the run, and evorca prints the run's lines as it leaves. The process exited 0:

telecom 12/51 task_6 answered 0.19 s
telecom 13/51 task_7 answered 0.19 s
› /exit
Stopping. evorca leaves once the run has stopped.
Stopped. Resume with: pnpm evorca evaluate --benchmark=openrca --resume=20261118T114701Z-random

Two ctrl+c in a row stop at once. The process exited 130, and no random container or network was left:

telecom 5/51 task_7 answered 0.22 s
telecom 6/51 task_1 answered 0.18 s
Stopping. Press ctrl+c again to stop at once.

--rows on the CLI, all with the random agent:

  • --rows=telecom:0,7 --rows=bank:3,4,10 --runs=2 recorded telecom [0,7] and bank [3,4,10].
  • --rows=0,7 over telecom and market-cloudbed-1 recorded [0,7] for both.
  • Refusals:
    • telecom rows: the run has [0,7], not [0]
    • --rows=bank:0 names bank, but the run covers telecom.
    • telecom has no row 99. Its tasks are rows 0 to 50.

An old run resumes. I checked out feat/cli-evaluate-view's packages/cli/src and started --rows=0,7 --runs=2 on telecom and bank, then stopped it with SIGINT. Back on this branch, --resume finished it: "bank: 2 tasks, 1 done", then run 2. Then pnpm evorca report printed both scenarios as 2 runs of random, rows 0,7.

Scoring points: every frame file, the scrollbacks included (25 files), was checked against every scoring-point line of the real query.csv of bank, telecom, and market-cloudbed-1 (287 lines). I also searched for "scoring" and "root cause component/reason/occurrence". Both checks found nothing. The shell tests also assert that the scoring_points of their fixture never reach a frame.

The shell's benchmark picker groups tasks by difficulty the way the scorer does, so difficultyOf and Difficulty move from the scoring summary to tasks.ts and the package exports them.
--rows can repeat, and a value that starts with a scenario, such as --rows=bank:3,4, gives the rows of that scenario only. A bare --rows=0,7,21 still gives the rows of every scenario that no other value names. run.json keeps its format, so a resume checks the rows of each scenario against its own record, and earlier runs resume and report as before. The scenario list moves to runs.ts so the parser and report share it.
/benchmark opens a picker of four steps: the scenarios, the tasks of each with their difficulty, the agent and its settings, and a confirmation with the evorca evaluate command that repeats the run and, for an agent that calls a model, a cost estimate from finished runs. The picker is a pure state machine in benchmark.ts, and benchmark-view.tsx only draws it. It never shows the scoring points.

The run goes in the shell's process, in the view of evorca watch, with the input line under the feed. ctrl+c stops it and returns to the transcript with the run's lines and the command that finishes it. A second ctrl+c stops at once. /exit asks first, then stops the run and leaves. The shell's palette and chrome move to parts.tsx because the shell now imports the watch view, which imported them from the shell.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant