Skip to content

feat(cli): show evorca evaluate full screen in a terminal - #28

Open
kottinov wants to merge 2 commits into
feat/cli-watchfrom
feat/cli-evaluate-view
Open

kottinov wants to merge 2 commits into
feat/cli-watchfrom
feat/cli-evaluate-view

Conversation

@kottinov

Copy link
Copy Markdown
Collaborator

Why

A 5-run evorca evaluate series takes hours, and you want to watch it in evorca's own terminal UI instead of a stream of lines. This PR builds on #27: in a terminal, evaluate now opens the evorca watch view on the run it starts, and the lines it prints show under the feed. When the run ends or stops, the view closes and the scrollback gets the same text as before, score tables included.

What changed

  • When stdin and stdout are both terminals, evorca evaluate shows the run full screen. It follows each run of a --runs series as the runner starts it, and the runner's last four lines show under the feed. --plain keeps the old line output.
  • While the view is open, evaluate-view.tsx captures console.log, info, warn, and error. When the run settles, the view closes, the terminal is restored, and every captured line goes to stdout or stderr in order.
  • The runner takes an AbortSignal and an onRun callback. It no longer handles SIGINT or SIGTERM and no longer calls process.exit. The evaluate command now owns the signals: it prints the same "Stopping." line, aborts the run, and exits with 130 or 143 as before.
  • Ctrl+c in the view sends SIGINT to the process, so a key press and kill -INT take the same path. When a SIGINT or SIGTERM comes and nothing else handles it, as with the second ctrl+c, the view closes and prints the lines first. Then the signal ends the process.
  • Watch takes a log getter and an optional onExit. Without onExit, the hints show ctrl+c stop instead of q quit.

Scope

  • No new evaluate behavior beyond the view. Plain output is the same as before, line for line.
  • The random agent has no trajectory, so its feed stays empty. A react run fills it as in feat(cli): watch a run full screen with evorca watch #27.
  • The runner can now run in-process without owning the process, which leaves room for a later shell /benchmark command. This PR does not add that command.

Blast Radius

  • Every evaluate run goes through the new signal handling in main.ts. A signal that comes before a run writes its folder, for example during the image build or between two runs of a series, now ends the command with 130 or 143 and writes nothing. Before, it killed the process at once. An error that the stop causes, such as a scorer that the terminal's SIGINT killed, still exits 130 as before. A second ctrl+c still stops at once.
  • The view only reads the run folder, and run.json is rewritten after each task, so a failing view loses no run data. If the view throws while it draws, the error shows in its place, and ctrl+c still stops the run because the view reads that key outside its components. While the view is open, OpenTUI catches uncaught errors. On one, the view closes and prints the lines and the error, and the run goes on with plain lines.
  • A run started in the background with & from a terminal still sees a terminal, so it opens the view. Setting up the terminal from a background job gets it suspended, as with any full-screen program. I did not try this. The README says to use --plain for a run in the background.

Verification

  • pnpm check passes. New tests cover the console capture, a reader that waits for the run the evaluation names (even when a newer run exists), and the view's log and stop hint. Each of 5 mutations to those parts fails a test.
  • Plain output: the same 3-row, 2-run random series on main and on this branch, with no terminal and EVORCA_SEED=7, gives the same stdout and stderr once run IDs, task seconds, the commit, and wall times are masked.
  • On a real pseudo-terminal with the random agent:
    • A 12-row, 2-run series opened the view, followed run 2, exited 0, and left both runs' accuracy tables and the series summary in the scrollback.
    • One ctrl+c during a 2-run series: the "Stopping." line showed in the view, and the runner recorded the stop. The view closed, and the scrollback ended with Stopped. Finish the 2 runs with: pnpm evorca evaluate --benchmark=openrca --resume=…. Under node, the exit code was 130, as on main.
    • The printed finish command resumed run 1, then ran run 2 in the view, and exited 0.
    • Two ctrl+c presses 100 ms apart: the view closed, and the process ended by SIGINT, as on main. No containers were left.
    • Two kill -TERM 20 ms apart: the view closed, the lines reached the scrollback, and the process ended by SIGTERM.
    • Ctrl+c while the view said "Starting the run…": the command exited 130 and wrote no run folder.
    • --plain with one ctrl+c printed the same lines as main and exited 130.
    • A fake evaluation whose run cannot be drawn showed the error, and ctrl+c still stopped the run. An uncaught error in a timer closed the view and printed the error, and the fake run went on to its end.
    • After every exit, the terminal was back in canonical mode with echo and signals, the alternate screen was closed, and the cursor showed.

The view during a run, after 5 of 51 tasks:

  evorca   telecom   run 1 of 2   20261118T103707Z-random-1
  5/51 tasks   5 answered   $0.0000 spent   1s elapsed   about 11s left, estimated

  • • • • • • • • • •   • • • • • • • • • •   • • • • • • • • • •   • • • • • • • • • •   • • • • • • • • • •   •

  task 005   task_1   running   $0.0000                                                                 live   tailing

  run 1/2 telecom 2/51 task_4 answered 0.22 s
  run 1/2 telecom 3/51 task_2 answered 0.22 s
  run 1/2 telecom 4/51 task_2 answered 0.23 s
  run 1/2 telecom 5/51 task_7 answered 0.25 s

  ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
  ↑↓ scroll   end tail   o output   ←→ task   tab running   f live   ctrl+c stop

  data/runs/openrca/telecom/20261118T103707Z-random-1                                                    evorca v0.0.0

The scrollback after that ctrl+c:

run 1/2 telecom 4/51 task_2 answered 0.23 s
run 1/2 telecom 5/51 task_7 answered 0.25 s

Stopping. Press ctrl+c again to stop at once.
Stopped. Finish the 2 runs with: pnpm evorca evaluate --benchmark=openrca --resume=20261118T103707Z-random-1

The runner takes the signal that stops it and returns once the run has
stopped, instead of handling SIGINT and SIGTERM and exiting itself. The
evaluate command owns the signals and the exit code, so another caller
can run an evaluation in the same process and go on afterwards. A signal
that comes before a run writes its folder now ends the command without
the run.
In a terminal, evorca evaluate shows the run in the view of evorca watch,
with the last lines that it printed under the feed. When the run ends or
stops, the view closes and the command prints every line, so the
scrollback holds the same text as before. --plain, or output without a
terminal, prints the lines as the run goes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant