Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/cli-evaluate-view.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@evorca/cli": minor
---

`evorca evaluate` shows the run full screen in a terminal, in the view of `evorca watch`, with the lines that it prints under the feed. When the run ends or stops, the view closes and the command prints those lines, so the scrollback holds the same text as before. `--plain` and output without a terminal print the lines as the run goes, as before.
7 changes: 5 additions & 2 deletions packages/cli/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,9 @@ evorca reproduce --benchmark=<benchmark> [--scenario=<scenario>...] [--archive=<
evorca evaluate --benchmark=<benchmark> --agent=<agent> [--scenario=<scenario>...]
[--runs=<count>] [--rows=<row>,...] [--jobs=<count>] [--model=<model>]
[--effort=<effort>] [--budget-usd=<dollars>] [--task-budget-usd=<dollars>]
[--plain]
evorca evaluate --benchmark=<benchmark> --resume=<run ID> [--scenario=<scenario>...]
[--jobs=<count>] [--budget-usd=<dollars>]
[--jobs=<count>] [--budget-usd=<dollars>] [--plain]
evorca report --benchmark=<benchmark> [--scenario=<scenario>...] <run ID>...
evorca watch --benchmark=<benchmark> [--scenario=<scenario>...] [<run ID>]
```
Expand Down Expand Up @@ -161,6 +162,8 @@ A run with the same seed writes the same predictions, byte for byte, whatever it

With `--jobs=4`, four containers run at once, and the lines come in the order that the tasks finish. If the agent's tasks together ask for more memory than Docker has, the command prints a warning before the first task. On Docker Desktop, set the memory under **Settings** > **Resources**.

In a terminal, the command shows the run full screen in the view of [`evorca watch`](#watch-a-run), with the last four lines that it printed under the feed. When the run ends or stops, the view closes and the command prints every line as above, the accuracy among them, so the scrollback holds the same text either way. `--plain` prints the lines as the run goes instead, as for a run in the background. Without a terminal, as in CI or with output to a file, the command prints the lines.

For a pilot, run a few rows with `--rows=0,7,21`. The command still writes a `predictions.csv` row for every task, because the evaluator matches predictions to tasks by position, and gives the other rows the empty prediction `{}`. The accuracy and `scores.json` cover only the rows of the run.

### Evaluate an agent that calls a model
Expand Down Expand Up @@ -199,7 +202,7 @@ Once the run's budget is spent, each task that has not started gets the outcome

### Resume a run

To stop a run, press ctrl+c. The command starts no more tasks and removes the run's containers. It waits for the model requests in flight, so the run records their cost. It records each task that was running with the outcome `stopped` and what its `requests.jsonl` says it spent, so the totals in `run.json` match the request logs at once. Then it removes the run's networks and prints the command that resumes the run. Press ctrl+c again to stop at once.
To stop a run, press ctrl+c. The command starts no more tasks and removes the run's containers. It waits for the model requests in flight, so the run records their cost. It records each task that was running with the outcome `stopped` and what its `requests.jsonl` says it spent, so the totals in `run.json` match the request logs at once. Then it removes the run's networks and prints the command that resumes the run. Press ctrl+c again to stop at once. In the full-screen view, ctrl+c does the same: the first stops the run, and the view shows the stop until the run has stopped. The second closes the view and then stops at once.

```sh
pnpm evorca evaluate --benchmark=openrca --resume=20261117T121537Z-random
Expand Down
42 changes: 42 additions & 0 deletions packages/cli/src/evaluate-view.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
import { join } from "node:path";
import { describe, expect, it } from "vitest";
import { captureConsole, followRuns, type LogLine } from "./evaluate-view.js";
import { runRecord, runsFolder } from "./watch-fixtures.js";

describe("captureConsole", () => {
it("sends what the console prints to a function until it restores the console", () => {
const original = console.log;
const lines: LogLine[] = [];
const restore = captureConsole((line) => lines.push(line));
console.log("Seed %d.", 7);
console.error("Stopped.");
restore();

expect(lines).toEqual([
{ text: "Seed 7.", stream: "stdout" },
{ text: "Stopped.", stream: "stderr" },
]);
expect(console.log).toBe(original);
});
});

describe("followRuns", () => {
it("waits for the run that the evaluation starts, then shows it even when a newer run exists", async () => {
const folder = await runsFolder();
const { read, onRun } = followRuns({
runsDir: folder.runsDir,
dataDir: join(folder.runsDir, "no-data"),
scenarios: ["telecom"],
});
await folder.writeRun(runRecord({ runId: "20261118T094052Z-react" }));
await folder.writeRun(runRecord({ runId: "20261119T000000Z-react" }));

expect(await read({ kind: "live" })).toEqual({ kind: "waiting", message: "Starting the run…" });

onRun("20261118T094052Z-react");
expect(await read({ kind: "live" })).toMatchObject({
kind: "run",
state: { record: { runId: "20261118T094052Z-react" } },
});
});
});
139 changes: 139 additions & 0 deletions packages/cli/src/evaluate-view.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
/**
* `evorca evaluate` in a terminal: the run full screen in the view of `evorca
* watch`, with the last lines that the runner printed under the feed. When
* the runner settles, the view closes, the terminal is restored, and every
* line that the runner printed goes to stdout or stderr in order. So the
* scrollback holds the same text as a plain run, the score tables among it.
*
* Ctrl+c in the view sends SIGINT to the process, as ctrl+c does in a plain
* terminal. When SIGINT or SIGTERM comes and nothing else handles it, so that
* the signal ends the process, the view closes and prints the lines first.
*
* A view that throws while it draws shows the error in place of the run, and
* ctrl+c still reaches the runner, because the key is read outside the view.
* An uncaught error closes the view and prints the lines and the error, and
* the run goes on with plain lines.
*
* @module
*/
// @jsxRuntime automatic
// @jsxImportSource @opentui/react
import { format } from "node:util";
import { createCliRenderer } from "@opentui/core";
import { createRoot } from "@opentui/react";
import { palette } from "./shell.js";
import { readRunFolders, type Frame, type RunFolders } from "./watch-folder.js";
import { Watch } from "./watch-view.js";
import type { Selection } from "./watch.js";

/** A line that the runner printed, and the stream it was meant for. */
export type LogLine = { readonly text: string; readonly stream: "stdout" | "stderr" };

const signals = ["SIGINT", "SIGTERM"] as const;
const methods = [
["log", "stdout"],
["info", "stdout"],
["warn", "stderr"],
["error", "stderr"],
] as const;

/**
* Sends what `console.log`, `console.info`, `console.warn`, and
* `console.error` print to `onLine` instead of the terminal, as the console
* formats it, until the returned function restores them.
*/
export function captureConsole(onLine: (line: LogLine) => void): () => void {
const originals = methods.map(([method]) => [method, console[method]] as const);
for (const [method, stream] of methods) {
console[method] = (...args: unknown[]) => onLine({ text: format(...args), stream });
}
return () => {
for (const [method, original] of originals) console[method] = original;
};
}

/**
* Reads the runs of an evaluation for {@link Watch}. The view waits until
* `onRun` names a run, then shows the run that `onRun` named last.
*/
export function followRuns(folders: Omit<RunFolders, "runId">): {
readonly read: (selection: Selection) => Promise<Frame>;
readonly onRun: (runId: string) => void;
} {
let runId: string | undefined;
const read = readRunFolders({ ...folders, runId: () => runId });
return {
read: (selection) =>
runId === undefined
? Promise.resolve({ kind: "waiting", message: "Starting the run…" })
: read(selection),
onRun: (id) => {
runId = id;
},
};
}

/**
* Calls `evaluate` and shows its runs full screen until it settles, then
* prints what it printed. `evaluate` passes the ID of each run to `onRun`
* once the run's folder holds `run.json`.
*/
export async function evaluateFullScreen<T>(
folders: Omit<RunFolders, "runId">,
evaluate: (onRun: (runId: string) => void) => Promise<T>,
): Promise<T> {
const lines: LogLine[] = [];
const { read, onRun } = followRuns(folders);
const renderer = await createCliRenderer({
exitOnCtrlC: false,
exitSignals: [],
consoleMode: "disabled",
openConsoleOnError: false,
useMouse: false,
backgroundColor: palette.background,
});
const restoreConsole = captureConsole((line) => lines.push(line));

let flushed = false;
const flush = () => {
if (flushed) return;
flushed = true;
restoreConsole();
process.off("exit", flush).off("uncaughtException", crash).off("unhandledRejection", crash);
for (const signal of signals) process.off(signal, onSignal);
for (const { text, stream } of lines) process[stream].write(`${text}\n`);
};
let closing: Promise<void> | undefined;
const close = () =>
(closing ??= (async () => {
renderer.destroy();
await renderer.closed.catch(() => {});
flush();
})());
const crash = (error: unknown) => {
if (renderer.isDestroyed) console.error(error);
void close();
};
const onSignal = (signal: NodeJS.Signals) => {
if (process.listenerCount(signal) > 1) return;
void close().then(() => process.kill(process.pid, signal));
};
process.once("exit", flush).on("uncaughtException", crash).on("unhandledRejection", crash);
for (const signal of signals) process.prependListener(signal, onSignal);
renderer.keyInput.on("keypress", (key) => {
if (key.ctrl && key.name === "c") process.kill(process.pid, "SIGINT");
});

createRoot(renderer).render(
<Watch
read={read}
wallTime={() => new Date()}
log={() => lines.flatMap((line) => line.text.split("\n"))}
/>,
);
try {
return await evaluate(onRun);
} finally {
await close();
}
}
34 changes: 29 additions & 5 deletions packages/cli/src/main.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,12 @@
* that copy on disk instead of downloading them.
* `evorca evaluate` runs an agent from `agents/` on those scenarios, one Docker
* container per task, scores its answers, and keeps each run in
* `data/runs/<benchmark>/<scenario>/`. `evorca report` prints the summary of
* finished runs from those folders. `evorca watch` shows a run full screen
* while it runs, from its folder, and follows the newest run without a run ID.
* `data/runs/<benchmark>/<scenario>/`. In a terminal, it shows the run in the
* view of `evorca watch` unless `--plain` asks for lines. On SIGINT or
* SIGTERM, it stops the run and exits with 130 or 143. `evorca report` prints
* the summary of finished runs from those folders. `evorca watch` shows a run
* full screen while it runs, from its folder, and follows the newest run
* without a run ID.
*
* The command takes the real API key from `ANTHROPIC_API_KEY` and then deletes
* the variable, so no Docker command, container, or evaluator that it starts
Expand Down Expand Up @@ -106,7 +109,25 @@ const commands = {
apiKey,
upstreamUrl: readUpstreamUrl(),
};
if (!(await benchmark.evaluate(options))) process.exitCode = 1;
const stop = new AbortController();
let stopSignal: NodeJS.Signals | undefined;
const onSignal = (signal: NodeJS.Signals) => {
stopSignal = signal;
console.error("\nStopping. Press ctrl+c again to stop at once.");
stop.abort();
};
process.once("SIGINT", onSignal).once("SIGTERM", onSignal);
const run = (onRun?: (runId: string) => void) =>
benchmark.evaluate({ ...options, signal: stop.signal, onRun });
const fullScreen = !flags.plain && process.stdin.isTTY && process.stdout.isTTY;
const view = fullScreen ? await import("./evaluate-view.js") : undefined;
const evaluation =
view === undefined ? run() : view.evaluateFullScreen({ ...dirs, scenarios }, run);
const ok = await evaluation.finally(() => {
process.off("SIGINT", onSignal).off("SIGTERM", onSignal);
if (stopSignal !== undefined) process.exit(stopSignal === "SIGINT" ? 130 : 143);
});
if (!ok) process.exitCode = 1;
},
async report({ benchmark, scenarios, runIds, runsDir }: Invocation) {
if (!(await benchmark.report({ scenarios, runIds, runsDir }))) process.exitCode = 1;
Expand Down Expand Up @@ -135,8 +156,9 @@ const usage = [
" evorca evaluate --benchmark=<benchmark> --agent=<agent> [--scenario=<scenario>...]",
" [--runs=<count>] [--rows=<row>,...] [--jobs=<count>] [--model=<model>]",
" [--effort=<effort>] [--budget-usd=<dollars>] [--task-budget-usd=<dollars>]",
" [--plain]",
" evorca evaluate --benchmark=<benchmark> --resume=<run id> [--scenario=<scenario>...]",
" [--jobs=<count>] [--budget-usd=<dollars>]",
" [--jobs=<count>] [--budget-usd=<dollars>] [--plain]",
" evorca report --benchmark=<benchmark> [--scenario=<scenario>...] <run id>...",
" evorca watch --benchmark=<benchmark> [--scenario=<scenario>...] [<run id>]",
].join("\n");
Expand Down Expand Up @@ -198,6 +220,7 @@ type Flags = {
readonly effort?: string | undefined;
readonly "budget-usd"?: string | undefined;
readonly "task-budget-usd"?: string | undefined;
readonly plain?: boolean | undefined;
};

function parseCommandLine() {
Expand All @@ -217,6 +240,7 @@ function parseCommandLine() {
effort: { type: "string" },
"budget-usd": { type: "string" },
"task-budget-usd": { type: "string" },
plain: { type: "boolean" },
},
});
} catch (error: unknown) {
Expand Down
Loading
Loading