Skip to content

feat(cli): add evorca evaluate and the random agent - #18

Merged
kottinov merged 4 commits into
mainfrom
feat/cli-evaluate
Nov 17, 2026
Merged

kottinov merged 4 commits into
mainfrom
feat/cli-evaluate

Conversation

@kottinov

Copy link
Copy Markdown
Collaborator

Why

evorca needs a first runnable OpenRCA baseline. It could reproduce OpenRCA's published scores but could not run an agent on the benchmark. pnpm evorca evaluate --benchmark=openrca --agent=random now runs the whole path with a chance-level agent that needs no model and no telemetry, before the ReAct baseline arrives.

What changed

  • evorca evaluate runs each task in its own Docker container, scores the answers with the official evaluator, and prints the same accuracy table as evorca reproduce. Each run lands in data/runs/openrca/<scenario>/<run ID>/ with predictions.csv, scores.json, the task folders, and run.json.
  • The agents table in packages/cli/src/agents.ts says whether an agent sees the candidates, whether it needs telemetry, and its timeout. The command builds the image from agents/<name>/, tagged by a hash of the folder.
  • agents/random/ answers each failure with a random minute in the task's window and a random candidate component and reason. EVORCA_SEED repeats its answers exactly.

Scope

  • Covers the container contract and one agent. Parallel tasks, resumed runs, more agents, and the model proxy for ReAct are follow-ups.
  • When query.csv is missing, the command prints pnpm evorca fetch, which also downloads the scenario's telemetry.

Blast Radius

The command and the agent package are new, and fetch and reproduce behave as before. A container gets no network and never sees query.csv or record.csv, so an agent cannot read the answers.

Verification

  • pnpm check passes. The unit tests cover the docker run arguments, the five outcomes, the random agent's repeatability, and an unknown --agent.
  • Two runs of EVORCA_SEED=7 pnpm evorca evaluate --benchmark=openrca --scenario=telecom --agent=random took 13 to 15 s each and wrote byte-identical predictions.csv files. Both scored 5.88% strict and 8.80% partial accuracy. A bank run scored 5.15% and 10.42%.
  • A throwaway probe agent, not committed, ran as nobody and saw only the telemetry folder at /data. It found no query.csv, record.csv, or scoring points, had no network, and could not write outside /out. It also produced each failure outcome once.

Telecom's record.csv ends with an empty line, and parseCsv failed on it
with "Invalid Record Length". The evaluator reads its CSV files with
pandas, which skips empty lines, so parseCsv now does the same.
RCA-Agent's prompt for each OpenRCA system lists the root cause
components and reasons that an answer can name. Whether an agent sees
these lists changes how hard the benchmark is, so evorca needs them as
data to give an agent the same view as RCA-Agent.

openRcaCandidates holds the lists per scenario. The maintainer script
scripts/extract-candidates.ts writes src/candidates.json from the
basic_prompt_<System>.py files at the pinned OpenRCA commit and trims
the trailing spaces that some prompt entries have. A test checks that
the lists include every component and reason in each scenario's
record.csv when the answer keys are on disk.
The thesis needs a chance-level floor for OpenRCA: an agent that reads
no telemetry and answers at random from the candidate root causes.

agents/random/ is a workspace package with a Dockerfile on a pinned
node:26-alpine digest. random.ts reads /task.json, finds the time window
and the number of failures in the instruction, and writes one cause per
failure to /out/prediction.json: a minute drawn uniformly from the
window, and a component and a reason drawn uniformly from the
candidates. Each draw comes from the SHA-256 of EVORCA_SEED, the
scenario, the row, and a counter, so a seed repeats a run exactly.
evorca could reproduce OpenRCA's published scores but could not run an
agent on the benchmark. `pnpm evorca evaluate --benchmark=openrca
--agent=<agent>` now runs each task in its own Docker container, scores
the answers with the official evaluator, and prints the accuracy table
that `evorca reproduce` prints.

The agents table in src/agents.ts says, per agent, whether it sees the
candidate root causes, whether it needs telemetry, and how long a task
may run. The command builds the agent's image from agents/<name>/,
tagged by a hash of the folder. A container gets /task.json read-only,
a writable /out, and the scenario's telemetry at /data only if the agent
needs it, never the answer keys. It has no network, a read-only root,
no capabilities, an unprivileged user, and limits on memory, CPU, and
processes. A missing or invalid prediction, a non-zero exit, and a
timeout each score {} and are recorded as the task's outcome.

Each run lands in data/runs/openrca/<scenario>/<run ID>/ with the
per-task folders, predictions.csv, scores.json, and run.json, which
records the image, the seed, the git commit, the pinned inputs, the
versions, and each task's outcome and time.
@kottinov
kottinov changed the base branch from feat/openrca-candidates to main November 17, 2026 12:53
@kottinov
kottinov merged commit 16a7123 into main Nov 17, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant