Repository navigation
feat(cli): add evorca evaluate and the random agent - #18
Merged
Merged
Conversation
Telecom's record.csv ends with an empty line, and parseCsv failed on it with "Invalid Record Length". The evaluator reads its CSV files with pandas, which skips empty lines, so parseCsv now does the same.
RCA-Agent's prompt for each OpenRCA system lists the root cause components and reasons that an answer can name. Whether an agent sees these lists changes how hard the benchmark is, so evorca needs them as data to give an agent the same view as RCA-Agent. openRcaCandidates holds the lists per scenario. The maintainer script scripts/extract-candidates.ts writes src/candidates.json from the basic_prompt_<System>.py files at the pinned OpenRCA commit and trims the trailing spaces that some prompt entries have. A test checks that the lists include every component and reason in each scenario's record.csv when the answer keys are on disk.
The thesis needs a chance-level floor for OpenRCA: an agent that reads no telemetry and answers at random from the candidate root causes. agents/random/ is a workspace package with a Dockerfile on a pinned node:26-alpine digest. random.ts reads /task.json, finds the time window and the number of failures in the instruction, and writes one cause per failure to /out/prediction.json: a minute drawn uniformly from the window, and a component and a reason drawn uniformly from the candidates. Each draw comes from the SHA-256 of EVORCA_SEED, the scenario, the row, and a counter, so a seed repeats a run exactly.
evorca could reproduce OpenRCA's published scores but could not run an
agent on the benchmark. `pnpm evorca evaluate --benchmark=openrca
--agent=<agent>` now runs each task in its own Docker container, scores
the answers with the official evaluator, and prints the accuracy table
that `evorca reproduce` prints.
The agents table in src/agents.ts says, per agent, whether it sees the
candidate root causes, whether it needs telemetry, and how long a task
may run. The command builds the agent's image from agents/<name>/,
tagged by a hash of the folder. A container gets /task.json read-only,
a writable /out, and the scenario's telemetry at /data only if the agent
needs it, never the answer keys. It has no network, a read-only root,
no capabilities, an unprivileged user, and limits on memory, CPU, and
processes. A missing or invalid prediction, a non-zero exit, and a
timeout each score {} and are recorded as the task's outcome.
Each run lands in data/runs/openrca/<scenario>/<run ID>/ with the
per-task folders, predictions.csv, scores.json, and run.json, which
records the image, the seed, the git commit, the pinned inputs, the
versions, and each task's outcome and time.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
evorca needs a first runnable OpenRCA baseline. It could reproduce OpenRCA's published scores but could not run an agent on the benchmark.
pnpm evorca evaluate --benchmark=openrca --agent=randomnow runs the whole path with a chance-level agent that needs no model and no telemetry, before the ReAct baseline arrives.What changed
evorca evaluateruns each task in its own Docker container, scores the answers with the official evaluator, and prints the same accuracy table asevorca reproduce. Each run lands indata/runs/openrca/<scenario>/<run ID>/withpredictions.csv,scores.json, the task folders, andrun.json.agentstable inpackages/cli/src/agents.tssays whether an agent sees the candidates, whether it needs telemetry, and its timeout. The command builds the image fromagents/<name>/, tagged by a hash of the folder.agents/random/answers each failure with a random minute in the task's window and a random candidate component and reason.EVORCA_SEEDrepeats its answers exactly.Scope
query.csvis missing, the command printspnpm evorca fetch, which also downloads the scenario's telemetry.Blast Radius
The command and the agent package are new, and
fetchandreproducebehave as before. A container gets no network and never seesquery.csvorrecord.csv, so an agent cannot read the answers.Verification
pnpm checkpasses. The unit tests cover thedocker runarguments, the five outcomes, the random agent's repeatability, and an unknown--agent.EVORCA_SEED=7 pnpm evorca evaluate --benchmark=openrca --scenario=telecom --agent=randomtook 13 to 15 s each and wrote byte-identicalpredictions.csvfiles. Both scored 5.88% strict and 8.80% partial accuracy. Abankrun scored 5.15% and 10.42%.nobodyand saw only the telemetry folder at/data. It found noquery.csv,record.csv, or scoring points, had no network, and could not write outside/out. It also produced each failure outcome once.