Skip to content

feat(openrca): export openRcaCandidates from RCA-Agent's prompts - #17

Merged
kottinov merged 2 commits into
mainfrom
feat/openrca-candidates
Nov 17, 2026
Merged

kottinov merged 2 commits into
mainfrom
feat/openrca-candidates

Conversation

@kottinov

Copy link
Copy Markdown
Collaborator

Why

evorca needs a first runnable OpenRCA baseline, and whether an agent sees the candidate root causes is a fairness choice that the thesis must state. RCA-Agent's prompts always list these candidates, so evorca needs them as data before it can give an agent the same view.

What changed

  • openRcaCandidates in @evorca/openrca maps each scenario to the components and reasons that an answer can name.
  • scripts/extract-candidates.ts writes src/candidates.json from basic_prompt_<System>.py at the pinned OpenRCA commit. It trims the trailing spaces that some prompt entries have, because the evaluator compares names exactly.
  • parseCsv skips empty lines, as the evaluator's pandas does, because Telecom's record.csv ends with one.

Scope

  • Adds the data and a check. No agent reads the lists yet. The next PR puts them in /task.json for agents that see candidates.
  • The coverage test reads record.csv from data/benchmarks/openrca/, so it runs only where the answer keys are on disk. CI skips it.

Blast Radius

The new export and JSON file touch nothing else. The parseCsv change applies to every CSV that the package reads, and evorca reproduce still matches all 335 archive scores after it.

Verification

  • pnpm check passes, and pnpm evorca reproduce --benchmark=openrca prints All scores match the archive.
  • With the answer keys present, src/candidates.test.ts finds every component and reason of each record.csv in its scenario's lists. When one name is removed from candidates.json, the test fails and names it.
  • A second run of pnpm --filter @evorca/openrca extract-candidates leaves git status clean.

Telecom's record.csv ends with an empty line, and parseCsv failed on it
with "Invalid Record Length". The evaluator reads its CSV files with
pandas, which skips empty lines, so parseCsv now does the same.
RCA-Agent's prompt for each OpenRCA system lists the root cause
components and reasons that an answer can name. Whether an agent sees
these lists changes how hard the benchmark is, so evorca needs them as
data to give an agent the same view as RCA-Agent.

openRcaCandidates holds the lists per scenario. The maintainer script
scripts/extract-candidates.ts writes src/candidates.json from the
basic_prompt_<System>.py files at the pinned OpenRCA commit and trims
the trailing spaces that some prompt entries have. A test checks that
the lists include every component and reason in each scenario's
record.csv when the answer keys are on disk.
@kottinov
kottinov merged commit 4563710 into main Nov 17, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant