Repository navigation
feat(openrca): export openRcaCandidates from RCA-Agent's prompts - #17
Merged
Merged
Conversation
Telecom's record.csv ends with an empty line, and parseCsv failed on it with "Invalid Record Length". The evaluator reads its CSV files with pandas, which skips empty lines, so parseCsv now does the same.
RCA-Agent's prompt for each OpenRCA system lists the root cause components and reasons that an answer can name. Whether an agent sees these lists changes how hard the benchmark is, so evorca needs them as data to give an agent the same view as RCA-Agent. openRcaCandidates holds the lists per scenario. The maintainer script scripts/extract-candidates.ts writes src/candidates.json from the basic_prompt_<System>.py files at the pinned OpenRCA commit and trims the trailing spaces that some prompt entries have. A test checks that the lists include every component and reason in each scenario's record.csv when the answer keys are on disk.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
evorca needs a first runnable OpenRCA baseline, and whether an agent sees the candidate root causes is a fairness choice that the thesis must state. RCA-Agent's prompts always list these candidates, so evorca needs them as data before it can give an agent the same view.
What changed
openRcaCandidatesin@evorca/openrcamaps each scenario to thecomponentsandreasonsthat an answer can name.scripts/extract-candidates.tswritessrc/candidates.jsonfrombasic_prompt_<System>.pyat the pinned OpenRCA commit. It trims the trailing spaces that some prompt entries have, because the evaluator compares names exactly.parseCsvskips empty lines, as the evaluator's pandas does, because Telecom'srecord.csvends with one.Scope
/task.jsonfor agents that see candidates.record.csvfromdata/benchmarks/openrca/, so it runs only where the answer keys are on disk. CI skips it.Blast Radius
The new export and JSON file touch nothing else. The
parseCsvchange applies to every CSV that the package reads, andevorca reproducestill matches all 335 archive scores after it.Verification
pnpm checkpasses, andpnpm evorca reproduce --benchmark=openrcaprintsAll scores match the archive.src/candidates.test.tsfinds every component and reason of eachrecord.csvin its scenario's lists. When one name is removed fromcandidates.json, the test fails and names it.pnpm --filter @evorca/openrca extract-candidatesleavesgit statusclean.