Repository navigation
feat(cli): run evaluate tasks in parallel and resume stopped runs - #22
Merged
Merged
Conversation
evorca evaluate takes --jobs, --rows, and --resume=<run ID>. It writes run.json before the first task and after each task, by renaming a staging file into place, so a stopped run keeps every finished task. A resumed run skips those tasks, removes containers that a killed run left, and refuses other agent, seed, or rows settings. Each agent's entry sets its container's memory, /tmp, CPUs, processes, and default jobs. The random agent keeps 1 GiB, 64 MiB, one CPU, 128 processes, and one task at a time.
This was referenced Nov 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A model agent spends minutes on each OpenRCA task. One container at a time makes a full pass take days, and a crash or a ctrl+c throws the run away. This PR lets
evorca evaluaterun tasks in parallel, run a few rows as a pilot, and resume a stopped run without redoing finished tasks. The next PR in the stack builds the model proxy on top of it.What changed
--jobs=Nruns N task containers at once. N workers share one queue. A task's prediction does not depend on the order, so one seed gives byte-identical predictions at any--jobs.--rows=0,7,21runs only those rows of each scenario.predictions.csvstill has a row for every task, with{}for the others, because the evaluator matches predictions by position. The accuracy andscores.jsoncover only the selected rows.--resume=<run ID>continues a run. It skips each task that has an outcome. It deletes the folders of the other tasks and runs them again. It first removes containers that a killed run left. It refuses a changed agent folder, agent entry,EVORCA_SEED, or--rows, and names each difference.run.jsonis written before the first task and after each task, through a staging file and a rename, so it is always whole. It gainsrows,jobs,resumes, andendedAt, which staysnullwhile a task has no outcome. Saves are serialized, and tasks are kept in row order.resources(memory,/tmpsize, CPUs, processes) and a defaultjobs. Each container carries the labelevorca.run=<run ID>.docker runclients start detached, so the signal reaches only evorca.jobstimes the agent's memory is more than Docker's memory.Scope
In scope is the evaluate runner. Model access is the next PR. The shell,
fetch,reproduce, and the agents are out of scope.Blast Radius
evorca evaluatechanges.randomkeeps its limits frommain: 1 GiB of memory, a 64 MiB/tmp, one CPU, 128 processes, and one task at a time. The limits now come from its table entry, not from constants.random's predictions do not change. Telecom withEVORCA_SEED=7gives the samepredictions.csv.run.jsongains fields. Only resume reads it.Verification
pnpm checkpasses.randomon Telecom withEVORCA_SEED=7givespredictions.csvsha256636a53f81623cdaff82963ee8d4944be1866df886d3dcf9705bfa2ea9d4dfa16with--jobs=1(16.5 s) and with--jobs=4(9.1 s).run.jsonholds 17 tasks andendedAt: null, and no container is left.EVORCA_SEED=8is refused withseed: the run has 7, not 8.--jobs=4runs the other 34 tasks. The predictions are byte-identical to an uninterrupted run.run.jsonlists 51 tasks in row order and one entry inresumes.--rows=0,7,21runs 3 tasks.--rows=0,99printstelecom has no row 99. Its tasks are rows 0 to 50.--jobs=0,--jobs=two,--rows=0,,7,--rows=-1, and--resume=../xexit with 1 and a message, covered bymain.test.ts.