Skip to content

feat(cli): run evaluate tasks in parallel and resume stopped runs - #22

Merged
kottinov merged 1 commit into
mainfrom
feat/cli-evaluate-resume
Nov 18, 2026
Merged

kottinov merged 1 commit into
mainfrom
feat/cli-evaluate-resume

Conversation

@kottinov

Copy link
Copy Markdown
Collaborator

Why

A model agent spends minutes on each OpenRCA task. One container at a time makes a full pass take days, and a crash or a ctrl+c throws the run away. This PR lets evorca evaluate run tasks in parallel, run a few rows as a pilot, and resume a stopped run without redoing finished tasks. The next PR in the stack builds the model proxy on top of it.

What changed

  • --jobs=N runs N task containers at once. N workers share one queue. A task's prediction does not depend on the order, so one seed gives byte-identical predictions at any --jobs.
  • --rows=0,7,21 runs only those rows of each scenario. predictions.csv still has a row for every task, with {} for the others, because the evaluator matches predictions by position. The accuracy and scores.json cover only the selected rows.
  • --resume=<run ID> continues a run. It skips each task that has an outcome. It deletes the folders of the other tasks and runs them again. It first removes containers that a killed run left. It refuses a changed agent folder, agent entry, EVORCA_SEED, or --rows, and names each difference.
  • run.json is written before the first task and after each task, through a staging file and a rename, so it is always whole. It gains rows, jobs, resumes, and endedAt, which stays null while a task has no outcome. Saves are serialized, and tasks are kept in row order.
  • Each entry of the agent table gains resources (memory, /tmp size, CPUs, processes) and a default jobs. Each container carries the label evorca.run=<run ID>.
  • ctrl+c and SIGTERM stop new tasks, remove the run's containers, print the resume command, and exit with 130 or 143. The docker run clients start detached, so the signal reaches only evorca.
  • A warning prints when jobs times the agent's memory is more than Docker's memory.

Scope

In scope is the evaluate runner. Model access is the next PR. The shell, fetch, reproduce, and the agents are out of scope.

Blast Radius

  • Only evorca evaluate changes.
  • random keeps its limits from main: 1 GiB of memory, a 64 MiB /tmp, one CPU, 128 processes, and one task at a time. The limits now come from its table entry, not from constants.
  • random's predictions do not change. Telecom with EVORCA_SEED=7 gives the same predictions.csv.
  • run.json gains fields. Only resume reads it.
  • Known gap, fixed in the next PR: a container that starts at the moment of ctrl+c can outlive the stop. A resume removes it.

Verification

  • pnpm check passes.
  • random on Telecom with EVORCA_SEED=7 gives predictions.csv sha256 636a53f81623cdaff82963ee8d4944be1866df886d3dcf9705bfa2ea9d4dfa16 with --jobs=1 (16.5 s) and with --jobs=4 (9.1 s).
  • ctrl+c after about 4 s exits with 130. run.json holds 17 tasks and endedAt: null, and no container is left.
  • Resuming that run with EVORCA_SEED=8 is refused with seed: the run has 7, not 8.
  • Resuming it with --jobs=4 runs the other 34 tasks. The predictions are byte-identical to an uninterrupted run. run.json lists 51 tasks in row order and one entry in resumes.
  • --rows=0,7,21 runs 3 tasks. --rows=0,99 prints telecom has no row 99. Its tasks are rows 0 to 50.
  • --jobs=0, --jobs=two, --rows=0,,7, --rows=-1, and --resume=../x exit with 1 and a message, covered by main.test.ts.

evorca evaluate takes --jobs, --rows, and --resume=<run ID>. It writes
run.json before the first task and after each task, by renaming a staging
file into place, so a stopped run keeps every finished task. A resumed run
skips those tasks, removes containers that a killed run left, and refuses
other agent, seed, or rows settings.

Each agent's entry sets its container's memory, /tmp, CPUs, processes, and
default jobs. The random agent keeps 1 GiB, 64 MiB, one CPU, 128
processes, and one task at a time.
@kottinov
kottinov merged commit fa2ad34 into main Nov 18, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant