Skip to content

feat(cli): run the react baseline through the model proxy - #24

Merged
kottinov merged 9 commits into
mainfrom
feat/react-baseline
Nov 18, 2026
Merged

kottinov merged 9 commits into
mainfrom
feat/react-baseline

Conversation

@kottinov

Copy link
Copy Markdown
Collaborator

Why

The thesis needs a real ReAct baseline score with Claude Sonnet 5. #21 built the agent, and #22 and #23 built the runner and the model proxy, in parallel. This PR joins them, so pnpm evorca evaluate --benchmark=openrca --scenario=telecom --agent=react runs the real agent image through the real proxy. It merges feat/agent-react and fixes the seams between the three PRs.

What changed

  • agents.react registers the agent with candidates, telemetry, and the model. It gets 3072 MiB, a 1024 MiB /tmp, two CPUs, 256 processes, an hour per task, and two tasks at a time.
  • Every container gets EVORCA_TASK_TIMEOUT_SECONDS. feat(agent-react): add the ReAct baseline agent #21's agent wraps up at 80% of it, but feat(cli): let evaluate run an agent that calls Claude through a proxy #23 never passed it, so the soft deadline never fired.
  • Only the proxy ends a model request. It answers 504 after 15 minutes and logs it, and the agent logs it and stops. The agent now waits 16 minutes for every response and asks for at most 32,000 output tokens. Before, it waited 10 minutes at 16,000 tokens, so the SDK could drop and retry a request that the proxy still served. At 64,000 tokens it waited 30 minutes for a request that the proxy cut at 15.

Scope

Tradeoffs

The proxy could wait 30 minutes and the agent keep 64,000 tokens. That doubles the worst-case request, which then cannot fit in the 12 minutes after the soft deadline. At most 32,000 tokens cost $0.32, and the SDK's planning rate of 128,000 tokens an hour puts them at 15 minutes.

Blast Radius

random gets one more variable, which it ignores. The agent changes only its output limit and its client timeout. The proxy and the runner keep their behavior.

Verification

  • pnpm check passes. random with EVORCA_SEED=7 still gives predictions.csv sha256 636a53f8….
  • Through the real CLI, image, proxy, and relay against a scripted mock API, on synthetic telemetry: 3 rows with --jobs=2 ran bash and DuckDB, then submitted. A $0.25 task budget ended in budget-exhausted after one 403. 50 tool calls wrapped up on steps. A 90 s timeout, as an uncommitted local edit, wrapped up on time at 74 s and submitted at 79 s.
  • From the agent's bash tool on a task network, only the relay's port 8080 is open. DNS and 1.1.1.1 fail, and the token is not an sk-ant key.
  • The real pilot, --rows=0,20,40 --budget-usd=6 on the real Telecom telemetry, answered 3 of 3 by submit_answer for $1.19 over 78 requests, with cache reads from turn 2. It scored 1 of 3. It ran at the commit before a comment and README rewording.

Parse /task.json, define the strict submit_answer tool, check an answer against the fields that the task asks for, and write it as OpenRCA prediction JSON.
One bash process runs the model's commands in turn. A command past the timeout ends the session's process group, and long output keeps its start and its end.
extract-background makes background.json from RCA-Agent's prompts at the pinned OpenRCA commit. buildPrompt adds the environment and the answer format, and keeps the system prompt and the tools the same for every task of a scenario.
investigate runs the model until it submits an answer. It wraps up at the step cap, the soft budget, or the soft deadline, and ends on a refusal or an API error with the best answer so far. runTask writes prediction.json, trajectory.jsonl, and summary.json.
The image adds Python 3.13 with pandas, pytz, and DuckDB from hashed pins. The smoke test runs one task in the image with the runner's restrictions against a mock Messages API on an internal network.
The react entry sees candidates, mounts the telemetry, and calls the model through the proxy, with 3 GiB of memory, a 1 GiB /tmp, two CPUs, 256 processes, an hour per task, and two tasks at a time. Every container now gets EVORCA_TASK_TIMEOUT_SECONDS, which the agent needs to wrap up before the runner kills it.
The proxy answers 504 after 15 minutes without a retry. The agent waited 10 minutes at 16,000 output tokens, so the SDK abandoned and retried a request that the proxy still served, and 30 minutes at 64,000, a limit that the proxy cut. The agent now waits 16 minutes for every response and asks for at most 32,000 tokens, which the SDK's own planning rate puts at 15 minutes.
@kottinov
kottinov changed the base branch from feat/cli-model-proxy to main November 18, 2026 06:07
@kottinov
kottinov merged commit 8d3a346 into main Nov 18, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant