Repository navigation
feat(cli): run the react baseline through the model proxy - #24
Merged
Merged
Conversation
Parse /task.json, define the strict submit_answer tool, check an answer against the fields that the task asks for, and write it as OpenRCA prediction JSON.
One bash process runs the model's commands in turn. A command past the timeout ends the session's process group, and long output keeps its start and its end.
extract-background makes background.json from RCA-Agent's prompts at the pinned OpenRCA commit. buildPrompt adds the environment and the answer format, and keeps the system prompt and the tools the same for every task of a scenario.
investigate runs the model until it submits an answer. It wraps up at the step cap, the soft budget, or the soft deadline, and ends on a refusal or an API error with the best answer so far. runTask writes prediction.json, trajectory.jsonl, and summary.json.
The image adds Python 3.13 with pandas, pytz, and DuckDB from hashed pins. The smoke test runs one task in the image with the runner's restrictions against a mock Messages API on an internal network.
The react entry sees candidates, mounts the telemetry, and calls the model through the proxy, with 3 GiB of memory, a 1 GiB /tmp, two CPUs, 256 processes, an hour per task, and two tasks at a time. Every container now gets EVORCA_TASK_TIMEOUT_SECONDS, which the agent needs to wrap up before the runner kills it.
The proxy answers 504 after 15 minutes without a retry. The agent waited 10 minutes at 16,000 output tokens, so the SDK abandoned and retried a request that the proxy still served, and 30 minutes at 64,000, a limit that the proxy cut. The agent now waits 16 minutes for every response and asks for at most 32,000 tokens, which the SDK's own planning rate puts at 15 minutes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The thesis needs a real ReAct baseline score with Claude Sonnet 5. #21 built the agent, and #22 and #23 built the runner and the model proxy, in parallel. This PR joins them, so
pnpm evorca evaluate --benchmark=openrca --scenario=telecom --agent=reactruns the real agent image through the real proxy. It mergesfeat/agent-reactand fixes the seams between the three PRs.What changed
agents.reactregisters the agent with candidates, telemetry, and the model. It gets 3072 MiB, a 1024 MiB/tmp, two CPUs, 256 processes, an hour per task, and two tasks at a time.EVORCA_TASK_TIMEOUT_SECONDS. feat(agent-react): add the ReAct baseline agent #21's agent wraps up at 80% of it, but feat(cli): let evaluate run an agent that calls Claude through a proxy #23 never passed it, so the soft deadline never fired.Scope
loop.tsandmain.ts. It can move into feat(agent-react): add the ReAct baseline agent #21 instead.timeout.Tradeoffs
The proxy could wait 30 minutes and the agent keep 64,000 tokens. That doubles the worst-case request, which then cannot fit in the 12 minutes after the soft deadline. At most 32,000 tokens cost $0.32, and the SDK's planning rate of 128,000 tokens an hour puts them at 15 minutes.
Blast Radius
randomgets one more variable, which it ignores. The agent changes only its output limit and its client timeout. The proxy and the runner keep their behavior.Verification
pnpm checkpasses.randomwithEVORCA_SEED=7still givespredictions.csvsha256636a53f8….--jobs=2ran bash and DuckDB, then submitted. A $0.25 task budget ended inbudget-exhaustedafter one 403. 50 tool calls wrapped up onsteps. A 90 s timeout, as an uncommitted local edit, wrapped up ontimeat 74 s and submitted at 79 s.1.1.1.1fail, and the token is not ansk-antkey.--rows=0,20,40 --budget-usd=6on the real Telecom telemetry, answered 3 of 3 bysubmit_answerfor $1.19 over 78 requests, with cache reads from turn 2. It scored 1 of 3. It ran at the commit before a comment and README rewording.