Repository navigation
feat(cli): let evaluate run an agent that calls Claude through a proxy - #23
Merged
Merged
Conversation
An agent with needsModel reaches only a model proxy that evorca evaluate runs on the host. A relay container forwards to it, and each task gets an internal network without a gateway address. The proxy holds the real key, gives each task its own token, forwards only POST /v1/messages for the run's model and effort, and records each task's tokens and cost. It answers 403 evorca_budget_exhausted once the task or the run has spent its budget, and tasks that the run's budget keeps from starting get the outcome budget-exhausted. The new flags are --model, --effort, --budget-usd, and --task-budget-usd. pnpm evorca reads ANTHROPIC_API_KEY from .env if the file exists.
The proxy takes a task's requests one at a time and at most 4 in flight, so parallel requests cannot pass a budget. It forwards the body it checked, serialized again, answers a 15-minute upstream timeout with a 504 that the SDK does not retry, and stops the run if it cannot write a task's requests.jsonl. The run budget counts every scenario of the run, even on a resume of only some of them. A running task that the run's budget cuts short gets the outcome budget-exhausted, because which tasks it cuts depends on timing. A resume writes run.json before it deletes the folders of abandoned tasks. ctrl+c starts no more tasks, sweeps the run's containers until the tasks settle, and waits for the model requests in flight so the run records their cost. The preflight check now also requires no default route, no DNS, and no route to 1.1.1.1. Each task network must have no gateway address, and the relay must still run after each task. EVORCA_UPSTREAM_URL accepts http only on a loopback address.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The ReAct baseline calls Claude from inside a task's container. That container must not hold the real API key, must not reach the internet with the telemetry, and must not spend without limit across hundreds of tasks. This PR gives an agent with
needsModel: trueone endpoint, a model proxy that holds the key, enforces the run's contract, and records the cost of each task. It stacks on #22. Review the last two commits.What changed
Container contract for
needsModelagents./task.json,/data, and/outstay as they are. The container gets these variables:ANTHROPIC_BASE_URLhttp://evorca-proxy:8080, the only endpoint it can reachANTHROPIC_API_KEYevorca-<random>, a token valid only while the task runsEVORCA_MODELclaude-sonnet-5by defaultEVORCA_EFFORThighby defaultEVORCA_TASK_BUDGET_USD2by defaultEVORCA_SEEDNetwork. The proxy runs inside
evorca evaluateon the host's 127.0.0.1, on a random port. One relay container per run (node:26-alpinepinned by digest, read-only, no capabilities, 64 MiB) forwards TCP from its port 8080 to the proxy throughhost.docker.internal. Each task gets its own--internalnetwork withcom.docker.network.bridge.inhibit_ipv4=true, which holds only the task's container and the relay, under the aliasevorca-proxy.I chose this after a prototype on Docker Desktop 28.2.2.
--internalalone, a container still reached the VM's rpcbind on port 111 and a service on 61883 through the bridge's gateway address.inhibit_ipv4removes the gateway address, and a full scan then found only the relay's 8080.The command checks the design on every run.
inhibit_ipv4.api.anthropic.com, and fail to reach1.1.1.1:443.Proxy. It forwards only
POST /v1/messageswithout streaming, for the run's model and effort. It swaps the task's token for the real key and sends the body it checked, serialized again, so duplicate keys cannot slip past a rule. It denies anything else with the API's error shape.evorca_budget_exhaustedonce the task or the run has spent its budget.anthropic-beta, server tools,mcp_servers, URL sources, or non-globalinference_geo. Server tools, MCP, and URL sources would let the API reach the internet for the agent.It reads the token from
x-api-keyor fromAuthorization: Bearer, which is how@anthropic-ai/sdk1.9 sends an API key. Upstream gets onlycontent-type,content-length,x-api-key, andanthropic-version. Upstream errors such as 429 and 529 pass through withretry-after, so the SDK retries them. The proxy's own errors setx-should-retry, so the SDK does not retry a 403. An upstream answer slower than 15 minutes gets 504timeout_errorwithout a retry.Budgets and cost.
usageis priced with a table inmodels.ts:anthropic-2026-11-17, with its source URL and date. Five-minute and one-hour cache writes are priced apart.budget-exhaustedand the empty prediction. So do running tasks that the run budget cuts short, because which tasks it cuts depends on timing. A resume runs both kinds again.requests.jsonlgets one line per request, with no prompt, response, or key.run.jsongainsmodel,budgetUsd, per-taskusage,abandoned, and a run total. Each progress line shows the task's cost and the run's cost so far.Flags.
--model,--effort,--budget-usd(default 50), and--task-budget-usd(default 2) apply only to a model agent and are refused for others. A resume keeps the model settings and refuses changes, naming the field, such asmodel.effort: the run has "high", not "low".--jobsand--budget-usdcan change. Without--jobs, a resume keeps the run's ownjobs.Stopping. ctrl+c starts no more tasks. It sweeps the run's containers every second until running tasks settle, so a container that was starting at that moment cannot outlive the stop. It waits for the model requests in flight, so the run records their cost, then exits 130. A second ctrl+c stops at once. A resume writes
run.jsonbefore it deletes the folders of abandoned tasks, so their spend cannot be lost.Key handling.
main.tsreadsANTHROPIC_API_KEYand deletes it fromprocess.envbefore evorca starts any Docker command, container, or evaluator. The rootevorcascript passes--env-file-if-exists=.env. A model agent without a key fails before any task.EVORCA_UPSTREAM_URLswaps the API for a mock. It must be https, or http only on a loopback address.Scope
In scope are the runner side, the proxy, the relay, the registry fields, the flags,
run.json, the docs, and the tests.agents/react/and its registry entry are out of scope, along with the shell and streaming. A lock against two sessions on one run is not included.Blast Radius
needsModelrun as before, with--network none.randomon Telecom withEVORCA_SEED=7still givespredictions.csvsha256636a53f8…at--jobs=1and--jobs=4, and itsrun.jsonhasmodel,budgetUsd, andusageset tonull.@anthropic-ai/sdkis a dev dependency of@evorca/cli, for the proxy tests only..envfile, Node prints.env not found. Continuing without it.on stderr for everypnpm evorcacommand.Verification
pnpm checkpasses, with 113 CLI tests. The new tests cover:budget-exhaustedre-runs;An independent review found 13 issues, and this PR fixes 12 of them. The 13th is two sessions on one run, which is out of scope.
The end-to-end runs used the real CLI and Docker, a mock upstream, and a sentinel fake key. The throwaway agents are not committed.
stream, another model, andweb_fetch.example.com,api.anthropic.com,host.docker.internal, andgateway.docker.internal.ENETUNREACHfor1.1.1.1:443and:53, example.com, api.anthropic.com, the host on 127.0.0.1 and 0.0.0.0 ports, the VM, anddocker0..1) and the task (.2).evorca-…, 39 characters, not ansk-antkey.--jobs=3. Three tasks ran at once, at $1.00 each.run.jsonshows the per-task and total usage, with $3.00 over 6 requests.evorca_budget_exhausted. The SDK raisedPermissionDeniedErrorwithout a retry, and the agent wrote its answer.--budget-usd=1.5, the tasks that had not started gotbudget-exhausted, and no container ran for them.--effort=low --task-budget-usd=5was refused.--budget-usd=2.5. The budget cut telecom row 0 after $0.50, and it gotbudget-exhausted.--scenario=telecom --budget-usd=4counted bank's $2. Its first line showedrun $3.5000.run.jsontotals bank $2 and telecom $3, with the cut spend inabandoned.requests.jsonland nothing left behind..envor key is on this machine.