Skip to content

feat(cli): let evaluate run an agent that calls Claude through a proxy - #23

Merged
kottinov merged 2 commits into
mainfrom
feat/cli-model-proxy
Nov 18, 2026
Merged

kottinov merged 2 commits into
mainfrom
feat/cli-model-proxy

Conversation

@kottinov

Copy link
Copy Markdown
Collaborator

Why

The ReAct baseline calls Claude from inside a task's container. That container must not hold the real API key, must not reach the internet with the telemetry, and must not spend without limit across hundreds of tasks. This PR gives an agent with needsModel: true one endpoint, a model proxy that holds the key, enforces the run's contract, and records the cost of each task. It stacks on #22. Review the last two commits.

What changed

Container contract for needsModel agents. /task.json, /data, and /out stay as they are. The container gets these variables:

Variable Value
ANTHROPIC_BASE_URL http://evorca-proxy:8080, the only endpoint it can reach
ANTHROPIC_API_KEY evorca-<random>, a token valid only while the task runs
EVORCA_MODEL claude-sonnet-5 by default
EVORCA_EFFORT high by default
EVORCA_TASK_BUDGET_USD 2 by default
EVORCA_SEED unchanged

Network. The proxy runs inside evorca evaluate on the host's 127.0.0.1, on a random port. One relay container per run (node:26-alpine pinned by digest, read-only, no capabilities, 64 MiB) forwards TCP from its port 8080 to the proxy through host.docker.internal. Each task gets its own --internal network with com.docker.network.bridge.inhibit_ipv4=true, which holds only the task's container and the relay, under the alias evorca-proxy.

I chose this after a prototype on Docker Desktop 28.2.2.

  • With --internal alone, a container still reached the VM's rpcbind on port 111 and a service on 61883 through the bridge's gateway address. inhibit_ipv4 removes the gateway address, and a full scan then found only the relay's 8080.
  • A proxy inside a container would need the key in that container. Keeping the proxy on the host keeps the key in the evorca process.
  • One network per task also keeps tasks from reaching each other.

The command checks the design on every run.

  • Each task network must have no IPAM gateway, which is how Docker records that it honored inhibit_ipv4.
  • Before the first task, a check container on a task network must get the proxy's 401 through the relay. It must also find no default route, fail to resolve api.anthropic.com, and fail to reach 1.1.1.1:443.
  • After each task, the relay must still run. Otherwise the run stops, and a resume runs the task again rather than scoring it as an agent failure.

Proxy. It forwards only POST /v1/messages without streaming, for the run's model and effort. It swaps the task's token for the real key and sends the body it checked, serialized again, so duplicate keys cannot slip past a rule. It denies anything else with the API's error shape.

  • 401 for an unknown or closed token.
  • 404 for another route.
  • 403 evorca_budget_exhausted once the task or the run has spent its budget.
  • 413 for a body over 32 MiB.
  • 429 for a fifth request of a task in flight.
  • 400 for streaming, another model or effort, anthropic-beta, server tools, mcp_servers, URL sources, or non-global inference_geo. Server tools, MCP, and URL sources would let the API reach the internet for the agent.

It reads the token from x-api-key or from Authorization: Bearer, which is how @anthropic-ai/sdk 1.9 sends an API key. Upstream gets only content-type, content-length, x-api-key, and anthropic-version. Upstream errors such as 429 and 529 pass through with retry-after, so the SDK retries them. The proxy's own errors set x-should-retry, so the SDK does not retry a 403. An upstream answer slower than 15 minutes gets 504 timeout_error without a retry.

Budgets and cost.

  • Each response's usage is priced with a table in models.ts: anthropic-2026-11-17, with its source URL and date. Five-minute and one-hour cache writes are priced apart.
  • The proxy takes a task's requests one at a time, so each budget check counts every earlier response. A task can pass its cap by one request, and the run by one request per running task.
  • The run budget covers every scenario of the run, even when a resume names only some of them.
  • Tasks that the spent run budget keeps from starting get the outcome budget-exhausted and the empty prediction. So do running tasks that the run budget cuts short, because which tasks it cuts depends on timing. A resume runs both kinds again.
  • Each task's requests.jsonl gets one line per request, with no prompt, response, or key.
  • run.json gains model, budgetUsd, per-task usage, abandoned, and a run total. Each progress line shows the task's cost and the run's cost so far.

Flags. --model, --effort, --budget-usd (default 50), and --task-budget-usd (default 2) apply only to a model agent and are refused for others. A resume keeps the model settings and refuses changes, naming the field, such as model.effort: the run has "high", not "low". --jobs and --budget-usd can change. Without --jobs, a resume keeps the run's own jobs.

Stopping. ctrl+c starts no more tasks. It sweeps the run's containers every second until running tasks settle, so a container that was starting at that moment cannot outlive the stop. It waits for the model requests in flight, so the run records their cost, then exits 130. A second ctrl+c stops at once. A resume writes run.json before it deletes the folders of abandoned tasks, so their spend cannot be lost.

Key handling. main.ts reads ANTHROPIC_API_KEY and deletes it from process.env before evorca starts any Docker command, container, or evaluator. The root evorca script passes --env-file-if-exists=.env. A model agent without a key fails before any task. EVORCA_UPSTREAM_URL swaps the API for a mock. It must be https, or http only on a loopback address.

Scope

In scope are the runner side, the proxy, the relay, the registry fields, the flags, run.json, the docs, and the tests. agents/react/ and its registry entry are out of scope, along with the shell and streaming. A lock against two sessions on one run is not included.

Blast Radius

  • Agents without needsModel run as before, with --network none. random on Telecom with EVORCA_SEED=7 still gives predictions.csv sha256 636a53f8… at --jobs=1 and --jobs=4, and its run.json has model, budgetUsd, and usage set to null.
  • ctrl+c changes for every agent. It now waits for running tasks to settle, then removes containers and networks.
  • @anthropic-ai/sdk is a dev dependency of @evorca/cli, for the proxy tests only.
  • Docker Desktop is required for model agents. On Linux, a container cannot reach the host's 127.0.0.1, so the preflight check fails before any task with a clear message.
  • Without a .env file, Node prints .env not found. Continuing without it. on stderr for every pnpm evorca command.

Verification

pnpm check passes, with 113 CLI tests. The new tests cover:

  • docker args, including the network and env and a sentinel key that never appears;
  • relay and network args;
  • cost math and usage parsing;
  • the proxy, through the real SDK against a mock upstream, covering forwards, every denial with zero upstream requests, the task and run 403s without retry, six parallel requests that stop at the task budget, a re-serialized body, a 529 retry, a 502, and close-after-in-flight;
  • resume conflicts and budget-exhausted re-runs;
  • upstream URLs that could send the key in clear.

An independent review found 13 issues, and this PR fixes 12 of them. The 13th is two sessions on one run, which is out of scope.

The end-to-end runs used the real CLI and Docker, a mock upstream, and a sentinel fake key. The throwaway agents are not committed.

  • Probe.
    • It reached the proxy with both token headers (200).
    • It got 401 without a token, and 400 for stream, another model, and web_fetch.
    • DNS failed for example.com, api.anthropic.com, host.docker.internal, and gateway.docker.internal.
    • TCP gave ENETUNREACH for 1.1.1.1:443 and :53, example.com, api.anthropic.com, the host on 127.0.0.1 and 0.0.0.0 ports, the VM, and docker0.
    • It had no default route. A full scan of the relay found only 8080. The task network held only the relay (.1) and the task (.2).
    • Its key was evorca-…, 39 characters, not an sk-ant key.
  • 3 rows with --jobs=3. Three tasks ran at once, at $1.00 each. run.json shows the per-task and total usage, with $3.00 over 6 requests.
  • Task budget. The fifth $0.50 request of a $2 task got 403 evorca_budget_exhausted. The SDK raised PermissionDeniedError without a retry, and the agent wrote its answer.
  • Run budget.
    • With --budget-usd=1.5, the tasks that had not started got budget-exhausted, and no container ran for them.
    • A resume with --effort=low --task-budget-usd=5 was refused.
    • A resume with a larger budget ran them.
  • Run budget across scenarios.
    • Bank and telecom ran under --budget-usd=2.5. The budget cut telecom row 0 after $0.50, and it got budget-exhausted.
    • A resume with --scenario=telecom --budget-usd=4 counted bank's $2. Its first line showed run $3.5000.
    • A last resume ran the cut task again. run.json totals bank $2 and telecom $3, with the cut spend in abandoned.
  • ctrl+c with a request in flight. SIGINT came 1 s into a 3 s request. The command exited 130 two seconds later, with the request's $0.50 in requests.jsonl and nothing left behind.
  • Kill and resume. The run had 6 rows and 2 jobs.
    • It was stopped by SIGKILL, then by SIGINT, then finished.
    • The mock's log shows one container for each task that had an outcome before a stop. Only the in-flight tasks ran again.
    • The run records $11 of the $12 the mock served. The difference is the two requests in flight at the SIGKILL.
  • Key. Upstream got the CLI's key on every request. The sentinel key appears in no run folder.
  • The real-key smoke test was skipped, because no .env or key is on this machine.

An agent with needsModel reaches only a model proxy that evorca evaluate
runs on the host. A relay container forwards to it, and each task gets an
internal network without a gateway address. The proxy holds the real key,
gives each task its own token, forwards only POST /v1/messages for the
run's model and effort, and records each task's tokens and cost. It
answers 403 evorca_budget_exhausted once the task or the run has spent
its budget, and tasks that the run's budget keeps from starting get the
outcome budget-exhausted.

The new flags are --model, --effort, --budget-usd, and --task-budget-usd.
pnpm evorca reads ANTHROPIC_API_KEY from .env if the file exists.
The proxy takes a task's requests one at a time and at most 4 in flight,
so parallel requests cannot pass a budget. It forwards the body it
checked, serialized again, answers a 15-minute upstream timeout with a
504 that the SDK does not retry, and stops the run if it cannot write a
task's requests.jsonl.

The run budget counts every scenario of the run, even on a resume of only
some of them. A running task that the run's budget cuts short gets the
outcome budget-exhausted, because which tasks it cuts depends on timing.
A resume writes run.json before it deletes the folders of abandoned
tasks.

ctrl+c starts no more tasks, sweeps the run's containers until the tasks
settle, and waits for the model requests in flight so the run records
their cost. The preflight check now also requires no default route, no
DNS, and no route to 1.1.1.1. Each task network must have no gateway
address, and the relay must still run after each task.

EVORCA_UPSTREAM_URL accepts http only on a loopback address.
@kottinov
kottinov changed the base branch from feat/cli-evaluate-resume to main November 18, 2026 06:06
@kottinov
kottinov merged commit a7cb035 into main Nov 18, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant