Repository navigation
feat: run a relay between the harness and the interception server - #3074
mikasenghaas wants to merge 14 commits into
Conversation
Add an in-box relay (`interception/relay`) that a harness in a remote box reaches the interception server through, at 127.0.0.1, instead of the tunnel URL (`agent.relay`, off by default). The relay retries a model call for up to 5 minutes while the tunnel cannot be reached, and marks each retry so the interception server joins it to the call it already serves. When a tunnel is lost for good, its server still takes no new rollouts, but no longer fails the ones that reach it through a relay: it opens a new tunnel, allows its host in each box's egress policy and reroutes each relay. Rollouts with tool servers or deployment services still fail, since those reach the old URL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Wiz Scan Summary
To detect these findings earlier in the dev lifecycle, try the Wiz Code extension for VS Code, JetBrains, or Visual Studio. |
ApprovabilityVerdict: Needs Human Approval Macroscope's review found this PR requires human approval — The relay lets remote harnesses survive tunnel replacement without restarting model calls. Unresolved concerns about competing egress-policy updates and retained old tunnel resources need resolution because they can disrupt rollouts, overwrite scoring policy, or consume tunnel-service resources. Needs human approval because:
No code changes detected at Adjust the Minimum Blocking Severity for this repo — including turning it Off — in Settings. You can add or adjust custom eligibility rules. Learn more. |
Drop `agent.relay`: a harness in a remote box always reaches the interception server through its relay, and a relay that does not start leaves it on the tunnel URL with a warning. - Give the relay only the model secret, which the harness already holds: a harness that reads the relay config gains no capability. - Run one relay per rollout (own dir, ephemeral port) and stop it when the rollout closes, so rollouts that share a box never share one. - Make a rollout movable before the server registers it; a tunnel lost while the relay starts waits for it. - Replace the tunnel again when it is lost during a replacement. - Retry a non-streaming body that breaks before any data, and a body that ends before its Content-Length. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The relay added about 4 s to setup (GSM8K, p50 6.4 s -> 10.6 s): eight sandbox calls one after another. Launch it while the harness sets up, with a placeholder upstream, and configure it once the interception slot is acquired: one write and one exec on the critical path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A `Relay` object owns the relay's lifecycle (launch alongside the harness setup, connect, reroute, stop), so the rollout keeps one field. The retry window is a constant; tests patch the script source. The relay verified the tunnel's certificate against the system CA store, which a task image may lack: terminal-bench-2's financial-document-processor image has no /etc/ssl/certs, so both its relayed rollouts failed on their first model call while the harness's own HTTP client (certifi) worked on main. Trust certifi's roots as well. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The relay is part of the machinery like the interception server: a relay that does not start fails the rollout with a SandboxError, which the rollout retry rules cover, instead of a silent fallback to the tunnel URL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add `RelayError` (an `InterceptionError`, next to `TunnelError`). A relay that does not start raises it. A relay that gives up on a request appends the reason to a failures file in its directory; when the harness then dies, the rollout checks that file and whether the relay still runs, and records `RelayError` with the relay's reason instead of the harness's `HarnessError`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Scripts prepared in parallel each ran the uv bootstrap under a lock of their own script: in an image without uv, two `pip install --user uv` runs or downloads to one path (or apt's lock) raced, and one failed. The relay prepares its script during the harness setup, so CI hit it: a random network-policy cell failed with zero branches. The first preparation per user now holds a bootstrap lock; once uv is there, scripts prepare in parallel. The integration grid's one-branch assertions show the trace's errors. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Mark a resend as a retry only after an attempt that may have reached the server; a connect failure or an edge page did not, so a body a later turn repeats is never answered from the replay cache. - Never frame SSE with a fixed length, and close an edge error page unread instead of draining it. - Count a new outage from its start once a response stayed up. - Record "ok" after a request goes through again, so a stale relay failure never names a later, unrelated harness death. - Stop the relay process also after a failed or cancelled launch. - Leave a closing rollout's egress alone on a reroute (shared scoring sets its own), and strip the trailing slash `host_url` may return. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The relay starts alongside the harness setup. Its install puts uv in a box that has none, and the setup of some harnesses (prime-agent, hermes-agent) does the same in a shell of its own. The two installs could overlap, and one then ran the other's half-written binary: "uv: Permission denied" from the relay start, seen once in CI. Co-Authored-By: Claude <noreply@anthropic.com>
# Conflicts: # verifiers/v1/interception/server.py
The relay took every 404, 502, 503 or 504 that was not JSON or SSE for the error page of a tunnel edge and retried it for five minutes. The interception server answers an unknown path with a plain-text 404, so a harness that asked for one waited that long. The server now marks each of its responses with a header, and only a response without it counts as the edge's. Co-Authored-By: Claude <noreply@anthropic.com> (cherry picked from commit a1b88d8)
| if not self.scanner.data: | ||
| # No body data went out: the retry can still answer. | ||
| return False | ||
| self.reset() |
There was a problem hiding this comment.
🟡 Medium relay/server.py:516
When an upstream body breaks or ends before Content-Length after body data has been sent, the relay records the request as successful: both branches call reset() and return True, so relay() calls record_success(). This omits the transport failure from the failures file (and can append ok after a prior failure), causing Rollout.step to report HarnessError instead of RelayError. Record these truncated-response failures before closing, and ensure they are not treated as successful completion.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/interception/relay/server.py around line 516:
When an upstream body breaks or ends before `Content-Length` after body data has been sent, the relay records the request as successful: both branches call `reset()` and return `True`, so `relay()` calls `record_success()`. This omits the transport failure from the failures file (and can append `ok` after a prior failure), causing `Rollout.step` to report `HarnessError` instead of `RelayError`. Record these truncated-response failures before closing, and ensure they are not treated as successful completion.
| upstream.attach(connection.sock) | ||
| # Only the connect is bounded: a model call may wait long for its first | ||
| # byte. | ||
| connection.sock.settimeout(None) |
There was a problem hiding this comment.
🟠 High relay/server.py:419
A retry that connects but receives no response headers can hang indefinitely instead of failing after RETRY_SECONDS; getresponse() blocks in open() and never reaches unreachable(). Setting the socket to blocking mode here removes the only timeout, while keepalive() continues sending comments and prevents the client’s idle timeout. Preserve an enforceable deadline or watchdog for stalled retry attempts.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/interception/relay/server.py around line 419:
A retry that connects but receives no response headers can hang indefinitely instead of failing after `RETRY_SECONDS`; `getresponse()` blocks in `open()` and never reaches `unreachable()`. Setting the socket to blocking mode here removes the only timeout, while `keepalive()` continues sending comments and prevents the client’s idle timeout. Preserve an enforceable deadline or watchdog for stalled retry attempts.
| logger.warning( | ||
| "interception tunnel replaced: url=%s -> %s", self.base_url, base_url | ||
| ) | ||
| self.base_url = base_url |
There was a problem hiding this comment.
Old tunnels stay on the stack
Medium Severity
Each replacement calls stack.enter_async_context(self.tunnel.expose(self.port)) and never exits the previous expose. The gone tunnel's frpc is therefore not sync_stopped until the whole interception server stops. A loss during replacement (_lost_again) stacks another live tunnel on the same host port, so a long rollout that flaps can leak processes, logins against a deleted registration, and rate-limit budget for the rest of that server's life.
Reviewed by Cursor Bugbot for commit 167b4b6. Configure here.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 69ecaad. Configure here.
| """Wait until `connect` finished; raise if the relay did not start.""" | ||
| await self._connected.wait() | ||
| if not self.upstream: | ||
| raise TunnelError("the relay did not start") |
There was a problem hiding this comment.
Relay start misattributed in the trace
Medium Severity
A relay that does not start or reload is recorded as SandboxError or TunnelError, not RelayError. boundary(RelayError) around connect lets those RolloutErrors through, and wait_connected raises TunnelError("the relay did not start"), so traces and stop conditions name the sandbox or tunnel instead of the relay.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit 69ecaad. Configure here.


Summary
127.0.0.1, instead of at the tunnel URL. The relay is part of the machinery like the interception server: always on for remote runtimes, with no option to turn it off.RelayError), not hidden behind aHarnessError.Why a process of our own in the box
Today a harness is launched with the interception server's URL and secret in its environment or argv. A third-party harness (claude-code, codex, opencode) cannot be told anything after launch. So a change on the host side breaks the rollout:
TunnelError, however long it ran.404 Tunnel Not Foundor 5xx page. Each harness handles that in its own way, and some exit.A relay gives the harness one address that never changes, and gives us a control point behind it. It follows a pattern verifiers already uses: tool servers and the ACP runner are verifiers processes in the box too (
prepare_uv_script+run_background).What the relay does
verifiers/v1/interception/relay/server.pyruns in the box;verifiers/v1/interception/relay/__init__.py(Relay) controls it from the host.x-stainless-retry-count), so the server joins it to the call it already serves: no second turn.uvin a box that has none, and the setup scripts of prime-agent and hermes-agent do the same in a shell of their own, so the two must not overlap (seen once in CI asuv: Permission deniedfrom the relay start).Example 1: the tunnel host changes (in this PR)
When a tunnel is lost for good, its server still takes no new rollouts, and the pools retire it once its rollouts leave. The rollouts on it no longer fail: the server opens a new tunnel, allows the new host in each box's egress policy and reroutes each relay. A model call in flight goes through the new tunnel. A rollout with tool servers, shared tool servers or deployment services still fails, because those reach the server at the tunnel URL.
Example 2: the secrets change (prepared)
Relay.reroute(upstream, tokens)moves the relay to a new URL and maps the harness's token to a new secret. Memory snapshots use this: the new host process registers the rollout under new secrets, and the resumed harness keeps the token it holds.Errors
RelayError(anInterceptionError, next toTunnelError) names the relay:RelayError.RelayErrorwith that reason (or "the relay exited") instead of the harness'sHarnessError. The interception server's own model-call failure still wins when there is one.Load test
uv run evalfrom prime-rl main, withdeps/verifiersat this branch's base (bce58f3e5) and at this branch (31d658b39), both arms side by side, bash harness on Prime, 64 rollouts in flight per run.RelayErrorsSandboxErroron exec), and the reward differences are within the noise of one rollout per task.financial-document-processor's image has no/etc/ssl/certs, so the relay's TLS trusted no certificate, and both of its rollouts failed on their first model call. The relay now trusts certifi's roots; 4 of 4 rollouts of that task then went through it.Limitations
RelayError); nothing restarts it.Verification
tests/integration/test_relay.pyon Prime: a model call is held while the server's tunnel is lost (registration deleted, frpc killed); the rollout finishes with no errors, one branch, and the held call reaches the model once. A relay killed during a held call is recorded asRelayError. Both pass.tests/unit/interception/test_relay.py: forwarding, the secret swap, streaming, resend on a reroute, edge-page retries, the interception's errors passed through, a body cut short, the retry window, the recorded failure reason.tests/unit/interception/test_tunnel.py: movable rollouts follow a new tunnel and the others fail; a second loss during a replacement; a failed replacement.pytest tests/unit: 241 passed, 19 skipped (no Docker, noPRIME_API_KEY, noharbor).ruffclean.prepare_uv_scriptnow bootstraps uv once per box at a time.🤖 Generated with Claude Code
🤖 Generated with Claude Code
Note
High Risk
Touches core model-call path, tunnel failover, and rollout networking; incorrect relay/retry/reroute logic could duplicate turns, mask provider errors, or leave rollouts stuck on dead tunnels.
Overview
Remote harness rollouts no longer talk to the interception server at the Prime tunnel URL. A per-rollout loopback relay runs in the sandbox at
127.0.0.1, forwards model HTTP/SSE to the current upstream, swaps bearer/API-key tokens, and retries transport failures and tunnel-edge error pages for up to five minutes with retry markers so the interception server can join duplicates to one in-flight call.Rollout starts the relay alongside harness setup, points the harness at the relay base URL, registers
RolloutSession.reroutewhen the rollout is “movable” (relay-only path: no tool servers, shared tools, or deployment services), and surfacesRelayErroron the trace when the relay fails instead of blaming the harness.On a permanently lost tunnel, the interception server still retires and refuses new sessions, but opens a replacement tunnel and calls each movable session’s
rerouteto repoint its relay; rollouts that still hit the tunnel URL directly keep failing withTunnelError. Responses from the interception server now carryx-vf-interceptionso the relay can distinguish real server answers from edge HTML.Also adds architecture docs, broad unit/integration coverage, slightly richer branch assertions in integration helpers, and serializes uv bootstrap in
prepare_uv_scriptso relay and harness setup do not race installinguv.Reviewed by Cursor Bugbot for commit 69ecaad. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Add relay between remote harness and interception server, with tunnel rerouting
Rollout._reroute; non-reroutable sessions get TunnelError and are released (server.py)RelayError(anInterceptionErrorsubtype) exported fromverifiers.v1, and attributes step failures to the relay when it records a failure or exit (errors.py)Runtime.prepare_uv_scriptto mark a user's uv bootstrap ready after success, so later preparations skip the shared lock (runtime.py)Rolloutharness setup in rollout.pyMacroscope summarized 69ecaad.