Skip to content

feat: run a relay between the harness and the interception server - #3074

Open
mikasenghaas wants to merge 14 commits into
mainfrom
feat/interception-relay
Open

mikasenghaas wants to merge 14 commits into
mainfrom
feat/interception-relay

Conversation

@mikasenghaas

@mikasenghaas mikasenghaas commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

Summary

  • A harness in a remote box now reaches the interception server through a relay that verifiers runs in the same box, at 127.0.0.1, instead of at the tunnel URL. The relay is part of the machinery like the interception server: always on for remote runtimes, with no option to turn it off.
  • Because we own the process next to the harness, the host can change where the harness's model traffic goes (the URL and the secrets) while the harness runs, without its cooperation.
  • First use: a lost tunnel no longer fails the rollouts on it. The server opens a new tunnel and reroutes each relay. Next use: memory snapshots, where a resumed box is served by a new host process with a new URL and new secrets.
  • A relay failure is named in the trace (RelayError), not hidden behind a HarnessError.

Why a process of our own in the box

Today a harness is launched with the interception server's URL and secret in its environment or argv. A third-party harness (claude-code, codex, opencode) cannot be told anything after launch. So a change on the host side breaks the rollout:

  • The tunnel host changes. Since improvement(interception): keep tunnels serving with prime-tunnel #3041, a tunnel whose registration is gone retires its server, and every rollout on it fails with TunnelError, however long it ran.
  • The host process and secrets change. A box resumed from a memory snapshot is served by a new host process: a new URL and new secrets for a harness that still holds the old ones.
  • A short outage. While frpc reconnects, the tunnel edge answers its own 404 Tunnel Not Found or 5xx page. Each harness handles that in its own way, and some exit.

A relay gives the harness one address that never changes, and gives us a control point behind it. It follows a pattern verifiers already uses: tool servers and the ACP runner are verifiers processes in the box too (prepare_uv_script + run_background).

What the relay does

verifiers/v1/interception/relay/server.py runs in the box; verifiers/v1/interception/relay/__init__.py (Relay) controls it from the host.

  • Forward and swap secrets. It forwards each request to its upstream and swaps the harness's bearer token for the upstream's. It holds only the model secret, which the harness holds anyway, so a harness that reads its config gains no capability.
  • Reroute. The host writes a new config and sends SIGHUP. The relay aborts its upstream attempts and resends each request that has sent the harness nothing but status, headers and SSE comments: a model call still waiting for its first event.
  • Retry transport failures for up to 5 minutes: a connect that fails, a request that breaks before its response, the tunnel edge's own error page (a non-JSON 404, 502, 503 or 504), a stream that breaks before any data. It never retries an answer of the interception server. Each resend is marked (x-stainless-retry-count), so the server joins it to the call it already serves: no second turn.
  • One per rollout, under its own directory and port, so rollouts that share a box never share a relay. It launches alongside the harness setup and stops when the rollout closes. The harness setup starts once the relay's program is installed: the install puts uv in a box that has none, and the setup scripts of prime-agent and hermes-agent do the same in a shell of their own, so the two must not overlap (seen once in CI as uv: Permission denied from the relay start).
  • A resend counts as a retry only when an earlier attempt may have reached the server. A connect failure or an edge page did not, so a body that a later turn repeats is never answered from the server's replay cache. It trusts certifi's CA roots as well as the image's, since an image may have no CA store.

Example 1: the tunnel host changes (in this PR)

When a tunnel is lost for good, its server still takes no new rollouts, and the pools retire it once its rollouts leave. The rollouts on it no longer fail: the server opens a new tunnel, allows the new host in each box's egress policy and reroutes each relay. A model call in flight goes through the new tunnel. A rollout with tool servers, shared tool servers or deployment services still fails, because those reach the server at the tunnel URL.

Example 2: the secrets change (prepared)

Relay.reroute(upstream, tokens) moves the relay to a new URL and maps the harness's token to a new secret. Memory snapshots use this: the new host process registers the rollout under new secrets, and the resumed harness keeps the token it holds.

Errors

RelayError (an InterceptionError, next to TunnelError) names the relay:

  • A relay that does not start fails the rollout with RelayError.
  • A relay that gives up on a request records the reason in its directory, and records "ok" once a request goes through again. If the harness then dies while a relay failure is current, the rollout records RelayError with that reason (or "the relay exited") instead of the harness's HarnessError. The interception server's own model-call failure still wins when there is one.

Load test

uv run eval from prime-rl main, with deps/verifiers at this branch's base (bce58f3e5) and at this branch (31d658b39), both arms side by side, bash harness on Prime, 64 rollouts in flight per run.

GSM8K base GSM8K relay terminal-bench-2 base terminal-bench-2 relay
Rollouts ok 256 of 256 256 of 256 88 of 89 88 of 89
Reward 0.984 0.977 0.568 0.602
Error rate 0% 0% 1.1% 1.1%
Timeouts 0 0 17 16
Rollout p50 / p90 20 / 30 s 22 / 32 s 266 / 3630 s 274 / 3633 s
Setup p50 / p90 6.8 / 11.9 s 8.7 / 14.1 s 4.2 / 15.2 s 7.6 / 19.7 s
RelayErrors — 0 — 0
  • Outcomes match: the errors are the same platform failures in both arms (SandboxError on exec), and the reward differences are within the noise of one rollout per task.
  • The relay costs about 2 to 3.5 s of setup at p50. In an image without uv, the relay's script preparation waits for the harness's (only one uv bootstrap at a time).
  • An earlier run found a bug: financial-document-processor's image has no /etc/ssl/certs, so the relay's TLS trusted no certificate, and both of its rollouts failed on their first model call. The relay now trusts certifi's roots; 4 of 4 rollouts of that task then went through it.

Limitations

  • Tool servers, shared tool servers, deployment services and proxied MCP routes still reach the interception server at the tunnel URL; a lost tunnel still fails their rollouts.
  • A relay that dies mid-rollout fails that rollout (RelayError); nothing restarts it.
  • The relay adds no setup requirement: every harness already prepares a uv script in setup (the chat program or the ACP runner), and a box has open network until the agent starts. It costs about 2 to 3.5 s of setup (see Load test).

Verification

  • tests/integration/test_relay.py on Prime: a model call is held while the server's tunnel is lost (registration deleted, frpc killed); the rollout finishes with no errors, one branch, and the held call reaches the model once. A relay killed during a held call is recorded as RelayError. Both pass.
  • tests/unit/interception/test_relay.py: forwarding, the secret swap, streaming, resend on a reroute, edge-page retries, the interception's errors passed through, a body cut short, the retry window, the recorded failure reason. tests/unit/interception/test_tunnel.py: movable rollouts follow a new tunnel and the others fail; a second loss during a replacement; a failed replacement.
  • pytest tests/unit: 241 passed, 19 skipped (no Docker, no PRIME_API_KEY, no harbor). ruff clean.
  • CI found a race this PR exposed: the relay prepares its uv script during the harness setup, and two uv bootstraps at once in an image without uv can fail. prepare_uv_script now bootstraps uv once per box at a time.

🤖 Generated with Claude Code

🤖 Generated with Claude Code


Note

High Risk
Touches core model-call path, tunnel failover, and rollout networking; incorrect relay/retry/reroute logic could duplicate turns, mask provider errors, or leave rollouts stuck on dead tunnels.

Overview
Remote harness rollouts no longer talk to the interception server at the Prime tunnel URL. A per-rollout loopback relay runs in the sandbox at 127.0.0.1, forwards model HTTP/SSE to the current upstream, swaps bearer/API-key tokens, and retries transport failures and tunnel-edge error pages for up to five minutes with retry markers so the interception server can join duplicates to one in-flight call.

Rollout starts the relay alongside harness setup, points the harness at the relay base URL, registers RolloutSession.reroute when the rollout is “movable” (relay-only path: no tool servers, shared tools, or deployment services), and surfaces RelayError on the trace when the relay fails instead of blaming the harness.

On a permanently lost tunnel, the interception server still retires and refuses new sessions, but opens a replacement tunnel and calls each movable session’s reroute to repoint its relay; rollouts that still hit the tunnel URL directly keep failing with TunnelError. Responses from the interception server now carry x-vf-interception so the relay can distinguish real server answers from edge HTML.

Also adds architecture docs, broad unit/integration coverage, slightly richer branch assertions in integration helpers, and serializes uv bootstrap in prepare_uv_script so relay and harness setup do not race installing uv.

Reviewed by Cursor Bugbot for commit 69ecaad. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Add relay between remote harness and interception server, with tunnel rerouting

  • Runs a relay process inside the remote runtime that proxies harness model calls to the interception server, swapping configured bearer and API-key tokens. The rollout uses the relay URL as the harness base and waits for relay installation (relay/init.py, rollout.py)
  • The relay streams SSE, retries connection failures, truncated bodies, and unmarked tunnel-edge error pages within a retry window, and emits SSE keepalives or error events while waiting. SIGHUP reloads the upstream and retries in-flight requests (relay/server.py)
  • On tunnel loss, the interception server now replaces the tunnel and reroutes movable rollout sessions (relay-backed, no toolsets/shared tools/deployment spec) via Rollout._reroute; non-reroutable sessions get TunnelError and are released (server.py)
  • Adds RelayError (an InterceptionError subtype) exported from verifiers.v1, and attributes step failures to the relay when it records a failure or exit (errors.py)
  • Fixes Runtime.prepare_uv_script to mark a user's uv bootstrap ready after success, so later preparations skip the shared lock (runtime.py)
  • Behavioral Change: remote-harness rollouts with toolsets, shared tools, or a deployment spec still use the tunnel URL directly and are failed (not rerouted) on tunnel loss; check Rollout harness setup in rollout.py

Macroscope summarized 69ecaad.

Add an in-box relay (`interception/relay`) that a harness in a remote
box reaches the interception server through, at 127.0.0.1, instead of
the tunnel URL (`agent.relay`, off by default). The relay retries a
model call for up to 5 minutes while the tunnel cannot be reached, and
marks each retry so the interception server joins it to the call it
already serves.

When a tunnel is lost for good, its server still takes no new
rollouts, but no longer fails the ones that reach it through a relay:
it opens a new tunnel, allows its host in each box's egress policy and
reroutes each relay. Rollouts with tool servers or deployment services
still fail, since those reach the old URL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wiz-prime-intellect

wiz-prime-intellect Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Wiz Scan Summary

Scanner Findings
Vulnerability Finding Vulnerabilities -
Data Finding Sensitive Data -
Secret Finding Secrets -
IaC Misconfiguration IaC Misconfigurations -
SAST Finding SAST Findings 1 Medium 1 Low
Software Management Finding Software Management Findings -
Total 1 Medium 1 Low

View scan details in Wiz

To detect these findings earlier in the dev lifecycle, try the Wiz Code extension for VS Code, JetBrains, or Visual Studio.

Comment thread verifiers/v1/rollout.py Resolved
Comment thread verifiers/v1/interception/server.py Resolved
Comment thread verifiers/v1/rollout.py Resolved

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread verifiers/v1/rollout.py Resolved
@macroscopeapp

macroscopeapp Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Needs Human Approval

Macroscope's review found this PR requires human approval — The relay lets remote harnesses survive tunnel replacement without restarting model calls. Unresolved concerns about competing egress-policy updates and retained old tunnel resources need resolution because they can disrupt rollouts, overwrite scoring policy, or consume tunnel-service resources.

Needs human approval because:

  • 2 blocking correctness issues found at or above your repo's Minimum Blocking Severity

No code changes detected at 69ecaad. Prior analysis still applies.

Adjust the Minimum Blocking Severity for this repo — including turning it Off — in Settings. You can add or adjust custom eligibility rules. Learn more.

Drop `agent.relay`: a harness in a remote box always reaches the
interception server through its relay, and a relay that does not start
leaves it on the tunnel URL with a warning.

- Give the relay only the model secret, which the harness already
  holds: a harness that reads the relay config gains no capability.
- Run one relay per rollout (own dir, ephemeral port) and stop it when
  the rollout closes, so rollouts that share a box never share one.
- Make a rollout movable before the server registers it; a tunnel lost
  while the relay starts waits for it.
- Replace the tunnel again when it is lost during a replacement.
- Retry a non-streaming body that breaks before any data, and a body
  that ends before its Content-Length.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread verifiers/v1/rollout.py Resolved
The relay added about 4 s to setup (GSM8K, p50 6.4 s -> 10.6 s): eight
sandbox calls one after another. Launch it while the harness sets up,
with a placeholder upstream, and configure it once the interception
slot is acquired: one write and one exec on the critical path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread verifiers/v1/rollout.py Resolved
mikasenghaas and others added 2 commits November 6, 2026 00:48
A `Relay` object owns the relay's lifecycle (launch alongside the
harness setup, connect, reroute, stop), so the rollout keeps one field.
The retry window is a constant; tests patch the script source.

The relay verified the tunnel's certificate against the system CA
store, which a task image may lack: terminal-bench-2's
financial-document-processor image has no /etc/ssl/certs, so both its
relayed rollouts failed on their first model call while the harness's
own HTTP client (certifi) worked on main. Trust certifi's roots as well.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The relay is part of the machinery like the interception server: a
relay that does not start fails the rollout with a SandboxError, which
the rollout retry rules cover, instead of a silent fallback to the
tunnel URL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mikasenghaas and others added 2 commits November 6, 2026 00:56
Add `RelayError` (an `InterceptionError`, next to `TunnelError`). A
relay that does not start raises it. A relay that gives up on a
request appends the reason to a failures file in its directory; when
the harness then dies, the rollout checks that file and whether the
relay still runs, and records `RelayError` with the relay's reason
instead of the harness's `HarnessError`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mikasenghaas mikasenghaas changed the title feat: keep rollouts through a lost interception tunnel feat: run a relay between the harness and the interception server Nov 6, 2026
Comment thread verifiers/v1/rollout.py Resolved
Comment thread verifiers/v1/rollout.py Resolved

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

mikasenghaas and others added 3 commits November 6, 2026 01:34
Scripts prepared in parallel each ran the uv bootstrap under a lock of
their own script: in an image without uv, two `pip install --user uv`
runs or downloads to one path (or apt's lock) raced, and one failed.
The relay prepares its script during the harness setup, so CI hit it:
a random network-policy cell failed with zero branches. The first
preparation per user now holds a bootstrap lock; once uv is there,
scripts prepare in parallel.

The integration grid's one-branch assertions show the trace's errors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Mark a resend as a retry only after an attempt that may have reached
  the server; a connect failure or an edge page did not, so a body a
  later turn repeats is never answered from the replay cache.
- Never frame SSE with a fixed length, and close an edge error page
  unread instead of draining it.
- Count a new outage from its start once a response stayed up.
- Record "ok" after a request goes through again, so a stale relay
  failure never names a later, unrelated harness death.
- Stop the relay process also after a failed or cancelled launch.
- Leave a closing rollout's egress alone on a reroute (shared scoring
  sets its own), and strip the trailing slash `host_url` may return.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The relay starts alongside the harness setup. Its install puts uv in
a box that has none, and the setup of some harnesses (prime-agent,
hermes-agent) does the same in a shell of its own. The two installs
could overlap, and one then ran the other's half-written binary:
"uv: Permission denied" from the relay start, seen once in CI.

Co-Authored-By: Claude <noreply@anthropic.com>
macroscopeapp[bot]
macroscopeapp Bot previously approved these changes Nov 11, 2026
mikasenghaas and others added 2 commits November 13, 2026 20:14
# Conflicts:
#	verifiers/v1/interception/server.py
The relay took every 404, 502, 503 or 504 that was not JSON or SSE
for the error page of a tunnel edge and retried it for five minutes.
The interception server answers an unknown path with a plain-text
404, so a harness that asked for one waited that long.

The server now marks each of its responses with a header, and only
a response without it counts as the edge's.

Co-Authored-By: Claude <noreply@anthropic.com>
(cherry picked from commit a1b88d8)
if not self.scanner.data:
# No body data went out: the retry can still answer.
return False
self.reset()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium relay/server.py:516

When an upstream body breaks or ends before Content-Length after body data has been sent, the relay records the request as successful: both branches call reset() and return True, so relay() calls record_success(). This omits the transport failure from the failures file (and can append ok after a prior failure), causing Rollout.step to report HarnessError instead of RelayError. Record these truncated-response failures before closing, and ensure they are not treated as successful completion.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/interception/relay/server.py around line 516:

When an upstream body breaks or ends before `Content-Length` after body data has been sent, the relay records the request as successful: both branches call `reset()` and return `True`, so `relay()` calls `record_success()`. This omits the transport failure from the failures file (and can append `ok` after a prior failure), causing `Rollout.step` to report `HarnessError` instead of `RelayError`. Record these truncated-response failures before closing, and ensure they are not treated as successful completion.

upstream.attach(connection.sock)
# Only the connect is bounded: a model call may wait long for its first
# byte.
connection.sock.settimeout(None)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High relay/server.py:419

A retry that connects but receives no response headers can hang indefinitely instead of failing after RETRY_SECONDS; getresponse() blocks in open() and never reaches unreachable(). Setting the socket to blocking mode here removes the only timeout, while keepalive() continues sending comments and prevents the client’s idle timeout. Preserve an enforceable deadline or watchdog for stalled retry attempts.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/interception/relay/server.py around line 419:

A retry that connects but receives no response headers can hang indefinitely instead of failing after `RETRY_SECONDS`; `getresponse()` blocks in `open()` and never reaches `unreachable()`. Setting the socket to blocking mode here removes the only timeout, while `keepalive()` continues sending comments and prevents the client’s idle timeout. Preserve an enforceable deadline or watchdog for stalled retry attempts.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread verifiers/v1/rollout.py Resolved
logger.warning(
"interception tunnel replaced: url=%s -> %s", self.base_url, base_url
)
self.base_url = base_url

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Old tunnels stay on the stack

Medium Severity

Each replacement calls stack.enter_async_context(self.tunnel.expose(self.port)) and never exits the previous expose. The gone tunnel's frpc is therefore not sync_stopped until the whole interception server stops. A loss during replacement (_lost_again) stacks another live tunnel on the same host port, so a long rollout that flaps can leak processes, logins against a deleted registration, and rate-limit budget for the rest of that server's life.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 167b4b6. Configure here.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 69ecaad. Configure here.

"""Wait until `connect` finished; raise if the relay did not start."""
await self._connected.wait()
if not self.upstream:
raise TunnelError("the relay did not start")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Relay start misattributed in the trace

Medium Severity

A relay that does not start or reload is recorded as SandboxError or TunnelError, not RelayError. boundary(RelayError) around connect lets those RolloutErrors through, and wait_connected raises TunnelError("the relay did not start"), so traces and stop conditions name the sandbox or tunnel instead of the relay.

Additional Locations (2)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 69ecaad. Configure here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant