Skip to content

exp: probe which harnesses echo their replies (do not merge) - #3170

Closed
mikasenghaas wants to merge 63 commits into
mainfrom
exp/echo-probe
Closed

mikasenghaas wants to merge 63 commits into
mainfrom
exp/echo-probe

Conversation

@mikasenghaas

@mikasenghaas mikasenghaas commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Throwaway experiment, not for merge. It sits on the branch of feat: restore rollouts from memory snapshots #3141 and will be closed when the CI run is read.
  • The run fixture of the integration tests reports, per trace, how many sampled replies the harness showed again in its next request in a form the message graph matched.
  • The workflow runs only the multi-turn jobs.

Question it answers: could a reply be committed to the trace when the box confirms it with its next request, for every bundled harness?

🤖 Generated with Claude Code

Note

Add rollout checkpoint, memory-restore, and relay-reroute support for harness probes

  • Adds a checkpoint system: RolloutCheckpointConfig in agent config, periodic snapshot capture with last/interval retention, trace rewind via Trace.rewind, and retry-from-checkpoint in Agent.run (agent.py, rollout.py, trace.py)
  • Adds memory snapshot/restore to PrimeBackend (pause/resume sandbox, attach to in-flight jobs) and SUPPORTS_MEMORY_RESTORE harness sessions, including ACP detached runners served over a Unix socket (prime.py, harness.py, acp)
  • Adds a Relay launched in the runtime for remote harness traffic; lost tunnels now trigger replacement and session rerouting instead of failing all sessions, with RolloutSession.abort/flow-control to hold turns (relay, server.py, session.py)
  • Adds integration fixtures (memory, subagent, tick restore) and an _echo_probe diagnostic in conftest.py to inspect which harnesses echo their replies, per the PR intent; the behavior matrix in tests.yml drops many scheduled entries during the probe
  • Behavioral Change: agent timing now reports checkpoint-wait time separately and excludes earlier-attempt calls; record_error gains attempt/stage kwargs; _run_once returns a Rollout instead of a Trace, so in-tree callers in agent.py are updated but external users of that private helper would break

Macroscope summarized 3be0b89.

mikasenghaas and others added 30 commits November 5, 2026 00:54
Add an in-box relay (`interception/relay`) that a harness in a remote
box reaches the interception server through, at 127.0.0.1, instead of
the tunnel URL (`agent.relay`, off by default). The relay retries a
model call for up to 5 minutes while the tunnel cannot be reached, and
marks each retry so the interception server joins it to the call it
already serves.

When a tunnel is lost for good, its server still takes no new
rollouts, but no longer fails the ones that reach it through a relay:
it opens a new tunnel, allows its host in each box's egress policy and
reroutes each relay. Rollouts with tool servers or deployment services
still fail, since those reach the old URL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drop `agent.relay`: a harness in a remote box always reaches the
interception server through its relay, and a relay that does not start
leaves it on the tunnel URL with a warning.

- Give the relay only the model secret, which the harness already
  holds: a harness that reads the relay config gains no capability.
- Run one relay per rollout (own dir, ephemeral port) and stop it when
  the rollout closes, so rollouts that share a box never share one.
- Make a rollout movable before the server registers it; a tunnel lost
  while the relay starts waits for it.
- Replace the tunnel again when it is lost during a replacement.
- Retry a non-streaming body that breaks before any data, and a body
  that ends before its Content-Length.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The relay added about 4 s to setup (GSM8K, p50 6.4 s -> 10.6 s): eight
sandbox calls one after another. Launch it while the harness sets up,
with a placeholder upstream, and configure it once the interception
slot is acquired: one write and one exec on the critical path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A `Relay` object owns the relay's lifecycle (launch alongside the
harness setup, connect, reroute, stop), so the rollout keeps one field.
The retry window is a constant; tests patch the script source.

The relay verified the tunnel's certificate against the system CA
store, which a task image may lack: terminal-bench-2's
financial-document-processor image has no /etc/ssl/certs, so both its
relayed rollouts failed on their first model call while the harness's
own HTTP client (certifi) worked on main. Trust certifi's roots as well.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The relay is part of the machinery like the interception server: a
relay that does not start fails the rollout with a SandboxError, which
the rollout retry rules cover, instead of a silent fallback to the
tunnel URL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add `RelayError` (an `InterceptionError`, next to `TunnelError`). A
relay that does not start raises it. A relay that gives up on a
request appends the reason to a failures file in its directory; when
the harness then dies, the rollout checks that file and whether the
relay still runs, and records `RelayError` with the relay's reason
instead of the harness's `HarnessError`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Scripts prepared in parallel each ran the uv bootstrap under a lock of
their own script: in an image without uv, two `pip install --user uv`
runs or downloads to one path (or apt's lock) raced, and one failed.
The relay prepares its script during the harness setup, so CI hit it:
a random network-policy cell failed with zero branches. The first
preparation per user now holds a bootstrap lock; once uv is there,
scripts prepare in parallel.

The integration grid's one-branch assertions show the trace's errors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Mark a resend as a retry only after an attempt that may have reached
  the server; a connect failure or an edge page did not, so a body a
  later turn repeats is never answered from the replay cache.
- Never frame SSE with a fixed length, and close an edge error page
  unread instead of draining it.
- Count a new outage from its start once a response stayed up.
- Record "ok" after a request goes through again, so a stale relay
  failure never names a later, unrelated harness death.
- Stop the relay process also after a failed or cancelled launch.
- Leave a closing rollout's egress alone on a reroute (shared scoring
  sets its own), and strip the trailing slash `host_url` may return.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An agent snapshots its prime box as model turns go upstream, and a
rollout retry restores the newest snapshot in a new box and continues
the trace from there instead of starting over. On by default
(`[env.agent.checkpoint]`: interval, keep, after, boxless).

The trace records each snapshot (`Trace.checkpoints`), rewinds to the
one a retry restores (`Trace.rewind`, `Trace.generation`), and gives
another process what it needs to continue it (`Trace.restorable`,
`RestorePoint`, `Env.run_slot(restore=...)`). null and bash relaunch on
the transcript; claude-code, codex and rlm load their ACP session from
the restored box (rlm pinned to nano-rlm with session/load).

A request hook that raises now ends the rollout with its error.

This is the filesystem-only state of verifiers#2994 as of ebfd9f8,
on today's main. Its env-server parts moved with the env server to
prime-rl and are not part of this change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Scope rollout checkpointing to retries within one Agent.run: drop the
cross-process restore (Env and Agent.run restore params, RestorePoint,
PinnedResult, Trace.restorable, Agents.only).

Call the runtime side a snapshot: backends return a typed Snapshot
(id, type) from snapshot(), wait_snapshot() and discard_snapshots()
take it, and restore_config() builds the config a new box boots from.
Runtimes and harnesses declare SUPPORTS_FILESYSTEM_SNAPSHOTS, and
RolloutCheckpointConfig.type takes a SnapshotType, so memory snapshots
can slot in behind the same API.

Also from review:
- snapshot every 10th turn by default
- copy state before the first await of a capture, so response hooks
  that run meanwhile do not leak into the checkpoint
- trim the checkpoints after the restored one by position, not node
- build the rollout params once per run, so an RLM retry keeps its
  skill home and the restored session loads
- discard snapshots on the shared leased client

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A turn cancelled while its reply waits for the snapshot capture now
still records its call and abandons its preview. The wait is shielded,
so the snapshot finishes in the background, where the rollout awaits
it.

A capture reads the turn's node and tail before its first await, like
the state copies, so everything a checkpoint describes comes from the
idle box.

Docs: show the checkpoint config as a TOML example, and shorten the
harness flag's entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Restore a rollout from a snapshot end to end on prime: per harness, at
the default interval, for rlm with skills, and for each retention
policy. The cases run once per SnapshotType, on every harness and
runtime that declare SUPPORTS_<TYPE>_SNAPSHOTS, so a new snapshot type
gets its cases without a test change. CI runs it as "Checkpoint
Restore".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A box-less checkpoint let a harness that runs no code restore from the
trace alone on a runtime without snapshots. It served restores in
another process, which this PR no longer does; for retries in one run
it only saves the few turns of a short chat rollout. Every checkpoint
now has a snapshot, and the boxless setting is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Record every snapshot the run's boxes take, across attempts, and check
each on the platform once the run is done: deleted, except the ones
the retention policy keeps, which stay restorable. Before, only the
checkpoints left on the final trace were checked, so a rewound
snapshot or one still uploading at the end could leak unnoticed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	verifiers/v1/agent.py
#	verifiers/v1/rollout.py
#	verifiers/v1/runtimes/backends/prime.py
When a rollout is cancelled (an agent timeout, shutdown), its harness
turn is cancelled and then joined. A second cancellation during that
join left the turn running while teardown began, where it could still
write the trace or use the box. Join it shielded, then re-raise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From review: keep the checkpoint config to what a user decides.

- interval counts from turn n (the n-th, 2n-th, ...), so a rollout
  shorter than n turns takes no snapshot; the turn a restore resumes
  at is not snapshotted again, so keep = "last" no longer deletes the
  snapshot that just restored
- every snapshot is deleted once the run is final: drop `after`
- drop keep = "all", which keep = 1 already expresses
- drop `type` from the config and SnapshotType from the API: the
  runtime takes the best snapshot it supports and records its type on
  the checkpoint (Snapshot.type)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A harness can continue a rollout in a box restored from a snapshot
when a new process of it can take the rollout over. For a harness that
resumes on the transcript (SUPPORTS_RESUME), a restore is that same
relaunch, so it needs nothing more. An ACP harness keeps its session
in a live process and needs its agent to load the session from its
files: ACPHarness.LOADS_SESSIONS (claude-code, codex, rlm). This
replaces the harness-side SUPPORTS_FILESYSTEM_SNAPSHOTS
(Harness.can_restore). browser-use now restores like any harness that
resumes on its transcript.

A restore whose attempt fails before its first new turn is marked
failed like one whose box never booted, so the next retry steps back
to an older snapshot or starts over instead of restoring the same
snapshot until the retries run out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The condition read `target is not None and A or B`, so B ran for runs
that restored nothing and read `restored_turns` on them. Group it as
`target is not None and (A or B)`. The new test's fake rollout also
called the run's `on_trace` hook, which it does not pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mem-ckpt

# Conflicts:
#	verifiers/v1/session.py
…at/mem-ckpt

# Conflicts:
#	verifiers/v1/interception/server.py
A memory snapshot (a pause of the box) holds the box's processes. A
retry boots a new box from it, reroutes the box's relay to the new
session and waits for the harness program the box still runs, so no
harness has to load a session or be told about the restore.

The ACP runner can run as a job of the box with one job per request,
because no connection from the host survives a stopped box.

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
mikasenghaas and others added 26 commits November 10, 2026 04:46
A box created from a memory snapshot continues the paused box's
idle clock. In a training run resumed 2 h 41 min after its
checkpoint, the platform ended 7 of 46 such boxes as idle before
they ran, and each episode failed without a retry.

Create the box again up to three times when it ends before it
runs. If the restore point of an owner still leads nowhere, start
the rollout over without spending one of its retries: that is what
it would have done without the point.

Co-Authored-By: Claude <noreply@anthropic.com>
A memory snapshot continues every process of the box, so it does
all that a filesystem snapshot with a new harness process did, and
needs nothing from the harness. Remove the filesystem path: the
snapshot type and its config field, the restore prompt, session
loading in the ACP runner, and the harness state paths it needed.

A rollout that cannot take memory snapshots (a runtime without
them, a harness that does not run detached, tool servers or
services of its own) takes no checkpoints, and a retry starts over.

Co-Authored-By: Claude <noreply@anthropic.com>
A snapshot at every 10th turn stops every healthy box for a few
seconds. Take the snapshot when it is needed instead: when a model
call fails with an error the run retries, before the harness gets
the error. The box that the retry restores still waits for the
reply, and asks for the call again.

`checkpoint.interval` is now off by default. It stays as an option
for failures that leave no box to snapshot (the box died, the
harness program crashed).

A model call that fails again right after a restore no longer
counts as a restore that led nowhere: the retry restores the same
snapshot.

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
A memory snapshot holds every process of the box, so the sub-agents
of a harness continue with it. Nothing tested that. The new case
saves an rlm rollout while two sub-agents are at work, and checks that
a second owner's box continues both without starting either again.

Co-Authored-By: Claude <noreply@anthropic.com>
The relay starts alongside the harness setup. Its install puts uv in
a box that has none, and the setup of some harnesses (prime-agent,
hermes-agent) does the same in a shell of its own. The two installs
could overlap, and one then ran the other's half-written binary:
"uv: Permission denied" from the relay start, seen once in CI.

Co-Authored-By: Claude <noreply@anthropic.com>
The relay starts alongside the harness setup. Its install puts uv in
a box that has none, and the setup of some harnesses (prime-agent,
hermes-agent) does the same in a shell of its own. The two installs
could overlap, and one then ran the other's half-written binary:
"uv: Permission denied" from the relay start, seen once in CI.

Co-Authored-By: Claude <noreply@anthropic.com>
An owner that saves every running rollout pauses all their boxes at
once. At thousands of rollouts that is a burst of pause and resume
calls the platform may refuse, and a box whose resume fails never
runs its rollout again. snapshots_per_min spreads the pauses out with
the bucket that already paces creates. Off by default.

Co-Authored-By: Claude <noreply@anthropic.com>
A saved rollout was a RestorePoint: the trace plus fields beside it
that only that class carried. Whoever held just the trace could not
continue it. save_all now returns the trace as plain data, with what
a restore needs on the entry of its checkpoint, and a restore takes
that trace.

Snapshots on a failed model call are now opt-in (checkpoint.on_error).
They help only when the harness would end on the error; a harness
that handles the failed call itself never uses the snapshot, and a
run without retries never takes one.

Co-Authored-By: Claude <noreply@anthropic.com>
Main moved blocking commands on Prime to native sessions, so a task
that clears scratch directories cannot delete the job files of a
running command. A session ends with its box, and a box that boots
from a memory snapshot is a new one. The harness's main program
therefore stays a server-owned job, which `attach` can continue to
wait on. Every other command takes main's session path.

Co-Authored-By: Claude <noreply@anthropic.com>
A box that boots from a memory snapshot keeps the sandboxd session of
the harness program, so the host can attach to it by its session id.
The program no longer needs a polled background job, and it gets the
same protection as every other command against a task that clears
the scratch directories.

A snapshot also ends the stream into the box that was paused. The
wait now attaches to the session again when the box runs.

Needs a prime-sandboxes release with attach_process.

Co-Authored-By: Claude <noreply@anthropic.com>
A memory snapshot resets the box's open connections, so the relay
asks again for the call that just failed. The interception server
keeps no failed attempt, so the box gets a new model call and the
harness rarely gets the error at all. The snapshot was a slow way to
retry a model call, and its outcome depended on when the relay's
request arrived.

A retry of a model call belongs at the call. Snapshots stay for a
box or a host that goes away: the owner's save, and `interval`.

A call that the box asks again at a snapshotted turn now takes no
second snapshot, or a call that keeps failing would take one per
request.

Co-Authored-By: Claude <noreply@anthropic.com>
The restore tests continued a rollout from one point and cut the
trace back at most one turn. This one saves a rollout at three
turns and continues each saved trace: with a scripted model, every
continuation must end with the messages of the first run. A counter
that a process holds only in memory shows a command that ran twice
or was skipped.

Co-Authored-By: Claude <noreply@anthropic.com>
A pause that the platform accepted stops the box. When the wait for
the paused state then failed, the box was never resumed and its
snapshot was dropped from the list that teardown frees. One failed
batched status call fails every box that waits in it, so a save of
many rollouts could stop them all for good.

Resume the box whatever the wait came to, keep the snapshot for
teardown, and ask again when a status poll fails.

Co-Authored-By: Claude <noreply@anthropic.com>
The relay took every 404, 502, 503 or 504 that was not JSON or SSE
for the error page of a tunnel edge and retried it for five minutes.
The interception server answers an unknown path with a plain-text
404, so a harness that asked for one waited that long.

The server now marks each of its responses with a header, and only
a response without it counts as the edge's.

Co-Authored-By: Claude <noreply@anthropic.com>
Three cases let a box see a reply that its checkpoint does not
record, or left a rollout stopped:

- A release (the owner's, or the timer) answered the box while
  another snapshot had it stopped. The snapshot now ends the hold
  itself when it is done.
- A save that was cancelled or failed kept the hold with no timer
  and left its snapshot pinned for an owner that never got it.
- A snapshot of a turn that waited for another snapshot recorded
  the trace with that turn's reply, which the box had not got. It
  is now skipped.

A save without a hold no longer ends the hold of an earlier one.

Co-Authored-By: Claude <noreply@anthropic.com>
The harness program now runs in a session and a restored box is
attached to by session id, which needs `attach_process`. Remove the
pin when a release has it.

Co-Authored-By: Claude <noreply@anthropic.com>
The fake backend finished its snapshot at once, so the save was never cancelled.

Co-Authored-By: Claude <noreply@anthropic.com>
# Conflicts:
#	verifiers/v1/interception/server.py
The relay took every 404, 502, 503 or 504 that was not JSON or SSE
for the error page of a tunnel edge and retried it for five minutes.
The interception server answers an unknown path with a plain-text
404, so a harness that asked for one waited that long.

The server now marks each of its responses with a header, and only
a response without it counts as the edge's.

Co-Authored-By: Claude <noreply@anthropic.com>
(cherry picked from commit a1b88d8)
# Conflicts:
#	verifiers/v1/interception/relay/__init__.py
#	verifiers/v1/rollout.py
Experiment only, not for merge: report per trace how many sampled replies the next request matched, and run only the multi-turn jobs.

Co-Authored-By: Claude <noreply@anthropic.com>
@wiz-prime-intellect

Copy link
Copy Markdown

Wiz Scan Summary

Scanner Findings
Vulnerability Finding Vulnerabilities -
Data Finding Sensitive Data -
Secret Finding Secrets -
IaC Misconfiguration IaC Misconfigurations -
SAST Finding SAST Findings 1 Medium 1 Low
Software Management Finding Software Management Findings -
Total 1 Medium 1 Low

View scan details in Wiz

To detect these findings earlier in the dev lifecycle, try the Wiz Code extension for VS Code, JetBrains, or Visual Studio.

@mikasenghaas

Copy link
Copy Markdown
Member Author

Read the run: 13 of 14 harnesses show every reply again in a form the graph matches; openclaw does not (6 of 11 replies ended as unmatched leaves). Closing the experiment.

@mikasenghaas
mikasenghaas deleted the exp/echo-probe branch November 13, 2026 20:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant