Skip to content

feat: restore rollouts from memory snapshots - #3141

Draft
mikasenghaas wants to merge 68 commits into
mainfrom
feat/mem-ckpt
Draft

mikasenghaas wants to merge 68 commits into
mainfrom
feat/mem-ckpt

Conversation

@mikasenghaas

@mikasenghaas mikasenghaas commented Nov 10, 2026 •

Copy link
Copy Markdown
Member

Summary

A running rollout can now be stopped and continued later, in a new sandbox and by another process, without losing work. The harness needs no changes and gets no notice.

  • Memory snapshots. A snapshot is a pause and resume of the Prime sandbox. A restore forks the snapshot into a new box that continues every process: the harness program, its sub-agents and shells, and the task's own processes.
  • A relay in the box. The harness talks to a small HTTP server on loopback, which forwards to the interception server on the host. After a restore the host is a different one; only the relay learns the new address.
  • The trace is the restore input. A saved trace holds its checkpoints, and a checkpoint names the snapshot. Another process continues the rollout from that trace alone.
  • Two triggers. The owner of a rollout saves it (save_all), or the rollout snapshots itself at every n-th turn (checkpoint.interval). No snapshot is taken by default.

This branch contains #3074 (the relay), which is still open: review the relay there and the rest here. It supersedes #3073 (closed).

Before merge: prime-sandboxes is pinned to the branch of PrimeIntellect-ai/prime#1147 (attach_process). Replace the pin with a release.

How it works

flowchart LR
  subgraph box["Sandbox (forked on restore)"]
    H["Harness program"] -->|"127.0.0.1"| R["Relay"]
  end
  R -->|"tunnel"| A["Interception server, host A"]
  R -.->|"after restore"| B["Interception server, host B"]
Loading

Snapshot

  1. The interception server holds every reply to the box: model replies, errors, tool verdicts and state writes. The stopped box has then seen no more than the trace that the checkpoint records.
  2. The runtime pauses the sandbox, which takes the snapshot, and resumes it.
  3. The checkpoint goes on the trace: the snapshot id and the node it belongs to. It also keeps what the host knows beside the trace: task state, rewritten tool results, tool verdicts, prepared user messages, and every reply that is on the trace and not yet in the box.
  4. The pause reset the box's connections. The relay asks again for its open model call, and the interception server answers with the stored reply.

Restore

  1. A new rollout creates a box from the snapshot. Every process continues where it stopped.
  2. The rollout points the relay at its own interception server and maps the secret that the harness holds to its own.
  3. The relay sends the open model call again, and the trace continues from the checkpoint's node.
  4. The host waits for the harness program that the box still runs (Runtime.attach).

API surface

Config

Field Default Meaning
[env.agent.checkpoint] on The rollout can be snapshotted. --no-env.agent.checkpoint turns it off.
checkpoint.interval none Snapshot at every n-th model turn.
checkpoint.keep "last" Snapshots kept while the rollout runs.
runtime.snapshots_per_min none Run-wide pace of snapshots, so a save of thousands of rollouts does not pause every box at once.

For the owner of a rollout (verifiers.v1.restore)

Call What it does
save_all(hold=False, skipped=None) Snapshots every running rollout and returns each trace as plain data. It returns when the boxes run again.
release_all() Ends the hold of save_all(hold=True), under which a saved rollout waits at its checkpoint. A hold ends by itself after 120 s.
RunSlot(task, id=..., restore=trace), Agent.run(task, restore=trace) Continues a saved rollout, in any process.
unsaveable(config) Names what rules snapshots out for an agent config.

Trace

Addition Meaning
Trace.checkpoints One Checkpoint per snapshot: id, snapshot, node, pinned, restored, deleted. A saved trace also carries what a restore needs (Checkpoint.restore).
Trace.rewind Cuts the trace back to a checkpoint. A retry of a failed run restores the newest snapshot and continues from there.
timing.agent.checkpoint Time the box was held for snapshots, split off the model time.

Runtime and harness

Addition Meaning
Runtime.snapshot, wait_snapshot, restore_config, discard Take a snapshot, wait until it is restorable, build the config of a restored box, delete a snapshot. Prime implements them (SUPPORTS_MEMORY_SNAPSHOTS).
Runtime.run(..., program=), Runtime.attach Run the harness's main program so that a restored box can be waited on again. On Prime every command runs in a native session, as on main; the host attaches to the program's session by id in the restored box, and again after a snapshot ended the stream.
Harness.SUPPORTS_MEMORY_RESTORE Declares that the harness runs detached from the host: null, bash and the ACP harnesses.
ACP runner Runs as a job of the box with one job per request over a Unix socket, because a streamed exec does not survive a pause.
Relay (#3074) connect, reroute, installed, state, failure, stop.
x-vf-interception header The interception server marks each of its responses. The relay retries a 404, 502, 503 or 504 only without the mark, so the server's own 404 for an unknown path reaches the harness at once.

Robustness on Prime

  • A resume that the platform refuses for lack of node memory is asked again for up to 600 s. A box left paused never runs its rollout again.
  • A box that ends before it runs from a snapshot is created again, up to three times. A saved trace that still leads nowhere starts the rollout over and costs no retry.
  • A saved rollout does not finish while its owner holds it, and a restored rollout gets the agent time that is left, not the full timeout.
  • checkpoint.interval counts turns from the newest snapshot, so conversations that run side by side in one box cannot step past it.
  • A box is resumed whatever the wait for its pause came to, and a failed status poll is asked again. The snapshot stays on the list that teardown frees.
  • Boxes that wait for a pause or a resume share batched status polls.
  • Every snapshot is deleted once the run is final. An owner's snapshot (Checkpoint.pinned) is the owner's to delete.

Limits

A rollout takes snapshots only when all of these hold. Otherwise a retry starts over, and save_all reports why it skipped the rollout:

  • The runtime is Prime.
  • The harness runs detached from the host (see above).
  • The rollout has no tool servers, shared tools or services of its own. They reach the host at an address that does not follow the box.
  • The rollout runs in its own box. A saved trace continues one rollout, so an owner can continue only single-agent episodes.

Two more:

  • A snapshot stops the box for several seconds and resets its open connections to the outside. The relay covers the model calls. A task process with its own long-lived connection sees a reset.
  • A failed model call takes no snapshot. The relay asks again after any pause, and the interception server answers a failed attempt with a new model call, so a retry of a model call belongs at the call.

Verification

CI on Prime sandboxes, job Checkpoint Restore: 16 passed, 4 skipped. Each test proves a restore with a nonce that exists only in the memory of a process in the box, and checks that no snapshot or sandbox leaks.

Test Harnesses What it shows
test_retry_restores_checkpoint null, bash, rlm, codex, claude-code, prime-agent, hermes-agent, opencode, stirrup, deepseek Every model call of the first attempt fails at one turn. The harness ends on the error, and the retry restores the snapshot of that turn.
test_another_owner_continues_a_saved_rollout bash, rlm save_all saves a running rollout, the rollout is cancelled, and a second env with its own server, tunnel and secrets continues it from the pickled trace.
test_another_owner_continues_running_sub_agents rlm The save happens while two sub-agents are at work. Both take their next turns in the second owner's box, and neither is started again.
test_every_saved_turn_continues_to_the_same_result bash One rollout is saved at turns 1, 2 and 3 and then finishes. A second owner continues each saved trace. With a scripted model, every continuation ends with the same messages as the first run, so no command ran twice and none was skipped.
test_a_restore_point_that_leads_nowhere_starts_the_rollout_over bash, rlm The saved trace names a snapshot that does not exist. The rollout starts over under the same trace id.

Skips: pi and openclaw finish their run despite the failed call; mini-swe-agent cannot continue from a memory snapshot; Kimi is rejected by the test endpoint.

Training runs through PrimeIntellect-ai/prime-rl#4456, with traces linked there:

Run Saved rollouts Result
bash harness, 6 nodes, fork of a checkpoint 90 85 finished, 1 started over, 4 still ran when the run was stopped
rlm harness, 3 nodes, SIGKILL and resume in place 116 All 116 continued in new boxes and finished
bash harness, 4 nodes, fork of a checkpoint that was 3 h old 186 147 finished before the run ended, 5 started over

The 3 h fork found a platform behaviour: a box from a snapshot older than the paused box's idle timeout ends as idle about 20 s after it runs. 46 creates were repeated for this. The create retry and the start-over above come from that.

Measured on Prime

Setup Snapshot stops the box for Fork
16 boxes at once (1 vCPU, 2 GB) 6 to 7 s at p50 3.5 s at p50
64 boxes at once (4 GB, 1 GB in use) 7 to 12 s at p50, up to 35 s

🤖 Generated with Claude Code

Note

Restore rollouts from memory snapshots

  • Adds checkpoint-and-restore support across the stack: RolloutCheckpointConfig in agent.py (snapshot type memory/filesystem, turn interval default 10, retention policy), a Checkpoint model and Trace.rewind in trace.py, and a RestorePoint serializer in restore.py
  • Agent.run in agent.py now retries failed owned-runtime rollouts from the newest eligible checkpoint instead of restarting from scratch; borrowed runtimes are still not retried. Failed restore targets are excluded from later selection
  • PrimeBackend in prime.py implements memory snapshots by pausing and resuming the sandbox, with an awake event that blocks other remote operations during a snapshot, and background-job polling that retries through pause errors
  • Harness layer gains capability flags (SUPPORTS_MEMORY_RESTORE, LOADS_SESSIONS) and restore hooks in harness.py; ACP sessions can run as detached socket-backed runners that checkpoint and restore in-flight prompts in runner.py
  • The interception server in server.py runs checkpoints alongside the model call, records held time, and reroutes sessions to a replacement tunnel instead of failing them when a tunnel is lost
  • Behavioral Change: harness state paths moved for Codex, Claude Code, and RLM (RLM cache is now /var/tmp/vf-rlm); RolloutSession.abort is first-wins; agent timing now splits checkpoint hold time out of model time; snapshots left by rewinds, retention, and strays are deleted at cleanup; check for Trace._split_agent_time, Relay._start, and ACPHarnessSession._stop_detached if regressions appear

Macroscope summarized 87e35c5. (Automatic summaries will resume when PR exits draft mode or review begins).

mikasenghaas and others added 30 commits November 5, 2026 00:54
Add an in-box relay (`interception/relay`) that a harness in a remote
box reaches the interception server through, at 127.0.0.1, instead of
the tunnel URL (`agent.relay`, off by default). The relay retries a
model call for up to 5 minutes while the tunnel cannot be reached, and
marks each retry so the interception server joins it to the call it
already serves.

When a tunnel is lost for good, its server still takes no new
rollouts, but no longer fails the ones that reach it through a relay:
it opens a new tunnel, allows its host in each box's egress policy and
reroutes each relay. Rollouts with tool servers or deployment services
still fail, since those reach the old URL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drop `agent.relay`: a harness in a remote box always reaches the
interception server through its relay, and a relay that does not start
leaves it on the tunnel URL with a warning.

- Give the relay only the model secret, which the harness already
  holds: a harness that reads the relay config gains no capability.
- Run one relay per rollout (own dir, ephemeral port) and stop it when
  the rollout closes, so rollouts that share a box never share one.
- Make a rollout movable before the server registers it; a tunnel lost
  while the relay starts waits for it.
- Replace the tunnel again when it is lost during a replacement.
- Retry a non-streaming body that breaks before any data, and a body
  that ends before its Content-Length.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The relay added about 4 s to setup (GSM8K, p50 6.4 s -> 10.6 s): eight
sandbox calls one after another. Launch it while the harness sets up,
with a placeholder upstream, and configure it once the interception
slot is acquired: one write and one exec on the critical path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A `Relay` object owns the relay's lifecycle (launch alongside the
harness setup, connect, reroute, stop), so the rollout keeps one field.
The retry window is a constant; tests patch the script source.

The relay verified the tunnel's certificate against the system CA
store, which a task image may lack: terminal-bench-2's
financial-document-processor image has no /etc/ssl/certs, so both its
relayed rollouts failed on their first model call while the harness's
own HTTP client (certifi) worked on main. Trust certifi's roots as well.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The relay is part of the machinery like the interception server: a
relay that does not start fails the rollout with a SandboxError, which
the rollout retry rules cover, instead of a silent fallback to the
tunnel URL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add `RelayError` (an `InterceptionError`, next to `TunnelError`). A
relay that does not start raises it. A relay that gives up on a
request appends the reason to a failures file in its directory; when
the harness then dies, the rollout checks that file and whether the
relay still runs, and records `RelayError` with the relay's reason
instead of the harness's `HarnessError`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Scripts prepared in parallel each ran the uv bootstrap under a lock of
their own script: in an image without uv, two `pip install --user uv`
runs or downloads to one path (or apt's lock) raced, and one failed.
The relay prepares its script during the harness setup, so CI hit it:
a random network-policy cell failed with zero branches. The first
preparation per user now holds a bootstrap lock; once uv is there,
scripts prepare in parallel.

The integration grid's one-branch assertions show the trace's errors.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Mark a resend as a retry only after an attempt that may have reached
  the server; a connect failure or an edge page did not, so a body a
  later turn repeats is never answered from the replay cache.
- Never frame SSE with a fixed length, and close an edge error page
  unread instead of draining it.
- Count a new outage from its start once a response stayed up.
- Record "ok" after a request goes through again, so a stale relay
  failure never names a later, unrelated harness death.
- Stop the relay process also after a failed or cancelled launch.
- Leave a closing rollout's egress alone on a reroute (shared scoring
  sets its own), and strip the trailing slash `host_url` may return.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An agent snapshots its prime box as model turns go upstream, and a
rollout retry restores the newest snapshot in a new box and continues
the trace from there instead of starting over. On by default
(`[env.agent.checkpoint]`: interval, keep, after, boxless).

The trace records each snapshot (`Trace.checkpoints`), rewinds to the
one a retry restores (`Trace.rewind`, `Trace.generation`), and gives
another process what it needs to continue it (`Trace.restorable`,
`RestorePoint`, `Env.run_slot(restore=...)`). null and bash relaunch on
the transcript; claude-code, codex and rlm load their ACP session from
the restored box (rlm pinned to nano-rlm with session/load).

A request hook that raises now ends the rollout with its error.

This is the filesystem-only state of verifiers#2994 as of ebfd9f8,
on today's main. Its env-server parts moved with the env server to
prime-rl and are not part of this change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Scope rollout checkpointing to retries within one Agent.run: drop the
cross-process restore (Env and Agent.run restore params, RestorePoint,
PinnedResult, Trace.restorable, Agents.only).

Call the runtime side a snapshot: backends return a typed Snapshot
(id, type) from snapshot(), wait_snapshot() and discard_snapshots()
take it, and restore_config() builds the config a new box boots from.
Runtimes and harnesses declare SUPPORTS_FILESYSTEM_SNAPSHOTS, and
RolloutCheckpointConfig.type takes a SnapshotType, so memory snapshots
can slot in behind the same API.

Also from review:
- snapshot every 10th turn by default
- copy state before the first await of a capture, so response hooks
  that run meanwhile do not leak into the checkpoint
- trim the checkpoints after the restored one by position, not node
- build the rollout params once per run, so an RLM retry keeps its
  skill home and the restored session loads
- discard snapshots on the shared leased client

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A turn cancelled while its reply waits for the snapshot capture now
still records its call and abandons its preview. The wait is shielded,
so the snapshot finishes in the background, where the rollout awaits
it.

A capture reads the turn's node and tail before its first await, like
the state copies, so everything a checkpoint describes comes from the
idle box.

Docs: show the checkpoint config as a TOML example, and shorten the
harness flag's entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Restore a rollout from a snapshot end to end on prime: per harness, at
the default interval, for rlm with skills, and for each retention
policy. The cases run once per SnapshotType, on every harness and
runtime that declare SUPPORTS_<TYPE>_SNAPSHOTS, so a new snapshot type
gets its cases without a test change. CI runs it as "Checkpoint
Restore".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A box-less checkpoint let a harness that runs no code restore from the
trace alone on a runtime without snapshots. It served restores in
another process, which this PR no longer does; for retries in one run
it only saves the few turns of a short chat rollout. Every checkpoint
now has a snapshot, and the boxless setting is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Record every snapshot the run's boxes take, across attempts, and check
each on the platform once the run is done: deleted, except the ones
the retention policy keeps, which stay restorable. Before, only the
checkpoints left on the final trace were checked, so a rewound
snapshot or one still uploading at the end could leak unnoticed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	verifiers/v1/agent.py
#	verifiers/v1/rollout.py
#	verifiers/v1/runtimes/backends/prime.py
When a rollout is cancelled (an agent timeout, shutdown), its harness
turn is cancelled and then joined. A second cancellation during that
join left the turn running while teardown began, where it could still
write the trace or use the box. Join it shielded, then re-raise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
From review: keep the checkpoint config to what a user decides.

- interval counts from turn n (the n-th, 2n-th, ...), so a rollout
  shorter than n turns takes no snapshot; the turn a restore resumes
  at is not snapshotted again, so keep = "last" no longer deletes the
  snapshot that just restored
- every snapshot is deleted once the run is final: drop `after`
- drop keep = "all", which keep = 1 already expresses
- drop `type` from the config and SnapshotType from the API: the
  runtime takes the best snapshot it supports and records its type on
  the checkpoint (Snapshot.type)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A harness can continue a rollout in a box restored from a snapshot
when a new process of it can take the rollout over. For a harness that
resumes on the transcript (SUPPORTS_RESUME), a restore is that same
relaunch, so it needs nothing more. An ACP harness keeps its session
in a live process and needs its agent to load the session from its
files: ACPHarness.LOADS_SESSIONS (claude-code, codex, rlm). This
replaces the harness-side SUPPORTS_FILESYSTEM_SNAPSHOTS
(Harness.can_restore). browser-use now restores like any harness that
resumes on its transcript.

A restore whose attempt fails before its first new turn is marked
failed like one whose box never booted, so the next retry steps back
to an older snapshot or starts over instead of restoring the same
snapshot until the retries run out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The condition read `target is not None and A or B`, so B ran for runs
that restored nothing and read `restored_turns` on them. Group it as
`target is not None and (A or B)`. The new test's fake rollout also
called the run's `on_trace` hook, which it does not pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mem-ckpt

# Conflicts:
#	verifiers/v1/session.py
…at/mem-ckpt

# Conflicts:
#	verifiers/v1/interception/server.py
A memory snapshot (a pause of the box) holds the box's processes. A
retry boots a new box from it, reroutes the box's relay to the new
session and waits for the harness program the box still runs, so no
harness has to load a session or be told about the restore.

The ACP runner can run as a job of the box with one job per request,
because no connection from the host survives a stopped box.

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
mikasenghaas and others added 2 commits November 10, 2026 22:24
A snapshot at every 10th turn stops every healthy box for a few
seconds. Take the snapshot when it is needed instead: when a model
call fails with an error the run retries, before the harness gets
the error. The box that the retry restores still waits for the
reply, and asks for the call again.

`checkpoint.interval` is now off by default. It stays as an option
for failures that leave no box to snapshot (the box died, the
harness program crashed).

A model call that fails again right after a restore no longer
counts as a restore that led nowhere: the retry restores the same
snapshot.

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
mikasenghaas and others added 7 commits November 11, 2026 21:08
A memory snapshot holds every process of the box, so the sub-agents
of a harness continue with it. Nothing tested that. The new case
saves an rlm rollout while two sub-agents are at work, and checks that
a second owner's box continues both without starting either again.

Co-Authored-By: Claude <noreply@anthropic.com>
The relay starts alongside the harness setup. Its install puts uv in
a box that has none, and the setup of some harnesses (prime-agent,
hermes-agent) does the same in a shell of its own. The two installs
could overlap, and one then ran the other's half-written binary:
"uv: Permission denied" from the relay start, seen once in CI.

Co-Authored-By: Claude <noreply@anthropic.com>
The relay starts alongside the harness setup. Its install puts uv in
a box that has none, and the setup of some harnesses (prime-agent,
hermes-agent) does the same in a shell of its own. The two installs
could overlap, and one then ran the other's half-written binary:
"uv: Permission denied" from the relay start, seen once in CI.

Co-Authored-By: Claude <noreply@anthropic.com>
An owner that saves every running rollout pauses all their boxes at
once. At thousands of rollouts that is a burst of pause and resume
calls the platform may refuse, and a box whose resume fails never
runs its rollout again. snapshots_per_min spreads the pauses out with
the bucket that already paces creates. Off by default.

Co-Authored-By: Claude <noreply@anthropic.com>
A saved rollout was a RestorePoint: the trace plus fields beside it
that only that class carried. Whoever held just the trace could not
continue it. save_all now returns the trace as plain data, with what
a restore needs on the entry of its checkpoint, and a restore takes
that trace.

Snapshots on a failed model call are now opt-in (checkpoint.on_error).
They help only when the harness would end on the error; a harness
that handles the failed call itself never uses the snapshot, and a
run without retries never takes one.

Co-Authored-By: Claude <noreply@anthropic.com>
Main moved blocking commands on Prime to native sessions, so a task
that clears scratch directories cannot delete the job files of a
running command. A session ends with its box, and a box that boots
from a memory snapshot is a new one. The harness's main program
therefore stays a server-owned job, which `attach` can continue to
wait on. Every other command takes main's session path.

Co-Authored-By: Claude <noreply@anthropic.com>
A box that boots from a memory snapshot keeps the sandboxd session of
the harness program, so the host can attach to it by its session id.
The program no longer needs a polled background job, and it gets the
same protection as every other command against a task that clears
the scratch directories.

A snapshot also ends the stream into the box that was paused. The
wait now attaches to the session again when the box runs.

Needs a prime-sandboxes release with attach_process.

Co-Authored-By: Claude <noreply@anthropic.com>
mikasenghaas and others added 12 commits November 13, 2026 01:49
A memory snapshot resets the box's open connections, so the relay
asks again for the call that just failed. The interception server
keeps no failed attempt, so the box gets a new model call and the
harness rarely gets the error at all. The snapshot was a slow way to
retry a model call, and its outcome depended on when the relay's
request arrived.

A retry of a model call belongs at the call. Snapshots stay for a
box or a host that goes away: the owner's save, and `interval`.

A call that the box asks again at a snapshotted turn now takes no
second snapshot, or a call that keeps failing would take one per
request.

Co-Authored-By: Claude <noreply@anthropic.com>
The restore tests continued a rollout from one point and cut the
trace back at most one turn. This one saves a rollout at three
turns and continues each saved trace: with a scripted model, every
continuation must end with the messages of the first run. A counter
that a process holds only in memory shows a command that ran twice
or was skipped.

Co-Authored-By: Claude <noreply@anthropic.com>
A pause that the platform accepted stops the box. When the wait for
the paused state then failed, the box was never resumed and its
snapshot was dropped from the list that teardown frees. One failed
batched status call fails every box that waits in it, so a save of
many rollouts could stop them all for good.

Resume the box whatever the wait came to, keep the snapshot for
teardown, and ask again when a status poll fails.

Co-Authored-By: Claude <noreply@anthropic.com>
The relay took every 404, 502, 503 or 504 that was not JSON or SSE
for the error page of a tunnel edge and retried it for five minutes.
The interception server answers an unknown path with a plain-text
404, so a harness that asked for one waited that long.

The server now marks each of its responses with a header, and only
a response without it counts as the edge's.

Co-Authored-By: Claude <noreply@anthropic.com>
Three cases let a box see a reply that its checkpoint does not
record, or left a rollout stopped:

- A release (the owner's, or the timer) answered the box while
  another snapshot had it stopped. The snapshot now ends the hold
  itself when it is done.
- A save that was cancelled or failed kept the hold with no timer
  and left its snapshot pinned for an owner that never got it.
- A snapshot of a turn that waited for another snapshot recorded
  the trace with that turn's reply, which the box had not got. It
  is now skipped.

A save without a hold no longer ends the hold of an earlier one.

Co-Authored-By: Claude <noreply@anthropic.com>
The harness program now runs in a session and a restored box is
attached to by session id, which needs `attach_process`. Remove the
pin when a release has it.

Co-Authored-By: Claude <noreply@anthropic.com>
The fake backend finished its snapshot at once, so the save was never cancelled.

Co-Authored-By: Claude <noreply@anthropic.com>
# Conflicts:
#	verifiers/v1/interception/server.py
The relay took every 404, 502, 503 or 504 that was not JSON or SSE
for the error page of a tunnel edge and retried it for five minutes.
The interception server answers an unknown path with a plain-text
404, so a harness that asked for one waited that long.

The server now marks each of its responses with a header, and only
a response without it counts as the edge's.

Co-Authored-By: Claude <noreply@anthropic.com>
(cherry picked from commit a1b88d8)
# Conflicts:
#	verifiers/v1/interception/relay/__init__.py
#	verifiers/v1/rollout.py
mikasenghaas and others added 6 commits November 13, 2026 21:21
A checkpoint is a copy of the host's view of the rollout, and the
copy missed things or caught them at the wrong moment:

- A reply was stored for replay when it was sent, not when it was
  committed. One that waited behind a hold was on the trace and
  could not be sent again, so a restored box got a second sample.
  A turn now stores its reply inside its commit, and a snapshot
  records the trace only outside that window.
- Tool verdicts and the prepared user messages were not copied. A
  call that a hook denied ran in the restored box, and request
  hooks ran twice on one message.
- A hold stopped only successful replies. Errors and "rollout
  stopped" now wait too, and a segment that ends under an owner's
  hold waits for the release, so a saved rollout does not finish
  while its owner records it as running.
- A save whose harness program ended meanwhile is skipped.

Smaller fixes with the same root:

- The pace of snapshots is waited for before the box is held.
- The interval counts turns from the newest snapshot, so
  conversations that run side by side cannot step past it.
- A restored rollout gets the agent time that is left, not the
  full timeout again.
- A rewind drops the rewrite records of the turns it cuts.
- A retry may pick the snapshot its owner offered, and no other
  snapshot of an owner.

Co-Authored-By: Claude <noreply@anthropic.com>
Main's lockfile changed format; the only difference to it is the prime-sandboxes pin.

Co-Authored-By: Claude <noreply@anthropic.com>
The harness launch passes the session's program to `Runtime.run`. The fake runtime of this new test on main did not accept it.

Co-Authored-By: Claude <noreply@anthropic.com>
Move the prime-sandboxes pin to the attach-process branch merged
with prime main (0.5.12), and relock from main's lockfile.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main moved the episode agent's override from run() to _run(), which
dropped the restore argument, and gave RolloutSession a route.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant