Repository navigation
feat: restore rollout retries from filesystem snapshots - #3073
mikasenghaas wants to merge 13 commits into
Conversation
ApprovabilityVerdict: Needs Human Approval Macroscope's review found this PR requires human approval — Supported Prime rollouts now create billed filesystem snapshots by default and resume retries from them. Human judgment is needed on that default and the unanswered concern that a transient failure immediately after restore can discard valid checkpoint progress. You can add or adjust custom eligibility rules. Learn more. |
An agent snapshots its prime box as model turns go upstream, and a rollout retry restores the newest snapshot in a new box and continues the trace from there instead of starting over. On by default (`[env.agent.checkpoint]`: interval, keep, after, boxless). The trace records each snapshot (`Trace.checkpoints`), rewinds to the one a retry restores (`Trace.rewind`, `Trace.generation`), and gives another process what it needs to continue it (`Trace.restorable`, `RestorePoint`, `Env.run_slot(restore=...)`). null and bash relaunch on the transcript; claude-code, codex and rlm load their ACP session from the restored box (rlm pinned to nano-rlm with session/load). A request hook that raises now ends the rollout with its error. This is the filesystem-only state of verifiers#2994 as of ebfd9f8, on today's main. Its env-server parts moved with the env server to prime-rl and are not part of this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Scope rollout checkpointing to retries within one Agent.run: drop the cross-process restore (Env and Agent.run restore params, RestorePoint, PinnedResult, Trace.restorable, Agents.only). Call the runtime side a snapshot: backends return a typed Snapshot (id, type) from snapshot(), wait_snapshot() and discard_snapshots() take it, and restore_config() builds the config a new box boots from. Runtimes and harnesses declare SUPPORTS_FILESYSTEM_SNAPSHOTS, and RolloutCheckpointConfig.type takes a SnapshotType, so memory snapshots can slot in behind the same API. Also from review: - snapshot every 10th turn by default - copy state before the first await of a capture, so response hooks that run meanwhile do not leak into the checkpoint - trim the checkpoints after the restored one by position, not node - build the rollout params once per run, so an RLM retry keeps its skill home and the restored session loads - discard snapshots on the shared leased client Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
022d5ed to
824ca30
Compare
A turn cancelled while its reply waits for the snapshot capture now still records its call and abandons its preview. The wait is shielded, so the snapshot finishes in the background, where the rollout awaits it. A capture reads the turn's node and tail before its first await, like the state copies, so everything a checkpoint describes comes from the idle box. Docs: show the checkpoint config as a TOML example, and shorten the harness flag's entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Restore a rollout from a snapshot end to end on prime: per harness, at the default interval, for rlm with skills, and for each retention policy. The cases run once per SnapshotType, on every harness and runtime that declare SUPPORTS_<TYPE>_SNAPSHOTS, so a new snapshot type gets its cases without a test change. CI runs it as "Checkpoint Restore". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A box-less checkpoint let a harness that runs no code restore from the trace alone on a runtime without snapshots. It served restores in another process, which this PR no longer does; for retries in one run it only saves the few turns of a short chat rollout. Every checkpoint now has a snapshot, and the boxless setting is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Record every snapshot the run's boxes take, across attempts, and check each on the platform once the run is done: deleted, except the ones the retention policy keeps, which stay restorable. Before, only the checkpoints left on the final trace were checked, so a rewound snapshot or one still uploading at the end could leak unnoticed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # verifiers/v1/agent.py # verifiers/v1/rollout.py # verifiers/v1/runtimes/backends/prime.py
When a rollout is cancelled (an agent timeout, shutdown), its harness turn is cancelled and then joined. A second cancellation during that join left the turn running while teardown began, where it could still write the trace or use the box. Join it shielded, then re-raise. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| - `APPENDS_SYSTEM_PROMPT`: passes `TaskData.system_prompt` as a separate system message; when `False`, `resolve_prompt` prepends it to the first user message instead. | ||
| - `SUPPORTS_MCP`: indicates that `launch` wires `mcp_urls` into the program. | ||
| - `SUPPORTS_RESUME`: indicates that a multi-turn exchange can restart from the conversation so far (programs with native session state should override `resume` or `session` instead). Note that live mid-turn steering requires a `HarnessSession` that implements `steer`; conversation resume support alone is not enough (see [message delivery](agent.md#messages-during-an-active-turn)). | ||
| - `SUPPORTS_FILESYSTEM_SNAPSHOTS`: indicates that a new process of the harness can continue a rollout in a box restored from a filesystem snapshot (`agent.checkpoint`). Keep per-rollout state in the box (for example under `/tmp/vf-<harness>-runs/<trace id>`), so the snapshot holds it. |
There was a problem hiding this comment.
why is this not the same as SUPPORTS_RESUME
There was a problem hiding this comment.
i went back and forth a bit with my clanker, latest understanding is that resume is for acp resume which assumes a live acp process in the harness while fs snapshot survives process failure (resume from session transcript on disk)
There was a problem hiding this comment.
but will double check
From review: keep the checkpoint config to what a user decides. - interval counts from turn n (the n-th, 2n-th, ...), so a rollout shorter than n turns takes no snapshot; the turn a restore resumes at is not snapshotted again, so keep = "last" no longer deletes the snapshot that just restored - every snapshot is deleted once the run is final: drop `after` - drop keep = "all", which keep = 1 already expresses - drop `type` from the config and SnapshotType from the API: the runtime takes the best snapshot it supports and records its type on the checkpoint (Snapshot.type) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A harness can continue a rollout in a box restored from a snapshot when a new process of it can take the rollout over. For a harness that resumes on the transcript (SUPPORTS_RESUME), a restore is that same relaunch, so it needs nothing more. An ACP harness keeps its session in a live process and needs its agent to load the session from its files: ACPHarness.LOADS_SESSIONS (claude-code, codex, rlm). This replaces the harness-side SUPPORTS_FILESYSTEM_SNAPSHOTS (Harness.can_restore). browser-use now restores like any harness that resumes on its transcript. A restore whose attempt fails before its first new turn is marked failed like one whose box never booted, so the next retry steps back to an older snapshot or starts over instead of restoring the same snapshot until the retries run out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 8b20e18. Configure here.
| # The restore led nowhere: the box never booted from it, or the | ||
| # attempt failed before its first new turn (a harness that | ||
| # cannot load its session fails the same way every time). The | ||
| # next retry steps back to an older snapshot, or starts over. |
There was a problem hiding this comment.
Restore skips snapshot after first-turn failure
High Severity
Any restored attempt that fails before a new sampled turn is recorded as a broken restore. That also matches a retryable ProviderError on the first restored model call, so the next retry blacklists a valid snapshot. With keep = "last", older snapshots are already gone, and a flaky first restored turn discards the rollout's progress and starts over.
Reviewed by Cursor Bugbot for commit 8b20e18. Configure here.
The condition read `target is not None and A or B`, so B ran for runs that restored nothing and read `restored_turns` on them. Group it as `target is not None and (A or B)`. The new test's fake rollout also called the run's `on_trace` hook, which it does not pass. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Superseded by #3141, which keeps the checkpoint plumbing of this PR (checkpoints on the trace, |


Summary
[env.agent.checkpoint]).Agent.run. Restoring in another process (an orchestrator restart), memory snapshots and whole-episode checkpoints are follow-ups.Harness.can_restore). One that resumes on the transcript (SUPPORTS_RESUME: null, bash, browser-use) relaunches on it, as a resume does; an ACP harness whose pinned agent implementssession/load(ACPHarness.LOADS_SESSIONS: claude-code, codex, rlm) loads its session from the restored box (rlm needs feat: load a session from its ledger over ACP nano-rlm#238).Snapshot(id, type); each checkpoint records it. Memory snapshots can replace it without a config change.docs/agent.md(Retries and checkpoints),docs/harnesses.md(LOADS_SESSIONS).Motivation
An agentic rollout can run for hours. Today, when it fails late (a provider error, a dead sandbox), a retry sets the task up again and samples every turn again. From scratch, an attempt finishes only if all N turns get through, so the cost grows exponentially with N; a restore loses only the turns since the last snapshot.
terminal-bench-2
fix-git(about 12 turns), 64 rollouts per run, a snapshot every turn, with an injected provider error on each model turn with probability p (measured on the original branch; the evals are linked in #2994):How it works
When. A snapshot is taken as every
interval-th model turn goes upstream (the n-th, the 2n-th, ...; a rollout shorter than n turns takes none), while the harness waits for the reply, so the box matches the prompt. It runs alongside the model call; the reply waits only until the snapshot's content is fixed (about 1 s), and durability is awaited in the background. It needs a runtime that takes snapshots (SUPPORTS_FILESYSTEM_SNAPSHOTS) and a harness that can restore; elsewhere the agent warns once and a retry starts over. No snapshot is taken while another model request of the trace is in flight.What is stored. A Prime snapshot of the box's root filesystem (incremental, billed by stored size), which also holds the harness's session files (
/tmp/vf-<harness>-runs/<trace id>). The trace records oneCheckpointper snapshot: itssnapshot(id and type), thenodeand uncommittedtailit follows, the harness state (e.g.acp_session_id), timing (captured,held,time),bytes, andrestored/deletedflags.Restore. Turns 1 to 3 succeed, turn 4 fails:
Agent.runpicks the newest restorable snapshot (durable, not deleted, not one whose restore already failed). A restore fails when its box does not boot, or when its attempt fails before its first new turn; the next retry then steps back to an older snapshot, or starts over.Trace.rewinddrops the later nodes and resets the typed state andinfo.RuntimeBackend.restore_config). Tasksetupis skipped, and the turn it resumes at is not snapshotted again.HarnessSession.restore) and getsRESTORE_PROMPT: the box was restored, and processes and in-memory state are gone.Deletion. While the rollout runs,
keepdeletes older snapshots once a newer one is durable. When the run is final, every snapshot left is deleted. An aborted attempt deletes its own, and a snapshot that never became durable is deleted with its box.Runtime and harness API. A runtime implements
snapshot(),wait_snapshot(snapshot),discard_snapshots(snapshots)andrestore_config(config, snapshot); only prime does today. A harness implementsHarnessSession.checkpoint_state()andrestore(state, tail); the default relaunches on the transcript, and ACP harnesses usesession/load.Known limitations
Breaking
--no-env.agent.checkpoint, or a largerinterval.@vf.intercepton aRequest) that raises now ends the rollout at once with that error; the harness's turn is cancelled. Before, the harness got an HTTP error (Claude Code retried it until the agent deadline). Migration: a hook that should not end the rollout must not raise; return a rewritten request or use a@stop.Verification
E2E on prime (
tests/integration/test_checkpoint.py, CI job "Checkpoint Restore"): 9 cases.keep = "last",keep = 2).attempt == 2, the earlier turns kept, one error, the restore notice, reward 1.0 (the earlier tool call's file is there,setupdid not rerun).Load tests through
uv run evalwith this PR pinned in prime-rl (PrimeIntellect-ai/prime-rl#4416), terminal-bench-2, bash harness, prime. They ran before the interval counted from turn n, so their snapshots started at turn 1.fix-git× 1000, 1000 concurrentFollow-ups
SnapshotTypebehind the same API, so processes, sub-agents and REPL state survive a restore.prime-rl
PrimeIntellect-ai/prime-rl#4416 pins this PR. A restoring retry bumps the trace's
attempt, so prime-rl's env server already resends a restored trace from scratch; #4416 adds streaming of the newcheckpointsfield.🤖 Generated with Claude Code
Note
High Risk
Changes default agent retry behavior on prime (snapshot cost/timing) and alters how request-hook failures end rollouts, touching core rollout, trace rewind, and ACP session restore paths.
Overview
Adds default-on rollout checkpointing: supported prime rollouts snapshot the box at configurable model-turn intervals and, on retryable failure,
Agent.runrewinds the trace and boots a new box from the newest durable checkpoint instead of rerunning task setup from scratch. This introducesRolloutCheckpointConfig, traceCheckpointrecords,Trace.rewind, capture/prune during turns, and aSnapshotruntime API implemented on Prime.Harnesses opt in via
Harness.can_restore()(transcript resume or ACPLOADS_SESSIONSwithsession/load); Claude Code, Codex, and RLM now keep per-run state under/tmpso snapshots include session files. Request@intercepthooks that raise now abort the rollout immediately (canceling the harness turn) rather than surfacing only an HTTP error to the program.Docs, CI Checkpoint Restore integration tests, and broad unit coverage exercise restore, retention, failed-restore fallback, and snapshot leak cleanup.
Reviewed by Cursor Bugbot for commit 014d638. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Restore agent rollout retries from filesystem snapshots
Rollout._capturein rollout.py).Agent.runretries now rewind to the latest durable checkpoint instead of restarting from the beginning, keeping one trace ID across attempts. Retries select the newest completed, non-deleted checkpoint not marked as a failed restore (agent.py).Trace.rewindrolls back the graph, restores checkpoint state and info, resets timing and run-result fields, and drops later checkpoints (trace.py).Snapshotmodel and backend snapshot contract in base.py, implemented byPrimeBackendwith a 300-second durability wait (prime.py). ACP sessions can save and reload an agent session ID on restore (runner.py).RolloutCheckpointConfig: filesystem, interval 10, keeplast, delete after run) but disabled with one warning per harness/runtime pair when either lacksSUPPORTS_FILESYSTEM_SNAPSHOTS(agent.py)./tmproots (/tmp/vf-rlm-runs,/tmp/vf-claude-code-runs,/tmp/vf-codex-runs), andInterceptionServerrequest-rewrite failures now abort the session via_abortinstead of returning an HTTP error.Changes since #3073 opened
SnapshotTypeabstraction from configs, runtimes and public exports [3a581f8]SUPPORTS_FILESYSTEM_SNAPSHOTSharness flag with acan_restore()classmethod and aLOADS_SESSIONSflag [8b20e18]Agentcheckpointing and retry handling to gate oncan_restore()and to skip restore snapshots that lead nowhere [8b20e18]can_restore()gating andrestored_turnstracking [8b20e18]target is not Noneguard inverifiers.v1.agent.Agent.run. [014d638]on_tracecallback invocation fromtests.unit.test_retry_ids.FakeRollout.__init__. [014d638]Macroscope summarized d77caf0.