Skip to content

feat: restore rollout retries from filesystem snapshots - #3073

Closed
mikasenghaas wants to merge 13 commits into
mainfrom
feat/fs-rollout-ckpt
Closed

mikasenghaas wants to merge 13 commits into
mainfrom
feat/fs-rollout-ckpt

Conversation

@mikasenghaas

@mikasenghaas mikasenghaas commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

Summary

  • A rollout on prime snapshots its box's filesystem at model turns (every 10th by default). When the rollout fails and its retry rules allow a retry, the retry restores the newest snapshot in a new box and continues the trace from there, instead of starting over. On by default ([env.agent.checkpoint]).
  • Scope: retries within one Agent.run. Restoring in another process (an orchestrator restart), memory snapshots and whole-episode checkpoints are follow-ups.
  • Harnesses: a harness can restore when a new process of it can take the rollout over (Harness.can_restore). One that resumes on the transcript (SUPPORTS_RESUME: null, bash, browser-use) relaunches on it, as a resume does; an ACP harness whose pinned agent implements session/load (ACPHarness.LOADS_SESSIONS: claude-code, codex, rlm) loads its session from the restored box (rlm needs feat: load a session from its ledger over ACP nano-rlm#238).
  • The runtime takes the best snapshot it supports (a filesystem snapshot on prime today) and returns a Snapshot(id, type); each checkpoint records it. Memory snapshots can replace it without a config change.
  • A request hook that raises now ends the rollout with its error (see Breaking).
  • Docs: docs/agent.md (Retries and checkpoints), docs/harnesses.md (LOADS_SESSIONS).

Motivation

An agentic rollout can run for hours. Today, when it fails late (a provider error, a dead sandbox), a retry sets the task up again and samples every turn again. From scratch, an attempt finishes only if all N turns get through, so the cost grows exponentially with N; a restore loses only the turns since the last snapshot.

terminal-bench-2 fix-git (about 12 turns), 64 rollouts per run, a snapshot every turn, with an injected provider error on each model turn with probability p (measured on the original branch; the evals are linked in #2994):

p Checkpoints Reward Rollouts ok Sampled turns / rollout
0.1 off / on 0.95 / 1.00 62 / 64 24.3 / 14.8
0.2 off / on 0.48 / 0.88 31 / 56 26.3 / 16.3

How it works

When. A snapshot is taken as every interval-th model turn goes upstream (the n-th, the 2n-th, ...; a rollout shorter than n turns takes none), while the harness waits for the reply, so the box matches the prompt. It runs alongside the model call; the reply waits only until the snapshot's content is fixed (about 1 s), and durability is awaited in the background. It needs a runtime that takes snapshots (SUPPORTS_FILESYSTEM_SNAPSHOTS) and a harness that can restore; elsewhere the agent warns once and a retry starts over. No snapshot is taken while another model request of the trace is in flight.

[env.agent.checkpoint]   # the defaults; --no-env.agent.checkpoint turns it off
interval = 10            # snapshot at the n-th, 2n-th, ... model turn
keep = "last"            # while the rollout runs: "last" | k (every k-th, plus the newest)

What is stored. A Prime snapshot of the box's root filesystem (incremental, billed by stored size), which also holds the harness's session files (/tmp/vf-<harness>-runs/<trace id>). The trace records one Checkpoint per snapshot: its snapshot (id and type), the node and uncommitted tail it follows, the harness state (e.g. acp_session_id), timing (captured, held, time), bytes, and restored/deleted flags.

Restore. Turns 1 to 3 succeed, turn 4 fails:

attempt 1   sys ─ user ─ A1 ─ tool1 ─ A2 ─ tool2 ─ A3      ✗ turn 4 fails
            C1 (turn 1)   C2 (turn 2)   C3 (turn 3)       snapshots

rewind      sys ─ user ─ A1 ─ tool1 ─ A2                   kept, still sampled
            tool2 ─ A3 dropped (A3's call stays on the trace, unlinked)

attempt 2   new box from C3, new harness process:
            ... A2 ─ tool2 ─ RESTORE_PROMPT ─ A3' ─ ...
  1. Agent.run picks the newest restorable snapshot (durable, not deleted, not one whose restore already failed). A restore fails when its box does not boot, or when its attempt fails before its first new turn; the next retry then steps back to an older snapshot, or starts over.
  2. Trace.rewind drops the later nodes and resets the typed state and info.
  3. A new box boots from the snapshot (RuntimeBackend.restore_config). Task setup is skipped, and the turn it resumes at is not snapshotted again.
  4. A new harness process continues the conversation (HarnessSession.restore) and gets RESTORE_PROMPT: the box was restored, and processes and in-memory state are gone.

Deletion. While the rollout runs, keep deletes older snapshots once a newer one is durable. When the run is final, every snapshot left is deleted. An aborted attempt deletes its own, and a snapshot that never became durable is deleted with its box.

Runtime and harness API. A runtime implements snapshot(), wait_snapshot(snapshot), discard_snapshots(snapshots) and restore_config(config, snapshot); only prime does today. A harness implements HarnessSession.checkpoint_state() and restore(state, tail); the default relaunches on the transcript, and ACP harnesses use session/load.

Known limitations

  • A snapshot holds the filesystem only: processes, sub-agents and REPL state are gone after a restore. An ACP agent continues from what its session files held, so a turn can be lost, and a harness that renders its loaded history differently forks the graph.
  • With concurrent sub-agents, one that is mid-tool can be snapshotted mid-write.
  • Prime only; single-target deployments (no Compose services, no tool-server VMs).

Breaking

  • Rollout checkpointing is on by default for prime with null, bash, browser-use, rlm, codex and claude-code: a snapshot every 10th turn, each snapshotted reply waits for its capture (about 0.2 to 0.4 s per rollout at the median), and snapshots are billed while they exist. Migration: --no-env.agent.checkpoint, or a larger interval.
  • A request hook (@vf.intercept on a Request) that raises now ends the rollout at once with that error; the harness's turn is cancelled. Before, the harness got an HTTP error (Claude Code retried it until the agent deadline). Migration: a hook that should not end the rollout must not raise; return a rewritten request or use a @stop.

Verification

E2E on prime (tests/integration/test_checkpoint.py, CI job "Checkpoint Restore"): 9 cases.

  • Restore per harness after a failure at turn 3 (null, bash, rlm, codex, claude-code), rlm with skills, and two retention policies (keep = "last", keep = 2).
  • With the default interval, a rollout that fails at turn 3 takes no snapshot and its retry starts over.
  • Each restore checks: a new box from the newest snapshot, attempt == 2, the earlier turns kept, one error, the restore notice, reward 1.0 (the earlier tool call's file is there, setup did not rerun).
  • Every case checks that the run leaks no snapshot: each snapshot its boxes took is deleted on the platform once the run is final.

Load tests through uv run eval with this PR pinned in prime-rl (PrimeIntellect-ai/prime-rl#4416), terminal-bench-2, bash harness, prime. They ran before the interval counted from turn n, so their snapshots started at turn 1.

89 tasks × 1, 64 concurrent fix-git × 1000, 1000 concurrent
Snapshot interval 10 2
Rollouts ok 78 / 89 (base without checkpoints: 74 / 88) 1000 / 1000, no retries
Rollout time vs. base 399 vs. 395 s p50 –
Reply held for captures, per rollout p50 0.40 s, p90 1.56 s p50 0.21 s, p90 2.91 s
Snapshots / restores 409 / 8 6,037 / 0
429s or rate-limit errors none none
Snapshot leaks none (97 sandboxes) none (1,000 sandboxes)
  • The 8 restores worked mechanically; the 7 rollouts behind them failed again for reasons outside checkpointing (a provider outage, and a task that runs its box out of memory in scoring on main too).
  • At 1,000 concurrent, captures slow down at the tail (max 16.6 s) but do not fail.

Follow-ups

  • Memory snapshots: a new SnapshotType behind the same API, so processes, sub-agents and REPL state survive a restore.
  • Restore in another process: continue in-flight rollouts after an orchestrator restart. It needs snapshots that outlive the run for their owner, and box-less checkpoints for harnesses that run no code.
  • Episode checkpointing: a consistent cut across a multi-agent episode's seats.

prime-rl

PrimeIntellect-ai/prime-rl#4416 pins this PR. A restoring retry bumps the trace's attempt, so prime-rl's env server already resends a restored trace from scratch; #4416 adds streaming of the new checkpoints field.

🤖 Generated with Claude Code


Note

High Risk
Changes default agent retry behavior on prime (snapshot cost/timing) and alters how request-hook failures end rollouts, touching core rollout, trace rewind, and ACP session restore paths.

Overview
Adds default-on rollout checkpointing: supported prime rollouts snapshot the box at configurable model-turn intervals and, on retryable failure, Agent.run rewinds the trace and boots a new box from the newest durable checkpoint instead of rerunning task setup from scratch. This introduces RolloutCheckpointConfig, trace Checkpoint records, Trace.rewind, capture/prune during turns, and a Snapshot runtime API implemented on Prime.

Harnesses opt in via Harness.can_restore() (transcript resume or ACP LOADS_SESSIONS with session/load); Claude Code, Codex, and RLM now keep per-run state under /tmp so snapshots include session files. Request @intercept hooks that raise now abort the rollout immediately (canceling the harness turn) rather than surfacing only an HTTP error to the program.

Docs, CI Checkpoint Restore integration tests, and broad unit coverage exercise restore, retention, failed-restore fallback, and snapshot leak cleanup.

Reviewed by Cursor Bugbot for commit 014d638. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Restore agent rollout retries from filesystem snapshots

  • Adds checkpointing to rollouts: eligible turns capture a backend filesystem snapshot plus trace, message, tool-result, and harness state (Rollout._capture in rollout.py).
  • Agent.run retries now rewind to the latest durable checkpoint instead of restarting from the beginning, keeping one trace ID across attempts. Retries select the newest completed, non-deleted checkpoint not marked as a failed restore (agent.py).
  • Trace.rewind rolls back the graph, restores checkpoint state and info, resets timing and run-result fields, and drops later checkpoints (trace.py).
  • New Snapshot model and backend snapshot contract in base.py, implemented by PrimeBackend with a 300-second durability wait (prime.py). ACP sessions can save and reload an agent session ID on restore (runner.py).
  • Checkpointing is on by default (RolloutCheckpointConfig: filesystem, interval 10, keep last, delete after run) but disabled with one warning per harness/runtime pair when either lacks SUPPORTS_FILESYSTEM_SNAPSHOTS (agent.py).
  • Behavioral Change: checkpointing defaults to enabled, so runs with supported backends now create and delete snapshots; RLM, Claude Code, and Codex per-trace state moved to /tmp roots (/tmp/vf-rlm-runs, /tmp/vf-claude-code-runs, /tmp/vf-codex-runs), and InterceptionServer request-rewrite failures now abort the session via _abort instead of returning an HTTP error.

Changes since #3073 opened

  • Changed checkpoint retention and configuration semantics so all snapshots are deleted when a run is final and the restored turn is tracked [3a581f8]
  • Removed the SnapshotType abstraction from configs, runtimes and public exports [3a581f8]
  • Updated unit tests, integration tests, test grid and docs for the new checkpoint behavior [3a581f8]
  • Replaced the SUPPORTS_FILESYSTEM_SNAPSHOTS harness flag with a can_restore() classmethod and a LOADS_SESSIONS flag [8b20e18]
  • Changed Agent checkpointing and retry handling to gate on can_restore() and to skip restore snapshots that lead nowhere [8b20e18]
  • Updated documentation and config descriptions for restore-capable harnesses and checkpointing [8b20e18]
  • Updated integration and unit tests for the can_restore() gating and restored_turns tracking [8b20e18]
  • Grouped the setup-error check and the no-new-turn check in the 'restore led nowhere' conditional under a single target is not None guard in verifiers.v1.agent.Agent.run. [014d638]
  • Removed the on_trace callback invocation from tests.unit.test_retry_ids.FakeRollout.__init__. [014d638]

Macroscope summarized d77caf0.

Comment thread verifiers/v1/rollout.py Resolved
Comment thread verifiers/v1/trace.py Resolved
Comment thread verifiers/v1/harnesses/rlm/harness.py Resolved

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread verifiers/v1/harnesses/rlm/harness.py Resolved
@macroscopeapp

macroscopeapp Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Needs Human Approval

Macroscope's review found this PR requires human approval — Supported Prime rollouts now create billed filesystem snapshots by default and resume retries from them. Human judgment is needed on that default and the unanswered concern that a transient failure immediately after restore can discard valid checkpoint progress.

You can add or adjust custom eligibility rules. Learn more.

An agent snapshots its prime box as model turns go upstream, and a
rollout retry restores the newest snapshot in a new box and continues
the trace from there instead of starting over. On by default
(`[env.agent.checkpoint]`: interval, keep, after, boxless).

The trace records each snapshot (`Trace.checkpoints`), rewinds to the
one a retry restores (`Trace.rewind`, `Trace.generation`), and gives
another process what it needs to continue it (`Trace.restorable`,
`RestorePoint`, `Env.run_slot(restore=...)`). null and bash relaunch on
the transcript; claude-code, codex and rlm load their ACP session from
the restored box (rlm pinned to nano-rlm with session/load).

A request hook that raises now ends the rollout with its error.

This is the filesystem-only state of verifiers#2994 as of ebfd9f8,
on today's main. Its env-server parts moved with the env server to
prime-rl and are not part of this change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread verifiers/v1/configs/agent.py Resolved
Comment thread verifiers/v1/configs/agent.py Resolved
Comment thread verifiers/v1/runtimes/backends/prime.py Resolved
Comment thread verifiers/v1/runtimes/base.py Resolved
Scope rollout checkpointing to retries within one Agent.run: drop the
cross-process restore (Env and Agent.run restore params, RestorePoint,
PinnedResult, Trace.restorable, Agents.only).

Call the runtime side a snapshot: backends return a typed Snapshot
(id, type) from snapshot(), wait_snapshot() and discard_snapshots()
take it, and restore_config() builds the config a new box boots from.
Runtimes and harnesses declare SUPPORTS_FILESYSTEM_SNAPSHOTS, and
RolloutCheckpointConfig.type takes a SnapshotType, so memory snapshots
can slot in behind the same API.

Also from review:
- snapshot every 10th turn by default
- copy state before the first await of a capture, so response hooks
  that run meanwhile do not leak into the checkpoint
- trim the checkpoints after the restored one by position, not node
- build the rollout params once per run, so an RLM retry keeps its
  skill home and the restored session loads
- discard snapshots on the shared leased client

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mikasenghaas mikasenghaas changed the title feat: restore rollouts from filesystem checkpoints feat: restore rollout retries from filesystem snapshots Nov 6, 2026

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread verifiers/v1/rollout.py Resolved
Comment thread verifiers/v1/interception/server.py Resolved
A turn cancelled while its reply waits for the snapshot capture now
still records its call and abandons its preview. The wait is shielded,
so the snapshot finishes in the background, where the rollout awaits
it.

A capture reads the turn's node and tail before its first await, like
the state copies, so everything a checkpoint describes comes from the
idle box.

Docs: show the checkpoint config as a TOML example, and shorten the
harness flag's entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread docs/agent.md Resolved
Comment thread docs/harnesses.md Resolved
mikasenghaas and others added 5 commits November 7, 2026 00:06
Restore a rollout from a snapshot end to end on prime: per harness, at
the default interval, for rlm with skills, and for each retention
policy. The cases run once per SnapshotType, on every harness and
runtime that declare SUPPORTS_<TYPE>_SNAPSHOTS, so a new snapshot type
gets its cases without a test change. CI runs it as "Checkpoint
Restore".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A box-less checkpoint let a harness that runs no code restore from the
trace alone on a runtime without snapshots. It served restores in
another process, which this PR no longer does; for retries in one run
it only saves the few turns of a short chat rollout. Every checkpoint
now has a snapshot, and the boxless setting is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Record every snapshot the run's boxes take, across attempts, and check
each on the platform once the run is done: deleted, except the ones
the retention policy keeps, which stay restorable. Before, only the
checkpoints left on the final trace were checked, so a rewound
snapshot or one still uploading at the end could leak unnoticed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread verifiers/v1/agent.py Resolved
# Conflicts:
#	verifiers/v1/agent.py
#	verifiers/v1/rollout.py
#	verifiers/v1/runtimes/backends/prime.py

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread verifiers/v1/rollout.py Resolved
When a rollout is cancelled (an agent timeout, shutdown), its harness
turn is cancelled and then joined. A second cancellation during that
join left the turn running while teardown began, where it could still
write the trace or use the box. Join it shielded, then re-raise.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread docs/agent.md Resolved
Comment thread docs/agent.md Resolved
Comment thread docs/harnesses.md Outdated
- `APPENDS_SYSTEM_PROMPT`: passes `TaskData.system_prompt` as a separate system message; when `False`, `resolve_prompt` prepends it to the first user message instead.
- `SUPPORTS_MCP`: indicates that `launch` wires `mcp_urls` into the program.
- `SUPPORTS_RESUME`: indicates that a multi-turn exchange can restart from the conversation so far (programs with native session state should override `resume` or `session` instead). Note that live mid-turn steering requires a `HarnessSession` that implements `steer`; conversation resume support alone is not enough (see [message delivery](agent.md#messages-during-an-active-turn)).
- `SUPPORTS_FILESYSTEM_SNAPSHOTS`: indicates that a new process of the harness can continue a rollout in a box restored from a filesystem snapshot (`agent.checkpoint`). Keep per-rollout state in the box (for example under `/tmp/vf-<harness>-runs/<trace id>`), so the snapshot holds it.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this not the same as SUPPORTS_RESUME

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i went back and forth a bit with my clanker, latest understanding is that resume is for acp resume which assumes a live acp process in the harness while fs snapshot survives process failure (resume from session transcript on disk)

@mikasenghaas mikasenghaas Nov 9, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but will double check

Comment thread verifiers/v1/configs/runtime.py Resolved
mikasenghaas and others added 2 commits November 9, 2026 23:39
From review: keep the checkpoint config to what a user decides.

- interval counts from turn n (the n-th, 2n-th, ...), so a rollout
  shorter than n turns takes no snapshot; the turn a restore resumes
  at is not snapshotted again, so keep = "last" no longer deletes the
  snapshot that just restored
- every snapshot is deleted once the run is final: drop `after`
- drop keep = "all", which keep = 1 already expresses
- drop `type` from the config and SnapshotType from the API: the
  runtime takes the best snapshot it supports and records its type on
  the checkpoint (Snapshot.type)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A harness can continue a rollout in a box restored from a snapshot
when a new process of it can take the rollout over. For a harness that
resumes on the transcript (SUPPORTS_RESUME), a restore is that same
relaunch, so it needs nothing more. An ACP harness keeps its session
in a live process and needs its agent to load the session from its
files: ACPHarness.LOADS_SESSIONS (claude-code, codex, rlm). This
replaces the harness-side SUPPORTS_FILESYSTEM_SNAPSHOTS
(Harness.can_restore). browser-use now restores like any harness that
resumes on its transcript.

A restore whose attempt fails before its first new turn is marked
failed like one whose box never booted, so the next retry steps back
to an older snapshot or starts over instead of restoring the same
snapshot until the retries run out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 8b20e18. Configure here.

Comment thread verifiers/v1/agent.py
# The restore led nowhere: the box never booted from it, or the
# attempt failed before its first new turn (a harness that
# cannot load its session fails the same way every time). The
# next retry steps back to an older snapshot, or starts over.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Restore skips snapshot after first-turn failure

High Severity

Any restored attempt that fails before a new sampled turn is recorded as a broken restore. That also matches a retryable ProviderError on the first restored model call, so the next retry blacklists a valid snapshot. With keep = "last", older snapshots are already gone, and a flaky first restored turn discards the rollout's progress and starts over.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 8b20e18. Configure here.

Comment thread verifiers/v1/agent.py Resolved
The condition read `target is not None and A or B`, so B ran for runs
that restored nothing and read `restored_turns` on them. Group it as
`target is not None and (A or B)`. The new test's fake rollout also
called the run's `on_trace` hook, which it does not pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mikasenghaas

Copy link
Copy Markdown
Member Author

Superseded by #3141, which keeps the checkpoint plumbing of this PR (checkpoints on the trace, Trace.rewind, retries from a checkpoint) and replaces the filesystem snapshots with memory snapshots.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants