Repository navigation
feat: rollout checkpointing - #2994
mikasenghaas wants to merge 5 commits into
Conversation
ApprovabilityVerdict: Needs Human Approval Macroscope's review found this PR requires human approval — Adds default filesystem recovery and opt-in memory checkpointing for agent rollouts. Unanswered review concerns about relay secrets in persisted traces and checkpoint/resource retention require resolution before rollout. Needs human approval because:
Adjust the Minimum Blocking Severity for this repo — including turning it Off — in Settings. You can add or adjust custom eligibility rules. Learn more. |
| if capture is not None: | ||
| held = time.monotonic() | ||
| try: | ||
| checkpoint = await capture |
There was a problem hiding this comment.
🟠 High interception/server.py:891
A timed-out rollout can finish checkpoint settlement and stop the runtime while session.checkpointer(turn) is still capturing a snapshot; that late capture can then append a checkpoint and start durability work after settlement, losing the latest recovery point. The capture task is only awaited in this handler’s finally, while _settle_checkpoints() gathers only durability tasks already created. Track and drain capture tasks before checkpoint settlement and runtime teardown, or cancel and drain them during release.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/interception/server.py around line 891:
A timed-out rollout can finish checkpoint settlement and stop the runtime while `session.checkpointer(turn)` is still capturing a snapshot; that late capture can then append a checkpoint and start durability work after settlement, losing the latest recovery point. The capture task is only awaited in this handler’s `finally`, while `_settle_checkpoints()` gathers only durability tasks already created. Track and drain capture tasks before checkpoint settlement and runtime teardown, or cancel and drain them during release.
| snapshots.update(checkpoint.id for checkpoint in trace.checkpoints) | ||
| deleted.update(run.deleted_checkpoints) | ||
| if restore is not None and any( | ||
| error.stage == "setup" for error in trace.errors[seen:] |
There was a problem hiding this comment.
🟠 High v1/agent.py:730
A deployment-provisioning timeout while restoring a checkpoint is not marked as a failed restore, so subsequent retries can select the same checkpoint again and exhaust the retry budget despite an older usable snapshot. Rollout.open() records this timeout without stage == "setup", so include provisioning-timeout failures in this check.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/agent.py around line 730:
A deployment-provisioning timeout while restoring a checkpoint is not marked as a failed restore, so subsequent retries can select the same checkpoint again and exhaust the retry budget despite an older usable snapshot. `Rollout.open()` records this timeout without `stage == "setup"`, so include provisioning-timeout failures in this check.
|
|
||
| timeout: TimeoutConfig = TimeoutConfig() | ||
| retries: RetryConfig = RetryConfig() | ||
| checkpoint: RolloutCheckpointConfig | None = None |
There was a problem hiding this comment.
lets turn this on by default
| live: dict[str, Any] = Field(default_factory=dict) | ||
| """What a memory restore needs to reattach to the harness program still running in | ||
| the resumed box (`HarnessSession.memory_state`), and the relay that program reaches | ||
| the interception through (`Rollout`).""" |
There was a problem hiding this comment.
Relay secrets persist on serialized checkpoints
Medium Severity
Checkpoint.live holds relay box tokens and interception secrets and is not exclude=True. Unlike tail, state, info, and pins, it is written into persisted traces instead of traveling only on RestorePoint.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 7c3a57b. Configure here.
| sandbox gone), for `start` to restore the checkpoint's filesystem instead.""" | ||
| assert snapshot.box is not None and self._client is not None | ||
| try: | ||
| await self._client.resume(snapshot.box) |
There was a problem hiding this comment.
🟠 High backends/prime.py:323
Restoring an older memory snapshot can resume a newer pause of the same sandbox, so execution continues with C2's state even though the rollout has rewound to C1. _resume() passes only snapshot.box to resume() and then marks the backend restored_memory=True without verifying that the sandbox's current pause belongs to snapshot.id; compare the current pause checkpoint ID with snapshot.id and fall back to filesystem restore when they differ.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/runtimes/backends/prime.py around line 323:
Restoring an older memory snapshot can resume a newer pause of the same sandbox, so execution continues with C2's state even though the rollout has rewound to C1. `_resume()` passes only `snapshot.box` to `resume()` and then marks the backend `restored_memory=True` without verifying that the sandbox's current pause belongs to `snapshot.id`; compare the current pause checkpoint ID with `snapshot.id` and fall back to filesystem restore when they differ.
| assert snapshot.box is not None and self._client is not None | ||
| try: | ||
| await self._client.resume(snapshot.box) | ||
| await self._settle(snapshot.box, "RUNNING") |
There was a problem hiding this comment.
🟡 Medium backends/prime.py:324
If cancellation arrives after resume() succeeds but while _settle() is waiting, the running sandbox is left undeleted because self.info.id is still None when teardown runs. Assign the ID immediately after resume() succeeds, before the cancellable wait.
await self._client.resume(snapshot.box)
+ self.info.id = snapshot.box
await self._settle(snapshot.box, "RUNNING")🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/runtimes/backends/prime.py around line 324:
If cancellation arrives after `resume()` succeeds but while `_settle()` is waiting, the running sandbox is left undeleted because `self.info.id` is still `None` when teardown runs. Assign the ID immediately after `resume()` succeeds, before the cancellable wait.
| try: | ||
| if snapshot.memory and snapshot.box is not None: | ||
| sandbox = await client.get(snapshot.box) | ||
| if sandbox.status == "PAUSED": |
There was a problem hiding this comment.
🟠 High backends/prime.py:435
Discarding an older memory snapshot deletes the paused sandbox even when a newer kept snapshot uses that same box, so the kept restore point loses its live memory state. discard_checkpoints() checks only whether the box is PAUSED; since _resume() and the next _pause() preserve the same box ID, verify that the discarded snapshot is the box’s current pause checkpoint before deleting the box.
Also found in 1 other location(s)
verifiers/v1/agent.py:877
Cleanup protects only checkpoint IDs, not the paused box backing a kept memory checkpoint. If attempt 0 takes memory checkpoint A, a retry resumes that box and takes checkpoint B, then the run ends with
memory_checkpoint.after='keep'(or is interrupted), B is kept but A goes intodrop. Both snapshots have the samebox:PrimeBackend._resumeadoptssnapshot.box, and_pauserecords that same ID.PrimeBackend.discard_checkpoints([A])deletes this box whenever it is PAUSED, destroying B's preserved processes/memory even though B remains advertised as restorable. Protect boxes referenced by kept snapshots when discarding older memory checkpoints. Prime's Pause & Resume documentation confirms resume restores in place; its SDK Clean Exit documentation identifiesdeleteas sandbox teardown.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/runtimes/backends/prime.py around line 435:
Discarding an older memory snapshot deletes the paused sandbox even when a newer kept snapshot uses that same box, so the kept restore point loses its live memory state. `discard_checkpoints()` checks only whether the box is `PAUSED`; since `_resume()` and the next `_pause()` preserve the same box ID, verify that the discarded snapshot is the box’s current pause checkpoint before deleting the box.
Also found in 1 other location(s):
- verifiers/v1/agent.py:877 -- Cleanup protects only checkpoint IDs, not the paused box backing a kept memory checkpoint. If attempt 0 takes memory checkpoint A, a retry resumes that box and takes checkpoint B, then the run ends with `memory_checkpoint.after='keep'` (or is interrupted), B is kept but A goes into `drop`. Both snapshots have the same `box`: `PrimeBackend._resume` adopts `snapshot.box`, and `_pause` records that same ID. `PrimeBackend.discard_checkpoints([A])` deletes this box whenever it is PAUSED, destroying B's preserved processes/memory even though B remains advertised as restorable. Protect boxes referenced by kept snapshots when discarding older memory checkpoints. Prime's Pause & Resume documentation confirms resume restores in place; its SDK Clean Exit documentation identifies `delete` as sandbox teardown.
| taken: wait on its program, still running in the resumed box, and check its | ||
| result as `turn` does. `state` is what `memory_state` returned.""" | ||
| async with boundary(HarnessError, f"harness {self.harness.config.id!r}"): | ||
| result = await self.runtime.attach(state["program"]) |
There was a problem hiding this comment.
🟠 High v1/harness.py:469
After reattach() attaches the resumed program, self._program remains None, so a subsequent interruption or provider failure makes memory_state() return None and prevents another memory checkpoint. Runtime.attach does not update this field; assign state["program"] to self._program before awaiting attach.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/harness.py around line 469:
After `reattach()` attaches the resumed program, `self._program` remains `None`, so a subsequent interruption or provider failure makes `memory_state()` return `None` and prevents another memory checkpoint. `Runtime.attach` does not update this field; assign `state["program"]` to `self._program` before awaiting `attach`.
| raise SandboxError(f"prime pause failed: {e}") from e | ||
| if not checkpoints: | ||
| raise SandboxError(f"prime pause of {self.info.id} left no checkpoint") | ||
| snapshot = Snapshot(id=checkpoints[-1].id, memory=True, box=self.info.id) |
There was a problem hiding this comment.
🟡 Medium backends/prime.py:395
If wait_checkpoint() fails for a memory snapshot, the paused sandbox and checkpoint are orphaned: _pause() sets _paused=True, so teardown skips deleting the sandbox, while this snapshot is absent from stray_checkpoints and cannot be cleaned up. Add the snapshot to stray_checkpoints here; wait_checkpoint() already removes it once durable.
| snapshot = Snapshot(id=checkpoints[-1].id, memory=True, box=self.info.id) | |
| snapshot = Snapshot(id=checkpoints[-1].id, memory=True, box=self.info.id) | |
| self.stray_checkpoints.append(snapshot) |
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/runtimes/backends/prime.py around line 395:
If `wait_checkpoint()` fails for a memory snapshot, the paused sandbox and checkpoint are orphaned: `_pause()` sets `_paused=True`, so teardown skips deleting the sandbox, while this snapshot is absent from `stray_checkpoints` and cannot be cleaned up. Add the snapshot to `stray_checkpoints` here; `wait_checkpoint()` already removes it once durable.
| return cls( | ||
| checkpoint=checkpoint, | ||
| tail=list(checkpoint.tail), | ||
| state=state.model_dump(mode="json") |
There was a problem hiding this comment.
🟠 High v1/trace.py:324
RestorePoint.of() raises UnicodeDecodeError for a checkpoint whose State.artifacts contains non-UTF-8 bytes, so DeltaStreamer.diff() and its final flush fail instead of exporting the restore point. model_dump(mode="json") applies Pydantic’s UTF-8 bytes encoding; preserve the bytes for the existing msgpack transport or encode them reversibly.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/trace.py around line 324:
`RestorePoint.of()` raises `UnicodeDecodeError` for a checkpoint whose `State.artifacts` contains non-UTF-8 bytes, so `DeltaStreamer.diff()` and its final flush fail instead of exporting the restore point. `model_dump(mode="json")` applies Pydantic’s UTF-8 bytes encoding; preserve the bytes for the existing msgpack transport or encode them reversibly.
| if self._memory_checkpoints or self._restored_memory: | ||
| # The harness reaches the interception through a relay in its box, | ||
| # which a resume reroutes to the new process's URL and secrets. | ||
| self._endpoint, self._secret = await self._route_relay( |
There was a problem hiding this comment.
🟠 High v1/rollout.py:501
When a memory restore reroutes the relay, outstanding model requests are retried against this session before restore() installs the checkpoint pins and marks the tail as intercepted. Those requests run hooks again without the pinned tool results, duplicating non-idempotent rewrites or state updates and potentially corrupting the resumed conversation. Install the restored session state before _route_relay() exposes it, or gate requests until restore preparation is complete.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/rollout.py around line 501:
When a memory restore reroutes the relay, outstanding model requests are retried against this session before `restore()` installs the checkpoint pins and marks the tail as intercepted. Those requests run hooks again without the pinned tool results, duplicating non-idempotent rewrites or state updates and potentially corrupting the resumed conversation. Install the restored session state before `_route_relay()` exposes it, or gate requests until restore preparation is complete.
| if self._session.checkpointer is not None and self._checkpointed is None: | ||
| pending = list(self.trace._pending.values()) | ||
| if len(pending) == 1 or self._memory_checkpoints: | ||
| await self._checkpoint(pending[0] if len(pending) == 1 else None, error) |
There was a problem hiding this comment.
🟠 High v1/rollout.py:890
Interrupting during setup can report success while the rollout continues sampling: the memory-checkpoint branch calls _checkpoint() before _harness_session exists, so _capture() raises before self._session.abort(error) runs. Skip checkpoint capture until the harness session exists, and ensure the abort is delivered even if capture fails.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/rollout.py around line 890:
Interrupting during setup can report success while the rollout continues sampling: the memory-checkpoint branch calls `_checkpoint()` before `_harness_session` exists, so `_capture()` raises before `self._session.abort(error)` runs. Skip checkpoint capture until the harness session exists, and ensure the abort is delivered even if capture fails.
| return { | ||
| "process": process.session, | ||
| "request": self._prompt, | ||
| "calls_before": len(self.trace.calls), |
There was a problem hiding this comment.
🟠 High acp/__init__.py:417
After a memory checkpoint taken after the prompt's model call commits, reattach() can raise ACP agent completed without committing a model turn even though that turn is retained. memory_state() stores the checkpoint-time len(self.trace.calls) instead of the prompt's original calls_before, so _require_model_turn() sees no calls after the baseline; preserve the original baseline in the checkpoint state.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/acp/__init__.py around line 417:
After a memory checkpoint taken after the prompt's model call commits, `reattach()` can raise `ACP agent completed without committing a model turn` even though that turn is retained. `memory_state()` stores the checkpoint-time `len(self.trace.calls)` instead of the prompt's original `calls_before`, so `_require_model_turn()` sees no calls after the baseline; preserve the original baseline in the checkpoint state.
| harness: dict[str, Any] | None = None | ||
| # Parallel sub-agents share the trace: with another request in flight the box | ||
| # is not idle, so no filesystem snapshot matches this turn's prefix. | ||
| if ( |
There was a problem hiding this comment.
🟠 High v1/rollout.py:935
A memory checkpoint that falls back to filesystem restore cannot resume an ACP session: harness is saved as {}, so ACP's restore() raises KeyError for acp_session_id and the rollout's prior progress is lost. _capture only calls checkpoint_state() when _checkpointing is enabled and turn is non-None, so memory-only checkpoints and interrupts between model calls omit that metadata; capture the restore metadata for memory checkpoints independently of those conditions.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/rollout.py around line 935:
A memory checkpoint that falls back to filesystem restore cannot resume an ACP session: `harness` is saved as `{}`, so ACP's `restore()` raises `KeyError` for `acp_session_id` and the rollout's prior progress is lost. `_capture` only calls `checkpoint_state()` when `_checkpointing` is enabled and `turn` is non-`None`, so memory-only checkpoints and interrupts between model calls omit that metadata; capture the restore metadata for memory checkpoints independently of those conditions.
| if self._session.checkpointer is not None and self._checkpointed is None: | ||
| pending = list(self.trace._pending.values()) | ||
| if len(pending) == 1 or self._memory_checkpoints: | ||
| await self._checkpoint(pending[0] if len(pending) == 1 else None, error) |
There was a problem hiding this comment.
🟠 High v1/rollout.py:894
An interrupt checkpoint can restore a trace containing committed assistant/tool nodes and the original turn.tail whose filesystem effects are absent from the snapshot, leaving the restored rollout inconsistent and potentially replaying the tail. interrupt() awaits _checkpoint() before calling _session.abort(error), so the pending request can complete and the harness can act while _capture() is in progress; quiesce the in-flight sampling and harness activity before capturing the checkpoint.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/rollout.py around line 894:
An interrupt checkpoint can restore a trace containing committed assistant/tool nodes and the original `turn.tail` whose filesystem effects are absent from the snapshot, leaving the restored rollout inconsistent and potentially replaying the tail. `interrupt()` awaits `_checkpoint()` before calling `_session.abort(error)`, so the pending request can complete and the harness can act while `_capture()` is in progress; quiesce the in-flight sampling and harness activity before capturing the checkpoint.
| taken = await asyncio.gather( | ||
| *(run.interrupt(timeout) for run in runs), return_exceptions=True | ||
| ) | ||
| return len(runs), [run.trace.id for run, ok in zip(runs, taken) if ok is True] |
There was a problem hiding this comment.
Interrupt count includes unfinished rollouts
Medium Severity
interrupt_all reports len(runs) as how many rollouts it interrupted, but interrupt now returns immediately for a trace that already has stop_condition set and does not abort it. Those rollouts stay in _LIVE through scoring and finalize, so InterruptResponse.interrupted overcounts them. Callers that treat “interrupted minus checkpointed” as work to start over will restart rollouts that are actually finishing.
Additional Locations (2)
Reviewed by Cursor Bugbot for commit d930ea3. Configure here.
A rollout checkpoints its box when a model call fails past its turn retries, or when its owner interrupts it. A filesystem checkpoint (`checkpoint`) restores the files into a new box; a memory checkpoint (`memory_checkpoint`) resumes the same box with its processes. The prime backend hides the two SDK paths behind `checkpoint(memory=...)`. A loopback relay in the box lets a resumed harness reach the new interception, and ACP harnesses reattach to their running runner through the SDK's `attach_process`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An env server's `save` request takes a filesystem checkpoint of each running rollout at its next model call, while the box is idle, and the rollout goes on. An owner that saves its run (prime-rl's interval checkpoints) can then continue its in-flight rollouts in another process instead of starting them over. The pool now sends `interrupt` and `save` to every worker and merges their replies; before, one worker answered for its own rollouts only. A final run also keeps every snapshot that `after="keep"` leaves to its owner. Before, it deleted all but the newest, which an owner may still hold. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A rollout being scored makes no more model calls, so its save would wait out the whole timeout and hold up its owner's checkpoint. It now takes none at once, and a save still waiting when the agent's segment ends gives up then. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A `save` request with `hold` keeps each rollout that took a checkpoint waiting at its model call until a `release` request, or for `hold` seconds at most. Its owner then records the rollout as it is at its checkpoint. Without the hold, the rollout went on while the others saved, and a resume from that record redid those turns. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A held save no longer waits for a rollout still setting up its box: it has no turn to lose. Its first model call before the release takes the checkpoint and holds there. Before, the rollout ran on unheld, and a box slow to boot made the save wait out its timeout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
d930ea3 to
3ca463e
Compare
Wiz Scan Summary
To detect these findings earlier in the dev lifecycle, try the Wiz Code extension for VS Code, JetBrains, or Visual Studio. |
| if restore is None and self.pending_restore is not None: | ||
| restore, self.pending_restore = self.pending_restore, None |
There was a problem hiding this comment.
🟡 Medium v1/agent.py:748
run(task, runtime=box, restore=...) continues the saved conversation against box without restoring the checkpoint's filesystem, so the trace can claim checkpoint state that is absent and execution can modify the wrong workspace. Reject restores when runtime is borrowed, checking pending_restore before consuming it.
| if restore is None and self.pending_restore is not None: | |
| restore, self.pending_restore = self.pending_restore, None | |
| if runtime is not None and (restore is not None or self.pending_restore is not None): | |
| raise ValueError("cannot restore into a borrowed runtime") | |
| if restore is None and self.pending_restore is not None: | |
| restore, self.pending_restore = self.pending_restore, None |
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/agent.py around lines 748-749:
`run(task, runtime=box, restore=...)` continues the saved conversation against `box` without restoring the checkpoint's filesystem, so the trace can claim checkpoint state that is absent and execution can modify the wrong workspace. Reject restores when `runtime` is borrowed, checking `pending_restore` before consuming it.
| c | ||
| for c in self.checkpoints | ||
| if c is not checkpoint and c.node is not None and c.node >= keep | ||
| ] | ||
| self.checkpoints = [c for c in self.checkpoints if c not in rewound] |
There was a problem hiding this comment.
🟠 High v1/trace.py:1147
Rewinding to checkpoint A leaves later checkpoint B restorable when both were captured at the same node (or both have node=None), so Agent.run can resume from the abandoned B state and B’s snapshot is omitted from cleanup. rewind filters by node number instead of checkpoint order; prune every checkpoint after the selected checkpoint in self.checkpoints.
- rewound = [
- c
- for c in self.checkpoints
- if c is not checkpoint and c.node is not None and c.node >= keep
- ]
- self.checkpoints = [c for c in self.checkpoints if c not in rewound]
+ checkpoint_index = next(
+ i for i, c in enumerate(self.checkpoints) if c is checkpoint
+ )
+ rewound = self.checkpoints[checkpoint_index + 1 :]
+ self.checkpoints = self.checkpoints[: checkpoint_index + 1]🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/trace.py around lines 1147-1151:
Rewinding to checkpoint A leaves later checkpoint B restorable when both were captured at the same node (or both have `node=None`), so `Agent.run` can resume from the abandoned B state and B’s snapshot is omitted from cleanup. `rewind` filters by node number instead of checkpoint order; prune every checkpoint after the selected checkpoint in `self.checkpoints`.
| interrupted = ( | ||
| run.trace.last_error is not None | ||
| and run.trace.last_error.type == "RolloutInterrupted" | ||
| ) |
There was a problem hiding this comment.
🟡 Medium v1/agent.py:866
A resumed trace that completes successfully still keeps its snapshot, so checkpoint.after="delete" leaks it on every interrupt/resume cycle. Trace.last_error includes the prior attempt's RolloutInterrupted because rewind preserves errors; check the current attempt's errors and require that it did not complete successfully before adding the checkpoint to keep.
- interrupted = (
- run.trace.last_error is not None
- and run.trace.last_error.type == "RolloutInterrupted"
- )
+ current_errors = run.trace.errors[seen:]
+ interrupted = (
+ not run.ok
+ and bool(current_errors)
+ and current_errors[-1].type == "RolloutInterrupted"
+ )🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/agent.py around lines 866-869:
A resumed trace that completes successfully still keeps its snapshot, so `checkpoint.after="delete"` leaks it on every interrupt/resume cycle. `Trace.last_error` includes the prior attempt's `RolloutInterrupted` because rewind preserves errors; check the current attempt's errors and require that it did not complete successfully before adding the checkpoint to `keep`.
| message = request.messages[position] | ||
| if position >= len(branch) and isinstance(message, ToolMessage): | ||
| self.pinned_tool_results.setdefault(key, message) | ||
| self.prepared_users.update( |
There was a problem hiding this comment.
🟡 Medium v1/session.py:427
After memory restore, a rewritten user message is intercepted again, so a stateful hook can repeat side effects or produce a different prompt. mark_intercepted records only the rewritten hash, but the replay uses the original request body, whose hash does not match prepared_users; preserve and apply the original-to-rewritten mapping on replay.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/session.py around line 427:
After memory restore, a rewritten user message is intercepted again, so a stateful hook can repeat side effects or produce a different prompt. `mark_intercepted` records only the rewritten hash, but the replay uses the original request body, whose hash does not match `prepared_users`; preserve and apply the original-to-rewritten mapping on replay.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 3 total unresolved issues (including 2 from previous reviews).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 3ca463e. Configure here.
| and run.trace.last_error.type == "RolloutInterrupted" | ||
| ) | ||
| if last.snapshot is not None and interrupted: | ||
| keep.add(last.id) |
There was a problem hiding this comment.
Restored interrupt leaks final checkpoint
High Severity
Cleanup treats a run as interrupted whenever last_error is RolloutInterrupted, but rewind keeps prior attempts' errors. After another process restores an interrupted run and it finishes, that leftover error still skips discard. The checkpoint, including a billed paused sandbox, is left behind.
Reviewed by Cursor Bugbot for commit 3ca463e. Configure here.


Supersedes #2741. GitHub closed #2741 when its branch was renamed to
feat/rollout-ckpt; its review history stays there.Summary
An agentic rollout can run for hours. When it fails late (a provider outage, a stopped run), a retry used to start from scratch. With this PR, a rollout checkpoints its box when it fails or when its owner asks, and the retry, or another process, continues from there. An earlier version of this PR snapshotted the box before every model turn; on a large RL run the replies waited about 20 % of the agent time for those snapshots (snapshot API rate limits). Checkpoints are now taken only on a failure or a request.
When a rollout checkpoints
AgentConfig.turn_retries): the interception retries a failed model call with backoff (408, 429, 5xx and transport failures; 5 retries, 1 s doubling to 30 s). The harness waits on the call and never sees an error that a retry recovered. Another status (a 400 such as a context overflow) still goes to the harness, which may handle it.ProviderError. The rollout retry (AgentConfig.retries) restores the checkpoint.Rollout.interrupt, orinterrupt_allfor every open rollout of the process) checkpoints the box and ends the rollout withRolloutInterrupted, for another process to continue it from its restore point (prime-rl does this when it pauses a run).Rollout.save, orsave_all) takes a filesystem checkpoint of the box at the rollout's next model call, while the box is idle, and the rollout goes on (prime-rl does this at each periodic checkpoint). Withhold, the rollout waits at that model call untilrelease_all, or untilholdseconds pass, so its owner records it as it is at the checkpoint. A rollout that is still setting up registers the save and checkpoints at its first model call; the save does not wait for it. A rollout whose agent is done (scoring) skips the save.SandboxErrortakes no checkpoint: the box is suspect, so its retry starts over in a new box.SandboxErrorand aProviderErrortwice each by default.Two kinds of checkpoints
AgentConfig.checkpoint, on by default): the runtime snapshots the box's files. A restore boots a new box from the snapshot and a new harness process, which continues the trace and gets a user turn that says the box was restored. Processes, sub-agents and in-memory state are gone. NeedsSUPPORTS_FILESYSTEM_CHECKPOINTSon the runtime and the harness (renamed fromSUPPORTS_CHECKPOINTS).AgentConfig.memory_checkpoint, off by default): the checkpoint also holds the box's processes and memory. A restore resumes the box, and the harness program carries on as it was, with its background processes, REPL state and sub-agents; the harness does not notice. NeedsSUPPORTS_MEMORY_CHECKPOINTSon the runtime and the harness. Memory goes first where both kinds apply, except for a save: a memory checkpoint stops the box.aftersets whether the checkpoints are deleted or kept when the run is final. With"keep", a run deletes only the checkpoints it took itself: a run restored from another process's checkpoint leaves that one to its owner.Runtime backend (
RuntimeBackend.checkpoint(memory=...),wait_checkpoint,discard_checkpoints,restore,restored_memory)Snapshot: a filesystem checkpoint id, and for memory the box it resumes in.checkpoint(); a memory checkpoint ispause(), which also freezes the disk into a checkpoint of its own. A memory restore resumes the paused sandbox; when it no longer resumes (resumed already, node gone, CPU changed), the restore boots a new box from the pause's disk checkpoint, as a filesystem restore would. Teardown leaves a paused sandbox to the checkpoint's owner. Today a pause resumes once and only on its node (platform#6498 adds cross-node resume); the backend hides that behind the same API.attachwaits on a background job again, andattach_processstreams a live process by its session, so a new host process can reattach to the programs a resumed box still runs.attach_processneeds the prime-sandboxes release with feat(sandboxes): attach to a live process by session prime#1147; without it, ACP harnesses take filesystem checkpoints only.Reattaching a harness after a memory restore
runtimes/relay_server.py). Loopback connections survive a pause. After a resume, the new host rewrites the relay's config to its own URL and secrets and signals it; the relay retries each request still waiting on its first event against the new upstream. No secrets of the old process need to live on.HarnessSession.reattachwaits on the program's background job again.session/load(feat: load a session from its ledger over ACP nano-rlm#238); the harness pins that commit.Trace
Checkpointholds itsSnapshot, thereasonit was taken on (an error type, orSaved), andlive(what a memory restore reattaches to). A checkpoint keeps every committed node, sub-agents' included.heldandtiming.agent.checkpointare gone.Trace.restorableis the newest checkpoint a restore can use, andRestorePointcarries it to another process (Env.run_slot(restore=...)).Trace.rewindtruncates the trace to a checkpoint and bumpsTrace.generation.prime_rl.env, feat: checkpoint in-flight and in-queue rollouts prime-rl#4296).Docs:
docs/agent.md(Retries and checkpoints: turn retries, the two kinds of checkpoints, interrupt and save) anddocs/harnesses.md(the two harness flags).Breaking
SUPPORTS_CHECKPOINTS(runtime backend and harness) is nowSUPPORTS_FILESYSTEM_CHECKPOINTS. Migration: rename the class attribute.ProviderErrorinstead of the harness getting the error. Migration:--env.agent.turn-retries.max-retries 0fails the attempt at the first such error; setstatus_codeto choose the statuses.SandboxErrorand aProviderErrortwice each by default (before: no retries). Migration: set[env.agent.retries]explicitly.RolloutCheckpointConfig.intervaland.keepare gone: a rollout takes one checkpoint per failure, interrupt or save.RuntimeBackend.checkpointreturns aSnapshot(before: an id), andwait_checkpoint/discard_checkpointstakeSnapshots.Verification
tests/unit/test_checkpoint.py(capture on failure, memory first with filesystem fallback, one checkpoint per attempt, interrupt with and without a turn in flight,interrupt_all, save, held save and its timeout, save of a rollout still setting up or whose agent is done, turn-retry backoff, capability warnings),tests/unit/test_relay.py(forwarding, token swap, streaming, retry on reroute),tests/unit/test_agent.py(a request hook that raises ends the rollout).pytest tests/unit: 262 passed, 19 skipped (no Docker, noPRIME_API_KEY, noharbor).tests/integration/test_checkpoint.py), with a scripted model that fails one turn past its retries:counter 2after the resume (counter 1before): the process and its memory survived. The bash program's waiting request reached the new interception through the relay.Connection refused), and the run carries on.kept = 41(same pid), and a persistent sub-agent spawned before the failure answers what it was told to remember after the resume. The new host attached to the running ACP runner.DURABLEcheckpoint, and resumed in 4 s with the same pid; a second resume returns 409.🤖 Generated with Claude Code
Note
High Risk
This touches core rollout lifecycle, sandbox pause/resume, model-call retry semantics, and cross-process restore—failures could lose progress, leak paused sandboxes, or change error handling for existing eval runs.
Overview
Adds failure-driven rollout checkpointing so agent retries (and cross-process resume) can continue from a saved box state instead of restarting the task from scratch.
Turn retries run inside the interception (
TurnRetryConfig, default 5× on 408/429/5xx/transport). When they are exhausted—or the owner interrupts the rollout—the attempt checkpoints the sandbox and fails; rollout retries (AgentConfig.retries, now default 2× forSandboxErrorandProviderError) restore the newest durable checkpoint on the same trace.SandboxErrorstill skips checkpointing and starts a fresh box.Two checkpoint modes: filesystem (default, snapshots files; new harness process + restore notice) and optional memory (pause/resume processes; harness reattach via in-box HTTP relay and ACP reply replay). Harnesses gain
SUPPORTS_FILESYSTEM_CHECKPOINTS/SUPPORTS_MEMORY_CHECKPOINTS;Rollout.save,interrupt_all, andRestorePointsupport owner-driven saves and handoff.Also changes request-hook failures to abort the segment immediately (no fake HTTP error to the harness), moves Claude/Codex/RLM session state under
/tmp/vf-*-runsfor checkpointable files, and documents the behavior indocs/agent.mdanddocs/harnesses.md.Reviewed by Cursor Bugbot for commit 3ca463e. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Add filesystem and memory checkpointing for agent rollouts with turn retries
AgentConfig: filesystem checkpoints default on (after: delete), memory checkpoints default off. Also addsTurnRetryConfig(5 retries for statuses 408, 429, 5xx, 1s initial / 30s max delay) and default retry budgets of 2 forSandboxErrorandProviderError.Rolloutcan now restore from aCheckpoint:Trace.rewindtruncates the graph and timing after the checkpoint, and memory restores reattach to the saved process via the harness session while filesystem restores restart the box and append a restore-notice user message. See rollout.py and trace.py.Agent.runretries resume from the newest eligible checkpoint instead of starting over. A failed restore during setup falls back to a fresh attempt without consuming the retry budget.PrimeBackendimplements checkpoint creation (memory via pause, filesystem via snapshot), durability waits, memory restore, process attach/detach, and a relay for rerouting model traffic on restored sandboxes (prime.py, relay.py).Bash,Null,RLM,Claude Code,Codex, ACP) declare checkpoint capability flags; the ACP harness can load saved sessions and resend in-flight prompts with their original request IDs. ExposesCheckpoint,Snapshot, andRolloutInterruptedin the public API./tmp/vf-rlm-runs,/tmp/vf-claude-code-runs,/tmp/vf-codex-runs),Tracespan duration now excludes calls that started before the span, request-mediation failures now abort the rollout and leave the handler suspended, andRollout.cleanupskips deleting sandboxes marked paused — checkSUPPORTS_*_CHECKPOINTSoverrides and any code relying on the old.vf-claudepath or_failresponses.Macroscope summarized 3ca463e.