Skip to content

feat: rollout checkpointing - #2994

Open
mikasenghaas wants to merge 5 commits into
mainfrom
feat/rollout-ckpt
Open

mikasenghaas wants to merge 5 commits into
mainfrom
feat/rollout-ckpt

Conversation

@mikasenghaas

@mikasenghaas mikasenghaas commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

Supersedes #2741. GitHub closed #2741 when its branch was renamed to feat/rollout-ckpt; its review history stays there.

Summary

An agentic rollout can run for hours. When it fails late (a provider outage, a stopped run), a retry used to start from scratch. With this PR, a rollout checkpoints its box when it fails or when its owner asks, and the retry, or another process, continues from there. An earlier version of this PR snapshotted the box before every model turn; on a large RL run the replies waited about 20 % of the agent time for those snapshots (snapshot API rate limits). Checkpoints are now taken only on a failure or a request.

When a rollout checkpoints

  • Turn retries (AgentConfig.turn_retries): the interception retries a failed model call with backoff (408, 429, 5xx and transport failures; 5 retries, 1 s doubling to 30 s). The harness waits on the call and never sees an error that a retry recovered. Another status (a 400 such as a context overflow) still goes to the harness, which may handle it.
  • When the turn retries run out, the rollout checkpoints its box at that turn and fails the attempt with the ProviderError. The rollout retry (AgentConfig.retries) restores the checkpoint.
  • An interrupt (Rollout.interrupt, or interrupt_all for every open rollout of the process) checkpoints the box and ends the rollout with RolloutInterrupted, for another process to continue it from its restore point (prime-rl does this when it pauses a run).
  • A save (Rollout.save, or save_all) takes a filesystem checkpoint of the box at the rollout's next model call, while the box is idle, and the rollout goes on (prime-rl does this at each periodic checkpoint). With hold, the rollout waits at that model call until release_all, or until hold seconds pass, so its owner records it as it is at the checkpoint. A rollout that is still setting up registers the save and checkpoints at its first model call; the save does not wait for it. A rollout whose agent is done (scoring) skips the save.
  • A SandboxError takes no checkpoint: the box is suspect, so its retry starts over in a new box.
  • Agent rollouts now retry a SandboxError and a ProviderError twice each by default.

Two kinds of checkpoints

  • Filesystem (AgentConfig.checkpoint, on by default): the runtime snapshots the box's files. A restore boots a new box from the snapshot and a new harness process, which continues the trace and gets a user turn that says the box was restored. Processes, sub-agents and in-memory state are gone. Needs SUPPORTS_FILESYSTEM_CHECKPOINTS on the runtime and the harness (renamed from SUPPORTS_CHECKPOINTS).
  • Memory (AgentConfig.memory_checkpoint, off by default): the checkpoint also holds the box's processes and memory. A restore resumes the box, and the harness program carries on as it was, with its background processes, REPL state and sub-agents; the harness does not notice. Needs SUPPORTS_MEMORY_CHECKPOINTS on the runtime and the harness. Memory goes first where both kinds apply, except for a save: a memory checkpoint stops the box.
  • after sets whether the checkpoints are deleted or kept when the run is final. With "keep", a run deletes only the checkpoints it took itself: a run restored from another process's checkpoint leaves that one to its owner.

Runtime backend (RuntimeBackend.checkpoint(memory=...), wait_checkpoint, discard_checkpoints, restore, restored_memory)

  • A checkpoint is a Snapshot: a filesystem checkpoint id, and for memory the box it resumes in.
  • Prime: a filesystem checkpoint is checkpoint(); a memory checkpoint is pause(), which also freezes the disk into a checkpoint of its own. A memory restore resumes the paused sandbox; when it no longer resumes (resumed already, node gone, CPU changed), the restore boots a new box from the pause's disk checkpoint, as a filesystem restore would. Teardown leaves a paused sandbox to the checkpoint's owner. Today a pause resumes once and only on its node (platform#6498 adds cross-node resume); the backend hides that behind the same API.
  • attach waits on a background job again, and attach_process streams a live process by its session, so a new host process can reattach to the programs a resumed box still runs. attach_process needs the prime-sandboxes release with feat(sandboxes): attach to a live process by session prime#1147; without it, ACP harnesses take filesystem checkpoints only.

Reattaching a harness after a memory restore

  • A harness that may be paused reaches the interception through a loopback relay in its box (runtimes/relay_server.py). Loopback connections survive a pause. After a resume, the new host rewrites the relay's config to its own URL and secrets and signals it; the relay retries each request still waiting on its first event against the new upstream. No secrets of the old process need to live on.
  • Chat-program harnesses (bash, null): HarnessSession.reattach waits on the program's background job again.
  • ACP harnesses (claude-code, codex, rlm): the session attaches to the running ACP runner by its session id and sends the prompt in flight again; the runner keeps its recent replies by request id and returns the one the old host missed. rlm restores a filesystem checkpoint with ACP session/load (feat: load a session from its ledger over ACP nano-rlm#238); the harness pins that commit.

Trace

  • Checkpoint holds its Snapshot, the reason it was taken on (an error type, or Saved), and live (what a memory restore reattaches to). A checkpoint keeps every committed node, sub-agents' included. held and timing.agent.checkpoint are gone.
  • Trace.restorable is the newest checkpoint a restore can use, and RestorePoint carries it to another process (Env.run_slot(restore=...)). Trace.rewind truncates the trace to a checkpoint and bumps Trace.generation.
  • The env server that sends these to prime-rl now lives in prime-rl (prime_rl.env, feat: checkpoint in-flight and in-queue rollouts prime-rl#4296).

Docs: docs/agent.md (Retries and checkpoints: turn retries, the two kinds of checkpoints, interrupt and save) and docs/harnesses.md (the two harness flags).

Breaking

  • SUPPORTS_CHECKPOINTS (runtime backend and harness) is now SUPPORTS_FILESYSTEM_CHECKPOINTS. Migration: rename the class attribute.
  • A model call that fails with 408, 429, 5xx or a transport error is retried in the interception up to 5 times before the harness sees anything; when the retries run out, the attempt fails with the ProviderError instead of the harness getting the error. Migration: --env.agent.turn-retries.max-retries 0 fails the attempt at the first such error; set status_code to choose the statuses.
  • Agent rollouts retry a SandboxError and a ProviderError twice each by default (before: no retries). Migration: set [env.agent.retries] explicitly.
  • RolloutCheckpointConfig.interval and .keep are gone: a rollout takes one checkpoint per failure, interrupt or save.
  • RuntimeBackend.checkpoint returns a Snapshot (before: an id), and wait_checkpoint / discard_checkpoints take Snapshots.
  • A request hook that raises now ends the rollout at once with its error; the harness never sees it.

Verification

  • Unit: tests/unit/test_checkpoint.py (capture on failure, memory first with filesystem fallback, one checkpoint per attempt, interrupt with and without a turn in flight, interrupt_all, save, held save and its timeout, save of a rollout still setting up or whose agent is done, turn-retry backoff, capability warnings), tests/unit/test_relay.py (forwarding, token swap, streaming, retry on reroute), tests/unit/test_agent.py (a request hook that raises ends the rollout). pytest tests/unit: 262 passed, 19 skipped (no Docker, no PRIME_API_KEY, no harbor).
  • Integration on prime (tests/integration/test_checkpoint.py), with a scripted model that fails one turn past its retries:
    • bash, memory: a background HTTP server started before the failure answers counter 2 after the resume (counter 1 before): the process and its memory survived. The bash program's waiting request reached the new interception through the relay.
    • bash, filesystem: the new box has the files; the server is gone (Connection refused), and the run carries on.
    • rlm, memory: the REPL kernel keeps kept = 41 (same pid), and a persistent sub-agent spawned before the failure answers what it was told to remember after the resume. The new host attached to the running ACP runner.
  • Live probes of the platform API: a pause took about 10 s, left one DURABLE checkpoint, and resumed in 4 s with the same pid; a second resume returns 409.
  • RL e2e through prime-rl#4296 (GLM-4.5-Air on scaleswe, a periodic checkpoint and a fork from it): 188 of 198 in-flight rollouts saved a restore point with no turns past it, and the fork continued all 188 with 0 failures. Details in that PR.

🤖 Generated with Claude Code


Note

High Risk
This touches core rollout lifecycle, sandbox pause/resume, model-call retry semantics, and cross-process restore—failures could lose progress, leak paused sandboxes, or change error handling for existing eval runs.

Overview
Adds failure-driven rollout checkpointing so agent retries (and cross-process resume) can continue from a saved box state instead of restarting the task from scratch.

Turn retries run inside the interception (TurnRetryConfig, default 5× on 408/429/5xx/transport). When they are exhausted—or the owner interrupts the rollout—the attempt checkpoints the sandbox and fails; rollout retries (AgentConfig.retries, now default 2× for SandboxError and ProviderError) restore the newest durable checkpoint on the same trace. SandboxError still skips checkpointing and starts a fresh box.

Two checkpoint modes: filesystem (default, snapshots files; new harness process + restore notice) and optional memory (pause/resume processes; harness reattach via in-box HTTP relay and ACP reply replay). Harnesses gain SUPPORTS_FILESYSTEM_CHECKPOINTS / SUPPORTS_MEMORY_CHECKPOINTS; Rollout.save, interrupt_all, and RestorePoint support owner-driven saves and handoff.

Also changes request-hook failures to abort the segment immediately (no fake HTTP error to the harness), moves Claude/Codex/RLM session state under /tmp/vf-*-runs for checkpointable files, and documents the behavior in docs/agent.md and docs/harnesses.md.

Reviewed by Cursor Bugbot for commit 3ca463e. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Add filesystem and memory checkpointing for agent rollouts with turn retries

  • Adds rollout checkpointing to AgentConfig: filesystem checkpoints default on (after: delete), memory checkpoints default off. Also adds TurnRetryConfig (5 retries for statuses 408, 429, 5xx, 1s initial / 30s max delay) and default retry budgets of 2 for SandboxError and ProviderError.
  • Rollout can now restore from a Checkpoint: Trace.rewind truncates the graph and timing after the checkpoint, and memory restores reattach to the saved process via the harness session while filesystem restores restart the box and append a restore-notice user message. See rollout.py and trace.py.
  • Agent.run retries resume from the newest eligible checkpoint instead of starting over. A failed restore during setup falls back to a fresh attempt without consuming the retry budget.
  • PrimeBackend implements checkpoint creation (memory via pause, filesystem via snapshot), durability waits, memory restore, process attach/detach, and a relay for rerouting model traffic on restored sandboxes (prime.py, relay.py).
  • Harnesses (Bash, Null, RLM, Claude Code, Codex, ACP) declare checkpoint capability flags; the ACP harness can load saved sessions and resend in-flight prompts with their original request IDs. Exposes Checkpoint, Snapshot, and RolloutInterrupted in the public API.
  • Behavioral Change: per-trace state dirs moved to absolute roots (/tmp/vf-rlm-runs, /tmp/vf-claude-code-runs, /tmp/vf-codex-runs), Trace span duration now excludes calls that started before the span, request-mediation failures now abort the rollout and leave the handler suspended, and Rollout.cleanup skips deleting sandboxes marked paused — check SUPPORTS_*_CHECKPOINTS overrides and any code relying on the old .vf-claude path or _fail responses.

Macroscope summarized 3ca463e.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread verifiers/v1/rollout.py Resolved
Comment thread verifiers/v1/runtimes/backends/prime.py Resolved
@macroscopeapp

macroscopeapp Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Needs Human Approval

Macroscope's review found this PR requires human approval — Adds default filesystem recovery and opt-in memory checkpointing for agent rollouts. Unanswered review concerns about relay secrets in persisted traces and checkpoint/resource retention require resolution before rollout.

Needs human approval because:

  • 17 blocking correctness issues found at or above your repo's Minimum Blocking Severity

Adjust the Minimum Blocking Severity for this repo — including turning it Off — in Settings. You can add or adjust custom eligibility rules. Learn more.

Comment thread verifiers/v1/harnesses/rlm/harness.py Resolved
Comment thread verifiers/v1/harnesses/rlm/harness.py Resolved
Comment thread verifiers/v1/trace.py Resolved

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread verifiers/v1/rollout.py Resolved
@mikasenghaas mikasenghaas changed the title feat: resume retries from runtime checkpoints feat: restore rollouts from sandbox checkpoints Oct 27, 2026

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread verifiers/v1/rollout.py Resolved
Comment thread verifiers/v1/interception/server.py Outdated
if capture is not None:
held = time.monotonic()
try:
checkpoint = await capture

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High interception/server.py:891

A timed-out rollout can finish checkpoint settlement and stop the runtime while session.checkpointer(turn) is still capturing a snapshot; that late capture can then append a checkpoint and start durability work after settlement, losing the latest recovery point. The capture task is only awaited in this handler’s finally, while _settle_checkpoints() gathers only durability tasks already created. Track and drain capture tasks before checkpoint settlement and runtime teardown, or cancel and drain them during release.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/interception/server.py around line 891:

A timed-out rollout can finish checkpoint settlement and stop the runtime while `session.checkpointer(turn)` is still capturing a snapshot; that late capture can then append a checkpoint and start durability work after settlement, losing the latest recovery point. The capture task is only awaited in this handler’s `finally`, while `_settle_checkpoints()` gathers only durability tasks already created. Track and drain capture tasks before checkpoint settlement and runtime teardown, or cancel and drain them during release.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread verifiers/v1/rollout.py Resolved
Comment thread verifiers/v1/agent.py
snapshots.update(checkpoint.id for checkpoint in trace.checkpoints)
deleted.update(run.deleted_checkpoints)
if restore is not None and any(
error.stage == "setup" for error in trace.errors[seen:]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High v1/agent.py:730

A deployment-provisioning timeout while restoring a checkpoint is not marked as a failed restore, so subsequent retries can select the same checkpoint again and exhaust the retry budget despite an older usable snapshot. Rollout.open() records this timeout without stage == "setup", so include provisioning-timeout failures in this check.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/agent.py around line 730:

A deployment-provisioning timeout while restoring a checkpoint is not marked as a failed restore, so subsequent retries can select the same checkpoint again and exhaust the retry budget despite an older usable snapshot. `Rollout.open()` records this timeout without `stage == "setup"`, so include provisioning-timeout failures in this check.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread verifiers/v1/agent.py Resolved
Comment thread verifiers/v1/rollout.py Resolved
Comment thread verifiers/v1/serve/delta.py Resolved
Comment thread verifiers/v1/serve/client.py Resolved

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread verifiers/v1/env.py Resolved
Comment thread verifiers/v1/serve/server.py Resolved

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread verifiers/v1/serve/delta.py Resolved

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this

Comment thread verifiers/v1/configs/agent.py Outdated

timeout: TimeoutConfig = TimeoutConfig()
retries: RetryConfig = RetryConfig()
checkpoint: RolloutCheckpointConfig | None = None

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets turn this on by default

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread verifiers/v1/agent.py Resolved
@mikasenghaas mikasenghaas changed the title feat: restore rollouts from sandbox checkpoints feat: rollout checkpointing Oct 29, 2026

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread verifiers/v1/rollout.py Resolved

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread verifiers/v1/rollout.py Resolved
Comment thread verifiers/v1/interception/server.py Resolved

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread verifiers/v1/harnesses/rlm/harness.py Resolved

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread verifiers/v1/runtimes/backends/prime.py Resolved
Comment thread verifiers/v1/runtimes/backends/prime.py Resolved
Comment thread verifiers/v1/rollout.py Resolved
Comment thread verifiers/v1/trace.py
live: dict[str, Any] = Field(default_factory=dict)
"""What a memory restore needs to reattach to the harness program still running in
the resumed box (`HarnessSession.memory_state`), and the relay that program reaches
the interception through (`Rollout`)."""

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Relay secrets persist on serialized checkpoints

Medium Severity

Checkpoint.live holds relay box tokens and interception secrets and is not exclude=True. Unlike tail, state, info, and pins, it is written into persisted traces instead of traveling only on RestorePoint.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 7c3a57b. Configure here.

sandbox gone), for `start` to restore the checkpoint's filesystem instead."""
assert snapshot.box is not None and self._client is not None
try:
await self._client.resume(snapshot.box)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High backends/prime.py:323

Restoring an older memory snapshot can resume a newer pause of the same sandbox, so execution continues with C2's state even though the rollout has rewound to C1. _resume() passes only snapshot.box to resume() and then marks the backend restored_memory=True without verifying that the sandbox's current pause belongs to snapshot.id; compare the current pause checkpoint ID with snapshot.id and fall back to filesystem restore when they differ.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/runtimes/backends/prime.py around line 323:

Restoring an older memory snapshot can resume a newer pause of the same sandbox, so execution continues with C2's state even though the rollout has rewound to C1. `_resume()` passes only `snapshot.box` to `resume()` and then marks the backend `restored_memory=True` without verifying that the sandbox's current pause belongs to `snapshot.id`; compare the current pause checkpoint ID with `snapshot.id` and fall back to filesystem restore when they differ.

assert snapshot.box is not None and self._client is not None
try:
await self._client.resume(snapshot.box)
await self._settle(snapshot.box, "RUNNING")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium backends/prime.py:324

If cancellation arrives after resume() succeeds but while _settle() is waiting, the running sandbox is left undeleted because self.info.id is still None when teardown runs. Assign the ID immediately after resume() succeeds, before the cancellable wait.

             await self._client.resume(snapshot.box)
+            self.info.id = snapshot.box
             await self._settle(snapshot.box, "RUNNING")
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/runtimes/backends/prime.py around line 324:

If cancellation arrives after `resume()` succeeds but while `_settle()` is waiting, the running sandbox is left undeleted because `self.info.id` is still `None` when teardown runs. Assign the ID immediately after `resume()` succeeds, before the cancellable wait.

try:
if snapshot.memory and snapshot.box is not None:
sandbox = await client.get(snapshot.box)
if sandbox.status == "PAUSED":

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High backends/prime.py:435

Discarding an older memory snapshot deletes the paused sandbox even when a newer kept snapshot uses that same box, so the kept restore point loses its live memory state. discard_checkpoints() checks only whether the box is PAUSED; since _resume() and the next _pause() preserve the same box ID, verify that the discarded snapshot is the box’s current pause checkpoint before deleting the box.

Also found in 1 other location(s)

verifiers/v1/agent.py:877

Cleanup protects only checkpoint IDs, not the paused box backing a kept memory checkpoint. If attempt 0 takes memory checkpoint A, a retry resumes that box and takes checkpoint B, then the run ends with memory_checkpoint.after='keep' (or is interrupted), B is kept but A goes into drop. Both snapshots have the same box: PrimeBackend._resume adopts snapshot.box, and _pause records that same ID. PrimeBackend.discard_checkpoints([A]) deletes this box whenever it is PAUSED, destroying B's preserved processes/memory even though B remains advertised as restorable. Protect boxes referenced by kept snapshots when discarding older memory checkpoints. Prime's Pause & Resume documentation confirms resume restores in place; its SDK Clean Exit documentation identifies delete as sandbox teardown.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/runtimes/backends/prime.py around line 435:

Discarding an older memory snapshot deletes the paused sandbox even when a newer kept snapshot uses that same box, so the kept restore point loses its live memory state. `discard_checkpoints()` checks only whether the box is `PAUSED`; since `_resume()` and the next `_pause()` preserve the same box ID, verify that the discarded snapshot is the box’s current pause checkpoint before deleting the box.

Also found in 1 other location(s):
- verifiers/v1/agent.py:877 -- Cleanup protects only checkpoint IDs, not the paused box backing a kept memory checkpoint. If attempt 0 takes memory checkpoint A, a retry resumes that box and takes checkpoint B, then the run ends with `memory_checkpoint.after='keep'` (or is interrupted), B is kept but A goes into `drop`. Both snapshots have the same `box`: `PrimeBackend._resume` adopts `snapshot.box`, and `_pause` records that same ID. `PrimeBackend.discard_checkpoints([A])` deletes this box whenever it is PAUSED, destroying B's preserved processes/memory even though B remains advertised as restorable. Protect boxes referenced by kept snapshots when discarding older memory checkpoints. Prime's Pause & Resume documentation confirms resume restores in place; its SDK Clean Exit documentation identifies `delete` as sandbox teardown.

Comment thread verifiers/v1/harness.py
taken: wait on its program, still running in the resumed box, and check its
result as `turn` does. `state` is what `memory_state` returned."""
async with boundary(HarnessError, f"harness {self.harness.config.id!r}"):
result = await self.runtime.attach(state["program"])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High v1/harness.py:469

After reattach() attaches the resumed program, self._program remains None, so a subsequent interruption or provider failure makes memory_state() return None and prevents another memory checkpoint. Runtime.attach does not update this field; assign state["program"] to self._program before awaiting attach.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/harness.py around line 469:

After `reattach()` attaches the resumed program, `self._program` remains `None`, so a subsequent interruption or provider failure makes `memory_state()` return `None` and prevents another memory checkpoint. `Runtime.attach` does not update this field; assign `state["program"]` to `self._program` before awaiting `attach`.

Comment on lines +395 to +400
raise SandboxError(f"prime pause failed: {e}") from e
if not checkpoints:
raise SandboxError(f"prime pause of {self.info.id} left no checkpoint")
snapshot = Snapshot(id=checkpoints[-1].id, memory=True, box=self.info.id)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium backends/prime.py:395

If wait_checkpoint() fails for a memory snapshot, the paused sandbox and checkpoint are orphaned: _pause() sets _paused=True, so teardown skips deleting the sandbox, while this snapshot is absent from stray_checkpoints and cannot be cleaned up. Add the snapshot to stray_checkpoints here; wait_checkpoint() already removes it once durable.

Suggested change
snapshot = Snapshot(id=checkpoints[-1].id, memory=True, box=self.info.id)
snapshot = Snapshot(id=checkpoints[-1].id, memory=True, box=self.info.id)
self.stray_checkpoints.append(snapshot)
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/runtimes/backends/prime.py around line 395:

If `wait_checkpoint()` fails for a memory snapshot, the paused sandbox and checkpoint are orphaned: `_pause()` sets `_paused=True`, so teardown skips deleting the sandbox, while this snapshot is absent from `stray_checkpoints` and cannot be cleaned up. Add the snapshot to `stray_checkpoints` here; `wait_checkpoint()` already removes it once durable.

Comment thread verifiers/v1/trace.py
return cls(
checkpoint=checkpoint,
tail=list(checkpoint.tail),
state=state.model_dump(mode="json")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High v1/trace.py:324

RestorePoint.of() raises UnicodeDecodeError for a checkpoint whose State.artifacts contains non-UTF-8 bytes, so DeltaStreamer.diff() and its final flush fail instead of exporting the restore point. model_dump(mode="json") applies Pydantic’s UTF-8 bytes encoding; preserve the bytes for the existing msgpack transport or encode them reversibly.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/trace.py around line 324:

`RestorePoint.of()` raises `UnicodeDecodeError` for a checkpoint whose `State.artifacts` contains non-UTF-8 bytes, so `DeltaStreamer.diff()` and its final flush fail instead of exporting the restore point. `model_dump(mode="json")` applies Pydantic’s UTF-8 bytes encoding; preserve the bytes for the existing msgpack transport or encode them reversibly.

Comment thread verifiers/v1/rollout.py
if self._memory_checkpoints or self._restored_memory:
# The harness reaches the interception through a relay in its box,
# which a resume reroutes to the new process's URL and secrets.
self._endpoint, self._secret = await self._route_relay(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High v1/rollout.py:501

When a memory restore reroutes the relay, outstanding model requests are retried against this session before restore() installs the checkpoint pins and marks the tail as intercepted. Those requests run hooks again without the pinned tool results, duplicating non-idempotent rewrites or state updates and potentially corrupting the resumed conversation. Install the restored session state before _route_relay() exposes it, or gate requests until restore preparation is complete.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/rollout.py around line 501:

When a memory restore reroutes the relay, outstanding model requests are retried against this session before `restore()` installs the checkpoint pins and marks the tail as intercepted. Those requests run hooks again without the pinned tool results, duplicating non-idempotent rewrites or state updates and potentially corrupting the resumed conversation. Install the restored session state before `_route_relay()` exposes it, or gate requests until restore preparation is complete.

Comment thread verifiers/v1/rollout.py
if self._session.checkpointer is not None and self._checkpointed is None:
pending = list(self.trace._pending.values())
if len(pending) == 1 or self._memory_checkpoints:
await self._checkpoint(pending[0] if len(pending) == 1 else None, error)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High v1/rollout.py:890

Interrupting during setup can report success while the rollout continues sampling: the memory-checkpoint branch calls _checkpoint() before _harness_session exists, so _capture() raises before self._session.abort(error) runs. Skip checkpoint capture until the harness session exists, and ensure the abort is delivered even if capture fails.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/rollout.py around line 890:

Interrupting during setup can report success while the rollout continues sampling: the memory-checkpoint branch calls `_checkpoint()` before `_harness_session` exists, so `_capture()` raises before `self._session.abort(error)` runs. Skip checkpoint capture until the harness session exists, and ensure the abort is delivered even if capture fails.

return {
"process": process.session,
"request": self._prompt,
"calls_before": len(self.trace.calls),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High acp/__init__.py:417

After a memory checkpoint taken after the prompt's model call commits, reattach() can raise ACP agent completed without committing a model turn even though that turn is retained. memory_state() stores the checkpoint-time len(self.trace.calls) instead of the prompt's original calls_before, so _require_model_turn() sees no calls after the baseline; preserve the original baseline in the checkpoint state.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/acp/__init__.py around line 417:

After a memory checkpoint taken after the prompt's model call commits, `reattach()` can raise `ACP agent completed without committing a model turn` even though that turn is retained. `memory_state()` stores the checkpoint-time `len(self.trace.calls)` instead of the prompt's original `calls_before`, so `_require_model_turn()` sees no calls after the baseline; preserve the original baseline in the checkpoint state.

Comment thread verifiers/v1/rollout.py
harness: dict[str, Any] | None = None
# Parallel sub-agents share the trace: with another request in flight the box
# is not idle, so no filesystem snapshot matches this turn's prefix.
if (

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High v1/rollout.py:935

A memory checkpoint that falls back to filesystem restore cannot resume an ACP session: harness is saved as {}, so ACP's restore() raises KeyError for acp_session_id and the rollout's prior progress is lost. _capture only calls checkpoint_state() when _checkpointing is enabled and turn is non-None, so memory-only checkpoints and interrupts between model calls omit that metadata; capture the restore metadata for memory checkpoints independently of those conditions.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/rollout.py around line 935:

A memory checkpoint that falls back to filesystem restore cannot resume an ACP session: `harness` is saved as `{}`, so ACP's `restore()` raises `KeyError` for `acp_session_id` and the rollout's prior progress is lost. `_capture` only calls `checkpoint_state()` when `_checkpointing` is enabled and `turn` is non-`None`, so memory-only checkpoints and interrupts between model calls omit that metadata; capture the restore metadata for memory checkpoints independently of those conditions.

Comment thread verifiers/v1/rollout.py
if self._session.checkpointer is not None and self._checkpointed is None:
pending = list(self.trace._pending.values())
if len(pending) == 1 or self._memory_checkpoints:
await self._checkpoint(pending[0] if len(pending) == 1 else None, error)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High v1/rollout.py:894

An interrupt checkpoint can restore a trace containing committed assistant/tool nodes and the original turn.tail whose filesystem effects are absent from the snapshot, leaving the restored rollout inconsistent and potentially replaying the tail. interrupt() awaits _checkpoint() before calling _session.abort(error), so the pending request can complete and the harness can act while _capture() is in progress; quiesce the in-flight sampling and harness activity before capturing the checkpoint.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/rollout.py around line 894:

An interrupt checkpoint can restore a trace containing committed assistant/tool nodes and the original `turn.tail` whose filesystem effects are absent from the snapshot, leaving the restored rollout inconsistent and potentially replaying the tail. `interrupt()` awaits `_checkpoint()` before calling `_session.abort(error)`, so the pending request can complete and the harness can act while `_capture()` is in progress; quiesce the in-flight sampling and harness activity before capturing the checkpoint.

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread verifiers/v1/rollout.py
taken = await asyncio.gather(
*(run.interrupt(timeout) for run in runs), return_exceptions=True
)
return len(runs), [run.trace.id for run, ok in zip(runs, taken) if ok is True]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interrupt count includes unfinished rollouts

Medium Severity

interrupt_all reports len(runs) as how many rollouts it interrupted, but interrupt now returns immediately for a trace that already has stop_condition set and does not abort it. Those rollouts stay in _LIVE through scoring and finalize, so InterruptResponse.interrupted overcounts them. Callers that treat “interrupted minus checkpointed” as work to start over will restart rollouts that are actually finishing.

Additional Locations (2)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit d930ea3. Configure here.

mikasenghaas and others added 5 commits November 3, 2026 18:39
A rollout checkpoints its box when a model call fails past its turn
retries, or when its owner interrupts it. A filesystem checkpoint
(`checkpoint`) restores the files into a new box; a memory checkpoint
(`memory_checkpoint`) resumes the same box with its processes. The
prime backend hides the two SDK paths behind `checkpoint(memory=...)`.

A loopback relay in the box lets a resumed harness reach the new
interception, and ACP harnesses reattach to their running runner
through the SDK's `attach_process`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An env server's `save` request takes a filesystem checkpoint of each
running rollout at its next model call, while the box is idle, and the
rollout goes on. An owner that saves its run (prime-rl's interval
checkpoints) can then continue its in-flight rollouts in another
process instead of starting them over.

The pool now sends `interrupt` and `save` to every worker and merges
their replies; before, one worker answered for its own rollouts only.

A final run also keeps every snapshot that `after="keep"` leaves to
its owner. Before, it deleted all but the newest, which an owner may
still hold.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A rollout being scored makes no more model calls, so its save would
wait out the whole timeout and hold up its owner's checkpoint. It now
takes none at once, and a save still waiting when the agent's segment
ends gives up then.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A `save` request with `hold` keeps each rollout that took a checkpoint
waiting at its model call until a `release` request, or for `hold`
seconds at most. Its owner then records the rollout as it is at its
checkpoint. Without the hold, the rollout went on while the others
saved, and a resume from that record redid those turns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A held save no longer waits for a rollout still setting up its box:
it has no turn to lose. Its first model call before the release takes
the checkpoint and holds there. Before, the rollout ran on unheld, and
a box slow to boot made the save wait out its timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wiz-prime-intellect

Copy link
Copy Markdown

Wiz Scan Summary

Scanner Findings
Vulnerability Finding Vulnerabilities -
Data Finding Sensitive Data -
Secret Finding Secrets -
IaC Misconfiguration IaC Misconfigurations -
SAST Finding SAST Findings 1 Medium 1 Low
Software Management Finding Software Management Findings -
Total 1 Medium 1 Low

View scan details in Wiz

To detect these findings earlier in the dev lifecycle, try the Wiz Code extension for VS Code, JetBrains, or Visual Studio.

Comment thread verifiers/v1/agent.py
Comment on lines +748 to +749
if restore is None and self.pending_restore is not None:
restore, self.pending_restore = self.pending_restore, None

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium v1/agent.py:748

run(task, runtime=box, restore=...) continues the saved conversation against box without restoring the checkpoint's filesystem, so the trace can claim checkpoint state that is absent and execution can modify the wrong workspace. Reject restores when runtime is borrowed, checking pending_restore before consuming it.

Suggested change
if restore is None and self.pending_restore is not None:
restore, self.pending_restore = self.pending_restore, None
if runtime is not None and (restore is not None or self.pending_restore is not None):
raise ValueError("cannot restore into a borrowed runtime")
if restore is None and self.pending_restore is not None:
restore, self.pending_restore = self.pending_restore, None
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/agent.py around lines 748-749:

`run(task, runtime=box, restore=...)` continues the saved conversation against `box` without restoring the checkpoint's filesystem, so the trace can claim checkpoint state that is absent and execution can modify the wrong workspace. Reject restores when `runtime` is borrowed, checking `pending_restore` before consuming it.

Comment thread verifiers/v1/trace.py
Comment on lines +1147 to +1151
c
for c in self.checkpoints
if c is not checkpoint and c.node is not None and c.node >= keep
]
self.checkpoints = [c for c in self.checkpoints if c not in rewound]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High v1/trace.py:1147

Rewinding to checkpoint A leaves later checkpoint B restorable when both were captured at the same node (or both have node=None), so Agent.run can resume from the abandoned B state and B’s snapshot is omitted from cleanup. rewind filters by node number instead of checkpoint order; prune every checkpoint after the selected checkpoint in self.checkpoints.

-        rewound = [
-            c
-            for c in self.checkpoints
-            if c is not checkpoint and c.node is not None and c.node >= keep
-        ]
-        self.checkpoints = [c for c in self.checkpoints if c not in rewound]
+        checkpoint_index = next(
+            i for i, c in enumerate(self.checkpoints) if c is checkpoint
+        )
+        rewound = self.checkpoints[checkpoint_index + 1 :]
+        self.checkpoints = self.checkpoints[: checkpoint_index + 1]
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/trace.py around lines 1147-1151:

Rewinding to checkpoint A leaves later checkpoint B restorable when both were captured at the same node (or both have `node=None`), so `Agent.run` can resume from the abandoned B state and B’s snapshot is omitted from cleanup. `rewind` filters by node number instead of checkpoint order; prune every checkpoint after the selected checkpoint in `self.checkpoints`.

Comment thread verifiers/v1/agent.py
Comment on lines +866 to +869
interrupted = (
run.trace.last_error is not None
and run.trace.last_error.type == "RolloutInterrupted"
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium v1/agent.py:866

A resumed trace that completes successfully still keeps its snapshot, so checkpoint.after="delete" leaks it on every interrupt/resume cycle. Trace.last_error includes the prior attempt's RolloutInterrupted because rewind preserves errors; check the current attempt's errors and require that it did not complete successfully before adding the checkpoint to keep.

-                interrupted = (
-                    run.trace.last_error is not None
-                    and run.trace.last_error.type == "RolloutInterrupted"
-                )
+                current_errors = run.trace.errors[seen:]
+                interrupted = (
+                    not run.ok
+                    and bool(current_errors)
+                    and current_errors[-1].type == "RolloutInterrupted"
+                )
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/agent.py around lines 866-869:

A resumed trace that completes successfully still keeps its snapshot, so `checkpoint.after="delete"` leaks it on every interrupt/resume cycle. `Trace.last_error` includes the prior attempt's `RolloutInterrupted` because rewind preserves errors; check the current attempt's errors and require that it did not complete successfully before adding the checkpoint to `keep`.

Comment thread verifiers/v1/session.py
message = request.messages[position]
if position >= len(branch) and isinstance(message, ToolMessage):
self.pinned_tool_results.setdefault(key, message)
self.prepared_users.update(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium v1/session.py:427

After memory restore, a rewritten user message is intercepted again, so a stateful hook can repeat side effects or produce a different prompt. mark_intercepted records only the rewritten hash, but the replay uses the original request body, whose hash does not match prepared_users; preserve and apply the original-to-rewritten mapping on replay.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/session.py around line 427:

After memory restore, a rewritten user message is intercepted again, so a stateful hook can repeat side effects or produce a different prompt. `mark_intercepted` records only the rewritten hash, but the replay uses the original request body, whose hash does not match `prepared_users`; preserve and apply the original-to-rewritten mapping on replay.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 3 total unresolved issues (including 2 from previous reviews).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 3ca463e. Configure here.

Comment thread verifiers/v1/agent.py
and run.trace.last_error.type == "RolloutInterrupted"
)
if last.snapshot is not None and interrupted:
keep.add(last.id)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Restored interrupt leaks final checkpoint

High Severity

Cleanup treats a run as interrupted whenever last_error is RolloutInterrupted, but rewind keeps prior attempts' errors. After another process restores an interrupted run and it finishes, that leftover error still skips discard. The checkpoint, including a billed paused sandbox, is left behind.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 3ca463e. Configure here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant