Skip to content

feat: resume retries from runtime checkpoints - #2741

Closed
mikasenghaas wants to merge 9 commits into
mainfrom
exp/checkpoint-resume
Closed

mikasenghaas wants to merge 9 commits into
mainfrom
exp/checkpoint-resume

Conversation

@mikasenghaas

@mikasenghaas mikasenghaas commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

Summary

Experimental: a retry resumes a rollout from the prime sandbox's newest filesystem checkpoint instead of starting over. One knob, retries.checkpoint (default on, effective only with a retry budget).

  • RuntimeBackend gains a filesystem checkpoint contract: SUPPORTS_CHECKPOINTS, checkpoint(), discard_checkpoints(ids). The prime backend implements it with the SDK's checkpoint, wait_for_checkpoint (batched durability polls via checkpoints:batchGet) and delete_checkpoint. A resumed attempt boots its backend from the runtime config with checkpoint set to the snapshot (PrimeConfig.checkpoint), so the trace's runtime info names the snapshot the box came from.
  • The box is snapshotted right before every model turn goes upstream (interception RolloutSession.checkpointer), when the box is idle, so the snapshot matches the prompt exactly. The call blocks until the snapshot is DURABLE. No snapshot is taken while another request shares the trace.
  • The trace records the snapshots (Trace.checkpoints: Checkpoint(id, node, resumed); the uncommitted prompt tail and the typed state at snapshot time stay live-only), stamps each error with the attempt that captured it and the stage it broke in (Error.attempt, Error.stage), and marks the checkpoint a retry resumed from (trace.has_resumed).
  • Resume: Trace.rewind drops the nodes after the checkpoint, keeps the earlier turns sampled, restores the typed state, resets the outcome stamps (rewards, dropped, stage spans) and dangling semantic links; the new box boots from the snapshot, task setup is skipped, and the harness continues the conversation (Harness.resume). Agent.run frees every snapshot its attempts took once the run is final (TODO in code: retaining snapshots beyond the run); a cancelled checkpoint deletes its snapshot; deletion happens after the box is gone, since the platform refuses to delete a checkpoint a live sandbox descends from.
  • A rewound trace is resent from scratch on the env-server delta stream (Trace.generation), with the reopen held until the discard has left the wire, so a client assembling deltas ends with the trace the worker finished with.
  • Fixture checkpoint-resume-v1 (first attempt fails as turn 3 goes upstream; setup stamps a nonce the prompt never reveals) and tests/integration/test_checkpoint.py on prime.

What the tests confirm

  • tests/integration/test_checkpoint.py -m prime, against the live platform, green after every revision including the merge with main: the first attempt fails as turn 3 goes upstream (2 turns done), the retry restores the turn-2 snapshot into a new sandbox, keeps turn 1 sampled, and finishes with reward 1.0; trace.errors == [ProviderError] stamped attempt 0 / stage agent, the restored checkpoint is the one marked resumed, every kept turn has its call, one branch. About 50 s wall clock.
  • A uv run eval of the same fixture through prime-rl's launcher-owned env server (the training path) did the same and scored 1.0, so the rewind survives the delta stream.
  • A real terminal-bench/fix-git rollout (bash harness, 9 turns, reward 1.0, 91 s) with checkpoints on: stored sizes 53.8 MB for the first snapshot (the delta since boot, mostly the harness install) and 174 B to 72 KB for each later one (incremental); durability waits 10.9 s for the first, 4.0 to 6.8 s after; all 9 snapshots deleted at the end.
  • Raw SDK on prod: checkpoint POST 0.9 s, durable after 3.0 s (tiny delta) and 10.7 s (100 MB delta), restore to running in 4.2 s, chained checkpoints, the restored box holds only the checkpointed files, checkpoints of a deleted sandbox delete at once while a live one's 409.
  • pytest tests/unit green (includes the rewound-trace delta tests, one with a failing discard send), ruff, ty clean on the touched files.

Known shortcomings

  • Concurrent sub-agents: the guard only skips a snapshot while another model request is in flight. A sub-agent that is mid-tool between requests can be snapshotted mid-write, and rewind/Harness.resume cut a multi-branch graph by node index, so a resume on such a trace can be inconsistent. Fix planned: snapshot and resume only while the graph is a single path.
  • Filesystem only: processes the agent left running (a dev server) are not in the snapshot; a restore cold boots.
  • Only harnesses that relaunch on a transcript resume (SUPPORTS_RESUME: bash, null, browser-use); the ACP harnesses take no checkpoints and retry from scratch, since their session resume starts a fresh process on user turns. Native session resume from the restored box's session files (claude-code --resume, codex resume) is not wired.
  • Cost: every turn waits for its snapshot, about 4 s in steady state, more for large writes (about 11 s per 100 MB); bounded at 300 s. On a short task like fix-git that roughly doubles wall clock. Not yet measured at training scale.
  • Snapshots live only as long as the run; nothing retains or reuses them.
  • Only the prime runtime; single-target deployments only (no Compose services, no tool-server VMs).

Roadmap

  1. Single-path guard for sub-agent harnesses, and native session resume for the ACP harnesses from their on-disk session files.
  2. Retain snapshots beyond the run (forks, inspection), with retention policy and cleanup.
  3. Async snapshot with an in-box position marker, since the per-turn wait is the dominant cost.
  4. Memory checkpoints once the platform offers them: no replay, any harness, stable interception session.
  5. Other runtimes (modal snapshot_filesystem, docker commit).

🤖 Generated with Claude Code


Note

High Risk
Changes core rollout retry, trace lifecycle, and sandbox provisioning on Prime; default-on checkpoint behavior adds per-turn latency and can affect all retried agent runs.

Overview
Adds experimental checkpoint resume for agent retries: when retries.checkpoint is on (default) and the runtime supports it, failed rollouts rewind the same trace and boot a new sandbox from the latest per-turn snapshot instead of restarting from scratch.

Retry path: Agent.run picks the newest Trace.checkpoints entry, calls Trace.rewind (truncate nodes/calls/scoring, restore typed state, bump generation), and opens a Rollout that binds the runtime with PrimeConfig.checkpoint set—task setup is skipped, harness continues via new Rollout.resume() / step(..., intercept=False) on the checkpoint tail. Errors are tagged with attempt and stage; checkpoint resume requires SUPPORTS_RESUME harnesses (e.g. bash).

Snapshots: Before each model turn goes upstream, interception invokes RolloutSession.checkpointer → Prime checkpoint() (blocks until durable). Snapshots are discarded when the run finishes.

Streaming: DeltaStreamer emits discard when trace generation changes so clients resync after rewind (reopen ordered after discard on send failure).

Tests: Integration fixture checkpoint-resume-v1 (mid-rollout ProviderError, resume keeps filesystem work); unit test for rewound delta stream.

Reviewed by Cursor Bugbot for commit 404a6dc. Bugbot is set up for automated code reviews on this repo. Configure here.

Note

Add checkpoint-based resume for agent retries via runtime snapshots

  • Agent.run retries can now resume from the latest trace checkpoint instead of restarting, when checkpoint retries are enabled and the runtime supports them (RetryConfig.checkpoint defaults to true).
  • Trace gains checkpoints, a generation counter, and a rewind method that restores graph, state, and timing to a checkpoint while clearing post-checkpoint results.
  • PrimeBackend implements durable snapshot checkpoint and best-effort discard_checkpoints; RuntimeBackend gains a SUPPORTS_CHECKPOINTS flag that defaults to false.
  • The delta streamer now sends trace checkpoints incrementally and instructs consumers to discard and re-stream a trace when its generation changes after a rewind; failed delta sends no longer let later deltas overtake them.
  • Recorded trace errors carry attempt and stage metadata, and successful rollouts report their stop condition rather than a stale last error.
  • Integration coverage added in test_checkpoint.py with a checkpoint-resume fixture, plus unit tests for rewind streaming.
  • Behavioral Change: Checkpoint is newly exported from verifiers.v1; rewound traces stream as a new generation, so existing delta consumers must handle the discard/reset sequence.
📊 Macroscope summarized 404a6dc. 10 files reviewed, 6 issues evaluated, 6 issues filtered, 0 comments posted

🗂️ Filtered Issues

verifiers/v1/agent.py — 0 comments posted, 1 evaluated, 1 filtered
  • line 760: Checkpoint cleanup is only on the normal return path. If Agent.run() is cancelled during the retry backoff at await asyncio.sleep(delay), the previous rollout has already closed, so _run_once()'s abort handler cannot run, and execution never reaches discard_checkpoints() here. All snapshots from that attempt are left undiscarded. Cancellation during a subsequent from-scratch attempt also leaves earlier attempts' snapshots behind because Rollout.abort() only knows its own trace. Put cleanup of the accumulated snapshots in a shielded finally covering the retry loop. [ Already posted ]
verifiers/v1/rollout.py — 0 comments posted, 2 evaluated, 2 filtered
  • line 707: System-message deduplication consults trace.task.data, not the data actually passed to the harness session. With no task system prompt and a restricted network plus network_notice=True, open() injects a system prompt into harness_data only. If the first model call fails after its checkpoint, its tail includes that generated system message, but this filter retains it because the original data.system_prompt is None. Harness.resume() and the bash/null chat program then prepend the session's system prompt again, so the retry sends two copies instead of the checkpointed prompt. Use the session's effective system-prompt placement when filtering the tail. [ Already posted ]
  • line 709: resume() passes the checkpoint tail with intercept=False, but that only bypasses step()'s pre-harness preparation. The resumed model request still goes through InterceptionServer.sample() → RolloutSession.rewrite_request(). The new session has empty pinned_tool_results and prepared_users, so already-intercepted, uncommitted tool/user messages in the saved tail are intercepted again. A non-idempotent hook (e.g. appending context or transforming a tool result) therefore changes the supposedly restored prompt a second time and can repeat hook side effects. Restore the relevant interception state or mark the saved tail as already prepared/pinned before resampling. [ Already posted ]
verifiers/v1/runtimes/backends/prime.py — 0 comments posted, 2 evaluated, 2 filtered
  • line 293: Cancellation during the checkpoint POST leaks the snapshot ID. run_shielded (verifiers/v1/utils/aio.py) finishes the inner operation but then re-raises cancellation instead of returning its result. Thus cancellation at this await never assigns checkpoint and bypasses the cancellation cleanup at lines 301–303 entirely. Rollout._checkpoint records IDs only after this method returns, so neither abort nor final-run cleanup can discard this snapshot. Capture the ID inside the shielded coroutine and cover that await with cancellation cleanup, as start() does for sandbox IDs. [ Already posted ]
  • line 301: Checkpoint-wait failures leave snapshots outside post-teardown cleanup. On cancellation this calls discard_checkpoints while the sandbox is still running and the snapshot may still be pending. The pinned prime-sandboxes 0.5.1 AsyncSandboxClient.delete_checkpoint documentation (prime repo, publishing commit 3f818cd4e8b42e1c5a0418cebad3ca88804ed273, packages/prime-sandboxes/src/prime_sandboxes/sandbox.py) explicitly says deletion returns HTTP 409 for pending checkpoints or a running sandbox's active tip. discard_checkpoints swallows that rejection; the ID is never returned to Rollout._checkpoint and therefore never enters the trace's later cleanup list. The ordinary exception branch also drops the ID without attempting cleanup when durability polling fails or times out. Retain unsuccessful checkpoint IDs and discard them after sandbox teardown, otherwise these snapshots escape the run's storage cleanup. [ Already posted ]
verifiers/v1/trace.py — 0 comments posted, 1 evaluated, 1 filtered
  • line 841: Resetting timing while retaining earlier model calls makes the resumed attempt's model/harness timing incorrect. Rollout.open starts a new agent span and Rollout.close calls split_agent_time(), which sums durations of all retained self.calls, including calls before the checkpoint. For example, retaining 60 seconds of earlier inference before a retry lasting 20 seconds causes the entire new span to be reported as model time and harness time as zero, even if most of the retry was tool execution. Restrict the split to calls in the new span, or retain compatible cumulative timing. [ Already posted ]

Comment thread verifiers/v1/runtimes/prime.py Resolved
Comment thread verifiers/v1/trace.py Resolved
Comment thread verifiers/v1/trace.py Resolved
Comment thread verifiers/v1/interception/server.py Resolved
Comment thread verifiers/v1/serve/delta.py Resolved
Comment thread verifiers/v1/trace.py Resolved
Comment thread verifiers/v1/runtimes/prime.py Resolved
Comment thread tests/v1/test_e2e.py Resolved

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread verifiers/v1/rollout.py Resolved
Comment thread verifiers/v1/rollout.py Resolved
Comment thread verifiers/v1/rollout.py Resolved
Comment thread verifiers/v1/trace.py Resolved
Comment thread verifiers/v1/trace.py Resolved
Comment thread verifiers/v1/runtimes/backends/prime.py Resolved

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread verifiers/v1/runtimes/backends/prime.py Resolved
@macroscopeapp

macroscopeapp Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Needs Human Approval

Macroscope's review found this PR requires human approval — This PR introduces a substantial cross-cutting checkpoint/resume capability that changes retry behavior, filesystem restoration, conversation traces, interception, streaming, and Prime runtime lifecycle handling. The supplied unresolved findings also identify material failure-mode and state-consistency risks requiring human review.

Needs human approval because:

  • 5 blocking correctness issues found at or above your repo's Minimum Blocking Severity

Adjust the Minimum Blocking Severity for this repo — including turning it Off — in Settings. You can add or adjust custom eligibility rules. Learn more.

The prime runtime checkpoints the sandbox filesystem right before each
model turn goes upstream, while the box is idle, and blocks until the
snapshot is DURABLE (the SDK's `wait_for_checkpoint`, whose polls are
batched across every rollout sharing the client). The trace records the
checkpoints with the typed state as of then; an agent retry resumes from
the newest one instead of starting over: the trace rewinds to it (earlier
turns stay sampled, the state and stage spans reset to match), the new
box boots from the snapshot, task setup is skipped, and the harness
continues the conversation.

A failed attempt bound for a retry keeps its checkpoints past teardown;
every other run discards them once its box is gone, since the platform
refuses to delete a checkpoint a live sandbox descends from. A cancelled
checkpoint deletes its snapshot, and no snapshot is taken while another
request shares the trace. Rewound traces stream again from scratch over
the env-server delta protocol, with the reopen held until the discard
has left the wire.

The retry verdict (`RetryState.next_error` consumes a rule budget) is
decided once per attempt and shared between the rollout's close and the
retry loop.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@mikasenghaas
mikasenghaas force-pushed the exp/checkpoint-resume branch from 4a5a479 to 0e6d961 Compare October 14, 2026 10:01
Comment thread verifiers/v1/rollout.py Resolved
Comment thread verifiers/v1/agent.py Resolved
mikasenghaas added a commit to PrimeIntellect-ai/prime-rl that referenced this pull request Oct 14, 2026
Bump deps/verifiers to PrimeIntellect-ai/verifiers#2741: the prime
runtime checkpoints the sandbox filesystem before every model turn, and
agent retries resume the same trace from the newest checkpoint instead
of starting over. Rewound traces stream again from scratch, so the
orchestrator assembles the trace the worker finished with.

Bump deps/renderers to 0.1.12 (verifiers main requires it) and relock,
which moves prime-sandboxes to 0.4.4, the released checkpoint SDK.

Add an eval integration test: one bash-harness rollout in a prime
sandbox on verifiers' checkpoint-resume-v1 fixture, whose first attempt
fails as its third turn goes upstream; the run's logs must show the
retry resuming from a checkpoint and the resumed rollout scoring 1.0.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
mikasenghaas and others added 2 commits October 14, 2026 10:19
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`Trace.attempts` holds one `Attempt` per run of the rollout (its box, the
checkpoint it resumed from, the turns it kept, its stage timing, how it
ended); errors and checkpoints point back to their attempt by index, and
errors name the stage they broke in. `has_resumed` and `num_attempts`
derive from it; `Trace.timing` is the final attempt's spans.

Every checkpoint is durable when taken, so `preserve_checkpoints` goes.
`Agent.run` frees all snapshots its attempts took once the run is final
(a TODO marks retaining them beyond the run); the rollout no longer needs
a retry verdict at close.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Comment thread verifiers/v1/rollout.py Resolved

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit b08a804. Configure here.

Comment thread verifiers/v1/agent.py Resolved
Comment thread verifiers/v1/agent.py Resolved
mikasenghaas and others added 6 commits October 14, 2026 13:12
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`AgentConfig.checkpoint` decides whether a run snapshots its box before
every turn (off by default: each turn waits for its snapshot);
`RetryConfig.from_checkpoint` decides whether a retry resumes from the
newest one (on by default) or starts from scratch.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Errors carry the attempt that captured them and the stage it broke in;
the checkpoint a retry resumed from is marked. That answers whether and
where a run retried without a second per-attempt structure.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`retries.checkpoint` both takes the snapshots and resumes from them;
checkpointing without resuming has no use yet. Its docstring names the
feature experimental and lists the known limits.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A checkpoint resume relaunches the program on the trace's transcript.
ACP harnesses only have a live-session resume that starts a fresh process
on user turns, so they retry from scratch and take no checkpoints.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Main split runtimes into a Runtime view over a RuntimeBackend that a
Deployment owns and starts. The checkpoint contract moves onto the
backend (SUPPORTS_CHECKPOINTS, checkpoint, discard_checkpoints); the
restore hook goes, since a resumed attempt boots its backend from the
runtime config with `checkpoint` set to the snapshot, which main's
PrimeConfig already supports and which the trace's runtime info then
names. Rewind also clears `dropped`, the new consumer-side flag main
added, since a rewound trace is scored again.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@mikasenghaas
mikasenghaas deleted the exp/checkpoint-resume branch October 27, 2026 18:51
@mikasenghaas mikasenghaas changed the title exp: resume retries from runtime checkpoints feat: resume retries from runtime checkpoints Oct 27, 2026
@mikasenghaas

Copy link
Copy Markdown
Member Author

Continued in #2994. GitHub closed this PR when its branch was renamed to feat/rollout-ckpt.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant