Repository navigation
feat: resume retries from runtime checkpoints - #2741
mikasenghaas wants to merge 9 commits into
Conversation
ApprovabilityVerdict: Needs Human Approval Macroscope's review found this PR requires human approval — This PR introduces a substantial cross-cutting checkpoint/resume capability that changes retry behavior, filesystem restoration, conversation traces, interception, streaming, and Prime runtime lifecycle handling. The supplied unresolved findings also identify material failure-mode and state-consistency risks requiring human review. Needs human approval because:
Adjust the Minimum Blocking Severity for this repo — including turning it Off — in Settings. You can add or adjust custom eligibility rules. Learn more. |
The prime runtime checkpoints the sandbox filesystem right before each model turn goes upstream, while the box is idle, and blocks until the snapshot is DURABLE (the SDK's `wait_for_checkpoint`, whose polls are batched across every rollout sharing the client). The trace records the checkpoints with the typed state as of then; an agent retry resumes from the newest one instead of starting over: the trace rewinds to it (earlier turns stay sampled, the state and stage spans reset to match), the new box boots from the snapshot, task setup is skipped, and the harness continues the conversation. A failed attempt bound for a retry keeps its checkpoints past teardown; every other run discards them once its box is gone, since the platform refuses to delete a checkpoint a live sandbox descends from. A cancelled checkpoint deletes its snapshot, and no snapshot is taken while another request shares the trace. Rewound traces stream again from scratch over the env-server delta protocol, with the reopen held until the discard has left the wire. The retry verdict (`RetryState.next_error` consumes a rule budget) is decided once per attempt and shared between the rollout's close and the retry loop. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
4a5a479 to
0e6d961
Compare
Bump deps/verifiers to PrimeIntellect-ai/verifiers#2741: the prime runtime checkpoints the sandbox filesystem before every model turn, and agent retries resume the same trace from the newest checkpoint instead of starting over. Rewound traces stream again from scratch, so the orchestrator assembles the trace the worker finished with. Bump deps/renderers to 0.1.12 (verifiers main requires it) and relock, which moves prime-sandboxes to 0.4.4, the released checkpoint SDK. Add an eval integration test: one bash-harness rollout in a prime sandbox on verifiers' checkpoint-resume-v1 fixture, whose first attempt fails as its third turn goes upstream; the run's logs must show the retry resuming from a checkpoint and the resumed rollout scoring 1.0. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`Trace.attempts` holds one `Attempt` per run of the rollout (its box, the checkpoint it resumed from, the turns it kept, its stage timing, how it ended); errors and checkpoints point back to their attempt by index, and errors name the stage they broke in. `has_resumed` and `num_attempts` derive from it; `Trace.timing` is the final attempt's spans. Every checkpoint is durable when taken, so `preserve_checkpoints` goes. `Agent.run` frees all snapshots its attempts took once the run is final (a TODO marks retaining them beyond the run); the rollout no longer needs a retry verdict at close. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit b08a804. Configure here.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`AgentConfig.checkpoint` decides whether a run snapshots its box before every turn (off by default: each turn waits for its snapshot); `RetryConfig.from_checkpoint` decides whether a retry resumes from the newest one (on by default) or starts from scratch. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Errors carry the attempt that captured them and the stage it broke in; the checkpoint a retry resumed from is marked. That answers whether and where a run retried without a second per-attempt structure. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
`retries.checkpoint` both takes the snapshots and resumes from them; checkpointing without resuming has no use yet. Its docstring names the feature experimental and lists the known limits. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A checkpoint resume relaunches the program on the trace's transcript. ACP harnesses only have a live-session resume that starts a fresh process on user turns, so they retry from scratch and take no checkpoints. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Main split runtimes into a Runtime view over a RuntimeBackend that a Deployment owns and starts. The checkpoint contract moves onto the backend (SUPPORTS_CHECKPOINTS, checkpoint, discard_checkpoints); the restore hook goes, since a resumed attempt boots its backend from the runtime config with `checkpoint` set to the snapshot, which main's PrimeConfig already supports and which the trace's runtime info then names. Rewind also clears `dropped`, the new consumer-side flag main added, since a rewound trace is scored again. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Continued in #2994. GitHub closed this PR when its branch was renamed to |

Summary
Experimental: a retry resumes a rollout from the prime sandbox's newest filesystem checkpoint instead of starting over. One knob,
retries.checkpoint(default on, effective only with a retry budget).RuntimeBackendgains a filesystem checkpoint contract:SUPPORTS_CHECKPOINTS,checkpoint(),discard_checkpoints(ids). The prime backend implements it with the SDK'scheckpoint,wait_for_checkpoint(batched durability polls viacheckpoints:batchGet) anddelete_checkpoint. A resumed attempt boots its backend from the runtime config withcheckpointset to the snapshot (PrimeConfig.checkpoint), so the trace's runtime info names the snapshot the box came from.RolloutSession.checkpointer), when the box is idle, so the snapshot matches the prompt exactly. The call blocks until the snapshot isDURABLE. No snapshot is taken while another request shares the trace.Trace.checkpoints:Checkpoint(id, node, resumed); the uncommitted prompt tail and the typed state at snapshot time stay live-only), stamps each error with the attempt that captured it and the stage it broke in (Error.attempt,Error.stage), and marks the checkpoint a retry resumed from (trace.has_resumed).Trace.rewinddrops the nodes after the checkpoint, keeps the earlier turns sampled, restores the typed state, resets the outcome stamps (rewards,dropped, stage spans) and dangling semantic links; the new box boots from the snapshot, tasksetupis skipped, and the harness continues the conversation (Harness.resume).Agent.runfrees every snapshot its attempts took once the run is final (TODO in code: retaining snapshots beyond the run); a cancelled checkpoint deletes its snapshot; deletion happens after the box is gone, since the platform refuses to delete a checkpoint a live sandbox descends from.Trace.generation), with the reopen held until the discard has left the wire, so a client assembling deltas ends with the trace the worker finished with.checkpoint-resume-v1(first attempt fails as turn 3 goes upstream;setupstamps a nonce the prompt never reveals) andtests/integration/test_checkpoint.pyon prime.What the tests confirm
tests/integration/test_checkpoint.py -m prime, against the live platform, green after every revision including the merge with main: the first attempt fails as turn 3 goes upstream (2 turns done), the retry restores the turn-2 snapshot into a new sandbox, keeps turn 1 sampled, and finishes with reward 1.0;trace.errors == [ProviderError]stamped attempt 0 / stageagent, the restored checkpoint is the one markedresumed, every kept turn has its call, one branch. About 50 s wall clock.uv run evalof the same fixture through prime-rl's launcher-owned env server (the training path) did the same and scored 1.0, so the rewind survives the delta stream.terminal-bench/fix-gitrollout (bash harness, 9 turns, reward 1.0, 91 s) with checkpoints on: stored sizes 53.8 MB for the first snapshot (the delta since boot, mostly the harness install) and 174 B to 72 KB for each later one (incremental); durability waits 10.9 s for the first, 4.0 to 6.8 s after; all 9 snapshots deleted at the end.pytest tests/unitgreen (includes the rewound-trace delta tests, one with a failing discard send),ruff,tyclean on the touched files.Known shortcomings
rewind/Harness.resumecut a multi-branch graph by node index, so a resume on such a trace can be inconsistent. Fix planned: snapshot and resume only while the graph is a single path.SUPPORTS_RESUME: bash, null, browser-use); the ACP harnesses take no checkpoints and retry from scratch, since their session resume starts a fresh process on user turns. Native session resume from the restored box's session files (claude-code--resume, codexresume) is not wired.Roadmap
snapshot_filesystem, dockercommit).🤖 Generated with Claude Code
Note
High Risk
Changes core rollout retry, trace lifecycle, and sandbox provisioning on Prime; default-on checkpoint behavior adds per-turn latency and can affect all retried agent runs.
Overview
Adds experimental checkpoint resume for agent retries: when
retries.checkpointis on (default) and the runtime supports it, failed rollouts rewind the same trace and boot a new sandbox from the latest per-turn snapshot instead of restarting from scratch.Retry path:
Agent.runpicks the newestTrace.checkpointsentry, callsTrace.rewind(truncate nodes/calls/scoring, restore typed state, bumpgeneration), and opens aRolloutthat binds the runtime withPrimeConfig.checkpointset—tasksetupis skipped, harness continues via newRollout.resume()/step(..., intercept=False)on the checkpoint tail. Errors are tagged withattemptandstage; checkpoint resume requiresSUPPORTS_RESUMEharnesses (e.g. bash).Snapshots: Before each model turn goes upstream, interception invokes
RolloutSession.checkpointer→ Primecheckpoint()(blocks until durable). Snapshots are discarded when the run finishes.Streaming:
DeltaStreameremitsdiscardwhen tracegenerationchanges so clients resync after rewind (reopen ordered after discard on send failure).Tests: Integration fixture
checkpoint-resume-v1(mid-rolloutProviderError, resume keeps filesystem work); unit test for rewound delta stream.Reviewed by Cursor Bugbot for commit 404a6dc. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Add checkpoint-based resume for agent retries via runtime snapshots
Agent.runretries can now resume from the latest trace checkpoint instead of restarting, when checkpoint retries are enabled and the runtime supports them (RetryConfig.checkpointdefaults to true).Tracegainscheckpoints, agenerationcounter, and arewindmethod that restores graph, state, and timing to a checkpoint while clearing post-checkpoint results.PrimeBackendimplements durable snapshotcheckpointand best-effortdiscard_checkpoints;RuntimeBackendgains aSUPPORTS_CHECKPOINTSflag that defaults to false.checkpointsincrementally and instructs consumers to discard and re-stream a trace when its generation changes after a rewind; failed delta sends no longer let later deltas overtake them.attemptandstagemetadata, and successful rollouts report their stop condition rather than a stale last error.Checkpointis newly exported fromverifiers.v1; rewound traces stream as a new generation, so existing delta consumers must handle the discard/reset sequence.📊 Macroscope summarized 404a6dc. 10 files reviewed, 6 issues evaluated, 6 issues filtered, 0 comments posted
🗂️ Filtered Issues
verifiers/v1/agent.py — 0 comments posted, 1 evaluated, 1 filtered
Agent.run()is cancelled during the retry backoff atawait asyncio.sleep(delay), the previous rollout has already closed, so_run_once()'s abort handler cannot run, and execution never reachesdiscard_checkpoints()here. All snapshots from that attempt are left undiscarded. Cancellation during a subsequent from-scratch attempt also leaves earlier attempts' snapshots behind becauseRollout.abort()only knows its own trace. Put cleanup of the accumulatedsnapshotsin a shieldedfinallycovering the retry loop. [ Already posted ]verifiers/v1/rollout.py — 0 comments posted, 2 evaluated, 2 filtered
trace.task.data, not the data actually passed to the harness session. With no task system prompt and a restricted network plusnetwork_notice=True,open()injects a system prompt intoharness_dataonly. If the first model call fails after its checkpoint, its tail includes that generated system message, but this filter retains it because the originaldata.system_promptisNone.Harness.resume()and the bash/null chat program then prepend the session's system prompt again, so the retry sends two copies instead of the checkpointed prompt. Use the session's effective system-prompt placement when filtering the tail. [ Already posted ]resume()passes the checkpoint tail withintercept=False, but that only bypassesstep()'s pre-harness preparation. The resumed model request still goes throughInterceptionServer.sample()→RolloutSession.rewrite_request(). The new session has emptypinned_tool_resultsandprepared_users, so already-intercepted, uncommitted tool/user messages in the saved tail are intercepted again. A non-idempotent hook (e.g. appending context or transforming a tool result) therefore changes the supposedly restored prompt a second time and can repeat hook side effects. Restore the relevant interception state or mark the saved tail as already prepared/pinned before resampling. [ Already posted ]verifiers/v1/runtimes/backends/prime.py — 0 comments posted, 2 evaluated, 2 filtered
run_shielded(verifiers/v1/utils/aio.py) finishes the inner operation but then re-raises cancellation instead of returning its result. Thus cancellation at this await never assignscheckpointand bypasses the cancellation cleanup at lines 301–303 entirely.Rollout._checkpointrecords IDs only after this method returns, so neither abort nor final-run cleanup can discard this snapshot. Capture the ID inside the shielded coroutine and cover that await with cancellation cleanup, asstart()does for sandbox IDs. [ Already posted ]discard_checkpointswhile the sandbox is still running and the snapshot may still be pending. The pinned prime-sandboxes 0.5.1AsyncSandboxClient.delete_checkpointdocumentation (prime repo, publishing commit3f818cd4e8b42e1c5a0418cebad3ca88804ed273, packages/prime-sandboxes/src/prime_sandboxes/sandbox.py) explicitly says deletion returns HTTP 409 for pending checkpoints or a running sandbox's active tip.discard_checkpointsswallows that rejection; the ID is never returned toRollout._checkpointand therefore never enters the trace's later cleanup list. The ordinary exception branch also drops the ID without attempting cleanup when durability polling fails or times out. Retain unsuccessful checkpoint IDs and discard them after sandbox teardown, otherwise these snapshots escape the run's storage cleanup. [ Already posted ]verifiers/v1/trace.py — 0 comments posted, 1 evaluated, 1 filtered
timingwhile retaining earlier model calls makes the resumed attempt's model/harness timing incorrect.Rollout.openstarts a new agent span andRollout.closecallssplit_agent_time(), which sums durations of all retainedself.calls, including calls before the checkpoint. For example, retaining 60 seconds of earlier inference before a retry lasting 20 seconds causes the entire new span to be reported as model time and harness time as zero, even if most of the retry was tool execution. Restrict the split to calls in the new span, or retain compatible cumulative timing. [ Already posted ]