Skip to content

feat: resume retries from runtime checkpoints - #2721

Closed
mikasenghaas wants to merge 2 commits into
mainfrom
feat/checkpoint-resume
Closed

mikasenghaas wants to merge 2 commits into
mainfrom
feat/checkpoint-resume

Conversation

@mikasenghaas

@mikasenghaas mikasenghaas commented Sep 30, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Add a filesystem checkpoint contract to Runtime: supports_checkpoints, checkpoint(), restore(id), preserve_checkpoints(ids), discard_checkpoints(ids).
  • Implement it on the prime runtime with the checkpoint SDK from feat(sandboxes): add checkpoint, restore, and batched durability waits prime#971 (checkpoint + wait_for_checkpoint, restore via CreateSandboxRequest(checkpoint_id=...), DELETE /sandbox/checkpoints/{id}), and on the docker/podman runtimes with commit / rmi.
  • Checkpoint the box right before every model turn goes upstream (interception RolloutSession.checkpointer), when the agent has retries: the box is idle then, so the snapshot matches the prompt exactly. The call blocks until the snapshot is restorable (docker commit returns captured; prime waits for DURABLE), so the turn's own writes can never leak into it.
  • Record checkpoints on the trace (Trace.checkpoints, Checkpoint(id, node); the uncommitted prompt tail stays live-only) and stamp the box a run restored from on RuntimeInfo.restored_from.
  • Resume retries from the newest checkpoint by default (RetryConfig.checkpoint = True): Trace.rewind drops the nodes after the checkpoint and keeps the earlier turns sampled, the new box boots from the snapshot, task setup is skipped, and the harness continues the conversation (Harness.resume). A failed attempt bound for a retry keeps its checkpoints past teardown; everything else is discarded.
  • Resend a rewound trace from scratch on the env-server delta stream (Trace.generation), so a client assembling deltas ends with the trace the worker finished with.
  • Add the checkpoint-resume-v1 fixture (a task whose first attempt fails as turn 3 goes upstream) and e2e rows for docker and prime, in-process and through the env-server pool.
  • Pin prime-sandboxes to the PR branch (13c8688) until it is published.

Verification

  • pytest tests/v1/test_e2e.py -k "retry_resumes_from_checkpoint and docker" (live model, docker): the first attempt fails at turn 3, the retry restores the turn-2 checkpoint image, keeps turn 1 sampled, finishes with reward 1.0, trace.errors == [ProviderError], restored_from set; every checkpoint image is removed afterwards. Passes in-process and through the env-server pool.
  • pytest tests/v1 -m "not e2e" (incl. the new test_rewound_trace_streams_again_from_scratch), ruff, ty check verifiers: green.
  • The prime rows are NOT verified live: the platform API (PrimeIntellect-ai/platform#5914) is not deployed, so POST /sandbox/{id}/checkpoints 404s on prod. The prime runtime was checked against a fake client following that contract (blocking checkpoint through wait_for_checkpoint, a FAILED checkpoint surfacing as SandboxError on restore, restore request without image/disk, delete).

Overhead

Docker commit on this box costs about 3 ms per MB of the container's writable layer, paid on every turn (the layer is the diff against the image, not against the last checkpoint): 0.03 s clean, 0.35 s at 100 MB written, 1.6 s at 500 MB. Prime's per-turn wait is the capture plus the upload of the delta since the previous checkpoint; unmeasured until the API is deployed, bounded at 300 s.

🤖 Generated with Claude Code

Runtimes that can snapshot their filesystem (prime through the checkpoint
SDK, docker/podman through `commit`) checkpoint the box right before each
model turn goes upstream, while the box is idle. The trace records them,
and an agent retry resumes from the newest one instead of starting over:
the trace rewinds to the checkpoint (earlier turns stay sampled), the new
box boots from the snapshot, task setup is skipped, and the harness
continues the conversation.

A failed attempt bound for a retry keeps its checkpoints past teardown;
every other run discards them. Rewound traces stream again from scratch
over the env-server delta protocol, so a client assembling deltas ends
with the trace the worker finished with.

`prime-sandboxes` is pinned to the branch of PrimeIntellect-ai/prime#971
until the checkpoint SDK is published; the platform API it needs
(PrimeIntellect-ai/platform#5914) is not deployed yet, so the prime rows
of the e2e test are unverified live.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Wait for the snapshot with the SDK's `wait_for_checkpoint` before the
turn goes upstream, so it can never include that turn's writes and a
teardown (which fails pending checkpoints) never loses one. Preserving
checkpoints at close therefore needs no prime-specific wait anymore.

Pin `prime-sandboxes` to the branch commit that adds the method.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@mikasenghaas

Copy link
Copy Markdown
Member Author

Superseded by the experimental PR #2741 (same commits on exp/checkpoint-resume).

@mikasenghaas
mikasenghaas deleted the feat/checkpoint-resume branch October 1, 2026 21:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant