Repository navigation
feat: resume retries from runtime checkpoints - #2721
Closed
mikasenghaas wants to merge 2 commits into
Closed
mikasenghaas wants to merge 2 commits into
mikasenghaas wants to merge 2 commits into
Conversation
Runtimes that can snapshot their filesystem (prime through the checkpoint SDK, docker/podman through `commit`) checkpoint the box right before each model turn goes upstream, while the box is idle. The trace records them, and an agent retry resumes from the newest one instead of starting over: the trace rewinds to the checkpoint (earlier turns stay sampled), the new box boots from the snapshot, task setup is skipped, and the harness continues the conversation. A failed attempt bound for a retry keeps its checkpoints past teardown; every other run discards them. Rewound traces stream again from scratch over the env-server delta protocol, so a client assembling deltas ends with the trace the worker finished with. `prime-sandboxes` is pinned to the branch of PrimeIntellect-ai/prime#971 until the checkpoint SDK is published; the platform API it needs (PrimeIntellect-ai/platform#5914) is not deployed yet, so the prime rows of the e2e test are unverified live. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Wait for the snapshot with the SDK's `wait_for_checkpoint` before the turn goes upstream, so it can never include that turn's writes and a teardown (which fails pending checkpoints) never loses one. Preserving checkpoints at close therefore needs no prime-specific wait anymore. Pin `prime-sandboxes` to the branch commit that adds the method. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Member
Author
|
Superseded by the experimental PR #2741 (same commits on |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Runtime:supports_checkpoints,checkpoint(),restore(id),preserve_checkpoints(ids),discard_checkpoints(ids).checkpoint+wait_for_checkpoint, restore viaCreateSandboxRequest(checkpoint_id=...),DELETE /sandbox/checkpoints/{id}), and on the docker/podman runtimes withcommit/rmi.RolloutSession.checkpointer), when the agent has retries: the box is idle then, so the snapshot matches the prompt exactly. The call blocks until the snapshot is restorable (dockercommitreturns captured; prime waits forDURABLE), so the turn's own writes can never leak into it.Trace.checkpoints,Checkpoint(id, node); the uncommitted prompt tail stays live-only) and stamp the box a run restored from onRuntimeInfo.restored_from.RetryConfig.checkpoint = True):Trace.rewinddrops the nodes after the checkpoint and keeps the earlier turns sampled, the new box boots from the snapshot, tasksetupis skipped, and the harness continues the conversation (Harness.resume). A failed attempt bound for a retry keeps its checkpoints past teardown; everything else is discarded.Trace.generation), so a client assembling deltas ends with the trace the worker finished with.checkpoint-resume-v1fixture (a task whose first attempt fails as turn 3 goes upstream) and e2e rows for docker and prime, in-process and through the env-server pool.prime-sandboxesto the PR branch (13c8688) until it is published.Verification
pytest tests/v1/test_e2e.py -k "retry_resumes_from_checkpoint and docker"(live model, docker): the first attempt fails at turn 3, the retry restores the turn-2 checkpoint image, keeps turn 1 sampled, finishes with reward 1.0,trace.errors == [ProviderError],restored_fromset; every checkpoint image is removed afterwards. Passes in-process and through the env-server pool.pytest tests/v1 -m "not e2e"(incl. the newtest_rewound_trace_streams_again_from_scratch),ruff,ty check verifiers: green.POST /sandbox/{id}/checkpoints404s on prod. The prime runtime was checked against a fake client following that contract (blockingcheckpointthroughwait_for_checkpoint, a FAILED checkpoint surfacing asSandboxErroron restore, restore request without image/disk, delete).Overhead
Docker
commiton this box costs about 3 ms per MB of the container's writable layer, paid on every turn (the layer is the diff against the image, not against the last checkpoint): 0.03 s clean, 0.35 s at 100 MB written, 1.6 s at 500 MB. Prime's per-turn wait is the capture plus the upload of the delta since the previous checkpoint; unmeasured until the API is deployed, bounded at 300 s.🤖 Generated with Claude Code