Repository navigation
Conversation
A paused Prime box refuses exec, so the harness's job polling failed the rollout with a sandbox error. Its processes are only frozen, so wait for the resume instead, without a time bound. - PrimeBackend.pause()/resume() freeze and continue the box, memory included (RuntimeBackend.SUPPORTS_PAUSE; other backends raise). - After a resume, shut down the outbound TCP connections the pause left dead (reset_sockets.py, run inside the box via pidfd_getfd). Otherwise the chat program waits out its 600 s read timeout on a model call that was in flight. The job poller does the same reset once when someone else paused and resumed the box. - The bash/null chat program retries a request for up to 15 minutes while the tunnel edge answers for the interception server (its HTML 404 "Tunnel Not Found" or 5xx pages). This covers frpc reconnecting or the host restarting behind the tunnel. - prime-sandboxes >= 0.5.2 for pause/resume. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ApprovabilityVerdict: Needs Human Approval Macroscope's review found this PR requires human approval — This adds pause/resume and cross-process restoration of live rollouts. The opt-in shutdown path retains billable boxes and tunnels—even when pausing fails—so resource inventory, cleanup ownership, and failure monitoring need operational confirmation. Needs human approval because:
Adjust the Minimum Blocking Severity for this repo — including turning it Off — in Settings. You can add or adjust custom eligibility rules. Learn more. |
A process that dies mid-rollout leaves its box running: the harness program holds its turns, background processes and shell state, and reaches the model through a tunnel URL and secrets that outlive the process. Let another process take the rollout over in that box. - A rollout another process can take over streams its live box once the harness program starts: a `restore` delta carrying `RestorePoint.live` (`LiveBox`: sandbox_id, tunnel_id, the tunnel's registration, the program's job, the model and state secrets, the typed state), collected in `EpisodeAssembly.restores`. `Trace.live` holds it, off the record. - `EnvClient.run(restore=RestoreRequest(trace, point), assembly=...)` continues a trace in another env server (`RunRequest.restore`, packed raw). The server rebuilds the trace (`EpisodeHook.restore` gives nodes back their arrays, `EpisodeHook.watch` sees it), and the episode's one agent continues it (`Agent.pending_restore`). - The run binds the sandbox (`RuntimeBackend.adopt`, resuming it if paused), skips task and harness setup, registers its session under the saved secrets on a server that serves the saved tunnel (`Interception.adopt`: one server per tunnel, shared by every adopted rollout behind it), and waits on the running program (`HarnessSession.attach`, `RuntimeBackend.attach`) instead of launching one. The trace is not rewound: turns the program took after the client's copy arrive with its next request and are recorded as context. A box that cannot be taken over starts the run over, unbudgeted. - Adoptable: bash and null (`Harness.SUPPORTS_ADOPT`) on Prime (`RuntimeBackend.SUPPORTS_ADOPT`), in an owned colocated box behind a prime tunnel, with no task tool servers or request hooks. - With VF_SUSPEND_DIR set, an env server that shuts down (SIGTERM, its parent gone) pauses the boxes of adoptable rollouts instead of deleting them, records each in <dir>/<hostname>-<pid>.jsonl, and keeps their tunnels registered (`runtimes.suspend`). A per-run cancel still deletes. - `EnvClient.detach()`: a cancelled run no longer cancels on the server. - A pool routes restores behind one tunnel to one worker (`run@<tunnel>`). - `pause_boxes` / `delete_boxes`: best-effort helpers for an orchestrator. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Before pausing the box itself, PrimeBackend lists its outbound connections (reset_sockets.py list); the recovery after the resume shuts down only those still open, never one opened since. After a pause someone else made, it still shuts down every outbound one. - Check the borrowed descriptor is the socket read from /proc before shutting it down: its owner may have reused the fd meanwhile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| if status == "PAUSED": | ||
| await self.resume() | ||
| elif status != "RUNNING": | ||
| raise SandboxError(f"sandbox {box} is {status}") |
There was a problem hiding this comment.
🟠 High backends/prime.py:363
Adopting a box that was already resumed leaves its pre-pause sockets untouched, so in-flight model calls can hang until their timeout and fail the restored rollout. _take_over() only recovers when it sees PAUSED; on the RUNNING path, call _recover(self._recovered) to reset connections left by an earlier pause.
| if status == "PAUSED": | |
| await self.resume() | |
| elif status != "RUNNING": | |
| raise SandboxError(f"sandbox {box} is {status}") | |
| if status == "PAUSED": | |
| await self.resume() | |
| elif status == "RUNNING": | |
| await self._recover(self._recovered) | |
| else: | |
| raise SandboxError(f"sandbox {box} is {status}") |
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/runtimes/backends/prime.py around lines 363-366:
Adopting a box that was already resumed leaves its pre-pause sockets untouched, so in-flight model calls can hang until their timeout and fail the restored rollout. `_take_over()` only recovers when it sees `PAUSED`; on the `RUNNING` path, call `_recover(self._recovered)` to reset connections left by an earlier pause.
- A takeover cancelled mid-way (the run shutting down again) no longer deletes the box at teardown: the next resume can still adopt it. A failed one still deletes it, since a paused box never expires. - A rollout with response interceptors is not adoptable: they may write state mid-run, and the live box carries the state as of the program's start. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An env server whose client (the orchestrator) died cancelled the runs it could no longer deliver, and the cancel deleted their boxes before the server itself was stopped: a crashed orchestrator lost most live boxes. - A run whose client is gone (a pool's broker cancels it with no request id, a lone server fails to send it a delta) is cancelled with its suspend scope lost: its teardown pauses and records the box like a shutdown does, while the server keeps running. - Until the box is paused, the interception server holds the requests of a suspending rollout's program instead of refusing them (401 ends the program); the pause freezes them, and they are retried on resume against the process that adopts the box. - test_suspend_on_shutdown also covers the client going away. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| if not suspending() or tunnel_id is None: | ||
| return False | ||
| return tunnel_id in _kept_tunnels or any(t == tunnel_id for _, t in _live.values()) |
There was a problem hiding this comment.
🟠 High runtimes/suspend.py:109
When one rollout has suspended, keeps_tunnel() returns False for its shared tunnel if the last user exits from a healthy run scope. The not suspending() check prevents _kept_tunnels from being consulted, so Interception.adopt() closes the registration and Tunnel.sync_stop() deletes the endpoint needed to readopt the paused box; check _kept_tunnels independently of the current scope.
| if not suspending() or tunnel_id is None: | |
| return False | |
| return tunnel_id in _kept_tunnels or any(t == tunnel_id for _, t in _live.values()) | |
| if tunnel_id is None: | |
| return False | |
| return tunnel_id in _kept_tunnels or ( | |
| suspending() and any(t == tunnel_id for _, t in _live.values()) | |
| ) |
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/runtimes/suspend.py around lines 109-111:
When one rollout has suspended, `keeps_tunnel()` returns `False` for its shared tunnel if the last user exits from a healthy run scope. The `not suspending()` check prevents `_kept_tunnels` from being consulted, so `Interception.adopt()` closes the registration and `Tunnel.sync_stop()` deletes the endpoint needed to readopt the paused box; check `_kept_tunnels` independently of the current scope.
| await self._harness_session.turn(messages) | ||
| job, self._attach = self._attach, None | ||
| if job is not None: | ||
| await self._harness_session.attach(job) |
There was a problem hiding this comment.
🟠 High v1/rollout.py:713
A restored trace can be marked agent_completed and scored with a stale answer when attach(job) reconnects to a job that already finished. The attach path does not replay the program’s final response into the trace, so a successful exit can appear complete despite the missing turn; recover the final transcript or restart instead of accepting that attach.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/rollout.py around line 713:
A restored trace can be marked `agent_completed` and scored with a stale answer when `attach(job)` reconnects to a job that already finished. The attach path does not replay the program’s final response into the trace, so a successful exit can appear complete despite the missing turn; recover the final transcript or restart instead of accepting that attach.
| ) | ||
| except Exception as e: # a sandbox/API failure is one rollout's problem | ||
| raise SandboxError(f"prime exec failed: {e}") from e | ||
| started = program_started.get() |
There was a problem hiding this comment.
🟠 High backends/prime.py:439
When launch_chat_program prepares a message-list prompt for an agent user, PrimeBackend.write runs chmod, the copy, and cleanup through run, so program_started publishes a helper job to Trace.live before the harness starts. A checkpoint during those awaits restores by attaching to that helper instead of launching the chat program, preventing the rollout from continuing. Notify program_started only for the harness-program launch, not preparatory runtime operations.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/v1/runtimes/backends/prime.py around line 439:
When `launch_chat_program` prepares a message-list prompt for an agent user, `PrimeBackend.write` runs `chmod`, the copy, and cleanup through `run`, so `program_started` publishes a helper job to `Trace.live` before the harness starts. A checkpoint during those awaits restores by attaching to that helper instead of launching the chat program, preventing the rollout from continuing. Notify `program_started` only for the harness-program launch, not preparatory runtime operations.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit e98859a. Configure here.

Prime can now pause a running VM box with its memory (
POST /sandbox/{id}/pause|resume, prime-sandboxes 0.5.2). This uses it to keep agentic rollouts alive when the process running them goes away, without snapshotting anything as turns go. It replaces #2994: there, every turn of every rollout snapshots the box's filesystem, and a restore boots a fresh box, relaunches the harness, tells the model the box was restored, and loses whatever ran in the box. Here the box itself is what survives: processes, shell state, servers it started, and the running harness program.Survive a pause
PrimeBackend.pause()/resume()(RuntimeBackend.SUPPORTS_PAUSE).pause()lists the box's outbound connections first; after the resume,reset_sockets.pyshuts down those still open (pidfd_getfd, checked against the inode), so the call fails at once and its retry lands on the interception server's replay. After someone else's pause (prime sandbox pause), the job poller resets every outbound one. Loopback and the guest agents' connections are left alone.404 Tunnel Not Foundor 5xx pages): frpc reconnecting, or the process behind the tunnel restarting.Adopt a live box in another process
Trace.live,LiveBox): the sandbox, the program's background job, the model and state secrets, and the tunnel's registration.RestoreRequest(trace, point),RunRequest.restore,EnvClient.run(restore=, assembly=),EpisodeAssembly.restoresandEpisodeHook.watch/.restorekeep feat: rollout checkpointing #2994's names and wire format.sampled=False), not samples. A box that cannot be taken over starts the run over, unbudgeted; a takeover cancelled mid-way leaves the box for the next resume.Harness.SUPPORTS_ADOPT) on Prime, in an owned colocated box behind a prime tunnel, without tool servers, tool VMs, compose, or request/response hooks (they write state mid-run). Anything else starts over, as on main.VF_SUSPEND_DIRset, a worker that shuts down (SIGTERM, or its parent gone) pauses its adoptable rollouts' boxes instead of deleting them (60 s shared budget), keeps their tunnels registered, and records each box in<dir>/<hostname>-<pid>.jsonl. So does a run whose client is gone (a crashed orchestrator: the pool's broker cancels the runs it cannot deliver, a lone server fails to send a delta), while the server keeps running; until its box is paused, the interception server holds that program's requests instead of refusing them, so the pause freezes them and they retry against the adopting process. A run its client cancels still deletes its box.run@<tunnel_id>method frame), which adopts the tunnel once.EnvClient.detach()(cancelled runs stop aborting server-side, so a shutdown can suspend them), and best-effortpause_boxes/delete_boxes.prime-rl uses this in PrimeIntellect-ai/prime-rl#4348 (
ckpt.rollouts): env servers suspend on shutdown, a resumed run pauses the boxes its checkpoint resumes in as soon as it has read it and deletes the rest, then adopts each one.Tests
On Prime with a scripted model, bash harness (new CI cells
PauseandReadopt):test_pause[tool]: 10 s pause during a 20 s sleeptest_pause[model]: pause during a model calltest_readopt[tool]: process SIGKILLed during a tool call, another adoptstest_readopt[model]: SIGKILLed with a model call in flighttest_readopt[box-gone]: box deleted before the restoretest_suspend_on_shutdown[shutdown]: detach, cancel, SIGTERM the servertest_suspend_on_shutdown[client-gone]: the client's socket closes without a wordControls for the pause path: without the socket reset the rollout hangs until the 180 s agent timeout; without the pause wait it fails with
sandbox_error; with the tunnel down for 40 s, main's harness dies withopenai.NotFoundError: 404.Not covered
_can_reconnect), and re-attaching it from a new process is untested. They start over.IDEMPOTENCY_CACHE_TTL_SECONDS.read/write/execstill refuse while paused.🤖 Generated with Claude Code
Note
Add pause, suspension, and live-box restore for rollouts and sandbox backends
LiveBoxrestore point (sandbox, tunnel, job handle, secrets) is recorded on the trace and streamed to clients; a restored run adopts the saved sandbox and attaches to its saved job instead of starting over.VF_SUSPEND_DIRenvironment variable in suspend.py. On shutdown (or a lost client), claimed backends are paused, kept, and recorded in a JSONL manifest; the tunnel and sandbox are not deleted.EnvClientcan run detached, restore from aRestoreRequest, and route requests by affinity so a restored run lands on the worker holding its tunnel.EnvServerrestores traces, marks runs lost when deltas cannot be delivered, and treats ID-less cancels as lost-client signals.PrimeBackendgainsadopt,attach,pause, andresume, retries provider calls refused during a pause, and resets outbound sockets on recovery via reset_sockets.py. Tunnel and tool-gate requests in core.py retry for up to 15 minutes during pauses.PrimeBackend.cleanupandteardownkeep the sandbox when suspension is requested; interception requests with held-but-unregistered secrets block on a future instead of returning unauthorized. Adoption is rejected for tasks with a deployment spec, tool servers, tool URLs, or interceptors (seeRollout._adoptablein rollout.py). Raisesprime-sandboxesminimum to 0.5.2.Macroscope summarized 84a5c2d.
Note
High Risk
Changes sandbox lifecycle, interception auth secrets, tunnel adoption, and env-server shutdown semantics—core orchestration paths where bugs could leak boxes, drop rollouts, or leave hung requests.
Overview
Rollouts can keep running in the same Prime sandbox when the host process pauses, shuts down, or is replaced—instead of snapshotting the filesystem each turn or booting a fresh box on restore.
Pause/resume on Prime (
prime-sandboxes0.5.2):PrimeBackendgainspause/resume, exec polling that waits through platform pause refusals, and post-resume TCP socket reset (reset_sockets.py) so stale outbound connections fail fast and model/tool retries reconnect. The bash/null chat program retries tunnel-edge HTML errors for up to 15 minutes while interception comes back.Live-box adoption: Once a harness program starts, the trace streams a
LiveBoxrestore point (sandbox id, background job, interception secrets, tunnel registration).RunRequest.restore/Agent.pending_restorecontinue that trace in another process: bind the sandbox, attach to the existing job, adopt the saved tunnel on one shared interception server per tunnel, and record gap turns as context (sampled=False) rather than rewinding. Failed or missing boxes start over unbudgeted.VF_SUSPEND_DIRmakes shutdown (or a detached client) pause and manifest adoptable boxes instead of deleting them; in-flight interception requests block until suspend completes.Serve/routing: Deltas carry
restorepayloads;EnvClient.run(restore=, assembly=)and poolrun@<tunnel_id>affinity pin restores to the worker that can hold the tunnel. Bash and null harnesses setSUPPORTS_ADOPT; adoption is rejected when compose, tool VMs, or mid-run hooks would diverge from the saved state.New CI integration suites
PauseandReadoptcover pause mid-tool/model, SIGKILL + restore, deleted box, and suspend-on-shutdown paths.Reviewed by Cursor Bugbot for commit 84a5c2d. Bugbot is set up for automated code reviews on this repo. Configure here.