[jameswnl]

Summary

On startup the runner destroyed every OpenShell sandbox labelled spawned-by=workflow-runner, including sandboxes that other runner replicas were still using. Against a fake gateway, the old reconcile destroyed this runner's, another runner's and an unlabelled sandbox alike.

Startup reconcile now destroys only sandboxes whose expiry has passed, and it deletes each one by its exact gateway name.

How expiry works

• Each spawned sandbox gets a cloud-agents/expires-at label: the stamp time (rounded up) + max(lifetime, SANDBOX_MIN_LIFETIME_SECONDS) + SANDBOX_EXPIRY_SKEW_SECONDS (defaults 86400 and 600; both are operational margins, not enforcement).
• lifetime is the sum of enforced deadlines from the stamp, defined in spawner/lifetime.py: spawn 420s (create and all post-create work, one budget starting at the stamp) + the step's run timeout + transcript fetch 30s + transcript save 30s + a 5s cancellation grace for each of those five phases + 120s destroy grace. With default timeouts that is 1225s before the floor and skew.
• Each phase runs through await_within / cancel_within: on its deadline the work is cancelled, given at most 5s to finish, then abandoned and logged. A timed-out run never goes on to fetch the transcript. This bounds how long the runner keeps using a sandbox; it does not stop a remote RPC that is already in flight or force-stop uncooperative code.
• Lifecycles without an enforced bound get no expiry and are never reaped by this mechanism. CLI sessions are in this group.
• Caller-supplied cloud-agents/expires-at labels are stripped, as is the retired cloud-agents/owner-instance label.

Other changes

• New AgentSpawner.list_sandbox_labels() and destroy_sandbox(name); reconcile no longer calls destroy(agent_name), which deleted every sandbox sharing an agent name. The base destroy_sandbox raises NotImplementedError instead of falling back.
• Sandbox listing pages through all results.
• A cancelled spawn deletes only the sandbox CreateSandbox returned and restores the concurrency count. If create never returned a name, nothing is deleted; a create that completes after our timeout keeps its expiry label and is reaped later.
• The Temporal heartbeat starts before spawn.
• Local-engine run timeouts raise a clear message and emit a sandbox_timeout audit event.

Rollout

Older runners still destroy every spawned-by=workflow-runner sandbox on startup. Stop or drain old runners before new runners accept work, and do the reverse on rollback. Sandboxes created before this change have no expiry label and are never reaped automatically; docs/sandbox-contract.md has a paged one-time cleanup snippet to run after old runners are drained.

Testing

• Tests written first throughout. Deletion and deadline regressions use the real OpenShellSpawner with a fake gateway and an advancing clock, covering same-agent-name expired/live/legacy collisions, a slow create, a trickling health check, stalled client closes after both timed-out and successful requests, progress cleanup that ignores cancellation, and stuck failure cleanup, for both the local and Temporal engines.
• Unit suite: 2267 passed, 5 skipped. Example and DEPLOYMENT.md YAML tests: 50 passed, 5 skipped. black and ruff clean. helm template renders.
• Not run: integration and e2e (need Temporal, Podman or Kind). Nothing was run against a live gateway, so label handling on a real gateway and real create/readiness timings against the 420s budget are unverified.

🤖 Generated with Claude Code

[beesarmy]

Reviewed merge-result head beb5ea9 against base a432a65. No blocking findings remain in the reviewed scope.

The source retains expired-only reconciliation and exact gateway-name deletion, without owner-identity-based deletion. Creation and post-create operations share a stamp-origin spawn budget; run, client closure, progress cleanup, transcript fetch/save and cancellation grace are covered by the reviewed lifetime policy. Timed-out parent flows do not proceed to transcript access. Unbounded lifecycles such as CLI sessions and legacy sandboxes remain without automatic expiry.

Inspected the semantic merge with the exec-output and dormant secret-helper changes. SDK output handling and the shared event path remain intact. Secret-file writes stay within the bounded spawn phase, their failures enter exact-target cleanup, and environment filtering remains in place. Added real-spawner/fake-gateway regressions cover failing and hanging secret writes and environment omission. The shared spawn budget may reject slow initialization; it is a fail-closed budget, not a performance guarantee for every number of mounts.

This was read-only code/test inspection; I did not independently rerun the reported local suite. Independently verified that build, lint, unit-tests, integration and e2e checks succeeded at this exact head. No live gateway, built sandbox image, real label acceptance or deployment timing was verified.

The deadlines bound caller waiting/control flow, not arbitrary background code or remote RPC termination. Cancellation-resistant work can be abandoned as documented. Preserve the rollout/rollback requirement to drain unsafe old runners before admitting new work. This review does not activate secret-bearing MCP or establish a general execution lease.