Skip to content

feat(local): a supervisor can terminate the sandboxes of a run it killed - #555

Merged
earakely-scale merged 5 commits into
mainfrom
edgararakelyan/fd-5375-local-sandbox-cleanup
Nov 9, 2026
Merged

earakely-scale merged 5 commits into
mainfrom
edgararakelyan/fd-5375-local-sandbox-cleanup

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Part of FD-5375. Answers #546.

A process running run_bundle that is killed outright, with no chance to handle a signal, tears none of its sandboxes down. An outside supervisor, such as REA's measurement helper in #546, needs a supported way to clean up exactly that run's sandboxes afterwards. This documents and tests one for local Docker, using what the local provider already does. The one addition is a public function that lists the sandboxes under a root.

How a supervisor cleans up

  1. Give each run its own AGENT_ENV_LOCAL_SANDBOX_DIR.
  2. After killing the run, from any process with the same variable set, terminate each id that local_sandbox_ids() lists and remove its folder:
for sandbox_id in local_sandbox_ids():
    await LocalSandboxProvider().terminate_sandbox(sandbox_id)
    remove_local_work_dir(sandbox_id)

This works because a local sandbox's work folder, <root>/agent-env-<id>-<random>, is made before anything the sandbox runs. A teardown rebuilt from that folder (get_sandbox) reaches everything the sandbox started:

  • its containers, networks and relays, through its agentenv.sandbox label and its container's name;
  • its compose stack, through docker compose down in the folder;
  • the processes its commands left on the machine, through their environment marker (Linux).

That answers the semantics #546 asks about:

Case Behavior
Cancelled during allocation the folder exists before any container or process, so it's found; a folder with nothing running is just removed
A sandbox arriving late, after the kill it has a folder, so the same loop finds it
Partial teardown, retry a folder is removed only once its sandbox is down, so whatever is left still has one; terminating again is safe
Ownership the root is the run's alone, so no other run is touched

For remote providers, a per-run value in config.toml's [sandbox.attribution] does the same job: every remote provider records it on each sandbox (a Modal tag, E2B metadata).

What changes

  • local_sandbox_ids(), in providers/sandbox_providers/local_docker/provider.py, lists the ids of the sandboxes with a folder under the root, so a supervisor needn't parse folder names. It lists only folders named agent-env-<id>-<suffix>, so a stray folder there can't stop the cleanup.
  • get_sandbox reads a port map that is missing or cut short as none. A run killed while writing that file is then still rebuilt and terminated, instead of raising.
  • run_bundle's docstring says what to do when a run is killed outright.
  • .env.example documents AGENT_ENV_LOCAL_SANDBOX_DIR, which only a docstring mentioned before.

Also, a fix for a flaky test from #549: test_disk_recipe.py::test_the_launcher_forwards_each_host_port_from_loopback_here_to_the_host_and_seeds_the_trust_files failed this PR's CodeBuild run twice, and PR #560's once, with assert 1 == 143. The cause is in the disk launcher. It runs under set -e, and its TERM trap kills the pid in qemu.pid, which isn't written until QEMU starts. A stop in between fails that kill, so the launcher exits 1 instead of 143. The test stops the launcher just after it writes the seed, so a slow runner hits that window. The trap now ignores a failed kill, as the EXIT trap beside it already does. test_a_launcher_stopped_before_its_computer_starts_exits_as_one_stopped_after_does stops the launcher while it decompresses its disk, before QEMU starts. It failed on the old launcher with the same 1 == 143, and with the fix, the whole launcher test file passed 6 runs out of 6 with every CPU of a 16-core box busy. Only the launcher image's tag changes; no composed disk's key does.

No teardown command or new teardown API, and sandbox labeling is unchanged. I first added the sandbox label to a container-mode sandbox's container, to cover a kill between its docker run and the folder recording its name. A check on main showed it wasn't needed: a teardown rebuilt from the folder removes the container agent-<id> by name anyway. It removed the container with its record deleted and no label on it.

Tests

  • Unit:
    • test_find_work_dir_only_returns_matching_sandbox_id also checks that local_sandbox_ids() lists the sandboxes under the root, and skips a stray file and a stray agent-env-notes directory.
    • test_a_sandbox_killed_while_writing_its_port_map_is_still_rebuilt_for_teardown rebuilds a sandbox from a folder whose .port-map.json is empty. Full unit suite on an EC2 box: 14,072 passed. Ruff is clean.
  • Integration: a new test in run_bundle_test.py runs a task with a VM-mode sandbox, a container-mode sandbox (nginx) and a long command, and SIGKILLs agent-env run once the command is running. It then terminates what local_sandbox_ids() finds, and asserts that no folder, process, labeled container or container named agent-<id> is left. _signalled gains an optional ready check, so the kill waits for the command. The whole file passes (10), and the kill tests passed three runs in a row.
  • End to end on an EC2 box: the same cleanup after SIGKILLing agent-env run at ten points. Each case was killed either alone or with its process group:
Killed Up when killed Left after cleanup
as the first step starts (twice) nothing yet nothing
2 s in (twice) a VM-mode sandbox's command and a container-mode sandbox nothing
mid-step (twice) the same nothing
after Ctrl-C started its own teardown already torn down nothing
as an MCP env starts deploying nothing yet nothing
6 s into the deploy its compose stack, 5 containers nothing
once the env is up its compose stack, 5 containers nothing

Not exercised: a kill in the middle of the run's own teardown. Here the teardown had finished before the SIGKILL landed.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 4/5

The existing requirement to name the readiness timings must be satisfied before merging; no new blocking issue was found.

Fix All in CursorFindings

  1. P2 Readiness timings lack names ▶
Fix with agent prompt
### Issue 1
tst/integration/cli/run_bundle_test.py:220-222
The new readiness loop uses inline `30` and `0.05` values. The repository requires descriptive names for magic numbers. Name the readiness timeout and polling interval so their purpose and units are clear. This repository requirement must be satisfied before merging.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds a supported way for supervisors to find local sandboxes after killing a run.

  • Supervisors can clean up local sandboxes after killing a run.
  • Disk launchers now report a stop even before QEMU starts.

The launcher also keeps its intended exit code when stopped before its PID file exists.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Give each run its own sandbox root] --> B[Run creates sandbox work folders]
    B --> C[Supervisor kills the run]
    C --> D[local_sandbox_ids lists IDs]
    D --> E[Terminate each sandbox]
    E --> F[Remove its work folder]
Loading

Reviews (12) · Last reviewed commit: "fix(computer): a disk launcher stopped b..." · Reviewed by Greptile

earakely-scale and others added 2 commits November 8, 2026 22:34
A run killed outright tears nothing down. A local sandbox's work folder is made before anything it runs, under the root
AGENT_ENV_LOCAL_SANDBOX_DIR names, and its teardown rebuilt from that folder reaches its containers, networks, compose
stack and processes. So a supervisor that gives a run a root of its own can terminate each sandbox found there
afterwards: local_sandbox_ids() lists them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale requested a review from a team as a code owner November 9, 2026 06:49
Comment thread tst/integration/cli/run_bundle_test.py Resolved
…ervisor's cleanup

get_sandbox reads a port map a killed run left empty as none, and local_sandbox_ids() lists only folders named as
sandboxes' are. The test also checks a container-mode sandbox's container, which is named, not labeled, is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
earakely-scale and others added 2 commits November 9, 2026 08:17
# Conflicts:
#	src/agent_env/providers/sandbox_providers/local_docker/provider.py
…ts 143

The launcher runs under set -e, and its TERM trap killed the pid in qemu.pid, which isn't written until QEMU starts. A
stop in between failed that kill and exited 1. test_the_launcher_forwards_each_host_port_from_loopback_here_to_the_host_and_seeds_the_trust_files
stops the launcher just after it writes the seed, so on a slow runner it hit that window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale merged commit d993069 into main Nov 9, 2026
35 checks passed
@earakely-scale
earakely-scale deleted the edgararakelyan/fd-5375-local-sandbox-cleanup branch November 9, 2026 17:12
Comment on lines +220 to +222
deadline = time.monotonic() + 30
while not ready() and time.monotonic() < deadline:
time.sleep(0.05)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Readiness timings lack names

The new readiness loop uses inline 30 and 0.05 values. The repository requires descriptive names for magic numbers. Name the readiness timeout and polling interval so their purpose and units are clear. This repository requirement must be satisfied before merging.

Rule Used: Store magic numbers as class or instance variables with descriptive names rather than using them inline in the code. (source)

Learned From
scaleapi/scaleapi#126388

Prompt To Fix With AI
This is a comment left during a code review.
Path: tst/integration/cli/run_bundle_test.py
Line: 220-222

Comment:
**Readiness timings lack names**

The new readiness loop uses inline `30` and `0.05` values. The repository requires descriptive names for magic numbers. Name the readiness timeout and polling interval so their purpose and units are clear. This repository requirement must be satisfied before merging.

**Rule Used:** Store magic numbers as class or instance variables with descriptive names rather than using them inline in the code. ([source](https://app.greptile.com/-/switch-tenant?tenant=scale-ai&next=%2Fcustom-context%3Fmemory%3D002e0051-41ad-46c1-9098-47433c580150))

**Learned From**
[scaleapi/scaleapi#126388](https://github.com/scaleapi/scaleapi/pull/126388)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fix in Cursor Fix in Claude Code Fix in Codex

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant