You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Expose sandbox cleanup ownership before run_bundle returns #546
REA runs run_bundle() in a supervised helper process to measure desktop tasks (downstream PR). If cancellation or teardown stalls and the helper must be forcibly stopped, REA has no structured sandbox identity with which to retry cleanup safely. Stopping the helper's process group does not establish that its Docker computer was removed.
This is a source-derived integration gap, not a reproduced real-Docker leak. We inspected the published wheels for 0.9.1431 (REA's task-measurement pin) and 0.9.1464. REA's controlled helper tests cover the missing-identity case; no real agentenv/Docker qualification was performed.
Provide a supported way for an external supervisor to obtain and retain exact sandbox cleanup ownership before the task finishes—for example, a structured ownership notification or an independently recoverable per-run cleanup handle.
Please define cancellation during allocation, late sandbox arrival, partial teardown, and retry semantics. Cleanup should identify only resources owned by that run and retain unresolved identities after partial success. If an existing public API or plugin hook already supports this, guidance would resolve the blocker without a new API.
Alternatives
REA now retains known identities returned in teardown reports and retries targeted removal. When forced termination occurs before an identity is reported, it reports cleanup as unknown and blocks integration. Parsing human progress text or deleting containers by a broad name match would not establish ownership and could affect unrelated runs.
Thanks for the careful write-up. The existing local Docker provider already supports this; #555 documents and tests the supported way, and adds one function to make it easy.
Local Docker (your sandbox="local" case):
Start each helper with its own AGENT_ENV_LOCAL_SANDBOX_DIR, for example a fresh temporary directory.
If you have to kill the helper, then from any process with that same variable set:
local_sandbox_ids() arrives with #555. Until it ships, each sandbox is the folder <root>/agent-env-<sandbox id>-<suffix>, and that is how it lists them.
Why this is exact: a sandbox's work folder is made under that root before anything the sandbox runs. A teardown rebuilt from the folder reaches everything the sandbox started:
its containers, networks and relays, by its label and its container's name;
its compose stack, through the folder;
processes its commands left on the machine, through their environment marker. This last part is Linux-only, since macOS doesn't let one process read another's environment.
The semantics you asked about:
Case
Behavior
Cancelled during allocation
the folder exists before any container or process, so it's found; one with nothing running is just removed
A sandbox arriving late
it has a folder, so the same loop finds it
Partial teardown, retry
a folder is removed only once its sandbox is down, so what's left keeps one; terminating again is safe
Ownership
the root belongs to that helper alone, so no other run is touched
We checked this on Linux by SIGKILLing agent-env run at ten points, to the process alone or to its group:
as allocation started;
mid-step, with a VM-mode sandbox's command and a container-mode sandbox up;
with an MCP env's 5-container compose stack up.
Each cleaned up completely: no folder, container, network, compose stack or process was left. #555 adds an integration test for the mid-step case.
Remote providers: give each helper its own value in config.toml's [sandbox.attribution]. Every remote provider records it on each sandbox, as a Modal tag or E2B metadata, so you can find exactly that helper's sandboxes there.
The per-helper sandbox root is useful, but there is a remaining verification gap in the published agentenv-framework==0.9.1464 provider: the folder can be removed after teardown reports success even when Docker removal failed.
I reproduced this with the real 0.9.1464 LocalSandbox and LocalSandboxProvider, a private sandbox root, and a controlled docker executable that always exits 1. terminate_sandbox(id) returned successfully; a subsequent remove_local_work_dir(id) removed the sandbox's folder. This is a controlled failure probe, not a reproduced leak against a real Docker daemon. The provider's container removal uses docker rm ... || true, and _remove_labeled can warn and return when listing fails, so successful return does not establish that all resources are gone.
Could the local teardown contract preserve the work folder and report an incomplete teardown unless container, labeled resources, compose cleanup and (on Linux) marked processes are confirmed gone? That would let an external supervisor retry safely. The macOS limitation on inspecting other host processes should remain explicit.
Downstream REA now gives each solve a private root, retries cleanup for folders that remain, independently checks Docker containers and networks before removing recovered folders, and retains failed cleanup for retry. A test with the real 0.9.1464 provider inside a Linux container removed an owned Docker container and marked process; on macOS, a Docker test removed the container but retained the folder because host processes could not be verified. Neither test was a full run_bundle SIGKILL test. If the helper stops before reporting and the provider has already removed the folder after an unverified teardown, REA still cannot recover that sandbox's identity, so its review thread remains open.
@earakely-scale Could you take a look at the remaining teardown verification gap and PR #561? The downstream REA tracking issue is morluto/rea#9676. The controlled failure probe and its limits are described in the preceding comment.
fix(local): retain sandbox folders after incomplete teardown #561 (merged; in the next release after 0.9.1483): terminate_sandbox now raises when anything of a sandbox is left. That covers a container, compose stack or network Docker couldn't list or remove, and a process that survived. So a terminate that returns means the sandbox is gone. One that raises means it isn't: keep its id and folder and retry, which your helper already does for unresolved ids. A sandbox that never used Docker still tears down when Docker isn't installed or its daemon is down.
Kill the helper's whole process group, as you do. On macOS, that's what stops commands a VM-mode sandbox ran on the machine, because cleanup can't read other processes' environments there to find them. The docs now say both points: #564
All three changes are released: #555 in 0.9.1482, and #561 and #564 in 0.9.1484 (PyPI is at 0.9.1493). On 0.9.1484 or later, REA's helper can find a killed run's sandboxes from its own AGENT_ENV_LOCAL_SANDBOX_DIR, and terminate_sandbox raises unless a sandbox is fully gone, so a return means it's cleaned up. Closing this for the local case; please reopen if the helper still sees a gap on 0.9.1484 or later. If you need the same guarantee for a remote sandbox provider, a new issue would be the place to track it.
Activity
earakely-scale commentedon Nov 9, 2026
Thanks for the careful write-up. The existing local Docker provider already supports this; #555 documents and tests the supported way, and adds one function to make it easy.
Local Docker (your
sandbox="local"case):AGENT_ENV_LOCAL_SANDBOX_DIR, for example a fresh temporary directory.local_sandbox_ids()arrives with #555. Until it ships, each sandbox is the folder<root>/agent-env-<sandbox id>-<suffix>, and that is how it lists them.Why this is exact: a sandbox's work folder is made under that root before anything the sandbox runs. A teardown rebuilt from the folder reaches everything the sandbox started:
The semantics you asked about:
We checked this on Linux by SIGKILLing
agent-env runat ten points, to the process alone or to its group:Each cleaned up completely: no folder, container, network, compose stack or process was left. #555 adds an integration test for the mid-step case.
Remote providers: give each helper its own value in
config.toml's[sandbox.attribution]. Every remote provider records it on each sandbox, as a Modal tag or E2B metadata, so you can find exactly that helper's sandboxes there.morluto commentedon Nov 9, 2026
The per-helper sandbox root is useful, but there is a remaining verification gap in the published
agentenv-framework==0.9.1464provider: the folder can be removed after teardown reports success even when Docker removal failed.I reproduced this with the real 0.9.1464
LocalSandboxandLocalSandboxProvider, a private sandbox root, and a controlleddockerexecutable that always exits 1.terminate_sandbox(id)returned successfully; a subsequentremove_local_work_dir(id)removed the sandbox's folder. This is a controlled failure probe, not a reproduced leak against a real Docker daemon. The provider's container removal usesdocker rm ... || true, and_remove_labeledcan warn and return when listing fails, so successful return does not establish that all resources are gone.Could the local teardown contract preserve the work folder and report an incomplete teardown unless container, labeled resources, compose cleanup and (on Linux) marked processes are confirmed gone? That would let an external supervisor retry safely. The macOS limitation on inspecting other host processes should remain explicit.
Downstream REA now gives each solve a private root, retries cleanup for folders that remain, independently checks Docker containers and networks before removing recovered folders, and retains failed cleanup for retry. A test with the real 0.9.1464 provider inside a Linux container removed an owned Docker container and marked process; on macOS, a Docker test removed the container but retained the folder because host processes could not be verified. Neither test was a full
run_bundleSIGKILL test. If the helper stops before reporting and the provider has already removed the folder after an unverified teardown, REA still cannot recover that sandbox's identity, so its review thread remains open.morluto commentedon Nov 9, 2026
@earakely-scale Could you take a look at the remaining teardown verification gap and PR #561? The downstream REA tracking issue is morluto/rea#9676. The controlled failure probe and its limits are described in the preceding comment.
earakely-scale commentedon Nov 9, 2026
An update for REA's helper: with #561 merged, a local cleanup result is now definitive, so you can drop the "cleanup unknown" outcome for local runs.
AGENT_ENV_LOCAL_SANDBOX_DIR. After a kill, terminate each idlocal_sandbox_ids()lists, then remove its folder.terminate_sandboxnow raises when anything of a sandbox is left. That covers a container, compose stack or network Docker couldn't list or remove, and a process that survived. So a terminate that returns means the sandbox is gone. One that raises means it isn't: keep its id and folder and retry, which your helper already does for unresolved ids. A sandbox that never used Docker still tears down when Docker isn't installed or its daemon is down.Kill the helper's whole process group, as you do. On macOS, that's what stops commands a VM-mode sandbox ran on the machine, because cleanup can't read other processes' environments there to find them. The docs now say both points: #564
earakely-scale commentedon Nov 10, 2026
All three changes are released: #555 in 0.9.1482, and #561 and #564 in 0.9.1484 (PyPI is at 0.9.1493). On 0.9.1484 or later, REA's helper can find a killed run's sandboxes from its own
AGENT_ENV_LOCAL_SANDBOX_DIR, andterminate_sandboxraises unless a sandbox is fully gone, so a return means it's cleaned up. Closing this for the local case; please reopen if the helper still sees a gap on 0.9.1484 or later. If you need the same guarantee for a remote sandbox provider, a new issue would be the place to track it.