Skip to content

Expose sandbox cleanup ownership before run_bundle returns #546

Closed

Description

@morluto

Problem

REA runs run_bundle() in a supervised helper process to measure desktop tasks (downstream PR). If cancellation or teardown stalls and the helper must be forcibly stopped, REA has no structured sandbox identity with which to retry cleanup safely. Stopping the helper's process group does not establish that its Docker computer was removed.

This is a source-derived integration gap, not a reproduced real-Docker leak. We inspected the published wheels for 0.9.1431 (REA's task-measurement pin) and 0.9.1464. REA's controlled helper tests cover the missing-identity case; no real agentenv/Docker qualification was performed.

In 0.9.1464, run_bundle exposes text progress and returns BundleRun. Its TaskStepContext is created internally and TaskRun is produced after teardown. TaskRun.torn_down and RunInterrupted.result help once a result reaches the caller; they cannot recover identities if the helper is killed before returning it.

Proposal

Provide a supported way for an external supervisor to obtain and retain exact sandbox cleanup ownership before the task finishes—for example, a structured ownership notification or an independently recoverable per-run cleanup handle.

Please define cancellation during allocation, late sandbox arrival, partial teardown, and retry semantics. Cleanup should identify only resources owned by that run and retain unresolved identities after partial success. If an existing public API or plugin hook already supports this, guidance would resolve the blocker without a new API.

Alternatives

REA now retains known identities returned in teardown reports and retries targeted removal. When forced termination occurs before an identity is reported, it reports cleanup as unknown and blocks integration. Parsing human progress text or deleting containers by a broad name match would not establish ownership and could affect unrelated runs.

Activity

  1. earakely-scale commented on Nov 9, 2026

    @earakely-scale
    Collaborator

    Thanks for the careful write-up. The existing local Docker provider already supports this; #555 documents and tests the supported way, and adds one function to make it easy.

    Local Docker (your sandbox="local" case):

    1. Start each helper with its own AGENT_ENV_LOCAL_SANDBOX_DIR, for example a fresh temporary directory.
    2. If you have to kill the helper, then from any process with that same variable set:
    from agent_env.providers.sandbox_providers.local_docker.provider import (
        LocalSandboxProvider, local_sandbox_ids, remove_local_work_dir)
    
    for sandbox_id in local_sandbox_ids():
        await LocalSandboxProvider().terminate_sandbox(sandbox_id)
        remove_local_work_dir(sandbox_id)

    local_sandbox_ids() arrives with #555. Until it ships, each sandbox is the folder <root>/agent-env-<sandbox id>-<suffix>, and that is how it lists them.

    Why this is exact: a sandbox's work folder is made under that root before anything the sandbox runs. A teardown rebuilt from the folder reaches everything the sandbox started:

    • its containers, networks and relays, by its label and its container's name;
    • its compose stack, through the folder;
    • processes its commands left on the machine, through their environment marker. This last part is Linux-only, since macOS doesn't let one process read another's environment.

    The semantics you asked about:

    Case Behavior
    Cancelled during allocation the folder exists before any container or process, so it's found; one with nothing running is just removed
    A sandbox arriving late it has a folder, so the same loop finds it
    Partial teardown, retry a folder is removed only once its sandbox is down, so what's left keeps one; terminating again is safe
    Ownership the root belongs to that helper alone, so no other run is touched

    We checked this on Linux by SIGKILLing agent-env run at ten points, to the process alone or to its group:

    • as allocation started;
    • mid-step, with a VM-mode sandbox's command and a container-mode sandbox up;
    • with an MCP env's 5-container compose stack up.

    Each cleaned up completely: no folder, container, network, compose stack or process was left. #555 adds an integration test for the mid-step case.

    Remote providers: give each helper its own value in config.toml's [sandbox.attribution]. Every remote provider records it on each sandbox, as a Modal tag or E2B metadata, so you can find exactly that helper's sandboxes there.

  2. morluto commented on Nov 9, 2026

    @morluto
    ContributorAuthor

    The per-helper sandbox root is useful, but there is a remaining verification gap in the published agentenv-framework==0.9.1464 provider: the folder can be removed after teardown reports success even when Docker removal failed.

    I reproduced this with the real 0.9.1464 LocalSandbox and LocalSandboxProvider, a private sandbox root, and a controlled docker executable that always exits 1. terminate_sandbox(id) returned successfully; a subsequent remove_local_work_dir(id) removed the sandbox's folder. This is a controlled failure probe, not a reproduced leak against a real Docker daemon. The provider's container removal uses docker rm ... || true, and _remove_labeled can warn and return when listing fails, so successful return does not establish that all resources are gone.

    Could the local teardown contract preserve the work folder and report an incomplete teardown unless container, labeled resources, compose cleanup and (on Linux) marked processes are confirmed gone? That would let an external supervisor retry safely. The macOS limitation on inspecting other host processes should remain explicit.

    Downstream REA now gives each solve a private root, retries cleanup for folders that remain, independently checks Docker containers and networks before removing recovered folders, and retains failed cleanup for retry. A test with the real 0.9.1464 provider inside a Linux container removed an owned Docker container and marked process; on macOS, a Docker test removed the container but retained the folder because host processes could not be verified. Neither test was a full run_bundle SIGKILL test. If the helper stops before reporting and the provider has already removed the folder after an unverified teardown, REA still cannot recover that sandbox's identity, so its review thread remains open.

  3. morluto commented on Nov 9, 2026

    @morluto
    ContributorAuthor

    @earakely-scale Could you take a look at the remaining teardown verification gap and PR #561? The downstream REA tracking issue is morluto/rea#9676. The controlled failure probe and its limits are described in the preceding comment.

  4. earakely-scale commented on Nov 9, 2026

    @earakely-scale
    Collaborator

    An update for REA's helper: with #561 merged, a local cleanup result is now definitive, so you can drop the "cleanup unknown" outcome for local runs.

    • feat(local): a supervisor can terminate the sandboxes of a run it killed #555 (in 0.9.1482): give each helper its own AGENT_ENV_LOCAL_SANDBOX_DIR. After a kill, terminate each id local_sandbox_ids() lists, then remove its folder.
    • fix(local): retain sandbox folders after incomplete teardown #561 (merged; in the next release after 0.9.1483): terminate_sandbox now raises when anything of a sandbox is left. That covers a container, compose stack or network Docker couldn't list or remove, and a process that survived. So a terminate that returns means the sandbox is gone. One that raises means it isn't: keep its id and folder and retry, which your helper already does for unresolved ids. A sandbox that never used Docker still tears down when Docker isn't installed or its daemon is down.

    Kill the helper's whole process group, as you do. On macOS, that's what stops commands a VM-mode sandbox ran on the machine, because cleanup can't read other processes' environments there to find them. The docs now say both points: #564

  5. earakely-scale commented on Nov 10, 2026

    @earakely-scale
    Collaborator

    All three changes are released: #555 in 0.9.1482, and #561 and #564 in 0.9.1484 (PyPI is at 0.9.1493). On 0.9.1484 or later, REA's helper can find a killed run's sandboxes from its own AGENT_ENV_LOCAL_SANDBOX_DIR, and terminate_sandbox raises unless a sandbox is fully gone, so a return means it's cleaned up. Closing this for the local case; please reopen if the helper still sees a gap on 0.9.1484 or later. If you need the same guarantee for a remote sandbox provider, a new issue would be the place to track it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions