bug(gateway): sandbox log/event buffers are re-created after delete and leak #5402

User Story

I run shared OpenShell gateways where agents create short-lived sandboxes all day (create, run a few commands, delete), and the gateways are expected to stay up for weeks. In HA runs, each gateway held 0.7–1.3 GiB with 300 sandboxes on a 5-node Kubernetes cluster, and 2.7 GiB with 1,080 sandboxes on a larger HA run: about 2–3.6 MiB per sandbox on every gateway. Deleted sandboxes leave memory behind, so gateway memory grows with every create/delete cycle until the gateway is OOM-killed, taking down every sandbox it serves.

Problem Statement

TracingLogBus::publish and PlatformEventBus::publish (crates/openshell-server/src/tracing_bus.rs) create a sandbox's buffers for any ID without checking that the sandbox still exists. After delete cleanup runs once, the buffers are re-created by:

• Late supervisor log pushes: PushSandboxLogs checks the sandbox only on a stream's first batch, then keeps publishing.
• Platform events that arrive after Deleted, such as pod termination events.

Cleanup never runs again for that sandbox, so the re-created buffers (~1.7 MiB each) leak permanently.

Impact / Why This Matters

Every create/delete cycle leaks memory. Restarting the gateway is the only way to reclaim it.

Acceptance Criteria

• Logs and events for a deleted sandbox do not create buffers.
• An open PushSandboxLogs stream stops being accepted once its sandbox is deleted.
• Repeated create/delete cycles do not grow gateway RSS.

Reproduction Steps

1. Deploy the gateway with the Helm chart (one replica).
2. Create 20 sandboxes, then delete them all; RSS stays flat (observed 113 → 115 MiB).
3. Create 8 new sandboxes, then delete them all.
4. RSS grows again and stays (observed 115 → 147 MiB), so nothing was freed or reused.

Environment

• OpenShell: 0.1.8-dev.51+gaf020160f
• Kubernetes driver, Helm chart

Notes

Likely fix: reject publishes for sandboxes that are deleting or deleted, and re-check the sandbox periodically on PushSandboxLogs streams.

Related

Other causes of the same gateway memory growth: #5399, #5400, #5401. Allocator follow-up: #5404