Repository navigation
bug(gateway): per-sandbox log/event buffers pre-allocate ~3.4 MiB even when idle #5399
Description
User Story
I run shared OpenShell gateways where agents create short-lived sandboxes all day (create, run a few commands, delete), and the gateways are expected to stay up for weeks. In HA runs, each gateway held 0.7–1.3 GiB with 300 sandboxes on a 5-node Kubernetes cluster, and 2.7 GiB with 1,080 sandboxes on a larger HA run: about 2–3.6 MiB per sandbox on every gateway. Even idle sandboxes cost each gateway about 3.4 MiB, so how many sandboxes a gateway can host is limited by memory, not by load.
Problem Statement
Each sandbox gets two buffers in the gateway (TracingLogBus and PlatformEventBus in crates/openshell-server/src/tracing_bus.rs). Each buffer pre-allocates a 1024-slot tokio::broadcast ring. Every slot is sized for the largest SandboxStreamEvent variant, the full Sandbox message (~1.6 KiB), although these buses only carry log lines (~0.2 KiB) and platform events.
The result is ~1.7 MiB per buffer and ~3.4 MiB per sandbox, allocated on the first publish or subscribe, even with no subscribers.
Impact / Why This Matters
Gateway memory scales with the number of sandboxes rather than with active sessions. This alone is enough to OOM a gateway at moderate fleet sizes.
Acceptance Criteria
- Idle gateway memory per sandbox is under 256 KiB.
- No broadcast ring is allocated for a sandbox until something subscribes to it.
Reproduction Steps
- Deploy the gateway with the Helm chart (one replica).
- Note the gateway container's RSS.
- Create 20 sandboxes and wait about a minute.
- RSS grows by ~3.8 MiB per sandbox (observed 38 → 113 MiB).
Environment
- OpenShell: 0.1.8-dev.51+gaf020160f
- Kubernetes driver, Helm chart
Notes
Likely fix: store Arc<CursoredEvent> in the ring and tail (or box the Sandbox variant), and create the channel lazily.
Related
Other causes of the same gateway memory growth: #5400, #5401, #5402. Allocator follow-up: #5404
Activity