bug(gateway): per-sandbox log tail holds up to ~3.7 MiB #5400

User Story

I run shared OpenShell gateways where agents create short-lived sandboxes all day (create, run a few commands, delete), and the gateways are expected to stay up for weeks. In HA runs, each gateway held 0.7–1.3 GiB with 300 sandboxes on a 5-node Kubernetes cluster, and 2.7 GiB with 1,080 sandboxes on a larger HA run: about 2–3.6 MiB per sandbox on every gateway. Every command's output stays in gateway memory, so busy sandboxes keep growing gateway memory even though nobody reads those logs.

Problem Statement

The gateway keeps up to 2,000 log lines per sandbox (TracingLogBus::DEFAULT_TAIL in crates/openshell-server/src/tracing_bus.rs). Each line is stored as a full SandboxStreamEvent (~1.6 KiB inline) plus its strings. A busy sandbox therefore holds ~3.7 MiB of log tail, in addition to its fixed buffers.

Impact / Why This Matters

Every exec and supervisor log push adds to the tail, so gateway memory climbs until each sandbox hits the cap. Across a fleet this is gigabytes held for replay that is rarely requested.

Acceptance Criteria

• The per-sandbox log tail is capped by bytes, not just line count.
• A full tail costs well under 1 MiB per sandbox.

Reproduction Steps

1. Deploy the gateway with the Helm chart.
2. Create a sandbox and run repeated openshell sandbox exec commands that produce output.
3. Gateway RSS grows with each exec until the sandbox reaches 2,000 lines.

Environment

• OpenShell: 0.1.8-dev.51+gaf020160f
• Kubernetes driver, Helm chart

Notes

Likely fix: store compact entries (Arc or log-only type) and enforce a byte budget per sandbox.

Related

Other causes of the same gateway memory growth: #5399, #5401, #5402. Allocator follow-up: #5404