Repository navigation
Description
User Story
I run shared OpenShell gateways with several replicas behind a load balancer, and hit this in perf/scale testing while chasing gateway memory growth (#5401). Supervisor sessions follow the gateway ring to the sandbox's owner, but a sandbox's logs and watch streams don't. So any replica can end up holding a sandbox's log buffers, and openshell logs shows a different, partial history depending on which replica the client lands on.
Problem Statement
Only the supervisor session (ConnectSupervisor) is placed on the ring owner. The rest of a sandbox's log path ignores ownership:
- Log push. The supervisor opens a separate
PushSandboxLogsconnection to the configured gateway endpoint (the Service), not to the owner it was redirected to. Any replica can receive it, and each reconnect can pick a new one. This applies to dedicated supervisors (openshell-supervisor/src/main.rs) and shared ones (sandbox_instance_host.rs). - Log reads and watches.
WatchSandboxandGetSandboxLogsserve only the local replica'sTracingLogBusand never forward to the owner. - Gateway logs. Any replica serving an RPC for a sandbox (create, exec, policy update) records its own lines for that sandbox locally.
- Platform events. Driver events are published only on the reconciler-lease holder.
Impact / Why This Matters
- Incomplete logs. A client sees only the lines its replica happened to receive. Lines pushed to another replica, and driver events published on the lease holder, are missing. Retrying can show a different history.
- Memory fan-out. Over a sandbox's lifetime, its buffers spread across many replicas, so per-replica memory grows with the whole fleet instead of that replica's share. fix(server): free deleted sandboxes' buffers on every replica #5460 frees these buffers after deletion, but doesn't stop the spread.
- No workaround. There is no way to point a client at the replica that has the full history.
Acceptance Criteria
-
openshell logs <sandbox>andopenshell logs <sandbox> --tailreturn the same complete history (supervisor lines, gateway lines, and platform events) through any replica. - A sandbox's live log buffers are held by about one replica, not by every replica that ever touched it.
- Supervisor log pushes and log reads keep working when the ring changes (scale up or down, rollout), without losing lines.
Reproduction Steps
- Deploy two or more gateway replicas with external Postgres (
deploy/helm/openshell/ci/values-high-availability.yaml). - Create a sandbox and run a few commands in it.
- Restart one gateway pod, so the supervisor's log push reconnects to another replica.
- Run
openshell logs <sandbox>several times, landing on different replicas. Each replica returns a different subset of lines.
Environment
- OpenShell: 0.1.8-pre.5 (main at 0603b15)
- Kubernetes driver, Helm chart, multi-replica gateway with Postgres
Notes
Found through code inspection while verifying #5460, where a non-owner replica held buffers for sandboxes it had only served RPCs for. A possible direction is to send log pushes, log reads, and watches to the ring owner the same way supervisor sessions and relays already are. The #5460 sweep is still needed after that, because ring changes and delete-path cleanup can leave buffers on a former owner.
Related
#5401 / #5460 (freeing buffers on every replica), #5399, #5400
Activity