Skip to content

bug: sandbox logs and watch streams ignore gateway ownership in multi-replica mode #5462

Open

Description

@FrostGod

User Story

I run shared OpenShell gateways with several replicas behind a load balancer, and hit this in perf/scale testing while chasing gateway memory growth (#5401). Supervisor sessions follow the gateway ring to the sandbox's owner, but a sandbox's logs and watch streams don't. So any replica can end up holding a sandbox's log buffers, and openshell logs shows a different, partial history depending on which replica the client lands on.

Problem Statement

Only the supervisor session (ConnectSupervisor) is placed on the ring owner. The rest of a sandbox's log path ignores ownership:

  • Log push. The supervisor opens a separate PushSandboxLogs connection to the configured gateway endpoint (the Service), not to the owner it was redirected to. Any replica can receive it, and each reconnect can pick a new one. This applies to dedicated supervisors (openshell-supervisor/src/main.rs) and shared ones (sandbox_instance_host.rs).
  • Log reads and watches. WatchSandbox and GetSandboxLogs serve only the local replica's TracingLogBus and never forward to the owner.
  • Gateway logs. Any replica serving an RPC for a sandbox (create, exec, policy update) records its own lines for that sandbox locally.
  • Platform events. Driver events are published only on the reconciler-lease holder.

Impact / Why This Matters

  • Incomplete logs. A client sees only the lines its replica happened to receive. Lines pushed to another replica, and driver events published on the lease holder, are missing. Retrying can show a different history.
  • Memory fan-out. Over a sandbox's lifetime, its buffers spread across many replicas, so per-replica memory grows with the whole fleet instead of that replica's share. fix(server): free deleted sandboxes' buffers on every replica #5460 frees these buffers after deletion, but doesn't stop the spread.
  • No workaround. There is no way to point a client at the replica that has the full history.

Acceptance Criteria

  • openshell logs <sandbox> and openshell logs <sandbox> --tail return the same complete history (supervisor lines, gateway lines, and platform events) through any replica.
  • A sandbox's live log buffers are held by about one replica, not by every replica that ever touched it.
  • Supervisor log pushes and log reads keep working when the ring changes (scale up or down, rollout), without losing lines.

Reproduction Steps

  1. Deploy two or more gateway replicas with external Postgres (deploy/helm/openshell/ci/values-high-availability.yaml).
  2. Create a sandbox and run a few commands in it.
  3. Restart one gateway pod, so the supervisor's log push reconnects to another replica.
  4. Run openshell logs <sandbox> several times, landing on different replicas. Each replica returns a different subset of lines.

Environment

  • OpenShell: 0.1.8-pre.5 (main at 0603b15)
  • Kubernetes driver, Helm chart, multi-replica gateway with Postgres

Notes

Found through code inspection while verifying #5460, where a non-owner replica held buffers for sandboxes it had only served RPCs for. A possible direction is to send log pushes, log reads, and watches to the ring owner the same way supervisor sessions and relays already are. The #5460 sweep is still needed after that, because ring changes and delete-path cleanup can leave buffers on a former owner.

Related

#5401 / #5460 (freeing buffers on every replica), #5399, #5400

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:gatewayGateway server and control-plane worktopic:observabilityLogging, metrics, and observability work

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions