[FrostGod]

Explainer for reviewers: open the one-page visual (gist source). It covers current vs. proposed behavior, the cap, the stay flag, rollouts, and the one trade-off.

Screenshot

Summary

Shared supervisors now spread evenly across gateway replicas. A supervisor that comes in through the Service goes to the first gateway clockwise on the existing gateway ring that serves fewer than ⌈supervisors ÷ gateways⌉, and a stopping gateway hands its supervisors off the same way. Supervisors move only when they reconnect through the Service; there is no rebalancer.

Related Issue

Closes #5434

Changes

• Protocol: HostGatewayMessage.redirect (HostRedirect { peer_endpoint }) and HostHello.stay.
• Gateway:
  • On attach, a hello without stay is redirected, before any session starts, when the ring and cap place the member elsewhere.
  • On shutdown, each stream ends with a redirect computed on the ring without this replica.
  • Counts come from the pool member rows the reconciler already reads (MemberView.gateway_replica).
  • New helper: GatewayRing::first_below.
• Supervisor:
  • Follows a redirect at once, without backoff, with stay.
  • Also sets stay on the sticky redial (token refresh, blips) and on the Service fallback after a failed redirect, so a stale ring cannot loop it.
  • A gateway always accepts stay.
• Metrics: openshell_server_supervisor_hosts (supervisors per gateway) and openshell_server_supervisor_host_instances_max (busiest supervisor's sandboxes). The second is a max rather than a per-supervisor label because member IDs are pod names, which would make the label set grow without bound.
• Docs: HA page, metrics catalog, and the supervisor-control README.

Known trade-off: nothing moves after placement, so two cases can stay uneven until supervisors restart:

• a gateway added without a rollout;
• a supervisor rolling update with surge, where a new supervisor can't know which old one leaves next.

Gateway rollouts and crashes, and supervisor restarts, end even.

Testing

• cargo clippy -D warnings and cargo fmt --check for openshell-server, openshell-supervisor, openshell-supervisor-control, and the e2e crate
• Unit tests: 2,646 in openshell-server, 293 in openshell-supervisor, 12 in openshell-supervisor-control, all pass. New tests:
  • first_below skips full replicas and falls back to the owner
  • arrivals under the cap fill every replica evenly
  • spread_home ignores the member's own record and leaving members
  • the supervisor follows a redirect at once with stay, then falls back to the Service with stay
• E2E on a local k3d cluster (2 gateways, 2 supervisors, Postgres) with this branch's images:
  • new supervisors_spread_evenly_across_gateways (restart each supervisor, then roll the gateways, and assert one supervisor per gateway): passed 3 of 3 runs
  • pool_sessions_follow_their_supervisors_gateway: passed
  • losing_a_supervisor_pod_moves_its_sandboxes_without_restarting_them stops at its first host-endpoint call with policy_dns_trusted_gateway_unavailable, before any supervisor or gateway moves. The hand-built cluster lacks the host-gateway DNS setup the CI lane provides.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-5436.docs.buildwithfern.com/openshell

[FrostGod]

/ok to test 0f34dee

[github-actions]

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

1. rust-sdk — No explicit risk acceptance found

Under openshell_sdk::raw::proto::supervisor_host::v1, HostHello gains stay and host_gateway_message::Payload gains Redirect, invalidating formerly complete literals and exhaustive matches.

• Impact: Raw SDK consumers constructing HostHello or exhaustively matching Payload fail to compile, although ordinary RPC calls and generated server-trait signatures remain unchanged.
• Action: Preserve the baseline exported Rust shapes for v0.1.8 by isolating the extended private protocol in separate types, or defer the break to a permitted minor train with guidance for stay/default initialization and Redirect handling.
• PR: feat(supervisor): spread shared supervisors evenly across gateway replicas #5436

Review details

• Execution: complete
• Candidate: PR #5436 (44164f0c877a9626ae9c59be58688ec1fdf2efb3)
• Baseline: 4f79018659447c6589dc6a340fda44712ae8ef16 (4f79018659447c6589dc6a340fda44712ae8ef16)
• Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run