[FrostGod]

Summary

• With multiple gateways, only the replica that served a delete (or holds the reconciler lease) freed the sandbox's log and event buffers.
• Every other replica kept them forever.
• Each gateway now sweeps its own idle buffers and frees the ones whose sandbox is gone.

Related Issue

Closes #5401

Changes

• New bus_sweeper task, every 30s, only on multi-replica (Postgres) gateways.
• Checks only this replica's quiet buffers (no activity for 2 min) with one batched lookup, including watched ones.
• Freeing a watched buffer ends its watch, so openshell logs --tail exits instead of hanging after a delete on another gateway.
• Frees a sandbox after its record is missing on two sweeps in a row; a store error frees nothing.
• New Store::existing_ids(): primary-key IN (...) lookup in batches of 500, Postgres and SQLite.
• TracingLogBus tracks last activity per buffer and lists idle sandboxes.

Release Notes

• Gateways in a multi-replica deployment no longer keep memory for deleted sandboxes.

Testing

• openshell-server fmt, clippy, and unit tests pass
• Unit tests added: frees after two misses, skips recently active buffers, frees watched buffers, a logs-only WatchSandbox ends after the sweep, store failure frees nothing, existing_ids batching
• Postgres test with two store connections as two replicas (mise run test:rust:postgres)
• Live test on k3d with 2 gateways + Postgres: 10 sandboxes deleted via the lease holder were freed on the other gateway ~2.5 min later (closed=10); a live sandbox kept its logs
• Live re-test: a sandbox deleted via the lease holder while logs --tail stayed open on the other gateway was freed ~2.5 min later and the tail exited; a live, watched sandbox kept its tail and logs
• E2E tests added/updated (if applicable)

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[TaylorMutch]

Reviewed commit bb9c594ed72f793d78926eb3358d888b6b011ab1 against #5401. Two findings:

1. A logs-only watch can keep a deleted sandbox's buffers alive indefinitely. Before this PR, deleting through another replica left the buffers allocated. With this PR, that remains true while a logs-only watch stays connected: TracingLogBus::idle_sandbox_ids() excludes any sandbox with a live receiver, but WatchSandbox(follow_status=false) does not subscribe to sandbox deletion notifications. CLI log tailing and SDK log watching use this mode. The stream can wait indefinitely while the sweeper keeps skipping the deleted sandbox, retaining its buffers. Please either check existence for watched buffers too or ensure every watch observes deletion. Add coverage for deletion through another replica while a logs-only watch remains connected.

   References: crates/openshell-server/src/bus_sweeper.rs:68, crates/openshell-server/src/tracing_bus.rs:87, and crates/openshell-server/src/grpc/sandbox.rs:2204.

2. The cleanup deadline does not match the issue's acceptance criterion. bug(gateway): sandbox log/event buffers are freed on only one replica #5401 asks for buffers to be gone within one sweep interval. This PR uses a 30-second interval, a two-minute idle threshold, and two consecutive missing-record observations. When the last activity coincides with deletion, cleanup takes roughly 150–180 seconds under normal scheduling. Please either meet the stated deadline or agree on revised acceptance criteria. This is an acceptance mismatch; the delay follows the implementation's documented design.

   Reference: crates/openshell-server/src/bus_sweeper.rs:24.

Reproduction results using the PR's bus and sweeper code in a standalone component harness:

PASS: 100 sweeps retain deleted sandbox with a live logs receiver; receiver remains open
PASS: dropping the receiver allows cleanup after two sweeps
t=30s: closed=0, buffered=1
t=60s: closed=0, buffered=1
t=90s: closed=0, buffered=1
t=120s: closed=0, buffered=1
t=150s: closed=1, buffered=0

The harness used a deterministic missing-record store response and controlled buffer ages. It exercised the bus/sweeper logic, not a live two-gateway deployment or the complete gRPC producer. The logs-only watch's lack of deletion observation was verified by inspecting the watch handler.

[FrostGod]

@TaylorMutch ,
Fixed in 921bb7e: the sweeper now checks quiet buffers even when watched, and freeing one ends the watch; added unit and WatchSandbox tests for a logs-only watch with deletion elsewhere, plus a live 2-gateway check.

Kept the timing as a deliberate safety margin and updated #5401's criterion to "within a few minutes (about 3 with the default sweep settings)".

[FrostGod]

/ok to test 921bb7e

[TaylorMutch]

Re-reviewed commit 921bb7e. Both previous findings are addressed:

• Quiet buffers are now checked even with live receivers. Component verification confirmed a deleted sandbox's buffers are freed and its receiver closes after two sweeps, while a live watched sandbox survives 100 sweeps with its logs and receiver intact. The added WatchSandbox handler test covers termination of a logs-only watch after deletion elsewhere.
• Issue #5401 now explicitly allows about three minutes for cleanup, matching the default idle threshold and two-sweep confirmation. Component verification confirmed cleanup at a simulated 150 seconds.

Relevant Rust, lint, and PostgreSQL CI checks pass. Deployment E2E jobs were skipped; the PR author reports a successful live two-gateway retest. Local verification was component-level, using deterministic store responses and controlled buffer ages.

No remaining blocking findings.