relay: live fan-out can deliver EVENT for a subscription ID it no longer owns #6319

Problem

The relay can deliver a live EVENT frame for a subscription ID that no longer belongs to the request that produced it. A WebSocket client can therefore receive an event it did not ask for, attributed to a subscription it does own, or receive an EVENT after the relay has already told it that subscription is CLOSED.

Mechanism

Live fan-out selects its recipients, then awaits access resolution, then queues frames — without revalidating ownership at the moment of insertion.

1. fan_out_event_to_local_subscribers snapshots matching (conn_id, sub_id) pairs from the registry (crates/buzz-relay/src/handlers/event.rs:246; the same shape appears at :306 and :431).
2. filter_fanout_by_access awaits channel-visibility and membership lookups (crates/buzz-relay/src/handlers/event.rs:189-212).
3. During that await, the subscription's lifecycle can cut over: either CLOSE x removes it and queues the CLOSED acknowledgement (crates/buzz-relay/src/handlers/close.rs:24-50), or a newer REQ reusing ID x commits and replaces the registry entry (crates/buzz-relay/src/handlers/req.rs:560-611).
4. Fan-out resumes and queues the frame from its stale snapshot via send_fanout_frames (crates/buzz-relay/src/handlers/event.rs:76-92), which calls ConnectionManager::send_to_text_bytes. That validates only that the connection exists and its buffer is not full (crates/buzz-relay/src/state.rs:586-605) — not subscription generation.

Registry ownership is checked when the recipient list is built, not when the frame is inserted into the outbound queue. The await turns the recipient list into a stale capability.

Two observable outcomes:

• After replacement: the client sees an EVENT matching the old filter, labelled with the ID the new subscription now owns. It cannot distinguish the two.
• After CLOSE: the client sees EVENT x arrive after CLOSED x, violating terminal-frame ordering.

Second instance: access-revocation eviction

evict_conn_channel_subscriptions removes registry entries, later awaits the connection map, then releases topics and queues a raw CLOSED (crates/buzz-relay/src/handlers/side_effects.rs:100-137). If a same-ID REQ commits during that window, eviction resumes and removes the newer subscription from the per-connection map and queues a CLOSED naming its ID, while the newer entry remains in the registry. The topic refcount can also be decremented for the evicted subscription after the newer one's retain.

Scope and provenance

This is pre-existing behavior on main, not a regression. It was found while reviewing #2382, which fences the REQ lease family — pre-registration rejection, registration commit, historical replay, CLOSE acknowledgement, same-ID replacement, disconnect drain — under a per-connection lifecycle lock. At #2382's head e8e0fcc34afe637530efd1e260bd75db5235d174, handlers/event.rs, handlers/side_effects.rs, and subscription.rs are byte-identical to main apart from one added line in an event.rs test fixture. #2382 was deliberately scoped to the request lease and does not address live fan-out; this issue tracks that remaining gap.

Suggested direction

Generation-aware live delivery: carry a subscription generation or lease handle on registry recipients and revalidate it at queue-insertion time, sharing the serialization boundary that CLOSE and replacement commit already use. Revocation eviction should participate in the same per-ID transaction or carry the same generation.

Acceptance

Deterministic regressions that pause live fan-out after recipient selection and prove no stale frame is queued, for all three orderings: fan-out vs same-ID replacement, fan-out vs CLOSE, and revocation eviction vs same-ID replacement.

Credit

Mechanism, orderings, and citations independently identified and corroborated by two reviewers during the #2382 concurrency review.