Repository navigation
fix(relay): publish group state from one locked snapshot - #8898
TheSentinel454 wants to merge 4 commits into
Conversation
kind:39000 and kind:39001 were built from an unlocked channel read and replaced on their own transactions. A membership emission (an admission auto-join, for example) could read a channel just before a kind:9002 edit committed and replace the edit's own 39000 after it, so the stale snapshot won with a later created_at. Only 39002 had a freshness fence. lock_member_snapshot now takes the replacement locks of all three group-state coordinates before the membership lock and reads the live channel row on the same transaction, and emit_group_discovery_events builds and replaces 39000, 39001 and 39002 from that one snapshot. Every channel writer commits before it emits, so a later emission either waits for an earlier one or reads the newer state. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Luke Tornquist <tornquist@squareup.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Luke Tornquist <tornquist@squareup.com>
The postgres-ci nextest profile only selects tests under a postgres_tests:: module, so the held-capture regression test was filtered out of CI. Nest it (and its helper) in one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Luke Tornquist <tornquist@squareup.com>
lock_member_snapshot read the roster identity and the live channel in two statements. A labels-only edit does not take the membership lock, so it could commit between them and one publication would carry mismatched t tags. Read the row once, keep its identity for a deleted channel, and expose it as the live channel only when not deleted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Luke Tornquist <tornquist@squareup.com>
🔐 Codex Security Review
Review SummaryOverall Risk: NONE
FindingsNo concrete security, correctness, or reliability findings were identified. Notes
Generated by Codex Security Review | |
wpfleger96
left a comment
There was a problem hiding this comment.
🤖 Automated review at f7d8043: no blocking findings.
This review covered two independent source reads of the diff plus a live run of the PR head against an isolated relay and database.
Lock order. The snapshot now takes the 39000, 39001 and 39002 replacement keys, then the membership key. No writer on this head holds the membership key, the TTL lock or the channel row lock and then waits on a group-state replacement key. Settings writes take membership, TTL, then the row lock. Kick paths take only membership. Migration 0032's trigger adds membership after the 39002 key. replace_addressable_event and the admin backfill hold one coordinate per transaction. So taking all three in kind order cannot invert against any of them, including relays still running the old build during a rollout.
Behavior parity. The old 39000/39001 path read the channel and roster with the same deleted_at IS NULL filters the snapshot uses, so a deleted channel still returns ChannelNotFound and deletion still removes the discovery heads rather than publishing a final set. The locked replace keeps the old path's stale-write check, follow-up hook and mention indexing. Roster validation stays 39002-only. No references to the removed helpers remain, and the new test hook is gated by #[cfg(test)] at both the call site and the module.
Live run. In 24 unforced rounds, a kind:9021 join raced a kind:9002 edit setting auto_join=false. In every round, the final 39000/39001/39002 heads matched SQL, arrived live on a channel subscription, and had no stale auto_join or label tags. A plain edit, a labels-only edit, a join, an admin promotion and a leave each published three fresh heads with strictly increasing timestamps, with correct signatures, identity tags, admin roles and roster.
Note (non-blocking). Sharing replacement keys makes old and new relays serialize, but it does not make old relays fresh: an old relay still reads the channel before locking. Group-state freshness is fully guaranteed only once every relay runs this build. The live run did not exercise a mixed-version deployment.
|
🤖 Rollout notes from McKeon, Brad Seiler's review agent, at This change has no flag: it takes effect when the relay image deploys. The PR body has no rollout section yet, so here's what I'd add:
|
|
Hi @TheSentinel454, I have resolved the merge conflicts with the latest
A clean, zero-conflict branch is available at: You can pull it directly into your branch: git fetch https://github.com/presentyourlove/buzz.git fix/group-metadata-fence-8898
git merge --ff-only FETCH_HEAD |
## Summary Cold login currently downloads full channel rosters to learn the viewer's memberships. This defines compact discovery in NIP-CL and reserves relay-only kind 39008 for a signed, request-bound response. Full kind-39002 rosters remain compatible. The contract binds a complete snapshot to the community, viewer, NIP-98 request and requested channel types. It defines a primary database cut, overflow errors, lazy-roster precedence, reconnect repair and safe fallback across mixed relay versions. The next PR implements both query modes. This PR does not enable either capability. A separate `membership_public` capability uses relay-only kind 39009 for lookups of any subject’s listed-open forum, stream and workflow memberships. It preserves roles and archived state while excluding private, unlisted, DM and system channels. Signed answers bind requester, subject and public scope; they cannot grant membership or sending permissions. The self-mode contract is unchanged. ### Related issue [BUZZ-615](https://linear.app/squareup/issue/BUZZ-615). No duplicate compact-membership implementation found. Related work: [#8898](#8898) protects full-roster signing; [#9060](#9060) caches membership lookups. ### Testing McKeon independently reviewed the contract and the fixes to request binding, per-channel race fencing and mixed-version fallback. Content checks parse every JSON example and resolve relative links. Paul’s review fixes remove implementation details from the normative text and move the existing 39009 registration and regression test into this PR. Protocol behavior is unchanged. Runtime validation belongs to the implementation PR. This description does not attest human testing. ### Implementation and operator notes Both kinds are registered in `crates/buzz-core/src/kind.rs`, included in `ALL_KINDS`, and rejected as client-submitted events by the relay-only guard. The dependent implementation uses one primary SQL statement’s MVCC snapshot over active `channel_members` joined to nondeleted channels, supported by the partial `(community_id, pubkey) WHERE removed_at IS NULL` index. Public requests share the existing community/requester `ApiCalls` quota configured by `human_api_calls_per_min`. `BUZZ_MEMBERSHIP_SNAPSHOT_ENABLED` and `BUZZ_MEMBERSHIP_PUBLIC_ENABLED` independently default to false. Deploy support across the fleet with both flags off, then enable capabilities independently. Disable the affected capability to roll back; client fallback and back-off behavior remain specified in NIP-CL. Client opt-in and production-shaped throttled validation follow separately. Generated with Codex --------- Signed-off-by: Brad Seiler <seiler@squareup.com> Co-authored-by: Honor <e0bf479eeebe47dde1e35705212a4df70616a7bc834d907ddfb18cf0405f8b1b@seiler-test.communities.public.builderlabstaging.xyz>
## Summary Cold login downloads other people's full rosters just to find the viewer's channels. This adds a compact projection from canonical memberships through the existing Nostr `/query` bridge. One primary SQL statement reads the viewer's rows and channel state, then the relay signs that exact result. Merged spec PR #9117 owns both relay-only kind registrations. Self mode (`membership_snapshot`, kind 39008) requires the viewer's own NIP-98 key. Public mode (`membership_public`, kind 39009) lets an admitted requester look up any subject's listed, open forum/stream/workflow memberships. It excludes private/unlisted channels, DMs, system channels and deleted or removed memberships, including shared private channels. Both modes require payload-bound NIP-98 and return signed request-bound envelopes; overflow or backend errors fail the whole request. Legacy kind-39002 reads and signatures stay intact. No schema migration or persisted per-user list is added. ### Related issue [BUZZ-615](https://linear.app/squareup/issue/BUZZ-615). Implements the compact discovery contract merged in #9117. Related: [#8898](#8898), [#9060](#9060). Client lazy-loading follows separately; this does not include [buzz-app#2627](block/buzz-app#2627). ### Testing Current head `5c0c35400a` rebases the five implementation patches unchanged onto main `c39ee507`, then repairs the removed Redis test API. McKeon independently verified the rebase and reviewed that repair. Full core/database/relay package tests, PostgreSQL membership tests (9/9), Clippy and every mandatory publication hook pass at this head. Old-head CI run 42132655155 failed six jobs on that same removed API; four summary checks propagated those failures. New-head hosted CI is pending. Main advanced to `e3a0f3d59d` during publication; a local merge-tree check found no conflicts. Historical runtime evidence below remains attributed to its tested heads. Prior self-mode validation at `6d2ffceeee` (historical evidence, not validation of the public-mode change): - `just ci`, pre-push hooks and all 2,221 core/DB/relay package tests passed. - All six new Postgres protocol tests passed. All five guard mutations failed assertions: mandatory auth, flag gate, single-statement cut, overflow probe and tenant predicate. A/B are older runs against production files unchanged from `9784df51`; C/D/E were rerun on final SQL/fixtures. The earlier surviving tenant mutation is superseded by the final duplicate-ID witness. - The full isolated Postgres run finished 1,288/1,294 green; six existing push/media/FI/startup cases failed. All six passed when rerun serially after the concurrent local gates stopped. The failed full run is retained and is not represented as a clean full-suite pass. The preceding correction `1c47dfc9` had passed all 1,294 tests. - `just test` passed its unit steps but integration setup was blocked by Docker Desktop organization sign-in enforcement. The waiting wrapper was stopped with SIGTERM; this is not a complete pass. - A real local relay rebuilt at this head returned 67,896 bytes for 468 memberships in 33 ms unthrottled. Signatures, request/viewer/tenant binding, content identity, wrong-viewer refusal, removal and role changes verified. The scale fixture adds five large rosters and 20,000 unrelated channels. Compact bytes stay unchanged as other members grow; EXPLAIN checks 468 indexed viewer rows and bounded channel-ID lookups. A same-UUID cross-tenant fixture verifies fields and duplicate rejection independently of row order. Durable local evidence: `WORK_LOGS/BUZZ_615_20261105/MANIFEST.md` and `MANIFEST.json` map commands/results to commits or verified mutant-file hashes and index the exported logs. Concrete human-test steps and the signed-request helper are included. Human testing remains pending; ready state follows the author’s explicit request. Native/browser 140/205 KB/s acceptance belongs to the client PRs. Public-mode validation at `c6e269170e`: McKeon independently ran all nine isolated PostgreSQL membership tests and five mutations. Every mutation failed its expected assertion: public selection predicate, subject tag, public routing, identity domain and self serialization order. The full core/DB/relay package run, all nine isolated PostgreSQL membership tests and the relay build passed at c6. A real signed HTTP workflow verified signatures, bindings, identity hashes, all visibility exclusions, archived guest roles, requester-independent content identity and malformed-request refusals. After a channel became private, the public list shrank from three entries to two; a role change appeared as admin. The local test relay was stopped afterward. The restacked child `8c56f97d6409` has the same code, tests and configuration as c6; only the spec differs. A new full core/DB/relay package run passed at that exact child head. The parent `17a481a89716` separately passed the full core package check. The first child attempt used the wrong local database and failed five existing tests on a missing table; its log is retained separately from the successful configured run. These checks do not attest human testing. ### Rollout and rollback This PR does not deploy or enable the feature. Both `BUZZ_MEMBERSHIP_SNAPSHOT_ENABLED` and `BUZZ_MEMBERSHIP_PUBLIC_ENABLED` default to false and independently gate advertisement and execution. An authorized operator sets them in the relay Helm environment managed by `squareup/block-coder-tf-stacks`. First deploy the image to every staging pod with both flags off; then enable each capability separately for staging validation. Enabling either mode does not require or enable the other. Repeat that dark-then-enable order for bb-block after staging passes. Opt-in client rollout follows the relay contract and the throttled desktop/browser workflow. Watch `buzz_membership_snapshot_query_seconds`, `buzz_membership_snapshot_encode_sign_seconds`, `buzz_membership_snapshot_response_bytes`, split by `mode=membership_snapshot` or `mode=membership_public`, plus `/query` 413/503 responses and primary-pool wait time for the Authorization operation. Confirm zero unexpected full-roster discovery downloads on supported clients and correct roles/membership against the canonical cut. Abort enablement for any binding/signature mismatch, access leakage, incorrect complete result, stale-response regrant, unexpected overflow for the 468-channel account, sustained 503s or a material regression in primary-pool wait/foreground reads. Turn the affected flag off to roll back. Self-mode clients recheck support and start a new legacy authority generation; public-mode clients recheck support and back off without treating public absence as membership loss. Keep cached display/user data. The first named usable channel target of five seconds at 140 KB/s remains a client acceptance target, not a result established by this relay PR. The 119-event approximately 5 MB read-state contention and native/browser throttled tests are required before broad client rollout. Generated with Codex --------- Signed-off-by: Brad Seiler <seiler@squareup.com> Co-authored-by: Honor <e0bf479eeebe47dde1e35705212a4df70616a7bc834d907ddfb18cf0405f8b1b@seiler-test.communities.public.builderlabstaging.xyz>
Summary
The relay builds kind:39000 (group metadata) and kind:39001 (admins) from a channel read that takes no lock, and replaces each one on its own transaction. A concurrent emission can therefore publish stale state with a newer
created_at:auto_join=false.created_atishead + 1, so the stale tags win.Only kind:39002 had a freshness fence. This race made
test_nip29_announce_channel_wirefail on main after #8594 (https://github.com/block/buzz/actions/runs/41341005385). The race itself is older: any membership change racing a metadata edit could hit it.Change
lock_member_snapshotnow takes the replacement locks of all three group-state coordinates (39000, 39001, 39002), in kind order, before the membership lock. It also reads the live channel row on the same transaction.lock_member_snapshotreads the channel row once: the roster's identity tags (kept for a deleted channel) and the live row behind 39000/39001 come from the same read, so a labels-only edit cannot land between them.replace_member_eventbecomesreplace_group_state_event. It accepts 39000, 39001 or 39002 for the locked channel and validates the roster only for 39002.emit_group_discovery_eventsbuilds all three events from that one snapshot, replaces them on its transaction, commits, and then fans out the events that became heads.Every channel writer commits before it emits. So an emission either finishes before a later one starts, or it reads the later writer's committed change. A capture of older state can no longer replace a newer one.
The replacement keys are the same ones
replace_addressable_eventtakes. During a rolling deploy, a relay that still writes each kind on its own transaction serializes with the new path. No writer holds one of these keys while waiting for another, so taking all three in order cannot deadlock.What got simpler
emit_addressable_discovery_eventis deleted, along with its separate unlockedcreated_atlookup.store_group_state_event), instead of two code paths with different freshness guarantees.Testing
Verified on Blox at
f7d8043:held_group_state_capture_cannot_replace_a_newer_metadata_edit(buzz-relay, Postgres):auto_join=true, commitauto_join=false, then start the edit's emission.auto_jointag.locked_member_snapshot_blocks_post_capture_membership_mutation(buzz-db) is extended:locked_member_snapshot_of_deleted_channel_keeps_identity(buzz-db): a deleted channel's snapshot has no live row but keeps its identity. Mutation: deriving identity from the filtered live row fails it.cargo fmt, clippy (-D warnings), and the full buzz-db (655) and buzz-relay (544) Postgres lanes pass; all three mutations turn their tests RED.auto_join=falserounds left 39000/39001/39002 matching SQL, and plain edit/join/leave publish fresh state live.buzz-review-completed
🤖 Generated with Claude Code