From 05ed52180226f911d7acee859f57052314b88677 Mon Sep 17 00:00:00 2001 From: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Date: Mon, 12 Oct 2026 09:54:35 -0400 Subject: [PATCH 01/12] docs(db): database target architecture and schema-change rules Add docs/database.md: the database end state (declarative schema applied by SchemaBot, no triggers/functions/foreign keys, backfills outside migrations, DDL-free relay) and the rules for changes now: every schema location, backward compatibility with the deployed relay, expand -> deploy -> backfill -> contract, and bounded locks. Point to it from AGENTS.md, ARCHITECTURE.md, and path-local AGENTS.md in migrations/ and schema/. Fix the stale claim that migrations auto-apply on startup. Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> --- AGENTS.md | 10 +- ARCHITECTURE.md | 5 + docs/database.md | 297 +++++++++++++++++++++++++++++++++++++++++++ migrations/AGENTS.md | 19 +++ schema/AGENTS.md | 19 +++ 5 files changed, 348 insertions(+), 2 deletions(-) create mode 100644 docs/database.md create mode 100644 migrations/AGENTS.md create mode 100644 schema/AGENTS.md diff --git a/AGENTS.md b/AGENTS.md index 9f75cbd26ac..817d2e44451 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -14,7 +14,12 @@ Before planning or reviewing a non-trivial change: 2. Read the `VISION_*.md` documents relevant to the affected product surface. 3. Read the applicable guidance in [TESTING.md](TESTING.md) and any package-local `TESTING.md`. -4. Check that the proposed design advances, or at least does not contradict, +4. For any change to `migrations/`, `schema/`, or schema-dependent `buzz-db` + queries, read [docs/database.md](docs/database.md): no new triggers, + functions, or foreign keys; no data backfills in migrations; update every + schema location; and keep every migration compatible with the relay that + is currently deployed. +5. Check that the proposed design advances, or at least does not contradict, that product intent. Call out any intentional tension explicitly. Implementation describes the product today; the vision documents describe the @@ -128,7 +133,8 @@ crates/ desktop/ # Tauri 2 + React 19 desktop app web/ # Browser web client (repo browser, served by the relay) mobile/ # Flutter mobile app -migrations/ # SQL migrations (auto-applied on relay startup) +migrations/ # SQL migrations (applied by operators; opt-in at relay startup via BUZZ_AUTO_MIGRATE) +schema/ # Desired-state schema (schema/tables/public/ is the SchemaBot source) scripts/ # Dev tooling .env.example # Config template — copy to .env before running ``` diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 5bb9896d713..12ae0f8bb11 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -525,6 +525,11 @@ shape is an explicit transition, not a flag day: behavior, pgschema bootstraps run reconciliation, and lock/pool metrics remain the operational evidence path for rollout safety. +The database end state (declarative schema applied by SchemaBot, no +triggers, functions, or foreign keys, backfills outside migrations) and the +expand → deploy → backfill → contract rules every schema change follows are in +[docs/database.md](docs/database.md). + **Does NOT:** cache queries, implement connection pooling logic (delegated to sqlx), or make network calls outside Postgres. --- diff --git a/docs/database.md b/docs/database.md new file mode 100644 index 00000000000..c5fc19687d4 --- /dev/null +++ b/docs/database.md @@ -0,0 +1,297 @@ +# Database: target architecture and schema-change rules + +`draft` + +This document states where the Buzz PostgreSQL database is going and the rules +every schema or data change must follow on the way there. Read it before +changing anything under `migrations/`, `schema/`, +`scripts/reconcile-schema-after-pgschema.sql`, or a `buzz-db` query whose +behavior depends on schema shape. + +The key words **MUST**, **MUST NOT**, **SHOULD**, and **MAY** are to be +interpreted as described in BCP 14 when they appear in all capitals. + +## Target architecture + +PostgreSQL stores data. The application owns behavior. + +| Concern | Today | Target | +|---|---|---| +| Schema source of truth | Two parallel sources: ordered sqlx migrations in `migrations/` and the desired-state manifest in `schema/` | The declarative desired state in `schema/tables/public/` is the source of truth. SchemaBot plans each change from it in review and applies it. | +| Who applies DDL | Opt-in startup migrations (`BUZZ_AUTO_MIGRATE`), `buzz-admin migrate`, and partition creation at relay startup and on a schedule | A reviewed external workflow. The serving relay runs no DDL, and its database role cannot. | +| Integrity between tables | Some foreign keys, including `ON DELETE CASCADE` and `ON DELETE SET NULL` | None in the database. The owning application transaction checks references and performs deletes explicitly. | +| Guards and follow-up writes | Triggers and PL/pgSQL functions (`schema/functions/`, `schema/triggers/`) | None in the database. Guards run in the owning writer's transaction; follow-up work is recorded in the same transaction and processed as durable, retryable application work. | +| Data repair | Some historical repair embedded in migrations | Backfill definitions in application code, run by the durable framework in [data-backfills.md](data-backfills.md). | +| Rollout | Effectively schema and code together | Expand, then deploy, then backfill, then contract. Each step is applied and verified on its own. | + +The directories in `schema/` other than `tables/` hold objects SchemaBot cannot +apply. They are expected to shrink toward zero. A change that adds to them +moves away from the target. + +### Why + +- **SchemaBot cannot manage what the target forbids.** SchemaBot's declarative + PostgreSQL format has no foreign keys. It refuses to pull tables that carry + triggers, comments, non-default relation options, or partitioning. It never + executes `DROP TABLE`, and its native apply path admits only `CREATE TABLE`, + `CREATE INDEX`, and bounded `ALTER TABLE` statements. Every new trigger, + function, or foreign key is another object that must be removed before + adoption. +- **Startup must not change the schema.** A relay that migrates at boot can + hold locks on hot tables during a rolling deploy. A relay that starts against + an already-compatible schema cannot. +- **Behavior in the database is hard to see, test, and roll back.** Rust + owns the rest of the write path. Logic split between Rust and PL/pgSQL is + logic with two owners. +- **Migrations are not a job runner.** A data rewrite inside a migration runs + unbounded, in one transaction, while holding locks, with no checkpoint, pause, + or recovery. The backfill framework exists to provide those properties. + +The work is tracked as the safe-startup and SchemaBot adoption effort. Until it +finishes, existing triggers, functions, and foreign keys stay in place as +backstops. They are removed one at a time, after their application replacement +has shipped and been verified. They are not removed early, and new ones are not +added. + +## Rules for changes now + +These rules apply to every change, starting now, including changes that touch +only one table. + +### 1. No new triggers or functions + +A change MUST NOT add `CREATE TRIGGER`, `CREATE FUNCTION`, `CREATE PROCEDURE`, +or `CREATE RULE`. It MUST NOT widen what an existing trigger or function +does. Put the behavior in the Rust transaction that owns the write: + +- **Validation or guards:** check inside the writer's transaction before + committing. Use the existing community-write and replica-floor lock helpers + described under the supported-writer contract in + [ARCHITECTURE.md](../ARCHITECTURE.md). +- **Follow-up writes** (counters, queues, materialized state): write them in the + same transaction, or record a durable work item atomically with the + originating write and process it idempotently. +- **Derived values:** compute them in the application. + +Dropping or narrowing an existing trigger or function is welcome. Follow the +retirement sequence below. + +**Transitional exception: the community write fence.** Until the fence moves +into application writers, every new table with a `community_id` column MUST be +fenced. Do this by calling the existing helper, +`SELECT attach_community_write_fence('')`, in the migration and in +`schema/triggers/
.sql`. The table also goes in the deletion store's +`EXPECTED_SCOPED_TABLES`. Without these steps, the migration lint fails, the +relay's serving-catalog check refuses to start, and community deletion reports +catalog drift. The call attaches an existing, generic trigger; it defines no +new behavior. No other trigger attachment is permitted. + +A `DO` block in a migration is not a persisted function. It MAY guard +idempotent DDL, as migration `0049` does. It MUST NOT be used to run data +repair. + +### 2. No foreign keys + +A change MUST NOT add `REFERENCES` or `FOREIGN KEY`, including in a new table. +For a new reference: + +- Store the referenced key, led by `community_id` for tenant-scoped data. +- Check that the referenced row exists, in the same community, inside the + writing transaction. Lock it or otherwise serialize against a concurrent + delete when a dangling reference would be unsafe. +- Delete or detach dependent rows explicitly in the transaction that deletes + the parent. Community deletion MUST cover the new table; see the fence + exception under Rule 1. +- Make readers tolerate a missing parent. Without a database constraint, a + dangling row is possible after an unsupported write, and readers MUST NOT + panic or leak data across communities when they meet one. + +Removing an existing foreign key requires its replacement checks and explicit +deletes to ship first, then the constraint drop as a separate contract step. + +### 3. Backfills are not migrations + +A migration MUST contain schema changes only. It MUST NOT `UPDATE`, `INSERT`, +or `DELETE` existing rows, and MUST NOT run any statement whose work grows with +table size. + +- Historical data repair MUST be a backfill definition that conforms to + [data-backfills.md](data-backfills.md): stable ID, finite upper bound, + bounded batches, atomic mutation plus checkpoint, and validation. +- The normal write path that maintains the new invariant MUST be fully rolled + out, with old writers drained, before the backfill starts. The backfill spec + explains why and what breaks otherwise. +- Constant seed rows that a new table or registry needs, such as an + `_operator_global_tables` entry, are schema, not backfill. They MAY ship in a + migration. They MUST also appear in `schema/seeds/` and + `scripts/reconcile-schema-after-pgschema.sql`. + +### 4. Change every schema location + +Until SchemaBot owns the schema, one logical change spans several files. A PR +that changes the schema MUST update all of these that apply: + +| Location | What goes there | +|---|---| +| `migrations/NNNN_.sql` | The forward migration for existing databases. Use the next free number; rebase and renumber on a collision. | +| `schema/tables/public/
.sql` | The table's final `CREATE TABLE` and its `CREATE INDEX` statements, one table per file. This is the SchemaBot-readable source. | +| `schema/schema.sql` | An `\i` line for each new file, after the tables it depends on. | +| `schema/types/`, `schema/partitions/`, `schema/seeds/` | The matching non-table objects. The layout test permits nothing else here, and Rule 1 forbids additions to `functions/` and `triggers/`. | +| `scripts/reconcile-schema-after-pgschema.sql` | Idempotent convergence, plus a live assertion, for anything `pgschema` does not preserve: seed rows, some storage parameters, partition attachment, and the shapes listed in AGENTS.md *Common Gotchas* #7. | +| `crates/buzz-push-gateway/migrations/` | Push-gateway schema, which lives in its own database. It has no desired-state directory today, but every rule here still applies. | + +A fresh desired-state database and a migrated database MUST converge on the +same live catalog. The layout test +(`desired_state_schema_layout_is_complete_and_schemabot_shaped`) checks file +structure only. Parity tests cover selected surfaces, and the full drift gate +does not exist yet. For tables and indexes you touch, add or extend a parity +assertion that compares the live catalog from both bootstrap paths, following +`admin_schema_parity_between_desired_state_and_migrations`. A string match +against `schema.sql` does not count. + +The existing migration lints still apply. Every tenant-scoped table has +`community_id NOT NULL`, every key on a scoped table leads with `community_id`, +and operator-global tables are listed explicitly. + +### 5. Every schema change is backward compatible with the running app + +Production does not run migrations automatically. An operator applies them +while the current relay keeps serving traffic, then deploys the new relay. +For a period, therefore, the **old relay runs against the new schema**. If the +new relay is rolled back, the old relay runs against the new schema again. +Every migration MUST be safe for the currently deployed release to read and +write against. + +The relay has no general schema-version compatibility check. It verifies only +specific catalog surfaces, such as the deletion serving fence and the replica +floor guard, so the ordering below is the main protection. When +`BUZZ_AUTO_MIGRATE` is on, an older binary refuses a database that has +migrations it does not know (sqlx `VersionMissing`). A rollback therefore +depends on the older binary not running the migrator. + +One feature therefore usually ships as several PRs, each deployed before the +next starts: + +1. **Expand (schema).** Add only things the old relay ignores: nullable columns, + columns with a constant default, new tables, new indexes. Add nothing the old + relay could violate. +2. **Deploy (code).** Ship the relay that writes the new shape on every write + path and reads both old and new rows correctly. +3. **Backfill (data).** After the new code is fully rolled out and old writers + are drained, repair historical rows with a backfill (Rule 3). Readers stay + correct for both repaired and unrepaired rows throughout. +4. **Contract (schema, then code).** After the backfill completes and + validates, add the constraints you wanted from the start: `NOT NULL`, + `CHECK`, `UNIQUE`. When nothing deployed still reads them, drop the old + columns, indexes, triggers, or functions. Each tightening is its own + migration, applied after the release that stops depending on the old shape. + +Skip a step only when it has nothing to do. For example, a new table with no +historical data needs no backfill. + +#### What breaks the old relay + +| Change | Why it breaks | Safe form | +|---|---|---| +| Add a `NOT NULL` column without a default | Old `INSERT`s omit it and fail | Add it nullable, deploy writers, backfill, then add the constraint (see below) | +| Rename a column or table | Old queries reference the old name | Add the new name, dual-write, backfill, move readers, then drop the old name | +| Change a column type | Old binds and decodes mismatch, and the change may rewrite the table | Add a new column and migrate through the expand/contract sequence | +| Add `CHECK`, `UNIQUE`, or a narrower index | Old writes violate it, or an old `ON CONFLICT` target no longer matches an index | Deploy writers that satisfy it, backfill, then add it. Keep every index an old `ON CONFLICT` names until no deployed code uses it | +| Add an enum value that new code writes | Old code fails to decode the unknown variant | Deploy readers that accept the value, then deploy writers that produce it | +| Drop a column, table, index, trigger, function, or FK | Old code reads it or relies on its side effect | Contract step only, after the release that stops relying on it is fully deployed | +| Add a table with a `community_id` column | The old relay's exact-set deletion catalog check (`EXPECTED_SCOPED_TABLES`) sees an unknown table, so community deletion fails closed until the new release deploys. Serving is unaffected. | Expected and safe. Keep the gap between expand and deploy short, and do not run community deletion during it | +| `SELECT *` or `RETURNING *` decoded by position | Column order differs between desired-state and migrated databases and changes when columns are added | Select named columns | + +#### Adding `NOT NULL` to a populated table + +`ALTER TABLE ... SET NOT NULL` scans the whole table under `ACCESS EXCLUSIVE`. +Instead: + +```sql +-- contract migration A +SET LOCAL lock_timeout = '5s'; +ALTER TABLE t ADD CONSTRAINT t_col_not_null CHECK (col IS NOT NULL) NOT VALID; + +-- contract migration B (separate step; scans but does not block writes) +SET LOCAL lock_timeout = '5s'; +ALTER TABLE t VALIDATE CONSTRAINT t_col_not_null; + +-- contract migration C (PostgreSQL 12+ uses the validated CHECK and skips the scan) +SET LOCAL lock_timeout = '5s'; +ALTER TABLE t ALTER COLUMN col SET NOT NULL; +ALTER TABLE t DROP CONSTRAINT t_col_not_null; +``` + +The desired-state file declares only the final `col ... NOT NULL`. + +#### Retiring a trigger, function, or foreign key + +1. Ship application code that does the trigger's work, or the foreign key's + check and cascade, while the database object is still in place. The two + paths MUST be idempotent with each other, so the duplicate work is harmless. +2. After that release is fully deployed, drop the database object in a + separate migration. Remove it from `schema/` in the same PR. +3. Removing a guard requires a falsifiable test showing that the application + path enforces what the guard enforced. + +### 6. Bound every lock and statement + +- Every migration MUST start with `SET LOCAL lock_timeout`, and SHOULD set + `SET LOCAL statement_timeout` when a statement can scan. A migration that + cannot get its lock fails and is retried. It MUST NOT queue behind + traffic on a hot table. +- An index on an existing populated table MUST be built with + `CREATE INDEX CONCURRENTLY`. CONCURRENTLY cannot run inside the migration + transaction, so follow the prebuild-then-verify procedure in + [thread-window-deployment.md](thread-window-deployment.md). Partitioned + parents need per-partition concurrent builds attached to an `ON ONLY` parent + index. +- A change MUST NOT rewrite or fully scan a large table under a + write-blocking lock. That includes column type changes, volatile defaults, + `SET NOT NULL` without a validated check, and changes to the generated + `events.search_tsv` column. Use the expand/contract forms above. +- Write changes that SchemaBot can apply. Its native path admits + `CREATE TABLE`, `CREATE INDEX`, and bounded `ALTER TABLE` on tables up to its + configured size ceiling (1 GiB by default), with short lock and statement + budgets. A multi-statement plan is not atomic. A change outside that envelope + needs a separately reviewed operational procedure, and the PR MUST say so. + +### 7. Old and new relays coexist during rollout + +Rolling deploys run two relay versions against one database. Durable records +written by the new release, such as backfill rows, outbox items, and new enum +values, MUST be handled safely by the old one. This means ignoring them, +leaving them for the new release, or failing closed. The old release MUST NOT +misinterpret them. + +## Review checklist + +A PR that touches the schema or a schema-dependent query states in its +description: + +- [ ] Which phase this is (expand, deploy, backfill, or contract), and which + PRs or releases come before and after it. +- [ ] That no trigger, function, procedure, rule, or foreign key is added. +- [ ] That no migration mutates existing rows, and the backfill ID for any + repair. +- [ ] That every location in Rule 4 is updated, with a catalog-level parity + test for each changed table and index. +- [ ] That the currently deployed relay works against the new schema, and how + that was checked. +- [ ] Lock and statement timeouts, concurrent index builds, and any operator + prebuild step. + +## Related documents + +- [ARCHITECTURE.md](../ARCHITECTURE.md): *Supported Writer Contract* and the + `buzz-db` crate. +- [data-backfills.md](data-backfills.md): the backfill specification. +- [multi-tenant-conformance.md](multi-tenant-conformance.md): tenant-scoping + obligations. +- [thread-window-deployment.md](thread-window-deployment.md): the brownfield + concurrent-index procedure. +- [crates/buzz-db/TESTING.md](../crates/buzz-db/TESTING.md): desired-state + and migration-mode tests. +- `schema/schema.sql` header: the desired-state layout. +- [SchemaBot PostgreSQL support](https://github.com/block/schemabot/blob/main/docs/postgresql.md): + what SchemaBot can plan and apply. diff --git a/migrations/AGENTS.md b/migrations/AGENTS.md new file mode 100644 index 00000000000..d8c214e95fd --- /dev/null +++ b/migrations/AGENTS.md @@ -0,0 +1,19 @@ +# Schema changes: read first + +Before changing anything in this directory, read +[docs/database.md](../docs/database.md). In short: + +- Add no triggers, functions, procedures, rules, or foreign keys. The one + exception is the community write-fence attachment described in that + document. +- Put no data backfills in migrations: no `UPDATE`, `INSERT`, or `DELETE` over + existing rows. Use the backfill framework in + [docs/data-backfills.md](../docs/data-backfills.md). +- Make each change in every schema location: `migrations/`, + `schema/tables/public/`, the `schema/` manifest and sibling directories, and + `scripts/reconcile-schema-after-pgschema.sql`. +- Keep every migration safe for the relay that is currently deployed. Ship in + this order: new schema, then new code, then backfill, then constraints and + cleanup. +- Bound every lock (`SET LOCAL lock_timeout`). Build indexes on populated + tables concurrently. diff --git a/schema/AGENTS.md b/schema/AGENTS.md new file mode 100644 index 00000000000..d8c214e95fd --- /dev/null +++ b/schema/AGENTS.md @@ -0,0 +1,19 @@ +# Schema changes: read first + +Before changing anything in this directory, read +[docs/database.md](../docs/database.md). In short: + +- Add no triggers, functions, procedures, rules, or foreign keys. The one + exception is the community write-fence attachment described in that + document. +- Put no data backfills in migrations: no `UPDATE`, `INSERT`, or `DELETE` over + existing rows. Use the backfill framework in + [docs/data-backfills.md](../docs/data-backfills.md). +- Make each change in every schema location: `migrations/`, + `schema/tables/public/`, the `schema/` manifest and sibling directories, and + `scripts/reconcile-schema-after-pgschema.sql`. +- Keep every migration safe for the relay that is currently deployed. Ship in + this order: new schema, then new code, then backfill, then constraints and + cleanup. +- Bound every lock (`SET LOCAL lock_timeout`). Build indexes on populated + tables concurrently. From 7621257232cab43c7837cda0bd707fa75b857b8c Mon Sep 17 00:00:00 2001 From: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Date: Mon, 12 Oct 2026 10:18:17 -0400 Subject: [PATCH 02/12] docs(db): address review on fence mechanism, Helm auto-migrate, gateway schema homes Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> --- AGENTS.md | 2 +- crates/buzz-push-gateway/migrations/AGENTS.md | 5 + docs/database.md | 96 ++++++++++++++----- migrations/AGENTS.md | 8 +- schema/AGENTS.md | 19 +--- 5 files changed, 88 insertions(+), 42 deletions(-) create mode 100644 crates/buzz-push-gateway/migrations/AGENTS.md diff --git a/AGENTS.md b/AGENTS.md index 817d2e44451..e8e8dd5124c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -133,7 +133,7 @@ crates/ desktop/ # Tauri 2 + React 19 desktop app web/ # Browser web client (repo browser, served by the relay) mobile/ # Flutter mobile app -migrations/ # SQL migrations (applied by operators; opt-in at relay startup via BUZZ_AUTO_MIGRATE) +migrations/ # SQL migrations (run at relay startup when BUZZ_AUTO_MIGRATE is on: binary default off, Helm default on; production applies them separately — see docs/database.md) schema/ # Desired-state schema (schema/tables/public/ is the SchemaBot source) scripts/ # Dev tooling .env.example # Config template — copy to .env before running diff --git a/crates/buzz-push-gateway/migrations/AGENTS.md b/crates/buzz-push-gateway/migrations/AGENTS.md new file mode 100644 index 00000000000..cfd08de1d52 --- /dev/null +++ b/crates/buzz-push-gateway/migrations/AGENTS.md @@ -0,0 +1,5 @@ +# Schema changes: read first + +Read [docs/database.md](../../../docs/database.md) before changing anything +here. The `push_gateway_*` tables also live in the relay schema (`migrations/` +and `schema/`); a change here goes there too. Never edit a merged migration. diff --git a/docs/database.md b/docs/database.md index c5fc19687d4..81b56424dcf 100644 --- a/docs/database.md +++ b/docs/database.md @@ -18,7 +18,7 @@ PostgreSQL stores data. The application owns behavior. | Concern | Today | Target | |---|---|---| | Schema source of truth | Two parallel sources: ordered sqlx migrations in `migrations/` and the desired-state manifest in `schema/` | The declarative desired state in `schema/tables/public/` is the source of truth. SchemaBot plans each change from it in review and applies it. | -| Who applies DDL | Opt-in startup migrations (`BUZZ_AUTO_MIGRATE`), `buzz-admin migrate`, and partition creation at relay startup and on a schedule | A reviewed external workflow. The serving relay runs no DDL, and its database role cannot. | +| Who applies DDL | Startup migrations when `BUZZ_AUTO_MIGRATE` is on (binary default off, Helm chart default on), `buzz-admin migrate`, the push gateway's pre-upgrade Helm Job, and partition creation at relay startup and on a schedule | A reviewed external workflow. The serving relay runs no DDL, and its database role cannot. Monthly partition creation moves out of the relay under BUZZ-178. Changes to partitioned tables (`events`, `delivery_log`) are outside SchemaBot's envelope: it refuses `CREATE TABLE ... PARTITION OF` on a live parent and concurrent index builds on partitioned parents. | | Integrity between tables | Some foreign keys, including `ON DELETE CASCADE` and `ON DELETE SET NULL` | None in the database. The owning application transaction checks references and performs deletes explicitly. | | Guards and follow-up writes | Triggers and PL/pgSQL functions (`schema/functions/`, `schema/triggers/`) | None in the database. Guards run in the owning writer's transaction; follow-up work is recorded in the same transaction and processed as durable, retryable application work. | | Data repair | Some historical repair embedded in migrations | Backfill definitions in application code, run by the durable framework in [data-backfills.md](data-backfills.md). | @@ -77,14 +77,54 @@ Dropping or narrowing an existing trigger or function is welcome. Follow the retirement sequence below. **Transitional exception: the community write fence.** Until the fence moves -into application writers, every new table with a `community_id` column MUST be -fenced. Do this by calling the existing helper, -`SELECT attach_community_write_fence('
')`, in the migration and in -`schema/triggers/
.sql`. The table also goes in the deletion store's -`EXPECTED_SCOPED_TABLES`. Without these steps, the migration lint fails, the -relay's serving-catalog check refuses to start, and community deletion reports -catalog drift. The call attaches an existing, generic trigger; it defines no -new behavior. No other trigger attachment is permitted. +into application writers (BUZZ-315), every new *tenant-owned* table with a +`community_id` column MUST be fenced by calling the existing helper, +`SELECT attach_community_write_fence('
')`. The call attaches an +existing, generic trigger; it defines no new behavior. No other trigger +attachment is permitted. Follow the checklist below. + +A table whose `community_id` only records provenance, and whose rows must stay +writable while that community is being deleted (the deletion control plane, +`product_feedback`, `rate_limit_violations`, `operator_listener_outbox`), MUST +NOT be fenced. Prefer naming the column something other than `community_id`, +as `relay_admin_actions.report_community_id` does, so the catalog does not +treat the table as scoped. If the column must be named `community_id`, the +table MUST be added to `community_write_fence_excluded_table` in the same +migration and in `schema/functions/`. That edit is part of this exception, and +the only permitted change to an existing function. + +**Nothing cheap catches a missing fence.** Know what actually fails: + +- Community deletion's exact-set catalog check + (`community deletion catalog drift (missing=…, unknown=…)`) and its + write-fence drift check fail closed. +- Relay startup fails only when the binary lists the table in + `EXPECTED_SCOPED_TABLES` and the database has no live fence on it. A binary + that does not list the table starts normally. +- Postgres-gated, migration-mode tests that run catalog validation on a fully + migrated database fail. + +No static lint requires the attach call. Desired-state bootstraps run +`schema/community_write_fence.sql`, which fences every `community_id` table +automatically. Most tests use desired state, so a migration that forgets the +call passes them. The PR MUST run the migration-mode tests. + +#### Checklist: adding a tenant-owned `community_id` table + +All in the same PR: + +- [ ] `SELECT attach_community_write_fence('
')` in the migration. +- [ ] `schema/triggers/
.sql` with the same call, plus its `\i` line. +- [ ] The table in `EXPECTED_SCOPED_TABLES` (`crates/buzz-db/src/store/deletion.rs`). +- [ ] The table in `PURGE_SCOPED_TABLES` in the same file, ordered child + before parent. A table missing here keeps its rows after the purge, and + deletion verification fails closed. +- [ ] The hard-coded fence set in + `deletion_surface_parity_between_migration_0029_and_schema_sql` + (`crates/buzz-db/src/runtime/migration.rs`). +- [ ] The migration count in `embedded_migrator_contains_consolidated_initial_schema`. +- [ ] A run of the Postgres-gated migration-mode tests (see + [crates/buzz-db/TESTING.md](../crates/buzz-db/TESTING.md)). A `DO` block in a migration is not a persisted function. It MAY guard idempotent DDL, as migration `0049` does. It MUST NOT be used to run data @@ -100,8 +140,8 @@ For a new reference: writing transaction. Lock it or otherwise serialize against a concurrent delete when a dangling reference would be unsafe. - Delete or detach dependent rows explicitly in the transaction that deletes - the parent. Community deletion MUST cover the new table; see the fence - exception under Rule 1. + the parent. Community deletion MUST cover the new table; see the checklist + under Rule 1. - Make readers tolerate a missing parent. Without a database constraint, a dangling row is possible after an unsupported write, and readers MUST NOT panic or leak data across communities when they meet one. @@ -112,8 +152,9 @@ deletes to ship first, then the constraint drop as a separate contract step. ### 3. Backfills are not migrations A migration MUST contain schema changes only. It MUST NOT `UPDATE`, `INSERT`, -or `DELETE` existing rows, and MUST NOT run any statement whose work grows with -table size. +or `DELETE` existing rows, and MUST NOT scan or rewrite a table while holding a +write-blocking lock. Scans that do not block writes, such as +`VALIDATE CONSTRAINT` or a bounded concurrent index build, are allowed (Rule 6). - Historical data repair MUST be a backfill definition that conforms to [data-backfills.md](data-backfills.md): stable ID, finite upper bound, @@ -123,8 +164,10 @@ table size. explains why and what breaks otherwise. - Constant seed rows that a new table or registry needs, such as an `_operator_global_tables` entry, are schema, not backfill. They MAY ship in a - migration. They MUST also appear in `schema/seeds/` and - `scripts/reconcile-schema-after-pgschema.sql`. + migration and MUST also appear in `schema/seeds/`. A seed row the running + relay reads (today only `replica_heartbeat`) MUST also be converged and + asserted in `scripts/reconcile-schema-after-pgschema.sql`. Rows read only by + the static lint, such as `_operator_global_tables`, do not go there. ### 4. Change every schema location @@ -133,12 +176,12 @@ that changes the schema MUST update all of these that apply: | Location | What goes there | |---|---| -| `migrations/NNNN_.sql` | The forward migration for existing databases. Use the next free number; rebase and renumber on a collision. | +| `migrations/NNNN_.sql` | The forward migration for existing databases. Use the next free number; rebase and renumber on a collision. Never edit a merged migration: its checksum changes, and relays that run the migrator fail with sqlx `VersionMismatch`. Fix forward in a new migration. | | `schema/tables/public/
.sql` | The table's final `CREATE TABLE` and its `CREATE INDEX` statements, one table per file. This is the SchemaBot-readable source. | | `schema/schema.sql` | An `\i` line for each new file, after the tables it depends on. | -| `schema/types/`, `schema/partitions/`, `schema/seeds/` | The matching non-table objects. The layout test permits nothing else here, and Rule 1 forbids additions to `functions/` and `triggers/`. | +| `schema/types/`, `schema/partitions/`, `schema/seeds/` | The matching non-table objects. The layout test permits nothing else here. Rule 1 forbids additions to `functions/` and `triggers/`, except the fence attachment and exclusion it describes. | | `scripts/reconcile-schema-after-pgschema.sql` | Idempotent convergence, plus a live assertion, for anything `pgschema` does not preserve: seed rows, some storage parameters, partition attachment, and the shapes listed in AGENTS.md *Common Gotchas* #7. | -| `crates/buzz-push-gateway/migrations/` | Push-gateway schema, which lives in its own database. It has no desired-state directory today, but every rule here still applies. | +| `crates/buzz-push-gateway/migrations/` **and** the relay copies | The six `push_gateway_*` tables have two homes. The gateway applies its own migration history to its own database, through a pre-install/pre-upgrade Helm Job ([push-gateway-deployment.md](push-gateway-deployment.md)). The relay schema also defines them: `migrations/` (`0015`, `0043`, `0045`, `0060`), `schema/tables/public/push_gateway_*.sql`, and `schema/seeds/push_gateway_*.sql`. A gateway schema change goes in both, as relay `0060` mirrors gateway `0006`. Every rule here applies to both. | A fresh desired-state database and a migrated database MUST converge on the same live catalog. The layout test @@ -155,8 +198,13 @@ and operator-global tables are listed explicitly. ### 5. Every schema change is backward compatible with the running app -Production does not run migrations automatically. An operator applies them -while the current relay keeps serving traffic, then deploys the new relay. +These rules assume migrations are applied separately from deploys. The relay +binary defaults `BUZZ_AUTO_MIGRATE` to off, but the Helm chart defaults +`migrate.autoMigrate` to `true`. Production MUST set +`migrate.autoMigrate=false`. With it on, the new pod migrates at boot, and +`helm rollback` starts an old binary that runs the migrator against a newer +schema and fails (see below). An operator applies the migrations while the +current relay keeps serving traffic, then deploys the new relay. For a period, therefore, the **old relay runs against the new schema**. If the new relay is rolled back, the old relay runs against the new schema again. Every migration MUST be safe for the currently deployed release to read and @@ -196,7 +244,7 @@ historical data needs no backfill. | Add a `NOT NULL` column without a default | Old `INSERT`s omit it and fail | Add it nullable, deploy writers, backfill, then add the constraint (see below) | | Rename a column or table | Old queries reference the old name | Add the new name, dual-write, backfill, move readers, then drop the old name | | Change a column type | Old binds and decodes mismatch, and the change may rewrite the table | Add a new column and migrate through the expand/contract sequence | -| Add `CHECK`, `UNIQUE`, or a narrower index | Old writes violate it, or an old `ON CONFLICT` target no longer matches an index | Deploy writers that satisfy it, backfill, then add it. Keep every index an old `ON CONFLICT` names until no deployed code uses it | +| Add `CHECK`, `UNIQUE`, or a narrower index | Old writes violate it, or an old `ON CONFLICT` target no longer matches an index | Deploy writers that satisfy it, backfill, then add it. Add a `CHECK` as `NOT VALID`, then `VALIDATE CONSTRAINT` in a later step. Keep every index an old `ON CONFLICT` names until no deployed code uses it | | Add an enum value that new code writes | Old code fails to decode the unknown variant | Deploy readers that accept the value, then deploy writers that produce it | | Drop a column, table, index, trigger, function, or FK | Old code reads it or relies on its side effect | Contract step only, after the release that stops relying on it is fully deployed | | Add a table with a `community_id` column | The old relay's exact-set deletion catalog check (`EXPECTED_SCOPED_TABLES`) sees an unknown table, so community deletion fails closed until the new release deploys. Serving is unaffected. | Expected and safe. Keep the gap between expand and deploy short, and do not run community deletion during it | @@ -222,7 +270,11 @@ ALTER TABLE t ALTER COLUMN col SET NOT NULL; ALTER TABLE t DROP CONSTRAINT t_col_not_null; ``` -The desired-state file declares only the final `col ... NOT NULL`. +The desired-state file declares only the final `col ... NOT NULL`, and that +change lands in the same PR as migration C. Until then, the desired state keeps +the column nullable. Between A and C, migrated databases carry +`t_col_not_null` and fresh ones do not, so exclude that constraint from any +parity assertion on the table for that window. #### Retiring a trigger, function, or foreign key diff --git a/migrations/AGENTS.md b/migrations/AGENTS.md index d8c214e95fd..cb505416454 100644 --- a/migrations/AGENTS.md +++ b/migrations/AGENTS.md @@ -10,8 +10,12 @@ Before changing anything in this directory, read existing rows. Use the backfill framework in [docs/data-backfills.md](../docs/data-backfills.md). - Make each change in every schema location: `migrations/`, - `schema/tables/public/`, the `schema/` manifest and sibling directories, and - `scripts/reconcile-schema-after-pgschema.sql`. + `schema/tables/public/`, the `schema/` manifest and sibling directories, + `scripts/reconcile-schema-after-pgschema.sql` where it applies, and, for + `push_gateway_*` tables, `crates/buzz-push-gateway/migrations/`. +- Never edit a merged migration. Fix forward. +- Adding a `community_id` table has its own checklist in that document. Run + the migration-mode tests; desired-state tests hide a missing fence. - Keep every migration safe for the relay that is currently deployed. Ship in this order: new schema, then new code, then backfill, then constraints and cleanup. diff --git a/schema/AGENTS.md b/schema/AGENTS.md index d8c214e95fd..b7e323b43e9 100644 --- a/schema/AGENTS.md +++ b/schema/AGENTS.md @@ -1,19 +1,4 @@ # Schema changes: read first -Before changing anything in this directory, read -[docs/database.md](../docs/database.md). In short: - -- Add no triggers, functions, procedures, rules, or foreign keys. The one - exception is the community write-fence attachment described in that - document. -- Put no data backfills in migrations: no `UPDATE`, `INSERT`, or `DELETE` over - existing rows. Use the backfill framework in - [docs/data-backfills.md](../docs/data-backfills.md). -- Make each change in every schema location: `migrations/`, - `schema/tables/public/`, the `schema/` manifest and sibling directories, and - `scripts/reconcile-schema-after-pgschema.sql`. -- Keep every migration safe for the relay that is currently deployed. Ship in - this order: new schema, then new code, then backfill, then constraints and - cleanup. -- Bound every lock (`SET LOCAL lock_timeout`). Build indexes on populated - tables concurrently. +This directory follows the same rules as `migrations/`. Read +[docs/database.md](../docs/database.md) and [migrations/AGENTS.md](../migrations/AGENTS.md) before changing anything here. From 62b53da7335ced1ac96c09d4c7fe4e8660a29f08 Mon Sep 17 00:00:00 2001 From: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Date: Mon, 12 Oct 2026 10:23:17 -0400 Subject: [PATCH 03/12] docs(db): give provenance-only tables a deletion disposition and registration checklist Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> --- docs/database.md | 34 ++++++++++++++++++++++++++++++++-- 1 file changed, 32 insertions(+), 2 deletions(-) diff --git a/docs/database.md b/docs/database.md index 81b56424dcf..f0f776ceeaa 100644 --- a/docs/database.md +++ b/docs/database.md @@ -91,7 +91,10 @@ as `relay_admin_actions.report_community_id` does, so the catalog does not treat the table as scoped. If the column must be named `community_id`, the table MUST be added to `community_write_fence_excluded_table` in the same migration and in `schema/functions/`. That edit is part of this exception, and -the only permitted change to an existing function. +the only permitted change to an existing function. Either route removes the +table from every automatic deletion surface, so it also needs the +provenance-only checklist below. Without it, rows that name a deleted +community survive deletion and no check fails. **Nothing cheap catches a missing fence.** Know what actually fails: @@ -126,6 +129,32 @@ All in the same PR: - [ ] A run of the Postgres-gated migration-mode tests (see [crates/buzz-db/TESTING.md](../crates/buzz-db/TESTING.md)). +#### Checklist: adding a provenance-only table + +For a table that records which community a row came from but is not owned by +that community. All in the same PR: + +- [ ] Register it as operator-global in all three places: the + `_operator_global_tables` seed row in the migration, `schema/seeds/`, and + the hard-coded name list in `operator_global_tables()` + (`crates/buzz-db/src/runtime/migration.rs`). The lint ignores a registry + row whose name is not on that list, and without registration the + `community_id NOT NULL` and key-leads-with-`community_id` lints fail. +- [ ] Choose and implement its community-deletion disposition in + `crates/buzz-db/src/store/deletion.rs`: + - **sever:** set the provenance column to `NULL` in the `BindingsRemoved` + step, as `product_feedback` and `rate_limit_violations` are; + - **delete and verify:** add it to `RELAY_ADMIN_TABLES` with a row source, + as the `relay_admin_*` tables are; or + - **retain on purpose:** add it to `CONTROL_PLANE_TABLES` and state the + reason in the PR. +- [ ] Use the `community_write_fence_excluded_table` edit only if the column + must be named `community_id`; otherwise give the column another name. +- [ ] A Postgres-gated deletion test that seeds a row for the target community + and asserts the chosen disposition after + `verify_postgres_logically_deleted`, as the existing tests do for + `product_feedback` and `relay_admin_*`. + A `DO` block in a migration is not a persisted function. It MAY guard idempotent DDL, as migration `0049` does. It MUST NOT be used to run data repair. @@ -194,7 +223,8 @@ against `schema.sql` does not count. The existing migration lints still apply. Every tenant-scoped table has `community_id NOT NULL`, every key on a scoped table leads with `community_id`, -and operator-global tables are listed explicitly. +and operator-global tables are registered in all three places listed in the +provenance-only checklist under Rule 1. ### 5. Every schema change is backward compatible with the running app From 66df3a32fdc61ff1213eddf4052b30267f83b52e Mon Sep 17 00:00:00 2001 From: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Date: Mon, 12 Oct 2026 10:25:41 -0400 Subject: [PATCH 04/12] docs(db): correct provenance-only deletion step, retention, and test precedent Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> --- docs/database.md | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/docs/database.md b/docs/database.md index f0f776ceeaa..05b82429514 100644 --- a/docs/database.md +++ b/docs/database.md @@ -142,18 +142,21 @@ that community. All in the same PR: `community_id NOT NULL` and key-leads-with-`community_id` lints fail. - [ ] Choose and implement its community-deletion disposition in `crates/buzz-db/src/store/deletion.rs`: - - **sever:** set the provenance column to `NULL` in the `BindingsRemoved` - step, as `product_feedback` and `rate_limit_violations` are; + - **sever:** set the provenance column to `NULL` in the `purge_postgres` + transaction, as `product_feedback` and `rate_limit_violations` are; - **delete and verify:** add it to `RELAY_ADMIN_TABLES` with a row source, as the `relay_admin_*` tables are; or - - **retain on purpose:** add it to `CONTROL_PLANE_TABLES` and state the - reason in the PR. + - **retain on purpose:** add no deletion code, and state the reason in a + comment beside the table's migration and in the PR, as + `operator_listener_outbox` is retained. No check enforces this choice. - [ ] Use the `community_write_fence_excluded_table` edit only if the column must be named `community_id`; otherwise give the column another name. - [ ] A Postgres-gated deletion test that seeds a row for the target community - and asserts the chosen disposition after - `verify_postgres_logically_deleted`, as the existing tests do for - `product_feedback` and `relay_admin_*`. + and asserts the chosen disposition. For delete and verify, call + `purge_postgres` and `verify_postgres_logically_deleted`, following + `purge_postgres_succeeds_with_relay_admin_action_rows`. For sever, call + `purge_postgres` and assert the column is `NULL`. This is new coverage: + no existing test drives the sever path through `purge_postgres`. A `DO` block in a migration is not a persisted function. It MAY guard idempotent DDL, as migration `0049` does. It MUST NOT be used to run data From 8da8f7fe44bc6d9322cc563a8e0ee8ad467be311 Mon Sep 17 00:00:00 2001 From: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Date: Tue, 13 Oct 2026 20:13:37 -0400 Subject: [PATCH 05/12] docs(database): drop fence exception, allow interim maintenance scripts, support both migration modes Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> --- AGENTS.md | 2 +- docs/database.md | 151 +++++++++++++++++++++++++------------------ migrations/AGENTS.md | 16 +++-- 3 files changed, 97 insertions(+), 72 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index e8e8dd5124c..d80bed1f1a7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -133,7 +133,7 @@ crates/ desktop/ # Tauri 2 + React 19 desktop app web/ # Browser web client (repo browser, served by the relay) mobile/ # Flutter mobile app -migrations/ # SQL migrations (run at relay startup when BUZZ_AUTO_MIGRATE is on: binary default off, Helm default on; production applies them separately — see docs/database.md) +migrations/ # SQL migrations (run at relay startup when BUZZ_AUTO_MIGRATE is on: binary default off, Helm default on; deployments at scale apply them out of band — see docs/database.md) schema/ # Desired-state schema (schema/tables/public/ is the SchemaBot source) scripts/ # Dev tooling .env.example # Config template — copy to .env before running diff --git a/docs/database.md b/docs/database.md index 05b82429514..14c97f228aa 100644 --- a/docs/database.md +++ b/docs/database.md @@ -17,8 +17,8 @@ PostgreSQL stores data. The application owns behavior. | Concern | Today | Target | |---|---|---| -| Schema source of truth | Two parallel sources: ordered sqlx migrations in `migrations/` and the desired-state manifest in `schema/` | The declarative desired state in `schema/tables/public/` is the source of truth. SchemaBot plans each change from it in review and applies it. | -| Who applies DDL | Startup migrations when `BUZZ_AUTO_MIGRATE` is on (binary default off, Helm chart default on), `buzz-admin migrate`, the push gateway's pre-upgrade Helm Job, and partition creation at relay startup and on a schedule | A reviewed external workflow. The serving relay runs no DDL, and its database role cannot. Monthly partition creation moves out of the relay under BUZZ-178. Changes to partitioned tables (`events`, `delivery_log`) are outside SchemaBot's envelope: it refuses `CREATE TABLE ... PARTITION OF` on a live parent and concurrent index builds on partitioned parents. | +| Schema source of truth | Two parallel sources: ordered sqlx migrations in `migrations/` and the desired-state manifest in `schema/` | The declarative desired state in `schema/tables/public/` is the source of truth. SchemaBot plans each change from it in review and applies it. `migrations/` stays, because auto-migration stays supported (next row). | +| Who applies DDL | Startup migrations when `BUZZ_AUTO_MIGRATE` is on (binary default off, Helm chart default on), `buzz-admin migrate`, the push gateway's pre-upgrade Helm Job, and partition creation at relay startup and on a schedule | Both modes stay supported. Deployments at scale, including Block's hosted deployment, apply schema changes out of band through a reviewed workflow before deploying, and their serving relay runs no DDL. Smaller and self-hosted deployments MAY keep auto-migration on. How the two modes share one schema source after SchemaBot adoption is still open. Monthly partition creation moves out of the relay under BUZZ-178. Changes to partitioned tables (`events`, `delivery_log`) are outside SchemaBot's envelope: it refuses `CREATE TABLE ... PARTITION OF` on a live parent and concurrent index builds on partitioned parents. | | Integrity between tables | Some foreign keys, including `ON DELETE CASCADE` and `ON DELETE SET NULL` | None in the database. The owning application transaction checks references and performs deletes explicitly. | | Guards and follow-up writes | Triggers and PL/pgSQL functions (`schema/functions/`, `schema/triggers/`) | None in the database. Guards run in the owning writer's transaction; follow-up work is recorded in the same transaction and processed as durable, retryable application work. | | Data repair | Some historical repair embedded in migrations | Backfill definitions in application code, run by the durable framework in [data-backfills.md](data-backfills.md). | @@ -37,9 +37,11 @@ moves away from the target. `CREATE INDEX`, and bounded `ALTER TABLE` statements. Every new trigger, function, or foreign key is another object that must be removed before adoption. -- **Startup must not change the schema.** A relay that migrates at boot can - hold locks on hot tables during a rolling deploy. A relay that starts against - an already-compatible schema cannot. +- **At scale, startup should not change the schema.** A relay that migrates at + boot can hold locks on hot tables during a rolling deploy. A relay that + starts against an already-compatible schema cannot. Auto-migration stays + supported for deployments where that risk is acceptable, so every change + must be safe in both modes (Rule 5). - **Behavior in the database is hard to see, test, and roll back.** Rust owns the rest of the write path. Logic split between Rust and PL/pgSQL is logic with two owners. @@ -76,63 +78,74 @@ does. Put the behavior in the Rust transaction that owns the write: Dropping or narrowing an existing trigger or function is welcome. Follow the retirement sequence below. -**Transitional exception: the community write fence.** Until the fence moves -into application writers (BUZZ-315), every new *tenant-owned* table with a -`community_id` column MUST be fenced by calling the existing helper, -`SELECT attach_community_write_fence('
')`. The call attaches an -existing, generic trigger; it defines no new behavior. No other trigger -attachment is permitted. Follow the checklist below. - -A table whose `community_id` only records provenance, and whose rows must stay -writable while that community is being deleted (the deletion control plane, -`product_feedback`, `rate_limit_violations`, `operator_listener_outbox`), MUST -NOT be fenced. Prefer naming the column something other than `community_id`, -as `relay_admin_actions.report_community_id` does, so the catalog does not -treat the table as scoped. If the column must be named `community_id`, the -table MUST be added to `community_write_fence_excluded_table` in the same -migration and in `schema/functions/`. That edit is part of this exception, and -the only permitted change to an existing function. Either route removes the -table from every automatic deletion surface, so it also needs the -provenance-only checklist below. Without it, rows that name a deleted -community survive deletion and no check fails. - -**Nothing cheap catches a missing fence.** Know what actually fails: - -- Community deletion's exact-set catalog check - (`community deletion catalog drift (missing=…, unknown=…)`) and its - write-fence drift check fail closed. -- Relay startup fails only when the binary lists the table in - `EXPECTED_SCOPED_TABLES` and the database has no live fence on it. A binary - that does not list the table starts normally. -- Postgres-gated, migration-mode tests that run catalog validation on a fully - migrated database fail. - -No static lint requires the attach call. Desired-state bootstraps run -`schema/community_write_fence.sql`, which fences every `community_id` table -automatically. Most tests use desired state, so a migration that forgets the -call passes them. The PR MUST run the migration-mode tests. +**No exception for the community write fence.** A new table MUST NOT get a +fence trigger, including by calling the existing helper +`attach_community_write_fence`. Writers of a new tenant-owned table MUST be +fenced in the application instead, by community admission: the transaction +takes the shared community admission lock for its community before it writes. +Today `AdmittedTx` (`crates/buzz-db/src/runtime/admitted_tx.rs`) does this for +the `events`, `reactions`, and `event_mentions` writers, and the compiler +rejects an unadmitted transaction. The design that extends admission to every +fenced table and names a chokepoint for each writer is BUZZ-314; the +implementation, which ends by dropping the fence triggers and functions, is +BUZZ-315. + +**Today this blocks new tenant-owned tables.** Community deletion's catalog +checks still assume that every tenant-scoped table carries a trigger fence: + +- The exact-set catalog check + (`community deletion catalog drift (missing=…, unknown=…)`) and the + write-fence drift check fail closed on a `community_id` table that has no + trigger. +- Relay startup refuses a binary that lists the table in + `EXPECTED_SCOPED_TABLES` when the database has no live fence on it + (`community serving write fences missing`). +- Desired-state bootstraps run `schema/community_write_fence.sql`, which + attaches a fence trigger to every `community_id` table automatically. + +So before a feature can add a tenant-owned `community_id` table, these checks +must recognise application-fenced tables, and the desired-state loop must stop +attaching triggers to them. That change belongs to the BUZZ-314/BUZZ-315 work. +Land it before the feature, or as the feature's first PR. Do not work around +it with a trigger, or with an entry in `community_write_fence_excluded_table` +for a table the community owns. #### Checklist: adding a tenant-owned `community_id` table -All in the same PR: +After the catalog change above has landed, all in the same PR: -- [ ] `SELECT attach_community_write_fence('
')` in the migration. -- [ ] `schema/triggers/
.sql` with the same call, plus its `\i` line. -- [ ] The table in `EXPECTED_SCOPED_TABLES` (`crates/buzz-db/src/store/deletion.rs`). +- [ ] Every writer of the table goes through community admission + (`AdmittedTx`, or the chokepoint BUZZ-314 defines for its writer). + Nothing writes the table outside it. +- [ ] No fence trigger: no `attach_community_write_fence` call in the + migration and no `schema/triggers/` file. +- [ ] The table in `EXPECTED_SCOPED_TABLES` (`crates/buzz-db/src/store/deletion.rs`), + plus any registration the catalog change introduces for + application-fenced tables. - [ ] The table in `PURGE_SCOPED_TABLES` in the same file, ordered child before parent. A table missing here keeps its rows after the purge, and deletion verification fails closed. -- [ ] The hard-coded fence set in - `deletion_surface_parity_between_migration_0029_and_schema_sql` - (`crates/buzz-db/src/runtime/migration.rs`). - [ ] The migration count in `embedded_migrator_contains_consolidated_initial_schema`. +- [ ] A Postgres-gated test showing that a write to the table is refused once + deletion of its community has started, and that the purge removes its + rows. - [ ] A run of the Postgres-gated migration-mode tests (see - [crates/buzz-db/TESTING.md](../crates/buzz-db/TESTING.md)). + [crates/buzz-db/TESTING.md](../crates/buzz-db/TESTING.md)). Most tests + use desired state, which builds the catalog differently. #### Checklist: adding a provenance-only table -For a table that records which community a row came from but is not owned by -that community. All in the same PR: +A table whose `community_id` only records which community a row came from, and +whose rows must stay writable while that community is being deleted (like the +deletion control plane, `product_feedback`, `rate_limit_violations`, and +`operator_listener_outbox`), is not owned by that community and is not fenced. +Name its column something other than `community_id`, as +`relay_admin_actions.report_community_id` does, so the catalog does not treat +the table as scoped. Adding it to `community_write_fence_excluded_table` +instead would change an existing function, which Rule 1 forbids. Renaming the +column also removes the table from every automatic deletion surface, so rows +that name a deleted community survive deletion and no check fails, unless the +PR follows this checklist. All in the same PR: - [ ] Register it as operator-global in all three places: the `_operator_global_tables` seed row in the migration, `schema/seeds/`, and @@ -149,8 +162,6 @@ that community. All in the same PR: - **retain on purpose:** add no deletion code, and state the reason in a comment beside the table's migration and in the PR, as `operator_listener_outbox` is retained. No check enforces this choice. -- [ ] Use the `community_write_fence_excluded_table` edit only if the column - must be named `community_id`; otherwise give the column another name. - [ ] A Postgres-gated deletion test that seeds a row for the target community and asserts the chosen disposition. For delete and verify, call `purge_postgres` and `verify_postgres_logically_deleted`, following @@ -191,6 +202,12 @@ write-blocking lock. Scans that do not block writes, such as - Historical data repair MUST be a backfill definition that conforms to [data-backfills.md](data-backfills.md): stable ID, finite upper bound, bounded batches, atomic mutation plus checkpoint, and validation. +- **Until the backfill framework ships**, a repair MAY instead be a one-off + SQL script in `scripts/maintenance/`, reviewed in its own PR and run by an + operator. The script MUST be idempotent and safe to rerun, MUST work in + bounded batches that each commit on their own, and MUST end with a + validation query whose expected result the PR states. Once the framework + ships, new repairs MUST use it. - The normal write path that maintains the new invariant MUST be fully rolled out, with old writers drained, before the backfill starts. The backfill spec explains why and what breaks otherwise. @@ -211,7 +228,7 @@ that changes the schema MUST update all of these that apply: | `migrations/NNNN_.sql` | The forward migration for existing databases. Use the next free number; rebase and renumber on a collision. Never edit a merged migration: its checksum changes, and relays that run the migrator fail with sqlx `VersionMismatch`. Fix forward in a new migration. | | `schema/tables/public/
.sql` | The table's final `CREATE TABLE` and its `CREATE INDEX` statements, one table per file. This is the SchemaBot-readable source. | | `schema/schema.sql` | An `\i` line for each new file, after the tables it depends on. | -| `schema/types/`, `schema/partitions/`, `schema/seeds/` | The matching non-table objects. The layout test permits nothing else here. Rule 1 forbids additions to `functions/` and `triggers/`, except the fence attachment and exclusion it describes. | +| `schema/types/`, `schema/partitions/`, `schema/seeds/` | The matching non-table objects. The layout test permits nothing else here. Rule 1 forbids any addition to `functions/` and `triggers/`. | | `scripts/reconcile-schema-after-pgschema.sql` | Idempotent convergence, plus a live assertion, for anything `pgschema` does not preserve: seed rows, some storage parameters, partition attachment, and the shapes listed in AGENTS.md *Common Gotchas* #7. | | `crates/buzz-push-gateway/migrations/` **and** the relay copies | The six `push_gateway_*` tables have two homes. The gateway applies its own migration history to its own database, through a pre-install/pre-upgrade Helm Job ([push-gateway-deployment.md](push-gateway-deployment.md)). The relay schema also defines them: `migrations/` (`0015`, `0043`, `0045`, `0060`), `schema/tables/public/push_gateway_*.sql`, and `schema/seeds/push_gateway_*.sql`. A gateway schema change goes in both, as relay `0060` mirrors gateway `0006`. Every rule here applies to both. | @@ -231,14 +248,17 @@ provenance-only checklist under Rule 1. ### 5. Every schema change is backward compatible with the running app -These rules assume migrations are applied separately from deploys. The relay -binary defaults `BUZZ_AUTO_MIGRATE` to off, but the Helm chart defaults -`migrate.autoMigrate` to `true`. Production MUST set -`migrate.autoMigrate=false`. With it on, the new pod migrates at boot, and -`helm rollback` starts an old binary that runs the migrator against a newer -schema and fails (see below). An operator applies the migrations while the -current relay keeps serving traffic, then deploys the new relay. -For a period, therefore, the **old relay runs against the new schema**. If the +Buzz supports two ways of applying migrations, and every change MUST be safe +under both: + +- **Auto-migration.** The relay binary defaults `BUZZ_AUTO_MIGRATE` to off, + but the Helm chart defaults `migrate.autoMigrate` to `true`. With it on, the + first new pod migrates at boot while old pods are still serving. +- **Out of band.** An operator applies the migrations while the current relay + keeps serving traffic, then deploys the new relay. Block's hosted + deployment works this way at scale, with `migrate.autoMigrate=false`. + +Either way, for a period the **old relay runs against the new schema**. If the new relay is rolled back, the old relay runs against the new schema again. Every migration MUST be safe for the currently deployed release to read and write against. @@ -247,8 +267,10 @@ The relay has no general schema-version compatibility check. It verifies only specific catalog surfaces, such as the deletion serving fence and the replica floor guard, so the ordering below is the main protection. When `BUZZ_AUTO_MIGRATE` is on, an older binary refuses a database that has -migrations it does not know (sqlx `VersionMissing`). A rollback therefore -depends on the older binary not running the migrator. +migrations it does not know (sqlx `VersionMissing`), so `helm rollback` with +auto-migration on fails at startup. A rollback therefore depends on the older +binary not running the migrator. Making rollback work under auto-migration is +part of supporting both modes. One feature therefore usually ships as several PRs, each deployed before the next starts: @@ -357,7 +379,8 @@ description: - [ ] Which phase this is (expand, deploy, backfill, or contract), and which PRs or releases come before and after it. - [ ] That no trigger, function, procedure, rule, or foreign key is added. -- [ ] That no migration mutates existing rows, and the backfill ID for any +- [ ] That no migration mutates existing rows, and the backfill ID (or, + until the framework ships, the `scripts/maintenance/` script) for any repair. - [ ] That every location in Rule 4 is updated, with a catalog-level parity test for each changed table and index. diff --git a/migrations/AGENTS.md b/migrations/AGENTS.md index cb505416454..cdce9f2bfbb 100644 --- a/migrations/AGENTS.md +++ b/migrations/AGENTS.md @@ -3,20 +3,22 @@ Before changing anything in this directory, read [docs/database.md](../docs/database.md). In short: -- Add no triggers, functions, procedures, rules, or foreign keys. The one - exception is the community write-fence attachment described in that - document. +- Add no triggers, functions, procedures, rules, or foreign keys. There are no + exceptions, including the community write-fence attachment. - Put no data backfills in migrations: no `UPDATE`, `INSERT`, or `DELETE` over existing rows. Use the backfill framework in - [docs/data-backfills.md](../docs/data-backfills.md). + [docs/data-backfills.md](../docs/data-backfills.md), or, until it ships, a + reviewed one-off script in `scripts/maintenance/`. - Make each change in every schema location: `migrations/`, `schema/tables/public/`, the `schema/` manifest and sibling directories, `scripts/reconcile-schema-after-pgschema.sql` where it applies, and, for `push_gateway_*` tables, `crates/buzz-push-gateway/migrations/`. - Never edit a merged migration. Fix forward. -- Adding a `community_id` table has its own checklist in that document. Run - the migration-mode tests; desired-state tests hide a missing fence. -- Keep every migration safe for the relay that is currently deployed. Ship in +- Adding a `community_id` table has its own checklist in that document. A + tenant-owned one is blocked until deletion's catalog checks accept + application-fenced tables (BUZZ-314/BUZZ-315). +- Keep every migration safe for the relay that is currently deployed, whether + migrations run at startup or out of band. Ship in this order: new schema, then new code, then backfill, then constraints and cleanup. - Bound every lock (`SET LOCAL lock_timeout`). Build indexes on populated From 6146fe2fdd88494393dfb23a0854e7510eeb5c9e Mon Sep 17 00:00:00 2001 From: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Date: Tue, 13 Oct 2026 20:24:48 -0400 Subject: [PATCH 06/12] docs(database): address re-review and owner comments (repair-script fencing, fail-closed contracts, gateway ownership, declarative wording) Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> --- AGENTS.md | 2 +- crates/buzz-push-gateway/migrations/AGENTS.md | 6 +- docs/database.md | 104 ++++++++++++------ migrations/AGENTS.md | 8 +- 4 files changed, 82 insertions(+), 38 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index d80bed1f1a7..355af9ba687 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -134,7 +134,7 @@ desktop/ # Tauri 2 + React 19 desktop app web/ # Browser web client (repo browser, served by the relay) mobile/ # Flutter mobile app migrations/ # SQL migrations (run at relay startup when BUZZ_AUTO_MIGRATE is on: binary default off, Helm default on; deployments at scale apply them out of band — see docs/database.md) -schema/ # Desired-state schema (schema/tables/public/ is the SchemaBot source) +schema/ # Desired-state schema (schema/tables/public/ is the declarative schema source) scripts/ # Dev tooling .env.example # Config template — copy to .env before running ``` diff --git a/crates/buzz-push-gateway/migrations/AGENTS.md b/crates/buzz-push-gateway/migrations/AGENTS.md index cfd08de1d52..685bad191d3 100644 --- a/crates/buzz-push-gateway/migrations/AGENTS.md +++ b/crates/buzz-push-gateway/migrations/AGENTS.md @@ -1,5 +1,7 @@ # Schema changes: read first Read [docs/database.md](../../../docs/database.md) before changing anything -here. The `push_gateway_*` tables also live in the relay schema (`migrations/` -and `schema/`); a change here goes there too. Never edit a merged migration. +here. The existing `push_gateway_*` tables are also duplicated in the relay +schema (`migrations/` and `schema/`); until that duplication is removed, a +change to them goes there too. The target is for the gateway to own its tables +alone, so put new gateway tables only here. Never edit a merged migration. diff --git a/docs/database.md b/docs/database.md index 14c97f228aa..73e5c665244 100644 --- a/docs/database.md +++ b/docs/database.md @@ -17,8 +17,8 @@ PostgreSQL stores data. The application owns behavior. | Concern | Today | Target | |---|---|---| -| Schema source of truth | Two parallel sources: ordered sqlx migrations in `migrations/` and the desired-state manifest in `schema/` | The declarative desired state in `schema/tables/public/` is the source of truth. SchemaBot plans each change from it in review and applies it. `migrations/` stays, because auto-migration stays supported (next row). | -| Who applies DDL | Startup migrations when `BUZZ_AUTO_MIGRATE` is on (binary default off, Helm chart default on), `buzz-admin migrate`, the push gateway's pre-upgrade Helm Job, and partition creation at relay startup and on a schedule | Both modes stay supported. Deployments at scale, including Block's hosted deployment, apply schema changes out of band through a reviewed workflow before deploying, and their serving relay runs no DDL. Smaller and self-hosted deployments MAY keep auto-migration on. How the two modes share one schema source after SchemaBot adoption is still open. Monthly partition creation moves out of the relay under BUZZ-178. Changes to partitioned tables (`events`, `delivery_log`) are outside SchemaBot's envelope: it refuses `CREATE TABLE ... PARTITION OF` on a live parent and concurrent index builds on partitioned parents. | +| Schema source of truth | Two parallel sources: ordered sqlx migrations in `migrations/` and the desired-state manifest in `schema/` | The declarative desired state in `schema/tables/public/` is the source of truth. Deployments that apply schema out of band plan each change from it; Block's hosted deployment uses SchemaBot for this, but the source does not depend on any one tool. `migrations/` stays, because auto-migration stays supported (next row). Each app owns its own migrations and declarative schema: the relay's copies of the push gateway's tables are debt to remove (Rule 4). | +| Who applies DDL | Startup migrations when `BUZZ_AUTO_MIGRATE` is on (binary default off, Helm chart default on), `buzz-admin migrate`, the push gateway's pre-upgrade Helm Job, and partition creation at relay startup and on a schedule | Both modes stay supported. Deployments at scale, including Block's hosted deployment, apply schema changes out of band through a reviewed workflow before deploying, and their serving relay runs no DDL. Smaller and self-hosted deployments MAY keep auto-migration on. Still open: how the two modes share one schema source, and how auto-migration orders a release whose expand, backfill, and contract steps arrive in one upgrade (Rule 5). Monthly partition creation moves out of the relay under BUZZ-178. Changes to partitioned tables (`events`, `delivery_log`) are outside SchemaBot's envelope: it refuses `CREATE TABLE ... PARTITION OF` on a live parent and concurrent index builds on partitioned parents. | | Integrity between tables | Some foreign keys, including `ON DELETE CASCADE` and `ON DELETE SET NULL` | None in the database. The owning application transaction checks references and performs deletes explicitly. | | Guards and follow-up writes | Triggers and PL/pgSQL functions (`schema/functions/`, `schema/triggers/`) | None in the database. Guards run in the owning writer's transaction; follow-up work is recorded in the same transaction and processed as durable, retryable application work. | | Data repair | Some historical repair embedded in migrations | Backfill definitions in application code, run by the durable framework in [data-backfills.md](data-backfills.md). | @@ -64,7 +64,8 @@ only one table. A change MUST NOT add `CREATE TRIGGER`, `CREATE FUNCTION`, `CREATE PROCEDURE`, or `CREATE RULE`. It MUST NOT widen what an existing trigger or function -does. Put the behavior in the Rust transaction that owns the write: +does; this document treats any edit to an existing function's body, other +than dropping or narrowing it, as widening. Put the behavior in the Rust transaction that owns the write: - **Validation or guards:** check inside the writer's transaction before committing. Use the existing community-write and replica-floor lock helpers @@ -81,22 +82,28 @@ retirement sequence below. **No exception for the community write fence.** A new table MUST NOT get a fence trigger, including by calling the existing helper `attach_community_write_fence`. Writers of a new tenant-owned table MUST be -fenced in the application instead, by community admission: the transaction -takes the shared community admission lock for its community before it writes. -Today `AdmittedTx` (`crates/buzz-db/src/runtime/admitted_tx.rs`) does this for -the `events`, `reactions`, and `event_mentions` writers, and the compiler -rejects an unadmitted transaction. The design that extends admission to every -fenced table and names a chokepoint for each writer is BUZZ-314; the -implementation, which ends by dropping the fence triggers and functions, is -BUZZ-315. +fenced in the application instead, by community admission. Today +`AdmittedTx` (`crates/buzz-db/src/runtime/admitted_tx.rs`) wraps event-write +transactions: event ingestion and the writes it makes in the same +transaction. It is constructed by `DeletionStore::guard_transaction`, which +takes the shared community deletion lock and refuses a community that is not +`active`. Helpers that take `&mut AdmittedTx` reject a raw transaction at +compile time, but crate code can still bypass the type, so the fence triggers +remain the authoritative backstop (ARCHITECTURE.md § *Supported Writer +Contract*). A table without a trigger has no database backstop. The design +that extends admission to every fenced table and names a chokepoint for each +writer is BUZZ-314; the implementation, which ends by dropping the fence +triggers and functions, is BUZZ-315. That work MUST also update +ARCHITECTURE.md § *Dual enforcement*. **Today this blocks new tenant-owned tables.** Community deletion's catalog checks still assume that every tenant-scoped table carries a trigger fence: -- The exact-set catalog check - (`community deletion catalog drift (missing=…, unknown=…)`) and the - write-fence drift check fail closed on a `community_id` table that has no - trigger. +- An unlisted `community_id` table, fenced or not, fails the exact-set check + against `EXPECTED_SCOPED_TABLES` + (`community deletion catalog drift (missing=…, unknown=…)`). Listing it + without a trigger fails the write-fence drift check. Either way, deletion + fails closed for every community, not only the new table's. - Relay startup refuses a binary that lists the table in `EXPECTED_SCOPED_TABLES` when the database has no live fence on it (`community serving write fences missing`). @@ -136,13 +143,15 @@ After the catalog change above has landed, all in the same PR: #### Checklist: adding a provenance-only table A table whose `community_id` only records which community a row came from, and -whose rows must stay writable while that community is being deleted (like the -deletion control plane, `product_feedback`, `rate_limit_violations`, and -`operator_listener_outbox`), is not owned by that community and is not fenced. -Name its column something other than `community_id`, as +whose rows must stay writable while that community is being deleted, is not +owned by that community and is not fenced. The existing ones (the deletion +control plane, `product_feedback`, `rate_limit_violations`, and +`operator_listener_outbox`) predate this rule and are listed in +`community_write_fence_excluded_table`. A new one MUST name its column +something other than `community_id`, as `relay_admin_actions.report_community_id` does, so the catalog does not treat -the table as scoped. Adding it to `community_write_fence_excluded_table` -instead would change an existing function, which Rule 1 forbids. Renaming the +the table as scoped. Adding a case to the exclusion function would widen an +existing function, which Rule 1 forbids. Renaming the column also removes the table from every automatic deletion surface, so rows that name a deleted community survive deletion and no check fails, unless the PR follows this checklist. All in the same PR: @@ -170,8 +179,8 @@ PR follows this checklist. All in the same PR: no existing test drives the sever path through `purge_postgres`. A `DO` block in a migration is not a persisted function. It MAY guard -idempotent DDL, as migration `0049` does. It MUST NOT be used to run data -repair. +idempotent DDL, as migration `0049` does, or make a contract migration refuse +to run (Rule 5). It MUST NOT be used to run data repair. ### 2. No foreign keys @@ -206,8 +215,19 @@ write-blocking lock. Scans that do not block writes, such as SQL script in `scripts/maintenance/`, reviewed in its own PR and run by an operator. The script MUST be idempotent and safe to rerun, MUST work in bounded batches that each commit on their own, and MUST end with a - validation query whose expected result the PR states. Once the framework - ships, new repairs MUST use it. + validation query whose expected result the PR states. Commit per batch + from a top-level `DO` block with `COMMIT` (outside any `BEGIN`) or from a + psql or shell loop; never with `CREATE FUNCTION` or `CREATE PROCEDURE`, + even a temporary one. Once the framework ships, new repairs MUST use it. +- A repair script MUST respect community deletion, as + [ARCHITECTURE.md](../ARCHITECTURE.md) § *Supported Writer Contract* + requires of operator maintenance. Each batch MUST take + `pg_advisory_xact_lock_shared(community_deletion_lock_key())` for + every community it touches, and MUST touch only rows whose community has + `deletion_state = 'active'` and `deleted_at IS NULL`, as + `DeletionStore::guard_transaction` does. Otherwise the PR MUST document a + reviewed procedure that keeps the routing fences closed. A script MUST NOT + disable triggers or set `session_replication_role`. - The normal write path that maintains the new invariant MUST be fully rolled out, with old writers drained, before the backfill starts. The backfill spec explains why and what breaks otherwise. @@ -220,17 +240,17 @@ write-blocking lock. Scans that do not block writes, such as ### 4. Change every schema location -Until SchemaBot owns the schema, one logical change spans several files. A PR +Until the declarative schema is the only source, one logical change spans several files. A PR that changes the schema MUST update all of these that apply: | Location | What goes there | |---|---| | `migrations/NNNN_.sql` | The forward migration for existing databases. Use the next free number; rebase and renumber on a collision. Never edit a merged migration: its checksum changes, and relays that run the migrator fail with sqlx `VersionMismatch`. Fix forward in a new migration. | -| `schema/tables/public/
.sql` | The table's final `CREATE TABLE` and its `CREATE INDEX` statements, one table per file. This is the SchemaBot-readable source. | +| `schema/tables/public/
.sql` | The table's final `CREATE TABLE` and its `CREATE INDEX` statements, one table per file. This is the declarative schema source. | | `schema/schema.sql` | An `\i` line for each new file, after the tables it depends on. | | `schema/types/`, `schema/partitions/`, `schema/seeds/` | The matching non-table objects. The layout test permits nothing else here. Rule 1 forbids any addition to `functions/` and `triggers/`. | | `scripts/reconcile-schema-after-pgschema.sql` | Idempotent convergence, plus a live assertion, for anything `pgschema` does not preserve: seed rows, some storage parameters, partition attachment, and the shapes listed in AGENTS.md *Common Gotchas* #7. | -| `crates/buzz-push-gateway/migrations/` **and** the relay copies | The six `push_gateway_*` tables have two homes. The gateway applies its own migration history to its own database, through a pre-install/pre-upgrade Helm Job ([push-gateway-deployment.md](push-gateway-deployment.md)). The relay schema also defines them: `migrations/` (`0015`, `0043`, `0045`, `0060`), `schema/tables/public/push_gateway_*.sql`, and `schema/seeds/push_gateway_*.sql`. A gateway schema change goes in both, as relay `0060` mirrors gateway `0006`. Every rule here applies to both. | +| `crates/buzz-push-gateway/migrations/` **and** the relay copies | The six `push_gateway_*` tables have two homes. The gateway applies its own migration history to its own database, through a pre-install/pre-upgrade Helm Job ([push-gateway-deployment.md](push-gateway-deployment.md)). The relay schema also defines them: `migrations/` (`0015`, `0043`, `0045`, `0060`), `schema/tables/public/push_gateway_*.sql`, and `schema/seeds/push_gateway_*.sql`. Until the duplication is removed, a gateway schema change goes in both, as relay `0060` mirrors gateway `0006`. Every rule here applies to both. The target is for the gateway to own its tables alone, in its own migrations and declarative schema; do not add new gateway tables to the relay schema. | A fresh desired-state database and a migrated database MUST converge on the same live catalog. The layout test @@ -268,7 +288,8 @@ specific catalog surfaces, such as the deletion serving fence and the replica floor guard, so the ordering below is the main protection. When `BUZZ_AUTO_MIGRATE` is on, an older binary refuses a database that has migrations it does not know (sqlx `VersionMissing`), so `helm rollback` with -auto-migration on fails at startup. A rollback therefore depends on the older +auto-migration on fails at startup, and so does any old pod that restarts +mid-rollout after a new pod has migrated. A rollback therefore depends on the older binary not running the migrator. Making rollback work under auto-migration is part of supporting both modes. @@ -292,6 +313,25 @@ next starts: Skip a step only when it has nothing to do. For example, a new table with no historical data needs no backfill. +**Under auto-migration the steps can collapse.** The migrator applies every +pending migration at one boot, before the new code serves and before any +backfill or repair script runs. An operator who upgrades across several +releases therefore applies expand and contract together. How auto-migration +should order such an upgrade is unresolved. Until it is, a contract migration +MUST fail closed when the step it depends on has not run: + +- `VALIDATE CONSTRAINT` and `SET NOT NULL` already fail on unrepaired rows. +- A contract that drops a column or table, or otherwise loses data, MUST first + check that the backfill or repair has completed, and `RAISE` from a + top-level `DO` block when unrepaired rows remain. +- The PR MUST add a Postgres-gated test that applies the expand and contract + migrations in one run against unrepaired rows and asserts the migration + errors without dropping data. + +A failed contract stops the relay from starting, which is an outage but not +data loss. The release notes MUST tell auto-migrating operators which repair +to run before upgrading past that release. + #### What breaks the old relay | Change | Why it breaks | Safe form | @@ -380,8 +420,10 @@ description: PRs or releases come before and after it. - [ ] That no trigger, function, procedure, rule, or foreign key is added. - [ ] That no migration mutates existing rows, and the backfill ID (or, - until the framework ships, the `scripts/maintenance/` script) for any - repair. + until the framework ships, the `scripts/maintenance/` script and how it + takes the community deletion lock) for any repair. +- [ ] For a contract migration, how it fails closed when the step it depends + on has not run. - [ ] That every location in Rule 4 is updated, with a catalog-level parity test for each changed table and index. - [ ] That the currently deployed relay works against the new schema, and how diff --git a/migrations/AGENTS.md b/migrations/AGENTS.md index cdce9f2bfbb..7c572455956 100644 --- a/migrations/AGENTS.md +++ b/migrations/AGENTS.md @@ -12,7 +12,8 @@ Before changing anything in this directory, read - Make each change in every schema location: `migrations/`, `schema/tables/public/`, the `schema/` manifest and sibling directories, `scripts/reconcile-schema-after-pgschema.sql` where it applies, and, for - `push_gateway_*` tables, `crates/buzz-push-gateway/migrations/`. + existing `push_gateway_*` tables, `crates/buzz-push-gateway/migrations/`. + Do not add new gateway tables here. - Never edit a merged migration. Fix forward. - Adding a `community_id` table has its own checklist in that document. A tenant-owned one is blocked until deletion's catalog checks accept @@ -20,6 +21,5 @@ Before changing anything in this directory, read - Keep every migration safe for the relay that is currently deployed, whether migrations run at startup or out of band. Ship in this order: new schema, then new code, then backfill, then constraints and - cleanup. -- Bound every lock (`SET LOCAL lock_timeout`). Build indexes on populated - tables concurrently. + cleanup. A contract migration fails closed if its backfill has not run. +- Bound every lock (`SET LOCAL lock_timeout`). From e9aee548c7719810b2b8267a342bee8c7c13859a Mon Sep 17 00:00:00 2001 From: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Date: Tue, 13 Oct 2026 20:28:05 -0400 Subject: [PATCH 07/12] docs(database): address re-review nits (lock-then-check, release-note ordering, declarative wording, wrapping) Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> --- ARCHITECTURE.md | 5 +++-- docs/database.md | 33 ++++++++++++++++++++------------- 2 files changed, 23 insertions(+), 15 deletions(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 12ae0f8bb11..c3990c3973b 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -525,8 +525,9 @@ shape is an explicit transition, not a flag day: behavior, pgschema bootstraps run reconciliation, and lock/pool metrics remain the operational evidence path for rollout safety. -The database end state (declarative schema applied by SchemaBot, no -triggers, functions, or foreign keys, backfills outside migrations) and the +The database end state (a declarative schema source, applied out of band at +scale, where Block uses SchemaBot; no triggers, functions, or foreign keys; +backfills outside migrations) and the expand → deploy → backfill → contract rules every schema change follows are in [docs/database.md](docs/database.md). diff --git a/docs/database.md b/docs/database.md index 73e5c665244..dbc6b9f0fd9 100644 --- a/docs/database.md +++ b/docs/database.md @@ -18,7 +18,7 @@ PostgreSQL stores data. The application owns behavior. | Concern | Today | Target | |---|---|---| | Schema source of truth | Two parallel sources: ordered sqlx migrations in `migrations/` and the desired-state manifest in `schema/` | The declarative desired state in `schema/tables/public/` is the source of truth. Deployments that apply schema out of band plan each change from it; Block's hosted deployment uses SchemaBot for this, but the source does not depend on any one tool. `migrations/` stays, because auto-migration stays supported (next row). Each app owns its own migrations and declarative schema: the relay's copies of the push gateway's tables are debt to remove (Rule 4). | -| Who applies DDL | Startup migrations when `BUZZ_AUTO_MIGRATE` is on (binary default off, Helm chart default on), `buzz-admin migrate`, the push gateway's pre-upgrade Helm Job, and partition creation at relay startup and on a schedule | Both modes stay supported. Deployments at scale, including Block's hosted deployment, apply schema changes out of band through a reviewed workflow before deploying, and their serving relay runs no DDL. Smaller and self-hosted deployments MAY keep auto-migration on. Still open: how the two modes share one schema source, and how auto-migration orders a release whose expand, backfill, and contract steps arrive in one upgrade (Rule 5). Monthly partition creation moves out of the relay under BUZZ-178. Changes to partitioned tables (`events`, `delivery_log`) are outside SchemaBot's envelope: it refuses `CREATE TABLE ... PARTITION OF` on a live parent and concurrent index builds on partitioned parents. | +| Who applies DDL | Startup migrations when `BUZZ_AUTO_MIGRATE` is on (binary default off, Helm chart default on), `buzz-admin migrate`, the push gateway's pre-upgrade Helm Job, and partition creation at relay startup and on a schedule | Both modes stay supported. Deployments at scale, including Block's hosted deployment, apply schema changes out of band through a reviewed workflow before deploying, and their serving relay runs no DDL. Smaller and self-hosted deployments MAY keep auto-migration on. Still open: how the two modes share one schema source, how auto-migration orders a release whose expand, backfill, and contract steps arrive in one upgrade, and how an operator recovers after a contract migration fails at boot (Rule 5). Monthly partition creation moves out of the relay under BUZZ-178. Changes to partitioned tables (`events`, `delivery_log`) are outside SchemaBot's envelope: it refuses `CREATE TABLE ... PARTITION OF` on a live parent and concurrent index builds on partitioned parents. | | Integrity between tables | Some foreign keys, including `ON DELETE CASCADE` and `ON DELETE SET NULL` | None in the database. The owning application transaction checks references and performs deletes explicitly. | | Guards and follow-up writes | Triggers and PL/pgSQL functions (`schema/functions/`, `schema/triggers/`) | None in the database. Guards run in the owning writer's transaction; follow-up work is recorded in the same transaction and processed as durable, retryable application work. | | Data repair | Some historical repair embedded in migrations | Backfill definitions in application code, run by the durable framework in [data-backfills.md](data-backfills.md). | @@ -65,7 +65,8 @@ only one table. A change MUST NOT add `CREATE TRIGGER`, `CREATE FUNCTION`, `CREATE PROCEDURE`, or `CREATE RULE`. It MUST NOT widen what an existing trigger or function does; this document treats any edit to an existing function's body, other -than dropping or narrowing it, as widening. Put the behavior in the Rust transaction that owns the write: +than dropping or narrowing it, as widening. Put the behavior in the Rust +transaction that owns the write: - **Validation or guards:** check inside the writer's transaction before committing. Use the existing community-write and replica-floor lock helpers @@ -223,9 +224,11 @@ write-blocking lock. Scans that do not block writes, such as [ARCHITECTURE.md](../ARCHITECTURE.md) § *Supported Writer Contract* requires of operator maintenance. Each batch MUST take `pg_advisory_xact_lock_shared(community_deletion_lock_key())` for - every community it touches, and MUST touch only rows whose community has - `deletion_state = 'active'` and `deleted_at IS NULL`, as - `DeletionStore::guard_transaction` does. Otherwise the PR MUST document a + every community it touches. After taking the lock, in the same + transaction, it MUST read the community's state and touch only rows whose + community has `deletion_state = 'active'` and `deleted_at IS NULL`, as + `DeletionStore::guard_transaction` does. Selecting active communities + before taking the lock races with deletion. Otherwise the PR MUST document a reviewed procedure that keeps the routing fences closed. A script MUST NOT disable triggers or set `session_replication_role`. - The normal write path that maintains the new invariant MUST be fully rolled @@ -240,8 +243,9 @@ write-blocking lock. Scans that do not block writes, such as ### 4. Change every schema location -Until the declarative schema is the only source, one logical change spans several files. A PR -that changes the schema MUST update all of these that apply: +Until the declarative schema is the only source, one logical change spans +several files. A PR that changes the schema MUST update all of these that +apply: | Location | What goes there | |---|---| @@ -289,9 +293,9 @@ floor guard, so the ordering below is the main protection. When `BUZZ_AUTO_MIGRATE` is on, an older binary refuses a database that has migrations it does not know (sqlx `VersionMissing`), so `helm rollback` with auto-migration on fails at startup, and so does any old pod that restarts -mid-rollout after a new pod has migrated. A rollback therefore depends on the older -binary not running the migrator. Making rollback work under auto-migration is -part of supporting both modes. +mid-rollout after a new pod has migrated. A rollback therefore depends on the +older binary not running the migrator. Making rollback work under +auto-migration is part of supporting both modes. One feature therefore usually ships as several PRs, each deployed before the next starts: @@ -328,9 +332,12 @@ MUST fail closed when the step it depends on has not run: migrations in one run against unrepaired rows and asserts the migration errors without dropping data. -A failed contract stops the relay from starting, which is an outage but not -data loss. The release notes MUST tell auto-migrating operators which repair -to run before upgrading past that release. +A failed contract stops the relay from starting. That is an outage, not data +loss, but recovery is not clean: the expand migrations before the contract +have already committed, so an old pod that restarts hits `VersionMissing`. +Because a repair needs the new writer fully rolled out (Rule 3), the release +notes MUST name the release to stop at (the one that ships the writer) and the +repair to run there before upgrading past the contract release. #### What breaks the old relay From 30d5294e69cf5a7a53ffbd1c148b21ad38c583de Mon Sep 17 00:00:00 2001 From: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Date: Wed, 14 Oct 2026 14:15:00 -0400 Subject: [PATCH 08/12] docs(database): scope CREATE INDEX CONCURRENTLY to migrations/ Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> --- docs/database.md | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/docs/database.md b/docs/database.md index dbc6b9f0fd9..8d09402bc10 100644 --- a/docs/database.md +++ b/docs/database.md @@ -394,12 +394,18 @@ parity assertion on the table for that window. `SET LOCAL statement_timeout` when a statement can scan. A migration that cannot get its lock fails and is retried. It MUST NOT queue behind traffic on a hot table. -- An index on an existing populated table MUST be built with - `CREATE INDEX CONCURRENTLY`. CONCURRENTLY cannot run inside the migration - transaction, so follow the prebuild-then-verify procedure in - [thread-window-deployment.md](thread-window-deployment.md). Partitioned - parents need per-partition concurrent builds attached to an `ON ONLY` parent - index. +- In `migrations/`, an index on an existing populated table MUST be built + with `CREATE INDEX CONCURRENTLY`, because auto-migrating deployments have no + online schema change tool to build it for them. CONCURRENTLY cannot run + inside the migration transaction, so follow the prebuild-then-verify + procedure in [thread-window-deployment.md](thread-window-deployment.md). + Partitioned parents need per-partition concurrent builds attached to an + `ON ONLY` parent index. +- The declarative schema source declares indexes with plain `CREATE INDEX`. + The tool that applies it out of band decides how to build them online, so + `schema/` files MUST NOT add `CONCURRENTLY`. SchemaBot refuses it, and + `desired_state_schema_layout_is_complete_and_schemabot_shaped` enforces + plain `CREATE INDEX`. - A change MUST NOT rewrite or fully scan a large table under a write-blocking lock. That includes column type changes, volatile defaults, `SET NOT NULL` without a validated check, and changes to the generated @@ -435,8 +441,8 @@ description: test for each changed table and index. - [ ] That the currently deployed relay works against the new schema, and how that was checked. -- [ ] Lock and statement timeouts, concurrent index builds, and any operator - prebuild step. +- [ ] Lock and statement timeouts, concurrent index builds in `migrations/`, + and any operator prebuild step. ## Related documents From fea5400c3b72113cc4677e2af1a8d4fc79cfd17a Mon Sep 17 00:00:00 2001 From: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Date: Fri, 16 Oct 2026 17:07:29 -0400 Subject: [PATCH 09/12] docs(database): drop tool and tracker names, separate writer prerequisites from data checks Describe the out-of-band apply constraints without naming a specific tool, remove internal tracker IDs and the draft label, and address review feedback: writer-release prerequisites are not provable by row checks, repair scripts that write historical channel events keep replica routing closed, community admission is named by its entry point after the fleet-sweep change, the NOT NULL example is three transactional migration files, and the old-relay table qualifies the new community_id table row. Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> --- AGENTS.md | 2 +- ARCHITECTURE.md | 2 +- docs/database.md | 156 +++++++++++++++++++++++-------------------- migrations/AGENTS.md | 5 +- 4 files changed, 87 insertions(+), 78 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 355af9ba687..3830f4879c3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -608,7 +608,7 @@ description. See [PR #803](https://github.com/block/buzz/pull/803). 4. **Worktrees: `cd` in the same command** — shell CWD doesn't persist between tool calls. Use `cd /path && cargo build` as one command. 5. **Desktop crate excluded from root workspace** — `cargo test` at repo root does NOT run desktop tests. Use `cargo test --manifest-path desktop/src-tauri/Cargo.toml` explicitly. 6. **React render perf: `React.memo` is all-or-nothing** — it only skips a re-render when *every* prop is reference-stable; one unstable prop (inline arrow/JSX, or a hook returning a fresh `{}`/`[]`/`Map` each render) defeats it. Two repeat offenders: (a) React Query results (`useMutation`/`useQuery`) are a **new object each render** — depend on the stable method (`mutation.mutateAsync`), not the object; (b) derived `Map`/array state that recomputes on a version bump — wrap in a content-equality ref cache (`shared/hooks/useStableReference.ts`). When chasing interaction lag, **measure with DevTools closed and no perf probes** (an open Web Inspector + per-keystroke `console.log` inflate the numbers), and isolate by removing one suspect at a time rather than guessing. -7. **`pgschema` omits seed DML and some storage parameters** — `schema/schema.sql` is an ordered `\i` manifest: `schema/tables/public/
.sql` holds only that table's `CREATE TABLE` and `CREATE INDEX` statements (one table per file, the shape SchemaBot reads), and types, functions, partitions, triggers, and seed rows live in the sibling `schema/` directories. Fresh desired-state bootstraps use `./bin/pgschema apply`, which does not execute `INSERT` statements or preserve every table storage parameter from `schema/schema.sql`. Put each unsupported invariant in `scripts/reconcile-schema-after-pgschema.sql` as an idempotent convergence statement plus a live catalog or data assertion. Every `pgschema apply` caller must run that script. A string assertion against `schema.sql` alone does not prove the pgschema-created database has the intended state. +7. **`pgschema` omits seed DML and some storage parameters** — `schema/schema.sql` is an ordered `\i` manifest: `schema/tables/public/
.sql` holds only that table's `CREATE TABLE` and `CREATE INDEX` statements (one table per file, the shape out-of-band schema tools read), and types, functions, partitions, triggers, and seed rows live in the sibling `schema/` directories. Fresh desired-state bootstraps use `./bin/pgschema apply`, which does not execute `INSERT` statements or preserve every table storage parameter from `schema/schema.sql`. Put each unsupported invariant in `scripts/reconcile-schema-after-pgschema.sql` as an idempotent convergence statement plus a live catalog or data assertion. Every `pgschema apply` caller must run that script. A string assertion against `schema.sql` alone does not prove the pgschema-created database has the intended state. --- diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index c3990c3973b..fcfb965ee9a 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -526,7 +526,7 @@ shape is an explicit transition, not a flag day: the operational evidence path for rollout safety. The database end state (a declarative schema source, applied out of band at -scale, where Block uses SchemaBot; no triggers, functions, or foreign keys; +scale; no triggers, functions, or foreign keys; backfills outside migrations) and the expand → deploy → backfill → contract rules every schema change follows are in [docs/database.md](docs/database.md). diff --git a/docs/database.md b/docs/database.md index 8d09402bc10..6421d009457 100644 --- a/docs/database.md +++ b/docs/database.md @@ -1,7 +1,5 @@ # Database: target architecture and schema-change rules -`draft` - This document states where the Buzz PostgreSQL database is going and the rules every schema or data change must follow on the way there. Read it before changing anything under `migrations/`, `schema/`, @@ -17,26 +15,26 @@ PostgreSQL stores data. The application owns behavior. | Concern | Today | Target | |---|---|---| -| Schema source of truth | Two parallel sources: ordered sqlx migrations in `migrations/` and the desired-state manifest in `schema/` | The declarative desired state in `schema/tables/public/` is the source of truth. Deployments that apply schema out of band plan each change from it; Block's hosted deployment uses SchemaBot for this, but the source does not depend on any one tool. `migrations/` stays, because auto-migration stays supported (next row). Each app owns its own migrations and declarative schema: the relay's copies of the push gateway's tables are debt to remove (Rule 4). | -| Who applies DDL | Startup migrations when `BUZZ_AUTO_MIGRATE` is on (binary default off, Helm chart default on), `buzz-admin migrate`, the push gateway's pre-upgrade Helm Job, and partition creation at relay startup and on a schedule | Both modes stay supported. Deployments at scale, including Block's hosted deployment, apply schema changes out of band through a reviewed workflow before deploying, and their serving relay runs no DDL. Smaller and self-hosted deployments MAY keep auto-migration on. Still open: how the two modes share one schema source, how auto-migration orders a release whose expand, backfill, and contract steps arrive in one upgrade, and how an operator recovers after a contract migration fails at boot (Rule 5). Monthly partition creation moves out of the relay under BUZZ-178. Changes to partitioned tables (`events`, `delivery_log`) are outside SchemaBot's envelope: it refuses `CREATE TABLE ... PARTITION OF` on a live parent and concurrent index builds on partitioned parents. | +| Schema source of truth | Two parallel sources: ordered sqlx migrations in `migrations/` and the desired-state manifest in `schema/` | The declarative desired state in `schema/tables/public/` is the source of truth. Deployments that apply schema out of band plan each change from it, with the tool of their choice. `migrations/` stays, because auto-migration stays supported (next row). Each app owns its own migrations and declarative schema: the relay's copies of the push gateway's tables are debt to remove (Rule 4). | +| Who applies DDL | Startup migrations when `BUZZ_AUTO_MIGRATE` is on (binary default off, Helm chart default on), `buzz-admin migrate`, the push gateway's pre-upgrade Helm Job, and partition creation at relay startup and on a schedule | Both modes stay supported. Deployments at scale, including Block's hosted deployment, apply schema changes out of band through a reviewed workflow before deploying, and their serving relay runs no DDL. Smaller and self-hosted deployments MAY keep auto-migration on. Still open: how the two modes share one schema source, how auto-migration orders a release whose expand, backfill, and contract steps arrive in one upgrade, and how an operator recovers after a contract migration fails at boot (Rule 5). Monthly partition creation is planned to move out of the relay. Changes to partitioned tables (`events`, `delivery_log`) are outside the out-of-band envelope (Rule 6): attaching a partition to a live parent and building an index on a partitioned parent need their own reviewed procedure. | | Integrity between tables | Some foreign keys, including `ON DELETE CASCADE` and `ON DELETE SET NULL` | None in the database. The owning application transaction checks references and performs deletes explicitly. | | Guards and follow-up writes | Triggers and PL/pgSQL functions (`schema/functions/`, `schema/triggers/`) | None in the database. Guards run in the owning writer's transaction; follow-up work is recorded in the same transaction and processed as durable, retryable application work. | | Data repair | Some historical repair embedded in migrations | Backfill definitions in application code, run by the durable framework in [data-backfills.md](data-backfills.md). | | Rollout | Effectively schema and code together | Expand, then deploy, then backfill, then contract. Each step is applied and verified on its own. | -The directories in `schema/` other than `tables/` hold objects SchemaBot cannot -apply. They are expected to shrink toward zero. A change that adds to them -moves away from the target. +The directories in `schema/` other than `tables/` hold objects that out-of-band +apply cannot manage. They are expected to shrink toward zero. A change that +adds to them moves away from the target. ### Why -- **SchemaBot cannot manage what the target forbids.** SchemaBot's declarative - PostgreSQL format has no foreign keys. It refuses to pull tables that carry - triggers, comments, non-default relation options, or partitioning. It never - executes `DROP TABLE`, and its native apply path admits only `CREATE TABLE`, - `CREATE INDEX`, and bounded `ALTER TABLE` statements. Every new trigger, - function, or foreign key is another object that must be removed before - adoption. +- **Out-of-band apply needs plain tables.** Applying changes online from a + declarative source works only for plain tables and indexes: no foreign keys, + and no triggers, comments, non-default relation options, or partitioning on + the tables it manages. It does not drop tables, and it applies only + `CREATE TABLE`, `CREATE INDEX`, and bounded `ALTER TABLE` statements. Every + new trigger, function, or foreign key is another object that must be removed + before the schema can be applied this way. - **At scale, startup should not change the schema.** A relay that migrates at boot can hold locks on hot tables during a rolling deploy. A relay that starts against an already-compatible schema cannot. Auto-migration stays @@ -49,7 +47,7 @@ moves away from the target. unbounded, in one transaction, while holding locks, with no checkpoint, pause, or recovery. The backfill framework exists to provide those properties. -The work is tracked as the safe-startup and SchemaBot adoption effort. Until it +The move to declarative, out-of-band schema changes is in progress. Until it finishes, existing triggers, functions, and foreign keys stay in place as backstops. They are removed one at a time, after their application replacement has shipped and been verified. They are not removed early, and new ones are not @@ -69,9 +67,7 @@ than dropping or narrowing it, as widening. Put the behavior in the Rust transaction that owns the write: - **Validation or guards:** check inside the writer's transaction before - committing. Use the existing community-write and replica-floor lock helpers - described under the supported-writer contract in - [ARCHITECTURE.md](../ARCHITECTURE.md). + committing. Tenant writes go through community admission, described below. - **Follow-up writes** (counters, queues, materialized state): write them in the same transaction, or record a durable work item atomically with the originating write and process it idempotently. @@ -83,19 +79,19 @@ retirement sequence below. **No exception for the community write fence.** A new table MUST NOT get a fence trigger, including by calling the existing helper `attach_community_write_fence`. Writers of a new tenant-owned table MUST be -fenced in the application instead, by community admission. Today -`AdmittedTx` (`crates/buzz-db/src/runtime/admitted_tx.rs`) wraps event-write -transactions: event ingestion and the writes it makes in the same -transaction. It is constructed by `DeletionStore::guard_transaction`, which -takes the shared community deletion lock and refuses a community that is not -`active`. Helpers that take `&mut AdmittedTx` reject a raw transaction at -compile time, but crate code can still bypass the type, so the fence triggers -remain the authoritative backstop (ARCHITECTURE.md § *Supported Writer -Contract*). A table without a trigger has no database backstop. The design -that extends admission to every fenced table and names a chokepoint for each -writer is BUZZ-314; the implementation, which ends by dropping the fence -triggers and functions, is BUZZ-315. That work MUST also update -ARCHITECTURE.md § *Dual enforcement*. +fenced in the application instead, by community admission. Serving writers, +including fleet sweeps (one community at a time), open their transaction with +`begin_community_write_transaction` or one of its variants +(`crates/buzz-db/src/runtime/mod.rs`). It returns an `AdmittedTx` +(`crates/buzz-db/src/runtime/admitted_tx.rs`), built by +`DeletionStore::guard_transaction`, which takes the shared community deletion +lock and refuses a community that is not `active`. Helpers that take +`&mut AdmittedTx` reject a raw transaction at compile time, but crate code can +still bypass the type, so the fence triggers remain the authoritative backstop +(ARCHITECTURE.md § *Supported Writer Contract*). A table without a trigger has +no database backstop. The remaining fence-retirement work makes every writer of +every fenced table go through admission, then drops the fence triggers and +functions. That work MUST also update ARCHITECTURE.md § *Dual enforcement*. **Today this blocks new tenant-owned tables.** Community deletion's catalog checks still assume that every tenant-scoped table carries a trigger fence: @@ -113,7 +109,7 @@ checks still assume that every tenant-scoped table carries a trigger fence: So before a feature can add a tenant-owned `community_id` table, these checks must recognise application-fenced tables, and the desired-state loop must stop -attaching triggers to them. That change belongs to the BUZZ-314/BUZZ-315 work. +attaching triggers to them. That change belongs to the fence-retirement work. Land it before the feature, or as the feature's first PR. Do not work around it with a trigger, or with an entry in `community_write_fence_excluded_table` for a table the community owns. @@ -123,8 +119,8 @@ for a table the community owns. After the catalog change above has landed, all in the same PR: - [ ] Every writer of the table goes through community admission - (`AdmittedTx`, or the chokepoint BUZZ-314 defines for its writer). - Nothing writes the table outside it. + (`begin_community_write_transaction` or another constructor that returns + `AdmittedTx`). Nothing writes the table outside it. - [ ] No fence trigger: no `attach_community_write_fence` call in the migration and no `schema/triggers/` file. - [ ] The table in `EXPECTED_SCOPED_TABLES` (`crates/buzz-db/src/store/deletion.rs`), @@ -228,9 +224,14 @@ write-blocking lock. Scans that do not block writes, such as transaction, it MUST read the community's state and touch only rows whose community has `deletion_state = 'active'` and `deleted_at IS NULL`, as `DeletionStore::guard_transaction` does. Selecting active communities - before taking the lock races with deletion. Otherwise the PR MUST document a - reviewed procedure that keeps the routing fences closed. A script MUST NOT - disable triggers or set `session_replication_role`. + before taking the lock races with deletion. +- A repair script that inserts channel-bearing `events` rows, or moves an + event's `created_at` backward, is outside the replica-floor proof: an + operator session does not set `buzz.created_at_floor`, so the commit-time + floor guard lets it through (`migrations/0021`). Its PR MUST document a + reviewed procedure that keeps replica routing closed from before the first + batch until its WAL has replayed on every replica. +- A script MUST NOT disable triggers or set `session_replication_role`. - The normal write path that maintains the new invariant MUST be fully rolled out, with old writers drained, before the backfill starts. The backfill spec explains why and what breaks otherwise. @@ -257,11 +258,11 @@ apply: | `crates/buzz-push-gateway/migrations/` **and** the relay copies | The six `push_gateway_*` tables have two homes. The gateway applies its own migration history to its own database, through a pre-install/pre-upgrade Helm Job ([push-gateway-deployment.md](push-gateway-deployment.md)). The relay schema also defines them: `migrations/` (`0015`, `0043`, `0045`, `0060`), `schema/tables/public/push_gateway_*.sql`, and `schema/seeds/push_gateway_*.sql`. Until the duplication is removed, a gateway schema change goes in both, as relay `0060` mirrors gateway `0006`. Every rule here applies to both. The target is for the gateway to own its tables alone, in its own migrations and declarative schema; do not add new gateway tables to the relay schema. | A fresh desired-state database and a migrated database MUST converge on the -same live catalog. The layout test -(`desired_state_schema_layout_is_complete_and_schemabot_shaped`) checks file -structure only. Parity tests cover selected surfaces, and the full drift gate -does not exist yet. For tables and indexes you touch, add or extend a parity -assertion that compares the live catalog from both bootstrap paths, following +same live catalog. The desired-state layout test in +`crates/buzz-db/src/runtime/migration.rs` checks file structure only. Parity +tests cover selected surfaces, and the full drift gate does not exist yet. For +tables and indexes you touch, add or extend a parity assertion that compares +the live catalog from both bootstrap paths, following `admin_schema_parity_between_desired_state_and_migrations`. A string match against `schema.sql` does not count. @@ -321,23 +322,30 @@ historical data needs no backfill. pending migration at one boot, before the new code serves and before any backfill or repair script runs. An operator who upgrades across several releases therefore applies expand and contract together. How auto-migration -should order such an upgrade is unresolved. Until it is, a contract migration -MUST fail closed when the step it depends on has not run: - -- `VALIDATE CONSTRAINT` and `SET NOT NULL` already fail on unrepaired rows. -- A contract that drops a column or table, or otherwise loses data, MUST first - check that the backfill or repair has completed, and `RAISE` from a - top-level `DO` block when unrepaired rows remain. -- The PR MUST add a Postgres-gated test that applies the expand and contract - migrations in one run against unrepaired rows and asserts the migration - errors without dropping data. - -A failed contract stops the relay from starting. That is an outage, not data -loss, but recovery is not clean: the expand migrations before the contract -have already committed, so an old pod that restarts hits `VersionMissing`. -Because a repair needs the new writer fully rolled out (Rule 3), the release -notes MUST name the release to stop at (the one that ships the writer) and the -repair to run there before upgrading past the contract release. +should order such an upgrade is unresolved. Until it is, a contract depends on +one or both of these, and each is handled differently: + +- **Data (a backfill or repair has run).** The contract migration MUST fail + closed when it has not. `VALIDATE CONSTRAINT` and `SET NOT NULL` already + fail on unrepaired rows. A contract that drops a column or table, or + otherwise loses data, MUST first check that the backfill or repair has + completed, and `RAISE` from a top-level `DO` block when unrepaired rows + remain. The PR MUST add a Postgres-gated test that applies the expand and + contract migrations in one run against unrepaired rows and asserts the + migration errors without dropping data. +- **Writers (a release that changes how the app writes is fully deployed).** + Dropping a trigger, function, or foreign key whose work moved into the app, + and adding a constraint that old writers could violate, depend on a release, + not on rows. No row check can prove it: the data may already be valid, or + there may be nothing to backfill, while old writers still serve. + Auto-migration does not protect this case. + +For either kind, the release notes MUST name the release to stop at (the one +that ships the writer) and any repair to run there, before upgrading past the +contract release. A failed data check stops the relay from starting. That is +an outage, not data loss, but recovery is not clean: the expand migrations +before the contract have already committed, so an old pod that restarts hits +`VersionMissing`. #### What breaks the old relay @@ -349,13 +357,15 @@ repair to run there before upgrading past the contract release. | Add `CHECK`, `UNIQUE`, or a narrower index | Old writes violate it, or an old `ON CONFLICT` target no longer matches an index | Deploy writers that satisfy it, backfill, then add it. Add a `CHECK` as `NOT VALID`, then `VALIDATE CONSTRAINT` in a later step. Keep every index an old `ON CONFLICT` names until no deployed code uses it | | Add an enum value that new code writes | Old code fails to decode the unknown variant | Deploy readers that accept the value, then deploy writers that produce it | | Drop a column, table, index, trigger, function, or FK | Old code reads it or relies on its side effect | Contract step only, after the release that stops relying on it is fully deployed | -| Add a table with a `community_id` column | The old relay's exact-set deletion catalog check (`EXPECTED_SCOPED_TABLES`) sees an unknown table, so community deletion fails closed until the new release deploys. Serving is unaffected. | Expected and safe. Keep the gap between expand and deploy short, and do not run community deletion during it | +| Add a table with a `community_id` column | The old relay's exact-set deletion catalog check (`EXPECTED_SCOPED_TABLES`) sees an unknown table, so community deletion fails closed until the new release deploys. Serving is unaffected, apart from the auto-migration restart case in Rule 5 | Allowed, after Rule 1's catalog change for a tenant-owned table. Keep the gap between expand and deploy short, and do not run community deletion during it | | `SELECT *` or `RETURNING *` decoded by position | Column order differs between desired-state and migrated databases and changes when columns are added | Select named columns | #### Adding `NOT NULL` to a populated table `ALTER TABLE ... SET NOT NULL` scans the whole table under `ACCESS EXCLUSIVE`. -Instead: +Instead, ship three migration files, each its own step. sqlx runs each file in +a transaction, which `SET LOCAL` needs; pasted into autocommit `psql`, +`SET LOCAL` only warns and has no effect. ```sql -- contract migration A @@ -384,7 +394,8 @@ parity assertion on the table for that window. check and cascade, while the database object is still in place. The two paths MUST be idempotent with each other, so the duplicate work is harmless. 2. After that release is fully deployed, drop the database object in a - separate migration. Remove it from `schema/` in the same PR. + separate migration. Remove it from `schema/` in the same PR. The release + notes name the step-1 release as a required stop (Rule 5). 3. Removing a guard requires a falsifiable test showing that the application path enforces what the guard enforced. @@ -403,18 +414,17 @@ parity assertion on the table for that window. `ON ONLY` parent index. - The declarative schema source declares indexes with plain `CREATE INDEX`. The tool that applies it out of band decides how to build them online, so - `schema/` files MUST NOT add `CONCURRENTLY`. SchemaBot refuses it, and - `desired_state_schema_layout_is_complete_and_schemabot_shaped` enforces - plain `CREATE INDEX`. + `schema/` files MUST NOT add `CONCURRENTLY`. The desired-state layout test + rejects it. - A change MUST NOT rewrite or fully scan a large table under a write-blocking lock. That includes column type changes, volatile defaults, `SET NOT NULL` without a validated check, and changes to the generated `events.search_tsv` column. Use the expand/contract forms above. -- Write changes that SchemaBot can apply. Its native path admits - `CREATE TABLE`, `CREATE INDEX`, and bounded `ALTER TABLE` on tables up to its - configured size ceiling (1 GiB by default), with short lock and statement - budgets. A multi-statement plan is not atomic. A change outside that envelope - needs a separately reviewed operational procedure, and the PR MUST say so. +- Write changes that out-of-band apply can make online: `CREATE TABLE`, + `CREATE INDEX`, and bounded `ALTER TABLE` on tables below a size ceiling, + with short lock and statement budgets. Do not rely on a multi-statement + change applying atomically. A change outside that envelope needs a + separately reviewed operational procedure, and the PR MUST say so. ### 7. Old and new relays coexist during rollout @@ -435,8 +445,8 @@ description: - [ ] That no migration mutates existing rows, and the backfill ID (or, until the framework ships, the `scripts/maintenance/` script and how it takes the community deletion lock) for any repair. -- [ ] For a contract migration, how it fails closed when the step it depends - on has not run. +- [ ] For a contract migration, how it fails closed when its backfill has not + run, and which writer release the release notes name as a required stop. - [ ] That every location in Rule 4 is updated, with a catalog-level parity test for each changed table and index. - [ ] That the currently deployed relay works against the new schema, and how @@ -456,5 +466,3 @@ description: - [crates/buzz-db/TESTING.md](../crates/buzz-db/TESTING.md): desired-state and migration-mode tests. - `schema/schema.sql` header: the desired-state layout. -- [SchemaBot PostgreSQL support](https://github.com/block/schemabot/blob/main/docs/postgresql.md): - what SchemaBot can plan and apply. diff --git a/migrations/AGENTS.md b/migrations/AGENTS.md index 7c572455956..8735eb6996c 100644 --- a/migrations/AGENTS.md +++ b/migrations/AGENTS.md @@ -17,9 +17,10 @@ Before changing anything in this directory, read - Never edit a merged migration. Fix forward. - Adding a `community_id` table has its own checklist in that document. A tenant-owned one is blocked until deletion's catalog checks accept - application-fenced tables (BUZZ-314/BUZZ-315). + application-fenced tables. - Keep every migration safe for the relay that is currently deployed, whether migrations run at startup or out of band. Ship in this order: new schema, then new code, then backfill, then constraints and - cleanup. A contract migration fails closed if its backfill has not run. + cleanup. A contract migration fails closed if its backfill has not run, and + its release notes name the writer release that upgrades must not skip. - Bound every lock (`SET LOCAL lock_timeout`). From 00ab4a48151be1e34f2aa922d8e3911a34adb9b4 Mon Sep 17 00:00:00 2001 From: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Date: Fri, 16 Oct 2026 17:10:29 -0400 Subject: [PATCH 10/12] docs(database): name guard_transaction as the admission point and every floor-bypassing event write Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> --- docs/database.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/docs/database.md b/docs/database.md index 6421d009457..6f17f18170c 100644 --- a/docs/database.md +++ b/docs/database.md @@ -80,12 +80,12 @@ retirement sequence below. fence trigger, including by calling the existing helper `attach_community_write_fence`. Writers of a new tenant-owned table MUST be fenced in the application instead, by community admission. Serving writers, -including fleet sweeps (one community at a time), open their transaction with -`begin_community_write_transaction` or one of its variants -(`crates/buzz-db/src/runtime/mod.rs`). It returns an `AdmittedTx` -(`crates/buzz-db/src/runtime/admitted_tx.rs`), built by +including fleet sweeps (one community at a time), are admitted by `DeletionStore::guard_transaction`, which takes the shared community deletion -lock and refuses a community that is not `active`. Helpers that take +lock and refuses a community that is not `active`. Most open their transaction +with `begin_community_write_transaction` or one of its variants +(`crates/buzz-db/src/runtime/mod.rs`), which returns an `AdmittedTx` +(`crates/buzz-db/src/runtime/admitted_tx.rs`). Helpers that take `&mut AdmittedTx` reject a raw transaction at compile time, but crate code can still bypass the type, so the fence triggers remain the authoritative backstop (ARCHITECTURE.md § *Supported Writer Contract*). A table without a trigger has @@ -225,8 +225,9 @@ write-blocking lock. Scans that do not block writes, such as community has `deletion_state = 'active'` and `deleted_at IS NULL`, as `DeletionStore::guard_transaction` does. Selecting active communities before taking the lock races with deletion. -- A repair script that inserts channel-bearing `events` rows, or moves an - event's `created_at` backward, is outside the replica-floor proof: an +- A repair script that writes a channel-bearing `events` row older than the + floor (inserting one, moving `created_at` backward, or setting `channel_id` + on an old row) is outside the replica-floor proof: an operator session does not set `buzz.created_at_floor`, so the commit-time floor guard lets it through (`migrations/0021`). Its PR MUST document a reviewed procedure that keeps replica routing closed from before the first From 235558de3db6b989e5a7fbac6abe86d9ec8da7b5 Mon Sep 17 00:00:00 2001 From: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Date: Tue, 20 Oct 2026 15:00:33 -0400 Subject: [PATCH 11/12] docs(database): describe the fence prerequisite against current deletion checks Startup and deletion validation no longer require a fence on any table, and every serving writer already goes through admission. The remaining blocker for a trigger-less tenant table is the desired-state fence loop. Also name the leased-completion admission exception and fix mid-sentence wraps. Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> --- ARCHITECTURE.md | 7 ++-- docs/database.md | 76 ++++++++++++++++++++++++-------------------- migrations/AGENTS.md | 9 +++--- 3 files changed, 48 insertions(+), 44 deletions(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index fcfb965ee9a..7ff26b4ef74 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -526,10 +526,9 @@ shape is an explicit transition, not a flag day: the operational evidence path for rollout safety. The database end state (a declarative schema source, applied out of band at -scale; no triggers, functions, or foreign keys; -backfills outside migrations) and the -expand → deploy → backfill → contract rules every schema change follows are in -[docs/database.md](docs/database.md). +scale; no triggers, functions, or foreign keys; backfills outside migrations) +and the expand → deploy → backfill → contract rules every schema change follows +are in [docs/database.md](docs/database.md). **Does NOT:** cache queries, implement connection pooling logic (delegated to sqlx), or make network calls outside Postgres. diff --git a/docs/database.md b/docs/database.md index 6f17f18170c..78ad9486dd4 100644 --- a/docs/database.md +++ b/docs/database.md @@ -79,53 +79,59 @@ retirement sequence below. **No exception for the community write fence.** A new table MUST NOT get a fence trigger, including by calling the existing helper `attach_community_write_fence`. Writers of a new tenant-owned table MUST be -fenced in the application instead, by community admission. Serving writers, -including fleet sweeps (one community at a time), are admitted by +fenced in the application instead, by community admission. New serving work, +including fleet sweeps (one community at a time), is admitted by `DeletionStore::guard_transaction`, which takes the shared community deletion -lock and refuses a community that is not `active`. Most open their transaction -with `begin_community_write_transaction` or one of its variants +lock and refuses a community that is not `active`. Most writers open their +transaction with `begin_community_write_transaction` or one of its variants (`crates/buzz-db/src/runtime/mod.rs`), which returns an `AdmittedTx` -(`crates/buzz-db/src/runtime/admitted_tx.rs`). Helpers that take -`&mut AdmittedTx` reject a raw transaction at compile time, but crate code can -still bypass the type, so the fence triggers remain the authoritative backstop -(ARCHITECTURE.md § *Supported Writer Contract*). A table without a trigger has -no database backstop. The remaining fence-retirement work makes every writer of -every fenced table go through admission, then drops the fence triggers and -functions. That work MUST also update ARCHITECTURE.md § *Dual enforcement*. - -**Today this blocks new tenant-owned tables.** Community deletion's catalog -checks still assume that every tenant-scoped table carries a trigger fence: - -- An unlisted `community_id` table, fenced or not, fails the exact-set check - against `EXPECTED_SCOPED_TABLES` - (`community deletion catalog drift (missing=…, unknown=…)`). Listing it - without a trigger fails the write-fence drift check. Either way, deletion - fails closed for every community, not only the new table's. -- Relay startup refuses a binary that lists the table in - `EXPECTED_SCOPED_TABLES` when the database has no live fence on it - (`community serving write fences missing`). +(`crates/buzz-db/src/runtime/admitted_tx.rs`). One exception is deliberate: +a write that finishes under an already-admitted serving lease +(`insert_event_with_serving_write_guard`, through +`AdmittedTx::admit_with_serving_lease`) is checked by +`guard_transaction_with_serving_lease`, which lets that exact unexpired lease +complete while the community is `quiescing`. Keep it. + +Helpers that take `&mut AdmittedTx` reject a raw transaction at compile time, +but crate code can still bypass the type, so the fence triggers remain the +database backstop until they are dropped (ARCHITECTURE.md § *Supported Writer +Contract*). A table without a trigger has no database backstop. Every serving +writer of a fenced table already goes through admission. The remaining +fence-retirement work drops the fence triggers and functions, and MUST also +update ARCHITECTURE.md § *Dual enforcement*. + +**Today this blocks new tenant-owned tables.** Startup and deletion validation +no longer require a fence on any table; they pass with every universal fence +object dropped (`migration_schema_fence_objects_are_optional_for_serving_and_deletion` +in `crates/buzz-db/src/store/deletion.rs`). Two things still matter: + +- Deletion's catalog check is an exact set. A `community_id` table missing from + `EXPECTED_SCOPED_TABLES`, fenced or not, fails it + (`community deletion catalog drift (missing=…, unknown=…)`), and deletion + fails closed for every community, not only the new table's. Listing the + table is enough to pass. - Desired-state bootstraps run `schema/community_write_fence.sql`, which - attaches a fence trigger to every `community_id` table automatically. + attaches a fence trigger to every `community_id` table not excluded by + `community_write_fence_excluded_table`. A fresh desired-state database would + put a trigger on the new table and a migrated one would not, so the two + schema paths would diverge. -So before a feature can add a tenant-owned `community_id` table, these checks -must recognise application-fenced tables, and the desired-state loop must stop -attaching triggers to them. That change belongs to the fence-retirement work. -Land it before the feature, or as the feature's first PR. Do not work around -it with a trigger, or with an entry in `community_write_fence_excluded_table` -for a table the community owns. +So before a feature can add a tenant-owned `community_id` table, the +desired-state bootstrap must stop attaching fence triggers. That change belongs +to the fence-retirement work. Land it before the feature, or as the feature's +first PR. Do not work around it with a trigger, or with an entry in +`community_write_fence_excluded_table` for a table the community owns. #### Checklist: adding a tenant-owned `community_id` table -After the catalog change above has landed, all in the same PR: +After the desired-state change above has landed, all in the same PR: - [ ] Every writer of the table goes through community admission (`begin_community_write_transaction` or another constructor that returns `AdmittedTx`). Nothing writes the table outside it. - [ ] No fence trigger: no `attach_community_write_fence` call in the migration and no `schema/triggers/` file. -- [ ] The table in `EXPECTED_SCOPED_TABLES` (`crates/buzz-db/src/store/deletion.rs`), - plus any registration the catalog change introduces for - application-fenced tables. +- [ ] The table in `EXPECTED_SCOPED_TABLES` (`crates/buzz-db/src/store/deletion.rs`). - [ ] The table in `PURGE_SCOPED_TABLES` in the same file, ordered child before parent. A table missing here keeps its rows after the purge, and deletion verification fails closed. @@ -358,7 +364,7 @@ before the contract have already committed, so an old pod that restarts hits | Add `CHECK`, `UNIQUE`, or a narrower index | Old writes violate it, or an old `ON CONFLICT` target no longer matches an index | Deploy writers that satisfy it, backfill, then add it. Add a `CHECK` as `NOT VALID`, then `VALIDATE CONSTRAINT` in a later step. Keep every index an old `ON CONFLICT` names until no deployed code uses it | | Add an enum value that new code writes | Old code fails to decode the unknown variant | Deploy readers that accept the value, then deploy writers that produce it | | Drop a column, table, index, trigger, function, or FK | Old code reads it or relies on its side effect | Contract step only, after the release that stops relying on it is fully deployed | -| Add a table with a `community_id` column | The old relay's exact-set deletion catalog check (`EXPECTED_SCOPED_TABLES`) sees an unknown table, so community deletion fails closed until the new release deploys. Serving is unaffected, apart from the auto-migration restart case in Rule 5 | Allowed, after Rule 1's catalog change for a tenant-owned table. Keep the gap between expand and deploy short, and do not run community deletion during it | +| Add a table with a `community_id` column | The old relay's exact-set deletion catalog check (`EXPECTED_SCOPED_TABLES`) sees an unknown table, so community deletion fails closed until the new release deploys. Serving is unaffected, apart from the auto-migration restart case in Rule 5 | Allowed. For a tenant-owned table, only after the desired-state change in Rule 1. Keep the gap between expand and deploy short, and do not run community deletion during it | | `SELECT *` or `RETURNING *` decoded by position | Column order differs between desired-state and migrated databases and changes when columns are added | Select named columns | #### Adding `NOT NULL` to a populated table diff --git a/migrations/AGENTS.md b/migrations/AGENTS.md index 8735eb6996c..f23d3d0543a 100644 --- a/migrations/AGENTS.md +++ b/migrations/AGENTS.md @@ -16,11 +16,10 @@ Before changing anything in this directory, read Do not add new gateway tables here. - Never edit a merged migration. Fix forward. - Adding a `community_id` table has its own checklist in that document. A - tenant-owned one is blocked until deletion's catalog checks accept - application-fenced tables. + tenant-owned one is blocked until desired-state bootstraps stop attaching + fence triggers to every `community_id` table. - Keep every migration safe for the relay that is currently deployed, whether - migrations run at startup or out of band. Ship in - this order: new schema, then new code, then backfill, then constraints and - cleanup. A contract migration fails closed if its backfill has not run, and + migrations run at startup or out of band. Ship in this order: new schema, + then new code, then backfill, then constraints and cleanup. A contract migration fails closed if its backfill has not run, and its release notes name the writer release that upgrades must not skip. - Bound every lock (`SET LOCAL lock_timeout`). From 09fc3778042a7e027cdbcb32456a5fba44f2cf8e Mon Sep 17 00:00:00 2001 From: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> Date: Tue, 20 Oct 2026 15:02:44 -0400 Subject: [PATCH 12/12] docs(migrations): rewrap the ship-order bullet Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz> --- migrations/AGENTS.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/migrations/AGENTS.md b/migrations/AGENTS.md index f23d3d0543a..82d72c94a92 100644 --- a/migrations/AGENTS.md +++ b/migrations/AGENTS.md @@ -20,6 +20,7 @@ Before changing anything in this directory, read fence triggers to every `community_id` table. - Keep every migration safe for the relay that is currently deployed, whether migrations run at startup or out of band. Ship in this order: new schema, - then new code, then backfill, then constraints and cleanup. A contract migration fails closed if its backfill has not run, and - its release notes name the writer release that upgrades must not skip. + then new code, then backfill, then constraints and cleanup. A contract + migration fails closed if its backfill has not run, and its release notes + name the writer release that upgrades must not skip. - Bound every lock (`SET LOCAL lock_timeout`).