Skip to content

buzz-db: drop the events_created_at_floor trigger (BUZZ-316 phase 2) - #8326

Draft
TheSentinel454 wants to merge 10 commits into
elrond/floor-guard-appfrom
elrond/drop-floor-guard
Draft

TheSentinel454 wants to merge 10 commits into
elrond/floor-guard-appfrom
elrond/drop-floor-guard

Conversation

@TheSentinel454

@TheSentinel454 TheSentinel454 commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Stack (merge bottom-up): main ← #8325 ← #8326 (0087) ← #8323 (0088) ← #8211 (0089). Migrations are pre-numbered in stack order, so each PR's migration.rs count assumes the ones below it.

Summary

Phase 2 of BUZZ-316. This is a destructive contract change, so it merges last. It is stacked on #8325 (phase 1), which adds the insert-time application floor check.

After phase 1, the app check at every channel-bearing INSERT INTO events is the only created_at floor guard. This PR removes the database backstop and everything that existed only to verify it:

  • Migration 0087 drops the deferred constraint trigger events_created_at_floor on events. That also removes its partition clones. It then loops over pg_partition_tree('events') to drop any child copy that was attached separately, and drops the events_created_at_floor_guard() function. Every drop uses IF EXISTS and none uses CASCADE. The migration sets lock_timeout to 5s. Migration 0021 stays byte-for-byte unchanged.
  • Desired state: removes the function file, its schema.sql include, the trigger in schema/triggers/events.sql, and the eight child-copy drops in scripts/reconcile-schema-after-pgschema.sql. Both boot paths stay catalog-equivalent, and admin_schema_parity_between_desired_state_and_migrations passes.
  • Runtime: deletes verify_floor_guard_catalog, verify_floor_guard_behavior, and the boot wrapper that gated the probe on them. Also deletes the writer-pool buzz.created_at_floor GUC arming and its CreatedAtFloor connection-setup metric step. The setup-series ceiling drops from 71 to 54. spawn_fence_probe becomes synchronous and returns bool, so the relay's startup error branch goes away. CREATED_AT_FLOOR_SECS (960s) stays because the app check and the fence wall both use it.
  • Kept: every other fail-closed probe check. That covers writer-pool isolation and timeouts, partition topology, the activity scan, the heartbeat/epoch checks, and closing on probe error. No other trigger, function, FK, or tenant fence is touched.

Deploy gate (required)

  1. Phase 1 must be running on every writer pod and job, with no rollback expected. Pods that are already running never recheck the guard, so a rolling deploy on its own is not enough.
  2. Stop out-of-band writers: direct SQL, COPY, restores, seed scripts, and anything that prepares transactions.
  3. Hold the replica breaker closed from before the drop until replay catches up and the new scheme is verified.
  4. Run the migration off-peak. DROP TRIGGER takes ACCESS EXCLUSIVE on events and every partition, so expect events reads and writes to stall for up to the 5s lock_timeout. A lock timeout aborts the migration transaction and is safe to retry. Under auto-migrate, the pod fails boot, restarts, and retries.

Mixed-version behavior:

  • A phase-1 pod that restarts after the drop, default config (migrate.autoMigrate: true in the chart): sqlx finds applied migration 87 missing from its embedded set and returns VersionMissing(87). Boot is fatal and the pod crash-loops. A binary rollback is therefore unsafe once 0087 has run.
  • The same pod with auto-migrate off: verify_floor_guard_catalog fails, the fence starts closed, and the relay degrades the ReplicaFenceProbe startup step. It keeps serving, with cursor reads on the writer.
  • This binary against a database that still has the trigger: boots and runs. It neither requires nor arms the guard.

Tests

crates/buzz-db/src/runtime/tests/floor_retirement.rs adds four PostgreSQL tests. CI's nextest filter picks them up through postgres_tests:::

  • the migration drops the parent trigger, its partition clones, and the function, and leaves every other trigger and FK unchanged;
  • the migration removes a separately attached child copy and can safely run twice;
  • the desired-state catalog has no floor objects;
  • the production writer pool no longer arms the GUC, and the probe runs against the retired schema.

All four fail when the change is reverted (0/4 pass) and pass with it (4/4). Four tests that only exercised the trigger or the probes are deleted. The outbox durable-failure regression is kept: it now injects an isolated BEFORE INSERT failure instead of arming the floor. The phase-1 floor fixture no longer disables a trigger that no longer exists. It still asserts that no enabled floor trigger can mask the app check.

With the trigger gone, the floor source policy (every_production_channel_event_insert_checks_the_floor_and_never_rewrites_it) is the only guard against a writer that skips the app check or rewrites created_at/channel_id on an existing row. Phase 1 already makes it scan every crate's src. This PR closes two matcher gaps the trigger used to cover:

  • UPDATE events SET(created_at, …) written with no space before the parenthesis;
  • INSERT INTO events … ON CONFLICT … DO UPDATE SET channel_id = …, which names no table but can re-home an existing row below the floor. Each DO UPDATE is bound to the INSERT it belongs to, so a watermark upsert that sets created_at next to a guarded events insert (the replaceable-event writer's shape) is not flagged.

Both gaps have negative fixtures, plus controls for a non-floor-column upsert, DO NOTHING, and the watermark shape.

The policy also now fails closed on SQL the per-function literal scan could not attribute to a writer (event_floor_policy_fails_closed_on_sql_it_cannot_attribute):

  • include_str! is expanded from the file it loads, and concat! becomes one literal, with an {opaque} placeholder for each non-literal argument. Every delimiter ((), [], {}) and path prefix (::core::concat!) is recognized.
  • Every SQL literal is checked on its own as well as per function. Anything the policy cannot attribute is a violation: a write target or SET column that is a placeholder (concat! or format!); a SET list of floor columns whose UPDATE is in another literal; an UPDATE events or DO UPDATE whose SET list continues in another literal or argument (empty, or an item with no =, as in format!("UPDATE events SET {}", …) or QueryBuilder::new("UPDATE events SET ").push(…)); an INSERT INTO with no table after it in the literal, or an events INSERT with a placeholder tail; events written by MERGE INTO/COPY; and event-write SQL in a const, static, or macro body outside any function. A placeholder in a WHERE or a value is allowed, so the retention deletes (concat! around a predicate macro) and the tenant-severing UPDATE {table} SET community_id = NULL pass.
  • An include_str! whose path is not one readable literal and any sqlx query_file*! macro are violations. Nothing uses query_file.
  • PREPARE TRANSACTION in any production literal, including a dangling PREPARE, is a violation (see below).
  • Function slices are cut to their own item, so a trailing const is judged at item level instead of being credited to the function before it.

Prepared transactions. The fence used to bound a prepared (2PC) transaction by pg_prepared_xacts.prepared. That only held because the deferred trigger checked its rows at PREPARE; the insert-time check runs earlier, at a time pg_prepared_xacts does not expose. So the probe no longer folds prepared into the wall: any prepared transaction fails the sample closed (ProbeError::PreparedTransactions). No supported writer prepares transactions (now source-policy enforced) and max_prepared_transactions defaults to 0, so in supported deployments this never fires. cluster_global_sample_writer_fails_closed_on_prepared_transaction asserts 2PC is unavailable at the default setting, and asserts the fail-closed error when the server allows it. CI runs only the default branch; the 2PC-enabled branch and a check-removed falsification run are pinned from the Blox gate. The prepared count is cluster-wide, like the activity scan.

What got simpler

The change is net −600 lines. Gone are one function, one parent constraint trigger plus its partition clones, two startup probes and their scratch helper, one boot wrapper, the per-connection GUC setup, one metric step, and an async startup branch that could no longer be reached.

Notes

Elrond and others added 10 commits November 10, 2026 08:45
Phase 2 of BUZZ-316. The insert-time application floor check (phase 1)
is now the only created_at floor guard, so drop the deferred constraint
trigger, its function, the writer-pool buzz.created_at_floor GUC, and the
startup catalog/behavior probes that required them.

Migration 0062 drops the trigger on events and every partition plus the
function, all IF EXISTS. Desired-state schema, pgschema reconciliation
and migration asserts drop the same objects. spawn_fence_probe becomes
synchronous; the ordered heartbeat probe still fails closed.

DESTRUCTIVE: deploy only after phase 1 runs on every writer, with no
rollback expected, out-of-band writers stopped, and the replica breaker
held closed through the drop until replay catches up.

Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Address review of the phase-2 floor retirement:
- the workspace-wide floor source policy now comes from phase 1's
  shared crates/*/src walk (which pins buzz-relay coverage); this
  commit only drops the GUC from its constant-parity message
- drop the legacy-0021 half of the writer/probe test, which could not
  fail, and rename the desired-state test to what it checks
- fix stale GUC prose and document the ACCESS EXCLUSIVE lock in 0062

Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
With the floor trigger retired, the source policy is the only guard
against rewriting created_at/channel_id on existing event rows. Close
two matcher gaps found in review:
- `UPDATE events SET(created_at, ...)` with no space before the
  parenthesis slipped past the " set " split
- `INSERT INTO events ... ON CONFLICT ... DO UPDATE SET channel_id = ...`
  names no table, so it passed even though it can re-home an existing
  row below the floor; bind each DO UPDATE to the INSERT it belongs to
  and treat it as an events update when that INSERT targets events

Share the SET-list check between both paths, and pin each gap with a
negative fixture plus controls: a non-floor-column events upsert, DO
NOTHING, and a guarded events insert next to a watermark upsert that
sets created_at (the shape of the replaceable-event writer).

Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
With the trigger gone the source policy is the only guard. Expose SQL the
per-function scan could not see: include_str! is expanded from the file
it loads and concat! literals are joined, so both reach the existing
insert/update checks. Fail closed on what stays opaque: a non-literal
include path, a concat! that spells SQL around a non-literal argument,
event-write SQL in a const, static or macro body outside any function,
and PREPARE TRANSACTION in any production writer. Cut each function
slice to its own item so trailing consts are judged at item level.

The replica fence no longer folds pg_prepared_xacts.prepared into the
wall: only the retired trigger checked prepared rows at PREPARE, and the
insert-time guard's clock is not exposed there. A prepared transaction
now fails the sample closed (PreparedTransactions).

Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
…annot read

Review of 8752ce1 found the opaque-concat rule red on production: the
two retention deletes concat a predicate macro into their WHERE, and any
"update " around a non-literal argument was a violation.

Replace that heuristic with one detector. concat! now becomes a single
literal with an {opaque} placeholder per non-literal argument, and every
SQL literal is checked on its own: a placeholder (from concat! or
format!) in a write target or SET column, a floor-column SET list whose
UPDATE is in another literal, an INSERT INTO with no table after it or
an events INSERT with a placeholder tail, events written by MERGE INTO
or COPY, and a dangling PREPARE are violations. A placeholder in a WHERE
or a value is allowed, so the retention deletes and the tenant-severing
UPDATE {table} SET community_id = NULL pass.

Also recognize concat!/include_str! with [] and {} delimiters, and ban
sqlx query_file*! macros, which load SQL the scan cannot see.

Document that the prepared-transaction count is cluster-wide on purpose,
like the activity scan.

Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
The per-literal UPDATE check only saw the SET list inside the literal
that holds SET, so format!("UPDATE events SET {}", ..), a split
[..].concat(), and QueryBuilder pushes after SET slipped through. A SET
list that is empty or holds an item with no '=' is now unattributable
when its target is events, a placeholder, or DO UPDATE. Prose that
merely contains update/set is still ignored.

Signed-off-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
The startup floor/roster probes are deleted with the triggers, so their
exemption lists and fixtures only kept dead names alive in the policy
test.

Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Luke Tornquist <tornquist@squareup.com>
Main grew a nip_fi_issuers argument on community bootstrap, ProbeTimings on
sample_writer, and a sync spawn_fence_probe; the rebased commits still used
the old shapes and the new PreparedTransactions variant had no span kind.

Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Luke Tornquist <tornquist@squareup.com>
Main's put_members_on upsert returns columns after its SET list, which the
policy read as an unreadable assignment and flagged. Also drop the writer
count floor to seven now that the startup probe is gone.

Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Luke Tornquist <tornquist@squareup.com>
The prebuilt case holds ROW EXCLUSIVE on events while it migrates. The
trigger retirements after 0086 need ACCESS EXCLUSIVE on events, so they
hit lock_timeout under that lock. Migrate through 0086 under the writer,
then apply the rest once it rolls back.

Co-authored-by: Elrond <28d6302a099e5225b02c4155ac4236e4912603df2ab08dbfc2f4fef08ce598c8@buzz.block.builderlab.xyz>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Luke Tornquist <tornquist@squareup.com>
@TheSentinel454
TheSentinel454 force-pushed the elrond/drop-floor-guard branch from 34f6527 to 138da61 Compare November 10, 2026 16:49

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant