Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 19 additions & 14 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -506,25 +506,30 @@ shape is an explicit transition, not a flag day:
being absent; inventories still record live fences for older executors.
Desired-state bootstraps install only explicitly declared legacy fences.
New tenant-owned tables rely on application admission, without a fence trigger.
- **Replica floor (transition foundation):** runtime now provides a shared
replica-floor lock helper for compliant channel-event writers, and the writer
probe handshake takes the exclusive counterpart before sampling `S`, scanning
activity, and committing the heartbeat token last. `probe_once` remains the
sole token/fence-wall publication path; there is no separately persisted active
floor cutoff. Commit-time trigger+GUC enforcement remains authoritative for all
paths.
- **Dual enforcement (current):** application-owned lock+precheck paths run in
front of the existing trigger/function enforcement; commit-time trigger checks
remain authoritative during this phase. Removing those backstops is a separate
gated migration, after every pod tolerates their absence and no rollback below
that version is expected.
- **Replica floor:** supported channel-event insertions check `created_at`
against `clock_timestamp()` on their admitted transaction using the shared
960-second floor. The ordered heartbeat handshake and activity scan retain
the same fence-wall calculation. Migration 0087 retires only the deferred
floor trigger/function; writer connections no longer arm its GUC and probe
startup no longer requires those objects.
- **Floor retirement deploy gate:** phase 1 must be deployed to every writer
pod/job, with no rollback expected, before applying the destructive contract.
Drain old connections/in-flight transactions, stop out-of-band writers, and
hold the replica breaker closed through the drop until replay catches up.
Already-running old pods do not reverify the trigger on each heartbeat.
- **Dual enforcement (community fence transition):** application-owned
lock+precheck paths run in front of the existing community trigger/function
enforcement. Removing those backstops is a separate gated migration, after
every pod tolerates their absence and no rollback below that version is
expected.
- **Role separation:** relay/runtime code owns admission and lock protocols;
operator maintenance/backfill workflows must either use those protocols or run
under explicit reviewed procedures that keep routing fences closed.
- **Unsupported shape:** direct owner SQL that bypasses supported transaction and
lock protocols is not a supported write path.
- **Fleet gates and reconciliation:** startup/fence probes verify guard catalog +
behavior, pgschema bootstraps run reconciliation, and lock/pool metrics remain
- **Fleet gates and reconciliation:** retained guards keep their catalog and
behavior checks; heartbeat/epoch and activity checks still fail closed.
pgschema bootstraps run reconciliation, and lock/pool metrics remain
the operational evidence path for rollout safety.

The database end state (a declarative schema source, applied out of band at
Expand Down
6 changes: 3 additions & 3 deletions crates/buzz-db/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -70,15 +70,15 @@ pub const DB_POOL_ACQUIRE_RAW_SERIES_PER_POD: usize =
runtime::observability::POOL_ACQUIRE_RAW_SERIES_PER_POD;

/// Valid database connection role/step pairs with a start counter.
pub const DB_CONNECTION_STARTED_STEPS: [(DbPoolRole, DbConnectionStep); 4] =
pub const DB_CONNECTION_STARTED_STEPS: [(DbPoolRole, DbConnectionStep); 3] =
runtime::CONNECTION_STARTED_STEPS;

/// Valid database connection role/step pairs with a duration histogram.
pub const DB_CONNECTION_DURATION_STEPS: [(DbPoolRole, DbConnectionStep); 4] =
pub const DB_CONNECTION_DURATION_STEPS: [(DbPoolRole, DbConnectionStep); 3] =
runtime::CONNECTION_DURATION_STEPS;

/// Valid database connection role/step/outcome terminal combinations.
pub const DB_CONNECTION_TERMINALS: [(DbPoolRole, DbConnectionStep, DbConnectionOutcome); 15] =
pub const DB_CONNECTION_TERMINALS: [(DbPoolRole, DbConnectionStep, DbConnectionOutcome); 12] =
runtime::CONNECTION_TERMINALS;

/// Raw Prometheus series ceiling per pod for connection-step telemetry.
Expand Down
39 changes: 9 additions & 30 deletions crates/buzz-db/src/runtime/connection_observability.rs
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,6 @@ pub enum DbConnectionStep {
WriterPool,
/// A physical connection has completed DNS/network/TLS/authentication.
PhysicalConnect,
/// Install the created-at replica-fence floor.
CreatedAtFloor,
/// Install lock, idle-transaction, and statement timeouts.
SessionTimeouts,
/// Verify READ COMMITTED transaction isolation.
Expand All @@ -29,10 +27,9 @@ pub enum DbConnectionStep {

impl DbConnectionStep {
/// Complete metric-label vocabulary.
pub const ALL: [Self; 6] = [
pub const ALL: [Self; 5] = [
Self::WriterPool,
Self::PhysicalConnect,
Self::CreatedAtFloor,
Self::SessionTimeouts,
Self::Isolation,
Self::Ready,
Expand All @@ -43,7 +40,6 @@ impl DbConnectionStep {
match self {
Self::WriterPool => "writer_pool",
Self::PhysicalConnect => "physical_connect",
Self::CreatedAtFloor => "created_at_floor",
Self::SessionTimeouts => "session_timeouts",
Self::Isolation => "isolation",
Self::Ready => "ready",
Expand Down Expand Up @@ -77,18 +73,17 @@ impl DbConnectionOutcome {
}

/// Valid role/step pairs with an explicit start counter.
pub const CONNECTION_STARTED_STEPS: [(DbPoolRole, DbConnectionStep); 4] = [
pub const CONNECTION_STARTED_STEPS: [(DbPoolRole, DbConnectionStep); 3] = [
(DbPoolRole::Writer, DbConnectionStep::WriterPool),
(DbPoolRole::Writer, DbConnectionStep::CreatedAtFloor),
(DbPoolRole::Writer, DbConnectionStep::SessionTimeouts),
(DbPoolRole::Writer, DbConnectionStep::Isolation),
];

/// Valid role/step pairs with a duration histogram.
pub const CONNECTION_DURATION_STEPS: [(DbPoolRole, DbConnectionStep); 4] = CONNECTION_STARTED_STEPS;
pub const CONNECTION_DURATION_STEPS: [(DbPoolRole, DbConnectionStep); 3] = CONNECTION_STARTED_STEPS;

/// Valid role/step/outcome terminal combinations.
pub const CONNECTION_TERMINALS: [(DbPoolRole, DbConnectionStep, DbConnectionOutcome); 15] = [
pub const CONNECTION_TERMINALS: [(DbPoolRole, DbConnectionStep, DbConnectionOutcome); 12] = [
(
DbPoolRole::Writer,
DbConnectionStep::WriterPool,
Expand All @@ -114,21 +109,6 @@ pub const CONNECTION_TERMINALS: [(DbPoolRole, DbConnectionStep, DbConnectionOutc
DbConnectionStep::PhysicalConnect,
DbConnectionOutcome::Succeeded,
),
(
DbPoolRole::Writer,
DbConnectionStep::CreatedAtFloor,
DbConnectionOutcome::Succeeded,
),
(
DbPoolRole::Writer,
DbConnectionStep::CreatedAtFloor,
DbConnectionOutcome::Failed,
),
(
DbPoolRole::Writer,
DbConnectionStep::CreatedAtFloor,
DbConnectionOutcome::Cancelled,
),
(
DbPoolRole::Writer,
DbConnectionStep::SessionTimeouts,
Expand Down Expand Up @@ -166,8 +146,8 @@ pub const CONNECTION_TERMINALS: [(DbPoolRole, DbConnectionStep, DbConnectionOutc
),
];

/// Four start counters + four 13-series histograms + fifteen terminal counters.
pub const CONNECTION_RAW_SERIES_PER_POD: usize = 4 + (4 * 13) + 15;
/// Three start counters + three 13-series histograms + twelve terminal counters.
pub const CONNECTION_RAW_SERIES_PER_POD: usize = 3 + (3 * 13) + 12;

pub(crate) struct DbConnectionStepAttempt {
pool_role: DbPoolRole,
Expand Down Expand Up @@ -282,13 +262,12 @@ mod tests {
[
"writer_pool",
"physical_connect",
"created_at_floor",
"session_timeouts",
"isolation",
"ready",
]
);
assert_eq!(CONNECTION_RAW_SERIES_PER_POD, 71);
assert_eq!(CONNECTION_RAW_SERIES_PER_POD, 54);
}

#[test]
Expand All @@ -297,7 +276,7 @@ mod tests {
let snapshotter = recorder.snapshotter();
let _guard = metrics::set_default_local_recorder(&recorder);
let attempt =
DbConnectionStepAttempt::start(DbPoolRole::Writer, DbConnectionStep::CreatedAtFloor);
DbConnectionStepAttempt::start(DbPoolRole::Writer, DbConnectionStep::SessionTimeouts);
drop(attempt);

let metrics = snapshotter.snapshot().into_vec();
Expand All @@ -309,7 +288,7 @@ mod tests {
.map(|label| (label.key(), label.value()))
.collect::<std::collections::BTreeMap<_, _>>();
assert_eq!(labels.get("pool_role"), Some(&"writer"));
assert_eq!(labels.get("step"), Some(&"created_at_floor"));
assert_eq!(labels.get("step"), Some(&"session_timeouts"));
assert!(!labels.contains_key("reason"));
assert!(!labels.contains_key("connection_ordinal"));
match value {
Expand Down
28 changes: 12 additions & 16 deletions crates/buzz-db/src/runtime/migration.rs
Original file line number Diff line number Diff line change
Expand Up @@ -58,14 +58,6 @@ async fn run_migrations_locked(conn: &mut PgConnection) -> Result<()> {
build_indexes_before_migration(conn, *version, doc, indexes).await?;
}
MIGRATOR.run(&mut *conn).await?;
// The replica-fence proof (see `replica_fence`) requires the commit-time
// `created_at` floor trigger from migration 0021 — correctly shaped — on
// the `events` parent and every partition. `CREATE TABLE .. PARTITION OF`
// clones parent triggers, but a partition attached with `ATTACH
// PARTITION` or created by an older code path would silently escape the
// guard, so migration fails closed if any is missing. (The fence probe
// re-runs this same check at startup on non-migrating relays.)
crate::replica_fence::verify_floor_guard_catalog(&mut *conn).await?;
Ok(())
}

Expand Down Expand Up @@ -902,7 +894,7 @@ mod postgres_tests {
let mut migrations: Vec<_> = MIGRATOR.iter().collect();
migrations.sort_by_key(|migration| migration.version);

assert_eq!(migrations.len(), 86);
assert_eq!(migrations.len(), 87);
assert!(migrations[83]
.sql
.as_str()
Expand All @@ -915,6 +907,11 @@ mod postgres_tests {
.sql
.as_str()
.contains("CREATE TABLE media_uploaders"));
// 0087 retires the deferred created_at floor trigger and its function.
assert_eq!(migrations[86].version, 87);
let retire_floor = strip_sql_comments(migrations[86].sql.as_str());
assert!(retire_floor.contains("DROP TRIGGER IF EXISTS events_created_at_floor ON events;"));
assert!(retire_floor.contains("DROP FUNCTION IF EXISTS events_created_at_floor_guard();"));
for (index, migration) in migrations.iter().enumerate() {
assert_eq!(migration.version, (index + 1) as i64);
}
Expand Down Expand Up @@ -1330,12 +1327,6 @@ mod postgres_tests {
.as_str()
.contains("CREATE TABLE join_policy_acceptances"));

// Replica-fence commit-time floor guard on channel-bearing events.
assert_eq!(migrations[20].version, 21);
assert!(migrations[20]
.sql
.as_str()
.contains("events_created_at_floor_guard"));
assert!(!migrations[0]
.sql
.as_str()
Expand Down Expand Up @@ -4133,10 +4124,15 @@ mod postgres_tests {
.await
.unwrap();
if prebuilt {
run_migrations(&pool)
// Stop at 0086: later trigger retirements take ACCESS
// EXCLUSIVE on events, which this held writer lock blocks.
run_migrations_through(&pool, 86)
.await
.expect("verified prebuild does not request an index-build lock");
writer.rollback().await.unwrap();
run_migrations(&pool)
.await
.expect("later migrations apply once the writer is gone");
} else {
let error = run_migrations(&pool).await.unwrap_err();
assert!(
Expand Down
Loading
Loading