Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions crates/buzz-db/src/error.rs
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,14 @@ pub enum DbError {
#[error("access denied: {0}")]
AccessDenied(String),

/// A permanent user-deletion restriction refuses this serving write.
#[error("identity is restricted for user deletion")]
UserDeletionRestricted,

/// Ownership changed while acquiring event-author locks; retry in a new transaction.
#[error("agent ownership changed; retry the write")]
EventAuthorOwnershipChanged,

/// Invitation admission or minting was denied by authoritative restrictions.
#[error("invitation principal is restricted")]
InviteRestricted,
Expand Down
9 changes: 9 additions & 0 deletions crates/buzz-db/src/runtime/admitted_tx.rs
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,15 @@ impl AdmittedTx {
&mut self.tx
}

pub(crate) async fn guard_event_author(&mut self, event: &nostr::Event) -> Result<()> {
crate::store::user_deletion_write_guard::guard_event_author(
&mut self.tx,
self.community,
event,
)
.await
}

/// Record that an event committed with this transaction belongs to
/// `channel_id`, so [`AdmittedTx::commit`] refreshes the channel's TTL.
pub(crate) fn record_channel_event(&mut self, channel_id: Option<Uuid>, kind: i32) {
Expand Down
100 changes: 96 additions & 4 deletions crates/buzz-db/src/store/archived_identities.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,14 @@
//! relay access, or repository permissions.
//! All pubkey and event ID values are lowercase hex strings.

use buzz_core::CommunityId;
use buzz_core::{CommunityId, StoredEvent};
use buzz_datastore_tracing::datastore_span;
use chrono::{DateTime, Utc};
use nostr::Event;
use sqlx::{PgPool, Row as _};

use crate::error::Result;
use crate::Db;
use crate::{AdmittedTx, Db};

/// A single archived identity record.
#[derive(Debug, Clone)]
Expand Down Expand Up @@ -70,6 +71,31 @@ pub async fn archive(
crate::observability::WriterOperation::EventWrite,
)
.await?;
let changed = archive_on(
&mut tx,
pubkey,
consent_path,
actor,
reason,
replaced_by,
request_event_id,
)
.await?;
tx.commit().await?;
Ok(changed)
}

#[allow(clippy::too_many_arguments)]
async fn archive_on(
tx: &mut AdmittedTx,
pubkey: &str,
consent_path: &str,
actor: &str,
reason: Option<&str>,
replaced_by: Option<&str>,
request_event_id: &str,
) -> Result<bool> {
let community_id = tx.community();
let result = sqlx::query(
"INSERT INTO archived_identities \
(community_id, pubkey, consent_path, actor, reason, replaced_by, request_event_id) \
Expand All @@ -85,7 +111,6 @@ pub async fn archive(
.bind(request_event_id)
.execute(tx.conn())
.await?;
tx.commit().await?;
Ok(result.rows_affected() > 0)
}

Expand All @@ -100,13 +125,19 @@ pub async fn unarchive(pool: &PgPool, community_id: CommunityId, pubkey: &str) -
crate::observability::WriterOperation::EventWrite,
)
.await?;
let changed = unarchive_on(&mut tx, pubkey).await?;
tx.commit().await?;
Ok(changed)
}

async fn unarchive_on(tx: &mut AdmittedTx, pubkey: &str) -> Result<bool> {
let community_id = tx.community();
let result =
sqlx::query("DELETE FROM archived_identities WHERE community_id = $1 AND pubkey = $2")
.bind(community_id.as_uuid())
.bind(pubkey)
.execute(tx.conn())
.await?;
tx.commit().await?;
Ok(result.rows_affected() > 0)
}

Expand Down Expand Up @@ -148,7 +179,68 @@ fn row_to_archived_identity(
})
}

/// Validated archive mutation accompanying a signed identity request.
pub struct IdentityArchiveChange<'a> {
/// Identity whose archive state changes.
pub target: &'a str,
/// Whether to archive (`true`) or unarchive (`false`).
pub archive: bool,
/// Validated consent path: self, owner, or admin.
pub consent_path: &'a str,
/// Optional reason supplied by the actor.
pub reason: Option<&'a str>,
/// Optional replacement identity.
pub replaced_by: Option<&'a str>,
}

impl Db {
/// Atomically persist the signed request, archive mutation, and relay delta.
///
/// The request passes the shared author fence before any archive mutation.
/// Duplicate requests never reapply a mutation; the persisted delta is
/// returned for post-commit delivery when newly inserted. Stored deltas
/// remain available for replay; duplicate requests emit no new delta.
pub async fn apply_identity_archive_event(
&self,
community: CommunityId,
request: &Event,
change: IdentityArchiveChange<'_>,
delta: &Event,
) -> Result<(StoredEvent, bool, Option<StoredEvent>)> {
let mut tx = self.begin_event_write_transaction(community).await?;
let (stored, inserted) =
crate::event::insert_event_in_transaction(&mut tx, request, None).await?;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 IMPORTANT — archive requests and deltas are missing from event_mentions. Before this change, the request was stored through Db::insert_event_with_thread_metadata and the delta through Db::insert_event. Both wrappers call insert_mentions_in_transaction after a new insert (event.rs L2289–2292 and L2829–2831). This function calls the low-level insert_event_in_transaction for both events and never indexes their p tags. Single-#p queries and counts use an INNER JOIN event_mentions (event.rs L628–641, L851–861; the relay pushes #p filters down to this path in req.rs L1220–1231). As a result, newly accepted 9035/9036 requests and their 8002/8003 deltas no longer show up in a #p: [target] subscription or history query, which is how a client replays one identity's archive history.

Fix: call crate::insert_mentions_in_transaction(&mut tx, event, None) for the request when inserted, and for the delta when it's inserted, inside this transaction. Please add a regression test that queries #p = target through the production path and expects both events.

let notification = if inserted {
let changed = if change.archive {
archive_on(
&mut tx,
change.target,
change.consent_path,
&request.pubkey.to_hex(),
change.reason,
change.replaced_by,
&request.id.to_hex(),
)
.await?
} else {
unarchive_on(&mut tx, change.target).await?
};
if changed {
Some(
crate::event::insert_event_in_transaction(&mut tx, delta, None)
.await?
.0,
)
} else {
None
}
} else {
None
};
tx.commit().await?;
Ok((stored, inserted, notification))
}

/// Returns `true` if `pubkey` (64-char hex) is archived in `community_id`.
#[datastore_span(name = "is_archived", system = "postgresql")]
pub async fn is_archived(&self, community_id: CommunityId, pubkey: &str) -> Result<bool> {
Expand Down
180 changes: 180 additions & 0 deletions crates/buzz-db/src/store/artifact_postgres_tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -699,6 +699,186 @@ async fn head_outside_authorized_channel_matches_missing_head() {
));
}

/// Exercise the production Move transaction, including its second, relay-signed
/// insert after domain row writes. Exclusive holders do not touch domain rows.
#[tokio::test]
#[ignore = "requires Postgres"]
async fn artifact_move_and_identity_fences_finish_without_partial_writes_or_deadlocks() {
use crate::user_deletion_fences::{self as fences, FenceIdentity};
use std::{sync::Arc, time::Duration};
for block_relay in [false, true] {
let f = Arc::new(Fixture::new().await);
let d = Uuid::new_v4();
let create = f.revision(d, "create", f.a, None, &f.owner, vec![]);
f.accept(&create, None).await;
let moved = f.revision(d, "move", f.b, Some(&create), &f.owner, vec![]);
let move_id = moved.id;
let store = f.db.entity_deletion_store();
let request =
f.db.user_deletion_store()
.submit(&format!("demo:{}", Uuid::new_v4()), "operator", None)
.await
.unwrap();
store
.approve(
request.id,
"reviewer",
request.inventory_digest.as_deref().unwrap(),
)
.await
.unwrap();
let lease = store
.claim(request.id, "worker", Duration::from_secs(300))
.await
.unwrap()
.unwrap();
let identity = FenceIdentity::user(f.owner.public_key());
let mut gate = f.db.pool.begin().await.unwrap();
if block_relay {
fences::lock_exclusive(&mut gate, &[FenceIdentity::user(f.relay.public_key())])
.await
.unwrap();
} else {
fences::insert(&mut gate, identity, &lease, None)
.await
.unwrap();
}
let gate_pid: i32 = sqlx::query_scalar("SELECT pg_backend_pid()")
.fetch_one(&mut *gate)
.await
.unwrap();
let task_f = f.clone();
let task = tokio::spawn(async move {
let env = artifact::validate(&moved).unwrap();
task_f
.db
.accept_artifact(
task_f.community,
&moved,
&env,
Some(task_f.a),
&task_f.relay,
)
.await
});
wait_for_artifact_blocker(&f.db.pool, gate_pid).await;
let writer_pid: i32 = sqlx::query_scalar("SELECT pid FROM pg_stat_activity WHERE datname=current_database() AND $1=ANY(pg_blocking_pids(pid)) LIMIT 1")
.bind(gate_pid).fetch_one(&f.db.pool).await.unwrap();
let coordinate_held: bool = sqlx::query_scalar(
"SELECT EXISTS(SELECT 1 FROM pg_locks WHERE pid=$1 AND locktype='advisory' AND granted AND mode='ExclusiveLock'
AND classid=((hashtextextended($2,0) >> 32) & 4294967295)::oid AND objid=(hashtextextended($2,0) & 4294967295)::oid)")
.bind(writer_pid).bind(format!("artifact:{}:{}", f.community.as_uuid(), d)).fetch_one(&f.db.pool).await.unwrap();
assert!(
coordinate_held,
"actual artifact caller must hold its domain lock before waiting"
);
let pending_user_fence = if block_relay {
let mut user_gate = f.db.pool.begin().await.unwrap();
let pid: i32 = sqlx::query_scalar("SELECT pg_backend_pid()")
.fetch_one(&mut *user_gate)
.await
.unwrap();
let fence_task = tokio::spawn(async move {
fences::insert(&mut user_gate, identity, &lease, None)
.await
.unwrap();
user_gate.commit().await.unwrap();
});
tokio::time::timeout(Duration::from_secs(3), async {
loop {
let blockers: Vec<i32> = sqlx::query_scalar("SELECT pg_blocking_pids($1)")
.bind(pid)
.fetch_one(&f.db.pool)
.await
.unwrap();
if blockers.contains(&writer_pid) {
break;
}
tokio::time::sleep(Duration::from_millis(10)).await;
}
})
.await
.expect("fence must wait on Move's already acquired author lock");
Some(fence_task)
} else {
None
};
gate.commit().await.unwrap();
let outcome = tokio::time::timeout(Duration::from_secs(8), task)
.await
.unwrap()
.unwrap();
if let Some(task) = pending_user_fence {
tokio::time::timeout(Duration::from_secs(8), task)
.await
.unwrap()
.unwrap();
}
if block_relay {
let ArtifactOutcome::Accepted(events) = outcome.unwrap() else {
panic!("Move should finish before queued author fence");
};
assert_eq!(events.len(), 2);
assert_eq!(events[1].event.kind.as_u16(), 45011);
} else {
assert!(matches!(
outcome,
Err(crate::DbError::UserDeletionRestricted)
));
}
let head: Vec<u8> = sqlx::query_scalar(
"SELECT event_id FROM artifact_heads WHERE community_id=$1 AND artifact_id=$2",
)
.bind(f.community.as_uuid())
.bind(d)
.fetch_one(&f.db.pool)
.await
.unwrap();
assert_eq!(
head,
if block_relay {
move_id.as_bytes()
} else {
create.id.as_bytes()
}
);
let revision_exists: bool = sqlx::query_scalar(
"SELECT EXISTS(SELECT 1 FROM artifact_revisions WHERE community_id=$1 AND event_id=$2)",
)
.bind(f.community.as_uuid())
.bind(move_id.as_bytes().as_slice())
.fetch_one(&f.db.pool)
.await
.unwrap();
assert_eq!(revision_exists, block_relay);
let removals: i64 =
sqlx::query_scalar("SELECT count(*) FROM events WHERE community_id=$1 AND kind=45011")
.bind(f.community.as_uuid())
.fetch_one(&f.db.pool)
.await
.unwrap();
assert_eq!(removals, i64::from(block_relay));
let deadlocks: i64 = sqlx::query_scalar(
"SELECT deadlocks FROM pg_stat_database WHERE datname=current_database()",
)
.fetch_one(&f.db.pool)
.await
.unwrap();
assert_eq!(deadlocks, 0);
}
}

async fn wait_for_artifact_blocker(pool: &PgPool, pid: i32) {
tokio::time::timeout(std::time::Duration::from_secs(3), async {
loop {
let waiting: bool = sqlx::query_scalar("SELECT EXISTS(SELECT 1 FROM pg_stat_activity WHERE datname=current_database() AND $1=ANY(pg_blocking_pids(pid)))")
.bind(pid).fetch_one(pool).await.unwrap();
if waiting { return; }
tokio::time::sleep(std::time::Duration::from_millis(10)).await;
}
}).await.expect("artifact writer must wait on identity lock");
}

/// Sync's complete-set read: keyset pages by `d` share one snapshot, so a
/// change committed after it opens is in no page. Mutation: drop the
/// `REPEATABLE READ` statement → the late create appears in a later page →
Expand Down
2 changes: 2 additions & 0 deletions crates/buzz-db/src/store/event.rs
Original file line number Diff line number Diff line change
Expand Up @@ -477,6 +477,7 @@ async fn insert_event_on(
return Err(DbError::EphemeralEventRejected(kind_u16));
}

tx.guard_event_author(event).await?;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 IMPORTANT — identity locks taken after domain row locks. The ordering rules in user_deletion_fences.rs are: community admission, then all global user locks in one call, then community-scoped agent locks. Because the guard runs at the insert call rather than when the transaction starts, some callers lock rows first. store/artifact.rs accept writes artifact_revisions and upserts artifact_heads before inserting the author's event. On a Move, it then inserts the relay-signed removal marker, which takes a second global user lock after the author's community-scoped lock. Nothing deadlocks today, because normal writers take only shared locks and no exclusive locker touches these rows. But the deletion worker will depend on this order, and once it holds exclusive identity locks and writes these rows, these paths will deadlock and abort.

Two ways to fix it: run the guard before any domain row lock (at admission, or at the top of each caller), or rewrite the ordering rules to match what callers actually do and state the worker-side requirement that follows.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Addressed using the second option in 21f8dc6, included in head 572e30e. The documentation now describes the actual call order: domain locks may precede the guard, and later relay-signed inserts can acquire another global identity lock after the first scoped lock. Per-call sorting is explicitly not a transaction-wide guarantee.

The future exclusive fencing transaction must write only restriction/request/checkpoint state while holding identity locks. It must not lock or mutate serving-domain rows; cleanup must use later transactions. Complete exclusive identity sets must also be audited against multi-signer transactions before enabling the worker.

Added production artifact Move concurrency regressions for a fence winning first and a Move waiting for the relay identity lock while a user fence queues. The tested interleavings complete with the expected persisted/rolled-back state and zero recorded database deadlocks. The channel-TTL race also passes. These tests do not establish universal deadlock safety; the real deletion worker remains disabled.

let id_bytes = event.id.as_bytes();
let pubkey_bytes = event.pubkey.to_bytes();
let sig_bytes = event.sig.serialize();
Expand Down Expand Up @@ -1880,6 +1881,7 @@ pub(crate) async fn insert_event_with_thread_metadata_tx(
return Err(DbError::EphemeralEventRejected(kind_u16));
}

tx.guard_event_author(event).await?;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 IMPORTANT — archive/unarchive takes effect despite the refusal. For kinds 9035/9036, ingest.rs L3082–3089 calls handle_identity_archive_event before the request reaches this inserter. That handler commits archived_identities changes in its own transaction and publishes the relay-signed 8002/8003 deltas and the 13535 snapshot. A restricted author's request therefore changes archive state, then gets refused here and is never stored. The published delta's ["e", request_id] then points at an event that doesn't exist.

Fix: apply the archive change and insert the request in one transaction that runs this guard, and publish after commit. A separate check before the handler would still race a restriction being installed. Please also add archive/unarchive to the HTTP/WS before-and-after-restriction tests, with archived_identities in the unchanged-state snapshot.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Fixed in 4688052, included in head 572e30e. Archive/unarchive validation now produces a plan without changing state. The guarded request insert, archived_identities mutation, and relay delta persist in one transaction; notification dispatch follows commit. A rejected request or failed delta insert rolls all three back. Duplicate requests do not repeat the mutation.

Added HTTP/WebSocket before-and-after-fence regressions, unchanged archive/snapshot assertions, delta-failure rollback, and fence commit/rollback races. These passed in the merged tree's PostgreSQL suite (1,189 tests).

Remaining boundary: the snapshot refresh is post-commit. Its failure propagates, and a duplicate retry refreshes it without repeating the archive mutation. There is no durable background refresh queue, so a crash after commit can leave the snapshot stale until a later refresh. This limitation is documented.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Follow-up fixed in 3564dc2: replaying an already-processed archive/unarchive request now leaves an already-current snapshot unchanged, including its event ID and timestamp, and does not broadcast a replacement. The publisher compares the saved snapshot with canonical archive state and rechecks that state before taking the no-op path. Missing/stale snapshots are still repaired after a failed publication; the existing post-write drift protection remains.

The new replay regression failed against the previous code, then passed with the fix. It covers repeated archive/unarchive requests and replay after a later opposite operation, asserting that all database rows remain unchanged. Recovery tests cover both missing and stale snapshots and a further retry after repair. A deterministic concurrency test changes canonical state while the publisher is paused before its no-op decision and verifies repair.

Validation: all 6 PostgreSQL archive/fencing tests and 8 existing archive consent/publisher tests passed, as did the normal pre-push hooks. The PostgreSQL fixture exercises the HTTP route and authenticated WebSocket event handler; I did not rerun the separate live WebSocket AUTH simulation. The full repository just ci run was not restarted.

let id_bytes = event.id.as_bytes();
let pubkey_bytes = event.pubkey.to_bytes();
let sig_bytes = event.sig.serialize();
Expand Down
Loading
Loading