diff --git a/docs/runbooks/admin-credential-bind-reconciliation.md b/docs/runbooks/admin-credential-bind-reconciliation.md new file mode 100644 index 00000000000..1a7584cfaae --- /dev/null +++ b/docs/runbooks/admin-credential-bind-reconciliation.md @@ -0,0 +1,182 @@ +--- +description: "Reconcile admin WorkOS credential binding failures, provider compensation, lifecycle quarantine, and uncertain outcomes." +"og:title": "AdCP Admin Credential Binding Reconciliation" +--- + +# Admin credential binding reconciliation + +This is the admin containment and compensation slice of #6827. It does not +close the issue or enable organization authorization enforcement. + +`POST /api/admin/users/:userId/linked-emails` creates a fresh WorkOS credential +and binds it without transferring memberships or other app state. An +identity-bearing admin is required. The journal records the exact authenticated +credential and its observed identity; both actor and host bindings are checked +again inside the local transaction. The host must remain primary. + +`POST /api/admin/users/:userId/credentials/:credentialId/promote` returns 409, +including requests with `consolidate: true` and orphan-primary repairs, because +its previous consolidation rewrote authority and membership provenance. + +## Mutation and replay rules + +The dedicated WorkOS client makes one attempt with a ten-second timeout. +WorkOS does not promise general create-user deduplication from an idempotency +header; see the [WorkOS SDK guidance](https://workos.com/docs/sdks/go). + +The `admin_credential_bind_operations` journal and its audit event commit before +the provider request. Its normalized-email hash uniquely reserves the address +across hosts while an operation is unresolved or committed. Session advisory +locks serialize concurrent requests for the address and host; uncertain lock +acquisition or release destroys the database connection. + +| Status | Meaning and operator action | +| --- | --- | +| `creating` | Intent exists; creation may be in flight or its result may have been lost. Keep replay blocked. | +| `provider_created` | The exact returned WorkOS user ID is recorded; local binding may be incomplete. Keep replay blocked. | +| `committed` | User, binding, authorization epochs and audit committed together. A retry returns success only if actor, host and credential remain live and the original binding and email still match under lifecycle locks. | +| `compensating` | Local rollback was acknowledged and a durable admission quarantine blocks callbacks. Deletion of the exact newly created WorkOS user may be in flight. Keep replay blocked. | +| `compensated` | Provider deletion returned success after a confirmed local rollback, and its terminal marker and audit committed under the lifecycle lock. A fresh attempt with a new provider ID is permitted. | +| `provider_rejected` | A non-retryable provider rejection was received without SDK retries. A fresh attempt is permitted. | +| `reconciliation_required` | Provider, commit, compensation or lock outcome is uncertain. Keep replay blocked until adjudicated. | + +A local write failure before COMMIT, followed by an acknowledged ROLLBACK, +permits compensation. Any COMMIT exception requires reconciliation: deleting +the credential could destroy a binding that actually committed. A local row +that appeared independently, including through a webhook, also requires +reconciliation instead of deleting potentially adopted state. + +The local bind and committed replay use #7459's seed-6827 credential lifecycle +locks on the same transaction client, followed by binding and identity row +locks. A deletion or quarantine marker blocks all local binding success writes. +Unknown lock/query/cardinality results require reconciliation. The exact actor +identity and active host primary are rechecked while these locks are held. + +Compensation first rechecks that the returned provider ID has no local user, +binding or terminal marker under that same fence. It commits +`identity_credential_admin_compensation_quarantined` together with the +`compensating` journal transition and checked audit cardinality. This admission +quarantine explicitly records `provider_delete_confirmed: false`; it does not +claim that WorkOS deleted the user. An unconfirmed quarantine COMMIT prevents +the upstream delete entirely. + +A second transaction reacquires the lifecycle lock and requires exactly that +operation's quarantine, matching the provider ID and exact admin credential +and identity, without other markers or local artifacts. It holds the lock +through the provider response and local COMMIT. Only a confirmed provider delete +records the separate `identity_credential_admin_compensation_deleted` marker +with `provider_delete_confirmed: true`. This evidence is distinct from a signed +provider-deletion event. Both lifecycle actions block delayed creation callbacks. +The durable quarantine covers the gap between transactions and survives failed +or ambiguous provider deletion and subsequent marker/audit/COMMIT failures. +Those outcomes remain `reconciliation_required`, with no invented confirmed +deletion claim. `confirmed_delete_evidence_failed` means the provider +acknowledged deletion but its local deletion marker/audit could not commit; +`compensation_commit_ambiguous` means that commit's acknowledgment is uncertain. +Neither result claims successful compensation. A later signed deletion still +records its own provider source and follows #7459's normal replay behavior. + +A provider timeout, transport failure, retryable status or failed compensation +never triggers an immediate GET to infer success or rollback. Delayed completion +can invalidate that read. There is no automatic stale-operation expiry. If a +later journal update fails, the previously committed intent remains blocking. + +## Adjudication + +Use the response's `operation_id` to inspect the journal and audit trail: + +```sql +SELECT id, host_user_id, host_identity_id, actor_user_id, actor_identity_id, + provider_user_id, status, failure_code, created_at, updated_at +FROM admin_credential_bind_operations +WHERE id = :operation_id; + +SELECT created_at, workos_user_id, details +FROM registry_audit_log +WHERE action IN ('admin_credential_bind', 'identity_credential_admin_compensation_deleted', + 'identity_credential_admin_compensation_quarantined') + AND resource_id = :operation_id +ORDER BY created_at; +``` + +Keep provider bodies, passwords, tokens and email addresses out of logs and +incident comments. The journal stores only identifiers, an email hash, fixed +status/failure codes and timestamps. It has no foreign keys so later deletion +cannot erase the evidence. + +Engineering must establish the final outcome of the original request through +provider request/event history and, where necessary, WorkOS support. A single +current GET, a 404, or elapsed time is insufficient. Match the recorded user ID +and original operation; never delete an account found merely by email. + +Inspect the local user, identity binding, epochs and audit in a transaction. +Determine whether a webhook or another flow adopted the created account and +whether it now has independent authority. Preserve those memberships and their +provenance. Repair only the reviewed operation, with an individually attributed +audit event recording the evidence and final disposition in the same transaction +as the journal transition. Mark `committed` only after confirming the expected +binding; release the email reservation as `compensated` only after establishing +that the original creation/deletion request has completed, the created account +is absent, and no local binding remains. Leave unresolved evidence blocking. +For a quarantined credential, retain its quarantine and record confirmed +deletion evidence under the shared credential lifecycle lock before committing +`compensated`. A quarantine alone is never proof of provider deletion. +There is intentionally no public retry-unblock or reconciliation endpoint. + +## Route inventory and scope + +The existing-credential `POST /api/admin/users/:userId/credentials` remains +contained by #7459 before its legacy WorkOS GET and generic merge handler. +The following admin provider-write routes were traced and do not subsequently +bind or promote a person identity: + +- `POST /api/admin/organizations/:orgId/add-users` +- `POST /api/admin/organizations/audit-admins` with `fix=true` (organization role promotion) +- `POST /api/admin/accounts/:id/migrate-members` +- `PUT /api/admin/accounts/:orgId/members/:userId/role` +- `POST /api/admin/domain-users/backfill-members` + +No membership mutation was added to fresh credential binding. Member primary +email mutation/reconciliation, automatic alias detection, generic merge +containment and detach integrity remain separate work. + +## Merge order with containment #7459 + +This PR is stacked on the exact [#7459](https://github.com/adcontextprotocol/adcp/pull/7459) +head `9a7cd5362787a0badbfccd0f1cb1dd002654c22f`. Land #7459 first, then retarget +and rebase this PR onto main. Its database, member, automatic alias, +existing-credential and promotion guards remain intact. Fresh binding does not +call or add an exception to `mergeUsers` or the consolidation policy. + +The only predecessor is corrected #7459. The admin saga has no API or schema +dependency on #7460 or migration 594; it uses existing identity, epoch and audit +tables plus its independent journal 593. Keep the corrected predecessor's +post-deletion missing-primary signal intact when rebasing. + +The shared `server/src/routes/admin/users.ts` hunk is resolved as follows: + +- Keep both the `refuseIdentityConsolidation` and `createAndBindAdminCredential` + imports. +- Keep `refuseIdentityConsolidation` on `POST /:userId/credentials`. +- Keep `POST /:userId/credentials/:credentialId/promote` as an unconditional + refusal using that middleware, with no legacy mutation body or orphan repair. +- Replace only `POST /:userId/linked-emails`'s blanket refusal and old merge body + with this PR's fresh-credential handler and durable saga. Requests to + consolidate or promote are still refused before any provider operation. + +The overlapping tests retain the fresh-bind/fault tests and #7459's +existing-credential and broader promotion containment assertions. Promotion +fixtures bind directly in SQL. Only fresh `/linked-emails` is removed from the +blanket-refusal matrix; new route tests cover identity-bearing authentication, +destructive intent rejection, exact actor attribution and provider/local failure +boundaries. Both PRs' focused suites and typecheck must pass after rebasing. + +Migration 593 was assigned after a read-only live ledger audit on 2026-09-13: +the highest applied version was 587; versions 588–592 and 593 were absent. +Main subsequently added unrelated migration 589; this change adds or modifies +none of 588–592. Migration 592 remains owned by member compensation +[#7463](https://github.com/adcontextprotocol/adcp/pull/7463). The local/origin-main +union and open-PR migration audit have no collision for 593. Recheck migration +numbering before merging parallel security PRs. If #7463 lands first, retain +both independent WorkOS client factories and both timeout tests when resolving +their shared insertion points; the admin journal does not depend on 592. diff --git a/server/public/admin-people.html b/server/public/admin-people.html index 1e9f7c318a3..e339f1f5c64 100644 --- a/server/public/admin-people.html +++ b/server/public/admin-people.html @@ -593,6 +593,8 @@

AAO site-ad data-cred-id="${escapeHtml(c.workos_user_id)}" data-cred-email="${escapeHtml(c.email || '')}" data-host-id="${escapeHtml(r.workos_user_id)}" + data-identity-id="${escapeHtml(data.identity_id)}" + data-authorization-epoch="${escapeHtml(c.authorization_epoch)}" data-target-label="${escapeHtml(targetLabel)}">Remove `; return ` @@ -639,7 +641,7 @@

Sign-in cre container.querySelectorAll('.unbind-cred-btn').forEach(btn => { btn.addEventListener('click', (ev) => { const b = ev.currentTarget; - unbindCredential(b.dataset.hostId, b.dataset.credId, b.dataset.credEmail, b.dataset.targetLabel); + unbindCredential(b.dataset.hostId, b.dataset.credId, b.dataset.credEmail, b.dataset.targetLabel, b.dataset.identityId, b.dataset.authorizationEpoch); }); }); container.querySelectorAll('.promote-cred-btn').forEach(btn => { @@ -751,10 +753,10 @@

Sign-in cre } } - async function unbindCredential(hostId, credId, credEmail, targetLabel) { + async function unbindCredential(hostId, credId, credEmail, targetLabel, identityId, authorizationEpoch) { const credLabel = credEmail || credId; if (credEmail) { - const typed = prompt(`Remove sign-in credential ${credLabel} from ${targetLabel}?\n\nThe WorkOS user stays alive in WorkOS but will no longer be associated with this account. Sign-ins via that email will reach an empty workspace.\n\nTo confirm, type the credential's email exactly:\n ${credEmail}`); + const typed = prompt(`Remove sign-in credential ${credLabel} from ${targetLabel}?\n\nThe WorkOS user stays alive in WorkOS but will no longer be associated with this account. Existing memberships and permissions stay with their current credential. This does not undo past account consolidation.\n\nTo confirm, type the credential's email exactly:\n ${credEmail}`); if (typed === null) return; if (typed.trim().toLowerCase() !== credEmail.trim().toLowerCase()) { alert('Confirmation email did not match. Cancelled.'); @@ -766,6 +768,8 @@

Sign-in cre try { const res = await AdminSidebar.fetch(`/api/admin/users/${encodeURIComponent(hostId)}/credentials/${encodeURIComponent(credId)}`, { method: 'DELETE', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ expected_identity_id: identityId, expected_authorization_epoch: authorizationEpoch }), }); const data = await res.json(); if (!res.ok) { diff --git a/server/src/addie/admin-status-lookup.ts b/server/src/addie/admin-status-lookup.ts index daf2ed7711a..b6e12753d78 100644 --- a/server/src/addie/admin-status-lookup.ts +++ b/server/src/addie/admin-status-lookup.ts @@ -43,9 +43,10 @@ const wgDb = new WorkingGroupDatabase(); export async function isWebUserAAOAdmin(workosUserId: string): Promise { const cache = getWebAdminStatusCache(); const cached = cache.get(workosUserId); - if (cached && cached.expiresAt > Date.now()) { + if (cached && !cached.isAdmin && cached.expiresAt > Date.now()) { return cached.isAdmin; } + if (cached?.isAdmin) cache.delete(workosUserId); try { const adminGroup = await wgDb.getWorkingGroupBySlug(AAO_ADMIN_WORKING_GROUP_SLUG); @@ -58,10 +59,12 @@ export async function isWebUserAAOAdmin(workosUserId: string): Promise } const isAdmin = await wgDb.isMember(adminGroup.id, workosUserId); - cache.set(workosUserId, { - isAdmin, - expiresAt: Date.now() + (isAdmin ? AAO_ADMIN_POSITIVE_CACHE_TTL_MS : AAO_ADMIN_NEGATIVE_CACHE_TTL_MS), - }); + if (!isAdmin) { + cache.set(workosUserId, { + isAdmin: false, + expiresAt: Date.now() + AAO_ADMIN_NEGATIVE_CACHE_TTL_MS, + }); + } logger.debug({ workosUserId, isAdmin }, 'Checked web user admin status'); return isAdmin; } catch (error) { diff --git a/server/src/addie/mcp/admin-tools.ts b/server/src/addie/mcp/admin-tools.ts index 2747ac97a22..63a421659b5 100644 --- a/server/src/addie/mcp/admin-tools.ts +++ b/server/src/addie/mcp/admin-tools.ts @@ -220,9 +220,10 @@ export async function isSlackUserAAOAdmin( ): Promise { // Check cache first const cached = adminStatusCache.get(slackUserId); - if (cached && cached.expiresAt > Date.now()) { + if (cached && !cached.isAdmin && cached.expiresAt > Date.now()) { return cached.isAdmin; } + if (cached?.isAdmin) adminStatusCache.delete(slackUserId); try { // Look up the Slack user mapping to get their WorkOS user ID @@ -259,10 +260,12 @@ export async function isSlackUserAAOAdmin( const isAdmin = await wgDb.isMember(adminGroup.id, mapping.workos_user_id); // Cache the result - adminStatusCache.set(slackUserId, { - isAdmin, - expiresAt: Date.now() + (isAdmin ? ADMIN_POSITIVE_CACHE_TTL_MS : ADMIN_NEGATIVE_CACHE_TTL_MS), - }); + if (!isAdmin) { + adminStatusCache.set(slackUserId, { + isAdmin: false, + expiresAt: Date.now() + ADMIN_NEGATIVE_CACHE_TTL_MS, + }); + } logger.info( { diff --git a/server/src/auth/workos-client.ts b/server/src/auth/workos-client.ts index fa155e88293..4037a7bc1f0 100644 --- a/server/src/auth/workos-client.ts +++ b/server/src/auth/workos-client.ts @@ -13,6 +13,7 @@ let _ownerlessPromotionWorkos: WorkOS | null = null; let _pipesWorkos: WorkOS | null = null; let _authorizationObserverWorkos: WorkOS | null = null; let _authorizationEnforcementWorkos: WorkOS | null = null; +let _adminCredentialMutationWorkos: WorkOS | null = null; let _clientId = ''; /** Returns the shared WorkOS client. Constructed on first call; WORKOS_API_KEY and WORKOS_CLIENT_ID must be set by then. */ @@ -26,6 +27,21 @@ export function getWorkos(): WorkOS { return _workos; } +/** One attempt only: a timed-out admin mutation requires durable adjudication. */ +export function getAdminCredentialMutationWorkos(): WorkOS { + if (!_adminCredentialMutationWorkos) { + if (!process.env.WORKOS_API_KEY || !process.env.WORKOS_CLIENT_ID) { + throw new Error('WorkOS configuration is required'); + } + _adminCredentialMutationWorkos = new WorkOS(process.env.WORKOS_API_KEY, { + clientId: process.env.WORKOS_CLIENT_ID, + timeout: 10_000, + maxRetries: 0, + }); + } + return _adminCredentialMutationWorkos; +} + /** Returns a WorkOS client with a shorter timeout for DB-lock-held promotion flows. */ export function getOwnerlessPromotionWorkos(): WorkOS { if (!_ownerlessPromotionWorkos) { diff --git a/server/src/db/authorization-epoch-db.ts b/server/src/db/authorization-epoch-db.ts index 493b3747d22..54b64d8c8f1 100644 --- a/server/src/db/authorization-epoch-db.ts +++ b/server/src/db/authorization-epoch-db.ts @@ -7,9 +7,10 @@ * the auth middleware stamps the observed fingerprint onto its cache entries * and drops any entry whose fingerprint no longer matches. * - * The fingerprint is compared for *inequality*, never ordering: a CASCADE - * delete (WorkOS `user.deleted`) removes a row, which moves the fingerprint - * backwards and must still invalidate. + * The fingerprint is compared for *inequality*, never ordering. Confirmed + * deletion contributes its durable, non-cascading audit tombstone, so even an + * epoch-0 credential moves from the empty fingerprint and cannot survive in a + * different process's session/JWT cache after the users-row CASCADE. */ import type { Pool, PoolClient } from 'pg'; @@ -18,6 +19,24 @@ import { query } from './client.js'; /** Anything that can run a parameterized statement — pool or in-transaction client. */ type Queryable = Pick; +export interface ActiveCredentialAuthorizationSnapshot { + workos_user_id: string; + email: string | null; + first_name: string | null; + last_name: string | null; + identity_id: string; + primary_workos_user_id: string | null; + fingerprint: string; +} + +export type CredentialAuthorizationLifecycle = + | { status: 'active'; snapshot: ActiveCredentialAuthorizationSnapshot } + | { + status: 'terminal'; + reason: 'deleted_or_quarantined' | 'missing_user' | 'missing_binding' | 'missing_primary'; + } + | { status: 'unavailable' }; + /** * Bump the authorization epoch for each credential, in the caller's * transaction. Call this in the same transaction as the binding mutation: @@ -31,17 +50,19 @@ type Queryable = Pick; export async function bumpAuthorizationEpochs( db: Queryable, workosUserIds: string[], -): Promise { +): Promise { const ids = [...new Set(workosUserIds.filter(Boolean))]; - if (ids.length === 0) return; + if (ids.length === 0) return []; - await db.query( + const bumped = await db.query<{ workos_user_id: string }>( `INSERT INTO authorization_epochs (workos_user_id, epoch) SELECT u.workos_user_id, 1 FROM users u WHERE u.workos_user_id = ANY($1) ON CONFLICT (workos_user_id) DO UPDATE - SET epoch = authorization_epochs.epoch + 1, updated_at = NOW()`, + SET epoch = authorization_epochs.epoch + 1, updated_at = NOW() + RETURNING workos_user_id`, [ids], ); + return bumped.rows.map((row) => row.workos_user_id); } /** @@ -60,11 +81,110 @@ export async function getAuthorizationFingerprint( if (ids.length === 0) return ''; const result = await query<{ fingerprint: string | null }>( - `SELECT string_agg(workos_user_id || ':' || epoch, ',' ORDER BY workos_user_id) - AS fingerprint - FROM authorization_epochs - WHERE workos_user_id = ANY($1)`, + `WITH fingerprint_parts AS ( + SELECT ae.workos_user_id || ':epoch:' || ae.epoch AS part + FROM authorization_epochs ae + WHERE ae.workos_user_id = ANY($1) + UNION ALL + SELECT ral.workos_user_id || + CASE WHEN ral.action = 'identity_credential_admin_compensation_quarantined' + THEN ':quarantined:' ELSE ':deleted:' END || ral.id AS part + FROM registry_audit_log ral + WHERE ral.workos_user_id = ANY($1) + AND ral.action IN ('identity_credential_deleted', 'identity_primary_deletion_quarantined', + 'identity_credential_admin_compensation_deleted', 'identity_credential_admin_compensation_quarantined') + ) + SELECT string_agg(part, ',' ORDER BY part) AS fingerprint + FROM fingerprint_parts`, [ids], ); return result.rows[0]?.fingerprint ?? ''; } + +/** + * Authoritative positive-authentication snapshot for one real credential. + * + * A fingerprint alone is not proof that the credential is live: after + * deletion the durable marker is itself a stable fingerprint. Positive auth + * cache acceptance and writes must therefore read the local user, identity + * binding, absence of the deletion tombstone, canonical route, and epoch in + * one database statement. Missing/corrupt/deleted credentials are terminal; + * database errors become an explicit unavailable state so callers fail closed. + */ +export async function readCredentialAuthorizationLifecycle( + workosUserId: string, +): Promise { + try { + const result = await query( + `WITH requested AS ( + SELECT $1::text AS workos_user_id + ) + SELECT requested.workos_user_id, + u.email, + u.first_name, + u.last_name, + iwu.identity_id, + primary_binding.workos_user_id AS primary_workos_user_id, + CASE WHEN ae.workos_user_id IS NULL THEN '' + ELSE ae.workos_user_id || ':epoch:' || ae.epoch + END AS fingerprint, + (u.workos_user_id IS NOT NULL) AS user_exists, + (iwu.workos_user_id IS NOT NULL) AS binding_exists, + COALESCE(primary_binding.primary_count, 0)::text AS primary_count, + EXISTS ( + SELECT 1 + FROM registry_audit_log ral + WHERE ral.workos_user_id = requested.workos_user_id + AND ral.action IN ( + 'identity_credential_deleted', + 'identity_primary_deletion_quarantined', + 'identity_credential_admin_compensation_deleted', + 'identity_credential_admin_compensation_quarantined' + ) + ) AS terminal_marker + FROM requested + LEFT JOIN users u + ON u.workos_user_id = requested.workos_user_id + LEFT JOIN identity_workos_users iwu + ON iwu.workos_user_id = requested.workos_user_id + LEFT JOIN LATERAL ( + SELECT COUNT(*) AS primary_count, + MIN(candidate.workos_user_id) AS workos_user_id + FROM identity_workos_users candidate + WHERE candidate.identity_id = iwu.identity_id + AND candidate.is_primary = TRUE + ) primary_binding ON TRUE + LEFT JOIN authorization_epochs ae + ON ae.workos_user_id = requested.workos_user_id`, + [workosUserId], + ); + const row = result.rows[0]; + if (!row || row.terminal_marker) { + return { status: 'terminal', reason: 'deleted_or_quarantined' }; + } + if (!row.user_exists) return { status: 'terminal', reason: 'missing_user' }; + if (!row.binding_exists) return { status: 'terminal', reason: 'missing_binding' }; + if (Number(row.primary_count) !== 1 || !row.primary_workos_user_id) { + return { status: 'terminal', reason: 'missing_primary' }; + } + return { + status: 'active', + snapshot: { + workos_user_id: row.workos_user_id, + email: row.email, + first_name: row.first_name, + last_name: row.last_name, + identity_id: row.identity_id, + primary_workos_user_id: row.primary_workos_user_id, + fingerprint: row.fingerprint, + }, + }; + } catch { + return { status: 'unavailable' }; + } +} diff --git a/server/src/db/credential-detach-db.ts b/server/src/db/credential-detach-db.ts new file mode 100644 index 00000000000..47775c30e0f --- /dev/null +++ b/server/src/db/credential-detach-db.ts @@ -0,0 +1,243 @@ +import { randomUUID } from 'node:crypto'; +import type { QueryResult, QueryResultRow } from 'pg'; +import { getPool } from './client.js'; + +export class CredentialDetachConflict extends Error {} + +export interface DetachCredentialInput { + hostUserId: string; + credentialId: string; + expectedIdentityId: string; + expectedAuthorizationEpoch: string; + actorUserId: string; + actorCredentialId: string; + actorIdentityId: string; +} + +function exactlyOne(result: QueryResult, operation: string): T { + if (result.rowCount !== 1 || result.rows.length !== 1 || !result.rows[0]) { + throw new Error(`Credential detach: ${operation} did not return exactly one row`); + } + return result.rows[0]; +} + +/** + * Detach only the observed non-primary credential/person binding. Authority + * rows are never consolidated or reconstructed, including after legacy merges. + * + * Lock order: sorted identity advisory keys, sorted identity rows, sorted + * binding rows, sorted credential lifecycle try-locks, sorted user rows, + * sorted bind-operation rows, sorted epoch writes. Every acquisition that can + * conflict with the parent's credential-before-binding protocol is nonblocking: + * NOWAIT/try-lock failure rolls back instead of creating an inverse-order wait. + * Session locks are checked and released BEFORE COMMIT while row locks still + * protect the binding: an unlock failure must roll back the entire operation. + * Any failure destroys the connection, releasing even an uncertain session lock. + */ +export async function detachCredential(input: DetachCredentialInput): Promise<{ + newIdentityId: string; + affectedCredentialIds: string[]; +}> { + const client = await getPool().connect(); + let destroyClient = false; + try { + await client.query('BEGIN'); + const identityIds = [...new Set([input.expectedIdentityId, input.actorIdentityId])].sort(); + const lockKeys = identityIds.map((id) => `identity-binding:${id}`); + for (const key of lockKeys) { + const lock = exactlyOne(await client.query<{ acquired: boolean }>( + 'SELECT pg_try_advisory_lock(hashtextextended($1, 0)) AS acquired', [key], + ), 'advisory lock'); + if (lock.acquired === false) throw new CredentialDetachConflict('Identity binding is busy; reload and retry'); + if (lock.acquired !== true) throw new Error('Credential detach: advisory lock was not acquired'); + } + + const identities = await client.query<{ id: string }>( + `SELECT id FROM identities WHERE id = ANY($1::uuid[]) ORDER BY id FOR UPDATE NOWAIT`, + [identityIds], + ); + if (identities.rowCount !== identityIds.length) { + throw new CredentialDetachConflict('Identity changed; reload credentials'); + } + const bindings = await client.query<{ + workos_user_id: string; identity_id: string; is_primary: boolean; + }>( + `SELECT workos_user_id, identity_id, is_primary FROM identity_workos_users + WHERE identity_id = ANY($1::uuid[]) ORDER BY workos_user_id FOR UPDATE NOWAIT`, + [identityIds], + ); + const credentialIds = bindings.rows.map((row) => row.workos_user_id).sort(); + if (bindings.rowCount !== credentialIds.length || new Set(credentialIds).size !== credentialIds.length || + bindings.rows.some((row) => !row.workos_user_id || !identityIds.includes(row.identity_id) || + typeof row.is_primary !== 'boolean')) { + throw new Error('Credential detach: binding locks were not confirmed'); + } + // Bind/deletion writers use this same credential fence. Acquiring it after + // our NOWAIT row locks avoids an unlocked discovery snapshot; try-locks + // ensure an earlier lifecycle writer always makes this detach fail closed. + for (const credentialId of credentialIds) { + const lock = exactlyOne(await client.query<{ acquired: boolean }>( + 'SELECT pg_try_advisory_xact_lock(hashtextextended($1, 6827)) AS acquired', [credentialId], + ), 'credential lifecycle lock'); + if (lock.acquired === false) throw new CredentialDetachConflict('Credential lifecycle is busy; reload and retry'); + if (lock.acquired !== true) throw new Error('Credential detach: lifecycle lock was not acquired'); + } + const host = bindings.rows.find((row) => row.workos_user_id === input.hostUserId); + const target = bindings.rows.find((row) => row.workos_user_id === input.credentialId); + const actor = bindings.rows.find((row) => row.workos_user_id === input.actorCredentialId); + const actorPrimaries = bindings.rows.filter((row) => row.identity_id === input.actorIdentityId && row.is_primary); + if (actor?.identity_id !== input.actorIdentityId || actorPrimaries.length !== 1 || + actorPrimaries[0].workos_user_id !== input.actorUserId) { + throw new CredentialDetachConflict('Acting identity changed; authenticate again'); + } + if (host?.identity_id !== input.expectedIdentityId || target?.identity_id !== input.expectedIdentityId) { + throw new CredentialDetachConflict('Credential binding changed; reload credentials'); + } + if (target.is_primary || input.hostUserId === input.credentialId) { + throw new CredentialDetachConflict('Cannot remove the primary credential'); + } + if (bindings.rows.filter((row) => row.identity_id === input.expectedIdentityId && row.is_primary).length !== 1) { + throw new CredentialDetachConflict('Identity has no primary credential; repair it before detaching'); + } + const affectedCredentialIds = bindings.rows + .filter((row) => row.identity_id === input.expectedIdentityId) + .map((row) => row.workos_user_id).sort(); + // Keep the complete actor/affected graphs live, including the canonical + // actor, until all required epochs and the attributed audit are committed. + const liveUsers = await client.query<{ workos_user_id: string }>( + `SELECT workos_user_id FROM users WHERE workos_user_id = ANY($1) + ORDER BY workos_user_id FOR UPDATE NOWAIT`, [credentialIds], + ); + if (liveUsers.rowCount !== credentialIds.length || liveUsers.rows.length !== credentialIds.length || + new Set(liveUsers.rows.map((row) => row.workos_user_id)).size !== credentialIds.length || + liveUsers.rows.some((row) => !credentialIds.includes(row.workos_user_id))) { + throw new CredentialDetachConflict('Credential lifecycle changed; reload credentials'); + } + async function requireActiveLifecycle() { + // Journal reconciliation can be recorded after a bind has released its + // lifecycle fence. Lock every matching row, including committed rows, + // before checking status so that transition serializes with this detach. + const operations = await client.query<{ id: string; provider_user_id: string; status: string }>( + `SELECT id, provider_user_id, status FROM admin_credential_bind_operations + WHERE provider_user_id = ANY($1) ORDER BY id FOR UPDATE NOWAIT`, [credentialIds], + ); + if (operations.rowCount !== operations.rows.length || + new Set(operations.rows.map((row) => row.id)).size !== operations.rows.length || + operations.rows.some((row) => !row.id || !credentialIds.includes(row.provider_user_id) || row.status !== 'committed')) { + throw new CredentialDetachConflict('Credential lifecycle requires reconciliation'); + } + const marker = exactlyOne(await client.query<{ blocked: boolean }>( + `SELECT EXISTS ( + SELECT 1 FROM registry_audit_log + WHERE (workos_user_id = ANY($1) AND action IN ( + 'identity_credential_deleted', 'identity_primary_deletion_quarantined', + 'identity_credential_admin_compensation_deleted', 'identity_credential_admin_compensation_quarantined')) + OR (action = 'admin_credential_bind' AND details->>'provider_user_id' = ANY($1) + AND details->>'status' IN ('reconciliation_required', 'compensating', 'compensated')) + ) AS blocked`, [credentialIds], + ), 'lifecycle marker check'); + // Bind audits attribute workos_user_id to the actor: only their explicit + // provider_user_id identifies the credential that requires reconciliation. + if (marker.blocked !== false) throw new CredentialDetachConflict('Credential lifecycle is terminal or requires reconciliation'); + } + await requireActiveLifecycle(); + const beforeEpochs = await client.query<{ workos_user_id: string; epoch: string }>( + `SELECT workos_user_id, epoch::text FROM authorization_epochs WHERE workos_user_id = ANY($1) + ORDER BY workos_user_id FOR UPDATE NOWAIT`, [affectedCredentialIds], + ); + const previousEpochs = new Map(beforeEpochs.rows.map((row) => [row.workos_user_id, row.epoch])); + if ((previousEpochs.get(input.credentialId) ?? '0') !== input.expectedAuthorizationEpoch) { + throw new CredentialDetachConflict('Credential authorization changed; reload credentials'); + } + + const newIdentityId = randomUUID(); + const newIdentity = exactlyOne(await client.query<{ id: string }>( + 'INSERT INTO identities (id) VALUES ($1) RETURNING id', [newIdentityId], + ), 'identity insert'); + if (newIdentity.id !== newIdentityId) throw new Error('Credential detach: identity insert failed'); + const detached = exactlyOne(await client.query<{ workos_user_id: string; identity_id: string; is_primary: boolean }>( + `UPDATE identity_workos_users SET identity_id = $1, is_primary = TRUE, bound_at = NOW() + WHERE workos_user_id = $2 AND identity_id = $3 AND is_primary = FALSE + RETURNING workos_user_id, identity_id, is_primary`, + [newIdentityId, input.credentialId, input.expectedIdentityId], + ), 'binding update'); + if (detached.workos_user_id !== input.credentialId || detached.identity_id !== newIdentityId || detached.is_primary !== true) { + throw new Error('Credential detach: binding update failed'); + } + + // Every session on the old identity must observe the binding change, even + // when the URL's host is itself secondary. Check suppression as well as errors. + const epochs = await client.query<{ workos_user_id: string; epoch: string }>( + `INSERT INTO authorization_epochs (workos_user_id, epoch) + SELECT workos_user_id, 1 FROM users WHERE workos_user_id = ANY($1) ORDER BY workos_user_id + ON CONFLICT (workos_user_id) DO UPDATE + SET epoch = authorization_epochs.epoch + 1, updated_at = NOW() + RETURNING workos_user_id, epoch::text`, [affectedCredentialIds], + ); + function verifyEpochs(result: typeof epochs) { + if (result.rowCount !== affectedCredentialIds.length || result.rows.length !== affectedCredentialIds.length || + new Set(result.rows.map((row) => row.workos_user_id)).size !== affectedCredentialIds.length || + result.rows.some((row) => !affectedCredentialIds.includes(row.workos_user_id) || !row.epoch || + BigInt(row.epoch) !== BigInt(previousEpochs.get(row.workos_user_id) ?? '0') + 1n)) { + throw new Error('Credential detach: authorization epochs were not advanced'); + } + } + verifyEpochs(epochs); + + const auditId = randomUUID(); + const details = JSON.stringify({ + operation_id: auditId, + host_user_id: input.hostUserId, + affected_workos_user_id: input.credentialId, + detached_from_identity_id: input.expectedIdentityId, + new_identity_id: newIdentityId, + acting_workos_user_id: input.actorCredentialId, + acting_identity_id: actor.identity_id, + expected_authorization_epoch: input.expectedAuthorizationEpoch, + affected_credential_ids: affectedCredentialIds, + }); + const audit = exactlyOne(await client.query<{ id: string }>( + `INSERT INTO registry_audit_log ( + id, workos_organization_id, workos_user_id, action, resource_type, resource_id, details + ) VALUES ($1, 'system', $2, 'unbind_credential', 'user', $3, $4) RETURNING id`, + [auditId, input.actorUserId, input.credentialId, details], + ), 'audit insert'); + if (audit.id !== auditId) throw new Error('Credential detach: audit insert returned an invalid id'); + // Flush deferred triggers before the final stored-state checks; otherwise + // they could suppress the audit or epoch advance during COMMIT itself. + await client.query('SET CONSTRAINTS ALL IMMEDIATE'); + await requireActiveLifecycle(); + verifyEpochs(await client.query<{ workos_user_id: string; epoch: string }>( + `SELECT workos_user_id, epoch::text FROM authorization_epochs WHERE workos_user_id = ANY($1) + ORDER BY workos_user_id`, [affectedCredentialIds], + )); + // RETURNING alone cannot detect an AFTER INSERT trigger deleting or + // duplicating the audit. Require exactly one stored record for this operation. + const storedAudit = exactlyOne(await client.query<{ id: string; matches: boolean }>( + `SELECT id, (workos_organization_id = 'system' AND workos_user_id = $2 + AND action = 'unbind_credential' AND resource_type = 'user' AND resource_id = $3 + AND details = $4::jsonb) AS matches + FROM registry_audit_log WHERE details->>'operation_id' = $1`, + [auditId, input.actorUserId, input.credentialId, details], + ), 'stored audit'); + if (storedAudit.id !== auditId || storedAudit.matches !== true) throw new Error('Credential detach: audit record is missing or changed'); + + for (const key of [...lockKeys].reverse()) { + const unlock = exactlyOne(await client.query<{ unlocked: boolean }>( + 'SELECT pg_advisory_unlock(hashtextextended($1, 0)) AS unlocked', [key], + ), 'advisory unlock'); + if (unlock.unlocked !== true) throw new Error('Credential detach: advisory unlock failed'); + } + await client.query('COMMIT'); + return { newIdentityId, affectedCredentialIds }; + } catch (error) { + destroyClient = true; + await client.query('ROLLBACK').catch(() => undefined); + if ((error as { code?: string }).code === '55P03') { + throw new CredentialDetachConflict('Identity binding is busy; reload and retry'); + } + throw error; + } finally { + client.release(destroyClient); + } +} diff --git a/server/src/db/identity-db.ts b/server/src/db/identity-db.ts index 9de63e54f0d..93667947a76 100644 --- a/server/src/db/identity-db.ts +++ b/server/src/db/identity-db.ts @@ -11,39 +11,1269 @@ */ import { getPool } from './client.js'; +import type { PoolClient } from 'pg'; +import { assertIdentityConsolidationAllowed } from './identity-mutation-policy.js'; import { bumpAuthorizationEpochs } from './authorization-epoch-db.js'; import { createLogger } from '../logger.js'; import { notifySystemError } from '../addie/error-notifier.js'; const logger = createLogger('identity-db'); +export const IDENTITY_RECOVERY_STATE = 'manual_primary_selection_required' as const; +export const IDENTITY_DELETION_SNAPSHOT_MAX_ROWS_PER_COLLECTION = 10_000; +export const IDENTITY_DELETION_SNAPSHOT_MAX_DETAILS_BYTES = 4 * 1024 * 1024; +export const IDENTITY_DELETION_SNAPSHOT_POLICY = { + completeness: 'complete_or_fail_closed', + max_rows_per_collection: IDENTITY_DELETION_SNAPSHOT_MAX_ROWS_PER_COLLECTION, + max_details_bytes: IDENTITY_DELETION_SNAPSHOT_MAX_DETAILS_BYTES, +} as const; + +export type IdentityCredentialDeletionSource = 'workos_webhook' | 'sync_users_backfill'; + +export interface IdentityBeforeGraph { + identity: Record; + identity_workos_users: Record[]; + users: Record[]; + organization_memberships: Record[]; + working_group_memberships: Record[]; + working_group_leaders: Record[]; + working_group_topic_subscriptions: Record[]; + slack_user_mappings: Record[]; + authorization_epochs: Record[]; + /** Every current ON DELETE CASCADE child of users, keyed by table name. */ + user_cascade_rows: Record[]>; + /** Attribution rows whose user FK is cleared with ON DELETE SET NULL. */ + user_set_null_attributions: Record[]>; +} + +export interface AAOAdminParallelRevocationEvidence { + event_type: 'revoked'; + target_user_id: string; + working_group_id: string; + mechanism: 'aao_admin_working_group'; + actor: { + type: 'workos_provider'; + source: IdentityCredentialDeletionSource; + workos_user_id: string; + }; + evidence_ledger: 'registry_audit_log'; + dedicated_ledger_limitation: 'provider_deletion_actor_not_representable'; +} + +export interface IdentityRecoveryQuarantine { + audit_id: string; + identity_id: string; + deleted_workos_user_id: string; + deletion_source: IdentityCredentialDeletionSource; + actor: { + type: 'workos_provider'; + source: IdentityCredentialDeletionSource; + workos_user_id: string; + }; + recovery_state: typeof IDENTITY_RECOVERY_STATE; + snapshot_policy: typeof IDENTITY_DELETION_SNAPSHOT_POLICY; + before_graph: IdentityBeforeGraph; + aao_admin_revocations: AAOAdminParallelRevocationEvidence[]; +} + +export interface IdentityCredentialDeletionResult { + deleted: boolean; + affectedUserIds: string[]; + affectedSlackUserIds: string[]; + quarantine: IdentityRecoveryQuarantine | null; + replay: { + audit_id: string; + kind: 'same_source' | 'different_source'; + original_deletion_source: IdentityCredentialDeletionSource; + requested_deletion_source: IdentityCredentialDeletionSource; + } | null; +} + +export interface WorkosUserUpsert { + id: string; + email: string; + firstName: string | null; + lastName: string | null; + emailVerified: boolean; + createdAt: string; + updatedAt: string; +} + +interface JsonRow { + row: Record; +} + /** - * When a WorkOS user that is the primary binding on a multi-credential - * identity is deleted (operator action, account closure, GDPR/CCPA erasure - * webhook), the CASCADE on `identity_workos_users.workos_user_id` drops the - * binding. The identity is left with zero primaries until an admin - * intervenes — `attachIdentityId` resolves `primary_workos_user_id` to NULL - * and skips the id-swap, so the surviving secondary signs in to an empty - * workspace (all their app-state was keyed on the now-deleted primary's - * workos_user_id). That's a denial-of-service against any non-primary user. - * - * Promote the longest-bound surviving secondary to primary in the same - * transaction, before the CASCADE fires. The id-swap in the auth middleware - * then routes both the dead binding and the surviving secondary to the new - * primary, keeping app-state reads intact. - * - * Returns the promoted credential's workos_user_id when a promotion ran, or - * null when there was no successor (single-credential identity — normal - * deletion, nothing to do) or the deleted user was not primary. - * - * On unexpected DB errors: log + ops alert, return null. The caller still - * returns 200 to WorkOS so the webhook doesn't retry-storm on a transient - * promotion failure; the binding is still deleted by the subsequent CASCADE, - * and admins can repair the identity manually. + * Current users-row CASCADE inventory. Keep this explicit and test it against + * pg_constraint so a future authority/provenance table cannot disappear + * without becoming part of the durable deletion snapshot. + */ +export const USER_CASCADE_SNAPSHOT_INVENTORY = [ + { table: 'authorization_epochs', columns: ['workos_user_id'] }, + { table: 'certification_attempts', columns: ['workos_user_id'] }, + { table: 'certification_learner_feedback', columns: ['workos_user_id'] }, + { table: 'community_points', columns: ['workos_user_id'] }, + { table: 'connections', columns: ['requester_user_id', 'recipient_user_id'] }, + { + table: 'email_link_tokens', + columns: ['primary_workos_user_id'], + redactColumns: ['token'], + }, + { table: 'identity_workos_users', columns: ['workos_user_id'] }, + { table: 'learner_progress', columns: ['workos_user_id'] }, + { + table: 'member_portraits', + columns: ['user_id'], + redactBinaryColumns: ['portrait_data'], + }, + { table: 'organization_credential_grants', columns: ['workos_user_id'] }, + { table: 'teaching_checkpoints', columns: ['workos_user_id'] }, + { + table: 'user_avatar_uploads', + columns: ['workos_user_id'], + redactBinaryColumns: ['image_data'], + }, + { table: 'user_badges', columns: ['workos_user_id'] }, + { table: 'user_credentials', columns: ['workos_user_id'] }, + { table: 'user_email_aliases', columns: ['workos_user_id'] }, +] as const; + +export const USER_SET_NULL_SNAPSHOT_INVENTORY = [ + { table: 'flagged_conversations', columns: ['reviewed_by'] }, + { table: 'known_media_contacts', columns: ['added_by'] }, +] as const; + +type UserSnapshotSpec = { + readonly table: string; + readonly columns: readonly string[]; + /** Preserve row metadata and payload length without retaining raw binary PII. */ + readonly redactBinaryColumns?: readonly string[]; + /** Preserve row provenance without retaining bearer-like secret values. */ + readonly redactColumns?: readonly string[]; +}; + +function assertSnapshotCollectionBound( + collection: string, + rows: readonly unknown[], +): void { + if (rows.length > IDENTITY_DELETION_SNAPSHOT_MAX_ROWS_PER_COLLECTION) { + throw new Error( + `Identity deletion snapshot ${collection} exceeds the fail-closed row limit`, + ); + } +} + +function quoteSnapshotIdentifier(identifier: string): string { + if (!/^[a-z_][a-z0-9_]*$/.test(identifier)) { + throw new Error(`Unsafe deletion snapshot identifier: ${identifier}`); + } + return `"${identifier}"`; +} + +async function snapshotAndLockUserChildren( + client: PoolClient, + specs: readonly UserSnapshotSpec[], + affectedUserIds: string[], +): Promise[]>> { + const snapshot: Record[]> = {}; + for (const spec of specs) { + const table = quoteSnapshotIdentifier(spec.table); + const binaryColumns = spec.redactBinaryColumns ?? []; + const binaryRedactedExpression = binaryColumns.reduce((expression, column) => { + const quotedColumn = quoteSnapshotIdentifier(column); + return `(${expression} - '${column}') || jsonb_build_object( + '${column}_redacted', child.${quotedColumn} IS NOT NULL, + '${column}_byte_length', octet_length(child.${quotedColumn}) + )`; + }, 'to_jsonb(child)'); + const jsonExpression = (spec.redactColumns ?? []).reduce((expression, column) => { + const quotedColumn = quoteSnapshotIdentifier(column); + return `(${expression} - '${column}') || jsonb_build_object( + '${column}_redacted', child.${quotedColumn} IS NOT NULL + )`; + }, binaryRedactedExpression); + const predicates = spec.columns + .map((column) => `child.${quoteSnapshotIdentifier(column)} = ANY($1)`) + .join(' OR '); + const result = await client.query( + `SELECT ${jsonExpression} AS row + FROM ${table} child + WHERE ${predicates} + ORDER BY to_jsonb(child)::text + LIMIT $2 + FOR UPDATE OF child`, + [affectedUserIds, IDENTITY_DELETION_SNAPSHOT_MAX_ROWS_PER_COLLECTION + 1], + ); + assertSnapshotCollectionBound(spec.table, result.rows); + snapshot[spec.table] = rowsAsJson(result); + } + return snapshot; +} + +export const CREDENTIAL_MUTATION_LOCK_TIMEOUT_MS = 1_000; +export const CREDENTIAL_MUTATION_MAX_ATTEMPTS = 3; +export const CREDENTIAL_MUTATION_ADMISSION_TIMEOUT_MS = 1_000; + +/** + * Keep one connection free for confirmed deletion and bound admission wait. + * Guard callbacks must use their supplied client and must not perform provider + * I/O, so each admitted mutation consumes exactly one pool connection. + */ +const credentialMutationAdmission = { + active: 0, + waiters: [] as Array<(release: () => void) => void>, +}; + +async function acquireCredentialMutationPoolCapacity(): Promise<() => void> { + const configuredMax = getPool().options.max; + if (configuredMax < 4) { + throw new Error('Credential event mutation requires a database pool with at least four connections'); + } + const limit = Math.max(1, configuredMax - 1); + + const makeRelease = (): (() => void) => { + let released = false; + return () => { + if (released) return; + released = true; + const next = credentialMutationAdmission.waiters.shift(); + if (next) { + // Transfer this occupied admission directly to the oldest waiter so + // a newly arriving mutation cannot steal capacity between turns. + next(makeRelease()); + } else { + credentialMutationAdmission.active--; + } + }; + }; + + if (credentialMutationAdmission.active < limit) { + credentialMutationAdmission.active++; + return makeRelease(); + } + return new Promise((resolve, reject) => { + const waiter = (release: () => void) => { + clearTimeout(timeout); + resolve(release); + }; + const timeout = setTimeout(() => { + const index = credentialMutationAdmission.waiters.indexOf(waiter); + if (index >= 0) credentialMutationAdmission.waiters.splice(index, 1); + reject(new Error('Credential event mutation admission timed out')); + }, CREDENTIAL_MUTATION_ADMISSION_TIMEOUT_MS); + credentialMutationAdmission.waiters.push(waiter); + }); +} + +function rowsAsJson(result: { rows: JsonRow[] }): Record[] { + return result.rows.map(({ row }) => row); +} + +async function lockCredentialMutations( + client: PoolClient, + workosUserIds: string[], +): Promise { + for (const workosUserId of [...new Set(workosUserIds)].sort()) { + const lock = await client.query(`SELECT pg_advisory_xact_lock(hashtextextended($1, 6827))`, [workosUserId]); + if (lock.command !== 'SELECT' || lock.rowCount !== 1 || lock.rows.length !== 1) { + throw new Error('Credential lifecycle lock was not confirmed'); + } + } +} + +export interface LockedCredentialLifecycle { + workos_user_id: string; + identity_id: string | null; + is_primary: boolean | null; + user_exists: boolean; + binding_exists: boolean; + primary_count: string; + terminal_marker: boolean; +} + +/** + * Inspect lifecycle under the deletion protocol's credential/binding/identity + * locks. The caller MUST already own a transaction on this supplied client and + * retain it through its local mutations. No transaction, epoch, audit, actor + * authorization, provider call or retry is performed on the caller's behalf. + * An unseen credential is distinct from a deleted or quarantined credential. + */ +export async function lockCredentialLifecyclesInTransaction( + client: PoolClient, + workosUserIds: string[], +): Promise { + const ids = [...new Set(workosUserIds)].sort(); + if (!ids.length || ids.some(id => !id)) throw new Error('Credential lifecycle IDs required'); + await lockCredentialMutations(client, ids); + const bindings = await client.query<{ workos_user_id: string; identity_id: string }>( + `SELECT workos_user_id, identity_id FROM identity_workos_users + WHERE workos_user_id = ANY($1) ORDER BY workos_user_id FOR UPDATE`, [ids], + ); + const byCredential = new Map(bindings.rows.map(row => [row.workos_user_id, row.identity_id])); + if (bindings.rowCount !== bindings.rows.length || byCredential.size !== bindings.rows.length + || bindings.rows.some(row => !ids.includes(row.workos_user_id) || typeof row.identity_id !== 'string')) { + throw new Error('Credential binding locks were not confirmed'); + } + const identityIds = [...new Set(bindings.rows.map(row => row.identity_id))].sort(); + if (identityIds.length) { + const identities = await client.query<{ id: string }>( + 'SELECT id FROM identities WHERE id = ANY($1) ORDER BY id FOR UPDATE', [identityIds]); + const lockedIds = identities.rows.map(row => row.id).sort(); + if (identities.rowCount !== identityIds.length || lockedIds.length !== identityIds.length + || lockedIds.some((id, index) => id !== identityIds[index])) { + throw new Error('Credential identity locks were not confirmed'); + } + } + const states = await client.query( + `SELECT requested.workos_user_id, iwu.identity_id, iwu.is_primary, + (u.workos_user_id IS NOT NULL) AS user_exists, + (iwu.workos_user_id IS NOT NULL) AS binding_exists, + (SELECT COUNT(*) FROM identity_workos_users primary_iwu + WHERE primary_iwu.identity_id = iwu.identity_id AND primary_iwu.is_primary)::text AS primary_count, + EXISTS (SELECT 1 FROM registry_audit_log ral + WHERE ral.workos_user_id = requested.workos_user_id + AND ral.action IN ('identity_credential_deleted', 'identity_primary_deletion_quarantined', + 'identity_credential_admin_compensation_deleted', + 'identity_credential_admin_compensation_quarantined')) AS terminal_marker + FROM unnest($1::text[]) AS requested(workos_user_id) + LEFT JOIN users u USING (workos_user_id) + LEFT JOIN identity_workos_users iwu USING (workos_user_id) + ORDER BY requested.workos_user_id`, [ids], + ); + // Database collations may order mixed-case WorkOS IDs differently from JS. + const rows = [...states.rows].sort((a, b) => + a.workos_user_id < b.workos_user_id ? -1 : a.workos_user_id > b.workos_user_id ? 1 : 0); + if (states.rowCount !== ids.length || rows.length !== ids.length + || rows.some((row, index) => row.workos_user_id !== ids[index] + || typeof row.user_exists !== 'boolean' || typeof row.binding_exists !== 'boolean' + || typeof row.terminal_marker !== 'boolean' + || row.binding_exists !== byCredential.has(row.workos_user_id) + || row.identity_id !== (byCredential.get(row.workos_user_id) ?? null) + || (row.identity_id !== null && typeof row.identity_id !== 'string') + || (row.is_primary !== null && typeof row.is_primary !== 'boolean') + || typeof row.primary_count !== 'string' + || !Number.isSafeInteger(Number(row.primary_count)) || Number(row.primary_count) < 0)) { + throw new Error('Credential lifecycle snapshot was not confirmed'); + } + return rows; +} + +interface AdminCredentialCompensationActor { + workosUserId: string; + operationId: string; + actorUserId: string; + actorIdentityId: string; +} + +/** Durable admission quarantine, not a claim that WorkOS deleted the user. */ +export async function recordAdminCredentialCompensationQuarantine( + client: PoolClient, + input: AdminCredentialCompensationActor, +): Promise { + const [state] = await lockCredentialLifecyclesInTransaction(client, [input.workosUserId]); + if (state.terminal_marker || state.user_exists || state.binding_exists + || state.identity_id !== null || state.is_primary !== null || Number(state.primary_count) !== 0) { + throw new Error('Admin compensation no longer owns an unseen credential'); + } + const audit = await client.query<{ id: string }>( + `INSERT INTO registry_audit_log + (workos_organization_id, workos_user_id, action, resource_type, resource_id, details) + VALUES ('system', $1::text, 'identity_credential_admin_compensation_quarantined', + 'admin_credential_bind_operation', $2::text, + jsonb_build_object('operation_id', $2::text, 'provider_user_id', $1::text, + 'acting_workos_user_id', $3::text, 'actor_identity_id', $4::text, + 'quarantine_reason', 'admin_credential_compensation_pending', 'provider_delete_confirmed', false)) + RETURNING id`, + [input.workosUserId, input.operationId, input.actorUserId, input.actorIdentityId], + ); + if (audit.rowCount !== 1 || audit.rows.length !== 1 || !audit.rows[0]?.id) { + throw new Error('Admin compensation quarantine was not recorded'); + } +} + +/** Reacquire the lifecycle fence and permit only this exact operation's quarantine. */ +export async function lockAdminCredentialCompensationQuarantine( + client: PoolClient, + input: AdminCredentialCompensationActor, +): Promise { + const [state] = await lockCredentialLifecyclesInTransaction(client, [input.workosUserId]); + if (!state.terminal_marker || state.user_exists || state.binding_exists + || state.identity_id !== null || state.is_primary !== null || Number(state.primary_count) !== 0) { + throw new Error('Admin compensation quarantine has contradictory local state'); + } + const markers = await client.query<{ + action: string; resource_type: string; resource_id: string; details: Record; + }>( + `SELECT action, resource_type, resource_id, details FROM registry_audit_log + WHERE workos_user_id = $1 + AND action IN ('identity_credential_deleted', 'identity_primary_deletion_quarantined', + 'identity_credential_admin_compensation_deleted', 'identity_credential_admin_compensation_quarantined')`, + [input.workosUserId], + ); + const marker = markers.rows[0]; + if (markers.rowCount !== 1 || markers.rows.length !== 1 + || marker?.action !== 'identity_credential_admin_compensation_quarantined' + || marker.resource_type !== 'admin_credential_bind_operation' || marker.resource_id !== input.operationId + || marker.details?.operation_id !== input.operationId || marker.details?.provider_user_id !== input.workosUserId + || marker.details?.acting_workos_user_id !== input.actorUserId || marker.details?.actor_identity_id !== input.actorIdentityId + || marker.details?.quarantine_reason !== 'admin_credential_compensation_pending' + || marker.details?.provider_delete_confirmed !== false) { + throw new Error('Admin compensation quarantine ownership was not confirmed'); + } +} + +/** + * Admin-specific evidence for a confirmed WorkOS compensation delete. The + * caller must have received deleteUser success and must retain its supplied + * transaction through this marker, the operation audit and COMMIT. This is + * not a signed webhook or a provider-deletion before-graph snapshot: its exact + * admin credential and operation remain explicit. Failed/ambiguous provider + * deletes must never call this helper. + */ +export async function recordConfirmedAdminCredentialCompensationDeletion( + client: PoolClient, + input: AdminCredentialCompensationActor, +): Promise { + await lockAdminCredentialCompensationQuarantine(client, input); + const audit = await client.query<{ id: string }>( + `INSERT INTO registry_audit_log + (workos_organization_id, workos_user_id, action, resource_type, resource_id, details) + VALUES ('system', $1::text, 'identity_credential_admin_compensation_deleted', + 'admin_credential_bind_operation', $2::text, + jsonb_build_object('operation_id', $2::text, 'deleted_workos_user_id', $1::text, + 'acting_workos_user_id', $3::text, 'actor_identity_id', $4::text, + 'deletion_source', 'admin_credential_compensation', 'provider_delete_confirmed', true)) + RETURNING id`, + [input.workosUserId, input.operationId, input.actorUserId, input.actorIdentityId], + ); + if (audit.rowCount !== 1 || audit.rows.length !== 1 || !audit.rows[0]?.id) { + throw new Error('Admin compensation deletion marker was not recorded'); + } +} + +async function lockCredentialThenIdentityMutation( + client: PoolClient, + workosUserId: string, +): Promise<{ identityId: string | undefined; identity: { rows: JsonRow[] } }> { + await lockCredentialMutations(client, [workosUserId]); + const identityLookup = await client.query<{ identity_id: string }>( + `SELECT identity_id FROM identity_workos_users + WHERE workos_user_id = $1 + FOR UPDATE`, + [workosUserId], + ); + const identityId = identityLookup.rows[0]?.identity_id; + const identity = identityId + ? await client.query( + `SELECT to_jsonb(i) AS row FROM identities i WHERE id = $1 FOR UPDATE OF i`, + [identityId], + ) + : { rows: [] as JsonRow[] }; + return { identityId, identity }; +} + +async function hasTerminalCredentialLifecycleMarker( + client: PoolClient, + workosUserId: string, +): Promise { + const marker = await client.query( + `SELECT 1 FROM registry_audit_log + WHERE workos_user_id = $1 + AND action IN ('identity_credential_deleted', 'identity_primary_deletion_quarantined', + 'identity_credential_admin_compensation_deleted', 'identity_credential_admin_compensation_quarantined') + LIMIT 1`, + [workosUserId], + ); + return marker.rowCount !== null && marker.rowCount > 0; +} + +type ConfirmedDeletionAuditAction = + | 'identity_credential_deleted' + | 'identity_primary_deletion_quarantined'; + +async function getConfirmedDeletionReplay( + client: PoolClient, + workosUserId: string, + requestedSource: IdentityCredentialDeletionSource, +): Promise { + const audits = await client.query<{ + id: string; + action: ConfirmedDeletionAuditAction; + details: Record; + }>( + `SELECT id, action, details + FROM registry_audit_log + WHERE workos_user_id = $1 + AND action IN ('identity_credential_deleted', 'identity_primary_deletion_quarantined') + ORDER BY created_at ASC, id ASC`, + [workosUserId], + ); + if (audits.rows.length === 0) return null; + + const prior = audits.rows[0]; + const details = prior.details; + const originalSource = details?.deletion_source; + const beforeGraph = details?.before_graph; + if ((originalSource !== 'workos_webhook' && originalSource !== 'sync_users_backfill') + || details?.deleted_workos_user_id !== workosUserId + || typeof beforeGraph !== 'object' + || beforeGraph === null + || Array.isArray(beforeGraph)) { + throw new Error('Confirmed credential deletion replay audit is malformed'); + } + + const graph = beforeGraph as unknown as IdentityBeforeGraph; + if (!Array.isArray(graph.identity_workos_users) + || !Array.isArray(graph.slack_user_mappings)) { + throw new Error('Confirmed credential deletion replay before graph is malformed'); + } + const affectedUserIds = [...new Set([ + workosUserId, + ...graph.identity_workos_users.map((row) => String(row.workos_user_id)), + ])].sort(); + const affectedSlackUserIds = graph.slack_user_mappings + .filter((row) => row.workos_user_id === workosUserId) + .map((row) => String(row.slack_user_id)); + + let quarantine: IdentityRecoveryQuarantine | null = null; + if (prior.action === 'identity_primary_deletion_quarantined') { + if (details.recovery_state !== IDENTITY_RECOVERY_STATE + || typeof details.identity_id !== 'string' + || !Array.isArray(details.aao_admin_revocations)) { + throw new Error('Confirmed credential deletion replay quarantine is malformed'); + } + quarantine = { + audit_id: prior.id, + identity_id: details.identity_id, + deleted_workos_user_id: workosUserId, + deletion_source: originalSource, + actor: details.actor as IdentityRecoveryQuarantine['actor'], + recovery_state: IDENTITY_RECOVERY_STATE, + snapshot_policy: IDENTITY_DELETION_SNAPSHOT_POLICY, + before_graph: graph, + aao_admin_revocations: + details.aao_admin_revocations as AAOAdminParallelRevocationEvidence[], + }; + } + + return { + // `deleted` means this invocation changed the live row. Replays return the + // original durable outcome below without incrementing caller counters. + deleted: false, + affectedUserIds, + affectedSlackUserIds, + quarantine, + replay: { + audit_id: prior.id, + kind: originalSource === requestedSource ? 'same_source' : 'different_source', + original_deletion_source: originalSource, + requested_deletion_source: requestedSource, + }, + }; +} + +function isRetryableCredentialLockError(error: unknown): boolean { + const code = typeof error === 'object' && error !== null && 'code' in error + ? String((error as { code?: unknown }).code) + : ''; + return code === '55P03' || code === '40P01'; +} + +export interface CredentialEventMutationResult { + applied: boolean; + value?: T; +} + +type CredentialMutationMode = 'active' | 'provider_upsert'; + +/** + * Run every authority-bearing side effect of one provider event while the + * same transaction-scoped credential lock used by deletion remains held. + * Tombstones are checked after locking. Deletion uses the same lock through + * its tombstone commit, so the marker cannot appear before this guard releases. + * Lock timeout/deadlock failures retry a bounded number of times, then throw + * so the signed webhook returns 500 and WorkOS can retry; no unlocked fallback + * is permitted. Once the callback starts it is never retried here: every + * local write must use the supplied client and a callback failure rolls the + * transaction back, but replaying arbitrary callback code is still unsafe. + */ +async function withCredentialEventMutation( + workosUserId: string, + mode: CredentialMutationMode, + mutation: (client: PoolClient) => Promise, +): Promise> { + const releaseCapacity = await acquireCredentialMutationPoolCapacity(); + try { + for (let attempt = 1; attempt <= CREDENTIAL_MUTATION_MAX_ATTEMPTS; attempt++) { + const client = await getPool().connect(); + let mutationStarted = false; + try { + await client.query('BEGIN'); + await client.query(`SET LOCAL lock_timeout = '${CREDENTIAL_MUTATION_LOCK_TIMEOUT_MS}ms'`); + const { identityId } = await lockCredentialThenIdentityMutation(client, workosUserId); + if (await hasTerminalCredentialLifecycleMarker(client, workosUserId)) { + await client.query('COMMIT'); + return { applied: false }; + } + const lifecycle = await client.query<{ + user_exists: boolean; + binding_exists: boolean; + primary_count: string; + }>( + `SELECT EXISTS ( + SELECT 1 FROM users WHERE workos_user_id = $1 + ) AS user_exists, + EXISTS ( + SELECT 1 FROM identity_workos_users WHERE workos_user_id = $1 + ) AS binding_exists, + COALESCE(( + SELECT COUNT(*) + FROM identity_workos_users primary_iwu + WHERE primary_iwu.identity_id = $2 + AND primary_iwu.is_primary = TRUE + ), 0)::text AS primary_count`, + [workosUserId, identityId ?? null], + ); + const state = lifecycle.rows[0]; + const isActive = state?.user_exists === true + && state.binding_exists === true + && Number(state.primary_count) === 1; + const isUnseen = state?.user_exists === false && state.binding_exists === false; + if ((mode === 'active' && !isActive) + || (mode === 'provider_upsert' && !isActive && !isUnseen)) { + await client.query('COMMIT'); + return { applied: false }; + } + + mutationStarted = true; + const value = await mutation(client); + const bumped = await bumpAuthorizationEpochs(client, [workosUserId]); + if (bumped.length !== 1 || bumped[0] !== workosUserId) { + throw new Error('Credential event mutation did not bump the live authorization epoch'); + } + await client.query('COMMIT'); + return { applied: true, value }; + } catch (error) { + await client.query('ROLLBACK').catch(() => undefined); + if (!mutationStarted + && attempt < CREDENTIAL_MUTATION_MAX_ATTEMPTS + && isRetryableCredentialLockError(error)) { + await new Promise((resolve) => setTimeout(resolve, attempt * 25)); + continue; + } + throw error; + } finally { + client.release(); + } + } + throw new Error('Credential event mutation retry bound exhausted'); + } finally { + releaseCapacity(); + } +} + +export async function withActiveCredentialEventMutation( + workosUserId: string, + mutation: (client: PoolClient) => Promise, +): Promise> { + return withCredentialEventMutation(workosUserId, 'active', mutation); +} + +/** Apply one atomic authority mutation to a sorted set of live credentials. */ +export async function withActiveCredentialSetMutation( + workosUserIds: string[], + mutation: (client: PoolClient) => Promise, +): Promise> { + const ids = [...new Set(workosUserIds.filter(Boolean))].sort(); + if (ids.length === 0) return { applied: false }; + const releaseCapacity = await acquireCredentialMutationPoolCapacity(); + try { + for (let attempt = 1; attempt <= CREDENTIAL_MUTATION_MAX_ATTEMPTS; attempt++) { + const client = await getPool().connect(); + let mutationStarted = false; + try { + await client.query('BEGIN'); + await client.query(`SET LOCAL lock_timeout = '${CREDENTIAL_MUTATION_LOCK_TIMEOUT_MS}ms'`); + await lockCredentialMutations(client, ids); + const bindings = await client.query<{ workos_user_id: string; identity_id: string }>( + `SELECT workos_user_id, identity_id + FROM identity_workos_users + WHERE workos_user_id = ANY($1) + ORDER BY workos_user_id + FOR UPDATE`, + [ids], + ); + const identityIds = [...new Set(bindings.rows.map((row) => row.identity_id))].sort(); + if (identityIds.length > 0) { + await client.query( + `SELECT id FROM identities + WHERE id = ANY($1) + ORDER BY id + FOR UPDATE`, + [identityIds], + ); + } + const lifecycle = await client.query<{ + workos_user_id: string; + user_exists: boolean; + binding_exists: boolean; + primary_count: string; + terminal_marker: boolean; + }>( + `SELECT requested.workos_user_id, + (u.workos_user_id IS NOT NULL) AS user_exists, + (iwu.workos_user_id IS NOT NULL) AS binding_exists, + COALESCE(( + SELECT COUNT(*) FROM identity_workos_users primary_iwu + WHERE primary_iwu.identity_id = iwu.identity_id + AND primary_iwu.is_primary = TRUE + ), 0)::text AS primary_count, + EXISTS ( + SELECT 1 FROM registry_audit_log ral + WHERE ral.workos_user_id = requested.workos_user_id + AND ral.action IN ( + 'identity_credential_deleted', + 'identity_primary_deletion_quarantined', + 'identity_credential_admin_compensation_deleted', + 'identity_credential_admin_compensation_quarantined' + ) + ) AS terminal_marker + FROM unnest($1::text[]) AS requested(workos_user_id) + LEFT JOIN users u USING (workos_user_id) + LEFT JOIN identity_workos_users iwu USING (workos_user_id) + ORDER BY requested.workos_user_id`, + [ids], + ); + if (lifecycle.rows.length !== ids.length + || lifecycle.rows.some((row) => !row.user_exists + || !row.binding_exists + || row.terminal_marker + || Number(row.primary_count) !== 1)) { + await client.query('COMMIT'); + return { applied: false }; + } + + mutationStarted = true; + const value = await mutation(client); + const bumped = (await bumpAuthorizationEpochs(client, ids)).sort(); + if (bumped.length !== ids.length || bumped.some((id, index) => id !== ids[index])) { + throw new Error('Authority mutation did not bump every live credential epoch'); + } + await client.query('COMMIT'); + return { applied: true, value }; + } catch (error) { + await client.query('ROLLBACK').catch(() => undefined); + if (!mutationStarted + && attempt < CREDENTIAL_MUTATION_MAX_ATTEMPTS + && isRetryableCredentialLockError(error)) { + await new Promise((resolve) => setTimeout(resolve, attempt * 25)); + continue; + } + throw error; + } finally { + client.release(); + } + } + throw new Error('Credential authority mutation retry bound exhausted'); + } finally { + releaseCapacity(); + } +} + +export async function withCredentialCreationEventMutation( + workosUserId: string, + mutation: (client: PoolClient) => Promise, +): Promise> { + return withCredentialEventMutation(workosUserId, 'provider_upsert', mutation); +} + +export async function upsertWorkosUserInCredentialEvent( + credentialLockClient: PoolClient, + user: WorkosUserUpsert, + nameConflictPolicy: 'provider_authoritative' | 'preserve_existing', +): Promise { + await credentialLockClient.query( + `INSERT INTO users ( + workos_user_id, email, first_name, last_name, + email_verified, workos_created_at, workos_updated_at, + created_at, updated_at + ) VALUES ($1, $2, $3, $4, $5, $6, $7, NOW(), NOW()) + ON CONFLICT (workos_user_id) DO UPDATE SET + email = EXCLUDED.email, + first_name = CASE WHEN $8::text = 'preserve_existing' + THEN COALESCE(NULLIF(TRIM(users.first_name), ''), EXCLUDED.first_name) + ELSE COALESCE(NULLIF(TRIM(EXCLUDED.first_name), ''), users.first_name) + END, + last_name = CASE WHEN $8::text = 'preserve_existing' + THEN COALESCE(NULLIF(TRIM(users.last_name), ''), EXCLUDED.last_name) + ELSE COALESCE(NULLIF(TRIM(EXCLUDED.last_name), ''), users.last_name) + END, + email_verified = EXCLUDED.email_verified, + workos_updated_at = EXCLUDED.workos_updated_at, + updated_at = NOW()`, + [ + user.id, + user.email, + user.firstName, + user.lastName, + user.emailVerified, + user.createdAt, + user.updatedAt, + nameConflictPolicy, + ], + ); +} + +/** + * Serialize provider upserts with confirmed deletion and refuse resurrection + * after a durable provider-deletion audit exists for the credential. + */ +export async function upsertWorkosUserUnlessConfirmedDeleted( + user: WorkosUserUpsert, +): Promise { + const result = await withCredentialCreationEventMutation(user.id, async (client) => { + await upsertWorkosUserInCredentialEvent(client, user, 'provider_authoritative'); + }); + return result.applied; +} + +/** Retrieve the durable, unresolved recovery state for an identity. */ +export async function getIdentityRecoveryQuarantine( + identityId: string, +): Promise { + const result = await getPool().query<{ id: string; details: IdentityRecoveryQuarantine }>( + `SELECT id, details + FROM registry_audit_log + WHERE action = 'identity_primary_deletion_quarantined' + AND resource_type = 'identity_recovery' + AND resource_id = $1 + AND details->>'recovery_state' = $2 + ORDER BY created_at DESC, id DESC + LIMIT 1`, + [identityId, IDENTITY_RECOVERY_STATE], + ); + if (result.rows.length === 0) return null; + return { + ...result.rows[0].details, + audit_id: result.rows[0].id, + }; +} + +/** + * Apply an authoritative provider deletion without selecting a new primary. + * Revoke cached routing for every affected credential in the same transaction + * as the binding disappears. Return the IDs to evict from local caches after + * commit; their persisted epochs handle caches on other instances. + */ +export async function deleteIdentityCredentialTransaction( + workosUserId: string, + deletionSource: IdentityCredentialDeletionSource, +): Promise { + const client = await getPool().connect(); + let result: IdentityCredentialDeletionResult | undefined; + + try { + await client.query('BEGIN'); + await client.query(`SET LOCAL lock_timeout = '${CREDENTIAL_MUTATION_LOCK_TIMEOUT_MS}ms'`); + const discoveredBinding = await client.query<{ identity_id: string }>( + `SELECT identity_id FROM identity_workos_users WHERE workos_user_id = $1`, + [workosUserId], + ); + const discoveredIdentityId = discoveredBinding.rows[0]?.identity_id; + const discoveredSiblings = discoveredIdentityId + ? await client.query<{ workos_user_id: string }>( + `SELECT workos_user_id FROM identity_workos_users + WHERE identity_id = $1 + ORDER BY workos_user_id + LIMIT $2`, + [discoveredIdentityId, IDENTITY_DELETION_SNAPSHOT_MAX_ROWS_PER_COLLECTION + 1], + ) + : { rows: [] as Array<{ workos_user_id: string }> }; + assertSnapshotCollectionBound('discovered_identity_workos_users', discoveredSiblings.rows); + const discoveredUserIds = [...new Set([ + workosUserId, + ...discoveredSiblings.rows.map((row) => row.workos_user_id), + ])].sort(); + await lockCredentialMutations(client, discoveredUserIds); + const lockedBinding = await client.query<{ identity_id: string }>( + `SELECT identity_id FROM identity_workos_users + WHERE workos_user_id = $1 + FOR UPDATE`, + [workosUserId], + ); + const identityId = lockedBinding.rows[0]?.identity_id; + const replay = await getConfirmedDeletionReplay(client, workosUserId, deletionSource); + if (replay) { + if (identityId) { + throw new Error('Confirmed credential deletion audit conflicts with a live identity binding'); + } + await client.query('COMMIT'); + return replay; + } + if (identityId !== discoveredIdentityId) { + throw new Error('Credential identity changed while acquiring deletion locks'); + } + const identity = identityId + ? await client.query( + `SELECT to_jsonb(i) AS row FROM identities i WHERE id = $1 FOR UPDATE OF i`, + [identityId], + ) + : { rows: [] as JsonRow[] }; + const bound = identityId + ? await client.query( + `SELECT to_jsonb(iwu) AS row + FROM identity_workos_users iwu + WHERE iwu.identity_id = $1 + ORDER BY iwu.workos_user_id + LIMIT $2 + FOR UPDATE OF iwu`, + [identityId, IDENTITY_DELETION_SNAPSHOT_MAX_ROWS_PER_COLLECTION + 1], + ) + : { rows: [] as JsonRow[] }; + assertSnapshotCollectionBound('identity_workos_users', bound.rows); + const bindingRows = rowsAsJson(bound); + const targetBinding = bindingRows.find((row) => row.workos_user_id === workosUserId); + if (identityId && !targetBinding) { + // Another confirmed deletion won the identity lock. Do not mutate epochs + // or emit a second audit on the replaying caller. + await client.query('COMMIT'); + return { + deleted: false, + affectedUserIds: [workosUserId], + affectedSlackUserIds: [], + quarantine: null, + replay: null, + }; + } + const affectedUserIds = [...new Set([ + workosUserId, + ...bindingRows.map((row) => String(row.workos_user_id)), + ])].sort(); + if (affectedUserIds.length !== discoveredUserIds.length + || affectedUserIds.some((id, index) => id !== discoveredUserIds[index])) { + throw new Error('Credential sibling graph changed while acquiring deletion locks'); + } + + // Lock and capture the complete authority-bearing before graph in stable + // orders. The graph becomes the durable recovery evidence if this is the + // primary-deletion fault path. + const users = await client.query( + `SELECT to_jsonb(u) AS row FROM users u + WHERE u.workos_user_id = ANY($1) + ORDER BY u.workos_user_id + LIMIT $2 + FOR UPDATE OF u`, + [affectedUserIds, IDENTITY_DELETION_SNAPSHOT_MAX_ROWS_PER_COLLECTION + 1], + ); + assertSnapshotCollectionBound('users', users.rows); + const organizationMemberships = await client.query( + `SELECT to_jsonb(om) AS row FROM organization_memberships om + WHERE om.workos_user_id = ANY($1) + ORDER BY om.workos_user_id, om.workos_organization_id + LIMIT $2 + FOR UPDATE OF om`, + [affectedUserIds, IDENTITY_DELETION_SNAPSHOT_MAX_ROWS_PER_COLLECTION + 1], + ); + assertSnapshotCollectionBound('organization_memberships', organizationMemberships.rows); + const slackUserMappings = await client.query( + `SELECT to_jsonb(sm) AS row FROM slack_user_mappings sm + WHERE sm.workos_user_id = ANY($1) + ORDER BY sm.workos_user_id, sm.slack_user_id + LIMIT $2 + FOR UPDATE OF sm`, + [affectedUserIds, IDENTITY_DELETION_SNAPSHOT_MAX_ROWS_PER_COLLECTION + 1], + ); + assertSnapshotCollectionBound('slack_user_mappings', slackUserMappings.rows); + const targetSlackUserIds = rowsAsJson(slackUserMappings) + .filter((row) => row.workos_user_id === workosUserId) + .map((row) => String(row.slack_user_id)); + const targetAuthorityPrincipalIds = [...new Set([ + workosUserId, + ...targetSlackUserIds, + ])].sort(); + const snapshotAuthorityPrincipalIds = [...new Set([ + ...affectedUserIds, + ...targetSlackUserIds, + ])].sort(); + const workingGroupMemberships = await client.query( + `SELECT to_jsonb(wgm) AS row FROM working_group_memberships wgm + WHERE wgm.workos_user_id = ANY($1) + ORDER BY wgm.workos_user_id, wgm.working_group_id + LIMIT $2 + FOR UPDATE OF wgm`, + [ + snapshotAuthorityPrincipalIds, + IDENTITY_DELETION_SNAPSHOT_MAX_ROWS_PER_COLLECTION + 1, + ], + ); + assertSnapshotCollectionBound('working_group_memberships', workingGroupMemberships.rows); + const workingGroupLeaders = await client.query( + `SELECT to_jsonb(wgl) AS row FROM working_group_leaders wgl + WHERE wgl.user_id = ANY($1) + ORDER BY wgl.user_id, wgl.working_group_id + LIMIT $2 + FOR UPDATE OF wgl`, + [ + snapshotAuthorityPrincipalIds, + IDENTITY_DELETION_SNAPSHOT_MAX_ROWS_PER_COLLECTION + 1, + ], + ); + assertSnapshotCollectionBound('working_group_leaders', workingGroupLeaders.rows); + const workingGroupTopicSubscriptions = await client.query( + `SELECT to_jsonb(wgts) AS row FROM working_group_topic_subscriptions wgts + WHERE wgts.workos_user_id = ANY($1) + ORDER BY wgts.workos_user_id, wgts.working_group_id + LIMIT $2 + FOR UPDATE OF wgts`, + [ + snapshotAuthorityPrincipalIds, + IDENTITY_DELETION_SNAPSHOT_MAX_ROWS_PER_COLLECTION + 1, + ], + ); + assertSnapshotCollectionBound( + 'working_group_topic_subscriptions', + workingGroupTopicSubscriptions.rows, + ); + const authorizationEpochs = await client.query( + `SELECT to_jsonb(ae) AS row FROM authorization_epochs ae + WHERE ae.workos_user_id = ANY($1) + ORDER BY ae.workos_user_id + LIMIT $2 + FOR UPDATE OF ae`, + [affectedUserIds, IDENTITY_DELETION_SNAPSHOT_MAX_ROWS_PER_COLLECTION + 1], + ); + assertSnapshotCollectionBound('authorization_epochs', authorizationEpochs.rows); + const userCascadeRows = await snapshotAndLockUserChildren( + client, + USER_CASCADE_SNAPSHOT_INVENTORY.filter( + ({ table }) => table !== 'identity_workos_users' && table !== 'authorization_epochs', + ), + affectedUserIds, + ); + userCascadeRows.identity_workos_users = bindingRows; + userCascadeRows.authorization_epochs = rowsAsJson(authorizationEpochs); + const userSetNullAttributions = await snapshotAndLockUserChildren( + client, + USER_SET_NULL_SNAPSHOT_INVENTORY, + affectedUserIds, + ); + + const userRows = rowsAsJson(users); + const targetUser = userRows.find((row) => row.workos_user_id === workosUserId); + const survivingBindings = bindingRows.filter((row) => row.workos_user_id !== workosUserId); + const needsQuarantine = identityId !== undefined + && survivingBindings.length > 0 + && survivingBindings.every((row) => row.is_primary !== true); + let quarantine: IdentityRecoveryQuarantine | null = null; + const beforeGraph: IdentityBeforeGraph = { + identity: identity.rows[0]?.row ?? (identityId ? { id: identityId } : {}), + identity_workos_users: bindingRows, + users: userRows, + organization_memberships: rowsAsJson(organizationMemberships), + working_group_memberships: rowsAsJson(workingGroupMemberships), + working_group_leaders: rowsAsJson(workingGroupLeaders), + working_group_topic_subscriptions: rowsAsJson(workingGroupTopicSubscriptions), + slack_user_mappings: rowsAsJson(slackUserMappings), + authorization_epochs: rowsAsJson(authorizationEpochs), + user_cascade_rows: userCascadeRows, + user_set_null_attributions: userSetNullAttributions, + }; + const targetMembership = beforeGraph.organization_memberships.find( + (row) => row.workos_user_id === workosUserId, + ); + const auditOrganizationId = String( + targetUser?.primary_organization_id + ?? targetMembership?.workos_organization_id + ?? 'identity-recovery-unscoped', + ); + const actor = { + type: 'workos_provider' as const, + source: deletionSource, + workos_user_id: workosUserId, + }; + const aaoAdminGroup = await client.query<{ id: string }>( + `SELECT id FROM working_groups WHERE slug = 'aao-admin' LIMIT 1`, + ); + const aaoAdminGroupId = aaoAdminGroup.rows[0]?.id; + // Migration 578 cannot truthfully encode a provider-deletion actor in + // actor_authorization_mechanism. Record compatible immutable revocation + // evidence in the generic deletion audit instead of forging one of its + // human/admin mechanisms; the full membership row is in before_graph. + const aaoAdminRevocations: AAOAdminParallelRevocationEvidence[] = aaoAdminGroupId + ? beforeGraph.working_group_memberships + .filter((row) => targetAuthorityPrincipalIds.includes(String(row.workos_user_id)) + && row.working_group_id === aaoAdminGroupId + && row.status === 'active') + .map((row) => ({ + event_type: 'revoked' as const, + target_user_id: String(row.workos_user_id), + working_group_id: aaoAdminGroupId, + mechanism: 'aao_admin_working_group' as const, + actor, + evidence_ledger: 'registry_audit_log' as const, + dedicated_ledger_limitation: 'provider_deletion_actor_not_representable' as const, + })) + : []; + const baseDetails = { + identity_id: identityId ?? null, + deleted_workos_user_id: workosUserId, + deletion_source: deletionSource, + actor, + snapshot_policy: IDENTITY_DELETION_SNAPSHOT_POLICY, + before_graph: beforeGraph, + aao_admin_revocations: aaoAdminRevocations, + }; + const details = needsQuarantine && identityId + ? { ...baseDetails, identity_id: identityId, recovery_state: IDENTITY_RECOVERY_STATE } + : baseDetails; + const auditAction = needsQuarantine + ? 'identity_primary_deletion_quarantined' + : 'identity_credential_deleted'; + const auditResourceType = needsQuarantine ? 'identity_recovery' : 'identity_credential'; + const auditResourceId = identityId ?? workosUserId; + const serializedDetails = JSON.stringify(details); + if (Buffer.byteLength(serializedDetails, 'utf8') + > IDENTITY_DELETION_SNAPSHOT_MAX_DETAILS_BYTES) { + throw new Error('Identity deletion snapshot exceeds the fail-closed byte limit'); + } + const audit = await client.query<{ id: string }>( + `INSERT INTO registry_audit_log ( + workos_organization_id, workos_user_id, action, + resource_type, resource_id, details + ) VALUES ($1, $2, $3, $4, $5, $6::jsonb) + RETURNING id`, + [auditOrganizationId, workosUserId, auditAction, auditResourceType, auditResourceId, serializedDetails], + ); + if (audit.rowCount !== 1 || audit.rows.length !== 1 || !audit.rows[0]?.id) { + throw new Error('Confirmed identity credential deletion audit was not durably recorded'); + } + if (needsQuarantine && identityId && 'recovery_state' in details) { + quarantine = { audit_id: audit.rows[0].id, ...details }; + } + + // Authentication checks the actual credential's epoch, so deleting the + // primary's epoch alone would leave each surviving credential's cache valid. + const expectedEpochUserIds = userRows + .map((row) => String(row.workos_user_id)) + .sort(); + const bumpedEpochUserIds = (await bumpAuthorizationEpochs(client, expectedEpochUserIds)).sort(); + if (bumpedEpochUserIds.length !== expectedEpochUserIds.length + || bumpedEpochUserIds.some((id, index) => id !== expectedEpochUserIds[index])) { + throw new Error('Confirmed credential deletion did not bump every live authorization epoch'); + } + const deletedOrganizationMemberships = await client.query( + `DELETE FROM organization_memberships WHERE workos_user_id = $1 RETURNING workos_organization_id`, + [workosUserId], + ); + const expectedOrganizationMemberships = beforeGraph.organization_memberships + .filter((row) => row.workos_user_id === workosUserId).length; + if (deletedOrganizationMemberships.rowCount !== expectedOrganizationMemberships) { + throw new Error('Confirmed credential deletion did not revoke every organization membership'); + } + // Preserve membership history while revoking live group authority. + const deactivatedWorkingGroupMemberships = await client.query<{ working_group_id: string }>( + `UPDATE working_group_memberships + SET status = 'inactive', updated_at = NOW() + WHERE workos_user_id = ANY($1) AND status <> 'inactive' + RETURNING working_group_id`, + [targetAuthorityPrincipalIds], + ); + const expectedWorkingGroupDeactivations = beforeGraph.working_group_memberships + .filter((row) => targetAuthorityPrincipalIds.includes(String(row.workos_user_id)) + && row.status !== 'inactive').length; + if (deactivatedWorkingGroupMemberships.rowCount !== expectedWorkingGroupDeactivations) { + throw new Error('Confirmed credential deletion did not deactivate every working-group membership'); + } + const deactivatedAAOAdminRows = aaoAdminGroupId + ? deactivatedWorkingGroupMemberships.rows + .filter((row) => row.working_group_id === aaoAdminGroupId).length + : 0; + if (deactivatedAAOAdminRows !== aaoAdminRevocations.length) { + throw new Error('AAO-admin revocation evidence cardinality did not match live deactivation'); + } + // Leadership rows have no inactive state; the complete rows are retained + // in before_graph before live authority is removed. + const deletedLeaderships = await client.query( + `DELETE FROM working_group_leaders WHERE user_id = ANY($1) RETURNING working_group_id`, + [targetAuthorityPrincipalIds], + ); + const expectedLeaderships = beforeGraph.working_group_leaders + .filter((row) => targetAuthorityPrincipalIds.includes(String(row.user_id))).length; + if (deletedLeaderships.rowCount !== expectedLeaderships) { + throw new Error('Confirmed credential deletion did not revoke every working-group leadership'); + } + const deletedTopicSubscriptions = await client.query( + `DELETE FROM working_group_topic_subscriptions + WHERE workos_user_id = ANY($1) + RETURNING working_group_id`, + [targetAuthorityPrincipalIds], + ); + const expectedTopicSubscriptions = beforeGraph.working_group_topic_subscriptions + .filter((row) => targetAuthorityPrincipalIds.includes(String(row.workos_user_id))).length; + if (deletedTopicSubscriptions.rowCount !== expectedTopicSubscriptions) { + throw new Error('Confirmed credential deletion did not revoke every topic subscription'); + } + // Preserve the Slack contact row while severing its WorkOS authority link. + const unlinkedSlackMappings = await client.query( + `UPDATE slack_user_mappings + SET workos_user_id = NULL, + mapping_status = 'unmapped', + mapping_source = NULL, + mapped_at = NULL, + mapped_by_user_id = NULL, + updated_at = NOW() + WHERE workos_user_id = $1 + RETURNING slack_user_id`, + [workosUserId], + ); + const expectedSlackMappings = beforeGraph.slack_user_mappings + .filter((row) => row.workos_user_id === workosUserId).length; + if (unlinkedSlackMappings.rowCount !== expectedSlackMappings) { + throw new Error('Confirmed credential deletion did not unlink every Slack mapping'); + } + let deletedRow = false; + if (targetUser) { + const deleted = await client.query<{ workos_user_id: string }>( + 'DELETE FROM users WHERE workos_user_id = $1 RETURNING workos_user_id', + [workosUserId], + ); + if (deleted.rowCount !== 1 || deleted.rows.length !== 1) { + throw new Error('Confirmed credential deletion did not delete exactly one locked user'); + } + deletedRow = true; + } + await client.query('COMMIT'); + result = { + deleted: deletedRow, + affectedUserIds, + affectedSlackUserIds: beforeGraph.slack_user_mappings + .filter((row) => row.workos_user_id === workosUserId) + .map((row) => String(row.slack_user_id)), + quarantine, + replay: null, + }; + } catch (err) { + await client.query('ROLLBACK').catch(() => undefined); + throw err; + } finally { + client.release(); + } + + return result; +} + +/** + * Legacy primary promotion is disabled for #6827: changing the canonical + * credential can transfer authority even without moving membership rows. + * Provider deletions use the identity-credential-deletion service without + * inferring a successor. + * @throws IdentityMutationDisabledError before any database access. */ export async function promoteSecondaryIfPrimaryDeleted( workosUserId: string, ): Promise<{ promotedUserId: string } | null> { + // A primary flip changes canonical authority even without moving rows. + assertIdentityConsolidationAllowed(); const pool = getPool(); const client = await pool.connect(); diff --git a/server/src/db/identity-mutation-policy.ts b/server/src/db/identity-mutation-policy.ts new file mode 100644 index 00000000000..7cba02b1527 --- /dev/null +++ b/server/src/db/identity-mutation-policy.ts @@ -0,0 +1,17 @@ +/** + * Temporary containment for #6827. Generic consolidation cannot preserve + * credential authority or membership provenance, in either direction. + * Operator confirmation, empty previews and caller privilege are not bypasses. + */ +export class IdentityMutationDisabledError extends Error { + readonly code = 'identity_mutation_disabled'; + + constructor() { + super('Identity consolidation is disabled until authority and provenance can be preserved.'); + this.name = 'IdentityMutationDisabledError'; + } +} + +export function assertIdentityConsolidationAllowed(): void { + throw new IdentityMutationDisabledError(); +} diff --git a/server/src/db/mcp-oauth-state-db.ts b/server/src/db/mcp-oauth-state-db.ts index affb95a31f5..7dd6177133f 100644 --- a/server/src/db/mcp-oauth-state-db.ts +++ b/server/src/db/mcp-oauth-state-db.ts @@ -9,8 +9,10 @@ import { query, isDatabaseInitialized } from './client.js'; import { encrypt, decrypt } from './encryption.js'; import { createLogger } from '../logger.js'; +import type { Pool, PoolClient } from 'pg'; const logger = createLogger('mcp-oauth-state-db'); +type Queryable = Pick; // --------------------------------------------------------------------------- @@ -120,11 +122,12 @@ const AUTH_CODE_TTL_MS = 5 * 60 * 1000; // 5 minutes export async function setAuthCode( code: string, data: AuthCodeData, + db?: Queryable, ): Promise { const expiresAt = new Date(Date.now() + AUTH_CODE_TTL_MS); const stored = encryptAuthCodeData(data); try { - await query( + await (db?.query.bind(db) ?? query)( `INSERT INTO mcp_oauth_auth_codes (code, data, expires_at) VALUES ($1, $2, $3)`, [code, JSON.stringify(stored), expiresAt], diff --git a/server/src/db/meetings-db.ts b/server/src/db/meetings-db.ts index 5d9582bf9f6..c21c0312723 100644 --- a/server/src/db/meetings-db.ts +++ b/server/src/db/meetings-db.ts @@ -1,4 +1,5 @@ import { query } from './client.js'; +import { withActiveCredentialEventMutation } from './identity-db.js'; import type { Meeting, MeetingSeries, @@ -652,15 +653,21 @@ export class MeetingsDatabase { * Update topic subscription (create if not exists) */ async updateTopicSubscription(input: UpdateTopicSubscriptionInput): Promise { - const result = await query( - `INSERT INTO working_group_topic_subscriptions (working_group_id, workos_user_id, topic_slugs) - VALUES ($1, $2, $3) - ON CONFLICT (working_group_id, workos_user_id) - DO UPDATE SET topic_slugs = $3, updated_at = NOW() - RETURNING *`, - [input.working_group_id, input.workos_user_id, input.topic_slugs] - ); - return result.rows[0]; + const guarded = await withActiveCredentialEventMutation(input.workos_user_id, async (client) => { + const result = await client.query( + `INSERT INTO working_group_topic_subscriptions (working_group_id, workos_user_id, topic_slugs) + VALUES ($1, $2, $3) + ON CONFLICT (working_group_id, workos_user_id) + DO UPDATE SET topic_slugs = $3, updated_at = NOW() + RETURNING *`, + [input.working_group_id, input.workos_user_id, input.topic_slugs], + ); + return result.rows[0]; + }); + if (!guarded.applied || !guarded.value) { + throw new Error('Cannot update topic subscriptions for an inactive credential'); + } + return guarded.value; } /** diff --git a/server/src/db/membership-db.ts b/server/src/db/membership-db.ts index 199570b5e3a..5f4cf9e37c5 100644 --- a/server/src/db/membership-db.ts +++ b/server/src/db/membership-db.ts @@ -10,10 +10,15 @@ import type { PoolClient } from 'pg'; import { getPool, getClient } from './client.js'; import { findPayingOrgForDomain } from './org-filters.js'; import { createLogger } from '../logger.js'; +import { withActiveCredentialEventMutation } from './identity-db.js'; const logger = createLogger('membership-db'); const OWNERLESS_PROMOTION_LOCK_TIMEOUT_MS = 5_000; +function shouldDeferAutomaticProviderAuthorityWrites(): boolean { + return true; +} + // ── Types ──────────────────────────────────────────────────────────── /** Tracks how each organization_memberships row came to exist. */ @@ -118,10 +123,21 @@ async function withOwnerlessOrgPromotionLock( */ export async function upsertOrganizationMembership( params: MembershipUpsertParams, + externalClient?: PoolClient, ): Promise { - const pool = getPool(); + if (!externalClient) { + const guarded = await withActiveCredentialEventMutation( + params.user_id, + (client) => upsertOrganizationMembership(params, client), + ); + if (!guarded.applied || !guarded.value) { + throw new Error('Cannot write organization authority for an inactive credential'); + } + return guarded.value; + } + const db = externalClient ?? getPool(); - const result = await pool.query<{ role: string }>( + const result = await db.query<{ role: string }>( `INSERT INTO organization_memberships ( workos_user_id, workos_organization_id, @@ -207,8 +223,20 @@ export async function resolveRoleWithWorkosFirstPromote(args: { }> { const { workos, membershipId, userId, organizationId, incomingRole } = args; + // Upstream role mutation needs a durable intent and reconciliation record: + // a timeout or concurrent credential deletion otherwise leaves an unknown + // provider-side authority outcome that cannot safely be replayed. Preserve + // the signed provider role and defer owner selection to an explicit workflow. + void workos; + logger.warn( + { membershipId, userId, orgId: organizationId, incomingRole }, + 'Automatic ownerless promotion deferred: no durable provider intent journal', + ); + return { role: incomingRole, promoted: false }; + // Only the 'member' input is eligible for ownerless-org promotion. Any // explicit role from WorkOS passes through unchanged. + /* c8 ignore start -- retained decisioning for an eventual journaled workflow */ if (incomingRole !== 'member') { return { role: incomingRole, promoted: false }; } @@ -273,6 +301,7 @@ export async function resolveRoleWithWorkosFirstPromote(args: { ); return { role: 'member', promoted: false, promotionError: err }; } + /* c8 ignore stop */ } // ── Delete ─────────────────────────────────────────────────────────── @@ -339,10 +368,11 @@ export async function deleteOrganizationMembership( export async function consumeInvitationSeatType( organizationId: string, email: string, + externalClient?: PoolClient, ): Promise<{ seat_type: string; source: ProvisioningSource | null } | null> { - const pool = getPool(); + const db = externalClient ?? getPool(); - const result = await pool.query<{ seat_type: string; source: string | null }>( + const result = await db.query<{ seat_type: string; source: string | null }>( `DELETE FROM invitation_seat_types WHERE workos_organization_id = $1 AND lower(email) = lower($2) RETURNING seat_type, source`, @@ -390,13 +420,16 @@ export async function setMembershipRole( organizationId: string, role: string, ): Promise { - const pool = getPool(); - - await pool.query( - `UPDATE organization_memberships SET role = $3, updated_at = NOW() - WHERE workos_user_id = $1 AND workos_organization_id = $2`, - [userId, organizationId, role], - ); + const guarded = await withActiveCredentialEventMutation(userId, async (client) => { + await client.query( + `UPDATE organization_memberships SET role = $3, updated_at = NOW() + WHERE workos_user_id = $1 AND workos_organization_id = $2`, + [userId, organizationId, role], + ); + }); + if (!guarded.applied) { + throw new Error('Cannot update organization authority for an inactive credential'); + } } // ── Auto-link by verified domain ──────────────────────────────────── @@ -411,8 +444,9 @@ export interface DomainLinkResult { * Check whether a user's email domain matches a verified domain on an * organization with an active subscription — directly or via the brand * registry hierarchy (e.g. AnalyticsIQ → Alliant). If so, and the org has - * consented to the relevant auto-provisioning class, create a WorkOS - * membership for them on the resolved paying org. + * consented to the relevant auto-provisioning class, this formerly created a + * WorkOS membership on the resolved paying org. Automatic upstream authority + * is currently deferred until a durable intent/reconciliation journal exists. * * Two consent flags on the resolved org, with different defaults: * - auto_provision_verified_domain (default true) gates DIRECT matches @@ -424,9 +458,9 @@ export interface DomainLinkResult { * LLM classification or admin PATCH, ages on M&A, and the joining user * gets no domain-level confirmation. Opt-in. * - * Idempotent: short-circuits when the user is already in the resolved org's - * local membership cache, and treats `organization_membership_already_exists` - * from WorkOS as success. Safe to call on every authenticated request. + * Calls currently fail closed without contacting WorkOS or staging a local + * membership. Keep the legacy decisioning below for an eventual explicitly + * journaled workflow; do not enable it as an untracked automatic mutation. * * Hierarchy walk uses the same trust gates as resolveEffectiveMembership * (high-confidence classifications, 180-day TTL, max 4 hops up) so pre-link @@ -437,6 +471,13 @@ export async function autoLinkByVerifiedDomain( userId: string, email: string, ): Promise { + if (shouldDeferAutomaticProviderAuthorityWrites()) { + logger.warn( + { userId }, + 'Verified-domain auto-provision deferred: no durable exactly-once provider intent journal', + ); + return null; + } const pool = getPool(); const emailDomain = email.split('@')[1]?.toLowerCase(); if (!emailDomain) return null; diff --git a/server/src/db/migrations/593_admin_credential_bind_operations.sql b/server/src/db/migrations/593_admin_credential_bind_operations.sql new file mode 100644 index 00000000000..3c773e81c82 --- /dev/null +++ b/server/src/db/migrations/593_admin_credential_bind_operations.sql @@ -0,0 +1,39 @@ +-- Durable intent and reconciliation evidence for admin WorkOS credential binds. +-- Deliberately no foreign keys: support evidence must survive user/identity deletion. +-- Migration numbers 588-592 are reserved for other work, including member email +-- mutation. The live application ledger was audited before assigning 593. + +CREATE TABLE IF NOT EXISTS admin_credential_bind_operations ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + email_hash TEXT NOT NULL, + host_user_id TEXT NOT NULL, + host_identity_id UUID NOT NULL, + actor_user_id TEXT NOT NULL, + actor_identity_id UUID NOT NULL, + provider_user_id TEXT, + status TEXT NOT NULL CHECK (status IN ( + 'creating', + 'provider_created', + 'committed', + 'compensating', + 'compensated', + 'provider_rejected', + 'reconciliation_required' + )), + failure_code TEXT, + created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW() +); + +CREATE UNIQUE INDEX IF NOT EXISTS admin_credential_bind_operations_active_email + ON admin_credential_bind_operations(email_hash) + WHERE status NOT IN ('compensated', 'provider_rejected'); + +CREATE INDEX IF NOT EXISTS admin_credential_bind_operations_host_created + ON admin_credential_bind_operations(host_user_id, created_at DESC); + +COMMENT ON TABLE admin_credential_bind_operations IS + 'Admin WorkOS bind intent and reconciliation evidence. Unresolved operations block replay; no credentials, tokens, email addresses, or provider bodies.'; + +COMMENT ON COLUMN admin_credential_bind_operations.actor_user_id IS + 'The exact WorkOS credential that authenticated the admin, before canonical identity resolution.'; diff --git a/server/src/db/organization-db.ts b/server/src/db/organization-db.ts index a305e6c087d..ed0be179e77 100644 --- a/server/src/db/organization-db.ts +++ b/server/src/db/organization-db.ts @@ -502,12 +502,13 @@ export async function getSeatUsage(orgId: string): Promise<{ contributor: number */ export async function canAddSeat( orgId: string, - seatType: SeatType + seatType: SeatType, + externalClient?: PoolClient, ): Promise<{ allowed: boolean; reason?: string }> { - const pool = getPool(); - const client = await pool.connect(); + const client = externalClient ?? await getPool().connect(); + const ownsTransaction = externalClient === undefined; try { - await client.query('BEGIN'); + if (ownsTransaction) await client.query('BEGIN'); // Lock the org row to serialize concurrent seat checks and resolve // the tier from the same (locked) read. @@ -554,7 +555,7 @@ export async function canAddSeat( community_only: (total - contributors) + pendingCommunity, }; - await client.query('COMMIT'); + if (ownsTransaction) await client.query('COMMIT'); const limit = seatType === 'contributor' ? limits.contributor : limits.community; const used = seatType === 'contributor' ? usage.contributor : usage.community_only; @@ -564,10 +565,10 @@ export async function canAddSeat( if (used >= limit) return { allowed: false, reason: `All ${limit} ${seatType === 'contributor' ? 'contributor' : 'community'} seats are in use. Upgrade at https://agenticadvertising.org/dashboard/membership to add more.` }; return { allowed: true }; } catch (error) { - await client.query('ROLLBACK'); + if (ownsTransaction) await client.query('ROLLBACK'); throw error; } finally { - client.release(); + if (ownsTransaction) client.release(); } } diff --git a/server/src/db/slack-db.ts b/server/src/db/slack-db.ts index e07928f4f77..2459caef3d3 100644 --- a/server/src/db/slack-db.ts +++ b/server/src/db/slack-db.ts @@ -7,6 +7,8 @@ import type { } from '../slack/types.js'; import { FREE_EMAIL_PROVIDER_DOMAINS } from '../services/identifier-normalization.js'; import { splitFullName } from '../utils/resolve-user-name.js'; +import type { PoolClient } from 'pg'; +import { withActiveCredentialEventMutation } from './identity-db.js'; /** * Escape LIKE pattern wildcards to prevent SQL injection @@ -116,8 +118,15 @@ export class SlackDatabase { workos_user_id: string; mapping_source: SlackMappingSource; mapped_by_user_id?: string; - }): Promise { - const result = await query( + }, externalClient?: PoolClient): Promise { + if (!externalClient) { + const guarded = await withActiveCredentialEventMutation( + input.workos_user_id, + (client) => this.mapUser(input, client), + ); + return guarded.applied ? guarded.value ?? null : null; + } + const result = await externalClient.query( `UPDATE slack_user_mappings SET workos_user_id = $1, mapping_status = 'mapped', @@ -146,7 +155,7 @@ export class SlackDatabase { const slackName = mapping.slack_real_name || mapping.slack_display_name; if (slackName?.trim()) { const { firstName, lastName } = splitFullName(slackName); - await query( + await externalClient.query( `UPDATE users SET first_name = COALESCE(NULLIF(TRIM(first_name), ''), $1), last_name = COALESCE(NULLIF(TRIM(last_name), ''), $2), diff --git a/server/src/db/user-merge-db.ts b/server/src/db/user-merge-db.ts index 6452b3752e5..fd0c477ed9e 100644 --- a/server/src/db/user-merge-db.ts +++ b/server/src/db/user-merge-db.ts @@ -9,6 +9,7 @@ */ import { getPool } from './client.js'; +import { assertIdentityConsolidationAllowed } from './identity-mutation-policy.js'; import { bumpAuthorizationEpochs } from './authorization-epoch-db.js'; import { createLogger } from '../logger.js'; const logger = createLogger('user-merge-db'); @@ -125,6 +126,10 @@ export interface MergeUsersOptions { } /** + * Disabled for #6827 containment; every caller is refused before database access. + * The legacy implementation below must not be re-enabled without preserving + * authority and provenance. Operator confirmation is not an exception. + * * Merge two user accounts. Moves all of the secondary user's app-state rows * to the primary, then binds the secondary's WorkOS user to the primary's * identity as a non-primary sign-in credential. The secondary WorkOS user @@ -145,6 +150,9 @@ export async function mergeUsers( mergedBy: string, options: MergeUsersOptions = {} ): Promise { + // No caller may move, copy or deduplicate authority through identity linkage. + // Keep this before pool access, including for promotion and automatic callers. + assertIdentityConsolidationAllowed(); const pool = getPool(); const client = await pool.connect(); diff --git a/server/src/db/working-group-db.ts b/server/src/db/working-group-db.ts index 50257e52505..3fe91c9c3e5 100644 --- a/server/src/db/working-group-db.ts +++ b/server/src/db/working-group-db.ts @@ -3,6 +3,10 @@ import type { PoolClient } from 'pg'; import { computeJourneyStage } from '../addie/services/journey-computation.js'; import { CommunityDatabase } from './community-db.js'; import { createLogger } from '../logger.js'; +import { + withActiveCredentialEventMutation, + withActiveCredentialSetMutation, +} from './identity-db.js'; import type { WorkingGroup, WorkingGroupLeader, @@ -95,11 +99,19 @@ export class WorkingGroupDatabase { * narrow bypass: they resolve the server-owned slug in their transaction * and do not call this generic mutation surface. */ - private async assertGenericMutationAllowed(workingGroupId: string): Promise { - const result = await query<{ slug: string }>( - 'SELECT slug FROM working_groups WHERE id = $1', - [workingGroupId], - ); + private async assertGenericMutationAllowed( + workingGroupId: string, + externalClient?: PoolClient, + ): Promise { + const result = externalClient + ? await externalClient.query<{ slug: string }>( + 'SELECT slug FROM working_groups WHERE id = $1', + [workingGroupId], + ) + : await query<{ slug: string }>( + 'SELECT slug FROM working_groups WHERE id = $1', + [workingGroupId], + ); if (result.rows[0]?.slug === AAO_ADMIN_WORKING_GROUP_SLUG) { throw new Error('AAO site-admin group can only be changed through the dedicated audited workflow'); } @@ -733,28 +745,12 @@ export class WorkingGroupDatabase { ); const subgroup = subgroupResult.rows[0]; - // 4. Promote subscribers to members - const subResult = await client.query<{ workos_user_id: string }>( - `SELECT DISTINCT workos_user_id FROM working_group_topic_subscriptions - WHERE working_group_id = $1 AND $2 = ANY(topic_slugs)`, - [parent.id, topicSlug] - ); - let addedMembers = 0; - for (const row of subResult.rows) { - const existing = await client.query<{ id: string }>( - `SELECT id FROM working_group_memberships - WHERE working_group_id = $1 AND workos_user_id = $2`, - [subgroup.id, row.workos_user_id] - ); - if (existing.rows[0]) continue; - await client.query( - `INSERT INTO working_group_memberships ( - working_group_id, workos_user_id, status, joined_at, updated_at - ) VALUES ($1, $2, 'active', NOW(), NOW())`, - [subgroup.id, row.workos_user_id] - ); - addedMembers++; - } + // 4. Do not turn subscriptions into authority automatically. Without a + // durable graduation intent journal, a subscriber can be deleted while + // this long content transaction is in flight and then be recreated as + // an active subgroup member. Operators may add active subscribers via + // the lifecycle-fenced membership workflow after graduation. + const addedMembers = 0; // 5. Strip the topic slug from parent subscription rows; delete empty ones await client.query( @@ -899,24 +895,30 @@ export class WorkingGroupDatabase { // Resolve Slack IDs to canonical WorkOS IDs to prevent duplicates const canonicalUserId = await this.resolveToCanonicalUserId(input.workos_user_id); - const result = await query( - `INSERT INTO working_group_memberships ( - working_group_id, workos_user_id, user_email, user_name, user_org_name, - workos_organization_id, added_by_user_id - ) VALUES ($1, $2, $3, $4, $5, $6, $7) - ON CONFLICT (working_group_id, workos_user_id) - DO UPDATE SET status = 'active', updated_at = NOW() - RETURNING *`, - [ - input.working_group_id, - canonicalUserId, - input.user_email || null, - input.user_name || null, - input.user_org_name || null, - input.workos_organization_id || null, - input.added_by_user_id || null, - ] - ); + const guarded = await withActiveCredentialEventMutation(canonicalUserId, async (client) => { + const result = await client.query( + `INSERT INTO working_group_memberships ( + working_group_id, workos_user_id, user_email, user_name, user_org_name, + workos_organization_id, added_by_user_id + ) VALUES ($1, $2, $3, $4, $5, $6, $7) + ON CONFLICT (working_group_id, workos_user_id) + DO UPDATE SET status = 'active', updated_at = NOW() + RETURNING *`, + [ + input.working_group_id, + canonicalUserId, + input.user_email || null, + input.user_name || null, + input.user_org_name || null, + input.workos_organization_id || null, + input.added_by_user_id || null, + ], + ); + return result.rows[0]; + }); + if (!guarded.applied || !guarded.value) { + throw new Error('Cannot add working-group authority to an inactive credential'); + } // Fire-and-forget journey recomputation const orgId = input.workos_organization_id || (await query<{ workos_organization_id: string }>( @@ -928,7 +930,7 @@ export class WorkingGroupDatabase { .catch((err) => { logger.error({ err, workingGroupId: input.working_group_id }, 'Journey stage computation failed'); }); } - return result.rows[0]; + return guarded.value; } /** @@ -990,9 +992,8 @@ export class WorkingGroupDatabase { async grantAAOAdminMembership( input: AAOAdminMembershipMutationInput, ): Promise { - const client = await getPool().connect(); - try { - await client.query('BEGIN'); + const targetUserId = await this.resolveToCanonicalUserId(input.targetUserId); + const guarded = await withActiveCredentialEventMutation(targetUserId, async (client) => { const groupResult = await client.query<{ id: string }>( 'SELECT id FROM working_groups WHERE slug = $1 FOR SHARE', [AAO_ADMIN_WORKING_GROUP_SLUG], @@ -1000,7 +1001,6 @@ export class WorkingGroupDatabase { const group = groupResult.rows[0]; if (!group) throw new Error('AAO admin working group not found'); - const targetUserId = await this.resolveCanonicalUserIdWithClient(client, input.targetUserId); const membershipResult = await client.query( `INSERT INTO working_group_memberships ( working_group_id, workos_user_id, added_by_user_id @@ -1017,16 +1017,12 @@ export class WorkingGroupDatabase { ) VALUES ('granted', $1, $2, 'aao_admin_working_group', $3, $4)`, [input.actorUserId, targetUserId, input.actorAuthorizationMechanism, input.reason], ); - await client.query('COMMIT'); return membershipResult.rows[0]; - } catch (error) { - await client.query('ROLLBACK').catch((rollbackError) => { - logger.warn({ err: rollbackError }, 'AAO admin grant rollback failed'); - }); - throw error; - } finally { - client.release(); + }); + if (!guarded.applied || !guarded.value) { + throw new Error('Cannot grant AAO admin authority to an inactive credential'); } + return guarded.value; } /** @@ -1351,26 +1347,36 @@ export class WorkingGroupDatabase { ); const oldLeaderIds = oldLeadersResult.rows.map(r => r.user_id); - // Remove existing leaders - await query( - 'DELETE FROM working_group_leaders WHERE working_group_id = $1', - [workingGroupId] - ); - - // Add new leaders in a single bulk insert - if (uniqueUserIds.length > 0) { - const values = uniqueUserIds.map((_, i) => `($1, $${i + 2})`).join(', '); - await query( - `INSERT INTO working_group_leaders (working_group_id, user_id) - VALUES ${values} - ON CONFLICT DO NOTHING`, - [workingGroupId, ...uniqueUserIds] - ); + if (uniqueUserIds.length === 0) { + await query('DELETE FROM working_group_leaders WHERE working_group_id = $1', [workingGroupId]); + } else { + const guarded = await withActiveCredentialSetMutation(uniqueUserIds, async (client) => { + await client.query( + 'DELETE FROM working_group_leaders WHERE working_group_id = $1', + [workingGroupId], + ); + const values = uniqueUserIds.map((_, i) => `($1, $${i + 2})`).join(', '); + await client.query( + `INSERT INTO working_group_leaders (working_group_id, user_id) + VALUES ${values} + ON CONFLICT DO NOTHING`, + [workingGroupId, ...uniqueUserIds], + ); + await client.query( + `INSERT INTO working_group_memberships (working_group_id, workos_user_id, status) + SELECT $1, user_id, 'active' + FROM working_group_leaders + WHERE working_group_id = $1 + ON CONFLICT (working_group_id, workos_user_id) + DO UPDATE SET status = 'active', updated_at = NOW()`, + [workingGroupId], + ); + }); + if (!guarded.applied) { + throw new Error('Cannot assign leadership to an inactive credential'); + } } - // Ensure leaders are members - await this.ensureLeadersAreMembers(workingGroupId); - // Fire-and-forget journey recomputation for affected orgs const allAffectedUserIds = [...new Set([...oldLeaderIds, ...uniqueUserIds])]; if (allAffectedUserIds.length > 0) { @@ -1388,20 +1394,40 @@ export class WorkingGroupDatabase { /** * Add a leader to a working group */ - async addLeader(workingGroupId: string, userId: string): Promise { - await this.assertGenericMutationAllowed(workingGroupId); + async addLeader( + workingGroupId: string, + userId: string, + externalClient?: PoolClient, + ): Promise { + await this.assertGenericMutationAllowed(workingGroupId, externalClient); // Resolve Slack IDs to canonical WorkOS IDs to prevent duplicates - const canonicalUserId = await this.resolveToCanonicalUserId(userId); + const canonicalUserId = externalClient + ? await this.resolveCanonicalUserIdWithClient(externalClient, userId) + : await this.resolveToCanonicalUserId(userId); - await query( - `INSERT INTO working_group_leaders (working_group_id, user_id) - VALUES ($1, $2) - ON CONFLICT DO NOTHING`, - [workingGroupId, canonicalUserId] - ); + if (externalClient) { + await externalClient.query( + `INSERT INTO working_group_leaders (working_group_id, user_id) + VALUES ($1, $2) + ON CONFLICT DO NOTHING`, + [workingGroupId, canonicalUserId], + ); + await externalClient.query( + `INSERT INTO working_group_memberships (working_group_id, workos_user_id, status) + VALUES ($1, $2, 'active') + ON CONFLICT (working_group_id, workos_user_id) + DO UPDATE SET status = 'active', updated_at = NOW()`, + [workingGroupId, canonicalUserId], + ); + return; + } - // Ensure leader is a member - await this.ensureLeadersAreMembers(workingGroupId); + const guarded = await withActiveCredentialEventMutation(canonicalUserId, async (client) => { + await this.addLeader(workingGroupId, canonicalUserId, client); + }); + if (!guarded.applied) { + throw new Error('Cannot assign leadership to an inactive credential'); + } // Fire-and-forget journey recomputation const orgResult = await query<{ workos_organization_id: string }>( @@ -2007,32 +2033,37 @@ export class WorkingGroupDatabase { // Resolve Slack IDs to canonical WorkOS IDs to prevent duplicates const canonicalUserId = await this.resolveToCanonicalUserId(input.workos_user_id); - const result = await query( - `INSERT INTO working_group_memberships ( - working_group_id, workos_user_id, user_email, user_name, user_org_name, - workos_organization_id, added_by_user_id, interest_level, interest_source - ) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9) - ON CONFLICT (working_group_id, workos_user_id) - DO UPDATE SET - status = 'active', - interest_level = COALESCE(EXCLUDED.interest_level, working_group_memberships.interest_level), - interest_source = COALESCE(EXCLUDED.interest_source, working_group_memberships.interest_source), - updated_at = NOW() - RETURNING *`, - [ - input.working_group_id, - canonicalUserId, - input.user_email || null, - input.user_name || null, - input.user_org_name || null, - input.workos_organization_id || null, - input.added_by_user_id || null, - input.interest_level || null, - input.interest_source || null, - ] - ); - - return result.rows[0]; + const guarded = await withActiveCredentialEventMutation(canonicalUserId, async (client) => { + const result = await client.query( + `INSERT INTO working_group_memberships ( + working_group_id, workos_user_id, user_email, user_name, user_org_name, + workos_organization_id, added_by_user_id, interest_level, interest_source + ) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9) + ON CONFLICT (working_group_id, workos_user_id) + DO UPDATE SET + status = 'active', + interest_level = COALESCE(EXCLUDED.interest_level, working_group_memberships.interest_level), + interest_source = COALESCE(EXCLUDED.interest_source, working_group_memberships.interest_source), + updated_at = NOW() + RETURNING *`, + [ + input.working_group_id, + canonicalUserId, + input.user_email || null, + input.user_name || null, + input.user_org_name || null, + input.workos_organization_id || null, + input.added_by_user_id || null, + input.interest_level || null, + input.interest_source || null, + ], + ); + return result.rows[0]; + }); + if (!guarded.applied || !guarded.value) { + throw new Error('Cannot add working-group authority to an inactive credential'); + } + return guarded.value; } /** diff --git a/server/src/http.ts b/server/src/http.ts index 773a82e1759..fc72a4560a5 100644 --- a/server/src/http.ts +++ b/server/src/http.ts @@ -57,11 +57,16 @@ import { PropertyDatabase } from "./db/property-db.js"; import * as manifestRefsDb from "./db/manifest-refs-db.js"; import { JoinRequestDatabase } from "./db/join-request-db.js"; import { SlackDatabase } from "./db/slack-db.js"; -import { autoLinkByVerifiedDomain } from "./db/membership-db.js"; +import { autoLinkByVerifiedDomain, upsertOrganizationMembership } from "./db/membership-db.js"; import { syncSlackUsers, getSyncStatus, tryAutoLinkWebsiteUserToSlack } from "./slack/sync.js"; import { isSlackConfigured, testSlackConnection } from "./slack/client.js"; import { handleSlashCommand } from "./slack/commands.js"; import { getCompanyDomain, getGoogleEmailAliases } from "./utils/email-domain.js"; +import { assertIdentityConsolidationAllowed } from "./db/identity-mutation-policy.js"; +import { + upsertWorkosUserInCredentialEvent, + withCredentialCreationEventMutation, +} from "./db/identity-db.js"; import { hasActiveSlackLink } from "./utils/slack-linkage.js"; import { isUuid } from "./utils/uuid.js"; import { resolveUserNameWithFallbacks, sanitizeName } from "./utils/resolve-user-name.js"; @@ -7749,25 +7754,27 @@ ${p.category ? `${p.category}\n` : ''}${publishedUrl}< // On INSERT, use WorkOS values — falling back to existing DB / Slack // mapping when WorkOS itself has empty names. On UPDATE, preserve // user-set names: only fill in names that are currently empty. - try { - const pool = getPool(); - const { firstName, lastName } = await resolveUserNameWithFallbacks( - pool, user.id, user.firstName, user.lastName, - ); - await pool.query( - `INSERT INTO users (workos_user_id, email, first_name, last_name, email_verified, workos_created_at, workos_updated_at, created_at, updated_at) - VALUES ($1, $2, $3, $4, $5, $6, $7, NOW(), NOW()) - ON CONFLICT (workos_user_id) DO UPDATE SET - email = EXCLUDED.email, - first_name = COALESCE(NULLIF(TRIM(users.first_name), ''), EXCLUDED.first_name), - last_name = COALESCE(NULLIF(TRIM(users.last_name), ''), EXCLUDED.last_name), - email_verified = EXCLUDED.email_verified, - workos_updated_at = EXCLUDED.workos_updated_at, - updated_at = NOW()`, - [user.id, user.email, firstName, lastName, user.emailVerified, user.createdAt, user.updatedAt] - ); - } catch (upsertError) { - logger.error({ error: upsertError, userId: user.id }, 'Failed to upsert user on login'); + const pool = getPool(); + const { firstName, lastName } = await resolveUserNameWithFallbacks( + pool, user.id, user.firstName, user.lastName, + ); + const localFinalize = await withCredentialCreationEventMutation(user.id, async (client) => { + await upsertWorkosUserInCredentialEvent(client, { + id: user.id, + email: user.email, + firstName, + lastName, + emailVerified: user.emailVerified, + createdAt: user.createdAt, + updatedAt: user.updatedAt, + }, 'preserve_existing'); + }); + if (!localFinalize.applied) { + logger.warn({ userId: user.id }, 'Refused OAuth callback for terminal local credential'); + return res.status(403).json({ + error: 'Account unavailable', + message: 'This account cannot be authenticated. Contact support if this is unexpected.', + }); } // Auto-merge duplicate accounts caused by Google email aliases. @@ -7802,6 +7809,9 @@ ${p.category ? `${p.category}\n` : ''}${publishedUrl}< const workosUsers = await workos.userManagement.listUsers({ email: aliasEmail }); const match = workosUsers.data.find(u => u.id !== user.id); if (match) { + duplicateAliasEmail = match.email; + // Containment before even creating a local duplicate credential. + assertIdentityConsolidationAllowed(); // Insert into local users table so mergeUsers can operate on it await pool.query( `INSERT INTO users (workos_user_id, email, first_name, last_name, email_verified, workos_created_at, workos_updated_at, created_at, updated_at) @@ -7821,6 +7831,9 @@ ${p.category ? `${p.category}\n` : ''}${publishedUrl}< if (existing) { duplicateAliasEmail = existing.email; + // Refuse before alias claims or WorkOS membership copies. Mailbox + // equivalence cannot authorize consolidation (#6827). + assertIdentityConsolidationAllowed(); // Claim the alias atomically — UNIQUE(LOWER(email)) prevents // two users from claiming the same target concurrently. @@ -8156,30 +8169,40 @@ ${p.category ? `${p.category}\n` : ''}${publishedUrl}< if (existingMapping && !existingMapping.workos_user_id) { // Link the Slack user to the newly authenticated WorkOS user - await slackDb.mapUser({ + const mapped = await slackDb.mapUser({ slack_user_id: slackUserIdToLink, workos_user_id: user.id, mapping_source: 'user_claimed', }); - accountLinked = true; - accountNewlyLinked = true; - logger.info( - { slackUserId: slackUserIdToLink, workosUserId: user.id }, - 'Auto-linked Slack account after signup' - ); + if (!mapped) { + logger.warn( + { slackUserId: slackUserIdToLink, workosUserId: user.id }, + 'Skipped Slack account link because the local credential is no longer active', + ); + } else { + accountLinked = true; + accountNewlyLinked = true; + logger.info( + { slackUserId: slackUserIdToLink, workosUserId: user.id }, + 'Auto-linked Slack account after signup' + ); + } - // Record account linking in the relationship system - try { - const { resolvePersonId } = await import('./db/relationship-db.js'); - const { recordEvent } = await import('./db/person-events-db.js'); - const personId = await resolvePersonId({ slack_user_id: slackUserIdToLink, workos_user_id: user.id }); - await recordEvent(personId, 'account_linked', { - channel: 'web', - data: { workos_user_id: user.id }, - }); - logger.info({ slackUserId: slackUserIdToLink, personId }, 'Recorded account_linked event'); - } catch (trackingError) { - logger.warn({ error: trackingError, slackUserId: slackUserIdToLink }, 'Failed to record account_linked event'); + // Record account linking only after the lifecycle-fenced map + // actually commits. + if (mapped) { + try { + const { resolvePersonId } = await import('./db/relationship-db.js'); + const { recordEvent } = await import('./db/person-events-db.js'); + const personId = await resolvePersonId({ slack_user_id: slackUserIdToLink, workos_user_id: user.id }); + await recordEvent(personId, 'account_linked', { + channel: 'web', + data: { workos_user_id: user.id }, + }); + logger.info({ slackUserId: slackUserIdToLink, personId }, 'Recorded account_linked event'); + } catch (trackingError) { + logger.warn({ error: trackingError, slackUserId: slackUserIdToLink }, 'Failed to record account_linked event'); + } } } else if (!existingMapping) { @@ -9244,13 +9267,21 @@ ${p.category ? `${p.category}\n` : ''}${publishedUrl}< role: roleSlug, }, 'User auto-added to organization via verified domain'); - // Mirror membership locally so it's visible immediately - const pool2 = getPool(); - await pool2.query(` - INSERT INTO organization_memberships (workos_user_id, workos_organization_id, email, role, created_at, updated_at, synced_at) - VALUES ($1, $2, $3, $4, NOW(), NOW(), NOW()) - ON CONFLICT (workos_user_id, workos_organization_id) DO UPDATE SET role = $4, updated_at = NOW() - `, [user.id, organization_id, user.email, roleSlug]); + // Mirror only while the exact credential is still active. A + // provider response that settles after deletion must not restore + // local organization authority. + await upsertOrganizationMembership({ + user_id: user.id, + organization_id, + membership_id: membership.id, + email: user.email, + first_name: user.firstName ?? null, + last_name: user.lastName ?? null, + role: roleSlug, + seat_type: 'community_only', + has_explicit_seat_type: false, + provisioning_source: 'verified_domain', + }); // Record audit log await orgDb.recordAuditLog({ diff --git a/server/src/mcp/oauth-provider.ts b/server/src/mcp/oauth-provider.ts index c3957f246a2..dd66b1ff1b3 100644 --- a/server/src/mcp/oauth-provider.ts +++ b/server/src/mcp/oauth-provider.ts @@ -22,6 +22,11 @@ import { verifyWorkOSJWT } from '../auth/workos-jwt.js'; import { getAuthorizationUrl, refreshTokenRaw, authenticateWithCodeForTokens } from '../auth/workos-client.js'; import * as mcpClientsDb from '../db/mcp-clients-db.js'; import * as mcpOAuthStateDb from '../db/mcp-oauth-state-db.js'; +import { + upsertWorkosUserInCredentialEvent, + withCredentialCreationEventMutation, +} from '../db/identity-db.js'; +import { readCredentialAuthorizationLifecycle } from '../db/authorization-epoch-db.js'; const logger = createLogger('mcp-oauth'); @@ -49,6 +54,17 @@ async function verifyAccessTokenJWT(token: string): Promise { throw new InvalidTokenError('Invalid or expired token'); } + if (!verified.isM2M && verified.sub) { + const lifecycle = await readCredentialAuthorizationLifecycle(verified.sub); + if (lifecycle.status !== 'active') { + logger.warn( + { sub: verified.sub, lifecycle: lifecycle.status }, + 'MCP OAuth: local credential lifecycle rejected token', + ); + throw new InvalidTokenError('Credential is unavailable'); + } + } + return { token, clientId: verified.clientId, @@ -253,46 +269,45 @@ export async function handleMCPOAuthCallback( return; } - // Upsert the user into our local table so downstream code (REST requireAuth, - // /api/me/*, dashboard queries) can find them by workos_user_id. The - // cookie-based /auth/callback path does the same upsert; without it, - // users who first arrive via the MCP OAuth flow don't exist locally and - // REST auth rejects their JWT. - // - // On failure we log and continue — we still issue the OAuth code so /mcp - // works. The user will see a 401 the first time they call /api/*, with a - // corresponding warn log ("Bearer JWT verified but user not found in - // local DB"). A retry (re-SSO) recovers; a persistent failure indicates - // a DB problem that operators need to investigate, not a per-user issue. + // Finalize the local credential and MCP authorization code atomically under + // the same terminal lifecycle lock used by provider deletion. The WorkOS + // exchange above performs provider I/O before lock acquisition. A deletion + // that wins during that exchange leaves a tombstone and this finalize emits + // no local code. + const localCode = crypto.randomBytes(32).toString('hex'); + let finalized = false; try { - const { getPool } = await import('../db/client.js'); const { user } = authResult; - await getPool().query( - `INSERT INTO users (workos_user_id, email, first_name, last_name, email_verified, workos_created_at, workos_updated_at, created_at, updated_at) - VALUES ($1, $2, $3, $4, $5, $6, $7, NOW(), NOW()) - ON CONFLICT (workos_user_id) DO UPDATE SET - email = EXCLUDED.email, - first_name = COALESCE(NULLIF(TRIM(users.first_name), ''), EXCLUDED.first_name), - last_name = COALESCE(NULLIF(TRIM(users.last_name), ''), EXCLUDED.last_name), - email_verified = EXCLUDED.email_verified, - workos_updated_at = EXCLUDED.workos_updated_at, - updated_at = NOW()`, - [user.id, user.email, user.firstName, user.lastName, user.emailVerified, user.createdAt, user.updatedAt], - ); - } catch (upsertErr) { - logger.error({ err: upsertErr }, 'MCP OAuth: Failed to upsert user on callback'); + const result = await withCredentialCreationEventMutation(user.id, async (client) => { + await upsertWorkosUserInCredentialEvent(client, { + id: user.id, + email: user.email, + firstName: user.firstName ?? null, + lastName: user.lastName ?? null, + emailVerified: user.emailVerified, + createdAt: user.createdAt, + updatedAt: user.updatedAt, + }, 'preserve_existing'); + await mcpOAuthStateDb.setAuthCode(localCode, { + clientId: pending.clientId, + codeChallenge: pending.codeChallenge, + redirectUri: pending.redirectUri, + accessToken: authResult.accessToken, + refreshToken: authResult.refreshToken, + }, client); + }); + finalized = result.applied; + } catch (finalizeErr) { + logger.error({ err: finalizeErr }, 'MCP OAuth: Failed closed during local callback finalize'); + } + if (!finalized) { + const errorUrl = new URL(pending.redirectUri); + errorUrl.searchParams.set('error', 'access_denied'); + errorUrl.searchParams.set('error_description', 'Local credential is unavailable'); + if (pending.state) errorUrl.searchParams.set('state', pending.state); + res.redirect(errorUrl.toString()); + return; } - - // Generate local authorization code - const localCode = crypto.randomBytes(32).toString('hex'); - - await mcpOAuthStateDb.setAuthCode(localCode, { - clientId: pending.clientId, - codeChallenge: pending.codeChallenge, - redirectUri: pending.redirectUri, - accessToken: authResult.accessToken, - refreshToken: authResult.refreshToken, - }); // Redirect to MCP client's callback URL const redirectUrl = new URL(pending.redirectUri); diff --git a/server/src/middleware/auth.ts b/server/src/middleware/auth.ts index b1e3c756f70..c9c2937e03e 100644 --- a/server/src/middleware/auth.ts +++ b/server/src/middleware/auth.ts @@ -14,7 +14,11 @@ import { isWorkOSApiKeyFormat } from './api-key-format.js'; import { verifyWorkOSJWT, looksLikeJWT } from '../auth/workos-jwt.js'; import { storeRefreshedSession, getRefreshedSession, cleanExpiredRefreshes } from '../db/session-refresh-db.js'; import { getPool } from '../db/client.js'; -import { getAuthorizationFingerprint } from '../db/authorization-epoch-db.js'; +import { + readCredentialAuthorizationLifecycle, + type ActiveCredentialAuthorizationSnapshot, + type CredentialAuthorizationLifecycle, +} from '../db/authorization-epoch-db.js'; import { getOrganizationAuthorizationUserId } from '../auth/organization-principal.js'; import { constantTimeEqual } from '../utils/constant-time-equal.js'; import { resolveEffectiveMembership } from '../db/org-filters.js'; @@ -380,34 +384,31 @@ export async function validateWorkOSBearerJWT(req: Request): Promise( - `SELECT first_name, last_name, email FROM users WHERE workos_user_id = $1`, - [verified.sub], - ); - if (localUser.rowCount === 0) { - logger.warn({ sub: verified.sub }, 'Bearer JWT verified but user not found in local DB'); + // A valid WorkOS signature is not enough: accept and cache only an + // authoritative local credential snapshot that includes the binding and + // proves no durable deletion tombstone exists. + const lifecycle = await readCredentialAuthorizationLifecycle(verified.sub); + if (lifecycle.status === 'unavailable') { + logger.warn({ sub: verified.sub }, 'Bearer JWT local credential lifecycle unavailable — failed closed'); + return null; + } + if (lifecycle.status === 'terminal') { + logger.warn({ sub: verified.sub }, 'Bearer JWT verified but credential is not active locally'); return null; } + const activeSnapshot = lifecycle.snapshot; - const row = localUser.rows[0]; - const email = verified.email ?? row.email ?? ''; + const email = verified.email ?? activeSnapshot.email ?? ''; const user: WorkOSUser = { id: verified.sub, email, - firstName: row.first_name?.trim() || undefined, - lastName: row.last_name?.trim() || undefined, + firstName: activeSnapshot.first_name?.trim() || undefined, + lastName: activeSnapshot.last_name?.trim() || undefined, emailVerified: true, createdAt: new Date().toISOString(), updatedAt: new Date().toISOString(), }; + applyActiveCredentialSnapshot(user, activeSnapshot); const tokenExpMs = verified.expiresAt ? verified.expiresAt * 1000 : Infinity; const cacheUntil = Math.min(now + BEARER_JWT_CACHE_TTL_MS, tokenExpMs); @@ -420,7 +421,7 @@ export async function validateWorkOSBearerJWT(req: Request): Promise { } } +function applyActiveCredentialSnapshot( + user: WorkOSUser, + snapshot: ActiveCredentialAuthorizationSnapshot, +): void { + const authenticatedWorkosUserId = snapshot.workos_user_id; + user.identityId = snapshot.identity_id; + if (snapshot.primary_workos_user_id + && snapshot.primary_workos_user_id !== authenticatedWorkosUserId) { + user.authWorkosUserId = authenticatedWorkosUserId; + user.id = snapshot.primary_workos_user_id; + } +} + /** * Authorization fingerprint for the credential that actually authenticated * this session. `attachIdentityId` may swap `user.id` to the identity's @@ -798,19 +812,11 @@ async function attachIdentityId(user: WorkOSUser): Promise { * Synthetic principals (admin API key, WorkOS API key) have no credential * row and no identity binding; they return '' and never mismatch. */ -async function authorizationFingerprintFor( +async function activeAuthorizationSnapshotFor( user: WorkOSUser, -): Promise { - if (isSyntheticUser(user.id)) return ''; - try { - return await getAuthorizationFingerprint([getOrganizationAuthorizationUserId(user)]); - } catch (err) { - // Never fail authentication over the stamp: an unstamped entry reads as - // stale on its next hit, so the session is re-validated rather than - // served from a fingerprint we could not confirm. - logger.warn({ err }, 'Authorization epoch stamp failed — session will not be served from cache'); - return undefined; - } +): Promise { + if (isSyntheticUser(user.id)) return { status: 'terminal', reason: 'missing_user' }; + return readCredentialAuthorizationLifecycle(getOrganizationAuthorizationUserId(user)); } /** @@ -828,9 +834,11 @@ async function isAuthorizationFingerprintCurrent( ): Promise { if (isSyntheticUser(user.id)) return true; try { - return (await authorizationFingerprintFor(user)) === (stamped ?? ''); + const lifecycle = await activeAuthorizationSnapshotFor(user); + return lifecycle.status === 'active' + && lifecycle.snapshot.fingerprint === (stamped ?? ''); } catch (err) { - logger.warn({ err }, 'Authorization epoch check failed — bypassing session cache'); + logger.warn({ err }, 'Authoritative credential check failed — bypassing session cache'); return false; } } @@ -908,7 +916,6 @@ export async function requireAuth(req: Request, res: Response, next: NextFunctio logger.warn({ err: banError, userId: jwtAuth.user.id, path: req.path }, 'Ban check failed — allowing request through'); } - await attachIdentityId(req.user); return next(); } @@ -1168,21 +1175,22 @@ export async function requireAuth(req: Request, res: Response, next: NextFunctio let firstName = result.user.firstName ?? undefined; let lastName = result.user.lastName ?? undefined; - // Verify the user exists in our local DB. This catches stale sessions - // from deleted/merged WorkOS accounts whose JWTs haven't expired yet. - // Also resolves names from local DB when WorkOS doesn't provide them. - try { - const pool = getPool(); - const localUser = await pool.query<{ - first_name: string | null; - last_name: string | null; - }>( - `SELECT first_name, last_name FROM users WHERE workos_user_id = $1`, - [result.user.id] - ); - if (localUser.rows.length === 0) { + // WorkOS session validity alone cannot authorize a deleted credential. + // Read the user, binding, canonical route, tombstone absence, and epoch + // atomically before accepting or caching this session. + const lifecycle = await readCredentialAuthorizationLifecycle(result.user.id); + if (lifecycle.status === 'unavailable') { + sessionCache.delete(cacheKey); + logger.warn({ userId: result.user.id, path: req.path }, + 'Local credential lifecycle unavailable — failing authentication closed'); + return res.status(503).json({ + error: 'Service temporarily unavailable', + message: 'Unable to verify your account right now. Please try again shortly.', + }); + } + if (lifecycle.status === 'terminal') { logger.warn({ userId: result.user.id, email: result.user.email, path: req.path }, - 'Authenticated user not found in local DB — forcing re-login'); + 'Authenticated credential is not active in local DB — forcing re-login'); sessionCache.delete(cacheKey); if (deadSessionCache.size >= DEAD_SESSION_MAX_SIZE) { const oldest = deadSessionCache.keys().next().value; @@ -1197,15 +1205,11 @@ export async function requireAuth(req: Request, res: Response, next: NextFunctio message: 'Your account could not be verified. Please log in again.', login_url: '/auth/login', }); - } - if (!firstName?.trim()) { - const row = localUser.rows[0]; - if (row.first_name?.trim()) firstName = row.first_name; - if (row.last_name?.trim()) lastName = row.last_name; - } - } catch (err) { - // Fail open: DB errors should not block authenticated users - logger.warn({ err, userId: result.user.id }, 'Local user check failed — allowing request through'); + } + const activeSnapshot = lifecycle.snapshot; + if (!firstName?.trim()) { + if (activeSnapshot.first_name?.trim()) firstName = activeSnapshot.first_name; + if (activeSnapshot.last_name?.trim()) lastName = activeSnapshot.last_name; } const user: WorkOSUser = { @@ -1227,8 +1231,7 @@ export async function requireAuth(req: Request, res: Response, next: NextFunctio ); } - // Resolve identityId once before caching so cache hits inherit it. - await attachIdentityId(user); + applyActiveCredentialSnapshot(user, activeSnapshot); // Cache the validated session sessionCache.set(cacheKey, { @@ -1236,7 +1239,7 @@ export async function requireAuth(req: Request, res: Response, next: NextFunctio accessToken: result.accessToken, expiresAt: now + SESSION_CACHE_TTL_MS, newSealedSession, - authorizationFingerprint: await authorizationFingerprintFor(user), + authorizationFingerprint: activeSnapshot.fingerprint, }); req.user = user; @@ -1938,7 +1941,6 @@ export async function optionalAuth(req: Request, res: Response, next: NextFuncti logger.warn({ err: banError, userId: jwtAuth.user.id, path: req.path }, 'Ban check failed — allowing optional-auth request through'); } - await attachIdentityId(req.user); return next(); } @@ -2119,10 +2121,21 @@ export async function optionalAuth(req: Request, res: Response, next: NextFuncti ); } - // Resolve identityId before caching — sessionCache is shared with - // requireAuth, so skipping it here would let an optionalAuth request - // poison subsequent requireAuth cache hits with identityId=undefined. - await attachIdentityId(user); + // The shared cache may be consumed by requireAuth. Only an authoritative + // active local credential snapshot may create a positive entry. + const lifecycle = await readCredentialAuthorizationLifecycle(user.id); + if (lifecycle.status !== 'active') { + sessionCache.delete(cacheKey); + return next(); + } + const activeSnapshot = lifecycle.snapshot; + if (!user.firstName?.trim() && activeSnapshot.first_name?.trim()) { + user.firstName = activeSnapshot.first_name; + } + if (!user.lastName?.trim() && activeSnapshot.last_name?.trim()) { + user.lastName = activeSnapshot.last_name; + } + applyActiveCredentialSnapshot(user, activeSnapshot); // Cache the validated session sessionCache.set(cacheKey, { @@ -2130,7 +2143,7 @@ export async function optionalAuth(req: Request, res: Response, next: NextFuncti accessToken: result.accessToken, expiresAt: now + SESSION_CACHE_TTL_MS, newSealedSession, - authorizationFingerprint: await authorizationFingerprintFor(user), + authorizationFingerprint: activeSnapshot.fingerprint, }); req.user = user; diff --git a/server/src/routes/account-linking.ts b/server/src/routes/account-linking.ts index 1e9853a3051..e1cfed0bb1f 100644 --- a/server/src/routes/account-linking.ts +++ b/server/src/routes/account-linking.ts @@ -4,7 +4,7 @@ import rateLimit from 'express-rate-limit'; import { createLogger } from '../logger.js'; import { requireAuth } from '../middleware/auth.js'; import { query, getPool } from '../db/client.js'; -import { mergeUsers } from '../db/user-merge-db.js'; +import { refuseIdentityConsolidation } from './identity-mutation-containment.js'; import { sendEmailLinkVerification } from '../notifications/email.js'; import { getWorkos } from '../auth/workos-client.js'; import { CachedPostgresStore } from '../middleware/pg-rate-limit-store.js'; @@ -215,7 +215,7 @@ export function createAccountLinkingRouter(): Router { }); // PUT /api/me/linked-emails/primary — swap a linked alias to be the primary email - router.put('/primary', requireAuth, verifyExecuteLimiter, async (req: Request, res: Response) => { + router.put('/primary', requireAuth, refuseIdentityConsolidation, verifyExecuteLimiter, async (req: Request, res: Response) => { try { const userId = req.user!.id; const { email } = req.body; diff --git a/server/src/routes/admin/users.ts b/server/src/routes/admin/users.ts index 639ecd74567..3b832f01752 100644 --- a/server/src/routes/admin/users.ts +++ b/server/src/routes/admin/users.ts @@ -16,12 +16,13 @@ import { import { SlackDatabase } from '../../db/slack-db.js'; import { WorkingGroupDatabase } from '../../db/working-group-db.js'; import { getPool } from '../../db/client.js'; -import { bumpAuthorizationEpochs } from '../../db/authorization-epoch-db.js'; -import { findSupersededMemberships } from '../../db/membership-consolidation-db.js'; +import { CredentialDetachConflict, detachCredential } from '../../db/credential-detach-db.js'; import { backfillOrganizationMemberships, backfillUsers, backfillOrganizationDomains } from '../workos-webhooks.js'; import { sendSlackInviteEmail, hasSlackInviteBeenSent } from '../../notifications/email.js'; import { getWorkos } from '../../auth/workos-client.js'; import { mergeUsers } from '../../db/user-merge-db.js'; +import { createAndBindAdminCredential } from '../../services/admin-credential-bind.js'; +import { refuseIdentityConsolidation } from '../identity-mutation-containment.js'; import { resolveMembershipTier, type MembershipTierRow } from '../../db/organization-db.js'; import { buildCountryMembersCsv, @@ -842,150 +843,30 @@ export function createAdminUsersRouter(): Router { } }); - // POST /api/admin/users/:userId/linked-emails - // - // Bind a new sign-in email to an existing user's identity. Creates a fresh - // WorkOS user for the new email and binds it as a non-primary credential - // under the same identity. After this, the user can sign in with either - // email — the auth middleware id-swaps non-primary logins to the canonical - // workos_user_id so they see the same workspace. - // - // Use case: a user lost access to an alias email after the old delete-the- - // secondary merge flow. Admin restores the alias by creating a new WorkOS - // user and binding it. - // - // Trust model: admin is asserting the email belongs to the person. No - // verification email is sent to the new address. Phase 3 may add one. + // Create a fresh credential and bind it without consolidating any app state + // or organization authority. Ambiguous provider outcomes block email replay. router.post('/:userId/linked-emails', ...requireGlobalAdmin, async (req, res) => { - const adminEmail = req.user!.email; - const adminUserId = req.user!.id; - const existingUserId = req.params.userId; - const rawEmail = (req.body?.email as string | undefined)?.trim(); - - if (!rawEmail) { - return res.status(400).json({ error: 'email is required' }); - } - const newEmail = rawEmail.toLowerCase(); - if (newEmail.length > 255 || !isPlausibleEmail(newEmail)) { - return res.status(400).json({ error: 'Invalid email address' }); - } - - const pool = getPool(); - - // Verify existing user - const existing = await pool.query<{ email: string; first_name: string | null; last_name: string | null }>( - `SELECT email, first_name, last_name FROM users WHERE workos_user_id = $1`, - [existingUserId] - ); - if (existing.rows.length === 0) { - return res.status(404).json({ error: 'User not found' }); - } - if (existing.rows[0].email.toLowerCase() === newEmail) { - return res.status(409).json({ error: 'This is already the user\'s primary email' }); + if (req.body?.consolidate === true || req.body?.promote === true) { + return res.status(409).json({ error: 'credential_consolidation_disabled' }); } - - // Refuse if this email is already a real WorkOS user in our DB. Binding - // an existing account is the high-risk path that the self-service merge - // flow blocks (see issue #3719); admin shouldn't silently merge two - // existing accounts via this endpoint either. Use the org-merge tool or - // a future "bind existing" admin path instead. - const claimed = await pool.query( - `SELECT workos_user_id FROM users WHERE LOWER(email) = $1`, - [newEmail] - ); - if (claimed.rows.length > 0) { - return res.status(409).json({ - error: 'This email already has an AAO account', - message: 'Use a separate consolidation flow to combine two existing accounts. This endpoint creates a fresh sign-in.', - }); + if (req.adminAccessMechanism === 'static_admin_api_key' || !req.user?.identityId) { + return res.status(403).json({ error: 'identity_bearing_admin_required' }); } - - const workos = getWorkos(); - let newWorkosUser: { id: string; email: string; firstName: string | null; lastName: string | null; emailVerified: boolean; createdAt: string; updatedAt: string }; - - try { - newWorkosUser = await workos.userManagement.createUser({ - email: newEmail, - emailVerified: true, - firstName: existing.rows[0].first_name ?? undefined, - lastName: existing.rows[0].last_name ?? undefined, - }); - } catch (err: any) { - // 422 = email already in use in WorkOS (we didn't see them in our DB - // but WorkOS knows about them). 400 = WorkOS rejected for other - // reasons — typically a still-living user at that email from a prior - // merge whose deleteUser silently failed. Surface clearly. - const status = err?.status ?? err?.response?.status; - const workosMsg = err?.message ?? err?.rawMessage ?? ''; - if (status === 422 || status === 409 || status === 400) { - logger.warn({ err, newEmail, existingUserId, status }, 'Admin bind-email: WorkOS rejected createUser'); - return res.status(409).json({ - error: 'WorkOS will not create a user at this email', - message: `WorkOS responded ${status}: ${workosMsg || 'no message'}. The email is likely already in use upstream — look it up in the WorkOS Dashboard and use the "Link existing WorkOS user" admin tool to bind by id.`, - }); - } - logger.error({ err, newEmail, existingUserId, status }, 'Admin bind-email: WorkOS createUser failed'); - return res.status(502).json({ - error: 'Failed to create sign-in email upstream', - message: workosMsg || `WorkOS responded with status ${status ?? 'unknown'}.`, - }); + const rawEmail = req.body?.email; + if (typeof rawEmail !== 'string' || !rawEmail.trim()) { + return res.status(400).json({ error: 'email is required' }); } - - // Insert into local users — fires the AFTER INSERT trigger which creates - // a singleton identity for the new WorkOS user. mergeUsers will then - // re-point the new user's binding to the existing user's identity. - try { - await pool.query( - `INSERT INTO users (workos_user_id, email, first_name, last_name, email_verified, - workos_created_at, workos_updated_at, created_at, updated_at) - VALUES ($1, $2, $3, $4, $5, $6, $7, NOW(), NOW()) - ON CONFLICT (workos_user_id) DO NOTHING`, - [newWorkosUser.id, newWorkosUser.email, newWorkosUser.firstName, newWorkosUser.lastName, - newWorkosUser.emailVerified, newWorkosUser.createdAt, newWorkosUser.updatedAt] - ); - - // Merge: moves zero data rows (the new user has nothing), rebinds the - // new user's identity_workos_users row to the existing user's identity - // as is_primary = FALSE, drops the new user's orphan singleton identity. - await mergeUsers(existingUserId, newWorkosUser.id, adminUserId); - } catch (err) { - logger.error( - { err, newWorkosUserId: newWorkosUser.id, existingUserId }, - 'Admin bind-email: local bind failed after WorkOS createUser succeeded — rolling back the WorkOS user' - ); - // Best-effort cleanup so a retry doesn't leave an orphan in WorkOS. - // If the delete itself fails, surface the WorkOS id for manual cleanup. - let cleanedUp = false; - try { - await workos.userManagement.deleteUser(newWorkosUser.id); - cleanedUp = true; - } catch (deleteErr) { - logger.error( - { err: deleteErr, newWorkosUserId: newWorkosUser.id }, - 'Admin bind-email: failed to roll back WorkOS user after local-bind failure' - ); - } - return res.status(500).json({ - error: 'Failed to bind sign-in email', - message: cleanedUp - ? 'The new WorkOS user was rolled back. Please retry.' - : 'Please contact engineering — a WorkOS user was created at the new email but is not yet linked, and rollback failed.', - ...(cleanedUp ? {} : { new_workos_user_id: newWorkosUser.id }), - }); + const email = rawEmail.trim().toLowerCase(); + if (email.length > 255 || !isPlausibleEmail(email)) { + return res.status(400).json({ error: 'Invalid email address' }); } - - logger.info( - { adminEmail, existingUserId, newEmail, newWorkosUserId: newWorkosUser.id }, - 'Admin bound new sign-in email to existing user' - ); - - return res.status(201).json({ - bound: true, - existing_user_id: existingUserId, - new_email: newEmail, - new_workos_user_id: newWorkosUser.id, - message: `${newEmail} is now a sign-in email for this user. They can sign in with either address.`, + const result = await createAndBindAdminCredential({ + hostUserId: req.params.userId, + email, + actorUserId: req.user.authWorkosUserId ?? req.user.id, + actorIdentityId: req.user.identityId, }); + return res.status(result.status).json(result.body); }); // GET /api/admin/users/:userId/credentials @@ -1009,14 +890,17 @@ export function createAdminUsersRouter(): Router { workos_user_id: string; is_primary: boolean; bound_at: string; + authorization_epoch: string; email: string | null; first_name: string | null; last_name: string | null; }>( `SELECT iwu.workos_user_id, iwu.is_primary, iwu.bound_at, + COALESCE(ae.epoch, 0)::text AS authorization_epoch, u.email, u.first_name, u.last_name FROM identity_workos_users iwu LEFT JOIN users u ON u.workos_user_id = iwu.workos_user_id + LEFT JOIN authorization_epochs ae ON ae.workos_user_id = iwu.workos_user_id WHERE iwu.identity_id = $1 ORDER BY iwu.is_primary DESC, iwu.bound_at ASC`, [identity.rows[0].identity_id] @@ -1041,7 +925,7 @@ export function createAdminUsersRouter(): Router { // its own app-state, mergeUsers moves that data to this user — admin is // asserting the two represent the same person. The trust model and // confirmation UX live on the admin frontend. - router.post('/:userId/credentials', ...requireGlobalAdmin, async (req, res) => { + router.post('/:userId/credentials', ...requireGlobalAdmin, refuseIdentityConsolidation, async (req, res) => { const adminEmail = req.user!.email; const adminUserId = req.user!.id; const adminAuthCredentialId = req.user!.authWorkosUserId ?? req.user!.id; @@ -1234,14 +1118,13 @@ export function createAdminUsersRouter(): Router { // separately via the WorkOS Dashboard if desired. // // Refuses if the credential is the primary — removing the primary would - // leave the identity with no canonical credential. Promote another - // credential to primary first (separate endpoint, not yet built). + // leave the identity with no canonical credential. Primary promotion + // is a separate operation. router.delete('/:userId/credentials/:credentialId', ...requireGlobalAdmin, async (req, res) => { const adminEmail = req.user!.email; const adminUserId = req.user!.id; - // For non-singleton admin identities (today: nobody — admins are still - // singleton-bound — but Phase 3+ may change), record the auth credential - // separately from the canonical id so forensics can tell them apart. + // Record the authenticated credential separately from the canonical user + // so the audit identifies the credential that actually initiated detach. const adminAuthCredentialId = req.user!.authWorkosUserId ?? req.user!.id; const userId = req.params.userId; const credId = req.params.credentialId; @@ -1250,95 +1133,59 @@ export function createAdminUsersRouter(): Router { return res.status(400).json({ error: 'Cannot remove the canonical user via this endpoint' }); } - const pool = getPool(); - const client = await pool.connect(); - try { - await client.query('BEGIN'); - - const check = await client.query<{ is_primary: boolean; identity_id: string }>( - `SELECT iwu.is_primary, iwu.identity_id - FROM identity_workos_users iwu - WHERE iwu.workos_user_id = $1 - AND iwu.identity_id = ( - SELECT identity_id FROM identity_workos_users WHERE workos_user_id = $2 - )`, - [credId, userId] - ); + if (req.adminAccessMechanism === 'static_admin_api_key' || !req.user!.identityId) { + return res.status(403).json({ error: 'identity_bearing_admin_required' }); + } + const expectedIdentityId = req.body?.expected_identity_id; + const expectedAuthorizationEpoch = req.body?.expected_authorization_epoch; + if (typeof expectedIdentityId !== 'string' || + !/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/.test(expectedIdentityId) || + typeof expectedAuthorizationEpoch !== 'string' || + !/^(0|[1-9][0-9]{0,18})$/.test(expectedAuthorizationEpoch)) { + return res.status(400).json({ + error: 'binding_precondition_required', + message: 'Reload credentials and supply the observed identity and authorization epoch.', + }); + } - if (check.rows.length === 0) { - await client.query('ROLLBACK'); - return res.status(404).json({ error: 'Credential not bound to this user' }); - } - if (check.rows[0].is_primary) { - await client.query('ROLLBACK'); + try { + const detached = await detachCredential({ + hostUserId: userId, + credentialId: credId, + expectedIdentityId, + expectedAuthorizationEpoch, + actorUserId: adminUserId, + actorCredentialId: adminAuthCredentialId, + actorIdentityId: req.user!.identityId, + }); + invalidateSessionsForUsers(detached.affectedCredentialIds); + } catch (err) { + if (err instanceof CredentialDetachConflict) { + // Logger hooks run before redaction. Retain only helper-owned constant + // reasons, never an exception object, stack, request or provider data. + const reason = [ + 'Identity binding is busy; reload and retry', + 'Identity changed; reload credentials', + 'Credential lifecycle is busy; reload and retry', + 'Acting identity changed; authenticate again', + 'Credential binding changed; reload credentials', + 'Cannot remove the primary credential', + 'Identity has no primary credential; repair it before detaching', + 'Credential lifecycle changed; reload credentials', + 'Credential lifecycle requires reconciliation', + 'Credential lifecycle is terminal or requires reconciliation', + 'Credential authorization changed; reload credentials', + ].find((knownReason) => knownReason === err.message) ?? '[REDACTED]'; + logger.warn({ failureCode: 'credential_binding_changed', reason }, 'Admin unbind-credential: conflict'); return res.status(409).json({ - error: 'Cannot remove the primary credential', - message: 'Promote another credential to primary before removing this one.', + error: 'credential_binding_changed', + message: 'Credential binding changed. Reload credentials and try again.', }); } - - const detachedIdentityId = check.rows[0].identity_id; - - // Unbind, then create a fresh singleton identity for the detached - // credential so the Phase 1 invariant ("every user has exactly one - // binding") holds. - await client.query( - `DELETE FROM identity_workos_users WHERE workos_user_id = $1`, - [credId] - ); - const newIdentity = await client.query<{ id: string }>( - `INSERT INTO identities DEFAULT VALUES RETURNING id` - ); - await client.query( - `INSERT INTO identity_workos_users (workos_user_id, identity_id, is_primary) - VALUES ($1, $2, TRUE)`, - [credId, newIdentity.rows[0].id] - ); - - // The detached credential stops routing to the host, so any session - // issued before this commit must lose that routing on every instance, - // not just the one handling this request. - await bumpAuthorizationEpochs(client, [userId, credId]); - - // Audit log - const auditOrg = await client.query<{ workos_organization_id: string }>( - `SELECT workos_organization_id FROM organization_memberships - WHERE workos_user_id = $1 LIMIT 1`, - [userId] - ); - const auditOrgId = auditOrg.rows[0]?.workos_organization_id || 'system'; - await client.query( - `INSERT INTO registry_audit_log ( - workos_organization_id, workos_user_id, action, resource_type, resource_id, details - ) VALUES ($1, $2, 'unbind_credential', 'user', $3, $4)`, - [ - auditOrgId, - adminUserId, - credId, - JSON.stringify({ - host_user_id: userId, - detached_from_identity_id: detachedIdentityId, - new_identity_id: newIdentity.rows[0].id, - // Auth credential the admin used (may differ from adminUserId - // post-id-swap if the admin has multiple bound credentials). - acting_workos_user_id: adminAuthCredentialId, - }), - ] - ); - - await client.query('COMMIT'); - } catch (err) { - await client.query('ROLLBACK'); - logger.error({ err, userId, credId }, 'Admin unbind-credential: failed'); + logger.error({ failureCode: 'credential_detach_failed', reason: '[REDACTED]' }, 'Admin unbind-credential: failed'); return res.status(500).json({ error: 'Failed to unbind credential' }); - } finally { - client.release(); } - // Invalidate cached sessions for both users so the swap is recomputed - // on the next request. - invalidateSessionsForUsers([userId, credId]); - logger.info( { adminEmail, userId, credId }, 'Admin unbound credential from identity' @@ -1348,263 +1195,13 @@ export function createAdminUsersRouter(): Router { removed: true, host_user_id: userId, removed_workos_user_id: credId, - message: 'Credential unbound. The WorkOS user is now a separate identity.', + message: 'Credential unbound. The WorkOS user is now a separate identity. Existing memberships and permissions are unchanged.', }); }); - // POST /api/admin/users/:userId/credentials/:credentialId/promote - // - // Make :credentialId the primary credential of the host's identity. - // Moves all of the current primary's app-state forward to :credentialId - // (so reads keyed on the canonical workos_user_id land on the right - // place), swaps `is_primary`, audit row. - // - // Use case: after a `link-existing` bind, the new credential ended up as - // the right one for the workspace the person actually wants (e.g., a - // work email that's a member of a paid org), but the canonical primary - // sits on a different credential whose org_memberships are a different - // (personal) workspace. Promote re-points the canonical so id-swap - // routes both sign-ins to the org-bearing credential. - // - // Implementation note: we run mergeUsers(newPrimary, currentPrimary) - // which moves data forward and demotes the old primary as a side - // effect (it becomes is_primary=FALSE). Both bindings are non-primary - // for a brief window between the mergeUsers commit and the follow-up - // UPDATE; during that window `attachIdentityId` finds no primary and - // skips the id-swap, so requests fall back to the auth user's slice of - // data — degraded but not broken. A failure of the follow-up UPDATE - // would persist that degraded state; the audit row records the intent - // and the recovery is a one-line UPDATE. - router.post('/:userId/credentials/:credentialId/promote', ...requireGlobalAdmin, async (req, res) => { - const adminEmail = req.user!.email; - const adminUserId = req.user!.id; - const adminIdentityId = req.user!.identityId; - const adminAuthCredentialId = req.user!.authWorkosUserId ?? req.user!.id; - const userId = req.params.userId; - const newPrimaryId = req.params.credentialId; - - if (newPrimaryId === userId) { - return res.status(400).json({ error: 'The credential to promote must differ from the host id in the URL' }); - } - - const pool = getPool(); - - // Validate: target is bound to host's identity, find current primary - // and target's email (for the audit row + caller display). - const check = await pool.query<{ - new_is_primary: boolean; - current_primary_id: string | null; - identity_id: string; - target_email: string | null; - }>( - `SELECT - target.is_primary AS new_is_primary, - primary_iwu.workos_user_id AS current_primary_id, - target.identity_id, - target_user.email AS target_email - FROM identity_workos_users target - LEFT JOIN identity_workos_users primary_iwu - ON primary_iwu.identity_id = target.identity_id - AND primary_iwu.is_primary = TRUE - LEFT JOIN users target_user - ON target_user.workos_user_id = target.workos_user_id - WHERE target.workos_user_id = $1 - AND target.identity_id = ( - SELECT identity_id FROM identity_workos_users WHERE workos_user_id = $2 - )`, - [newPrimaryId, userId] - ); - - if (check.rows.length === 0) { - return res.status(404).json({ error: 'Credential not bound to this user' }); - } - if (check.rows[0].new_is_primary) { - return res.json({ promoted: true, message: 'Already primary — no change.' }); - } - - const identityId = check.rows[0].identity_id; - const currentPrimaryId = check.rows[0].current_primary_id; - const targetEmail = check.rows[0].target_email; - - // Refuse self-promote: an admin shouldn't mutate their own identity via - // this admin endpoint (it would shuffle their own session's app-state - // mid-request). If they need to promote one of their own credentials, - // they sign in as the target person and use the user-facing flow (or - // another admin handles it). - if (adminIdentityId && adminIdentityId === identityId) { - return res.status(409).json({ - error: 'Cannot promote your own credential', - message: 'This identity belongs to the signed-in admin. Have a different admin perform the promote.', - }); - } - - // Edge case: identity has no current primary (broken invariant from a - // prior partial promote, manual SQL, etc.). Just set the target as - // primary; nothing to move forward. - if (!currentPrimaryId) { - const repairClient = await pool.connect(); - try { - await repairClient.query('BEGIN'); - await repairClient.query( - `UPDATE identity_workos_users SET is_primary = TRUE WHERE workos_user_id = $1`, - [newPrimaryId] - ); - // Same transaction as the primary flip: a bump that commits - // separately leaves a window where the routing changed but stale - // sessions still validate. Every credential on the identity is - // bumped, not just the new primary — the flip changes where each - // sibling's canonical reads land. - const repairBound = await repairClient.query<{ workos_user_id: string }>( - `SELECT workos_user_id FROM identity_workos_users WHERE identity_id = $1`, - [identityId] - ); - await bumpAuthorizationEpochs(repairClient, [ - newPrimaryId, - ...repairBound.rows.map((row) => row.workos_user_id), - ]); - await repairClient.query('COMMIT'); - } catch (err) { - await repairClient.query('ROLLBACK').catch(() => undefined); - logger.error({ err, userId, newPrimaryId }, 'Promote: orphan-primary repair failed'); - return res.status(500).json({ error: 'Failed to promote credential' }); - } finally { - repairClient.release(); - } - logger.info( - { adminEmail, userId, newPrimaryId, identityId, recovered_orphan: true }, - 'Promote: identity had no current primary; set target as primary directly' - ); - invalidateSessionsForUsers([newPrimaryId]); - return res.json({ - promoted: true, - message: 'Promoted (no current primary to demote — invariant repaired).', - }); - } - - // A corporate credential must remain canonical once it carries an - // entitled corporate membership. Otherwise an operator could safely bind - // a personal credential onto it, then promote that credential and move - // the corporate membership onto the personal account. - const currentMembershipRows = await pool.query( - `SELECT o.workos_organization_id, o.membership_tier, - o.subscription_price_lookup_key, o.subscription_status, - o.subscription_amount, o.subscription_interval, o.is_personal, - o.subscription_canceled_at - FROM organization_memberships om - JOIN organizations o - ON o.workos_organization_id = om.workos_organization_id - WHERE om.workos_user_id = $1 - AND o.subscription_canceled_at IS NULL - ORDER BY o.workos_organization_id`, - [currentPrimaryId], - ); - const corporateMemberships = resolveEntitledMemberships(currentMembershipRows.rows) - .filter((membership) => isCorporateMembershipTier(membership.membership_tier)); - if (corporateMemberships.length > 0) { - return res.status(409).json({ - error: 'corporate_membership_primary_required', - message: 'Cannot promote another credential while the current primary holds an entitled corporate membership. Keep the corporate credential canonical.', - corporate_memberships: corporateMemberships, - }); - } - - // Foot-gun gate: promote runs the same consolidation as link-credential. - // Memberships whose partner key the incoming credential already holds are - // DELETED by it — the outgoing row's role, seat, upstream membership id, - // and join provenance are not recoverable, and an unlink cannot put them - // back. Require stated intent, matching the `consolidate: true` gate on - // the bind path above. - if (req.body?.consolidate !== true) { - const superseded = await findSupersededMemberships(currentPrimaryId, newPrimaryId); - const supersededCount = - superseded.organizationIds.length + superseded.workingGroupIds.length; - if (supersededCount > 0) { - return res.status(409).json({ - error: 'Promoting would delete memberships', - message: `The outgoing primary holds ${supersededCount} membership(s) whose organization or working group the incoming credential already belongs to. Promoting deletes those rows — their role, seat, WorkOS membership id, and join provenance cannot be recovered. Certification progress, credentials, badges, and committee interest are consolidated in the same operation and deduplicated on conflict; they are not enumerated here. Re-submit with \`"consolidate": true\` to confirm this is intended.`, - consolidate_confirmation_required: true, - superseded_organization_ids: superseded.organizationIds, - superseded_working_group_ids: superseded.workingGroupIds, - }); - } - } - - if (req.body?.consolidate === true && req.adminAccessMechanism === 'static_admin_api_key') { - return res.status(403).json({ - error: 'identity_bearing_admin_required', - message: 'Destructive consolidation must be confirmed by an identity-bearing SSO admin session; the static admin API key cannot provide individual actor attribution.', - }); - } - - // Run mergeUsers with ensurePrimaryFlag so the data move, the secondary - // rebind, AND the new primary's is_primary=TRUE flip all happen in one - // transaction. This closes the window where the identity has zero - // primaries. - try { - await mergeUsers(newPrimaryId, currentPrimaryId, adminUserId, { - ensurePrimaryFlag: true, - auditContext: { - acting_workos_user_id: adminAuthCredentialId, - admin_access_mechanism: req.adminAccessMechanism ?? null, - consolidation_confirmed: req.body?.consolidate === true, - }, - }); - } catch (err) { - logger.error( - { err, userId, newPrimaryId, currentPrimaryId }, - 'Promote: mergeUsers failed' - ); - return res.status(500).json({ error: 'Failed to promote credential' }); - } - - // Audit row. mergeUsers writes its own merge_user audit; this adds the - // promote-specific record with target email + identity context. Failure - // here doesn't unwind the promote (the data + primary swap are - // committed) — log loud so we notice. - try { - const auditOrg = await pool.query<{ workos_organization_id: string }>( - `SELECT workos_organization_id FROM organization_memberships - WHERE workos_user_id = $1 LIMIT 1`, - [newPrimaryId] - ); - const auditOrgId = auditOrg.rows[0]?.workos_organization_id || 'system'; - await pool.query( - `INSERT INTO registry_audit_log ( - workos_organization_id, workos_user_id, action, resource_type, resource_id, details - ) VALUES ($1, $2, 'promote_credential_to_primary', 'user', $3, $4)`, - [ - auditOrgId, - adminUserId, - newPrimaryId, - JSON.stringify({ - host_user_id: userId, - identity_id: identityId, - previous_primary_id: currentPrimaryId, - new_primary_id: newPrimaryId, - target_email: targetEmail, - acting_workos_user_id: adminAuthCredentialId, - }), - ] - ); - } catch (err) { - logger.error({ err, userId, newPrimaryId }, 'Promote: audit row insert failed (operation already committed)'); - } - - invalidateSessionsForUsers([userId, newPrimaryId, currentPrimaryId]); - - logger.info( - { adminEmail, identityId, previous_primary_id: currentPrimaryId, new_primary_id: newPrimaryId }, - 'Admin promoted credential to primary' - ); - - return res.json({ - promoted: true, - identity_id: identityId, - previous_primary_id: currentPrimaryId, - new_primary_id: newPrimaryId, - message: 'Credential is now primary. Sign-ins via either bound credential will route here.', - }); - }); + // Primary promotion currently consolidates state and rewrites membership + // provenance. Contain this admin surface until a preserving operation exists. + router.post('/:userId/credentials/:credentialId/promote', ...requireGlobalAdmin, refuseIdentityConsolidation); return router; } diff --git a/server/src/routes/identity-mutation-containment.ts b/server/src/routes/identity-mutation-containment.ts new file mode 100644 index 00000000000..67db3eb793b --- /dev/null +++ b/server/src/routes/identity-mutation-containment.ts @@ -0,0 +1,8 @@ +import type { RequestHandler } from 'express'; +import { IdentityMutationDisabledError } from '../db/identity-mutation-policy.js'; + +/** Refuse after authentication, before any provider or local mutation. */ +export const refuseIdentityConsolidation: RequestHandler = (_req, res) => { + const error = new IdentityMutationDisabledError(); + res.status(409).json({ error: error.code, message: error.message }); +}; diff --git a/server/src/routes/workos-webhooks.ts b/server/src/routes/workos-webhooks.ts index 3e35d58cc50..a99b32e39c1 100644 --- a/server/src/routes/workos-webhooks.ts +++ b/server/src/routes/workos-webhooks.ts @@ -19,17 +19,23 @@ */ import { Router, Request, Response } from 'express'; +import type { PoolClient } from 'pg'; import { createLogger } from '../logger.js'; import { getPool } from '../db/client.js'; -import { invalidateSessionsForUsers } from '../middleware/auth.js'; -import { promoteSecondaryIfPrimaryDeleted } from '../db/identity-db.js'; +import { deleteIdentityCredential } from '../services/identity-credential-deletion.js'; +import { + upsertWorkosUserInCredentialEvent, + upsertWorkosUserUnlessConfirmedDeleted, + withActiveCredentialEventMutation, + withCredentialCreationEventMutation, +} from '../db/identity-db.js'; import { upsertWorkosDomain, autoPromotePrimaryIfNone, removeWorkosDomainAndReselectPrimary, } from '../db/organization-domains-db.js'; import { BrandDatabase } from '../db/brand-db.js'; -import { getWorkos, getOwnerlessPromotionWorkos } from '../auth/workos-client.js'; +import { getWorkos } from '../auth/workos-client.js'; import { invalidateUnifiedUsersCache } from '../cache/unified-users.js'; import { tryAutoLinkWebsiteUserToSlack } from '../slack/sync.js'; import { resolveUserNameWithFallbacks } from '../utils/resolve-user-name.js'; @@ -40,7 +46,6 @@ import { notifyBrandClaimOpportunity } from '../notifications/registry.js'; import { getNudgeDismissal, recordNudgeDismissal } from '../db/user-nudges-db.js'; import { getCompanyDomain } from '../utils/email-domain.js'; import { canonicalizeBrandDomain, assertClaimableBrandDomain } from '../services/identifier-normalization.js'; -import { resolvePreferredOrganization, backfillPrimaryOrganization } from '../db/users-db.js'; import { notifySystemError } from '../addie/error-notifier.js'; import { canAddSeat, type SeatType, OrganizationDatabase } from '../db/organization-db.js'; import { sendToOrgAdmins, escapeSlackMrkdwn } from '../slack/org-group-dm.js'; @@ -48,10 +53,6 @@ import { upsertOrganizationMembership, deleteOrganizationMembership, consumeInvitationSeatType, - findSuccessorForPromotion, - setMembershipRole, - autoLinkByVerifiedDomain, - resolveRoleWithWorkosFirstPromote, } from '../db/membership-db.js'; import { boundedRawJson, type RawJsonRequest } from '../middleware/bounded-raw-json.js'; @@ -81,6 +82,42 @@ interface UserData { updated_at: string; } +const PROVIDER_PREFETCH_TIMEOUT_MS = Number( + process.env.WORKOS_PROVIDER_PREFETCH_TIMEOUT_MS ?? 5_000, +); + +async function boundedProviderPrefetch(operation: Promise): Promise { + let timeout: ReturnType | undefined; + try { + return await Promise.race([ + operation, + new Promise((_resolve, reject) => { + timeout = setTimeout( + () => reject(new Error('Provider prefetch timed out')), + PROVIDER_PREFETCH_TIMEOUT_MS, + ); + }), + ]); + } finally { + if (timeout) clearTimeout(timeout); + } +} + +async function fetchMembershipUser(membership: OrganizationMembershipData): Promise { + const workosUser = await boundedProviderPrefetch( + getWorkos().userManagement.getUser(membership.user_id), + ); + return { + id: workosUser.id, + email: workosUser.email, + first_name: workosUser.firstName, + last_name: workosUser.lastName, + email_verified: workosUser.emailVerified, + created_at: workosUser.createdAt, + updated_at: workosUser.updatedAt, + }; +} + interface OrganizationDomainData { domain: string; state: 'verified' | 'pending'; @@ -195,34 +232,17 @@ async function notifyAdminsOfRefusedMembership(input: { async function upsertMembership( membership: OrganizationMembershipData, - user?: UserData + user: UserData, + client: PoolClient, ): Promise { - // If we don't have user data, fetch it from WorkOS - let userData = user; - if (!userData) { - try { - const workosUser = await getWorkos().userManagement.getUser(membership.user_id); - userData = { - id: workosUser.id, - email: workosUser.email, - first_name: workosUser.firstName, - last_name: workosUser.lastName, - email_verified: workosUser.emailVerified, - created_at: workosUser.createdAt, - updated_at: workosUser.updatedAt, - }; - } catch (error) { - logger.error({ error, userId: membership.user_id }, 'Failed to fetch user from WorkOS'); - return; - } - } + const userData = user; if (membership.status !== 'active') { logger.info( { membershipId: membership.id, status: membership.status, userId: membership.user_id, orgId: membership.organization_id }, 'Removing non-active organization membership from local cache', ); - await deleteInactiveMembershipCache(membership); + await deleteInactiveMembershipCache(membership, client); return; } @@ -232,19 +252,24 @@ async function upsertMembership( // auto-promote this user (ownerless-org safety net), push the change to // WorkOS first and only then write the resolved role locally. WorkOS is // the source of truth — local must never get ahead of it. - const resolution = await resolveRoleWithWorkosFirstPromote({ - workos: getOwnerlessPromotionWorkos(), - membershipId: membership.id, - userId: membership.user_id, - organizationId: membership.organization_id, - incomingRole, - }); + // Provider-side owner promotion cannot be made exactly-once without a + // durable event/intent journal. Preserve the signed WorkOS role locally and + // defer ownerless reconciliation rather than perform an upstream authority + // write whose outcome could become unknown during deletion. + const resolution: { role: string; promoted: boolean; promotionError?: unknown } = { + role: incomingRole, + promoted: false, + }; // Consume any pending seat_type + provisioning_source staged by the // endpoint that triggered this membership creation. Falls back to defaults // when no row was staged (e.g. someone added the membership directly in // WorkOS rather than through one of our endpoints). - const consumed = await consumeInvitationSeatType(membership.organization_id, userData.email); + const consumed = await consumeInvitationSeatType( + membership.organization_id, + userData.email, + client, + ); const hasExplicitSeatType = consumed !== null; const seatType: SeatType = consumed?.seat_type === 'contributor' ? 'contributor' : 'community_only'; const provisioningSource = consumed?.source || 'webhook'; @@ -258,7 +283,7 @@ async function upsertMembership( // and notify so a multi-user company can't squeeze onto a 1-seat // individual sub by adding members in WorkOS directly. if (!hasExplicitSeatType) { - const availability = await canAddSeat(membership.organization_id, seatType); + const availability = await canAddSeat(membership.organization_id, seatType, client); if (!availability.allowed) { logger.warn({ orgId: membership.organization_id, @@ -288,7 +313,7 @@ async function upsertMembership( seat_type: seatType, has_explicit_seat_type: hasExplicitSeatType, provisioning_source: provisioningSource, - }); + }, client); // Audit-log both outcomes of the ownerless-org auto-promote path so future // role-drift questions have a paper trail in registry_audit_log (and we @@ -340,17 +365,6 @@ async function upsertMembership( } } - // Set primary_organization_id if not already set (prefer paying orgs). - // Best-effort — same rationale as upsertUser: a transient backfill failure - // shouldn't fail the membership webhook. Integrity invariant catches drift. - try { - const preferredOrg = await resolvePreferredOrganization(membership.user_id); - if (preferredOrg) { - await backfillPrimaryOrganization(membership.user_id, preferredOrg); - } - } catch (err) { - logger.warn({ err, userId: membership.user_id }, 'primary_organization_id backfill failed during membership upsert'); - } } /** @@ -360,8 +374,15 @@ async function upsertMembership( * remove local access immediately, but it must not run owner-succession logic: * promotion is reserved for explicit organization_membership.deleted events. */ -async function deleteInactiveMembershipCache(membership: OrganizationMembershipData): Promise { - const deletedRole = await deleteOrganizationMembership(membership.user_id, membership.organization_id); +async function deleteInactiveMembershipCache( + membership: OrganizationMembershipData, + client: PoolClient, +): Promise { + const deletedRole = await deleteOrganizationMembership( + membership.user_id, + membership.organization_id, + client, + ); logger.info({ membershipId: membership.id, @@ -385,51 +406,17 @@ async function deleteMembership(membership: OrganizationMembershipData): Promise role: deletedRole, }, 'Deleted organization membership'); - // If an admin/owner was removed, check if the org still has one. - // Promote the longest-tenured remaining member to prevent ownerless orgs. + // Do not automatically promote a successor in WorkOS. Without a durable + // provider-intent journal, deletion can race the upstream call and leave an + // unknown authority outcome that is unsafe to replay. The signed deletion + // remains locally revoked; operators reconcile an ownerless organization + // through the explicit administration workflow. if (deletedRole === 'admin' || deletedRole === 'owner') { - try { - const target = await findSuccessorForPromotion(membership.organization_id); - if (!target) return; - - // Promote in WorkOS first, then mirror locally - let promotedInWorkos = false; - if (target.workos_membership_id) { - await getWorkos().userManagement.updateOrganizationMembership( - target.workos_membership_id, - { roleSlug: 'owner' } - ); - promotedInWorkos = true; - } else { - // No cached membership ID — look it up from WorkOS - const memberships = await getWorkos().userManagement.listOrganizationMemberships({ - organizationId: membership.organization_id, - userId: target.workos_user_id, - }); - if (memberships.data.length > 0) { - await getWorkos().userManagement.updateOrganizationMembership( - memberships.data[0].id, - { roleSlug: 'owner' } - ); - promotedInWorkos = true; - } else { - logger.warn({ - orgId: membership.organization_id, - userId: target.workos_user_id, - }, 'Successor has no WorkOS membership — cannot promote, org may be ownerless'); - } - } - if (promotedInWorkos) { - await setMembershipRole(target.workos_user_id, membership.organization_id, 'owner'); - logger.info({ - orgId: membership.organization_id, - promotedUserId: target.workos_user_id, - previousOwnerId: membership.user_id, - }, 'Promoted longest-tenured member to owner after admin/owner removal'); - } - } catch (err) { - logger.warn({ err, orgId: membership.organization_id }, 'Failed to promote successor after owner removal'); - } + logger.warn({ + orgId: membership.organization_id, + previousOwnerId: membership.user_id, + recoveryState: 'manual_owner_selection_required', + }, 'Automatic successor promotion deferred: no durable provider intent journal'); } } @@ -437,42 +424,33 @@ async function deleteMembership(membership: OrganizationMembershipData): Promise * Upsert user to local users table * Called on user.created and user.updated events */ -async function upsertUser(user: UserData): Promise { +async function upsertUser(user: UserData, credentialClient?: PoolClient): Promise { const pool = getPool(); const { firstName, lastName } = await resolveUserNameWithFallbacks( - pool, user.id, user.first_name, user.last_name, + credentialClient ?? pool, user.id, user.first_name, user.last_name, ); - await pool.query( - `INSERT INTO users ( - workos_user_id, - email, - first_name, - last_name, - email_verified, - workos_created_at, - workos_updated_at, - created_at, - updated_at - ) VALUES ($1, $2, $3, $4, $5, $6, $7, NOW(), NOW()) - ON CONFLICT (workos_user_id) DO UPDATE SET - email = EXCLUDED.email, - first_name = COALESCE(NULLIF(TRIM(EXCLUDED.first_name), ''), users.first_name), - last_name = COALESCE(NULLIF(TRIM(EXCLUDED.last_name), ''), users.last_name), - email_verified = EXCLUDED.email_verified, - workos_updated_at = EXCLUDED.workos_updated_at, - updated_at = NOW()`, - [ - user.id, - user.email, - firstName, - lastName, - user.email_verified, - user.created_at, - user.updated_at, - ] - ); + const normalizedUser = { + id: user.id, + email: user.email, + firstName, + lastName, + emailVerified: user.email_verified, + createdAt: user.created_at, + updatedAt: user.updated_at, + }; + const upserted = credentialClient + ? (await upsertWorkosUserInCredentialEvent( + credentialClient, + normalizedUser, + 'provider_authoritative', + ), true) + : await upsertWorkosUserUnlessConfirmedDeleted(normalizedUser); + if (!upserted) { + logger.warn({ userId: user.id }, 'Ignored stale WorkOS user event after confirmed deletion'); + return false; + } // Close the user.created vs organization_membership.created race: if a // membership webhook fired first, its backfill UPDATE was a no-op because @@ -483,31 +461,8 @@ async function upsertUser(user: UserData): Promise { // Failures here don't fail the user upsert — the integrity invariant // surfaces any pointer that doesn't get set, and the next authenticated // request opportunistically backfills via resolvePrimaryOrganization. - try { - const preferredOrg = await resolvePreferredOrganization(user.id); - if (preferredOrg) { - await backfillPrimaryOrganization(user.id, preferredOrg); - } - } catch (err) { - logger.warn({ err, userId: user.id }, 'primary_organization_id backfill failed during user upsert'); - } - logger.info({ userId: user.id, email: user.email }, 'Upserted user'); -} - -/** - * Delete user from local users table - * Called on user.deleted events - */ -async function deleteUser(userId: string): Promise { - const pool = getPool(); - - await pool.query( - `DELETE FROM users WHERE workos_user_id = $1`, - [userId] - ); - - logger.info({ userId }, 'Deleted user'); + return true; } /** @@ -515,10 +470,8 @@ async function deleteUser(userId: string): Promise { * Reads the resolved name from the users table (which may have been enriched * from Slack) rather than using raw WorkOS data that might be empty. */ -async function updateUserAcrossMemberships(user: UserData): Promise { - const pool = getPool(); - - const result = await pool.query( +async function updateUserAcrossMemberships(user: UserData, client: PoolClient): Promise { + const result = await client.query( `UPDATE organization_memberships om SET email = $1, first_name = COALESCE(NULLIF(TRIM(u.first_name), ''), om.first_name), @@ -534,7 +487,7 @@ async function updateUserAcrossMemberships(user: UserData): Promise { // person_relationships denormalizes users.email too. Keep it in sync so // the admin person-detail header doesn't lag the canonical email after a // WorkOS-side update (dashboard edit, OIDC profile sync, primary swap). - await pool.query( + await client.query( `UPDATE person_relationships SET email = $1, updated_at = NOW() WHERE workos_user_id = $2 AND email IS DISTINCT FROM $1`, @@ -547,23 +500,6 @@ async function updateUserAcrossMemberships(user: UserData): Promise { }, 'Updated user details across memberships'); } -/** - * Delete all memberships for a user - */ -async function deleteUserMemberships(userId: string): Promise { - const pool = getPool(); - - const result = await pool.query( - `DELETE FROM organization_memberships WHERE workos_user_id = $1`, - [userId] - ); - - logger.info({ - userId, - deletedCount: result.rowCount, - }, 'Deleted all memberships for user'); -} - /** * Sync organization domains from WorkOS * This upserts domains and removes any that are no longer in WorkOS @@ -967,23 +903,18 @@ export function createWorkOSWebhooksRouter(): Router { switch (event.event) { case 'organization_membership.created': { const membership = event.data as unknown as OrganizationMembershipData; - await upsertMembership(membership); + const userData = await fetchMembershipUser(membership); + const guarded = await withActiveCredentialEventMutation(membership.user_id, async (client) => { + await upsertMembership(membership, userData, client); + }); // Try to auto-link to Slack account by email (in case user.created didn't catch it) - if (membership.status === 'active') { - let workosUser: any; - try { - workosUser = await getWorkos().userManagement.getUser(membership.user_id); - } catch (error) { - logger.debug({ error, userId: membership.user_id }, 'Could not fetch user for auto-link on membership'); - } - - if (workosUser) { + if (guarded.applied && membership.status === 'active') { // Slack auto-link try { - const linkResult = await tryAutoLinkWebsiteUserToSlack(membership.user_id, workosUser.email); + const linkResult = await tryAutoLinkWebsiteUserToSlack(membership.user_id, userData.email); if (linkResult.linked) { logger.info( - { userId: membership.user_id, email: workosUser.email, slackUserId: linkResult.slack_user_id }, + { userId: membership.user_id, email: userData.email, slackUserId: linkResult.slack_user_id }, 'Auto-linked website user to Slack account on membership creation' ); } @@ -996,19 +927,22 @@ export function createWorkOSWebhooksRouter(): Router { const { matchExpectationToUser } = await import('../db/certification-db.js'); const matched = await matchExpectationToUser( membership.organization_id, - workosUser.email, + userData.email, membership.user_id ); if (matched) { logger.info( - { userId: membership.user_id, email: workosUser.email, orgId: membership.organization_id }, + { userId: membership.user_id, email: userData.email, orgId: membership.organization_id }, 'Matched certification expectation to new org member' ); } } catch (certErr) { logger.warn({ error: certErr, userId: membership.user_id }, 'Could not match cert expectation on membership'); } - } + } + if (!guarded.applied) { + logger.warn({ userId: membership.user_id }, + 'Ignored stale WorkOS membership.created event after confirmed deletion'); } invalidateUnifiedUsersCache(); break; @@ -1016,7 +950,14 @@ export function createWorkOSWebhooksRouter(): Router { case 'organization_membership.updated': { const membership = event.data as unknown as OrganizationMembershipData; - await upsertMembership(membership); + const userData = await fetchMembershipUser(membership); + const guarded = await withActiveCredentialEventMutation(membership.user_id, async (client) => { + await upsertMembership(membership, userData, client); + }); + if (!guarded.applied) { + logger.warn({ userId: membership.user_id }, + 'Ignored stale WorkOS membership.updated event after confirmed deletion'); + } invalidateUnifiedUsersCache(); break; } @@ -1030,7 +971,10 @@ export function createWorkOSWebhooksRouter(): Router { case 'user.created': { const user = event.data as unknown as UserData; - await upsertUser(user); + const guarded = await withCredentialCreationEventMutation(user.id, async (client) => { + await upsertUser(user, client); + }); + if (guarded.applied) { // Try to auto-link to Slack account by email const linkResult = await tryAutoLinkWebsiteUserToSlack(user.id, user.email); if (linkResult.linked) { @@ -1043,18 +987,10 @@ export function createWorkOSWebhooksRouter(): Router { // Verified email is the trust gate: skip when WorkOS hasn't confirmed it yet // (the /api/me/* paths will retry after the user signs in and the email verifies). if (user.email_verified) { - try { - const linked = await autoLinkByVerifiedDomain(getWorkos(), user.id, user.email); - if (linked) { - logger.info( - { userId: user.id, email: user.email, orgId: linked.organizationId, role: linked.role }, - 'Auto-provisioned new user into verified-domain organization' - ); - } - } catch (linkErr) { - logger.warn({ err: linkErr, userId: user.id, email: user.email }, - 'Failed to auto-provision new user into verified-domain organization'); - } + logger.warn( + { userId: user.id }, + 'Verified-domain upstream auto-provision deferred: no durable exactly-once provider intent journal', + ); } // Fire-and-forget prospect triage + brand research for business emails. if (user.email) { @@ -1140,36 +1076,32 @@ export function createWorkOSWebhooksRouter(): Router { } } } + } + if (!guarded.applied) { + logger.warn({ userId: user.id }, 'Ignored stale WorkOS user.created event after confirmed deletion'); + } invalidateUnifiedUsersCache(); break; } case 'user.updated': { const user = event.data as unknown as UserData; - await upsertUser(user); - await updateUserAcrossMemberships(user); + const guarded = await withActiveCredentialEventMutation(user.id, async (client) => { + await upsertUser(user, client); + await updateUserAcrossMemberships(user, client); + }); + if (!guarded.applied) { + logger.warn({ userId: user.id }, 'Ignored stale WorkOS user.updated event after confirmed deletion'); + } invalidateUnifiedUsersCache(); break; } case 'user.deleted': { const user = event.data as unknown as UserData; - // Promote a surviving secondary BEFORE the CASCADE on - // identity_workos_users.workos_user_id fires (via deleteUser). - // Without this, the identity is left with zero primaries and the - // surviving secondary signs in to an empty workspace — a DoS - // vector reachable via GDPR/CCPA-driven WorkOS deletions. - const promoted = await promoteSecondaryIfPrimaryDeleted(user.id); - await deleteUser(user.id); - await deleteUserMemberships(user.id); - // Close the 60-second session/JWT cache window where a cached - // pre-deletion swap would still route reads to the dead binding. - // Invalidate both the deleted user and the promoted successor so - // the next request re-resolves identity from the DB. - const sessionsToInvalidate = [user.id]; - if (promoted) sessionsToInvalidate.push(promoted.promotedUserId); - invalidateSessionsForUsers(sessionsToInvalidate); - invalidateUnifiedUsersCache(); + // Provider deletion revokes this credential. Do not infer a new + // primary: that would redirect surviving credentials' authority. + await deleteIdentityCredential(user.id, 'workos_webhook'); break; } @@ -1317,18 +1249,25 @@ export async function backfillOrganizationMemberships(): Promise<{ if (membership && membership.status === 'active') { seenMemberships.add(`${user.id}:${org.workos_organization_id}`); const role = membership.role?.slug || 'member'; - await upsertOrganizationMembership({ - user_id: user.id, - organization_id: org.workos_organization_id, - membership_id: membership.id, - email: user.email, - first_name: user.firstName, - last_name: user.lastName, - role, - seat_type: 'community_only', - has_explicit_seat_type: false, + const guarded = await withActiveCredentialEventMutation(user.id, async (client) => { + await upsertOrganizationMembership({ + user_id: user.id, + organization_id: org.workos_organization_id, + membership_id: membership.id, + email: user.email, + first_name: user.firstName, + last_name: user.lastName, + role, + seat_type: 'community_only', + has_explicit_seat_type: false, + }, client); }); - result.membershipsCreated++; + if (guarded.applied) { + result.membershipsCreated++; + } else { + logger.warn({ userId: user.id, orgId: org.workos_organization_id }, + 'Backfill: ignored stale membership after confirmed user deletion'); + } } } catch (memberError) { const msg = `Failed to process membership for user ${user.id}: ${memberError}`; @@ -1399,6 +1338,17 @@ export async function backfillOrganizationMemberships(): Promise<{ * any org), then lists users per org as a safety net. Removes local rows * for users confirmed deleted from WorkOS (verified via getUser). */ +export const BACKFILL_DELETION_MAX_ABSOLUTE = 100; +export const BACKFILL_DELETION_MIN_RATIO_COUNT = 10; +export const BACKFILL_DELETION_MAX_RATIO = 0.25; + +export function shouldAbortBackfillDeletion(localCount: number, candidateCount: number): boolean { + if (localCount <= 0 || candidateCount <= 0) return false; + return candidateCount > BACKFILL_DELETION_MAX_ABSOLUTE + || (candidateCount > BACKFILL_DELETION_MIN_RATIO_COUNT + && candidateCount / localCount > BACKFILL_DELETION_MAX_RATIO); +} + export async function backfillUsers(): Promise<{ usersProcessed: number; usersCreated: number; @@ -1426,23 +1376,13 @@ export async function backfillUsers(): Promise<{ processedUserIds.add(user.id); try { - await pool.query( - `INSERT INTO users ( - workos_user_id, email, first_name, last_name, - email_verified, workos_created_at, workos_updated_at, - created_at, updated_at - ) VALUES ($1, $2, $3, $4, $5, $6, $7, NOW(), NOW()) - ON CONFLICT (workos_user_id) DO UPDATE SET - email = EXCLUDED.email, - first_name = COALESCE(NULLIF(TRIM(EXCLUDED.first_name), ''), users.first_name), - last_name = COALESCE(NULLIF(TRIM(EXCLUDED.last_name), ''), users.last_name), - email_verified = EXCLUDED.email_verified, - workos_updated_at = EXCLUDED.workos_updated_at, - updated_at = NOW()`, - [user.id, user.email, user.firstName, user.lastName, - user.emailVerified, user.createdAt, user.updatedAt] - ); - result.usersCreated++; + const upserted = await upsertWorkosUserUnlessConfirmedDeleted(user); + if (upserted) { + result.usersCreated++; + } else { + result.usersSkipped++; + logger.warn({ userId: user.id }, 'Backfill: ignored stale user after confirmed deletion'); + } } catch (userError) { logger.warn({ error: userError, userId: user.id }, 'Backfill: failed to upsert user'); result.errors.push(`Failed to upsert user ${user.id}`); @@ -1532,47 +1472,53 @@ export async function backfillUsers(): Promise<{ logger.info({ count: candidates.length }, 'Backfill: verifying deletion candidates against WorkOS'); - // Process candidates sequentially to avoid pool connection deadlock. - // Each iteration may acquire a pool connection for the DELETE transaction, - // and the pool max is small (3). Using Promise.all with a batch size > pool - // max would cause all connections to be held simultaneously, blocking any - // new connect() calls and deadlocking. - for (const row of candidates) { - try { - // Confirm the user is actually gone from WorkOS before deleting - await getWorkos().userManagement.getUser(row.workos_user_id); - // User still exists in WorkOS — skip deletion - result.usersSkipped++; - } catch (getErr: any) { - if (getErr?.status === 404 || getErr?.code === 'entity_not_found') { - const client = await pool.connect(); - try { - await client.query('BEGIN'); - await client.query(`DELETE FROM organization_memberships WHERE workos_user_id = $1`, [row.workos_user_id]); - await client.query(`DELETE FROM users WHERE workos_user_id = $1`, [row.workos_user_id]); - await client.query('COMMIT'); - result.usersRemoved++; - logger.info({ userId: row.workos_user_id }, 'Backfill: removed user confirmed deleted from WorkOS'); - } catch (err) { - await client.query('ROLLBACK'); - // FK constraints prevent deletion of users with platform activity - // (community_points, certifications, etc.) — this is expected + if (shouldAbortBackfillDeletion(localUsers.rows.length, candidates.length)) { + const message = `Deletion circuit breaker: refused ${candidates.length} candidates out of ${localUsers.rows.length} local users`; + result.errors.push(message); + result.usersSkipped += candidates.length; + logger.error({ + localUsers: localUsers.rows.length, + deletionCandidates: candidates.length, + }, 'Backfill: deletion circuit breaker refused an unsafe batch'); + } else { + // Process candidates sequentially to avoid pool connection deadlock. + // Each iteration may acquire a pool connection for the DELETE transaction, + // and the pool max is small (3). Using Promise.all with a batch size > pool + // max would cause all connections to be held simultaneously, blocking any + // new connect() calls and deadlocking. + for (const row of candidates) { + try { + // Confirm the user is actually gone from WorkOS before deleting + await getWorkos().userManagement.getUser(row.workos_user_id); + // User still exists in WorkOS — skip deletion + result.usersSkipped++; + } catch (getErr: any) { + if (getErr?.status === 404 || getErr?.code === 'entity_not_found') { + try { + const deletion = await deleteIdentityCredential(row.workos_user_id, 'sync_users_backfill'); + if (deletion.deleted) { + result.usersRemoved++; + logger.info({ userId: row.workos_user_id }, 'Backfill: removed user confirmed deleted from WorkOS'); + } else { + result.usersSkipped++; + } + } catch (err) { + result.usersSkipped++; + logger.warn({ error: err, userId: row.workos_user_id }, 'Backfill: confirmed user deletion failed locally'); + } + } else { + // WorkOS API error — don't delete, log full error server-side only + logger.warn({ error: getErr, userId: row.workos_user_id }, 'Backfill: WorkOS API error during user verification'); + result.errors.push(`Could not verify user ${row.workos_user_id}: WorkOS API error (status ${getErr?.status || 'unknown'})`); result.usersSkipped++; - logger.info({ error: err, userId: row.workos_user_id }, 'Backfill: user deleted from WorkOS but retained locally due to platform activity'); - } finally { - client.release(); } - } else { - // WorkOS API error — don't delete, log full error server-side only - logger.warn({ error: getErr, userId: row.workos_user_id }, 'Backfill: WorkOS API error during user verification'); - result.errors.push(`Could not verify user ${row.workos_user_id}: WorkOS API error (status ${getErr?.status || 'unknown'})`); - result.usersSkipped++; } } } } - // Invalidate cache after backfill + // Upserts need one final invalidation; confirmed deletions invalidate + // their session and unified-user caches inside deleteIdentityCredential. invalidateUnifiedUsersCache(); logger.info(result, 'Completed users backfill'); diff --git a/server/src/services/admin-credential-bind.ts b/server/src/services/admin-credential-bind.ts new file mode 100644 index 00000000000..5d7a1e23df7 --- /dev/null +++ b/server/src/services/admin-credential-bind.ts @@ -0,0 +1,406 @@ +import { createHash } from 'node:crypto'; +import type { Pool, PoolClient } from 'pg'; +import type { WorkOS } from '@workos-inc/node'; +import { getPool } from '../db/client.js'; +import { bumpAuthorizationEpochs } from '../db/authorization-epoch-db.js'; +import { + lockAdminCredentialCompensationQuarantine, + lockCredentialLifecyclesInTransaction, + recordAdminCredentialCompensationQuarantine, + recordConfirmedAdminCredentialCompensationDeletion, + type LockedCredentialLifecycle, +} from '../db/identity-db.js'; +import { getAdminCredentialMutationWorkos } from '../auth/workos-client.js'; +import { createLogger } from '../logger.js'; + +const logger = createLogger('admin-credential-bind'); +type Provider = Pick; +type Status = 'creating' | 'provider_created' | 'committed' | 'compensating' | + 'compensated' | 'provider_rejected' | 'reconciliation_required'; +interface Operation { + id: string; + host_user_id: string; + host_identity_id: string; + actor_user_id: string; + actor_identity_id: string; + provider_user_id: string | null; + status: Status; +} +interface Input { + hostUserId: string; + email: string; + actorUserId: string; + actorIdentityId: string; +} +interface Result { status: number; body: Record } + +// Only allowlisted fields reach the audit log; never serialize SDK errors, +// request/response bodies, email addresses, tokens, or credential material. +const auditSql = `INSERT INTO registry_audit_log + (workos_organization_id, workos_user_id, action, resource_type, resource_id, details) + SELECT 'system', actor_user_id, 'admin_credential_bind', + 'admin_credential_bind_operation', id::text, + jsonb_build_object('status', status, 'failure_code', failure_code, + 'acting_workos_user_id', actor_user_id, 'actor_identity_id', actor_identity_id, + 'host_user_id', host_user_id, 'host_identity_id', host_identity_id, + 'provider_user_id', provider_user_id) + FROM operation`; + +async function transition(db: Pick, op: Operation, + status: Status, failureCode: string | null = null): Promise { + const result = await db.query( + `WITH operation AS ( + UPDATE admin_credential_bind_operations + SET status = $2, failure_code = $3, provider_user_id = $4, updated_at = NOW() + WHERE id = $1 RETURNING * + ) ${auditSql}`, + [op.id, status, failureCode, op.provider_user_id], + ); + if (result.rowCount !== 1) throw new Error('Missing operation journal'); + op.status = status; +} + +function reconciliation(op?: Operation): Result { + return { status: 503, body: { + error: 'reconciliation_required', reconciliation_required: true, + message: 'The outcome requires engineering adjudication before another attempt.', + ...(op ? { operation_id: op.id, new_workos_user_id: op.provider_user_id } : {}), + } }; +} + +function success(op: Operation, email: string, replay = false): Result { + return { status: replay ? 200 : 201, body: { + bound: true, operation_id: op.id, existing_user_id: op.host_user_id, + new_email: email, new_workos_user_id: op.provider_user_id, + message: 'Sign-in credential bound. No organization memberships were transferred.', + } }; +} + +function active(state: LockedCredentialLifecycle | undefined): boolean { + return !!state && !state.terminal_marker && state.user_exists && state.binding_exists + && !!state.identity_id && Number(state.primary_count) === 1; +} + +function unseen(state: LockedCredentialLifecycle): boolean { + return !state.terminal_marker && !state.user_exists && !state.binding_exists + && state.identity_id === null && state.is_primary === null && Number(state.primary_count) === 0; +} + +/** + * Admin-only create-and-bind saga. The journal commits before WorkOS is called. + * This deliberately does not share merge/member-email transaction semantics: + * it never consolidates app state or writes organization memberships. + * + * A crash at any external boundary leaves a durable blocking state. No GET is + * used to infer the outcome of an in-flight request. Only a confirmed local + * rollback permits deleting the exact user returned by a successful create. + */ +export async function createAndBindAdminCredential(input: Input, dependencies: { + pool?: Pool; + workos?: Provider; +} = {}): Promise { + const pool = dependencies.pool ?? getPool(); + const email = input.email.trim().toLowerCase(); + const emailHash = createHash('sha256').update(email).digest('hex'); + const keys = [`admin-credential-bind:email:${emailHash}`, `admin-credential-bind:host:${input.hostUserId}`].sort(); + const held: string[] = []; + let client: PoolClient | undefined; + let op: Operation | undefined; + let destroy = false; + let result: Result = { status: 503, body: { error: 'credential_bind_unavailable' } }; + + async function rollback(): Promise { + if (!client) return false; + try { + if ((await client.query('ROLLBACK')).command === 'ROLLBACK') return true; + } catch { /* An unconfirmed rollback must never authorize compensation. */ } + client.release(true); + client = undefined; + return false; + } + + async function markUncertain(code: string): Promise { + if (op) { + try { await transition(client ?? pool, op, 'reconciliation_required', code); } + catch { + // Release a broken session before using a second connection, so a + // pool full of failing requests cannot deadlock its own evidence writes. + client?.release(true); + client = undefined; + try { await transition(pool, op, 'reconciliation_required', code); } + catch { + // The precommitted intent still blocks replay during a DB outage. + logger.error({ operationId: op.id, failureCode: 'journal_update_failed' }, + 'Admin credential binding requires reconciliation'); + } + } + } + return reconciliation(op); + } + + async function run(): Promise { + client = await pool.connect(); + for (const key of keys) { + try { + const lock = await client.query<{ locked: boolean }>( + 'SELECT pg_try_advisory_lock(hashtextextended($1, 0)) AS locked', [key], + ); + if (lock.rows[0]?.locked === false) { + return { status: 409, body: { error: 'credential_bind_in_progress' } }; + } + if (lock.rows[0]?.locked !== true) throw new Error('Unconfirmed lock acquisition'); + held.push(key); + } catch { + // The server may have granted a lock whose reply was lost. Never put + // that session back in the pool, even though it is not in held yet. + destroy = true; + throw new Error('Unconfirmed lock acquisition'); + } + } + + // Require a real credential and the identity observed by authentication. + const actor = await client.query( + `SELECT 1 FROM identity_workos_users WHERE workos_user_id = $1 AND identity_id = $2`, + [input.actorUserId, input.actorIdentityId], + ); + if (actor.rowCount !== 1) return { status: 403, body: { error: 'actor_identity_changed' } }; + + const prior = await client.query( + `SELECT * FROM admin_credential_bind_operations WHERE email_hash = $1 + AND status NOT IN ('compensated', 'provider_rejected')`, [emailHash], + ); + if (prior.rows[0]) { + const existing = prior.rows[0]; + op = existing; + if (existing.status === 'committed' && existing.host_user_id === input.hostUserId + && existing.provider_user_id) { + // A stable journal is not proof that the credential is still live. + // Serialize replay with the same lifecycle writers as a fresh bind. + try { + await client.query('BEGIN'); + await client.query("SET LOCAL lock_timeout = '5s'"); + const states = await lockCredentialLifecyclesInTransaction(client, + [input.actorUserId, input.hostUserId, existing.provider_user_id]); + const actorState = states.find(state => state.workos_user_id === input.actorUserId); + if (!states.every(active) || actorState?.identity_id !== input.actorIdentityId) { + throw new Error('Replay lifecycle changed'); + } + const bound = await client.query( + `SELECT 1 FROM identity_workos_users host + JOIN identity_workos_users credential ON credential.identity_id = host.identity_id + JOIN users u ON u.workos_user_id = credential.workos_user_id + WHERE host.workos_user_id = $1 AND host.identity_id = $2 AND host.is_primary + AND credential.workos_user_id = $3 AND NOT credential.is_primary AND LOWER(u.email) = $4`, + [input.hostUserId, existing.host_identity_id, existing.provider_user_id, email], + ); + if (bound.rowCount !== 1 || bound.rows.length !== 1) throw new Error('Replay binding changed'); + if ((await client.query('COMMIT')).command !== 'COMMIT') throw new Error('Unconfirmed replay commit'); + return success(existing, email, true); + } catch { + await rollback(); + return await markUncertain('committed_lifecycle_unavailable'); + } + } + return reconciliation(existing); + } + + const hostResult = await client.query<{ + email: string; first_name: string | null; last_name: string | null; + identity_id: string | null; is_primary: boolean | null; + }>( + `SELECT u.email, u.first_name, u.last_name, b.identity_id, b.is_primary + FROM users u LEFT JOIN identity_workos_users b USING (workos_user_id) + WHERE u.workos_user_id = $1`, [input.hostUserId], + ); + const host = hostResult.rows[0]; + if (!host) return { status: 404, body: { error: 'User not found' } }; + if (!host.identity_id || !host.is_primary) return { status: 409, body: { error: 'host_primary_required' } }; + if (host.email.toLowerCase() === email) return { status: 409, body: { error: 'This is already the user\'s primary email' } }; + const claimed = await client.query('SELECT 1 FROM users WHERE LOWER(email) = $1', [email]); + if (claimed.rowCount !== 0) return { status: 409, body: { error: 'This email already has an AAO account' } }; + + // Construct the client before recording intent; missing configuration has + // no provider outcome to adjudicate. SDK retries are disabled for mutations. + const workos = dependencies.workos ?? getAdminCredentialMutationWorkos().userManagement; + const intent = await client.query( + `WITH operation AS ( + INSERT INTO admin_credential_bind_operations + (email_hash, host_user_id, host_identity_id, actor_user_id, actor_identity_id, status) + VALUES ($1, $2, $3, $4, $5, 'creating') RETURNING * + ), audit AS (${auditSql}) SELECT * FROM operation`, + [emailHash, input.hostUserId, host.identity_id, input.actorUserId, input.actorIdentityId], + ); + op = intent.rows[0]; + if (!op) throw new Error('Missing operation intent'); + + let created: Awaited>; + try { + created = await workos.createUser({ + email, emailVerified: true, + firstName: host.first_name ?? undefined, lastName: host.last_name ?? undefined, + }); + } catch (error) { + const status = (error as { status?: number })?.status; + // No retryable status (408/429/5xx), transport failure or timeout can + // establish that the write did not happen. + if (status && [400, 401, 403, 404, 409, 422].includes(status)) { + await transition(client, op, 'provider_rejected', 'provider_rejected'); + return { status: 409, body: { error: 'provider_rejected', operation_id: op.id, + message: 'WorkOS rejected creation. Check the account in the WorkOS Dashboard.' } }; + } + return await markUncertain('provider_create_ambiguous'); + } + + if (typeof created?.id === 'string' && /^user_[a-zA-Z0-9_]{1,250}$/.test(created.id)) { + op.provider_user_id = created.id; + } + if (!op.provider_user_id || + created.id === input.hostUserId || created.email?.toLowerCase() !== email) { + return await markUncertain('provider_response_invalid'); + } + await transition(client, op, 'provider_created'); + + let commitStarted = false; + let rollbackConfirmed = false; + let lifecycleValidated = false; + let lifecycleFailure = 'credential_lifecycle_unavailable'; + try { + await client.query('BEGIN'); + await client.query("SET LOCAL lock_timeout = '5s'"); + const states = await lockCredentialLifecyclesInTransaction(client, + [input.actorUserId, input.hostUserId, created.id]); + if (states.some(state => state.terminal_marker)) { + lifecycleFailure = 'credential_lifecycle_terminal'; + throw new Error('Terminal credential'); + } + const newState = states.find(state => state.workos_user_id === created.id); + if (!newState || !unseen(newState)) { + lifecycleFailure = 'local_user_already_exists'; + throw new Error('Credential already has local state'); + } + const currentHost = states.find(state => state.workos_user_id === input.hostUserId); + const currentActor = states.find(state => state.workos_user_id === input.actorUserId); + if (!active(currentHost) || !active(currentActor)) throw new Error('Inactive actor or host'); + lifecycleValidated = true; + if (currentHost?.identity_id !== op.host_identity_id || !currentHost.is_primary + || currentActor?.identity_id !== input.actorIdentityId) { + throw new Error('Binding changed'); + } + + // No upsert: a webhook or another operation may already own local state. + // The insert, singleton binding move, epoch and audit commit together. + const inserted = await client.query( + `INSERT INTO users (workos_user_id, email, first_name, last_name, email_verified, + workos_created_at, workos_updated_at, created_at, updated_at) + VALUES ($1, $2, $3, $4, $5, $6, $7, NOW(), NOW())`, + [created.id, email, created.firstName, created.lastName, created.emailVerified, + created.createdAt, created.updatedAt], + ); + if (inserted.rowCount !== 1) throw new Error('Credential insert cardinality changed'); + const singleton = await client.query<{ identity_id: string }>( + `SELECT identity_id FROM identity_workos_users WHERE workos_user_id = $1 FOR UPDATE`, [created.id], + ); + if (singleton.rowCount !== 1) throw new Error('Missing new binding'); + const bound = await client.query( + `UPDATE identity_workos_users SET identity_id = $2, is_primary = FALSE, bound_at = NOW() + WHERE workos_user_id = $1 AND identity_id = $3`, + [created.id, op.host_identity_id, singleton.rows[0].identity_id], + ); + if (bound.rowCount !== 1) throw new Error('Binding changed'); + await client.query( + `DELETE FROM identities WHERE id = $1 + AND NOT EXISTS (SELECT 1 FROM identity_workos_users WHERE identity_id = $1)`, + [singleton.rows[0].identity_id], + ); + const siblings = await client.query<{ workos_user_id: string }>( + 'SELECT workos_user_id FROM identity_workos_users WHERE identity_id = $1', [op.host_identity_id], + ); + const siblingIds = [...new Set(siblings.rows.map(row => row.workos_user_id))].sort(); + if (!siblingIds.includes(created.id) || !siblingIds.includes(input.hostUserId)) { + throw new Error('Incomplete credential binding set'); + } + const bumped = (await bumpAuthorizationEpochs(client, siblingIds)).sort(); + if (bumped.length !== siblingIds.length || bumped.some((id, index) => id !== siblingIds[index])) { + throw new Error('Incomplete credential epoch update'); + } + await transition(client, op, 'committed'); + commitStarted = true; + const commit = await client.query('COMMIT'); + if (commit.command !== 'COMMIT') throw new Error('Unconfirmed commit'); + result = success(op, email); + } catch { + rollbackConfirmed = await rollback(); + if (commitStarted || !rollbackConfirmed) { + // A lost COMMIT reply may mean a real binding exists. Never delete it. + result = await markUncertain('local_commit_ambiguous'); + } else if (!lifecycleValidated) { + // A marker or uncertain lifecycle observation cannot authorize an + // upstream delete, even though our own local writes rolled back. + result = await markUncertain(lifecycleFailure); + } else { + // Commit a truthful admission quarantine before deleting upstream. + // It survives provider ambiguity and deletion-evidence write failures. + // Reacquire the same fence and verify exact quarantine ownership, then + // retain it through delete, the confirmed-deletion marker and COMMIT. + const compensationClient = client; + if (!compensationClient) return await markUncertain('compensation_lifecycle_unavailable'); + const compensationActor = { + workosUserId: created.id, operationId: op.id, + actorUserId: op.actor_user_id, actorIdentityId: op.actor_identity_id, + }; + let compensationFailure = 'compensation_quarantine_failed'; + try { + await compensationClient.query('BEGIN'); + await compensationClient.query("SET LOCAL lock_timeout = '5s'"); + await recordAdminCredentialCompensationQuarantine(compensationClient, compensationActor); + await transition(compensationClient, op, 'compensating', 'local_write_failed'); + compensationFailure = 'compensation_quarantine_commit_ambiguous'; + if ((await compensationClient.query('COMMIT')).command !== 'COMMIT') throw new Error('Unconfirmed quarantine commit'); + + compensationFailure = 'compensation_lifecycle_unavailable'; + await compensationClient.query('BEGIN'); + await compensationClient.query("SET LOCAL lock_timeout = '5s'"); + await lockAdminCredentialCompensationQuarantine(compensationClient, compensationActor); + compensationFailure = 'provider_compensation_failed'; + await workos.deleteUser(created.id); + compensationFailure = 'confirmed_delete_evidence_failed'; + await recordConfirmedAdminCredentialCompensationDeletion(compensationClient, compensationActor); + await transition(compensationClient, op, 'compensated', 'local_write_failed'); + compensationFailure = 'compensation_commit_ambiguous'; + if ((await compensationClient.query('COMMIT')).command !== 'COMMIT') throw new Error('Unconfirmed compensation commit'); + } catch { + await rollback(); + return await markUncertain(compensationFailure); + } + result = { status: 500, body: { error: 'Failed to bind sign-in email', operation_id: op.id, + compensated: true, message: 'The new WorkOS user was rolled back. Please retry.' } }; + } + } + return result; + } + try { + result = await run(); + } catch { + result = op ? await markUncertain('operation_interrupted') + : { status: 503, body: { error: 'credential_bind_unavailable' } }; + } finally { + if (client) { + for (const key of held.reverse()) { + try { + const unlock = await client.query<{ unlocked: boolean }>( + 'SELECT pg_advisory_unlock(hashtextextended($1, 0)) AS unlocked', [key], + ); + if (unlock.rows[0]?.unlocked !== true) destroy = true; + } catch { destroy = true; } + } + client.release(destroy); + client = undefined; + if (destroy) { + logger.error({ operationId: op?.id, failureCode: 'lock_release_unconfirmed' }, + 'Admin credential binding connection discarded'); + result = await markUncertain('lock_release_unconfirmed'); + } + } + } + return result; +} diff --git a/server/src/services/identity-credential-deletion.ts b/server/src/services/identity-credential-deletion.ts new file mode 100644 index 00000000000..21e684c87f2 --- /dev/null +++ b/server/src/services/identity-credential-deletion.ts @@ -0,0 +1,73 @@ +/** + * Single route-facing owner for confirmed WorkOS credential deletion. + * + * The DB helper commits locks, membership deletion, epochs, and durable audit + * together. This service then evicts process-local caches and raises the + * operator alert. No caller may infer or promote a replacement primary. + */ + +import { + deleteIdentityCredentialTransaction, + type IdentityCredentialDeletionResult, + type IdentityCredentialDeletionSource, +} from '../db/identity-db.js'; +import { invalidateSessionsForUsers } from '../middleware/auth.js'; +import { invalidateUnifiedUsersCache } from '../cache/unified-users.js'; +import { notifySystemError } from '../addie/error-notifier.js'; +import { createLogger } from '../logger.js'; +import { + invalidateSlackAdminStatusCache, + invalidateWebAdminStatusCache, +} from '../addie/admin-status-cache.js'; +import { invalidateMemberContextCache } from '../addie/member-context-cache.js'; +import { invalidateHomeCache } from '../addie/home/cache.js'; + +const logger = createLogger('identity-credential-deletion'); + +export async function deleteIdentityCredential( + workosUserId: string, + deletionSource: IdentityCredentialDeletionSource, +): Promise { + const result = await deleteIdentityCredentialTransaction(workosUserId, deletionSource); + + if (result.replay) return result; + + invalidateSessionsForUsers(result.affectedUserIds); + invalidateUnifiedUsersCache(); + for (const workosUserId of result.affectedUserIds) { + invalidateWebAdminStatusCache(workosUserId); + } + for (const slackUserId of result.affectedSlackUserIds) { + invalidateSlackAdminStatusCache(slackUserId); + invalidateMemberContextCache(slackUserId); + invalidateHomeCache(slackUserId); + } + + if (result.quarantine) { + const { quarantine } = result; + logger.warn({ + identityId: quarantine.identity_id, + deletedWorkosUserId: quarantine.deleted_workos_user_id, + recoveryState: quarantine.recovery_state, + auditId: quarantine.audit_id, + }, 'Credential deletion left an identity without a primary'); + try { + notifySystemError({ + source: `identity-primary-deletion-quarantine:${quarantine.identity_id}`, + errorMessage: [ + `Credential ${quarantine.deleted_workos_user_id} was deleted without promotion.`, + `Identity ${quarantine.identity_id} is quarantined.`, + `recovery_state=${quarantine.recovery_state}.`, + `audit_id=${quarantine.audit_id}.`, + ].join(' '), + }); + } catch (err) { + // Durable recovery state is already committed. Alert transport failure + // must not undo containment or prevent post-commit cache eviction. + logger.error({ err, identityId: quarantine.identity_id, auditId: quarantine.audit_id }, + 'Failed to deliver primary-deletion recovery alert'); + } + } + + return result; +} diff --git a/server/src/slack/sync.ts b/server/src/slack/sync.ts index 58cc55fe663..3af2586734b 100644 --- a/server/src/slack/sync.ts +++ b/server/src/slack/sync.ts @@ -550,11 +550,18 @@ export async function tryAutoLinkWebsiteUserToSlack( } // Link the accounts - await slackDb.mapUser({ + const mapped = await slackDb.mapUser({ slack_user_id: slackUser.slack_user_id, workos_user_id: workosUserId, mapping_source: 'email_auto', }); + if (!mapped) { + logger.warn( + { workosUserId, slackUserId: slackUser.slack_user_id }, + 'Skipped Slack auto-link because the local credential is not active', + ); + return { linked: false, reason: 'credential_inactive' }; + } logger.info( { workosUserId, slackUserId: slackUser.slack_user_id, email }, @@ -625,6 +632,13 @@ export async function checkAndAssignOrganizationByDomain( previousOrgName?: string; error?: string; } | null> { + logger.warn( + { workosUserId }, + 'Automatic domain organization assignment deferred: no durable provider intent journal', + ); + return null; + + /* c8 ignore start -- retained discovery for an eventual journaled workflow */ const pool = getPool(); try { @@ -721,6 +735,7 @@ export async function checkAndAssignOrganizationByDomain( error: error instanceof Error ? error.message : 'Unknown error', }; } + /* c8 ignore stop */ } /** @@ -819,6 +834,10 @@ export async function autoAddVerifiedDomainUsersAsMembers(): Promise<{ skipped: number; errors: number; }> { + logger.warn('Automatic verified-domain membership sweep deferred: no durable provider intent journal'); + return { added: 0, skipped: 0, errors: 0 }; + + /* c8 ignore start -- retained discovery for an eventual journaled workflow */ const pool = getPool(); const result = await pool.query<{ @@ -943,4 +962,5 @@ export async function autoAddVerifiedDomainUsersAsMembers(): Promise<{ } return { added: totalAdded, skipped: totalSkipped, errors: totalErrors }; + /* c8 ignore stop */ } diff --git a/server/tests/integration/admin-add-users.test.ts b/server/tests/integration/admin-add-users.test.ts index 9ca424fcb0a..946910b3d90 100644 --- a/server/tests/integration/admin-add-users.test.ts +++ b/server/tests/integration/admin-add-users.test.ts @@ -108,6 +108,11 @@ describe("admin add-users", () => { "DELETE FROM organization_memberships WHERE workos_organization_id IN ($1, $2)", [TARGET_ORG_ID, SOURCE_ORG_ID] ); + await pool.query( + "DELETE FROM identities WHERE id IN (SELECT identity_id FROM identity_workos_users WHERE workos_user_id = $1)", + [USER_ID] + ); + await pool.query("DELETE FROM users WHERE workos_user_id = $1", [USER_ID]); await pool.query( "DELETE FROM organizations WHERE workos_organization_id IN ($1, $2)", [TARGET_ORG_ID, SOURCE_ORG_ID] @@ -115,6 +120,14 @@ describe("admin add-users", () => { } async function seedSourceMembership() { + await pool.query( + `INSERT INTO users ( + workos_user_id, email, first_name, last_name, email_verified, + workos_created_at, workos_updated_at, created_at, updated_at, + primary_organization_id + ) VALUES ($1, 'user@example.test', 'Ada', 'Admin', TRUE, NOW(), NOW(), NOW(), NOW(), $2)`, + [USER_ID, SOURCE_ORG_ID] + ); await pool.query( `INSERT INTO organization_memberships (workos_user_id, workos_organization_id, workos_membership_id, email, first_name, last_name, role, seat_type, created_at, updated_at, synced_at) diff --git a/server/tests/integration/admin-bind-email.test.ts b/server/tests/integration/admin-bind-email.test.ts index c82f7549f02..d30d5f9014f 100644 --- a/server/tests/integration/admin-bind-email.test.ts +++ b/server/tests/integration/admin-bind-email.test.ts @@ -1,15 +1,5 @@ -/** - * Admin "create + bind sign-in email" integration test (Phase 2b). - * - * Exercises POST /api/admin/users/:userId/linked-emails: - * - Creates a fresh WorkOS user for the new email (mocked). - * - Inserts into local users (trigger fires, creating a singleton identity). - * - mergeUsers re-points the new user's binding to the existing user's - * identity as is_primary = FALSE; drops the orphan singleton. - * - * After this, the existing user has two bound WorkOS users; the auth - * middleware will id-swap a non-primary login to the canonical id. - */ +/** Admin fresh credential binding route tests; provider faults live in + * admin-credential-bind-compensation.test.ts. No merge or authority transfer. */ import { describe, it, expect, beforeAll, afterAll, beforeEach, vi } from 'vitest'; @@ -22,18 +12,24 @@ vi.hoisted(() => { process.env.WORKOS_COOKIE_PASSWORD ??= 'test-cookie-password-at-least-32-chars-long'; }); import request from 'supertest'; -import { initializeDatabase, closeDatabase, getPool } from '../../src/db/client.js'; +import express from 'express'; +import { initializeDatabase, closeDatabase } from '../../src/db/client.js'; import { runMigrations } from '../../src/db/migrate.js'; import type { Pool } from 'pg'; +const actor = vi.hoisted(() => ({ identityId: '' })); + // Mock auth middleware: synthetic admin user. Use importOriginal so that // helpers like createRequireWorkingGroupLeader (used by other mounted // routers) are still available — env vars are set via vi.hoisted above // so the WorkOS constructor at module load doesn't blow up. vi.mock('../../src/middleware/auth.js', async (importOriginal) => { const mockedRequireAuth = (req: any, _res: any, next: any) => { + req.adminAccessMechanism = req.headers['x-test-admin-access-mechanism']; req.user = { id: 'user_test_admin_bind', + authWorkosUserId: 'user_test_admin_bind_credential', + identityId: actor.identityId, email: 'admin@test.local', emailVerified: true, createdAt: new Date().toISOString(), @@ -77,16 +73,16 @@ vi.mock('../../src/auth/workos-client.js', () => { return { workos: mockWorkos, getWorkos: () => mockWorkos, + getAdminCredentialMutationWorkos: () => mockWorkos, }; }); -import { HTTPServer } from '../../src/http.js'; +import { createAdminUsersRouter } from '../../src/routes/admin/users.js'; const EXISTING_USER_ID = 'user_test_bind_existing'; describe('POST /api/admin/users/:userId/linked-emails (admin bind)', () => { - let server: HTTPServer; - let app: any; + let app: express.Express; let pool: Pool; beforeAll(async () => { @@ -94,14 +90,13 @@ describe('POST /api/admin/users/:userId/linked-emails (admin bind)', () => { connectionString: process.env.DATABASE_URL || 'postgresql://adcp:localdev@localhost:5432/adcp_test', }); await runMigrations(); - server = new HTTPServer(); - await server.start(0); - app = server.app; + app = express(); + app.use(express.json()); + app.use('/api/admin/users', createAdminUsersRouter()); }, 60000); afterAll(async () => { await cleanup(); - await server?.stop(); await closeDatabase(); }); @@ -113,6 +108,14 @@ describe('POST /api/admin/users/:userId/linked-emails (admin bind)', () => { VALUES ($1, 'existing@test.example', 'Existing', 'User', true, NOW(), NOW(), NOW(), NOW())`, [EXISTING_USER_ID] ); + await pool.query(`INSERT INTO users (workos_user_id, email) VALUES + ('user_test_admin_bind', 'alex@pinnacle.example'), + ('user_test_admin_bind_credential', 'alex@personal.example')`); + const identity = await pool.query(`SELECT identity_id FROM identity_workos_users + WHERE workos_user_id = 'user_test_admin_bind'`); + actor.identityId = identity.rows[0].identity_id; + await pool.query(`UPDATE identity_workos_users SET identity_id = $1, is_primary = FALSE + WHERE workos_user_id = 'user_test_admin_bind_credential'`, [actor.identityId]); mockCreateUser.mockReset(); mockDeleteUser.mockReset(); mockCreateUser.mockImplementation(async ({ email, firstName, lastName }: any) => ({ @@ -128,6 +131,11 @@ describe('POST /api/admin/users/:userId/linked-emails (admin bind)', () => { }); async function cleanup() { + await pool.query(`DELETE FROM registry_audit_log WHERE action = 'admin_credential_bind' + AND workos_user_id = 'user_test_admin_bind_credential'`); + await pool.query(`DELETE FROM admin_credential_bind_operations WHERE host_user_id = $1`, [EXISTING_USER_ID]); + await pool.query(`DELETE FROM users WHERE workos_user_id IN + ('user_test_admin_bind', 'user_test_admin_bind_credential')`); await pool.query(`DELETE FROM users WHERE workos_user_id IN ($1, $2)`, [ EXISTING_USER_ID, MOCK_NEW_WORKOS_USER_ID, @@ -217,9 +225,8 @@ describe('POST /api/admin/users/:userId/linked-emails (admin bind)', () => { .send({ email: 'taken-upstream@test.example' }) .expect(409); - // The error wording surfaces the WorkOS status + message and points - // admins at the "Link existing WorkOS user" tool. - expect(response.body.message).toMatch(/already in use upstream|Link existing WorkOS user/i); + expect(response.body.error).toBe('provider_rejected'); + expect(JSON.stringify(response.body)).not.toContain('Email already in use'); // No local users row should have been created. const localCheck = await pool.query( `SELECT 1 FROM users WHERE LOWER(email) = 'taken-upstream@test.example'` @@ -227,21 +234,36 @@ describe('POST /api/admin/users/:userId/linked-emails (admin bind)', () => { expect(localCheck.rows).toEqual([]); }); - it('rolls back the WorkOS user when local bind fails after createUser succeeded', async () => { - // Pre-create the new WorkOS user id locally so the post-createUser INSERT - // would succeed, but force mergeUsers to fail by deleting the trigger- - // created identity binding for the existing user (Phase 1 trigger - // guarantees normally — we break the invariant to simulate a partial - // failure path). + it('rejects a broken host binding before provider creation', async () => { await pool.query(`DELETE FROM identity_workos_users WHERE workos_user_id = $1`, [EXISTING_USER_ID]); - const response = await request(app) .post(`/api/admin/users/${EXISTING_USER_ID}/linked-emails`) - .send({ email: 'rollback@test.example' }) - .expect(500); + .send({ email: 'rollback@test.example' }).expect(409); + expect(response.body.error).toBe('host_primary_required'); + expect(mockCreateUser).not.toHaveBeenCalled(); + expect(mockDeleteUser).not.toHaveBeenCalled(); + }); + + it('refuses destructive intent before provider writes', async () => { + await request(app).post(`/api/admin/users/${EXISTING_USER_ID}/linked-emails`) + .send({ email: 'fresh@test.example', consolidate: true }).expect(409); + expect(mockCreateUser).not.toHaveBeenCalled(); + }); + + it('requires an identity-bearing authenticated admin', async () => { + await request(app).post(`/api/admin/users/${EXISTING_USER_ID}/linked-emails`) + .set('x-test-admin-access-mechanism', 'static_admin_api_key') + .send({ email: 'fresh@test.example' }).expect(403); + expect(mockCreateUser).not.toHaveBeenCalled(); + }); - expect(response.body.error).toMatch(/failed to bind/i); - expect(response.body.message).toMatch(/rolled back|retry/i); - expect(mockDeleteUser).toHaveBeenCalledWith(MOCK_NEW_WORKOS_USER_ID); + it('journals the authenticated credential instead of the canonical user', async () => { + const response = await request(app).post(`/api/admin/users/${EXISTING_USER_ID}/linked-emails`) + .send({ email: 'fresh@test.example' }).expect(201); + const journal = await pool.query(`SELECT actor_user_id, actor_identity_id + FROM admin_credential_bind_operations WHERE id = $1`, [response.body.operation_id]); + expect(journal.rows[0]).toEqual({ + actor_user_id: 'user_test_admin_bind_credential', actor_identity_id: actor.identityId, + }); }); }); diff --git a/server/tests/integration/admin-credential-bind-compensation.test.ts b/server/tests/integration/admin-credential-bind-compensation.test.ts new file mode 100644 index 00000000000..6915df9736e --- /dev/null +++ b/server/tests/integration/admin-credential-bind-compensation.test.ts @@ -0,0 +1,723 @@ +/** + * Admin create-and-bind uses real PostgreSQL transactions and a mocked WorkOS + * boundary. Transport faults deliberately leave the provider outcome unknown; + * a stale provider read cannot authorize replay or destructive compensation. + */ +import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest'; +import type { Pool, PoolClient } from 'pg'; + +const provider = vi.hoisted(() => ({ + createUser: vi.fn(), + deleteUser: vi.fn(), + getUser: vi.fn(), + listUsers: vi.fn(), + createOrganizationMembership: vi.fn(), + updateOrganizationMembership: vi.fn(), + deleteOrganizationMembership: vi.fn(), +})); +const logs = vi.hoisted(() => ({ + info: vi.fn(), warn: vi.fn(), error: vi.fn(), debug: vi.fn(), +})); + +vi.mock('../../src/auth/workos-client.js', () => ({ + getAdminCredentialMutationWorkos: () => ({ userManagement: provider }), +})); +vi.mock('../../src/logger.js', () => ({ createLogger: () => logs })); +vi.mock('../../src/addie/error-notifier.js', () => ({ notifySystemError: vi.fn() })); + +import { closeDatabase, initializeDatabase } from '../../src/db/client.js'; +import { runMigrations } from '../../src/db/migrate.js'; +import { getAuthorizationFingerprint } from '../../src/db/authorization-epoch-db.js'; +import { createAndBindAdminCredential } from '../../src/services/admin-credential-bind.js'; + +const PREFIX = 'user_admin_bind_compensation_'; +const HOST = `${PREFIX}sam`; +const HOST_SIBLING = `${PREFIX}sam_existing_credential`; +const OTHER_HOST = `${PREFIX}priya`; +const ACTOR_PRIMARY = `${PREFIX}jordan`; +const ACTOR_CREDENTIAL = `${PREFIX}jordan_credential`; +const CREATED = `${PREFIX}sam_new_credential`; +const ORG = 'org_admin_bind_compensation_pinnacle'; +const EMAIL = 'sam.new@admin-bind.example'; +const SECRET = 'provider-body-with-password-and-access-token-must-not-be-retained'; + +interface Operation { + id: string; + email_hash: string; + host_user_id: string; + host_identity_id: string; + actor_user_id: string; + actor_identity_id: string; + provider_user_id: string | null; + status: string; + failure_code: string | null; +} + +function deferred() { + let resolve!: (value: T) => void; + const promise = new Promise((complete) => { resolve = complete; }); + return { promise, resolve }; +} + +describe('admin credential create-and-bind compensation', () => { + let pool: Pool; + let actorIdentityId: string; + let hostIdentityId: string; + const fixtureIdentities = new Set(); + + beforeAll(async () => { + pool = initializeDatabase({ + connectionString: process.env.DATABASE_URL || 'postgresql://adcp:localdev@localhost:5432/adcp_test', + }); + await runMigrations(); + }, 60000); + + afterAll(async () => { + if (pool) await cleanup(); + await closeDatabase(); + }); + + beforeEach(async () => { + await cleanup(); + vi.clearAllMocks(); + for (const mock of Object.values(provider)) mock.mockReset(); + provider.createUser.mockResolvedValue(createdUser()); + provider.deleteUser.mockResolvedValue(undefined); + // These intentionally stale responses would conceal delayed provider + // completion if the implementation used immediate reads as adjudication. + provider.getUser.mockRejectedValue(Object.assign(new Error('not found'), { status: 404 })); + provider.listUsers.mockResolvedValue({ data: [] }); + + await insertUser(HOST, 'sam@admin-bind.example', 'Sam', 'Adeyemi'); + await insertUser(HOST_SIBLING, 'sam.existing@admin-bind.example', 'Sam', 'Adeyemi'); + await insertUser(OTHER_HOST, 'priya@admin-bind.example', 'Priya', 'Nair'); + await insertUser(ACTOR_PRIMARY, 'jordan@admin-bind.example', 'Jordan', 'Ochoa'); + await insertUser(ACTOR_CREDENTIAL, 'jordan.credential@admin-bind.example', 'Jordan', 'Ochoa'); + actorIdentityId = await identityOf(ACTOR_PRIMARY); + hostIdentityId = await identityOf(HOST); + await pool.query( + `UPDATE identity_workos_users SET identity_id = $1, is_primary = false WHERE workos_user_id = $2`, + [hostIdentityId, HOST_SIBLING], + ); + await pool.query( + `UPDATE identity_workos_users SET identity_id = $1, is_primary = false WHERE workos_user_id = $2`, + [actorIdentityId, ACTOR_CREDENTIAL], + ); + await pool.query( + `INSERT INTO organizations (workos_organization_id, name) VALUES ($1, 'Pinnacle Agency')`, + [ORG], + ); + await pool.query( + `INSERT INTO organization_memberships + (workos_user_id, workos_organization_id, email, role) + VALUES ($1, $2, 'sam@admin-bind.example', 'admin')`, + [HOST, ORG], + ); + await pool.query('UPDATE users SET primary_organization_id = $1 WHERE workos_user_id = $2', [ORG, HOST]); + }); + + function createdUser() { + return { + id: CREATED, email: EMAIL, firstName: 'Sam', lastName: 'Adeyemi', + emailVerified: false, createdAt: '2026-09-01T00:00:00.000Z', + updatedAt: '2026-09-01T00:00:00.000Z', + }; + } + + function input(overrides: Partial[0]> = {}) { + return { hostUserId: HOST, email: EMAIL, actorUserId: ACTOR_CREDENTIAL, actorIdentityId, ...overrides }; + } + + async function insertUser(id: string, email: string, firstName: string, lastName: string) { + await pool.query( + `INSERT INTO users (workos_user_id, email, first_name, last_name, email_verified, + workos_created_at, workos_updated_at) VALUES ($1, $2, $3, $4, true, NOW(), NOW())`, + [id, email, firstName, lastName], + ); + fixtureIdentities.add(await identityOf(id)); + } + + async function identityOf(userId: string) { + const result = await pool.query<{ identity_id: string }>( + 'SELECT identity_id FROM identity_workos_users WHERE workos_user_id = $1', [userId], + ); + return result.rows[0].identity_id; + } + + async function cleanup() { + await pool.query('DROP TRIGGER IF EXISTS admin_bind_test_fail_insert ON users'); + await pool.query('DROP FUNCTION IF EXISTS admin_bind_test_fail_insert()'); + await pool.query('DROP TRIGGER IF EXISTS admin_bind_test_fail_audit ON registry_audit_log'); + await pool.query('DROP TRIGGER IF EXISTS admin_bind_test_fail_intent ON admin_credential_bind_operations'); + await pool.query('DROP TRIGGER IF EXISTS admin_bind_test_fail_reconciliation ON admin_credential_bind_operations'); + await pool.query('DROP FUNCTION IF EXISTS admin_bind_test_fail_journal()'); + const identities = await pool.query<{ identity_id: string }>( + 'SELECT identity_id FROM identity_workos_users WHERE starts_with(workos_user_id, $1)', [PREFIX], + ); + for (const row of identities.rows) fixtureIdentities.add(row.identity_id); + await pool.query( + `DELETE FROM registry_audit_log WHERE resource_id IN + (SELECT id::text FROM admin_credential_bind_operations WHERE starts_with(host_user_id, $1))`, + [PREFIX], + ); + await pool.query('DELETE FROM admin_credential_bind_operations WHERE starts_with(host_user_id, $1)', [PREFIX]); + await pool.query('DELETE FROM organization_memberships WHERE workos_organization_id = $1', [ORG]); + await pool.query('DELETE FROM organizations WHERE workos_organization_id = $1', [ORG]); + await pool.query('DELETE FROM users WHERE starts_with(workos_user_id, $1)', [PREFIX]); + if (fixtureIdentities.size) { + await pool.query('DELETE FROM identities WHERE id = ANY($1::uuid[])', [[...fixtureIdentities]]); + fixtureIdentities.clear(); + } + } + + async function operations() { + return (await pool.query( + `SELECT * FROM admin_credential_bind_operations WHERE starts_with(host_user_id, $1) + ORDER BY created_at, id`, [PREFIX], + )).rows; + } + + async function expectNoLocalCredential() { + const rows = await pool.query('SELECT workos_user_id FROM users WHERE workos_user_id = $1', [CREATED]); + expect(rows.rows).toEqual([]); + // Confirmed compensation now records a terminal marker in the fingerprint. + // It must still leave no binding epochs or altered host authority. + expect((await pool.query('SELECT 1 FROM authorization_epochs WHERE workos_user_id = ANY($1)', + [[HOST, HOST_SIBLING, CREATED]])).rows).toEqual([]); + expect(await identityOf(HOST)).toBe(hostIdentityId); + } + + async function failLocalInsert() { + await pool.query(`CREATE FUNCTION admin_bind_test_fail_insert() RETURNS trigger LANGUAGE plpgsql AS $$ + BEGIN + RAISE EXCEPTION 'injected local credential write failure'; + END; + $$`); + await pool.query(`CREATE TRIGGER admin_bind_test_fail_insert BEFORE INSERT ON users + FOR EACH ROW WHEN (NEW.workos_user_id = '${CREATED}') + EXECUTE FUNCTION admin_bind_test_fail_insert()`); + } + + function assertNoProviderReadsOrMembershipWrites() { + expect(provider.getUser).not.toHaveBeenCalled(); + expect(provider.listUsers).not.toHaveBeenCalled(); + expect(provider.createOrganizationMembership).not.toHaveBeenCalled(); + expect(provider.updateOrganizationMembership).not.toHaveBeenCalled(); + expect(provider.deleteOrganizationMembership).not.toHaveBeenCalled(); + } + + /** Preserve real transactions; change only a specific transport response. */ + function faultPool(mode: 'commit_reply_lost' | 'rollback_reply_lost' | 'lock_false' | 'unlock_false' | 'lock_reply_lost' | 'lock_reply_malformed') { + let injected = false; + const releases: Array = []; + const proxy = new Proxy(pool, { + get(target, property) { + if (property === 'connect') return async () => { + const client = await target.connect(); + const query = client.query.bind(client); + const release = client.release.bind(client); + let localWrite = false; + client.query = (async (text: string, params?: unknown[]) => { + if (text.includes('INSERT INTO users')) localWrite = true; + if (!injected && mode === 'lock_false' && text.includes('pg_try_advisory_lock')) { + injected = true; + return { rows: [{ locked: false, acquired: false, pg_try_advisory_lock: false }], rowCount: 1 }; + } + const result = await query(text, params); + if (!injected && text.includes('pg_try_advisory_lock') && mode.startsWith('lock_reply_')) { + injected = true; + if (mode === 'lock_reply_lost') throw Object.assign(new Error('lock transport reply lost'), { code: 'ECONNRESET' }); + return { ...result, rows: [] }; + } + if (!injected && mode === 'unlock_false' && text.includes('pg_advisory_unlock')) { + injected = true; + return { ...result, rows: [{ unlocked: false, pg_advisory_unlock: false }] }; + } + if (!injected && mode === 'commit_reply_lost' && localWrite && text.trim() === 'COMMIT') { + injected = true; + throw Object.assign(new Error('commit transport reply lost'), { code: 'ECONNRESET' }); + } + if (!injected && mode === 'rollback_reply_lost' && localWrite && text.trim() === 'ROLLBACK') { + injected = true; + throw Object.assign(new Error('rollback transport reply lost'), { code: 'ECONNRESET' }); + } + return result; + }) as PoolClient['query']; + client.release = (error?: boolean | Error) => { + releases.push(error); + client.query = query; + client.release = release; + release(error); + }; + return client; + }; + const value = Reflect.get(target, property); + return typeof value === 'function' ? value.bind(target) : value; + }, + }); + return { pool: proxy, releases, wasInjected: () => injected }; + } + + it('binds only the created credential, revokes all host sessions, and records the exact authenticated actor', async () => { + const result = await createAndBindAdminCredential(input()); + + expect(result.status).toBe(201); + const journal = await operations(); + expect(journal).toHaveLength(1); + expect(journal[0]).toMatchObject({ + status: 'committed', host_user_id: HOST, host_identity_id: hostIdentityId, + actor_user_id: ACTOR_CREDENTIAL, actor_identity_id: actorIdentityId, provider_user_id: CREATED, + }); + expect(journal[0].email_hash).not.toContain(EMAIL); + const binding = await pool.query( + 'SELECT identity_id, is_primary FROM identity_workos_users WHERE workos_user_id = $1', [CREATED], + ); + expect(binding.rows).toEqual([{ identity_id: hostIdentityId, is_primary: false }]); + expect(await getAuthorizationFingerprint([HOST, CREATED])).toBe( + `${HOST}:epoch:1,${CREATED}:epoch:1`, + ); + expect(await getAuthorizationFingerprint([HOST_SIBLING])).toBe(`${HOST_SIBLING}:epoch:1`); + const organizationPointers = await pool.query( + `SELECT workos_user_id, primary_organization_id FROM users WHERE workos_user_id = ANY($1) + ORDER BY workos_user_id`, [[HOST, CREATED]], + ); + expect(organizationPointers.rows).toEqual([ + { workos_user_id: HOST, primary_organization_id: ORG }, + { workos_user_id: CREATED, primary_organization_id: null }, + ]); + const memberships = await pool.query( + `SELECT workos_user_id, role FROM organization_memberships WHERE workos_organization_id = $1`, [ORG], + ); + expect(memberships.rows).toEqual([{ workos_user_id: HOST, role: 'admin' }]); + const audit = await pool.query( + `SELECT details FROM registry_audit_log WHERE action = 'admin_credential_bind' + AND resource_id = $1 ORDER BY created_at`, [journal[0].id], + ); + expect(audit.rows.length).toBeGreaterThan(0); + expect(audit.rows).toEqual(expect.arrayContaining([{ + details: expect.objectContaining({ + status: 'committed', acting_workos_user_id: ACTOR_CREDENTIAL, actor_identity_id: actorIdentityId, + }), + }])); + expect(provider.deleteUser).not.toHaveBeenCalled(); + assertNoProviderReadsOrMembershipWrites(); + }); + + it('records a definite provider rejection before any local credential mutation and permits a safe retry', async () => { + provider.createUser.mockRejectedValueOnce(Object.assign(new Error(SECRET), { status: 422 })); + + const rejected = await createAndBindAdminCredential(input()); + + expect(rejected.status).toBeGreaterThanOrEqual(400); + await expectNoLocalCredential(); + expect((await operations())[0].status).toBe('provider_rejected'); + expect(provider.deleteUser).not.toHaveBeenCalled(); + const retry = await createAndBindAdminCredential(input()); + expect(retry.status).toBe(201); + expect(provider.createUser).toHaveBeenCalledTimes(2); + expect(JSON.stringify(logs.error.mock.calls)).not.toContain(SECRET); + expect(JSON.stringify(logs.warn.mock.calls)).not.toContain(SECRET); + }); + + it('retains a safe provider ID for adjudication when creation returns a different email', async () => { + provider.createUser.mockResolvedValueOnce({ ...createdUser(), email: 'priya.new@admin-bind.example' }); + + const result = await createAndBindAdminCredential(input()); + + expect(result.body).toMatchObject({ reconciliation_required: true, new_workos_user_id: CREATED }); + const journal = (await operations())[0]; + expect(journal).toMatchObject({ + status: 'reconciliation_required', provider_user_id: CREATED, failure_code: 'provider_response_invalid', + }); + const audit = await pool.query('SELECT details FROM registry_audit_log WHERE resource_id = $1', [journal.id]); + expect(audit.rows).toContainEqual({ details: expect.objectContaining({ provider_user_id: CREATED }) }); + await expectNoLocalCredential(); + expect((await createAndBindAdminCredential(input())).body.reconciliation_required).toBe(true); + expect(provider.createUser).toHaveBeenCalledTimes(1); + expect(provider.deleteUser).not.toHaveBeenCalled(); + assertNoProviderReadsOrMembershipWrites(); + }); + + it('does not adopt or delete a provider credential materialized locally by a concurrent webhook', async () => { + let webhookIdentityId: string | undefined; + provider.createUser.mockImplementationOnce(async () => { + await insertUser(CREATED, EMAIL, 'Sam', 'Adeyemi'); + webhookIdentityId = await identityOf(CREATED); + await pool.query( + `INSERT INTO organization_memberships (workos_user_id, workos_organization_id, email, role) + VALUES ($1, $2, $3, 'member')`, [CREATED, ORG, EMAIL], + ); + return createdUser(); + }); + + const result = await createAndBindAdminCredential(input()); + + expect(result.body.reconciliation_required).toBe(true); + expect((await operations())[0]).toMatchObject({ + status: 'reconciliation_required', provider_user_id: CREATED, failure_code: 'local_user_already_exists', + }); + expect(await identityOf(CREATED)).toBe(webhookIdentityId); + expect(webhookIdentityId).not.toBe(hostIdentityId); + const memberships = await pool.query( + 'SELECT role FROM organization_memberships WHERE workos_user_id = $1 AND workos_organization_id = $2', + [CREATED, ORG], + ); + expect(memberships.rows).toEqual([{ role: 'member' }]); + expect(await getAuthorizationFingerprint([HOST, HOST_SIBLING, CREATED])).toBe(''); + expect((await createAndBindAdminCredential(input())).body.reconciliation_required).toBe(true); + expect(provider.createUser).toHaveBeenCalledTimes(1); + expect(provider.deleteUser).not.toHaveBeenCalled(); + assertNoProviderReadsOrMembershipWrites(); + }); + + it('rolls back the local transaction and deletes exactly the newly created provider user', async () => { + await failLocalInsert(); + provider.deleteUser.mockImplementation(async (id) => { + expect(id).toBe(CREATED); + await expectNoLocalCredential(); + }); + + const result = await createAndBindAdminCredential(input()); + + expect(result.status).toBeGreaterThanOrEqual(400); + expect(provider.createUser).toHaveBeenCalledTimes(1); + expect(provider.deleteUser).toHaveBeenCalledExactlyOnceWith(CREATED); + await expectNoLocalCredential(); + expect((await operations())[0]).toMatchObject({ status: 'compensated', provider_user_id: CREATED }); + assertNoProviderReadsOrMembershipWrites(); + }); + + it('permits a new operation after verified compensation without replaying the old operation', async () => { + await failLocalInsert(); + await createAndBindAdminCredential(input()); + const first = (await operations())[0]; + expect(first.status).toBe('compensated'); + await pool.query('DROP TRIGGER admin_bind_test_fail_insert ON users'); + // WorkOS allocates a new ID; a deleted ID must never be resurrected. + provider.createUser.mockResolvedValueOnce({ ...createdUser(), id: `${CREATED}_retry` }); + + expect((await createAndBindAdminCredential(input())).status).toBe(201); + + const journal = await operations(); + expect(journal).toHaveLength(2); + expect(journal.find((operation) => operation.id === first.id)?.status).toBe('compensated'); + expect(provider.createUser).toHaveBeenCalledTimes(2); + expect(provider.deleteUser).toHaveBeenCalledTimes(1); + }); + + it('rolls back the inserted user, binding, and epochs if the transactional commit audit fails', async () => { + await pool.query(`CREATE FUNCTION admin_bind_test_fail_journal() RETURNS trigger LANGUAGE plpgsql AS $$ + BEGIN RAISE EXCEPTION 'injected journal write failure'; END; + $$`); + await pool.query(`CREATE TRIGGER admin_bind_test_fail_audit BEFORE INSERT ON registry_audit_log + FOR EACH ROW WHEN (NEW.action = 'admin_credential_bind' + AND NEW.details->>'status' = 'committed' AND NEW.details->>'host_user_id' = '${HOST}') + EXECUTE FUNCTION admin_bind_test_fail_journal()`); + + const result = await createAndBindAdminCredential(input()); + + expect(result.body.compensated).toBe(true); + await expectNoLocalCredential(); + expect(provider.deleteUser).toHaveBeenCalledExactlyOnceWith(CREATED); + const journal = (await operations())[0]; + expect(journal.status).toBe('compensated'); + const committed = await pool.query( + `SELECT 1 FROM registry_audit_log WHERE resource_id = $1 AND details->>'status' = 'committed'`, [journal.id], + ); + expect(committed.rows).toEqual([]); + }); + + it('refuses provider creation if durable intent cannot be recorded', async () => { + await pool.query(`CREATE FUNCTION admin_bind_test_fail_journal() RETURNS trigger LANGUAGE plpgsql AS $$ + BEGIN RAISE EXCEPTION 'injected journal write failure'; END; + $$`); + await pool.query(`CREATE TRIGGER admin_bind_test_fail_intent BEFORE INSERT ON admin_credential_bind_operations + FOR EACH ROW WHEN (NEW.host_user_id = '${HOST}') EXECUTE FUNCTION admin_bind_test_fail_journal()`); + + const result = await createAndBindAdminCredential(input()); + + expect(result.status).toBeGreaterThanOrEqual(400); + expect(provider.createUser).not.toHaveBeenCalled(); + expect(await operations()).toEqual([]); + await expectNoLocalCredential(); + }); + + it('retains support-safe durable evidence when compensation fails and blocks replay across hosts', async () => { + await failLocalInsert(); + provider.deleteUser.mockRejectedValueOnce(Object.assign(new Error(SECRET), { + status: 503, response: { accessToken: SECRET }, + })); + + const failed = await createAndBindAdminCredential(input()); + + expect(failed.body.reconciliation_required).toBe(true); + const journal = (await operations())[0]; + expect(journal).toMatchObject({ status: 'reconciliation_required', provider_user_id: CREATED }); + expect(journal.failure_code).toBeTruthy(); + await expectNoLocalCredential(); + for (const hostUserId of [HOST, OTHER_HOST]) { + const replay = await createAndBindAdminCredential(input({ hostUserId, email: ` ${EMAIL.toUpperCase()} ` })); + expect(replay.body.reconciliation_required).toBe(true); + } + expect(provider.createUser).toHaveBeenCalledTimes(1); + expect(provider.deleteUser).toHaveBeenCalledTimes(1); + const audit = await pool.query('SELECT details FROM registry_audit_log WHERE resource_id = $1', [journal.id]); + expect(audit.rows.length).toBeGreaterThan(0); + expect(JSON.stringify({ journal, audit: audit.rows, failed, logs: Object.values(logs).map((log) => log.mock.calls) })) + .not.toContain(SECRET); + assertNoProviderReadsOrMembershipWrites(); + }); + + it('retains the durable compensating intent if compensation and the reconciliation journal update both fail', async () => { + await failLocalInsert(); + await pool.query(`CREATE FUNCTION admin_bind_test_fail_journal() RETURNS trigger LANGUAGE plpgsql AS $$ + BEGIN RAISE EXCEPTION 'injected reconciliation write failure'; END; + $$`); + await pool.query(`CREATE TRIGGER admin_bind_test_fail_reconciliation BEFORE UPDATE ON admin_credential_bind_operations + FOR EACH ROW WHEN (NEW.host_user_id = '${HOST}' AND NEW.status = 'reconciliation_required') + EXECUTE FUNCTION admin_bind_test_fail_journal()`); + provider.deleteUser.mockRejectedValueOnce(new Error(SECRET)); + + const result = await createAndBindAdminCredential(input()); + + expect(result.body.reconciliation_required).toBe(true); + const journal = (await operations())[0]; + expect(journal).toMatchObject({ status: 'compensating', provider_user_id: CREATED, failure_code: 'local_write_failed' }); + const audit = await pool.query('SELECT details FROM registry_audit_log WHERE resource_id = $1', [journal.id]); + expect(audit.rows).toContainEqual({ details: expect.objectContaining({ + status: 'compensating', provider_user_id: CREATED, acting_workos_user_id: ACTOR_CREDENTIAL, + }) }); + for (const hostUserId of [HOST, OTHER_HOST]) { + expect((await createAndBindAdminCredential(input({ hostUserId }))).body.reconciliation_required).toBe(true); + } + expect(provider.createUser).toHaveBeenCalledTimes(1); + expect(provider.deleteUser).toHaveBeenCalledExactlyOnceWith(CREATED); + expect(logs.error).toHaveBeenCalledWith({ operationId: journal.id, failureCode: 'journal_update_failed' }, expect.any(String)); + expect(JSON.stringify({ journal, audit: audit.rows, result, logs: Object.values(logs).map((log) => log.mock.calls) })) + .not.toContain(SECRET); + await expectNoLocalCredential(); + assertNoProviderReadsOrMembershipWrites(); + }); + + it('blocks replay after a timeout even when provider creation finishes after stale reads would report absence', async () => { + const delayed = deferred(); + let providerCompleted = false; + provider.createUser.mockImplementationOnce(async () => { + void delayed.promise.then(() => { providerCompleted = true; }); + throw Object.assign(new Error('provider response timeout'), { code: 'ETIMEDOUT' }); + }); + + const timeout = await createAndBindAdminCredential(input()); + expect(timeout.body.reconciliation_required).toBe(true); + expect(providerCompleted).toBe(false); + expect((await operations())[0].status).toBe('reconciliation_required'); + expect((await createAndBindAdminCredential(input())).body.reconciliation_required).toBe(true); + delayed.resolve(); + await delayed.promise; + expect(providerCompleted).toBe(true); + expect((await createAndBindAdminCredential(input({ hostUserId: OTHER_HOST }))).body.reconciliation_required).toBe(true); + expect(provider.createUser).toHaveBeenCalledTimes(1); + expect(provider.deleteUser).not.toHaveBeenCalled(); + await expectNoLocalCredential(); + assertNoProviderReadsOrMembershipWrites(); + }); + + it('does not infer compensation success after a delete timeout with delayed provider completion', async () => { + await failLocalInsert(); + const delayed = deferred(); + let providerDeleted = false; + provider.deleteUser.mockImplementationOnce(async () => { + void delayed.promise.then(() => { providerDeleted = true; }); + throw Object.assign(new Error('delete response timeout'), { code: 'ETIMEDOUT' }); + }); + + const result = await createAndBindAdminCredential(input()); + expect(result.body.reconciliation_required).toBe(true); + expect(providerDeleted).toBe(false); + delayed.resolve(); + await delayed.promise; + expect(providerDeleted).toBe(true); + expect((await createAndBindAdminCredential(input())).body.reconciliation_required).toBe(true); + expect(provider.createUser).toHaveBeenCalledTimes(1); + expect(provider.deleteUser).toHaveBeenCalledTimes(1); + expect((await operations())[0].status).toBe('reconciliation_required'); + assertNoProviderReadsOrMembershipWrites(); + }); + + it('returns a verified committed replay without another provider write or epoch bump', async () => { + expect((await createAndBindAdminCredential(input())).status).toBe(201); + const before = await getAuthorizationFingerprint([HOST, CREATED]); + + const replay = await createAndBindAdminCredential(input({ email: ` ${EMAIL.toUpperCase()} ` })); + + expect(replay.status).toBe(200); + expect(await operations()).toHaveLength(1); + expect(provider.createUser).toHaveBeenCalledTimes(1); + expect(await getAuthorizationFingerprint([HOST, CREATED])).toBe(before); + }); + + it('refuses a committed replay when the credential no longer belongs to the recorded host identity', async () => { + await createAndBindAdminCredential(input()); + await pool.query('UPDATE identity_workos_users SET identity_id = $1 WHERE workos_user_id = $2', [ + await identityOf(OTHER_HOST), CREATED, + ]); + + const replay = await createAndBindAdminCredential(input()); + + expect(replay.status).toBeGreaterThanOrEqual(400); + expect(replay.body.reconciliation_required).toBe(true); + expect(provider.createUser).toHaveBeenCalledTimes(1); + expect(provider.deleteUser).not.toHaveBeenCalled(); + }); + + it('serializes concurrent normalized-email requests across two host identities', async () => { + const entered = deferred(); + const finish = deferred>(); + provider.createUser.mockImplementationOnce(() => { + entered.resolve(); + return finish.promise; + }); + const first = createAndBindAdminCredential(input()); + await entered.promise; + // Keep WorkOS creation in flight until the competing request is refused; + // this tests overlapping operations, not merely two sequential replays. + const second = await createAndBindAdminCredential(input({ hostUserId: OTHER_HOST, email: EMAIL.toUpperCase() })); + finish.resolve(createdUser()); + + const firstResult = await first; + + expect(firstResult.status).toBe(201); + expect(second.status).toBe(409); + expect(provider.createUser).toHaveBeenCalledTimes(1); + expect(await identityOf(CREATED)).toBe(hostIdentityId); + expect(await identityOf(OTHER_HOST)).not.toBe(hostIdentityId); + }); + + it.each(['absent credential', 'wrong identity'] as const)( + 'rejects an %s actor before creating a provider credential', async (scenario) => { + const invalidActor = scenario === 'absent credential' + ? { actorUserId: `${PREFIX}missing` } + : { actorIdentityId: hostIdentityId }; + + const result = await createAndBindAdminCredential(input(invalidActor)); + + expect(result.status).toBe(403); + expect(provider.createUser).not.toHaveBeenCalled(); + expect(await operations()).toEqual([]); + }, + ); + + it('rechecks the actor identity after provider creation and compensates if the credential moved', async () => { + provider.createUser.mockImplementationOnce(async () => { + await pool.query('UPDATE identity_workos_users SET identity_id = $1 WHERE workos_user_id = $2', [ + await identityOf(OTHER_HOST), ACTOR_CREDENTIAL, + ]); + return createdUser(); + }); + + const result = await createAndBindAdminCredential(input()); + + expect(result.status).toBeGreaterThanOrEqual(400); + expect(provider.deleteUser).toHaveBeenCalledExactlyOnceWith(CREATED); + expect((await operations())[0].status).toBe('compensated'); + await expectNoLocalCredential(); + }); + + it('refuses a non-primary host before provider creation', async () => { + const result = await createAndBindAdminCredential(input({ hostUserId: ACTOR_CREDENTIAL })); + + expect(result.status).toBeGreaterThanOrEqual(400); + expect(provider.createUser).not.toHaveBeenCalled(); + expect(await operations()).toEqual([]); + }); + + it('compensates instead of binding to a host whose identity changed during provider creation', async () => { + provider.createUser.mockImplementationOnce(async () => { + await pool.query('UPDATE identity_workos_users SET identity_id = $1, is_primary = false WHERE workos_user_id = $2', [ + await identityOf(OTHER_HOST), HOST, + ]); + return createdUser(); + }); + + const result = await createAndBindAdminCredential(input()); + + expect(result.body.compensated).toBe(true); + expect(provider.deleteUser).toHaveBeenCalledExactlyOnceWith(CREATED); + expect((await operations())[0].status).toBe('compensated'); + const local = await pool.query('SELECT 1 FROM users WHERE workos_user_id = $1', [CREATED]); + expect(local.rows).toEqual([]); + expect((await pool.query('SELECT 1 FROM authorization_epochs WHERE workos_user_id = ANY($1)', + [[HOST, HOST_SIBLING, CREATED]])).rows).toEqual([]); + }); + + it('preserves the provider credential after a lost local COMMIT reply and requires adjudication', async () => { + const fault = faultPool('commit_reply_lost'); + + const result = await createAndBindAdminCredential(input(), { pool: fault.pool, workos: provider }); + + expect(fault.wasInjected()).toBe(true); + expect(result.body.reconciliation_required).toBe(true); + expect(provider.deleteUser).not.toHaveBeenCalled(); + // The transaction really committed; deleting WorkOS here would break the + // successfully bound credential despite the caller receiving an error. + expect(await identityOf(CREATED)).toBe(hostIdentityId); + expect((await operations())[0].status).toBe('reconciliation_required'); + expect((await createAndBindAdminCredential(input())).body.reconciliation_required).toBe(true); + expect(provider.createUser).toHaveBeenCalledTimes(1); + }); + + it('discards the connection and preserves the provider credential when local rollback is unconfirmed', async () => { + await failLocalInsert(); + const fault = faultPool('rollback_reply_lost'); + + const result = await createAndBindAdminCredential(input(), { pool: fault.pool, workos: provider }); + + expect(fault.wasInjected()).toBe(true); + expect(fault.releases.some((release) => release === true || release instanceof Error)).toBe(true); + expect(result.body.reconciliation_required).toBe(true); + expect(result.body.compensated).not.toBe(true); + expect((await operations())[0]).toMatchObject({ status: 'reconciliation_required', provider_user_id: CREATED }); + expect(provider.deleteUser).not.toHaveBeenCalled(); + await expectNoLocalCredential(); + expect((await createAndBindAdminCredential(input())).body.reconciliation_required).toBe(true); + expect(provider.createUser).toHaveBeenCalledTimes(1); + assertNoProviderReadsOrMembershipWrites(); + }); + + it('refuses provider writes when advisory lock acquisition returns false', async () => { + const fault = faultPool('lock_false'); + + const result = await createAndBindAdminCredential(input(), { pool: fault.pool, workos: provider }); + + expect(fault.wasInjected()).toBe(true); + expect(result.status).toBeGreaterThanOrEqual(400); + expect(provider.createUser).not.toHaveBeenCalled(); + expect(await operations()).toEqual([]); + }); + + it('destroys the pooled connection and reports reconciliation if advisory unlock is not confirmed', async () => { + const fault = faultPool('unlock_false'); + + const result = await createAndBindAdminCredential(input(), { pool: fault.pool, workos: provider }); + + expect(fault.wasInjected()).toBe(true); + expect(result.body.reconciliation_required).toBe(true); + expect(fault.releases.some((release) => release === true || release instanceof Error)).toBe(true); + expect((await operations())[0].status).toBe('reconciliation_required'); + expect(provider.deleteUser).not.toHaveBeenCalled(); + }); + + it.each(['lock_reply_lost', 'lock_reply_malformed'] as const)( + 'destroys the pooled connection when acquisition outcome is uncertain (%s)', async (mode) => { + const fault = faultPool(mode); + + const result = await createAndBindAdminCredential(input(), { pool: fault.pool, workos: provider }); + + expect(fault.wasInjected()).toBe(true); + expect(result.status).toBeGreaterThanOrEqual(400); + expect(fault.releases.some((release) => release === true || release instanceof Error)).toBe(true); + expect(provider.createUser).not.toHaveBeenCalled(); + expect(await operations()).toEqual([]); + }, + ); +}); diff --git a/server/tests/integration/admin-credential-bind-lifecycle.test.ts b/server/tests/integration/admin-credential-bind-lifecycle.test.ts new file mode 100644 index 00000000000..ccebd8c2199 --- /dev/null +++ b/server/tests/integration/admin-credential-bind-lifecycle.test.ts @@ -0,0 +1,1014 @@ +/** + * The admin saga and authoritative WorkOS deletion share one PostgreSQL + * credential lifecycle fence. These tests keep database transactions real and + * control provider replies / individual SQL replies at explicit boundaries. + */ +import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest'; +import type { Pool, PoolClient, QueryResult } from 'pg'; + +const provider = vi.hoisted(() => ({ createUser: vi.fn(), deleteUser: vi.fn() })); +const logs = vi.hoisted(() => ({ info: vi.fn(), warn: vi.fn(), error: vi.fn(), debug: vi.fn() })); +vi.mock('../../src/auth/workos-client.js', () => ({ + getAdminCredentialMutationWorkos: () => ({ userManagement: provider }), +})); +vi.mock('../../src/logger.js', () => ({ createLogger: () => logs })); +vi.mock('../../src/addie/error-notifier.js', () => ({ notifySystemError: vi.fn() })); + +import { initializeDatabase, closeDatabase } from '../../src/db/client.js'; +import { runMigrations } from '../../src/db/migrate.js'; +import { deleteIdentityCredentialTransaction, withCredentialCreationEventMutation } from '../../src/db/identity-db.js'; +import { getAuthorizationFingerprint, readCredentialAuthorizationLifecycle } from '../../src/db/authorization-epoch-db.js'; +import { createAndBindAdminCredential } from '../../src/services/admin-credential-bind.js'; + +const PREFIX = 'user_admin_bind_lifecycle_'; +const HOST = `${PREFIX}sam`; +const HOST_SIBLING = `${PREFIX}sam_existing`; +const ACTOR_PRIMARY = `${PREFIX}jordan`; +const ACTOR = `${PREFIX}jordan_credential`; +const CREATED = `${PREFIX}sam_new`; +const EMAIL = 'sam.new@admin-bind-lifecycle.example'; +const SECRET = 'provider-response-token-must-not-be-recorded'; + +function deferred() { + let resolve!: (value: T) => void; + const promise = new Promise((complete) => { resolve = complete; }); + return { promise, resolve }; +} + +type RawQuery = (sql: string, params?: unknown[]) => Promise; +interface QueryHooks { + before?: (sql: string, params: unknown[] | undefined, query: RawQuery) => Promise; + after?: (sql: string, params: unknown[] | undefined, result: QueryResult) => Promise; +} + +describe('admin credential binding and terminal lifecycle', () => { + let pool: Pool; + let actorIdentityId: string; + let hostIdentityId: string; + const identityIds = new Set(); + + beforeAll(async () => { + pool = initializeDatabase({ + connectionString: process.env.DATABASE_URL || 'postgresql://adcp:localdev@localhost:5432/adcp_test', + }); + await runMigrations(); + }, 60000); + + afterAll(async () => { + if (pool) await cleanup(); + await closeDatabase(); + }); + + beforeEach(async () => { + await cleanup(); + vi.clearAllMocks(); + provider.createUser.mockReset().mockResolvedValue(createdUser()); + provider.deleteUser.mockReset().mockResolvedValue(undefined); + await insertUser(HOST, 'sam@admin-bind-lifecycle.example', 'Sam', 'Adeyemi'); + await insertUser(HOST_SIBLING, 'sam.existing@admin-bind-lifecycle.example', 'Sam', 'Adeyemi'); + await insertUser(ACTOR_PRIMARY, 'jordan@admin-bind-lifecycle.example', 'Jordan', 'Ochoa'); + await insertUser(ACTOR, 'jordan.credential@admin-bind-lifecycle.example', 'Jordan', 'Ochoa'); + actorIdentityId = await identityOf(ACTOR_PRIMARY); + hostIdentityId = await identityOf(HOST); + for (const [identityId, userId] of [[actorIdentityId, ACTOR], [hostIdentityId, HOST_SIBLING]]) { + await pool.query( + 'UPDATE identity_workos_users SET identity_id = $1, is_primary = false WHERE workos_user_id = $2', + [identityId, userId], + ); + } + }); + + function input() { return { hostUserId: HOST, email: EMAIL, actorUserId: ACTOR, actorIdentityId }; } + function createdUser() { + return { + id: CREATED, email: EMAIL, firstName: 'Sam', lastName: 'Adeyemi', emailVerified: true, + createdAt: '2026-09-14T00:00:00.000Z', updatedAt: '2026-09-14T00:00:00.000Z', + }; + } + async function identityOf(userId: string) { + const binding = (await pool.query<{ identity_id: string }>( + 'SELECT identity_id FROM identity_workos_users WHERE workos_user_id = $1', [userId], + )).rows[0]; + if (!binding) throw new Error('Expected fixture credential binding'); + return binding.identity_id; + } + async function insertUser(id: string, email: string, firstName: string, lastName: string) { + await pool.query( + `INSERT INTO users (workos_user_id, email, first_name, last_name, email_verified, + workos_created_at, workos_updated_at) VALUES ($1, $2, $3, $4, true, NOW(), NOW())`, + [id, email, firstName, lastName], + ); + identityIds.add(await identityOf(id)); + } + async function cleanup() { + const identities = await pool.query<{ identity_id: string }>( + 'SELECT identity_id FROM identity_workos_users WHERE starts_with(workos_user_id, $1)', [PREFIX], + ); + identities.rows.forEach((row) => identityIds.add(row.identity_id)); + await pool.query( + `DELETE FROM registry_audit_log WHERE starts_with(workos_user_id, $1) + OR (action = 'admin_credential_bind' AND details->>'host_user_id' = $2)`, [PREFIX, HOST], + ); + await pool.query('DELETE FROM admin_credential_bind_operations WHERE host_user_id = $1', [HOST]); + await pool.query('DELETE FROM users WHERE starts_with(workos_user_id, $1)', [PREFIX]); + if (identityIds.size) await pool.query('DELETE FROM identities WHERE id = ANY($1::uuid[])', [[...identityIds]]); + identityIds.clear(); + } + async function operations() { + return (await pool.query( + `SELECT id, status, provider_user_id, failure_code FROM admin_credential_bind_operations + WHERE host_user_id = $1 ORDER BY created_at, id`, [HOST], + )).rows; + } + async function addTerminalMarker(userId: string, action = 'identity_credential_deleted') { + await pool.query( + `INSERT INTO registry_audit_log + (workos_organization_id, workos_user_id, action, resource_type, resource_id, details) + VALUES ('system', $1::text, $2::text, 'identity_credential', $1::text, '{}'::jsonb)`, [userId, action], + ); + } + async function expectNoCreatedArtifacts(observedIdentities: Set = new Set()) { + for (const table of ['users', 'identity_workos_users', 'authorization_epochs']) { + expect((await pool.query(`SELECT 1 FROM ${table} WHERE workos_user_id = $1`, [CREATED])).rows).toEqual([]); + } + if (observedIdentities.size) { + expect((await pool.query('SELECT 1 FROM identities WHERE id = ANY($1::uuid[])', [[...observedIdentities]])).rows).toEqual([]); + } + expect((await pool.query( + `SELECT 1 FROM registry_audit_log WHERE action = 'admin_credential_bind' + AND details->>'host_user_id' = $1 AND details->>'status' = 'committed'`, [HOST], + )).rows).toEqual([]); + } + async function expectBlockedReplay(operationId: string) { + const replay = await createAndBindAdminCredential(input()); + expect(replay.status).toBeGreaterThanOrEqual(400); + expect(replay.body).toMatchObject({ reconciliation_required: true, operation_id: operationId }); + expect(provider.createUser).toHaveBeenCalledTimes(1); + } + function delayedCreationCallback() { + return vi.fn(async (client: PoolClient) => { + await client.query( + `INSERT INTO users (workos_user_id, email, first_name, last_name, email_verified, + workos_created_at, workos_updated_at) VALUES ($1, $2, 'Sam', 'Adeyemi', true, NOW(), NOW())`, + [CREATED, EMAIL], + ); + }); + } + async function expectDelayedCreationBlocked() { + const callback = delayedCreationCallback(); + expect(await withCredentialCreationEventMutation(CREATED, callback)).toEqual({ applied: false }); + expect(callback).not.toHaveBeenCalled(); + } + async function expectOwnCompensationQuarantine(operationId: string) { + const quarantine = await pool.query<{ resource_type: string; resource_id: string; details: Record }>( + `SELECT resource_type, resource_id, details FROM registry_audit_log WHERE workos_user_id = $1 + AND action = 'identity_credential_admin_compensation_quarantined'`, [CREATED], + ); + expect(quarantine.rows).toHaveLength(1); + expect(quarantine.rows[0]).toMatchObject({ resource_type: 'admin_credential_bind_operation', resource_id: operationId }); + expect(quarantine.rows[0].details).toMatchObject({ + operation_id: operationId, provider_user_id: CREATED, + acting_workos_user_id: ACTOR, actor_identity_id: actorIdentityId, + quarantine_reason: 'admin_credential_compensation_pending', provider_delete_confirmed: false, + }); + } + + function observedPool(hooks: QueryHooks = {}) { + let pid: number | undefined; + const createdIdentities = new Set(); + const statements: string[] = []; + const proxy = new Proxy(pool, { + get(target, property) { + if (property === 'connect') return async () => { + const client = await target.connect(); + const query: RawQuery = client.query.bind(client); + const release = client.release.bind(client); + pid = (await query('SELECT pg_backend_pid() AS pid')).rows[0].pid; + client.query = (async (sql: string, params?: unknown[]) => { + statements.push(sql); + await hooks.before?.(sql, params, query); + const result = await query(sql, params); + if (sql.includes('INSERT INTO users') && params?.[0] === CREATED) { + const identity = await query('SELECT identity_id FROM identity_workos_users WHERE workos_user_id = $1', [CREATED]); + if (identity.rows[0]) { + createdIdentities.add(identity.rows[0].identity_id); + identityIds.add(identity.rows[0].identity_id); + } + } + return (await hooks.after?.(sql, params, result)) ?? result; + }) as PoolClient['query']; + client.release = (error?: boolean | Error) => { + client.query = query as PoolClient['query']; + client.release = release; + release(error); + }; + return client; + }; + const value = Reflect.get(target, property); + return typeof value === 'function' ? value.bind(target) : value; + }, + }); + return { pool: proxy, createdIdentities, statements, pid: () => pid }; + } + + it('refuses recreation when deletion of the unseen ID commits before the provider create reply', async () => { + provider.createUser.mockImplementationOnce(async () => { + expect((await deleteIdentityCredentialTransaction(CREATED, 'workos_webhook')).deleted).toBe(false); + expect(await readCredentialAuthorizationLifecycle(CREATED)).toEqual({ status: 'terminal', reason: 'deleted_or_quarantined' }); + return createdUser(); + }); + const observed = observedPool(); + + const result = await createAndBindAdminCredential(input(), { pool: observed.pool, workos: provider }); + + expect(result.status).toBeGreaterThanOrEqual(400); + expect(result.body.reconciliation_required).toBe(true); + const [operation] = await operations(); + expect(operation).toMatchObject({ status: 'reconciliation_required', provider_user_id: CREATED }); + expect(observed.statements.some((sql) => sql.includes('INSERT INTO users'))).toBe(false); + await expectNoCreatedArtifacts(observed.createdIdentities); + await expectBlockedReplay(operation.id); + expect(provider.deleteUser).not.toHaveBeenCalled(); + }); + + it('refuses a binding when the host is quarantined by primary deletion during provider creation', async () => { + provider.createUser.mockImplementationOnce(async () => { + const deletion = await deleteIdentityCredentialTransaction(HOST, 'workos_webhook'); + expect(deletion.quarantine).not.toBeNull(); + return createdUser(); + }); + + const result = await createAndBindAdminCredential(input()); + + expect(result.body.reconciliation_required).toBe(true); + expect(await readCredentialAuthorizationLifecycle(HOST)).toEqual({ status: 'terminal', reason: 'deleted_or_quarantined' }); + await expectNoCreatedArtifacts(); + const [operation] = await operations(); + expect(operation.status).toBe('reconciliation_required'); + await expectBlockedReplay(operation.id); + expect(provider.deleteUser).not.toHaveBeenCalled(); + expect(await identityOf(HOST_SIBLING)).toBe(hostIdentityId); + }); + + it('requires reconciliation when the authenticated actor loses its primary during provider creation', async () => { + provider.createUser.mockImplementationOnce(async () => { + await deleteIdentityCredentialTransaction(ACTOR_PRIMARY, 'workos_webhook'); + return createdUser(); + }); + + const result = await createAndBindAdminCredential(input()); + + expect(await readCredentialAuthorizationLifecycle(ACTOR)).toEqual({ status: 'terminal', reason: 'missing_primary' }); + expect((await pool.query( + `SELECT 1 FROM registry_audit_log WHERE workos_user_id = $1 + AND action IN ('identity_credential_deleted', 'identity_primary_deletion_quarantined')`, [ACTOR], + )).rows).toEqual([]); + expect(result.status).toBeGreaterThanOrEqual(400); + expect(result.body.reconciliation_required).toBe(true); + await expectNoCreatedArtifacts(); + const [operation] = await operations(); + expect(operation).toMatchObject({ status: 'reconciliation_required', provider_user_id: CREATED }); + await expectBlockedReplay(operation.id); + expect(provider.deleteUser).not.toHaveBeenCalled(); + }); + + it('blocks a quarantined new credential even when no local user or binding exists', async () => { + provider.createUser.mockImplementationOnce(async () => { + await addTerminalMarker(CREATED, 'identity_primary_deletion_quarantined'); + return createdUser(); + }); + + const result = await createAndBindAdminCredential(input()); + + expect(result.body.reconciliation_required).toBe(true); + expect(await readCredentialAuthorizationLifecycle(CREATED)).toEqual({ status: 'terminal', reason: 'deleted_or_quarantined' }); + await expectNoCreatedArtifacts(); + await expectBlockedReplay((await operations())[0].id); + expect(provider.deleteUser).not.toHaveBeenCalled(); + }); + + it.each([HOST, ACTOR])('rejects a terminal marker on %s that appears during provider creation', async (userId) => { + provider.createUser.mockImplementationOnce(async () => { + await addTerminalMarker(userId); + return createdUser(); + }); + + const result = await createAndBindAdminCredential(input()); + + expect(result.body.reconciliation_required).toBe(true); + expect(await readCredentialAuthorizationLifecycle(userId)).toEqual({ status: 'terminal', reason: 'deleted_or_quarantined' }); + await expectNoCreatedArtifacts(); + await expectBlockedReplay((await operations())[0].id); + expect(provider.deleteUser).not.toHaveBeenCalled(); + }); + + it('keeps a successful replay on the same operation without provider calls or additional epoch bumps', async () => { + const first = await createAndBindAdminCredential(input()); + expect(first.status).toBe(201); + const fingerprint = await getAuthorizationFingerprint([HOST, HOST_SIBLING, CREATED]); + + const replay = await createAndBindAdminCredential(input()); + + expect(replay.status).toBe(200); + expect(replay.body.operation_id).toBe(first.body.operation_id); + expect(await operations()).toHaveLength(1); + expect(await getAuthorizationFingerprint([HOST, HOST_SIBLING, CREATED])).toBe(fingerprint); + expect(provider.createUser).toHaveBeenCalledTimes(1); + expect(provider.deleteUser).not.toHaveBeenCalled(); + }); + + it.each([ + [CREATED, 'identity_credential_deleted'], + [CREATED, 'identity_primary_deletion_quarantined'], + [HOST, 'identity_credential_deleted'], + [ACTOR, 'identity_credential_deleted'], + ])('refuses committed replay when %s has marker %s despite intact local rows', async (userId, action) => { + const first = await createAndBindAdminCredential(input()); + expect(first.status).toBe(201); + await addTerminalMarker(userId, action); + expect(await identityOf(CREATED)).toBe(hostIdentityId); + const fingerprint = await getAuthorizationFingerprint([HOST, HOST_SIBLING, CREATED]); + + await expectBlockedReplay(String(first.body.operation_id)); + + expect(await getAuthorizationFingerprint([HOST, HOST_SIBLING, CREATED])).toBe(fingerprint); + expect(provider.deleteUser).not.toHaveBeenCalled(); + }); + + it('serializes deletion behind a binding commit and removes the credential on provider redelivery', async () => { + const atCommit = deferred(); + const releaseCommit = deferred(); + let inserted = false; + let paused = false; + const observed = observedPool({ + before: async (sql) => { + if (inserted && !paused && sql.trim() === 'COMMIT') { + paused = true; + atCommit.resolve(); + await releaseCommit.promise; + } + }, + after: async (sql) => { if (sql.includes('INSERT INTO users')) inserted = true; }, + }); + const binding = createAndBindAdminCredential(input(), { pool: observed.pool, workos: provider }); + await atCommit.promise; + const deletion = deleteIdentityCredentialTransaction(CREATED, 'workos_webhook') + .then((value) => ({ value, error: undefined }), (error: Error) => ({ value: undefined, error })); + try { + await expect.poll(async () => (await pool.query<{ blocked: boolean }>( + `SELECT EXISTS ( + SELECT 1 FROM pg_locks waiting JOIN pg_locks holder + ON waiting.locktype = holder.locktype AND waiting.database = holder.database + AND waiting.classid = holder.classid AND waiting.objid = holder.objid + AND waiting.objsubid = holder.objsubid + WHERE waiting.locktype = 'advisory' AND NOT waiting.granted AND holder.granted + AND holder.pid = $1 + AND holder.classid = ((hashtextextended($2, 6827) >> 32) & 4294967295)::oid + AND holder.objid = (hashtextextended($2, 6827) & 4294967295)::oid + ) AS blocked`, [observed.pid(), CREATED], + )).rows[0].blocked, { timeout: 750, interval: 10 }).toBe(true); + } finally { + releaseCommit.resolve(); + } + const result = await binding; + expect(result.status).toBe(201); + const firstDeletion = await deletion; + // Discovery preceded the uncommitted INSERT. The existing deletion helper + // rejects its stale topology snapshot; WorkOS redelivery then sees it. + expect(firstDeletion.error?.message).toBe('Credential identity changed while acquiring deletion locks'); + expect((await deleteIdentityCredentialTransaction(CREATED, 'workos_webhook')).deleted).toBe(true); + expect(await readCredentialAuthorizationLifecycle(CREATED)).toEqual({ status: 'terminal', reason: 'deleted_or_quarantined' }); + for (const table of ['users', 'identity_workos_users', 'authorization_epochs']) { + expect((await pool.query(`SELECT 1 FROM ${table} WHERE workos_user_id = $1`, [CREATED])).rows).toEqual([]); + } + expect((await pool.query('SELECT 1 FROM identities WHERE id = ANY($1::uuid[])', [[...observed.createdIdentities]])).rows).toEqual([]); + await expectBlockedReplay(String(result.body.operation_id)); + expect(provider.deleteUser).not.toHaveBeenCalled(); + }); + + it.each(['lock_query', 'lock_cardinality', 'snapshot_query', 'snapshot_cardinality'] as const)( + 'requires reconciliation without deletion when the lifecycle fence fails (%s)', async (fault) => { + let injected = false; + const observed = observedPool({ + before: async (sql) => { + if (!injected && ((fault === 'lock_query' && sql.includes('pg_advisory_xact_lock')) + || (fault === 'snapshot_query' && sql.includes('terminal_marker') && sql.includes('requested.workos_user_id')))) { + injected = true; + throw new Error(SECRET); + } + }, + after: async (sql, _params, result) => { + if (!injected && ((fault === 'lock_cardinality' && sql.includes('pg_advisory_xact_lock')) + || (fault === 'snapshot_cardinality' && sql.includes('terminal_marker') && sql.includes('requested.workos_user_id')))) { + injected = true; + return { ...result, rows: [], rowCount: 0 }; + } + }, + }); + + const result = await createAndBindAdminCredential(input(), { pool: observed.pool, workos: provider }); + + expect(injected).toBe(true); + expect(result.body.reconciliation_required).toBe(true); + const [operation] = await operations(); + expect(operation).toMatchObject({ status: 'reconciliation_required', provider_user_id: CREATED }); + expect(operation.failure_code).toBeTruthy(); + await expectNoCreatedArtifacts(observed.createdIdentities); + await expectBlockedReplay(operation.id); + expect(provider.deleteUser).not.toHaveBeenCalled(); + expect(JSON.stringify({ result, operation, logs: Object.values(logs).map((log) => log.mock.calls) })).not.toContain(SECRET); + }, + ); + + it('rolls back the complete binding if the success audit reports the wrong cardinality', async () => { + let injected = false; + const observed = observedPool({ + after: async (sql, params, result) => { + if (!injected && sql.includes('UPDATE admin_credential_bind_operations') && params?.[1] === 'committed') { + injected = true; + return { ...result, rowCount: 0 }; + } + }, + }); + + const result = await createAndBindAdminCredential(input(), { pool: observed.pool, workos: provider }); + + expect(injected).toBe(true); + expect(result.body.compensated).toBe(true); + expect(observed.createdIdentities.size).toBe(1); + await expectNoCreatedArtifacts(observed.createdIdentities); + expect((await operations())[0].status).toBe('compensated'); + expect(provider.deleteUser).toHaveBeenCalledExactlyOnceWith(CREATED); + expect(await getAuthorizationFingerprint([HOST, HOST_SIBLING])).toBe(''); + }); + + it('rolls back every binding and epoch when the bump receipt omits the new credential', async () => { + let injected = false; + const observed = observedPool({ + after: async (sql, _params, result) => { + if (!injected && sql.includes('INSERT INTO authorization_epochs')) { + injected = true; + const rows = result.rows.filter((row) => row.workos_user_id !== CREATED); + return { ...result, rows, rowCount: rows.length }; + } + }, + }); + + const result = await createAndBindAdminCredential(input(), { pool: observed.pool, workos: provider }); + + expect(injected).toBe(true); + expect(result.body.compensated).toBe(true); + expect(observed.createdIdentities.size).toBe(1); + await expectNoCreatedArtifacts(observed.createdIdentities); + expect(await getAuthorizationFingerprint([HOST, HOST_SIBLING])).toBe(''); + expect(provider.deleteUser).toHaveBeenCalledExactlyOnceWith(CREATED); + }); + + it.each(['deletion', 'local_user'] as const)( + 'does not compensate a provider credential whose lifecycle changed after local rollback (%s)', async (change) => { + let failed = false; + let changed = false; + let racedIdentity: string | undefined; + const observed = observedPool({ + before: async (sql, params, query) => { + if (!failed && sql.includes('INSERT INTO users') && params?.[0] === CREATED) { + failed = true; + // Abort the real local transaction, then let the independent + // deletion/upsert commit before the compensation fence is read. + await query('SELECT 1 / 0'); + } + }, + after: async (sql) => { + if (failed && !changed && sql.trim() === 'ROLLBACK') { + changed = true; + if (change === 'deletion') await deleteIdentityCredentialTransaction(CREATED, 'workos_webhook'); + else { + await insertUser(CREATED, EMAIL, 'Sam', 'Adeyemi'); + racedIdentity = await identityOf(CREATED); + } + } + }, + }); + + const result = await createAndBindAdminCredential(input(), { pool: observed.pool, workos: provider }); + + expect(failed && changed).toBe(true); + expect(result.body.reconciliation_required).toBe(true); + expect(provider.deleteUser).not.toHaveBeenCalled(); + const [operation] = await operations(); + expect(operation).toMatchObject({ status: 'reconciliation_required', provider_user_id: CREATED }); + await expectBlockedReplay(operation.id); + if (change === 'deletion') await expectNoCreatedArtifacts(); + else { + expect(await identityOf(CREATED)).toBe(racedIdentity); + expect(racedIdentity).not.toBe(hostIdentityId); + expect(await getAuthorizationFingerprint([HOST, HOST_SIBLING, CREATED])).toBe(''); + } + }, + ); + + it.each(['confirmed', 'timeout', 'error'] as const)( + 'records compensation deletion only after confirmation (%s) and keeps provider deletion replay distinct', async (outcome) => { + let failed = false; + const observed = observedPool({ + before: async (sql, params, query) => { + if (!failed && sql.includes('INSERT INTO users') && params?.[0] === CREATED) { + failed = true; + await query('SELECT 1 / 0'); + } + }, + }); + if (outcome === 'timeout') { + provider.deleteUser.mockRejectedValueOnce(Object.assign(new Error(SECRET), { code: 'ETIMEDOUT' })); + } else if (outcome === 'error') { + provider.deleteUser.mockRejectedValueOnce(Object.assign(new Error(SECRET), { status: 503 })); + } + + const result = await createAndBindAdminCredential(input(), { pool: observed.pool, workos: provider }); + + expect(failed).toBe(true); + if (outcome === 'confirmed') expect(result.body.compensated).toBe(true); + else expect(result.body.reconciliation_required).toBe(true); + expect(provider.deleteUser).toHaveBeenCalledExactlyOnceWith(CREATED); + const beforeProviderDeletion = await pool.query<{ action: string }>( + `SELECT action FROM registry_audit_log WHERE workos_user_id = $1 + AND action IN ('identity_credential_deleted', 'identity_credential_admin_compensation_deleted')`, + [CREATED], + ); + const delayedCreation = delayedCreationCallback(); + await expectOwnCompensationQuarantine((await operations())[0].id); + if (outcome === 'confirmed') { + expect(beforeProviderDeletion.rows).toEqual([{ action: 'identity_credential_admin_compensation_deleted' }]); + } else { + // The pre-delete quarantine is truthful containment. It blocks late + // creation without falsely attesting that the provider user is gone. + expect(beforeProviderDeletion.rows).toEqual([]); + await expectBlockedReplay((await operations())[0].id); + } + expect(await readCredentialAuthorizationLifecycle(CREATED)).toEqual({ status: 'terminal', reason: 'deleted_or_quarantined' }); + expect(await withCredentialCreationEventMutation(CREATED, delayedCreation)).toEqual({ applied: false }); + expect(delayedCreation).not.toHaveBeenCalled(); + await expectNoCreatedArtifacts(observed.createdIdentities); + + // A later signed deletion event remains distinct from the local + // confirmed compensation and produces its own truthful replay record. + const confirmedDeletion = await deleteIdentityCredentialTransaction(CREATED, 'workos_webhook'); + expect(confirmedDeletion.deleted).toBe(false); + expect(confirmedDeletion.replay).toBeNull(); + const deletionReplay = await deleteIdentityCredentialTransaction(CREATED, 'workos_webhook'); + expect(deletionReplay.replay).toMatchObject({ kind: 'same_source', original_deletion_source: 'workos_webhook' }); + expect((await pool.query( + `SELECT 1 FROM registry_audit_log WHERE workos_user_id = $1 AND action = 'identity_credential_deleted'`, [CREATED], + )).rows).toHaveLength(1); + expect(await withCredentialCreationEventMutation(CREATED, delayedCreation)).toEqual({ applied: false }); + expect(delayedCreation).not.toHaveBeenCalled(); + expect(JSON.stringify({ result, logs: Object.values(logs).map((log) => log.mock.calls) })).not.toContain(SECRET); + }, + ); + + it('requires reconciliation if the confirmed compensation marker cannot be durably recorded', async () => { + let localFailed = false; + let markerFailed = false; + const observed = observedPool({ + before: async (sql, params, query) => { + if (!localFailed && sql.includes('INSERT INTO users') && params?.[0] === CREATED) { + localFailed = true; + await query('SELECT 1 / 0'); + } + }, + after: async (sql, _params, result) => { + if (!markerFailed && sql.includes('INSERT INTO registry_audit_log') + && sql.includes('identity_credential_admin_compensation_deleted')) { + markerFailed = true; + return { ...result, rowCount: 0 }; + } + }, + }); + + const result = await createAndBindAdminCredential(input(), { pool: observed.pool, workos: provider }); + + expect(localFailed && markerFailed).toBe(true); + expect(result.body.reconciliation_required).toBe(true); + expect(result.body.compensated).not.toBe(true); + expect(provider.deleteUser).toHaveBeenCalledExactlyOnceWith(CREATED); + const [operation] = await operations(); + expect(operation).toMatchObject({ status: 'reconciliation_required', provider_user_id: CREATED }); + expect(operation.failure_code).toBeTruthy(); + expect((await pool.query( + `SELECT 1 FROM registry_audit_log WHERE workos_user_id = $1 + AND action = 'identity_credential_admin_compensation_deleted'`, [CREATED], + )).rows).toEqual([]); + await expectOwnCompensationQuarantine(operation.id); + await expectDelayedCreationBlocked(); + await expectNoCreatedArtifacts(); + await expectBlockedReplay(operation.id); + expect(provider.deleteUser).toHaveBeenCalledTimes(1); + }); + + it('preserves a committed compensation marker after a lost COMMIT reply without claiming success or replaying deletion', async () => { + let localFailed = false; + let replyLost = false; + const observed = observedPool({ + before: async (sql, params, query) => { + if (!localFailed && sql.includes('INSERT INTO users') && params?.[0] === CREATED) { + localFailed = true; + await query('SELECT 1 / 0'); + } + }, + after: async (sql) => { + if (!replyLost && sql.trim() === 'COMMIT' && provider.deleteUser.mock.calls.length === 1) { + replyLost = true; + throw Object.assign(new Error('compensation commit reply lost'), { code: 'ECONNRESET' }); + } + }, + }); + + const result = await createAndBindAdminCredential(input(), { pool: observed.pool, workos: provider }); + + expect(localFailed && replyLost).toBe(true); + expect(result.body.reconciliation_required).toBe(true); + expect(result.body.compensated).not.toBe(true); + expect(provider.deleteUser).toHaveBeenCalledExactlyOnceWith(CREATED); + const [operation] = await operations(); + expect(operation).toMatchObject({ status: 'reconciliation_required', provider_user_id: CREATED }); + expect((await pool.query( + `SELECT 1 FROM registry_audit_log WHERE workos_user_id = $1 + AND action = 'identity_credential_admin_compensation_deleted'`, [CREATED], + )).rows).toHaveLength(1); + expect(await readCredentialAuthorizationLifecycle(CREATED)).toEqual({ status: 'terminal', reason: 'deleted_or_quarantined' }); + await expectOwnCompensationQuarantine(operation.id); + await expectDelayedCreationBlocked(); + await expectNoCreatedArtifacts(); + await expectBlockedReplay(operation.id); + expect(provider.deleteUser).toHaveBeenCalledTimes(1); + }); + + it.each(['insert_error', 'insert_cardinality', 'audit_cardinality', 'commit_error', 'commit_receipt', 'commit_reply_lost'] as const)( + 'does not delete upstream before compensation quarantine is confirmed (%s)', async (fault) => { + let localFailed = false; + let quarantineStarted = false; + let injected = false; + const observed = observedPool({ + before: async (sql, params, query) => { + if (!localFailed && sql.includes('INSERT INTO users') && params?.[0] === CREATED) { + localFailed = true; + await query('SELECT 1 / 0'); + } + if (sql.includes('INSERT INTO registry_audit_log') + && sql.includes('identity_credential_admin_compensation_quarantined')) { + quarantineStarted = true; + if (fault === 'insert_error' && !injected) { + injected = true; + throw new Error(SECRET); + } + } + if (!injected && fault === 'commit_error' && quarantineStarted + && sql.trim() === 'COMMIT' && provider.deleteUser.mock.calls.length === 0) { + injected = true; + throw new Error(SECRET); + } + }, + after: async (sql, params, result) => { + if (!injected && fault === 'insert_cardinality' && sql.includes('INSERT INTO registry_audit_log') + && sql.includes('identity_credential_admin_compensation_quarantined')) { + injected = true; + return { ...result, rows: [], rowCount: 0 }; + } + if (!injected && fault === 'audit_cardinality' && sql.includes('UPDATE admin_credential_bind_operations') + && params?.[1] === 'compensating') { + injected = true; + return { ...result, rowCount: 0 }; + } + if (!injected && quarantineStarted && sql.trim() === 'COMMIT' + && provider.deleteUser.mock.calls.length === 0 && ['commit_receipt', 'commit_reply_lost'].includes(fault)) { + injected = true; + if (fault === 'commit_reply_lost') throw Object.assign(new Error(SECRET), { code: 'ECONNRESET' }); + return { ...result, command: 'ROLLBACK' }; + } + }, + }); + + const result = await createAndBindAdminCredential(input(), { pool: observed.pool, workos: provider }); + + expect(localFailed && quarantineStarted && injected).toBe(true); + expect(result.body.reconciliation_required).toBe(true); + expect(result.body.compensated).not.toBe(true); + expect(provider.deleteUser).not.toHaveBeenCalled(); + const [operation] = await operations(); + expect(operation).toMatchObject({ status: 'reconciliation_required', provider_user_id: CREATED }); + if (fault === 'commit_receipt' || fault === 'commit_reply_lost') { + // The actual quarantine COMMIT succeeded; its reply did not attest + // that outcome, so upstream deletion still must not be attempted. + await expectOwnCompensationQuarantine(operation.id); + await expectDelayedCreationBlocked(); + } else { + expect((await pool.query( + `SELECT 1 FROM registry_audit_log WHERE workos_user_id = $1 + AND action = 'identity_credential_admin_compensation_quarantined'`, [CREATED], + )).rows).toEqual([]); + } + await expectNoCreatedArtifacts(); + await expectBlockedReplay(operation.id); + expect(JSON.stringify({ result, operation, logs: Object.values(logs).map((log) => log.mock.calls) })).not.toContain(SECRET); + }, + ); + + it.each(['operation_id', 'provider_user_id', 'acting_workos_user_id', 'actor_identity_id', 'resource_id', 'resource_type'] as const)( + 'refuses upstream deletion if committed quarantine has conflicting %s', async (field) => { + let localFailed = false; + let quarantineStarted = false; + let changed = false; + let changedRows: number | null = null; + const observed = observedPool({ + before: async (sql, params, query) => { + if (!localFailed && sql.includes('INSERT INTO users') && params?.[0] === CREATED) { + localFailed = true; + await query('SELECT 1 / 0'); + } + }, + after: async (sql) => { + if (sql.includes('INSERT INTO registry_audit_log') + && sql.includes('identity_credential_admin_compensation_quarantined')) quarantineStarted = true; + if (!changed && quarantineStarted && sql.trim() === 'COMMIT' && provider.deleteUser.mock.calls.length === 0) { + changed = true; + const conflictingValues = { + operation_id: '00000000-0000-0000-0000-000000000001', + provider_user_id: HOST, + acting_workos_user_id: ACTOR_PRIMARY, + actor_identity_id: hostIdentityId, + resource_id: '00000000-0000-0000-0000-000000000001', + resource_type: 'identity_credential', + }; + if (field === 'resource_id' || field === 'resource_type') { + const sql = field === 'resource_id' + ? `UPDATE registry_audit_log SET resource_id = $1::text + WHERE workos_user_id = $2 AND action = 'identity_credential_admin_compensation_quarantined'` + : `UPDATE registry_audit_log SET resource_type = $1::text + WHERE workos_user_id = $2 AND action = 'identity_credential_admin_compensation_quarantined'`; + changedRows = (await pool.query(sql, [conflictingValues[field], CREATED])).rowCount; + } else { + changedRows = (await pool.query( + `UPDATE registry_audit_log SET details = jsonb_set(details, ARRAY[$1::text], to_jsonb($2::text), true) + WHERE workos_user_id = $3 AND action = 'identity_credential_admin_compensation_quarantined'`, + [field, conflictingValues[field], CREATED], + )).rowCount; + } + } + }, + }); + + const result = await createAndBindAdminCredential(input(), { pool: observed.pool, workos: provider }); + + expect(localFailed && changed).toBe(true); + expect(changedRows).toBe(1); + expect(result.body.reconciliation_required).toBe(true); + expect(provider.deleteUser).not.toHaveBeenCalled(); + await expectDelayedCreationBlocked(); + await expectNoCreatedArtifacts(); + await expectBlockedReplay((await operations())[0].id); + }, + ); + + it('rejects a creation callback delivered between quarantine commit and lifecycle lock reacquisition', async () => { + let localFailed = false; + let quarantineStarted = false; + let delivered = false; + const callback = delayedCreationCallback(); + const observed = observedPool({ + before: async (sql, params, query) => { + if (!localFailed && sql.includes('INSERT INTO users') && params?.[0] === CREATED) { + localFailed = true; + await query('SELECT 1 / 0'); + } + }, + after: async (sql) => { + if (sql.includes('INSERT INTO registry_audit_log') + && sql.includes('identity_credential_admin_compensation_quarantined')) quarantineStarted = true; + if (!delivered && quarantineStarted && sql.trim() === 'COMMIT' && provider.deleteUser.mock.calls.length === 0) { + delivered = true; + const [operation] = await operations(); + expect(operation.status).toBe('compensating'); + await expectOwnCompensationQuarantine(operation.id); + expect(await withCredentialCreationEventMutation(CREATED, callback)).toEqual({ applied: false }); + } + }, + }); + + const result = await createAndBindAdminCredential(input(), { pool: observed.pool, workos: provider }); + + expect(localFailed && delivered).toBe(true); + expect(result.body.compensated).toBe(true); + expect(callback).not.toHaveBeenCalled(); + expect(provider.deleteUser).toHaveBeenCalledExactlyOnceWith(CREATED); + await expectOwnCompensationQuarantine((await operations())[0].id); + await expectNoCreatedArtifacts(); + }); + + it('preserves signed deletion delivered between quarantine commit and reacquisition without another provider delete', async () => { + let localFailed = false; + let quarantineStarted = false; + let delivered = false; + let signedDeletion: Awaited> | undefined; + const observed = observedPool({ + before: async (sql, params, query) => { + if (!localFailed && sql.includes('INSERT INTO users') && params?.[0] === CREATED) { + localFailed = true; + await query('SELECT 1 / 0'); + } + }, + after: async (sql) => { + if (sql.includes('INSERT INTO registry_audit_log') + && sql.includes('identity_credential_admin_compensation_quarantined')) quarantineStarted = true; + if (!delivered && quarantineStarted && sql.trim() === 'COMMIT' && provider.deleteUser.mock.calls.length === 0) { + delivered = true; + signedDeletion = await deleteIdentityCredentialTransaction(CREATED, 'workos_webhook'); + } + }, + }); + + const result = await createAndBindAdminCredential(input(), { pool: observed.pool, workos: provider }); + + expect(localFailed && delivered).toBe(true); + expect(signedDeletion?.deleted).toBe(false); + expect(signedDeletion?.replay).toBeNull(); + expect(result.status).toBeGreaterThanOrEqual(400); + expect(result.body.reconciliation_required).toBe(true); + expect(provider.deleteUser).not.toHaveBeenCalled(); + const [operation] = await operations(); + await expectOwnCompensationQuarantine(operation.id); + expect((await pool.query( + `SELECT details->>'deletion_source' AS deletion_source, + details->'actor'->>'type' AS actor_type + FROM registry_audit_log WHERE workos_user_id = $1 AND action = 'identity_credential_deleted'`, [CREATED], + )).rows).toEqual([{ deletion_source: 'workos_webhook', actor_type: 'workos_provider' }]); + const replay = await deleteIdentityCredentialTransaction(CREATED, 'workos_webhook'); + expect(replay.replay).toMatchObject({ kind: 'same_source', original_deletion_source: 'workos_webhook' }); + await expectDelayedCreationBlocked(); + await expectNoCreatedArtifacts(); + await expectBlockedReplay(operation.id); + }); + + it('holds the lifecycle lock from provider deletion through confirmed evidence and compensation commit', async () => { + const atProviderDelete = deferred(); + const releaseProviderDelete = deferred(); + const atCompensationCommit = deferred(); + const releaseCompensationCommit = deferred(); + let localFailed = false; + let confirmedMarkerInserted = false; + let creationSettled = false; + const callback = delayedCreationCallback(); + provider.deleteUser.mockImplementationOnce(async () => { + atProviderDelete.resolve(); + await releaseProviderDelete.promise; + }); + const observed = observedPool({ + before: async (sql, params, query) => { + if (!localFailed && sql.includes('INSERT INTO users') && params?.[0] === CREATED) { + localFailed = true; + await query('SELECT 1 / 0'); + } + if (confirmedMarkerInserted && sql.trim() === 'COMMIT') { + atCompensationCommit.resolve(); + await releaseCompensationCommit.promise; + } + }, + after: async (sql) => { + if (sql.includes('INSERT INTO registry_audit_log') + && sql.includes('identity_credential_admin_compensation_deleted')) confirmedMarkerInserted = true; + }, + }); + const binding = createAndBindAdminCredential(input(), { pool: observed.pool, workos: provider }); + await Promise.race([ + atProviderDelete.promise, + binding.then(() => { throw new Error('Compensation did not enter the provider deletion barrier'); }), + ]); + const creation = withCredentialCreationEventMutation(CREATED, callback).then( + (value) => { creationSettled = true; return { value, error: undefined }; }, + (error: Error) => { creationSettled = true; return { value: undefined, error }; }, + ); + async function expectCreationWaitingOnLifecycleLock() { + await expect.poll(async () => (await pool.query<{ blocked: boolean }>( + `SELECT EXISTS ( + SELECT 1 FROM pg_locks waiting JOIN pg_locks holder + ON waiting.locktype = holder.locktype AND waiting.database = holder.database + AND waiting.classid = holder.classid AND waiting.objid = holder.objid + AND waiting.objsubid = holder.objsubid + WHERE waiting.locktype = 'advisory' AND NOT waiting.granted AND holder.granted + AND holder.pid = $1 + AND holder.classid = ((hashtextextended($2, 6827) >> 32) & 4294967295)::oid + AND holder.objid = (hashtextextended($2, 6827) & 4294967295)::oid + ) AS blocked`, [observed.pid(), CREATED], + )).rows[0].blocked, { timeout: 750, interval: 10 }).toBe(true); + } + try { + await expectCreationWaitingOnLifecycleLock(); + expect(creationSettled).toBe(false); + releaseProviderDelete.resolve(); + await Promise.race([ + atCompensationCommit.promise, + binding.then(() => { throw new Error('Compensation did not reach its commit barrier'); }), + ]); + expect(confirmedMarkerInserted).toBe(true); + // The marker is written on the saga session but remains uncommitted; + // another session must still wait instead of observing a writable gap. + expect((await pool.query( + `SELECT 1 FROM registry_audit_log WHERE workos_user_id = $1 + AND action = 'identity_credential_admin_compensation_deleted'`, [CREATED], + )).rows).toEqual([]); + await expectCreationWaitingOnLifecycleLock(); + expect(creationSettled).toBe(false); + expect(callback).not.toHaveBeenCalled(); + } finally { + releaseProviderDelete.resolve(); + releaseCompensationCommit.resolve(); + } + + const result = await binding; + const delivery = await creation; + expect(result.body.compensated).toBe(true); + expect(delivery.error).toBeUndefined(); + expect(delivery.value).toEqual({ applied: false }); + expect(callback).not.toHaveBeenCalled(); + expect(provider.deleteUser).toHaveBeenCalledExactlyOnceWith(CREATED); + expect((await pool.query( + `SELECT 1 FROM registry_audit_log WHERE workos_user_id = $1 + AND action = 'identity_credential_admin_compensation_deleted'`, [CREATED], + )).rows).toHaveLength(1); + await expectOwnCompensationQuarantine((await operations())[0].id); + await expectNoCreatedArtifacts(); + }); + + it('does not delete upstream when the quarantine ownership query has an incomplete receipt', async () => { + let localFailed = false; + let injected = false; + const observed = observedPool({ + before: async (sql, params, query) => { + if (!localFailed && sql.includes('INSERT INTO users') && params?.[0] === CREATED) { + localFailed = true; + await query('SELECT 1 / 0'); + } + }, + after: async (sql, _params, result) => { + if (!injected && provider.deleteUser.mock.calls.length === 0 + && sql.includes('SELECT action, resource_type, resource_id, details FROM registry_audit_log')) { + injected = true; + return { ...result, rows: [], rowCount: 0 }; + } + }, + }); + + const result = await createAndBindAdminCredential(input(), { pool: observed.pool, workos: provider }); + + expect(localFailed && injected).toBe(true); + expect(result.body.reconciliation_required).toBe(true); + expect(provider.deleteUser).not.toHaveBeenCalled(); + const [operation] = await operations(); + await expectOwnCompensationQuarantine(operation.id); + await expectDelayedCreationBlocked(); + await expectNoCreatedArtifacts(); + await expectBlockedReplay(operation.id); + }); + + it.each(['audit_cardinality', 'commit_error'] as const)( + 'retains quarantine if confirmed-deletion evidence rolls back (%s)', async (fault) => { + let localFailed = false; + let injected = false; + const observed = observedPool({ + before: async (sql, params, query) => { + if (!localFailed && sql.includes('INSERT INTO users') && params?.[0] === CREATED) { + localFailed = true; + await query('SELECT 1 / 0'); + } + if (!injected && fault === 'commit_error' && sql.trim() === 'COMMIT' + && provider.deleteUser.mock.calls.length === 1) { + injected = true; + throw new Error(SECRET); + } + }, + after: async (sql, params, result) => { + if (!injected && fault === 'audit_cardinality' && sql.includes('UPDATE admin_credential_bind_operations') + && params?.[1] === 'compensated') { + injected = true; + return { ...result, rowCount: 0 }; + } + }, + }); + + const result = await createAndBindAdminCredential(input(), { pool: observed.pool, workos: provider }); + + expect(localFailed && injected).toBe(true); + expect(result.body.reconciliation_required).toBe(true); + expect(provider.deleteUser).toHaveBeenCalledExactlyOnceWith(CREATED); + const [operation] = await operations(); + expect(operation).toMatchObject({ status: 'reconciliation_required', provider_user_id: CREATED }); + await expectOwnCompensationQuarantine(operation.id); + expect((await pool.query( + `SELECT 1 FROM registry_audit_log WHERE workos_user_id = $1 + AND action = 'identity_credential_admin_compensation_deleted'`, [CREATED], + )).rows).toEqual([]); + await expectDelayedCreationBlocked(); + await expectNoCreatedArtifacts(); + await expectBlockedReplay(operation.id); + }, + ); +}); diff --git a/server/tests/integration/admin-link-unlink-credential.test.ts b/server/tests/integration/admin-link-unlink-credential.test.ts index 45cedc66bbc..5d76610f9ca 100644 --- a/server/tests/integration/admin-link-unlink-credential.test.ts +++ b/server/tests/integration/admin-link-unlink-credential.test.ts @@ -11,7 +11,6 @@ import { describe, it, expect, beforeAll, afterAll, beforeEach, vi } from 'vitest'; import request from 'supertest'; import type { Pool } from 'pg'; -import { readFileSync } from 'node:fs'; vi.hoisted(() => { process.env.WORKOS_API_KEY ??= 'sk_test_mock_key'; @@ -19,7 +18,21 @@ vi.hoisted(() => { process.env.WORKOS_COOKIE_PASSWORD ??= 'test-cookie-password-at-least-32-chars-long'; }); -const { mockGetUser } = vi.hoisted(() => ({ mockGetUser: vi.fn() })); +const { mockGetUser, adminContext, routeLogger } = vi.hoisted(() => ({ + mockGetUser: vi.fn(), adminContext: { identityId: '' }, + routeLogger: { info: vi.fn(), warn: vi.fn(), error: vi.fn() }, +})); + +vi.mock('../../src/logger.js', async (importOriginal) => { + const actual = await importOriginal(); + return { ...actual, createLogger: (...args: Parameters) => + args[0] === 'admin-users-routes' ? routeLogger : actual.createLogger(...args) }; +}); + +vi.mock('../../src/db/credential-detach-db.js', async (importOriginal) => { + const actual = await importOriginal(); + return { ...actual, detachCredential: vi.fn(actual.detachCredential) }; +}); vi.mock('../../src/auth/workos-client.js', () => { const mockUserManagement = { getUser: mockGetUser }; @@ -33,6 +46,7 @@ vi.mock('../../src/middleware/auth.js', async (importOriginal) => { req.user = { id: 'user_test_admin_link', authWorkosUserId: 'user_test_admin_link_credential', + identityId: adminContext.identityId, email: 'admin@test.local', emailVerified: true, createdAt: new Date().toISOString(), @@ -58,15 +72,18 @@ vi.mock('../../src/middleware/csrf.js', () => ({ csrfProtection: (_req: any, _res: any, next: any) => next(), })); -import { initializeDatabase, closeDatabase, getPool } from '../../src/db/client.js'; +import { initializeDatabase, closeDatabase } from '../../src/db/client.js'; import { runMigrations } from '../../src/db/migrate.js'; -import { HTTPServer } from '../../src/http.js'; +import express from 'express'; +import { createAdminUsersRouter } from '../../src/routes/admin/users.js'; +import { stopAuthTimers } from '../../src/middleware/auth.js'; +import { CredentialDetachConflict, detachCredential } from '../../src/db/credential-detach-db.js'; const HOST_USER_ID = 'user_test_link_host'; const TARGET_USER_ID = 'user_test_link_target'; +const AUTHORITY_ORG_ID = 'org_test_unlink_authority'; describe('admin link / unlink credential', () => { - let server: HTTPServer; let app: any; let pool: Pool; @@ -75,20 +92,23 @@ describe('admin link / unlink credential', () => { connectionString: process.env.DATABASE_URL || 'postgresql://adcp:localdev@localhost:5432/adcp_test', }); await runMigrations(); - server = new HTTPServer(); - await server.start(0); - app = server.app; + app = express(); + app.use(express.json()); + app.use('/api/admin/users', createAdminUsersRouter()); }, 60000); afterAll(async () => { await cleanup(); - await server?.stop(); + await pool.query(`DELETE FROM users WHERE workos_user_id IN ('user_test_admin_link', 'user_test_admin_link_credential')`); + stopAuthTimers(); await closeDatabase(); }); beforeEach(async () => { await cleanup(); mockGetUser.mockReset(); + vi.mocked(detachCredential).mockClear(); + for (const log of Object.values(routeLogger)) log.mockClear(); await pool.query( `INSERT INTO users (workos_user_id, email, first_name, last_name, email_verified, workos_created_at, workos_updated_at, created_at, updated_at) @@ -98,307 +118,61 @@ describe('admin link / unlink credential', () => { }); async function cleanup() { + await pool.query(`DELETE FROM organization_memberships WHERE workos_organization_id = $1`, [AUTHORITY_ORG_ID]); + await pool.query(`DELETE FROM organizations WHERE workos_organization_id = $1`, [AUTHORITY_ORG_ID]); + await pool.query(`DELETE FROM registry_audit_log WHERE resource_id IN ($1, $2)`, [HOST_USER_ID, TARGET_USER_ID]); await pool.query(`DELETE FROM users WHERE workos_user_id IN ($1, $2)`, [HOST_USER_ID, TARGET_USER_ID]); } - describe('POST /credentials (bind existing)', () => { - it('keeps the declared global-admin and people-UI confirmation contracts', () => { - const routeSource = readFileSync( - new URL('../../src/routes/admin/users.ts', import.meta.url), - 'utf8', - ); - const bindRoute = routeSource.match( - /router\.post\('\/:userId\/credentials', \.\.\.requireGlobalAdmin, async \(req, res\) => \{[\s\S]*?\n \}\);/ - )?.[0]; - expect(bindRoute).toContain('...requireGlobalAdmin'); - - const peopleUi = readFileSync( - new URL('../../public/admin-people.html', import.meta.url), - 'utf8', - ); - expect(peopleUi).toContain('data.consolidate_confirmation_required'); - expect(peopleUi).toContain('JSON.stringify({ workos_user_id: trimmed, consolidate: true })'); - }); - - it('binds an existing WorkOS user (already in local users) under host\'s identity', async () => { - // Insert the target locally so the trigger creates its singleton identity - await pool.query( - `INSERT INTO users (workos_user_id, email, first_name, last_name, email_verified, - workos_created_at, workos_updated_at, created_at, updated_at) - VALUES ($1, 'target@test.example', 'Target', 'User', true, NOW(), NOW(), NOW(), NOW())`, - [TARGET_USER_ID] - ); - - const response = await request(app) - .post(`/api/admin/users/${HOST_USER_ID}/credentials`) - .send({ workos_user_id: TARGET_USER_ID }) - .expect(201); - - expect(response.body).toMatchObject({ - linked: true, - existing_user_id: HOST_USER_ID, - bound_workos_user_id: TARGET_USER_ID, - }); - - // Both bound to one identity; target is non-primary. - const result = await pool.query( - `SELECT iwu.workos_user_id, iwu.is_primary, iwu.identity_id - FROM identity_workos_users iwu - WHERE iwu.workos_user_id IN ($1, $2) - ORDER BY iwu.is_primary DESC`, - [HOST_USER_ID, TARGET_USER_ID] - ); - expect(result.rows).toHaveLength(2); - expect(result.rows[0].workos_user_id).toBe(HOST_USER_ID); - expect(result.rows[0].is_primary).toBe(true); - expect(result.rows[1].workos_user_id).toBe(TARGET_USER_ID); - expect(result.rows[1].is_primary).toBe(false); - expect(result.rows[0].identity_id).toBe(result.rows[1].identity_id); - }); - - it('fetches the WorkOS user and upserts when not in local users yet', async () => { - mockGetUser.mockResolvedValueOnce({ - id: TARGET_USER_ID, - email: 'fetched@test.example', - firstName: 'Fetched', - lastName: 'User', - emailVerified: true, - createdAt: new Date().toISOString(), - updatedAt: new Date().toISOString(), - }); - - await request(app) - .post(`/api/admin/users/${HOST_USER_ID}/credentials`) - .send({ workos_user_id: TARGET_USER_ID }) - .expect(201); - - expect(mockGetUser).toHaveBeenCalledWith(TARGET_USER_ID); - - const local = await pool.query(`SELECT email FROM users WHERE workos_user_id = $1`, [TARGET_USER_ID]); - expect(local.rows[0].email).toBe('fetched@test.example'); - }); - - it('400s on missing or malformed workos_user_id', async () => { - await request(app) - .post(`/api/admin/users/${HOST_USER_ID}/credentials`) - .send({}) - .expect(400); - - await request(app) - .post(`/api/admin/users/${HOST_USER_ID}/credentials`) - .send({ workos_user_id: 'not-a-workos-id' }) - .expect(400); - }); - - it('400s on self-bind', async () => { - await request(app) - .post(`/api/admin/users/${HOST_USER_ID}/credentials`) - .send({ workos_user_id: HOST_USER_ID }) - .expect(400); - }); - - it('idempotent: returns linked: true with no change when already bound', async () => { - await pool.query( - `INSERT INTO users (workos_user_id, email, first_name, last_name, email_verified, - workos_created_at, workos_updated_at, created_at, updated_at) - VALUES ($1, 'target@test.example', 'Target', 'User', true, NOW(), NOW(), NOW(), NOW())`, - [TARGET_USER_ID] - ); - await request(app) - .post(`/api/admin/users/${HOST_USER_ID}/credentials`) - .send({ workos_user_id: TARGET_USER_ID }) - .expect(201); - - // Second call is a no-op. - const response = await request(app) - .post(`/api/admin/users/${HOST_USER_ID}/credentials`) - .send({ workos_user_id: TARGET_USER_ID }) - .expect(200); - expect(response.body.linked).toBe(true); - }); - - it('404s when host user does not exist', async () => { - await request(app) - .post(`/api/admin/users/user_does_not_exist/credentials`) - .send({ workos_user_id: TARGET_USER_ID }) - .expect(404); - }); - - it('404s when WorkOS getUser returns 404 for the credential id', async () => { - const err: any = new Error('Not found'); - err.status = 404; - mockGetUser.mockRejectedValueOnce(err); - - const response = await request(app) - .post(`/api/admin/users/${HOST_USER_ID}/credentials`) - .send({ workos_user_id: TARGET_USER_ID }) - .expect(404); - expect(response.body.error).toMatch(/not found/i); - }); - - it('refuses to consolidate a cred with existing app-state without explicit opt-in', async () => { - // Insert cred locally + give it an organization_membership (real AAO data) + describe('POST /credentials containment', () => { + it.each([false, true])('refuses existing-account consolidation even with consolidate=%s', async (consolidate) => { await pool.query( `INSERT INTO users (workos_user_id, email, first_name, last_name, email_verified, workos_created_at, workos_updated_at, created_at, updated_at) VALUES ($1, 'target@test.example', 'Target', 'User', true, NOW(), NOW(), NOW(), NOW())`, - [TARGET_USER_ID] + [TARGET_USER_ID], ); - const orgId = 'org_test_link_consolidate'; - await pool.query( - `INSERT INTO organizations (workos_organization_id, name, created_at, updated_at) - VALUES ($1, 'Test Org', NOW(), NOW()) - ON CONFLICT (workos_organization_id) DO NOTHING`, - [orgId] + const before = await pool.query( + `SELECT * FROM identity_workos_users WHERE workos_user_id IN ($1, $2) ORDER BY workos_user_id`, + [HOST_USER_ID, TARGET_USER_ID], ); - await pool.query( - `INSERT INTO organization_memberships (workos_user_id, workos_organization_id, email, role, created_at, updated_at) - VALUES ($1, $2, 'target@test.example', 'member', NOW(), NOW())`, - [TARGET_USER_ID, orgId] - ); - - try { - const refused = await request(app) - .post(`/api/admin/users/${HOST_USER_ID}/credentials`) - .send({ workos_user_id: TARGET_USER_ID }) - .expect(409); - expect(refused.body.consolidate_confirmation_required).toBe(true); - expect(refused.body.message).toMatch(/consolidate/i); - - // A static global-admin key has no person-level principal, so it may - // observe the preflight but cannot acknowledge a destructive merge. - await request(app) - .post(`/api/admin/users/${HOST_USER_ID}/credentials`) - .set('X-Test-Admin-Access-Mechanism', 'static_admin_api_key') - .send({ workos_user_id: TARGET_USER_ID, consolidate: true }) - .expect(403) - .expect(({ body }) => { - expect(body.error).toBe('identity_bearing_admin_required'); - }); - - // Same call with consolidate: true succeeds and moves the membership + for (const accessMechanism of ['sso', 'static_admin_api_key']) { await request(app) .post(`/api/admin/users/${HOST_USER_ID}/credentials`) - .send({ workos_user_id: TARGET_USER_ID, consolidate: true }) - .expect(201); - - const moved = await pool.query( - `SELECT workos_user_id FROM organization_memberships WHERE workos_organization_id = $1`, - [orgId] - ); - expect(moved.rows).toHaveLength(1); - expect(moved.rows[0].workos_user_id).toBe(HOST_USER_ID); - - // The destructive, server-confirmed consolidation is attributed to - // the authenticated global admin by mergeUsers' audit record. - const audit = await pool.query<{ workos_user_id: string; details: any }>( - `SELECT workos_user_id, details - FROM registry_audit_log - WHERE action = 'merge_user' AND resource_id = $1 - ORDER BY created_at DESC LIMIT 1`, - [TARGET_USER_ID], - ); - expect(audit.rows).toHaveLength(1); - expect(audit.rows[0].workos_user_id).toBe('user_test_admin_link'); - expect(audit.rows[0].details).toMatchObject({ - primary_user_id: HOST_USER_ID, - secondary_user_id: TARGET_USER_ID, - audit_context: { - acting_workos_user_id: 'user_test_admin_link_credential', - consolidation_confirmed: true, - }, - }); - } finally { - await pool.query(`DELETE FROM organization_memberships WHERE workos_organization_id = $1`, [orgId]); - await pool.query(`DELETE FROM organizations WHERE workos_organization_id = $1`, [orgId]); + .set('X-Test-Admin-Access-Mechanism', accessMechanism) + .send({ workos_user_id: TARGET_USER_ID, consolidate }) + .expect(409) + .expect(({ body }) => expect(body.error).toBe('identity_mutation_disabled')); } + expect((await pool.query( + `SELECT * FROM identity_workos_users WHERE workos_user_id IN ($1, $2) ORDER BY workos_user_id`, + [HOST_USER_ID, TARGET_USER_ID], + )).rows).toEqual(before.rows); + expect(mockGetUser).not.toHaveBeenCalled(); }); - it('blocks a corporate-tier credential from being absorbed by an individual-tier host', async () => { - await pool.query( - `INSERT INTO users (workos_user_id, email, first_name, last_name, email_verified, - workos_created_at, workos_updated_at, created_at, updated_at) - VALUES ($1, 'corporate@test.example', 'Corporate', 'User', true, NOW(), NOW(), NOW(), NOW())`, - [TARGET_USER_ID], - ); - const personalOrgId = 'org_test_link_personal_tier'; - const corporateOrgId = 'org_test_link_corporate_tier'; - await pool.query( - `INSERT INTO organizations - (workos_organization_id, name, is_personal, membership_tier, subscription_status, created_at, updated_at) - VALUES - ($1, 'Personal test workspace', true, 'individual_professional', 'active', NOW(), NOW()), - ($2, 'Corporate test workspace', false, NULL, 'active', NOW(), NOW())`, - [personalOrgId, corporateOrgId], - ); - await pool.query( - `UPDATE organizations - SET subscription_price_lookup_key = 'aao_membership_member' - WHERE workos_organization_id = $1`, - [corporateOrgId], - ); - await pool.query( - `INSERT INTO organization_memberships - (workos_user_id, workos_organization_id, email, role, created_at, updated_at) - VALUES - ($1, $2, 'host@test.example', 'member', NOW(), NOW()), - ($3, $4, 'corporate@test.example', 'member', NOW(), NOW())`, - [HOST_USER_ID, personalOrgId, TARGET_USER_ID, corporateOrgId], - ); - - try { - for (const body of [ - { workos_user_id: TARGET_USER_ID }, - { workos_user_id: TARGET_USER_ID, consolidate: true }, - ]) { - const response = await request(app) - .post(`/api/admin/users/${HOST_USER_ID}/credentials`) - .send(body) - .expect(409); - expect(response.body).toMatchObject({ - error: 'corporate_membership_personal_host_conflict', - corporate_memberships: [{ - workos_organization_id: corporateOrgId, - membership_tier: 'company_icl', - }], - personal_memberships: [{ - workos_organization_id: personalOrgId, - membership_tier: 'individual_professional', - }], - }); - } - - const [membership, bindings] = await Promise.all([ - pool.query( - `SELECT workos_user_id FROM organization_memberships - WHERE workos_organization_id = $1`, - [corporateOrgId], - ), - pool.query<{ identity_id: string }>( - `SELECT identity_id FROM identity_workos_users - WHERE workos_user_id IN ($1, $2) - ORDER BY workos_user_id`, - [HOST_USER_ID, TARGET_USER_ID], - ), - ]); - expect(membership.rows).toEqual([{ workos_user_id: TARGET_USER_ID }]); - expect(bindings.rows).toHaveLength(2); - expect(bindings.rows[0].identity_id).not.toBe(bindings.rows[1].identity_id); - } finally { - await pool.query( - `DELETE FROM organization_memberships - WHERE workos_organization_id IN ($1, $2)`, - [personalOrgId, corporateOrgId], - ); - await pool.query( - `DELETE FROM organizations - WHERE workos_organization_id IN ($1, $2)`, - [personalOrgId, corporateOrgId], - ); - } + it('refuses before fetching or upserting a missing WorkOS credential', async () => { + await request(app) + .post(`/api/admin/users/${HOST_USER_ID}/credentials`) + .send({ workos_user_id: TARGET_USER_ID, consolidate: true }) + .expect(409) + .expect(({ body }) => expect(body.error).toBe('identity_mutation_disabled')); + expect(mockGetUser).not.toHaveBeenCalled(); + expect((await pool.query(`SELECT 1 FROM users WHERE workos_user_id = $1`, [TARGET_USER_ID])).rows).toEqual([]); }); }); + // Fixtures model existing bindings; the now-disabled merge route must never + // be used to prepare read-only listing or binding-only revocation tests. + async function bindExistingFixture() { + await pool.query( + `UPDATE identity_workos_users SET identity_id = ( + SELECT identity_id FROM identity_workos_users WHERE workos_user_id = $1 + ), is_primary = FALSE WHERE workos_user_id = $2`, + [HOST_USER_ID, TARGET_USER_ID], + ); + } + describe('GET /credentials', () => { it('lists bound credentials for the host\'s identity', async () => { await pool.query( @@ -407,10 +181,7 @@ describe('admin link / unlink credential', () => { VALUES ($1, 'target@test.example', 'Target', 'User', true, NOW(), NOW(), NOW(), NOW())`, [TARGET_USER_ID] ); - await request(app) - .post(`/api/admin/users/${HOST_USER_ID}/credentials`) - .send({ workos_user_id: TARGET_USER_ID }) - .expect(201); + await bindExistingFixture(); const response = await request(app) .get(`/api/admin/users/${HOST_USER_ID}/credentials`) @@ -434,30 +205,107 @@ describe('admin link / unlink credential', () => { }); describe('DELETE /credentials/:credentialId', () => { + let precondition: { expected_identity_id: string; expected_authorization_epoch: string }; + beforeEach(async () => { + await pool.query(`INSERT INTO users (workos_user_id, email) VALUES + ('user_test_admin_link', 'admin-link@example.test'), + ('user_test_admin_link_credential', 'admin-credential@example.test') + ON CONFLICT (workos_user_id) DO NOTHING`); + const admin = await pool.query(`SELECT identity_id FROM identity_workos_users WHERE workos_user_id = 'user_test_admin_link'`); + adminContext.identityId = admin.rows[0].identity_id; + await pool.query(`UPDATE identity_workos_users SET identity_id = $1, is_primary = FALSE + WHERE workos_user_id = 'user_test_admin_link_credential'`, [adminContext.identityId]); await pool.query( `INSERT INTO users (workos_user_id, email, first_name, last_name, email_verified, workos_created_at, workos_updated_at, created_at, updated_at) VALUES ($1, 'target@test.example', 'Target', 'User', true, NOW(), NOW(), NOW(), NOW())`, [TARGET_USER_ID] ); - await request(app) - .post(`/api/admin/users/${HOST_USER_ID}/credentials`) - .send({ workos_user_id: TARGET_USER_ID }) - .expect(201); + await bindExistingFixture(); + const listed = await request(app).get(`/api/admin/users/${HOST_USER_ID}/credentials`).expect(200); + precondition = { + expected_identity_id: listed.body.identity_id, + expected_authorization_epoch: listed.body.credentials.find((c: any) => c.workos_user_id === TARGET_USER_ID).authorization_epoch, + }; + }); + + it('requires an observed binding and an identity-bearing actor', async () => { + await request(app).delete(`/api/admin/users/${HOST_USER_ID}/credentials/${TARGET_USER_ID}`) + .expect(400); + await request(app).delete(`/api/admin/users/${HOST_USER_ID}/credentials/${TARGET_USER_ID}`) + .send(precondition).set('X-Test-Admin-Access-Mechanism', 'static_admin_api_key').expect(403); + const before = await pool.query(`SELECT identity_id FROM identity_workos_users WHERE workos_user_id = $1`, [TARGET_USER_ID]); + expect(before.rows[0].identity_id).toBe(precondition.expected_identity_id); }); - it('unbinds a non-primary credential and gives it a fresh singleton identity', async () => { + it.each([true, false])('keeps arbitrary exception details out of the response and logs (conflict=%s)', async (conflict) => { + const sensitive = 'internal-fixture-only: bearer secret-token person@example.test provider-response-body'; + const error = conflict ? new CredentialDetachConflict(sensitive) : new Error(sensitive); + Object.assign(error, { stack: sensitive, cause: new Error(sensitive), + access_token: sensitive, response: { body: sensitive } }); + vi.mocked(detachCredential).mockRejectedValueOnce(error); + const before = (await pool.query( + 'SELECT * FROM identity_workos_users WHERE workos_user_id = $1', [TARGET_USER_ID], + )).rows; + const response = await request(app) + .delete(`/api/admin/users/${HOST_USER_ID}/credentials/${TARGET_USER_ID}`) + .send(precondition).expect(conflict ? 409 : 500); + expect(response.body).toEqual(conflict + ? { error: 'credential_binding_changed', message: 'Credential binding changed. Reload credentials and try again.' } + : { error: 'Failed to unbind credential' }); + const log = conflict ? routeLogger.warn : routeLogger.error; + expect(log).toHaveBeenCalledExactlyOnceWith({ + failureCode: conflict ? 'credential_binding_changed' : 'credential_detach_failed', reason: '[REDACTED]', + }, conflict ? 'Admin unbind-credential: conflict' : 'Admin unbind-credential: failed'); + const emitted = response.text + JSON.stringify(Object.values(routeLogger).flatMap((mock) => mock.mock.calls)); + for (const value of [sensitive, 'secret-token', 'person@example.test', 'provider-response-body', HOST_USER_ID, TARGET_USER_ID]) { + expect(emitted).not.toContain(value); + } + expect((await pool.query('SELECT * FROM identity_workos_users WHERE workos_user_id = $1', [TARGET_USER_ID])).rows).toEqual(before); + expect((await pool.query("SELECT 1 FROM registry_audit_log WHERE action = 'unbind_credential' AND resource_id = $1", [TARGET_USER_ID])).rows).toEqual([]); + }); + + it("unbinds a non-primary credential without moving either credential's organization authority", async () => { + await pool.query( + `INSERT INTO organizations (workos_organization_id, name) VALUES ($1, 'Unlink authority fixture')`, + [AUTHORITY_ORG_ID], + ); + await pool.query( + `INSERT INTO organization_memberships (workos_user_id, workos_organization_id, email, role) + VALUES ($1, $3, 'host@test.example', 'admin'), ($2, $3, 'target@test.example', 'member')`, + [HOST_USER_ID, TARGET_USER_ID, AUTHORITY_ORG_ID], + ); + const authorityBefore = await pool.query( + `SELECT * FROM organization_memberships WHERE workos_organization_id = $1 ORDER BY workos_user_id`, + [AUTHORITY_ORG_ID], + ); const beforeIdentity = await pool.query<{ identity_id: string }>( `SELECT identity_id FROM identity_workos_users WHERE workos_user_id = $1`, [HOST_USER_ID] ); const hostIdentity = beforeIdentity.rows[0].identity_id; - const response = await request(app) - .delete(`/api/admin/users/${HOST_USER_ID}/credentials/${TARGET_USER_ID}`) - .expect(200); + // API keys are provider-owned: detach must not issue any HTTP write + // (or even read) to WorkOS's /organizations/:id/api_keys inventory. + const providerFetch = vi.spyOn(globalThis, 'fetch').mockRejectedValue(new Error('Unexpected provider call')); + mockGetUser.mockClear(); + let response; + try { + response = await request(app) + .delete(`/api/admin/users/${HOST_USER_ID}/credentials/${TARGET_USER_ID}`) + .send(precondition) + .expect(200); + expect(providerFetch).not.toHaveBeenCalled(); + expect(mockGetUser).not.toHaveBeenCalled(); + } finally { + providerFetch.mockRestore(); + } expect(response.body.removed).toBe(true); + expect((await pool.query( + `SELECT * FROM organization_memberships WHERE workos_organization_id = $1 ORDER BY workos_user_id`, + [AUTHORITY_ORG_ID], + )).rows).toEqual(authorityBefore.rows); const after = await pool.query<{ identity_id: string; is_primary: boolean }>( `SELECT identity_id, is_primary FROM identity_workos_users WHERE workos_user_id = $1`, @@ -475,19 +323,25 @@ describe('admin link / unlink credential', () => { [TARGET_USER_ID] ); expect(audit.rows).toHaveLength(1); - expect(audit.rows[0].details.host_user_id).toBe(HOST_USER_ID); + expect(audit.rows[0].details).toMatchObject({ + host_user_id: HOST_USER_ID, + affected_workos_user_id: TARGET_USER_ID, + acting_workos_user_id: 'user_test_admin_link_credential', + acting_identity_id: adminContext.identityId, + detached_from_identity_id: precondition.expected_identity_id, + }); }); it('refuses to remove the primary credential', async () => { const response = await request(app) .delete(`/api/admin/users/${HOST_USER_ID}/credentials/${HOST_USER_ID}`) + .send(precondition) .expect(400); // self-id check fires before the primary check expect(response.body.error).toMatch(/canonical user/i); }); it('refuses to remove a credential that primary-bound on another identity\'s scope', async () => { - // Set TARGET's binding to is_primary = TRUE inside HOST's identity (corrupt - // state, but verifies the guard) — easier: try to unbind via wrong host id. + // A stale URL host cannot detach a binding from another identity. const otherUserId = 'user_test_link_other'; await pool.query( `INSERT INTO users (workos_user_id, email, first_name, last_name, email_verified, @@ -498,24 +352,31 @@ describe('admin link / unlink credential', () => { try { const response = await request(app) .delete(`/api/admin/users/${otherUserId}/credentials/${TARGET_USER_ID}`) - .expect(404); - expect(response.body.error).toMatch(/not bound/i); + .send(precondition) + .expect(409); + expect(response.body.error).toBe('credential_binding_changed'); } finally { await pool.query(`DELETE FROM users WHERE workos_user_id = $1`, [otherUserId]); } }); - it('404s when the credential is not bound to this host', async () => { + it('rejects replay when the credential is no longer bound to this host', async () => { // Unbind first await request(app) .delete(`/api/admin/users/${HOST_USER_ID}/credentials/${TARGET_USER_ID}`) + .send(precondition) .expect(200); - // Second call should 404 — credential is no longer bound here + // Second call should conflict — credential is no longer bound here const response = await request(app) .delete(`/api/admin/users/${HOST_USER_ID}/credentials/${TARGET_USER_ID}`) - .expect(404); - expect(response.body.error).toMatch(/not bound/i); + .send(precondition) + .expect(409); + expect(response.body.error).toBe('credential_binding_changed'); + expect(response.body.message).toBe('Credential binding changed. Reload credentials and try again.'); + expect(routeLogger.warn).toHaveBeenLastCalledWith({ + failureCode: 'credential_binding_changed', reason: 'Credential binding changed; reload credentials', + }, 'Admin unbind-credential: conflict'); }); }); }); diff --git a/server/tests/integration/admin-promote-credential.test.ts b/server/tests/integration/admin-promote-credential.test.ts index 302da4ed7b0..e2b3b0dcd44 100644 --- a/server/tests/integration/admin-promote-credential.test.ts +++ b/server/tests/integration/admin-promote-credential.test.ts @@ -1,12 +1,4 @@ -/** - * Admin "promote credential to primary" integration test. - * - * Exercises POST /api/admin/users/:userId/credentials/:credentialId/promote. - * The new primary should: - * - hold all app-state previously on the old primary (org_memberships, etc.) - * - have is_primary = TRUE; the old primary, FALSE - * - record an audit row - */ +/** Identity promotion containment preserves both privilege directions. */ import { describe, it, expect, beforeAll, afterAll, beforeEach, vi } from 'vitest'; import request from 'supertest'; @@ -18,8 +10,10 @@ vi.hoisted(() => { process.env.WORKOS_COOKIE_PASSWORD ??= 'test-cookie-password-at-least-32-chars-long'; }); +const provider = vi.hoisted(() => ({ getUser: vi.fn(), createUser: vi.fn(), updateUser: vi.fn(), deleteUser: vi.fn() })); + vi.mock('../../src/auth/workos-client.js', () => { - const mockUserManagement = { getUser: vi.fn(), createUser: vi.fn(), deleteUser: vi.fn() }; + const mockUserManagement = provider; const mockWorkos = { userManagement: mockUserManagement }; return { workos: mockWorkos, getWorkos: () => mockWorkos }; }); @@ -34,10 +28,6 @@ vi.mock('../../src/middleware/auth.js', async (importOriginal) => { emailVerified: true, createdAt: new Date().toISOString(), updatedAt: new Date().toISOString(), - // Tests can set X-Test-Admin-Identity header to simulate an admin - // who is bound to the same identity as the target — exercises the - // self-promote guard. - identityId: req.headers['x-test-admin-identity'] || undefined, }; next(); }; @@ -59,9 +49,12 @@ vi.mock('../../src/middleware/csrf.js', () => ({ csrfProtection: (_req: any, _res: any, next: any) => next(), })); -import { initializeDatabase, closeDatabase, getPool } from '../../src/db/client.js'; +import { initializeDatabase, closeDatabase } from '../../src/db/client.js'; import { runMigrations } from '../../src/db/migrate.js'; -import { HTTPServer } from '../../src/http.js'; +import express from 'express'; +import { handleEmailLinkVerification } from '../../src/routes/account-linking.js'; +import { createAdminUsersRouter } from '../../src/routes/admin/users.js'; +import { stopAuthTimers } from '../../src/middleware/auth.js'; const HOST_USER_ID = 'user_test_promote_host'; const TARGET_USER_ID = 'user_test_promote_target'; @@ -69,7 +62,6 @@ const HOST_ORG_ID = 'org_test_promote_host'; const TARGET_ORG_ID = 'org_test_promote_target'; describe('admin promote credential to primary', () => { - let server: HTTPServer; let app: any; let pool: Pool; @@ -78,19 +70,23 @@ describe('admin promote credential to primary', () => { connectionString: process.env.DATABASE_URL || 'postgresql://adcp:localdev@localhost:5432/adcp_test', }); await runMigrations(); - server = new HTTPServer(); - await server.start(0); - app = server.app; + // Isolate the verification route's durable limiter across repeated runs. + await pool.query(`DELETE FROM rate_limit_hits WHERE key LIKE 'verify-email-exec:%'`); + app = express(); + app.use(express.json()); + app.use('/api/admin/users', createAdminUsersRouter()); + handleEmailLinkVerification(app); }, 60000); afterAll(async () => { await cleanup(); - await server?.stop(); + stopAuthTimers(); await closeDatabase(); }); beforeEach(async () => { await cleanup(); + vi.clearAllMocks(); // Insert two users; trigger creates a singleton identity for each await pool.query( `INSERT INTO users (workos_user_id, email, first_name, last_name, email_verified, @@ -123,299 +119,117 @@ describe('admin promote credential to primary', () => { ); } - /** - * Replicates the Ahmed shape: a host with one org, a bound target with - * its own org, target gets promoted to primary; both orgs end up on - * target's workos_user_id. - */ - async function setupBoundPair() { - // Host has Host Org + async function snapshotAuthority() { + const snapshot = await Promise.all([ + pool.query(`SELECT * FROM users WHERE workos_user_id IN ($1, $2) ORDER BY workos_user_id`, [HOST_USER_ID, TARGET_USER_ID]), + pool.query(`SELECT * FROM identity_workos_users WHERE workos_user_id IN ($1, $2) ORDER BY workos_user_id`, [HOST_USER_ID, TARGET_USER_ID]), + pool.query(`SELECT * FROM organization_memberships WHERE workos_user_id IN ($1, $2) ORDER BY workos_user_id, workos_organization_id`, [HOST_USER_ID, TARGET_USER_ID]), + pool.query(`SELECT * FROM organizations WHERE workos_organization_id IN ($1, $2) ORDER BY workos_organization_id`, [HOST_ORG_ID, TARGET_ORG_ID]), + ]); + return snapshot.map(result => result.rows); + } + + // Seed an existing linked identity directly. Containment must not require a + // destructive merge to succeed as part of test preparation. + async function setupBoundPair(hostRole: string, targetRole: string) { await pool.query( `INSERT INTO organization_memberships (workos_user_id, workos_organization_id, email, role, created_at, updated_at) - VALUES ($1, $2, 'host@test.example', 'admin', NOW(), NOW())`, - [HOST_USER_ID, HOST_ORG_ID] + VALUES ($1, $3, 'host@test.example', $4, NOW(), NOW()), + ($2, $3, 'target@test.example', $5, NOW(), NOW())`, + [HOST_USER_ID, TARGET_USER_ID, HOST_ORG_ID, hostRole, targetRole], ); - // Target has Target Org await pool.query( - `INSERT INTO organization_memberships (workos_user_id, workos_organization_id, email, role, created_at, updated_at) - VALUES ($1, $2, 'target@test.example', 'admin', NOW(), NOW())`, - [TARGET_USER_ID, TARGET_ORG_ID] + `UPDATE users SET primary_organization_id = $1 WHERE workos_user_id = $2`, + [HOST_ORG_ID, HOST_USER_ID], ); - // Bind target as non-primary under host's identity (mergeUsers does this); - // mergeUsers also moves target's data to host, so we set it up by hand - // post-bind to keep target_org membership on the target's workos_user_id. - await request(app) - .post(`/api/admin/users/${HOST_USER_ID}/credentials`) - .send({ workos_user_id: TARGET_USER_ID, consolidate: true }) - .expect(201); - // After bind, target_org_membership was moved to HOST_USER_ID. Move it - // back to TARGET_USER_ID to simulate the post-resync Ahmed state where - // an org membership lands on the non-primary credential (because WorkOS - // org_membership webhook routed it to that user_id). await pool.query( - `UPDATE organization_memberships SET workos_user_id = $1 - WHERE workos_organization_id = $2`, - [TARGET_USER_ID, TARGET_ORG_ID] - ); - } - - it('promotes the target credential and moves the old primary\'s app-state forward', async () => { - await setupBoundPair(); - - const response = await request(app) - .post(`/api/admin/users/${HOST_USER_ID}/credentials/${TARGET_USER_ID}/promote`) - .expect(200); - - expect(response.body).toMatchObject({ - promoted: true, - previous_primary_id: HOST_USER_ID, - new_primary_id: TARGET_USER_ID, - }); - - // is_primary swapped - const bindings = await pool.query<{ workos_user_id: string; is_primary: boolean }>( - `SELECT workos_user_id, is_primary FROM identity_workos_users - WHERE workos_user_id IN ($1, $2) - ORDER BY is_primary DESC`, - [HOST_USER_ID, TARGET_USER_ID] - ); - expect(bindings.rows).toHaveLength(2); - expect(bindings.rows.find(r => r.workos_user_id === TARGET_USER_ID)?.is_primary).toBe(true); - expect(bindings.rows.find(r => r.workos_user_id === HOST_USER_ID)?.is_primary).toBe(false); - - // Both orgs now on TARGET_USER_ID (host's app-state moved forward; - // target's stayed since it was already there) - const memberships = await pool.query<{ workos_user_id: string }>( - `SELECT workos_user_id FROM organization_memberships - WHERE workos_organization_id IN ($1, $2) - ORDER BY workos_organization_id`, - [HOST_ORG_ID, TARGET_ORG_ID] + `UPDATE users SET primary_organization_id = $1 WHERE workos_user_id = $2`, + [TARGET_ORG_ID, TARGET_USER_ID], ); - expect(memberships.rows).toHaveLength(2); - expect(memberships.rows.every(r => r.workos_user_id === TARGET_USER_ID)).toBe(true); - }); - - /** - * Both credentials hold a membership in the same organization. The - * consolidation deletes the outgoing primary's row, so the endpoint must - * refuse without stated intent. - */ - async function setupOverlappingPair() { - await setupBoundPair(); - // Give the target its own membership in the host's org, so promoting - // would delete the host's row for that org rather than move it. await pool.query( - `INSERT INTO organization_memberships (workos_user_id, workos_organization_id, email, role, created_at, updated_at) - VALUES ($1, $2, 'target@test.example', 'member', NOW(), NOW())`, - [TARGET_USER_ID, HOST_ORG_ID] + `UPDATE identity_workos_users SET identity_id = ( + SELECT identity_id FROM identity_workos_users WHERE workos_user_id = $1 + ), is_primary = FALSE WHERE workos_user_id = $2`, + [HOST_USER_ID, TARGET_USER_ID], ); } - it('409s rather than silently deleting a membership the target already holds', async () => { - await setupOverlappingPair(); - - const response = await request(app) - .post(`/api/admin/users/${HOST_USER_ID}/credentials/${TARGET_USER_ID}/promote`) - .expect(409); - - expect(response.body).toMatchObject({ - consolidate_confirmation_required: true, - superseded_organization_ids: [HOST_ORG_ID], - superseded_working_group_ids: [], - }); - - // Nothing moved: the host is still primary and still holds its row. - const bindings = await pool.query<{ is_primary: boolean }>( - `SELECT is_primary FROM identity_workos_users WHERE workos_user_id = $1`, - [HOST_USER_ID] - ); - expect(bindings.rows[0].is_primary).toBe(true); - }); - - it('promotes once the caller confirms the consolidation', async () => { - await setupOverlappingPair(); - - await request(app) - .post(`/api/admin/users/${HOST_USER_ID}/credentials/${TARGET_USER_ID}/promote`) - .send({ consolidate: true }) - .expect(200); - - const bindings = await pool.query<{ workos_user_id: string; is_primary: boolean }>( - `SELECT workos_user_id, is_primary FROM identity_workos_users - WHERE workos_user_id IN ($1, $2)`, - [HOST_USER_ID, TARGET_USER_ID] - ); - expect(bindings.rows.find(r => r.workos_user_id === TARGET_USER_ID)?.is_primary).toBe(true); - }); - - it('requires an identity-bearing admin to confirm destructive promotion', async () => { - await setupOverlappingPair(); - - const response = await request(app) - .post(`/api/admin/users/${HOST_USER_ID}/credentials/${TARGET_USER_ID}/promote`) - .set('X-Test-Admin-Access-Mechanism', 'static_admin_api_key') - .send({ consolidate: true }) - .expect(403); - expect(response.body.error).toBe('identity_bearing_admin_required'); - }); - - it('promotes without confirmation when the memberships do not overlap', async () => { - await setupBoundPair(); - - await request(app) - .post(`/api/admin/users/${HOST_USER_ID}/credentials/${TARGET_USER_ID}/promote`) - .expect(200); + it.each([ + ['admin', 'member'], + ['member', 'admin'], + ])('preserves %s primary and %s target authority through concurrent promotion, consolidation and replay', async (hostRole, targetRole) => { + await setupBoundPair(hostRole, targetRole); + const before = await snapshotAuthority(); + const calls = [ + `/api/admin/users/${HOST_USER_ID}/credentials/${TARGET_USER_ID}/promote`, + `/api/admin/users/${TARGET_USER_ID}/credentials/${HOST_USER_ID}/promote`, + `/api/admin/users/${HOST_USER_ID}/credentials`, + `/api/admin/users/${TARGET_USER_ID}/credentials`, + ]; + const invoke = (path: string, consolidate: boolean) => request(app).post(path) + .send({ workos_user_id: path.includes(HOST_USER_ID) ? TARGET_USER_ID : HOST_USER_ID, consolidate }) + .expect(409) + .expect(({ body }) => expect(body.error).toBe('identity_mutation_disabled')); + await Promise.all(calls.flatMap(path => [invoke(path, false), invoke(path, true)])); + for (const path of calls) await invoke(path, true); + expect(await snapshotAuthority()).toEqual(before); + for (const mock of Object.values(provider)) expect(mock).not.toHaveBeenCalled(); }); - it('blocks promotion away from a corporate primary after a personal credential was bound', async () => { + it.each([ + [HOST_USER_ID, TARGET_USER_ID], + [TARGET_USER_ID, HOST_USER_ID], + ])('refuses an in-flight member merge token from %s to %s and its replay', async (primaryId, targetId) => { + await setupBoundPair('admin', 'member'); + const token = `containment-${primaryId}-${targetId}`; await pool.query( - `UPDATE organizations - SET is_personal = false, membership_tier = 'company_standard', subscription_status = 'active' - WHERE workos_organization_id = $1`, - [HOST_ORG_ID], - ); - await pool.query( - `UPDATE organizations - SET is_personal = true, membership_tier = 'individual_professional', subscription_status = 'active' - WHERE workos_organization_id = $1`, - [TARGET_ORG_ID], - ); - await pool.query( - `INSERT INTO organization_memberships (workos_user_id, workos_organization_id, email, role, created_at, updated_at) - VALUES ($1, $2, 'host@test.example', 'member', NOW(), NOW()), - ($3, $4, 'target@test.example', 'member', NOW(), NOW())`, - [HOST_USER_ID, HOST_ORG_ID, TARGET_USER_ID, TARGET_ORG_ID], - ); - - // This is the supported direction: the corporate credential remains - // primary when the personal credential is bound. - await request(app) - .post(`/api/admin/users/${HOST_USER_ID}/credentials`) - .send({ workos_user_id: TARGET_USER_ID, consolidate: true }) - .expect(201); - - const response = await request(app) - .post(`/api/admin/users/${HOST_USER_ID}/credentials/${TARGET_USER_ID}/promote`) - .expect(409); - expect(response.body).toMatchObject({ - error: 'corporate_membership_primary_required', - corporate_memberships: [{ - workos_organization_id: HOST_ORG_ID, - membership_tier: 'company_standard', - }], - }); - - const [corporateMembership, bindings] = await Promise.all([ - pool.query( - `SELECT workos_user_id FROM organization_memberships - WHERE workos_organization_id = $1`, - [HOST_ORG_ID], - ), - pool.query<{ workos_user_id: string; is_primary: boolean }>( - `SELECT workos_user_id, is_primary FROM identity_workos_users - WHERE workos_user_id IN ($1, $2)`, - [HOST_USER_ID, TARGET_USER_ID], - ), - ]); - expect(corporateMembership.rows).toEqual([{ workos_user_id: HOST_USER_ID }]); - expect(bindings.rows.find((row) => row.workos_user_id === HOST_USER_ID)?.is_primary).toBe(true); - expect(bindings.rows.find((row) => row.workos_user_id === TARGET_USER_ID)?.is_primary).toBe(false); + `INSERT INTO email_link_tokens (token, primary_workos_user_id, target_email, target_workos_user_id, expires_at) + VALUES ($1, $2, 'target@test.example', $3, NOW() + INTERVAL '1 hour')`, + [token, primaryId, targetId], + ); + const before = await snapshotAuthority(); + for (let attempt = 0; attempt < 2; attempt++) { + const response = await request(app).post('/verify-email-link') + .type('form').send({ token, consolidate: true }).expect(200); + expect(response.text).toContain('Verification Failed'); + expect(await snapshotAuthority()).toEqual(before); + } + expect((await pool.query(`SELECT status FROM email_link_tokens WHERE token = $1`, [token])).rows) + .toEqual([{ status: 'revoked' }]); + expect((await pool.query(`SELECT 1 FROM user_email_aliases WHERE workos_user_id IN ($1, $2)`, [HOST_USER_ID, TARGET_USER_ID])).rows) + .toEqual([]); + for (const mock of Object.values(provider)) expect(mock).not.toHaveBeenCalled(); }); - it('writes a promote_credential_to_primary audit row', async () => { - await setupBoundPair(); - await request(app) - .post(`/api/admin/users/${HOST_USER_ID}/credentials/${TARGET_USER_ID}/promote`) - .expect(200); - - const audit = await pool.query<{ details: any }>( - `SELECT details FROM registry_audit_log - WHERE action = 'promote_credential_to_primary' AND resource_id = $1 - ORDER BY created_at DESC LIMIT 1`, - [TARGET_USER_ID] - ); - expect(audit.rows).toHaveLength(1); - expect(audit.rows[0].details).toMatchObject({ - previous_primary_id: HOST_USER_ID, - new_primary_id: TARGET_USER_ID, - }); + it('preserves bare alias verification without moving authority or calling WorkOS', async () => { + await setupBoundPair('admin', 'member'); + const token = 'containment-bare-alias'; + await pool.query( + `INSERT INTO email_link_tokens (token, primary_workos_user_id, target_email, expires_at) + VALUES ($1, $2, 'alias@test.example', NOW() + INTERVAL '1 hour')`, + [token, HOST_USER_ID], + ); + const before = await snapshotAuthority(); + const response = await request(app).post('/verify-email-link') + .type('form').send({ token }).expect(200); + expect(response.text).toContain('Email Linked'); + expect(await snapshotAuthority()).toEqual(before); + expect((await pool.query(`SELECT email FROM user_email_aliases WHERE workos_user_id = $1`, [HOST_USER_ID])).rows) + .toEqual([{ email: 'alias@test.example' }]); + for (const mock of Object.values(provider)) expect(mock).not.toHaveBeenCalled(); }); - it('is idempotent: promoting an already-primary credential returns 200 with no change', async () => { - // Promote target so it's the primary - await setupBoundPair(); + it('refuses promotion without a current primary even for a static admin API key', async () => { + await setupBoundPair('admin', 'member'); + await pool.query(`UPDATE identity_workos_users SET is_primary = FALSE WHERE workos_user_id = $1`, [HOST_USER_ID]); + const before = await snapshotAuthority(); await request(app) .post(`/api/admin/users/${HOST_USER_ID}/credentials/${TARGET_USER_ID}/promote`) - .expect(200); - - // Calling again on the same credential should be a no-op - const response = await request(app) - .post(`/api/admin/users/${HOST_USER_ID}/credentials/${TARGET_USER_ID}/promote`) - .expect(200); - expect(response.body.promoted).toBe(true); - expect(response.body.message).toMatch(/already primary/i); - - // Bindings unchanged - const bindings = await pool.query<{ workos_user_id: string; is_primary: boolean }>( - `SELECT workos_user_id, is_primary FROM identity_workos_users - WHERE workos_user_id IN ($1, $2)`, - [HOST_USER_ID, TARGET_USER_ID] - ); - expect(bindings.rows.find(r => r.workos_user_id === TARGET_USER_ID)?.is_primary).toBe(true); - }); - - it('400s when the credentialId in the URL matches the host id', async () => { - const response = await request(app) - .post(`/api/admin/users/${HOST_USER_ID}/credentials/${HOST_USER_ID}/promote`) - .expect(400); - expect(response.body.error).toMatch(/must differ/i); - }); - - it('404s when the credential is not bound to the host\'s identity', async () => { - // Target is its own singleton identity, not bound to host - const response = await request(app) - .post(`/api/admin/users/${HOST_USER_ID}/credentials/${TARGET_USER_ID}/promote`) - .expect(404); - expect(response.body.error).toMatch(/not bound/i); - }); - - it('refuses when the admin is signed in as a member of the target identity', async () => { - await setupBoundPair(); - - // Find the host's identity id - const identityRow = await pool.query<{ identity_id: string }>( - `SELECT identity_id FROM identity_workos_users WHERE workos_user_id = $1`, - [HOST_USER_ID] - ); - const hostIdentityId = identityRow.rows[0].identity_id; - - // Simulate an admin whose identityId == host's identityId via the test - // header the auth mock reads. - const response = await request(app) - .post(`/api/admin/users/${HOST_USER_ID}/credentials/${TARGET_USER_ID}/promote`) - .set('X-Test-Admin-Identity', hostIdentityId) - .expect(409); - expect(response.body.error).toMatch(/own credential/i); - }); - - it('repairs an orphan-no-primary state by setting the target as primary directly', async () => { - await setupBoundPair(); - // Manually break the primary so the identity has no current primary - await pool.query( - `UPDATE identity_workos_users SET is_primary = FALSE WHERE workos_user_id = $1`, - [HOST_USER_ID] - ); - - const response = await request(app) - .post(`/api/admin/users/${HOST_USER_ID}/credentials/${TARGET_USER_ID}/promote`) - .expect(200); - expect(response.body.message).toMatch(/no current primary|invariant repaired/i); - - const bindings = await pool.query<{ workos_user_id: string; is_primary: boolean }>( - `SELECT workos_user_id, is_primary FROM identity_workos_users - WHERE workos_user_id IN ($1, $2)`, - [HOST_USER_ID, TARGET_USER_ID] - ); - expect(bindings.rows.find(r => r.workos_user_id === TARGET_USER_ID)?.is_primary).toBe(true); + .set('X-Test-Admin-Access-Mechanism', 'static_admin_api_key') + .send({ consolidate: true }) + .expect(409) + .expect(({ body }) => expect(body.error).toBe('identity_mutation_disabled')); + expect(await snapshotAuthority()).toEqual(before); + for (const mock of Object.values(provider)) expect(mock).not.toHaveBeenCalled(); }); }); diff --git a/server/tests/integration/authorization-deletion-cross-replica.test.ts b/server/tests/integration/authorization-deletion-cross-replica.test.ts new file mode 100644 index 00000000000..14ea78e8ffe --- /dev/null +++ b/server/tests/integration/authorization-deletion-cross-replica.test.ts @@ -0,0 +1,163 @@ +/** + * Exact cross-replica deletion-cache regression (#6827). + * + * Two independently loaded auth modules own distinct in-memory session caches + * and distinct PostgreSQL pools. Replica A deletes an epoch-0 credential and + * receives the local eviction. Replica B must reject its still-cached session + * on the immediate next hit by observing the durable deletion fingerprint. + */ + +import type { NextFunction, Request, Response } from 'express'; +import type { Pool } from 'pg'; +import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest'; + +const USER_ID = 'user_authz_cross_replica_deleted'; + +const mocks = vi.hoisted(() => ({ + authenticate: vi.fn(), + loadSealedSession: vi.fn(), +})); + +vi.hoisted(() => { + delete process.env.DEV_USER_EMAIL; + delete process.env.DEV_USER_ID; + process.env.WORKOS_API_KEY = process.env.WORKOS_API_KEY ?? 'sk_test'; + process.env.WORKOS_CLIENT_ID = process.env.WORKOS_CLIENT_ID ?? 'client_test'; + process.env.WORKOS_COOKIE_PASSWORD = + process.env.WORKOS_COOKIE_PASSWORD ?? 'placeholder-cookie-password-32-bytes-min'; +}); + +vi.mock('@workos-inc/node', () => ({ + WorkOS: vi.fn(function WorkOS() { + return { + userManagement: { loadSealedSession: mocks.loadSealedSession }, + apiKeys: { createValidation: vi.fn() }, + }; + }), +})); + +vi.mock('../../src/db/bans-db.js', () => ({ + bansDb: { + checkPlatformBan: vi.fn().mockResolvedValue({ banned: false, ban: null }), + checkPlatformBanForApiKey: vi.fn().mockResolvedValue({ banned: false, ban: null }), + }, +})); + +function requestFor(cookie: string): Request { + return { + headers: {}, + cookies: { 'wos-session': cookie }, + path: '/api/me', + originalUrl: '/api/me', + accepts: () => false, + } as unknown as Request; +} + +function response(): Response { + return { + cookie: vi.fn(), + status: vi.fn().mockReturnThis(), + json: vi.fn().mockReturnThis(), + redirect: vi.fn().mockReturnThis(), + } as unknown as Response; +} + +describe('durable deletion fingerprint across independent replicas', () => { + let poolA: Pool; + let closeA: () => Promise<void>; + let closeB: () => Promise<void>; + let deleteOnA: (userId: string, source: 'workos_webhook') => Promise<unknown>; + let fingerprintOnA: (userIds: string[]) => Promise<string>; + let requireAuthA: (req: Request, res: Response, next: NextFunction) => Promise<unknown>; + let requireAuthB: (req: Request, res: Response, next: NextFunction) => Promise<unknown>; + let invalidateA: (userIds: string[]) => void; + + beforeAll(async () => { + const connectionString = process.env.DATABASE_URL + || 'postgresql://adcp:localdev@localhost:5432/adcp_test'; + + vi.resetModules(); + const dbA = await import('../../src/db/client.js'); + poolA = dbA.initializeDatabase({ connectionString }); + closeA = dbA.closeDatabase; + const migrationA = await import('../../src/db/migrate.js'); + await migrationA.runMigrations(); + const identityA = await import('../../src/db/identity-db.js'); + const epochA = await import('../../src/db/authorization-epoch-db.js'); + const authA = await import('../../src/middleware/auth.js'); + deleteOnA = identityA.deleteIdentityCredentialTransaction; + fingerprintOnA = epochA.getAuthorizationFingerprint; + requireAuthA = authA.requireAuth; + invalidateA = authA.invalidateSessionsForUsers; + + vi.resetModules(); + const dbB = await import('../../src/db/client.js'); + dbB.initializeDatabase({ connectionString }); + closeB = dbB.closeDatabase; + const authB = await import('../../src/middleware/auth.js'); + requireAuthB = authB.requireAuth; + }, 60000); + + afterAll(async () => { + await poolA.query(`DELETE FROM registry_audit_log WHERE workos_user_id = $1`, [USER_ID]); + await poolA.query(`DELETE FROM users WHERE workos_user_id = $1`, [USER_ID]); + await closeB(); + await closeA(); + }); + + beforeEach(async () => { + vi.clearAllMocks(); + await poolA.query(`DELETE FROM registry_audit_log WHERE workos_user_id = $1`, [USER_ID]); + await poolA.query(`DELETE FROM users WHERE workos_user_id = $1`, [USER_ID]); + await poolA.query( + `INSERT INTO users ( + workos_user_id, email, first_name, last_name, email_verified, + workos_created_at, workos_updated_at, created_at, updated_at + ) VALUES ($1, 'cross-replica@authz.test', 'Cross', 'Replica', true, + NOW(), NOW(), NOW(), NOW())`, + [USER_ID], + ); + mocks.authenticate.mockResolvedValue({ + authenticated: true, + user: { + id: USER_ID, + email: 'cross-replica@authz.test', + firstName: 'Cross', + lastName: 'Replica', + emailVerified: true, + createdAt: '2026-01-01T00:00:00.000Z', + updatedAt: '2026-01-01T00:00:00.000Z', + }, + accessToken: 'cross-replica-access-token', + }); + mocks.loadSealedSession.mockReturnValue({ + authenticate: mocks.authenticate, + refresh: vi.fn(), + }); + }); + + it('denies replica B immediately after replica A deletes an epoch-0 credential', async () => { + const nextA = vi.fn() as NextFunction; + const nextB = vi.fn() as NextFunction; + const cookieA = `sealed-replica-a-${Date.now()}`; + const cookieB = `sealed-replica-b-${Date.now()}`; + + expect(await fingerprintOnA([USER_ID])).toBe(''); + await requireAuthA(requestFor(cookieA), response(), nextA); + await requireAuthB(requestFor(cookieB), response(), nextB); + expect(mocks.authenticate).toHaveBeenCalledTimes(2); + + await deleteOnA(USER_ID, 'workos_webhook'); + invalidateA([USER_ID]); + expect(await fingerprintOnA([USER_ID])).toMatch( + /^user_authz_cross_replica_deleted:deleted:[0-9a-f-]{36}$/, + ); + + const replicaBResponse = response(); + await requireAuthB(requestFor(cookieB), replicaBResponse, nextB); + + expect(nextB).toHaveBeenCalledTimes(1); + expect(mocks.authenticate).toHaveBeenCalledTimes(3); + expect(replicaBResponse.status).toHaveBeenCalledWith(401); + }); +}); diff --git a/server/tests/integration/authorization-epoch.test.ts b/server/tests/integration/authorization-epoch.test.ts index c6d8fc2f5a8..ff41f84a25d 100644 --- a/server/tests/integration/authorization-epoch.test.ts +++ b/server/tests/integration/authorization-epoch.test.ts @@ -7,8 +7,7 @@ * - bumping is monotonic per credential * - a CASCADE delete moves the fingerprint (so callers must compare for * inequality, not ordering) - * - mergeUsers bumps both credentials inside its own transaction - * - a failed mergeUsers leaves the fingerprint untouched + * - refused generic merges and promotions leave fingerprints untouched */ import { describe, it, expect, beforeAll, afterAll, beforeEach } from 'vitest'; @@ -21,6 +20,7 @@ import { } from '../../src/db/authorization-epoch-db.js'; import { mergeUsers } from '../../src/db/user-merge-db.js'; import { promoteSecondaryIfPrimaryDeleted } from '../../src/db/identity-db.js'; +import { deleteIdentityCredential } from '../../src/services/identity-credential-deletion.js'; const TEST_USER_PREFIX = 'user_authz_epoch_test_'; @@ -37,11 +37,13 @@ describe('Authorization epoch (migration 565)', () => { afterAll(async () => { await pool.query(`DELETE FROM users WHERE workos_user_id LIKE $1`, [`${TEST_USER_PREFIX}%`]); + await pool.query(`DELETE FROM registry_audit_log WHERE workos_user_id LIKE $1`, [`${TEST_USER_PREFIX}%`]); await closeDatabase(); }); beforeEach(async () => { await pool.query(`DELETE FROM users WHERE workos_user_id LIKE $1`, [`${TEST_USER_PREFIX}%`]); + await pool.query(`DELETE FROM registry_audit_log WHERE workos_user_id LIKE $1`, [`${TEST_USER_PREFIX}%`]); }); async function insertUser(suffix: string): Promise<string> { @@ -68,10 +70,10 @@ describe('Authorization epoch (migration 565)', () => { const userId = await insertUser('monotonic'); await bumpAuthorizationEpochs(pool, [userId]); - expect(await getAuthorizationFingerprint([userId])).toBe(`${userId}:1`); + expect(await getAuthorizationFingerprint([userId])).toBe(`${userId}:epoch:1`); await bumpAuthorizationEpochs(pool, [userId]); - expect(await getAuthorizationFingerprint([userId])).toBe(`${userId}:2`); + expect(await getAuthorizationFingerprint([userId])).toBe(`${userId}:epoch:2`); }); it('ignores credentials with no users row instead of failing the transaction', async () => { @@ -79,7 +81,7 @@ describe('Authorization epoch (migration 565)', () => { await bumpAuthorizationEpochs(pool, [userId, `${TEST_USER_PREFIX}absent`]); - expect(await getAuthorizationFingerprint([userId])).toBe(`${userId}:1`); + expect(await getAuthorizationFingerprint([userId])).toBe(`${userId}:epoch:1`); }); it('changes the fingerprint when a bumped credential is deleted', async () => { @@ -94,54 +96,64 @@ describe('Authorization epoch (migration 565)', () => { expect(after).toBe(''); }); - it('bumps both credentials when mergeUsers binds them to one identity', async () => { - const primaryId = await insertUser('merge_primary'); - const secondaryId = await insertUser('merge_secondary'); + it('retains a deletion fingerprint for an epoch-0 credential after users-row cascade', async () => { + const userId = await insertUser('epoch_zero_confirmed_delete'); + expect(await getAuthorizationFingerprint([userId])).toBe(''); - await mergeUsers(primaryId, secondaryId, primaryId); + await expect(deleteIdentityCredential(userId, 'workos_webhook')) + .resolves.toMatchObject({ deleted: true }); - expect(await getAuthorizationFingerprint([primaryId])).toBe(`${primaryId}:1`); - expect(await getAuthorizationFingerprint([secondaryId])).toBe(`${secondaryId}:1`); + expect(await getAuthorizationFingerprint([userId])) + .toMatch(new RegExp(`^${userId}:deleted:[0-9a-f-]+$`)); }); - it('bumps both credentials when the primary is deleted and a secondary is promoted', async () => { - const primaryId = await insertUser('promote_primary'); - const secondaryId = await insertUser('promote_secondary'); - await mergeUsers(primaryId, secondaryId, primaryId); + it('preserves both fingerprints when generic merging is refused', async () => { + const primaryId = await insertUser('merge_primary'); + const secondaryId = await insertUser('merge_secondary'); + await bumpAuthorizationEpochs(pool, [primaryId, secondaryId]); + const before = await getAuthorizationFingerprint([primaryId, secondaryId]); - const promoted = await promoteSecondaryIfPrimaryDeleted(primaryId); + await expect(mergeUsers(primaryId, secondaryId, primaryId)).rejects.toMatchObject({ + code: 'identity_mutation_disabled', + }); - expect(promoted).toEqual({ promotedUserId: secondaryId }); - expect(await getAuthorizationFingerprint([primaryId])).toBe(`${primaryId}:2`); - expect(await getAuthorizationFingerprint([secondaryId])).toBe(`${secondaryId}:2`); + expect(await getAuthorizationFingerprint([primaryId, secondaryId])).toBe(before); }); - it('bumps every bound credential on promotion, not just the successor', async () => { - // Three credentials on one identity. Deleting the primary promotes the - // longest-bound secondary; the third credential's canonical routing moves - // to that successor too, so its sessions must be revalidated as well. + it('preserves every bound fingerprint when automatic promotion is refused', async () => { const primaryId = await insertUser('fanout_primary'); const successorId = await insertUser('fanout_successor'); const bystanderId = await insertUser('fanout_bystander'); - await mergeUsers(primaryId, successorId, primaryId); - await mergeUsers(primaryId, bystanderId, primaryId); - - const bystanderBefore = await getAuthorizationFingerprint([bystanderId]); + const credentials = [primaryId, successorId, bystanderId]; + // Model a historical multi-credential identity without invoking the + // disabled generic merge as test setup. + const original = await pool.query<{ identity_id: string }>( + `SELECT identity_id FROM identity_workos_users WHERE workos_user_id = ANY($1)`, + [[successorId, bystanderId]], + ); + await pool.query( + `UPDATE identity_workos_users + SET identity_id = (SELECT identity_id FROM identity_workos_users WHERE workos_user_id = $1), + is_primary = FALSE + WHERE workos_user_id = ANY($2)`, + [primaryId, [successorId, bystanderId]], + ); + await pool.query(`DELETE FROM identities WHERE id = ANY($1)`, [original.rows.map((row) => row.identity_id)]); + await bumpAuthorizationEpochs(pool, credentials); + const before = await getAuthorizationFingerprint(credentials); - const promoted = await promoteSecondaryIfPrimaryDeleted(primaryId); + await expect(promoteSecondaryIfPrimaryDeleted(primaryId)).rejects.toMatchObject({ + code: 'identity_mutation_disabled', + }); - expect(promoted).toEqual({ promotedUserId: successorId }); - expect(await getAuthorizationFingerprint([bystanderId])).not.toBe(bystanderBefore); + expect(await getAuthorizationFingerprint(credentials)).toBe(before); }); - it('leaves the fingerprint untouched when mergeUsers rolls back', async () => { - const secondaryId = await insertUser('rollback_secondary'); - const missingPrimaryId = `${TEST_USER_PREFIX}rollback_missing_primary`; - - await expect(mergeUsers(missingPrimaryId, secondaryId, secondaryId)).rejects.toThrow( - 'Primary user does not exist' - ); - + it('refuses before checking for a missing primary or changing the fingerprint', async () => { + const secondaryId = await insertUser('missing_secondary'); + await expect(mergeUsers(`${TEST_USER_PREFIX}missing_primary`, secondaryId, secondaryId)).rejects.toMatchObject({ + code: 'identity_mutation_disabled', + }); expect(await getAuthorizationFingerprint([secondaryId])).toBe(''); }); }); diff --git a/server/tests/integration/credential-detach-composition.test.ts b/server/tests/integration/credential-detach-composition.test.ts new file mode 100644 index 00000000000..876b5284776 --- /dev/null +++ b/server/tests/integration/credential-detach-composition.test.ts @@ -0,0 +1,420 @@ +/** Real transactions; query barriers establish ownership before each competing mutation. */ +import { randomUUID } from 'node:crypto'; +import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest'; +import type { Pool, PoolClient, QueryResult } from 'pg'; + +const provider = vi.hoisted(() => ({ + createUser: vi.fn(), deleteUser: vi.fn(), getUser: vi.fn(), listUsers: vi.fn(), + createOrganizationMembership: vi.fn(), updateOrganizationMembership: vi.fn(), deleteOrganizationMembership: vi.fn(), +})); +vi.mock('../../src/auth/workos-client.js', () => ({ + getAdminCredentialMutationWorkos: () => ({ userManagement: provider }), +})); +vi.mock('../../src/logger.js', () => ({ createLogger: () => ({ info: vi.fn(), warn: vi.fn(), error: vi.fn(), debug: vi.fn() }) })); +vi.mock('../../src/addie/error-notifier.js', () => ({ notifySystemError: vi.fn() })); +vi.mock('../../src/db/client.js', async (importOriginal) => { + const actual = await importOriginal<typeof import('../../src/db/client.js')>(); + return { ...actual, getPool: vi.fn(actual.getPool) }; +}); + +import { closeDatabase, getPool, initializeDatabase } from '../../src/db/client.js'; +import { runMigrations } from '../../src/db/migrate.js'; +import { CredentialDetachConflict, detachCredential } from '../../src/db/credential-detach-db.js'; +import { deleteIdentityCredentialTransaction } from '../../src/db/identity-db.js'; +import { readCredentialAuthorizationLifecycle } from '../../src/db/authorization-epoch-db.js'; +import { createAndBindAdminCredential } from '../../src/services/admin-credential-bind.js'; + +const PREFIX = `user_detach_composition_${randomUUID().slice(0, 8)}_`; +const [HOST, SIBLING, OTHER_HOST, ACTOR_PRIMARY, ACTOR, CREATED] = + ['sam', 'sam_existing', 'priya', 'jordan', 'jordan_credential', 'sam_new'].map(role => `${PREFIX}${role}`); +const EXISTING = [HOST, SIBLING, OTHER_HOST, ACTOR_PRIMARY, ACTOR]; +const USERS = [...EXISTING, CREATED]; +const ORG = `org_${PREFIX}pinnacle`; +const EMAIL = `${PREFIX}sam.new@example.test`; +type RawQuery = (sql: string, params?: unknown[]) => Promise<QueryResult>; +type QueryHook = (sql: string, params?: unknown[]) => Promise<void>; + +function deferred() { + let resolve!: () => void; + const promise = new Promise<void>(complete => { resolve = complete; }); + return { promise, resolve }; +} +function pauseBefore(predicate: (sql: string, params?: unknown[]) => boolean) { + const reached = deferred(); + const release = deferred(); + let paused = false; + const before: QueryHook = async (sql, params) => { + if (!paused && predicate(sql.replace(/\s+/g, ' '), params)) { + paused = true; + reached.resolve(); + await release.promise; + } + }; + return { before, reached: reached.promise, release: release.resolve }; +} +function settled<T>(promise: Promise<T>) { + return promise.then(value => ({ value, error: undefined }), (error: Error) => ({ value: undefined, error })); +} +async function reach<T>(barrier: ReturnType<typeof pauseBefore>, pending: Promise<T>) { + await Promise.race([barrier.reached, pending.then(() => { throw new Error('Operation finished before its query barrier'); })]); +} + +describe('detach composes with credential binding and provider deletion', () => { + let pool: Pool; + let hostIdentityId: string; + let actorIdentityId: string; + const identities = new Set<string>(); + + beforeAll(async () => { + pool = initializeDatabase({ connectionString: process.env.DATABASE_URL || 'postgresql://adcp:localdev@localhost:5432/adcp_test' }); + await runMigrations(); + }, 60000); + afterAll(async () => { if (pool) await cleanup(); await closeDatabase(); }); + beforeEach(async () => { + await cleanup(); + vi.clearAllMocks(); + for (const mock of Object.values(provider)) mock.mockReset(); + provider.createUser.mockResolvedValue({ id: CREATED, email: EMAIL, firstName: 'Sam', lastName: 'Adeyemi', + emailVerified: true, createdAt: '2026-09-14T00:00:00.000Z', updatedAt: '2026-09-14T00:00:00.000Z' }); + provider.deleteUser.mockResolvedValue(undefined); + await pool.query(`INSERT INTO organizations (workos_organization_id, name, subscription_status, stripe_subscription_id) + VALUES ($1, 'Pinnacle Agency', 'active', $2)`, [ORG, `${PREFIX}subscription`]); + for (const id of EXISTING) { + await pool.query('INSERT INTO users (workos_user_id, email, primary_organization_id) VALUES ($1, $2, $3)', + [id, `${id}@example.test`, ORG]); + identities.add((await binding(id)).identity_id); + } + hostIdentityId = (await binding(HOST)).identity_id; + actorIdentityId = (await binding(ACTOR_PRIMARY)).identity_id; + for (const [id, identity] of [[SIBLING, hostIdentityId], [ACTOR, actorIdentityId]]) { + await pool.query('UPDATE identity_workos_users SET identity_id = $1, is_primary = FALSE WHERE workos_user_id = $2', [identity, id]); + } + await pool.query(`INSERT INTO organization_memberships + (workos_user_id, workos_organization_id, workos_membership_id, email, role, provisioning_source) + VALUES ($1, $3, $4, 'sam@example.test', 'owner', 'invitation'), + ($2, $3, $5, 'sam.credential@example.test', 'admin', 'sso')`, + [HOST, SIBLING, ORG, `${PREFIX}membership1`, `${PREFIX}membership2`]); + await pool.query(`INSERT INTO organization_credential_grants + (workos_organization_id, workos_user_id, role, granted_by_workos_user_id, reason) + VALUES ($1, $2, 'admin', $3, 'Original independent credential grant')`, [ORG, SIBLING, ACTOR]); + const group = (await pool.query<{ id: string }>( + "INSERT INTO working_groups (name, slug) VALUES ('Detach composition fixture', $1) RETURNING id", [PREFIX], + )).rows[0].id; + await pool.query(`INSERT INTO working_group_memberships (working_group_id, workos_user_id, added_by_user_id) + VALUES ($1, $2, $3), ((SELECT id FROM working_groups WHERE slug = 'aao-admin'), $2, $3)`, [group, SIBLING, ACTOR]); + await pool.query('INSERT INTO working_group_leaders (working_group_id, user_id) VALUES ($1, $2)', [group, SIBLING]); + await pool.query(`INSERT INTO working_group_topic_subscriptions (working_group_id, workos_user_id, topic_slugs) + VALUES ($1, $2, '{original-topic}')`, [group, SIBLING]); + await pool.query(`INSERT INTO subscription_line_items + (workos_organization_id, stripe_subscription_id, stripe_subscription_item_id, price_id, quantity, metadata) + VALUES ($1, $2, $3, 'price_original', 3, '{"source":"billing"}')`, [ORG, `${PREFIX}subscription`, `${PREFIX}item`]); + }); + + async function binding(id: string) { + return (await pool.query<{ identity_id: string; is_primary: boolean }>( + 'SELECT identity_id, is_primary FROM identity_workos_users WHERE workos_user_id = $1', [id], + )).rows[0]; + } + async function cleanup() { + for (const row of (await pool.query<{ identity_id: string }>( + 'SELECT identity_id FROM identity_workos_users WHERE workos_user_id = ANY($1)', [USERS], + )).rows) identities.add(row.identity_id); + await pool.query(`DELETE FROM registry_audit_log WHERE workos_user_id = ANY($1) + OR details->>'host_user_id' = ANY($1) OR resource_id = ANY($1)`, [USERS]); + await pool.query('DELETE FROM admin_credential_bind_operations WHERE host_user_id = ANY($1)', [USERS]); + await pool.query('DELETE FROM working_group_memberships WHERE workos_user_id = ANY($1)', [USERS]); + await pool.query('DELETE FROM working_groups WHERE slug = $1', [PREFIX]); + await pool.query('DELETE FROM organization_memberships WHERE workos_user_id = ANY($1)', [USERS]); + await pool.query('DELETE FROM users WHERE workos_user_id = ANY($1)', [USERS]); + await pool.query('DELETE FROM organizations WHERE workos_organization_id = $1', [ORG]); + if (identities.size) await pool.query('DELETE FROM identities WHERE id = ANY($1::uuid[])', [[...identities]]); + identities.clear(); + } + function bindInput(hostUserId = HOST) { return { hostUserId, email: EMAIL, actorUserId: ACTOR, actorIdentityId }; } + async function detachInput(credentialId = SIBLING) { + const epoch = (await pool.query<{ epoch: string }>('SELECT epoch::text FROM authorization_epochs WHERE workos_user_id = $1', [credentialId])).rows[0]?.epoch ?? '0'; + return { hostUserId: HOST, credentialId, expectedIdentityId: hostIdentityId, expectedAuthorizationEpoch: epoch, + actorUserId: ACTOR_PRIMARY, actorCredentialId: ACTOR, actorIdentityId }; + } + async function authoritySnapshot(ids = EXISTING) { + const queries = [ + ['users', 'workos_user_id = ANY($1)', ids], + ['organization_memberships', 'workos_user_id = ANY($1)', ids], + ['organization_credential_grants', 'workos_user_id = ANY($1)', ids], + ['working_group_memberships', 'workos_user_id = ANY($1)', ids], + ['working_group_leaders', 'user_id = ANY($1)', ids], + ['working_group_topic_subscriptions', 'workos_user_id = ANY($1)', ids], + ['organizations', 'workos_organization_id = ANY($1)', [ORG]], + ['subscription_line_items', 'workos_organization_id = ANY($1)', [ORG]], + ] as const; + return Object.fromEntries(await Promise.all(queries.map(async ([table, where, values]) => [table, + (await pool.query(`SELECT to_jsonb(t) AS row FROM ${table} t WHERE ${where} ORDER BY to_jsonb(t)::text`, [values])).rows, + ]))); + } + async function detachSnapshot() { + return { + bindings: (await pool.query('SELECT * FROM identity_workos_users WHERE workos_user_id = ANY($1) ORDER BY workos_user_id', [USERS])).rows, + epochs: (await pool.query('SELECT * FROM authorization_epochs WHERE workos_user_id = ANY($1) ORDER BY workos_user_id', [USERS])).rows, + audits: (await pool.query("SELECT * FROM registry_audit_log WHERE action = 'unbind_credential' AND resource_id = ANY($1) ORDER BY id", [USERS])).rows, + }; + } + async function operations() { + return (await pool.query('SELECT * FROM admin_credential_bind_operations WHERE host_user_id = ANY($1) ORDER BY id', [USERS])).rows; + } + async function fullState() { + return { detach: await detachSnapshot(), authority: await authoritySnapshot(USERS), operations: await operations(), + audits: (await pool.query('SELECT * FROM registry_audit_log WHERE workos_user_id = ANY($1) ORDER BY id', [USERS])).rows }; + } + async function expectEpochs(expected: Record<string, string>) { + const rows = (await pool.query<{ workos_user_id: string; epoch: string }>( + 'SELECT workos_user_id, epoch::text FROM authorization_epochs WHERE workos_user_id = ANY($1)', [USERS], + )).rows; + expect(Object.fromEntries(rows.map(row => [row.workos_user_id, row.epoch]))).toEqual(expected); + } + function expectProviderCalls(creates: number) { + expect(provider.createUser).toHaveBeenCalledTimes(creates); + for (const [name, mock] of Object.entries(provider)) if (name !== 'createUser') expect(mock).not.toHaveBeenCalled(); + } + async function expectOneDetach(credentialId: string) { + const audit = (await detachSnapshot()).audits; + expect(audit).toHaveLength(1); + expect(audit[0]).toMatchObject({ workos_user_id: ACTOR_PRIMARY, resource_id: credentialId, + details: { acting_workos_user_id: ACTOR, acting_identity_id: actorIdentityId, + affected_workos_user_id: credentialId, detached_from_identity_id: hostIdentityId } }); + expect((await binding(credentialId)).is_primary).toBe(true); + expect((await binding(credentialId)).identity_id).not.toBe(hostIdentityId); + } + function observedPool(before: QueryHook) { + let pid: number | undefined; + const proxy = new Proxy(pool, { get(target, property) { + if (property === 'connect') return async () => { + const client = await target.connect(); + const query: RawQuery = client.query.bind(client); + const release = client.release.bind(client); + pid = (await query('SELECT pg_backend_pid() AS pid')).rows[0].pid; + client.query = (async (sql: string, params?: unknown[]) => { + await before(sql, params); + return query(sql, params); + }) as PoolClient['query']; + client.release = (error?: boolean | Error) => { + client.query = query as PoolClient['query']; client.release = release; release(error); + }; + return client; + }; + const value = Reflect.get(target, property); + return typeof value === 'function' ? value.bind(target) : value; + } }); + return { pool: proxy, pid: () => pid }; + } + async function expectAdvisoryWait(waiter: () => number | undefined, holder: () => number | undefined) { + await expect.poll(async () => (await pool.query<{ blocked: boolean }>( + `SELECT EXISTS (SELECT 1 FROM pg_locks waiting JOIN pg_locks holder + ON waiting.locktype = holder.locktype AND waiting.database = holder.database + AND waiting.classid = holder.classid AND waiting.objid = holder.objid AND waiting.objsubid = holder.objsubid + WHERE waiting.locktype = 'advisory' AND NOT waiting.granted AND holder.granted + AND waiting.pid = $1 AND holder.pid = $2) AS blocked`, [waiter(), holder()], + )).rows[0].blocked, { timeout: 2000, interval: 10 }).toBe(true); + } + + it.each(['credential fence', 'binding rows', 'identity rows'] as const)( + 'rejects detach while a fresh bind owns %s without changing detach or authority state', async (phase) => { + const input = await detachInput(); + const authority = await authoritySnapshot(); + const barrier = pauseBefore((sql, params) => phase === 'credential fence' + ? sql.includes('SELECT workos_user_id, identity_id FROM identity_workos_users') + : phase === 'binding rows' ? sql.includes('SELECT id FROM identities') + : sql.includes('INSERT INTO users') && params?.[0] === CREATED); + const observed = observedPool(barrier.before); + const pending = settled(createAndBindAdminCredential(bindInput(), { pool: observed.pool, workos: provider })); + try { + await reach(barrier, pending); + const before = await fullState(); + await expect(detachCredential(input)).rejects.toBeInstanceOf(CredentialDetachConflict); + expect(await fullState()).toEqual(before); + } finally { barrier.release(); await pending; } + expect((await pending).value?.status).toBe(201); + expect(await authoritySnapshot()).toEqual(authority); + expect((await detachSnapshot()).audits).toEqual([]); + await expectEpochs({ [HOST]: '1', [SIBLING]: '1', [CREATED]: '1' }); + expectProviderCalls(1); + }); + + it('serializes a fresh bind after detach and bumps only the remaining host graph', async () => { + const input = await detachInput(); + const authority = await authoritySnapshot(); + const barrier = pauseBefore(sql => sql.includes('INSERT INTO registry_audit_log') && sql.includes('unbind_credential')); + const observedDetach = observedPool(barrier.before); + vi.mocked(getPool).mockReturnValueOnce(observedDetach.pool); + const detaching = settled(detachCredential(input)); + let bindingOperation: ReturnType<typeof settled<Awaited<ReturnType<typeof createAndBindAdminCredential>>>> | undefined; + try { + await reach(barrier, detaching); + const observedBind = observedPool(async () => {}); + bindingOperation = settled(createAndBindAdminCredential(bindInput(), { pool: observedBind.pool, workos: provider })); + await expectAdvisoryWait(observedBind.pid, observedDetach.pid); + } finally { barrier.release(); await detaching; await bindingOperation; } + expect((await detaching).error).toBeUndefined(); + expect((await bindingOperation!).value?.status).toBe(201); + expect(await authoritySnapshot()).toEqual(authority); + await expectOneDetach(SIBLING); + expect((await binding(CREATED)).identity_id).toBe(hostIdentityId); + await expectEpochs({ [HOST]: '2', [SIBLING]: '1', [CREATED]: '1' }); + expectProviderCalls(1); + }); + + it('rejects detach while authoritative deletion owns the credential fence', async () => { + const input = await detachInput(); + const barrier = pauseBefore((sql, params) => sql.includes('SELECT identity_id FROM identity_workos_users') + && sql.includes('FOR UPDATE') && params?.[0] === SIBLING); + const observed = observedPool(barrier.before); + vi.mocked(getPool).mockReturnValueOnce(observed.pool); + const deleting = settled(deleteIdentityCredentialTransaction(SIBLING, 'workos_webhook')); + try { + await reach(barrier, deleting); + const before = await fullState(); + await expect(detachCredential(input)).rejects.toBeInstanceOf(CredentialDetachConflict); + expect(await fullState()).toEqual(before); + } finally { barrier.release(); await deleting; } + expect((await deleting).value?.deleted).toBe(true); + expect((await detachSnapshot()).audits).toEqual([]); + expect(await binding(SIBLING)).toBeUndefined(); + expect((await pool.query("SELECT 1 FROM registry_audit_log WHERE workos_user_id = $1 AND action = 'identity_credential_deleted'", [SIBLING])).rows).toHaveLength(1); + await expectEpochs({ [HOST]: '1' }); + expectProviderCalls(0); + }); + + it('forces deletion to re-read a detached credential before its authoritative retry', async () => { + const input = await detachInput(); + const authority = await authoritySnapshot(); + const barrier = pauseBefore(sql => sql.includes('INSERT INTO registry_audit_log') && sql.includes('unbind_credential')); + const observedDetach = observedPool(barrier.before); + vi.mocked(getPool).mockReturnValueOnce(observedDetach.pool); + const detaching = settled(detachCredential(input)); + let deleting: ReturnType<typeof settled<Awaited<ReturnType<typeof deleteIdentityCredentialTransaction>>>> | undefined; + try { + await reach(barrier, detaching); + const observedDelete = observedPool(async () => {}); + vi.mocked(getPool).mockReturnValueOnce(observedDelete.pool); + deleting = settled(deleteIdentityCredentialTransaction(SIBLING, 'workos_webhook')); + await expectAdvisoryWait(observedDelete.pid, observedDetach.pid); + } finally { barrier.release(); await detaching; await deleting; } + expect((await detaching).error).toBeUndefined(); + expect((await deleting!).error?.message).toBe('Credential identity changed while acquiring deletion locks'); + expect(await authoritySnapshot()).toEqual(authority); + await expectOneDetach(SIBLING); + await expectEpochs({ [HOST]: '1', [SIBLING]: '1' }); + expect((await pool.query("SELECT 1 FROM registry_audit_log WHERE workos_user_id = $1 AND action = 'identity_credential_deleted'", [SIBLING])).rows).toEqual([]); + expect((await deleteIdentityCredentialTransaction(SIBLING, 'workos_webhook')).deleted).toBe(true); + expect((await detachSnapshot()).audits).toHaveLength(1); + expect(await binding(SIBLING)).toBeUndefined(); + expect((await pool.query("SELECT 1 FROM registry_audit_log WHERE workos_user_id = $1 AND action = 'identity_credential_deleted'", [SIBLING])).rows).toHaveLength(1); + await expectEpochs({ [HOST]: '1' }); + expectProviderCalls(0); + }); + + it('rejects a captured secondary-actor request when its canonical primary is deleted before detach locks', async () => { + const input = await detachInput(); + const barrier = pauseBefore(sql => sql.includes('pg_try_advisory_lock(')); + const observed = observedPool(barrier.before); + vi.mocked(getPool).mockReturnValueOnce(observed.pool); + const detaching = settled(detachCredential(input)); + let afterDeletion: Awaited<ReturnType<typeof fullState>> | undefined; + try { + await reach(barrier, detaching); + expect((await deleteIdentityCredentialTransaction(ACTOR_PRIMARY, 'workos_webhook')).deleted).toBe(true); + expect(await readCredentialAuthorizationLifecycle(ACTOR)).toEqual({ status: 'terminal', reason: 'missing_primary' }); + afterDeletion = await fullState(); + } finally { barrier.release(); await detaching; } + expect((await detaching).error).toBeInstanceOf(CredentialDetachConflict); + expect(await fullState()).toEqual(afterDeletion); + expect((await binding(SIBLING)).identity_id).toBe(hostIdentityId); + expect((await detachSnapshot()).audits).toEqual([]); + await expectEpochs({ [ACTOR]: '1' }); + expectProviderCalls(0); + }); + + it.each(['same host', 'conflicting host'] as const)('keeps bind → detach → %s replay from rebinding or changing authority', async (replay) => { + const created = await createAndBindAdminCredential(bindInput(), { workos: provider }); + expect(created.status).toBe(201); + const journalBefore = await operations(); + const authority = await authoritySnapshot(USERS); + await detachCredential(await detachInput(CREATED)); + expect(await operations()).toEqual(journalBefore); + await expectOneDetach(CREATED); + await expectEpochs({ [HOST]: '2', [SIBLING]: '2', [CREATED]: '2' }); + const detached = await detachSnapshot(); + const replayInput = bindInput(replay === 'same host' ? HOST : OTHER_HOST); + for (let attempt = 0; attempt < 2; attempt++) { + const result = await createAndBindAdminCredential(replayInput, { workos: provider }); + expect(result.status).toBe(503); + expect(result.body).toMatchObject({ reconciliation_required: true, operation_id: created.body.operation_id }); + expect(await detachSnapshot()).toEqual(detached); + expect(await authoritySnapshot(USERS)).toEqual(authority); + } + const journal = await operations(); + expect(journal).toHaveLength(1); + expect(journal[0]).toMatchObject({ id: created.body.operation_id, + status: replay === 'same host' ? 'reconciliation_required' : 'committed', provider_user_id: CREATED }); + expectProviderCalls(1); + }); + + it('rejects a concurrent journal reconciliation transition, then rejects its committed status without mutation', async () => { + const created = await createAndBindAdminCredential(bindInput(), { workos: provider }); + expect(created.status).toBe(201); + const input = await detachInput(CREATED); + const before = await fullState(); + const writer = await pool.connect(); + try { + await writer.query('BEGIN'); + // Parent replay can transition this row after releasing its lifecycle transaction. + await writer.query(`UPDATE admin_credential_bind_operations + SET status = 'reconciliation_required', updated_at = NOW() WHERE id = $1`, [created.body.operation_id]); + await expect(detachCredential(input)).rejects.toBeInstanceOf(CredentialDetachConflict); + expect(await fullState()).toEqual(before); + await writer.query('COMMIT'); + const reconciled = await fullState(); + await expect(detachCredential(input)).rejects.toBeInstanceOf(CredentialDetachConflict); + expect(await fullState()).toEqual(reconciled); + } finally { await writer.query('ROLLBACK'); writer.release(); } + expect((await binding(CREATED)).identity_id).toBe(hostIdentityId); + expect((await detachSnapshot()).audits).toEqual([]); + await expectEpochs({ [HOST]: '1', [SIBLING]: '1', [CREATED]: '1' }); + expectProviderCalls(1); + }); + + it('holds the committed bind operation row through detach commit before a later reconciliation write', async () => { + const created = await createAndBindAdminCredential(bindInput(), { workos: provider }); + expect(created.status).toBe(201); + const input = await detachInput(CREATED); + const authority = await authoritySnapshot(USERS); + const barrier = pauseBefore(sql => sql.includes('INSERT INTO registry_audit_log') && sql.includes('unbind_credential')); + const observed = observedPool(barrier.before); + vi.mocked(getPool).mockReturnValueOnce(observed.pool); + const detaching = settled(detachCredential(input)); + const writer = await pool.connect(); + let writing: ReturnType<typeof settled<QueryResult>> | undefined; + try { + await reach(barrier, detaching); + await writer.query('BEGIN'); + const writerPid = (await writer.query<{ pid: number }>('SELECT pg_backend_pid() AS pid')).rows[0].pid; + writing = settled(writer.query(`UPDATE admin_credential_bind_operations + SET status = 'reconciliation_required', updated_at = NOW() WHERE id = $1`, [created.body.operation_id])); + await expect.poll(async () => (await pool.query<{ blocked: boolean }>( + 'SELECT $1::integer = ANY(pg_blocking_pids($2::integer)) AS blocked', [observed.pid(), writerPid], + )).rows[0].blocked, { timeout: 2000, interval: 10 }).toBe(true); + } finally { + barrier.release(); + await detaching; + await writing; + await writer.query('COMMIT'); + writer.release(); + } + expect((await detaching).error).toBeUndefined(); + expect((await writing!).error).toBeUndefined(); + expect((await operations())[0].status).toBe('reconciliation_required'); + await expectOneDetach(CREATED); + await expectEpochs({ [HOST]: '2', [SIBLING]: '2', [CREATED]: '2' }); + expect(await authoritySnapshot(USERS)).toEqual(authority); + expectProviderCalls(1); + }); +}); diff --git a/server/tests/integration/credential-detach-db.test.ts b/server/tests/integration/credential-detach-db.test.ts new file mode 100644 index 00000000000..f52d035cdcf --- /dev/null +++ b/server/tests/integration/credential-detach-db.test.ts @@ -0,0 +1,455 @@ +import { randomUUID } from 'node:crypto'; +import { afterAll, afterEach, beforeAll, describe, expect, it } from 'vitest'; +import type { Pool } from 'pg'; +import { closeDatabase, initializeDatabase } from '../../src/db/client.js'; +import { runMigrations } from '../../src/db/migrate.js'; +import { CredentialDetachConflict, detachCredential, type DetachCredentialInput } from '../../src/db/credential-detach-db.js'; + +const prefix = `user_detach_integrity_${randomUUID().slice(0, 8)}_`; +const users = ['personal', 'corporate', 'sibling', 'actor'].map((suffix) => `${prefix}${suffix}`); +const [personal, corporate, sibling, actor] = users; +const orgs = [`${prefix}personal_org`, `${prefix}corporate_org`]; +let pool: Pool; +let groupId: string; + +// Full rows include identifiers, role/provisioning/grant provenance and timestamps. +// WorkOS API keys have no local table; the detach helper has no provider dependency. +async function authoritySnapshot() { + const queries = [ + ['users', 'workos_user_id = ANY($1)', users], + ['organization_memberships', 'workos_user_id = ANY($1)', users], + ['organization_credential_grants', 'workos_user_id = ANY($1)', users], + ['working_group_memberships', 'workos_user_id = ANY($1)', users], + ['working_group_leaders', 'user_id = ANY($1)', users], + ['organizations', 'workos_organization_id = ANY($1)', orgs], + ['subscription_line_items', 'workos_organization_id = ANY($1)', orgs], + ] as const; + return Object.fromEntries(await Promise.all(queries.map(async ([table, where, ids]) => [table, + (await pool.query(`SELECT to_jsonb(t) AS row FROM ${table} t WHERE ${where} ORDER BY to_jsonb(t)::text`, [ids])).rows, + ]))); +} + +async function bindingSnapshot() { + return { + bindings: (await pool.query('SELECT * FROM identity_workos_users WHERE workos_user_id = ANY($1) ORDER BY workos_user_id', [users])).rows, + epochs: (await pool.query('SELECT * FROM authorization_epochs WHERE workos_user_id = ANY($1) ORDER BY workos_user_id', [users])).rows, + audits: (await pool.query(`SELECT * FROM registry_audit_log WHERE resource_id = ANY($1) + OR workos_user_id = ANY($1) OR details->>'provider_user_id' = ANY($1) ORDER BY id`, [users])).rows, + bindOperations: (await pool.query(`SELECT * FROM admin_credential_bind_operations + WHERE host_user_id = ANY($1) OR provider_user_id = ANY($1) ORDER BY id`, [users])).rows, + identities: (await pool.query('SELECT * FROM identities ORDER BY id')).rows, + }; +} + +async function fixture(direction: 'personal' | 'corporate' = 'personal'): Promise<DetachCredentialInput> { + await pool.query(`INSERT INTO organizations (workos_organization_id, name, is_personal, subscription_status, + stripe_subscription_id, subscription_metadata) + VALUES ($1, 'Personal workspace', TRUE, NULL, NULL, NULL), + ($2, 'Pinnacle Agency', FALSE, 'active', $3, '{"source":"original-subscription"}')`, + [orgs[0], orgs[1], `${prefix}subscription`]); + for (const [index, user] of users.entries()) { + await pool.query(`INSERT INTO users (workos_user_id, email, primary_organization_id) + VALUES ($1, $2, $3)`, [user, `${user}@${index === 0 ? 'personal.example' : 'pinnacle.example'}`, orgs[index === 0 ? 0 : 1]]); + } + const hostUserId = direction === 'personal' ? personal : corporate; + const credentialId = direction === 'personal' ? corporate : personal; + const initialBindings = (await pool.query<{ workos_user_id: string; identity_id: string }>( + 'SELECT workos_user_id, identity_id FROM identity_workos_users WHERE workos_user_id = ANY($1)', [users], + )).rows; + const expectedIdentityId = initialBindings.find((row) => row.workos_user_id === hostUserId)!.identity_id; + const actorIdentityId = initialBindings.find((row) => row.workos_user_id === actor)!.identity_id; + await pool.query(`UPDATE identity_workos_users SET identity_id = $1, is_primary = FALSE + WHERE workos_user_id = ANY($2)`, [expectedIdentityId, [credentialId, sibling]]); + await pool.query('DELETE FROM identities WHERE id = ANY($1::uuid[])', [initialBindings + .filter((row) => [credentialId, sibling].includes(row.workos_user_id)).map((row) => row.identity_id)]); + await pool.query(`INSERT INTO authorization_epochs (workos_user_id, epoch) VALUES ($1, 7), ($2, 11)`, [credentialId, actor]); + + // The unattributed membership is deliberately left as found after a legacy merge. + await pool.query(`INSERT INTO organization_memberships + (workos_user_id, workos_organization_id, workos_membership_id, email, role, provisioning_source) + VALUES ($1, $3, $5, 'personal@personal.example', 'owner', 'invitation'), + ($2, $4, $6, 'corporate@pinnacle.example', 'admin', 'sso'), + ($1, $4, $7, 'personal@personal.example', 'member', NULL)`, + [personal, corporate, orgs[0], orgs[1], `${prefix}membership1`, `${prefix}membership2`, `${prefix}legacy`]); + await pool.query(`INSERT INTO organization_credential_grants + (workos_organization_id, workos_user_id, role, granted_by_workos_user_id, reason) + VALUES ($1, $3, 'admin', $5, 'Original grant'), ($2, $4, 'member', $5, 'Independent grant')`, + [orgs[1], orgs[0], personal, corporate, actor]); + groupId = (await pool.query<{ id: string }>( + `INSERT INTO working_groups (name, slug) VALUES ('Detach integrity fixture', $1) RETURNING id`, [prefix], + )).rows[0].id; + const adminGroup = (await pool.query<{ id: string }>(`SELECT id FROM working_groups WHERE slug = 'aao-admin'`)).rows[0]; + expect(adminGroup).toBeDefined(); + await pool.query(`INSERT INTO working_group_memberships (working_group_id, workos_user_id, added_by_user_id) + VALUES ($1, $3, $5), ($2, $4, $5)`, [groupId, adminGroup.id, personal, corporate, actor]); + await pool.query('INSERT INTO working_group_leaders (working_group_id, user_id) VALUES ($1, $2)', [groupId, corporate]); + await pool.query(`INSERT INTO subscription_line_items + (workos_organization_id, stripe_subscription_id, stripe_subscription_item_id, price_id, quantity, metadata) + VALUES ($1, $2, $3, 'price_original', 3, '{"source":"billing"}')`, + [orgs[1], `${prefix}subscription`, `${prefix}item`]); + return { hostUserId, credentialId, expectedIdentityId, expectedAuthorizationEpoch: '7', + actorUserId: actor, actorCredentialId: actor, actorIdentityId }; +} + +async function fixtureWithSecondaryActor(): Promise<DetachCredentialInput> { + const input = await fixture(); + await pool.query('UPDATE identity_workos_users SET identity_id = $1 WHERE workos_user_id = $2', + [input.actorIdentityId, sibling]); + return { ...input, actorCredentialId: sibling }; +} + +const lifecycleRoles = ['target', 'host', 'actor', 'canonical'] as const; +function graphCredential(input: DetachCredentialInput, role: typeof lifecycleRoles[number]): string { + return { target: input.credentialId, host: input.hostUserId, + actor: input.actorCredentialId, canonical: input.actorUserId }[role]; +} + +async function insertBindOperation(input: DetachCredentialInput, providerUserId: string, status: string) { + return (await pool.query<{ id: string }>(`INSERT INTO admin_credential_bind_operations + (email_hash, host_user_id, host_identity_id, actor_user_id, actor_identity_id, provider_user_id, status) + VALUES ($1, $2, $3, $4, $5, $6, $7) RETURNING id`, + [randomUUID(), input.hostUserId, input.expectedIdentityId, input.actorCredentialId, + input.actorIdentityId, providerUserId, status])).rows[0].id; +} + +async function expectConflictWithoutMutation(input: DetachCredentialInput) { + const before = await bindingSnapshot(); + const authority = await authoritySnapshot(); + await expect(detachCredential(input)).rejects.toBeInstanceOf(CredentialDetachConflict); + expect(await bindingSnapshot()).toEqual(before); + expect(await authoritySnapshot()).toEqual(authority); +} + +async function installFault( + table: 'registry_audit_log' | 'authorization_epochs', timing: 'BEFORE' | 'AFTER', body: string, deferred = false, +) { + await pool.query(`CREATE FUNCTION detach_integrity_fault() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN ${body} END $$`); + const column = table === 'registry_audit_log' ? 'resource_id' : 'workos_user_id'; + await pool.query(`CREATE ${deferred ? 'CONSTRAINT ' : ''}TRIGGER detach_integrity_fault ${timing} INSERT OR UPDATE ON ${table} + ${deferred ? 'DEFERRABLE INITIALLY DEFERRED' : ''} + FOR EACH ROW WHEN (NEW.${column} LIKE '${prefix}%') EXECUTE FUNCTION detach_integrity_fault()`); +} + +describe('binding-only credential detach (real PostgreSQL)', () => { + beforeAll(async () => { + pool = initializeDatabase({ connectionString: process.env.DATABASE_URL || 'postgresql://adcp:localdev@localhost:5432/adcp_test' }); + await runMigrations(); + }, 60000); + + afterEach(async () => { + await pool.query('DROP TRIGGER IF EXISTS detach_integrity_fault ON registry_audit_log'); + await pool.query('DROP TRIGGER IF EXISTS detach_integrity_fault ON authorization_epochs'); + await pool.query('DROP FUNCTION IF EXISTS detach_integrity_fault()'); + const identities = (await pool.query<{ identity_id: string }>( + 'SELECT identity_id FROM identity_workos_users WHERE workos_user_id = ANY($1)', [users], + )).rows.map((row) => row.identity_id); + await pool.query(`DELETE FROM registry_audit_log WHERE resource_id = ANY($1) + OR workos_user_id = ANY($1) OR details->>'provider_user_id' = ANY($1)`, [users]); + await pool.query(`DELETE FROM admin_credential_bind_operations + WHERE host_user_id = ANY($1) OR provider_user_id = ANY($1)`, [users]); + await pool.query('DELETE FROM working_group_memberships WHERE workos_user_id = ANY($1)', [users]); + await pool.query('DELETE FROM working_groups WHERE slug = $1', [prefix]); + await pool.query('DELETE FROM organization_memberships WHERE workos_user_id = ANY($1)', [users]); + await pool.query('DELETE FROM users WHERE workos_user_id = ANY($1)', [users]); + await pool.query('DELETE FROM identities WHERE id = ANY($1::uuid[])', [identities]); + await pool.query('DELETE FROM organizations WHERE workos_organization_id = ANY($1)', [orgs]); + }); + afterAll(() => closeDatabase()); + + it.each(['personal', 'corporate'] as const)('detaches from a %s host without changing any authority provenance', async (direction) => { + const input = await fixture(direction); + const authority = await authoritySnapshot(); + const before = await bindingSnapshot(); + const result = await detachCredential(input); + expect(await authoritySnapshot()).toEqual(authority); + const after = await bindingSnapshot(); + expect(after.identities).toHaveLength(before.identities.length + 1); + expect(after.bindings.filter((row) => row.workos_user_id !== input.credentialId)) + .toEqual(before.bindings.filter((row) => row.workos_user_id !== input.credentialId)); + expect(after.bindings.find((row) => row.workos_user_id === input.credentialId)) + .toMatchObject({ identity_id: result.newIdentityId, is_primary: true }); + expect(result.affectedCredentialIds).toEqual([personal, corporate, sibling].sort()); + expect(Object.fromEntries(after.epochs.map((row) => [row.workos_user_id, row.epoch]))) + .toEqual({ [input.hostUserId]: '1', [input.credentialId]: '8', [sibling]: '1', [actor]: '11' }); + expect(after.audits).toHaveLength(1); + expect(after.audits[0]).toMatchObject({ action: 'unbind_credential', workos_user_id: actor, resource_id: input.credentialId, + details: { operation_id: after.audits[0].id, acting_workos_user_id: actor, acting_identity_id: input.actorIdentityId, + affected_workos_user_id: input.credentialId, detached_from_identity_id: input.expectedIdentityId, + new_identity_id: result.newIdentityId, expected_authorization_epoch: '7' } }); + expect(after.epochs.find((row) => row.workos_user_id === actor)).toEqual(before.epochs.find((row) => row.workos_user_id === actor)); + }); + + it('accepts a sibling URL host and advances all old-identity epochs', async () => { + const input = await fixture(); + const authority = await authoritySnapshot(); + const result = await detachCredential({ ...input, hostUserId: sibling }); + expect(result.affectedCredentialIds).toEqual([personal, corporate, sibling].sort()); + expect(await authoritySnapshot()).toEqual(authority); + expect((await bindingSnapshot()).epochs).toEqual(expect.arrayContaining([ + expect.objectContaining({ workos_user_id: personal, epoch: '1' }), + expect.objectContaining({ workos_user_id: sibling, epoch: '1' }), + expect.objectContaining({ workos_user_id: corporate, epoch: '8' }), + ])); + }); + + it('records the authenticated secondary actor credential separately from its canonical user', async () => { + const input = await fixture(); + await detachCredential({ ...input, actorCredentialId: sibling, actorUserId: personal, actorIdentityId: input.expectedIdentityId }); + expect((await bindingSnapshot()).audits[0]).toMatchObject({ workos_user_id: personal, + details: { acting_workos_user_id: sibling, acting_identity_id: input.expectedIdentityId } }); + }); + + it('preserves a valid separate actor identity and attributes its authenticated secondary credential', async () => { + const input = await fixtureWithSecondaryActor(); + const authority = await authoritySnapshot(); + const before = await bindingSnapshot(); + await detachCredential(input); + const after = await bindingSnapshot(); + expect(await authoritySnapshot()).toEqual(authority); + expect(after.bindings.filter((row) => [actor, sibling].includes(row.workos_user_id))) + .toEqual(before.bindings.filter((row) => [actor, sibling].includes(row.workos_user_id))); + expect(after.epochs.filter((row) => [actor, sibling].includes(row.workos_user_id))) + .toEqual(before.epochs.filter((row) => [actor, sibling].includes(row.workos_user_id))); + expect(after.bindOperations).toEqual(before.bindOperations); + expect(after.audits).toHaveLength(1); + expect(after.audits[0]).toMatchObject({ workos_user_id: actor, resource_id: input.credentialId, + details: { acting_workos_user_id: sibling, acting_identity_id: input.actorIdentityId } }); + }); + + const terminalActions = ['identity_credential_deleted', 'identity_primary_deletion_quarantined', + 'identity_credential_admin_compensation_deleted', 'identity_credential_admin_compensation_quarantined'] as const; + it.each(terminalActions.flatMap((action) => lifecycleRoles.map((role) => ({ action, role }))))( + 'rejects $action on the $role credential without changing authority or lifecycle evidence', async ({ action, role }) => { + const input = await fixtureWithSecondaryActor(); + const userId = graphCredential(input, role); + await pool.query(`INSERT INTO registry_audit_log + (workos_organization_id, workos_user_id, action, resource_type, resource_id, details) + VALUES ('system', $1::text, $2, 'identity_credential', $1::text, '{}'::jsonb)`, [userId, action]); + await expectConflictWithoutMutation(input); + }, + ); + + it.each(lifecycleRoles)('rejects a reconciliation journal provider_user_id matching the %s credential', async (role) => { + const input = await fixtureWithSecondaryActor(); + await insertBindOperation(input, graphCredential(input, role), 'reconciliation_required'); + await expectConflictWithoutMutation(input); + }); + + it.each(['creating', 'provider_created', 'compensating', 'compensated', 'provider_rejected'] as const)( + 'rejects a %s journal for the affected credential without rewriting its status', async (status) => { + const input = await fixture(); + await insertBindOperation(input, input.credentialId, status); + await expectConflictWithoutMutation(input); + }, + ); + + const bindAuditStatuses = ['reconciliation_required', 'compensating', 'compensated'] as const; + it.each(bindAuditStatuses.flatMap((status) => lifecycleRoles.map((role) => ({ status, role }))))( + 'rejects immutable bind $status evidence identifying the $role credential', async ({ status, role }) => { + const input = await fixtureWithSecondaryActor(); + const providerUserId = graphCredential(input, role); + // An older operator outside this graph created the record. Only the immutable + // provider_user_id payload identifies the affected credential, not the actor column. + const historicalOperator = `${prefix}historical_operator`; + expect(users).not.toContain(historicalOperator); + await pool.query(`INSERT INTO registry_audit_log + (workos_organization_id, workos_user_id, action, resource_type, resource_id, details) + VALUES ('system', $1, 'admin_credential_bind', 'admin_credential_bind_operation', $2, $3)`, + [historicalOperator, randomUUID(), { status, provider_user_id: providerUserId }]); + expect((await bindingSnapshot()).bindOperations).toEqual([]); + await expectConflictWithoutMutation(input); + }, + ); + + it('preserves bind evidence about another provider credential when its audit actor belongs to this graph', async () => { + const input = await fixtureWithSecondaryActor(); + const providerUserId = `${prefix}outside_graph`; + expect(users).not.toContain(providerUserId); + await pool.query(`INSERT INTO registry_audit_log + (workos_organization_id, workos_user_id, action, resource_type, resource_id, details) + VALUES ('system', $1, 'admin_credential_bind', 'admin_credential_bind_operation', $2, $3)`, + [input.credentialId, randomUUID(), { status: 'reconciliation_required', provider_user_id: providerUserId }]); + const before = await bindingSnapshot(); + const authority = await authoritySnapshot(); + await detachCredential(input); + const after = await bindingSnapshot(); + const detachAudits = after.audits.filter((row) => row.action === 'unbind_credential'); + expect(detachAudits).toHaveLength(1); + expect(after.audits.filter((row) => row.id !== detachAudits[0].id)).toEqual(before.audits); + expect(after.bindOperations).toEqual(before.bindOperations); + expect(await authoritySnapshot()).toEqual(authority); + }); + + it.each(['zero-primary', 'canonical-user-is-secondary'] as const)( + 'rejects an actor identity with %s without changing its graph', async (fault) => { + const input = await fixtureWithSecondaryActor(); + if (fault === 'zero-primary') { + await pool.query('UPDATE identity_workos_users SET is_primary = FALSE WHERE workos_user_id = $1', [actor]); + } else { + input.actorUserId = sibling; + } + await expectConflictWithoutMutation(input); + }, + ); + + it('rejects replay without another audit or epoch increment', async () => { + const input = await fixture(); + await detachCredential(input); + const before = await bindingSnapshot(); + await expect(detachCredential(input)).rejects.toBeInstanceOf(CredentialDetachConflict); + expect(await bindingSnapshot()).toEqual(before); + }); + + it('rejects an old request even when the credential is rebound to the same identity', async () => { + const input = await fixture(); + const detached = await detachCredential(input); + await pool.query('UPDATE identity_workos_users SET identity_id = $1, is_primary = FALSE WHERE workos_user_id = $2', + [input.expectedIdentityId, input.credentialId]); + await pool.query('DELETE FROM identities WHERE id = $1', [detached.newIdentityId]); + const before = await bindingSnapshot(); + await expect(detachCredential(input)).rejects.toThrow('authorization changed'); + expect(await bindingSnapshot()).toEqual(before); + }); + + it.each(['identity', 'epoch', 'actor', 'primary', 'same-user', 'missing-primary'])( + 'rejects stale or invalid %s state without mutation', async (fault) => { + const input = await fixture(); + if (fault === 'identity') input.expectedIdentityId = input.actorIdentityId; + if (fault === 'epoch') input.expectedAuthorizationEpoch = '6'; + if (fault === 'actor') input.actorCredentialId = sibling; + if (fault === 'primary') { input.credentialId = personal; input.hostUserId = sibling; } + if (fault === 'same-user') input.hostUserId = input.credentialId; + if (fault === 'missing-primary') await pool.query('UPDATE identity_workos_users SET is_primary = FALSE WHERE workos_user_id = $1', [personal]); + const before = await bindingSnapshot(); + const authority = await authoritySnapshot(); + await expect(detachCredential(input)).rejects.toBeInstanceOf(CredentialDetachConflict); + expect(await bindingSnapshot()).toEqual(before); + expect(await authoritySnapshot()).toEqual(authority); + }, + ); + + const auditFaults = [ + ['BEFORE RETURN NULL suppresses INSERT (rowCount 0)', 'BEFORE', 'RETURN NULL;'], + ['thrown audit error', 'BEFORE', "RAISE EXCEPTION 'injected audit failure';"], + ['AFTER DELETE removes the returned row', 'AFTER', 'DELETE FROM registry_audit_log WHERE id = NEW.id; RETURN NEW;'], + ['AFTER duplicate creates two durable records', 'AFTER', `IF pg_trigger_depth() = 1 THEN + INSERT INTO registry_audit_log (workos_organization_id, workos_user_id, action, resource_type, resource_id, details) + VALUES (NEW.workos_organization_id, NEW.workos_user_id, NEW.action, NEW.resource_type, NEW.resource_id, NEW.details); + END IF; RETURN NEW;`], + ['AFTER duplicate alters its payload but keeps operation_id', 'AFTER', `IF pg_trigger_depth() = 1 THEN + INSERT INTO registry_audit_log (workos_organization_id, workos_user_id, action, resource_type, resource_id, details) + VALUES (NEW.workos_organization_id, NEW.workos_user_id, NEW.action, NEW.resource_type, NEW.resource_id, + NEW.details || '{"duplicate":true}'::jsonb); END IF; RETURN NEW;`], + ['BEFORE changes the returned audit id', 'BEFORE', 'NEW.id := gen_random_uuid(); RETURN NEW;'], + ['BEFORE removes actor attribution', 'BEFORE', "NEW.details := NEW.details - 'acting_workos_user_id'; RETURN NEW;"], + ] as const; + it.each(auditFaults)('rolls back binding, epochs and audit when %s', async (_name, timing, body) => { + const input = await fixture(); + const before = await bindingSnapshot(); + const authority = await authoritySnapshot(); + await installFault('registry_audit_log', timing, body); + await expect(detachCredential(input)).rejects.toThrow(); + expect(await bindingSnapshot()).toEqual(before); + expect(await authoritySnapshot()).toEqual(authority); + }); + + it.each([ + ['BEFORE', 'RETURN NULL;'], + ['BEFORE', "RAISE EXCEPTION 'injected epoch failure';"], + ['BEFORE', "IF TG_OP = 'UPDATE' THEN RETURN OLD; END IF; RETURN NEW;"], + ['AFTER', `IF TG_OP = 'UPDATE' AND pg_trigger_depth() = 1 THEN + UPDATE authorization_epochs SET epoch = OLD.epoch WHERE workos_user_id = NEW.workos_user_id; + END IF; RETURN NEW;`], + ] as const)( + 'rolls back on authorization epoch suppression or failure: %s %s', async (timing, body) => { + const input = await fixture(); + const before = await bindingSnapshot(); + const authority = await authoritySnapshot(); + await installFault('authorization_epochs', timing, body); + await expect(detachCredential(input)).rejects.toThrow(); + expect(await bindingSnapshot()).toEqual(before); + expect(await authoritySnapshot()).toEqual(authority); + }, + ); + + it('rolls back even after writing the audit if advisory unlock fails', async () => { + const input = await fixture(); + const before = await bindingSnapshot(); + const authority = await authoritySnapshot(); + await installFault('registry_audit_log', 'AFTER', 'PERFORM pg_advisory_unlock_all(); RETURN NEW;'); + await expect(detachCredential(input)).rejects.toThrow('advisory unlock failed'); + expect(await bindingSnapshot()).toEqual(before); + expect(await authoritySnapshot()).toEqual(authority); + }); + + it.each(['registry_audit_log', 'authorization_epochs'] as const)( + 'flushes deferred %s triggers and rejects changed durable state before commit', async (table) => { + const input = await fixture(); + const before = await bindingSnapshot(); + const authority = await authoritySnapshot(); + const body = table === 'registry_audit_log' + ? 'DELETE FROM registry_audit_log WHERE id = NEW.id; RETURN NEW;' + : `IF NEW.epoch <> 7 THEN + UPDATE authorization_epochs SET epoch = 7 WHERE workos_user_id = NEW.workos_user_id; + END IF; RETURN NEW;`; + await installFault(table, 'AFTER', body, true); + await expect(detachCredential(input)).rejects.toThrow(); + expect(await bindingSnapshot()).toEqual(before); + expect(await authoritySnapshot()).toEqual(authority); + }, + ); + + it('rolls back detach and its audit when a deferred trigger adds a terminal lifecycle marker', async () => { + const input = await fixture(); + await installFault('registry_audit_log', 'AFTER', `IF NEW.action = 'unbind_credential' THEN + INSERT INTO registry_audit_log + (workos_organization_id, workos_user_id, action, resource_type, resource_id, details) + VALUES ('system', NEW.resource_id, 'identity_credential_admin_compensation_quarantined', + 'identity_credential', NEW.resource_id, '{}'::jsonb); + END IF; RETURN NEW;`, true); + await expectConflictWithoutMutation(input); + }); + + it('allows exactly one concurrent detach, with one durable audit and one epoch bump', async () => { + const input = await fixture(); + const authority = await authoritySnapshot(); + const blocker = await pool.connect(); + const barrier = 6827001; + await blocker.query('SELECT pg_advisory_lock($1)', [barrier]); + await installFault('registry_audit_log', 'BEFORE', `PERFORM pg_advisory_xact_lock(${barrier}); RETURN NEW;`); + const first = detachCredential(input); + try { + // Wait for the first transaction to reach the audit INSERT while still + // holding its binding locks, rather than relying on a scheduling sleep. + await expect.poll(async () => (await pool.query( + `SELECT 1 FROM pg_stat_activity WHERE datname = current_database() + AND wait_event = 'advisory' AND query LIKE 'INSERT INTO registry_audit_log%'`, + )).rowCount, { timeout: 5000 }).toBe(1); + await expect(detachCredential(input)).rejects.toBeInstanceOf(CredentialDetachConflict); + } finally { + expect((await blocker.query('SELECT pg_advisory_unlock($1) AS unlocked', [barrier])).rows[0].unlocked).toBe(true); + blocker.release(); + await first; + } + expect(await authoritySnapshot()).toEqual(authority); + const after = await bindingSnapshot(); + expect(after.audits).toHaveLength(1); + expect(after.epochs.find((row) => row.workos_user_id === corporate)?.epoch).toBe('8'); + }); + + it.each(['identity', 'epoch'])('rejects a concurrent %s writer without waiting or changing state', async (kind) => { + const input = await fixture(); + const before = await bindingSnapshot(); + const writer = await pool.connect(); + await writer.query('BEGIN'); + try { + if (kind === 'identity') await writer.query('SELECT id FROM identities WHERE id = $1 FOR UPDATE', [input.expectedIdentityId]); + else await writer.query('SELECT workos_user_id FROM authorization_epochs WHERE workos_user_id = $1 FOR UPDATE', [corporate]); + await expect(detachCredential(input)).rejects.toBeInstanceOf(CredentialDetachConflict); + expect(await bindingSnapshot()).toEqual(before); + } finally { + await writer.query('ROLLBACK'); + writer.release(); + } + }); +}); diff --git a/server/tests/integration/mcp-oauth-deletion-race.test.ts b/server/tests/integration/mcp-oauth-deletion-race.test.ts new file mode 100644 index 00000000000..8f1eba9454e --- /dev/null +++ b/server/tests/integration/mcp-oauth-deletion-race.test.ts @@ -0,0 +1,114 @@ +import type { Response } from 'express'; +import type { Pool } from 'pg'; +import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest'; + +const USER_ID = 'user_mcp_oauth_deletion_race'; +const PENDING_ID = 'pending_mcp_oauth_deletion_race'; + +const mocks = vi.hoisted(() => ({ + authenticateWithCodeForTokens: vi.fn(), +})); + +vi.mock('../../src/auth/workos-client.js', async (importOriginal) => ({ + ...(await importOriginal<typeof import('../../src/auth/workos-client.js')>()), + authenticateWithCodeForTokens: mocks.authenticateWithCodeForTokens, +})); + +const { initializeDatabase, closeDatabase } = await import('../../src/db/client.js'); +const { runMigrations } = await import('../../src/db/migrate.js'); +const mcpOAuthStateDb = await import('../../src/db/mcp-oauth-state-db.js'); +const { deleteIdentityCredential } = await import('../../src/services/identity-credential-deletion.js'); +const { handleMCPOAuthCallback } = await import('../../src/mcp/oauth-provider.js'); + +describe('MCP OAuth callback vs confirmed credential deletion', () => { + let pool: Pool; + + beforeAll(async () => { + pool = initializeDatabase({ + connectionString: process.env.DATABASE_URL + || 'postgresql://adcp:localdev@localhost:5432/adcp_test', + }); + await runMigrations(); + }, 60_000); + + afterAll(async () => { + await cleanup(); + await closeDatabase(); + }); + + beforeEach(async () => { + vi.clearAllMocks(); + await cleanup(); + await pool.query( + `INSERT INTO users ( + workos_user_id, email, first_name, last_name, email_verified, + workos_created_at, workos_updated_at, created_at, updated_at + ) VALUES ($1, 'mcp-race@example.test', 'MCP', 'Race', TRUE, + NOW(), NOW(), NOW(), NOW())`, + [USER_ID], + ); + await mcpOAuthStateDb.setPendingAuth(PENDING_ID, { + clientId: 'client_mcp_race', + redirectUri: 'https://client.example.test/callback', + codeChallenge: 'challenge', + scopes: ['openid'], + state: 'roundtrip-state', + }); + }); + + async function cleanup() { + if (!pool) return; + await pool.query(`DELETE FROM mcp_oauth_pending_auths WHERE id = $1`, [PENDING_ID]); + await pool.query( + `DELETE FROM mcp_oauth_auth_codes + WHERE data->>'clientId' = 'client_mcp_race'`, + ); + await pool.query(`DELETE FROM registry_audit_log WHERE workos_user_id = $1`, [USER_ID]); + await pool.query(`DELETE FROM users WHERE workos_user_id = $1`, [USER_ID]); + } + + it('issues no local code when deletion commits after exchange starts but before local finalize', async () => { + let exchangeEntered!: () => void; + let releaseExchange!: () => void; + const entered = new Promise<void>((resolve) => { exchangeEntered = resolve; }); + const release = new Promise<void>((resolve) => { releaseExchange = resolve; }); + mocks.authenticateWithCodeForTokens.mockImplementation(async () => { + exchangeEntered(); + await release; + return { + user: { + id: USER_ID, + email: 'mcp-race@example.test', + firstName: 'MCP', + lastName: 'Race', + emailVerified: true, + createdAt: new Date().toISOString(), + updatedAt: new Date().toISOString(), + }, + accessToken: 'access-token', + refreshToken: 'refresh-token', + }; + }); + const res = { + status: vi.fn().mockReturnThis(), + json: vi.fn().mockReturnThis(), + redirect: vi.fn().mockReturnThis(), + } as unknown as Response; + + const callback = handleMCPOAuthCallback({} as never, res, 'workos-code', PENDING_ID); + await entered; + await deleteIdentityCredential(USER_ID, 'workos_webhook'); + releaseExchange(); + await callback; + + expect(res.redirect).toHaveBeenCalledTimes(1); + const redirect = new URL(vi.mocked(res.redirect).mock.calls[0][0] as string); + expect(redirect.searchParams.get('error')).toBe('access_denied'); + expect(redirect.searchParams.has('code')).toBe(false); + expect((await pool.query( + `SELECT 1 FROM mcp_oauth_auth_codes WHERE data->>'clientId' = 'client_mcp_race'`, + )).rows).toEqual([]); + expect((await pool.query(`SELECT 1 FROM users WHERE workos_user_id = $1`, [USER_ID])).rows) + .toEqual([]); + }); +}); diff --git a/server/tests/integration/membership-webhook.test.ts b/server/tests/integration/membership-webhook.test.ts index b5b4161691b..fe9505e71bf 100644 --- a/server/tests/integration/membership-webhook.test.ts +++ b/server/tests/integration/membership-webhook.test.ts @@ -37,11 +37,19 @@ describe('Membership webhook DB operations', () => { connectionString: process.env.DATABASE_URL || 'postgresql://adcp:localdev@localhost:5432/adcp_test', }); await runMigrations(); + await pool.query( + `INSERT INTO users (workos_user_id, email, created_at, updated_at) + VALUES ($1, 'one@membership.test', NOW(), NOW()), + ($2, 'two@membership.test', NOW(), NOW()) + ON CONFLICT (workos_user_id) DO NOTHING`, + [TEST_USER_1, TEST_USER_2], + ); }, 60000); afterAll(async () => { await pool.query('DELETE FROM invitation_seat_types WHERE workos_organization_id = $1', [TEST_ORG_ID]); await pool.query('DELETE FROM organization_memberships WHERE workos_organization_id = $1', [TEST_ORG_ID]); + await pool.query('DELETE FROM users WHERE workos_user_id = ANY($1)', [[TEST_USER_1, TEST_USER_2]]); await closeDatabase(); }); @@ -267,8 +275,8 @@ describe('Membership webhook DB operations', () => { expect(updateOrganizationMembership).not.toHaveBeenCalled(); }); - it('promotes to owner in WorkOS first when the org has no other admin/owner', async () => { - const { client, updateOrganizationMembership } = makeWorkos({ + it('defers ownerless promotion when the org has no other admin/owner', async () => { + const { client, updateOrganizationMembership, listOrganizationMemberships } = makeWorkos({ memberships: [{ userId: TEST_USER_1, role: { slug: 'member' } }], }); @@ -280,12 +288,13 @@ describe('Membership webhook DB operations', () => { incomingRole: 'member', }); - expect(result).toEqual({ role: 'owner', promoted: true }); - expect(updateOrganizationMembership).toHaveBeenCalledWith('om_target', { roleSlug: 'owner' }); + expect(result).toEqual({ role: 'member', promoted: false }); + expect(listOrganizationMemberships).not.toHaveBeenCalled(); + expect(updateOrganizationMembership).not.toHaveBeenCalled(); }); it('leaves role as member when another admin already exists in WorkOS', async () => { - const { client, updateOrganizationMembership } = makeWorkos({ + const { client, updateOrganizationMembership, listOrganizationMemberships } = makeWorkos({ memberships: [ { userId: TEST_USER_2, role: { slug: 'admin' } }, { userId: TEST_USER_1, role: { slug: 'member' } }, @@ -301,11 +310,12 @@ describe('Membership webhook DB operations', () => { }); expect(result).toEqual({ role: 'member', promoted: false }); + expect(listOrganizationMemberships).not.toHaveBeenCalled(); expect(updateOrganizationMembership).not.toHaveBeenCalled(); }); - it('falls back to member when the WorkOS update fails (no drift)', async () => { - const { client } = makeWorkos({ + it('does not attempt an owner update even when the client would fail', async () => { + const { client, updateOrganizationMembership } = makeWorkos({ memberships: [{ userId: TEST_USER_1, role: { slug: 'member' } }], updateFails: true, }); @@ -320,11 +330,12 @@ describe('Membership webhook DB operations', () => { expect(result.role).toBe('member'); expect(result.promoted).toBe(false); - expect(result.promotionError).toBeDefined(); + expect(result.promotionError).toBeUndefined(); + expect(updateOrganizationMembership).not.toHaveBeenCalled(); }); - it('falls back to member when WorkOS membership list fails (refuses to promote blind)', async () => { - const { client, updateOrganizationMembership } = makeWorkos({ + it('does not list provider memberships while promotion is deferred', async () => { + const { client, updateOrganizationMembership, listOrganizationMemberships } = makeWorkos({ memberships: [], listFails: true, }); @@ -339,7 +350,8 @@ describe('Membership webhook DB operations', () => { expect(result.role).toBe('member'); expect(result.promoted).toBe(false); - expect(result.promotionError).toBeDefined(); + expect(result.promotionError).toBeUndefined(); + expect(listOrganizationMemberships).not.toHaveBeenCalled(); expect(updateOrganizationMembership).not.toHaveBeenCalled(); }); }); @@ -547,24 +559,17 @@ describe('Membership webhook DB operations', () => { ); } - it('always creates membership as member; upsert path handles auto-promotion atomically', async () => { + it('defers automatic direct-domain provider authority without an intent journal', async () => { await seedOrgWithVerifiedDomain('autolink.com'); const workos = makeWorkOSMock(); const result = await autoLinkByVerifiedDomain(workos, AUTOLINK_USER, 'matt@autolink.com'); - expect(result).not.toBeNull(); - expect(result!.organizationId).toBe(TEST_AUTOLINK_ORG_ID); - expect(result!.organizationName).toBe('AutoLink Corp'); - expect(result!.role).toBe('member'); - expect(workos.userManagement.createOrganizationMembership).toHaveBeenCalledWith({ - userId: AUTOLINK_USER, - organizationId: TEST_AUTOLINK_ORG_ID, - roleSlug: 'member', - }); + expect(result).toBeNull(); + expect(workos.userManagement.createOrganizationMembership).not.toHaveBeenCalled(); }); - it('still creates as member when org already has an admin', async () => { + it('defers automatic provider authority even when the org already has an admin', async () => { await seedOrgWithVerifiedDomain('autolink.com'); await pool.query( `INSERT INTO organization_memberships (workos_user_id, workos_organization_id, email, role, seat_type, created_at, updated_at, synced_at) @@ -575,10 +580,8 @@ describe('Membership webhook DB operations', () => { const result = await autoLinkByVerifiedDomain(workos, AUTOLINK_USER, 'matt@autolink.com'); - expect(result).not.toBeNull(); - expect(workos.userManagement.createOrganizationMembership).toHaveBeenCalledWith( - expect.objectContaining({ roleSlug: 'member' }), - ); + expect(result).toBeNull(); + expect(workos.userManagement.createOrganizationMembership).not.toHaveBeenCalled(); }); it('returns null when no matching verified domain exists', async () => { @@ -623,13 +626,13 @@ describe('Membership webhook DB operations', () => { expect(result).toBeNull(); }); - it('handles membership_already_exists gracefully', async () => { + it('does not call WorkOS even when an already-exists response would have been safe to retry', async () => { await seedOrgWithVerifiedDomain('autolink.com'); const workos = makeWorkOSMock({ shouldFail: true, errorCode: 'organization_membership_already_exists' }); const result = await autoLinkByVerifiedDomain(workos, AUTOLINK_USER, 'matt@autolink.com'); - expect(result).not.toBeNull(); - expect(result!.organizationId).toBe(TEST_AUTOLINK_ORG_ID); + expect(result).toBeNull(); + expect(workos.userManagement.createOrganizationMembership).not.toHaveBeenCalled(); }); it('returns null on other WorkOS errors', async () => { @@ -676,7 +679,7 @@ describe('Membership webhook DB operations', () => { expect(workos.userManagement.createOrganizationMembership).not.toHaveBeenCalled(); }); - it('still creates membership when user has memberships in OTHER orgs (the Triton case)', async () => { + it('defers automatic authority when the user has memberships in other orgs', async () => { await seedOrgWithVerifiedDomain('autolink.com'); // Seed an unrelated personal org membership for the user. await pool.query( @@ -689,14 +692,8 @@ describe('Membership webhook DB operations', () => { const result = await autoLinkByVerifiedDomain(workos, AUTOLINK_USER, 'matt@autolink.com'); - expect(result).not.toBeNull(); - expect(result!.organizationId).toBe(TEST_AUTOLINK_ORG_ID); - expect(workos.userManagement.createOrganizationMembership).toHaveBeenCalledWith( - expect.objectContaining({ - userId: AUTOLINK_USER, - organizationId: TEST_AUTOLINK_ORG_ID, - }), - ); + expect(result).toBeNull(); + expect(workos.userManagement.createOrganizationMembership).not.toHaveBeenCalled(); // Cleanup the personal-org seed await pool.query( @@ -704,12 +701,12 @@ describe('Membership webhook DB operations', () => { ); }); - it('stages provisioning_source=verified_domain so the webhook can record it', async () => { + it('does not stage a membership when automatic provider authority is deferred', async () => { await seedOrgWithVerifiedDomain('autolink.com'); const workos = makeWorkOSMock(); const result = await autoLinkByVerifiedDomain(workos, AUTOLINK_USER, 'matt@autolink.com'); - expect(result).not.toBeNull(); + expect(result).toBeNull(); // The staging row should now exist for the org+email pair so the // organization_membership.created webhook can consume it. @@ -718,9 +715,7 @@ describe('Membership webhook DB operations', () => { WHERE workos_organization_id = $1 AND lower(email) = lower($2)`, [TEST_AUTOLINK_ORG_ID, 'matt@autolink.com'], ); - expect(staged.rows[0]).toBeDefined(); - expect(staged.rows[0].seat_type).toBe('community_only'); - expect(staged.rows[0].source).toBe('verified_domain'); + expect(staged.rows).toEqual([]); }); // ───────────────────────────────────────────────────────────────── @@ -776,19 +771,14 @@ describe('Membership webhook DB operations', () => { await clearBrandHierarchy(); }); - it('links @child.com user to parent when org has opted into hierarchical auto-provisioning', async () => { + it('defers hierarchical provider authority even when the org opted in', async () => { await seedBrandHierarchy(); const workos = makeWorkOSMock(); const result = await autoLinkByVerifiedDomain(workos, AUTOLINK_USER, `mike@${CHILD_DOMAIN}`); - expect(result).not.toBeNull(); - expect(result!.organizationId).toBe(TEST_AUTOLINK_ORG_ID); - expect(workos.userManagement.createOrganizationMembership).toHaveBeenCalledWith({ - userId: AUTOLINK_USER, - organizationId: TEST_AUTOLINK_ORG_ID, - roleSlug: 'member', - }); + expect(result).toBeNull(); + expect(workos.userManagement.createOrganizationMembership).not.toHaveBeenCalled(); }); it('does NOT auto-link via hierarchy by default (auto_provision_brand_hierarchy_children = false)', async () => { @@ -830,7 +820,7 @@ describe('Membership webhook DB operations', () => { expect(workos.userManagement.createOrganizationMembership).not.toHaveBeenCalled(); }); - it('honors direct auto-provisioning opt-out independently for direct matches', async () => { + it('defers hierarchy authority independently of the direct-domain flag', async () => { // Direct=false (opt-out), hierarchy=true. A user matching the // child via hierarchy still gets in. await seedBrandHierarchy(); @@ -841,9 +831,9 @@ describe('Membership webhook DB operations', () => { ); const workos = makeWorkOSMock(); - // Hierarchical match still works. const inheritedResult = await autoLinkByVerifiedDomain(workos, AUTOLINK_USER, `mike@${CHILD_DOMAIN}`); - expect(inheritedResult).not.toBeNull(); + expect(inheritedResult).toBeNull(); + expect(workos.userManagement.createOrganizationMembership).not.toHaveBeenCalled(); }); it('short-circuits when the user is already in the resolved parent org', async () => { @@ -893,7 +883,7 @@ describe('Membership webhook DB operations', () => { await pool.query('DELETE FROM users WHERE workos_user_id = $1', [AUTOLINK_USER]); }); - it('cohort: still auto-links a NEW user (created after flag flip) via hierarchy', async () => { + it('cohort: defers a new user created after the hierarchy flag flip', async () => { await seedBrandHierarchy({ hierarchyOptIn: false }); // start opted out // Flip flag at T0 (sets enabled_at = NOW()). await pool.query( @@ -912,13 +902,13 @@ describe('Membership webhook DB operations', () => { const workos = makeWorkOSMock(); const result = await autoLinkByVerifiedDomain(workos, AUTOLINK_USER, `mike@${CHILD_DOMAIN}`); - expect(result).not.toBeNull(); - expect(result!.organizationId).toBe(TEST_AUTOLINK_ORG_ID); + expect(result).toBeNull(); + expect(workos.userManagement.createOrganizationMembership).not.toHaveBeenCalled(); await pool.query('DELETE FROM users WHERE workos_user_id = $1', [AUTOLINK_USER]); }); - it('cohort: auto-links when no users row exists yet (just-created via webhook)', async () => { + it('cohort: does not treat a missing user row as permission to auto-link', async () => { // Webhook race: autoLink fires before user.created webhook lands the // local users row. Don't block on a missing row — treat as new joiner. await seedBrandHierarchy(); // hierarchyOptIn defaults to true @@ -929,11 +919,11 @@ describe('Membership webhook DB operations', () => { const result = await autoLinkByVerifiedDomain(workos, AUTOLINK_USER, `mike@${CHILD_DOMAIN}`); - expect(result).not.toBeNull(); - expect(result!.organizationId).toBe(TEST_AUTOLINK_ORG_ID); + expect(result).toBeNull(); + expect(workos.userManagement.createOrganizationMembership).not.toHaveBeenCalled(); }); - it('prefers a direct verified-domain match over a hierarchical one when both exist', async () => { + it('does not choose a direct or hierarchical target while automatic writes are deferred', async () => { // Direct match wins: even if the brand registry says child→parent, // an explicit organization_domains row on the child takes priority. await seedBrandHierarchy(); @@ -955,8 +945,8 @@ describe('Membership webhook DB operations', () => { const workos = makeWorkOSMock(); const result = await autoLinkByVerifiedDomain(workos, AUTOLINK_USER, `mike@${CHILD_DOMAIN}`); - expect(result).not.toBeNull(); - expect(result!.organizationId).toBe(DIRECT_ORG); + expect(result).toBeNull(); + expect(workos.userManagement.createOrganizationMembership).not.toHaveBeenCalled(); // Cleanup await pool.query('DELETE FROM organization_domains WHERE workos_organization_id = $1', [DIRECT_ORG]); diff --git a/server/tests/integration/merge-bind-not-delete.test.ts b/server/tests/integration/merge-bind-not-delete.test.ts index 083632b173f..de818823f60 100644 --- a/server/tests/integration/merge-bind-not-delete.test.ts +++ b/server/tests/integration/merge-bind-not-delete.test.ts @@ -1,34 +1,38 @@ /** - * mergeUsers — bind, don't delete (Phase 2a) integration tests. - * - * The new contract: - * - All of the secondary user's app-state rows move to the primary (same - * as before). - * - The secondary WorkOS user stays alive in `users`. Each linked email is - * a real sign-in credential. - * - Both WorkOS users end up bound to the primary's identity. The - * secondary's binding is `is_primary = FALSE`. - * - The secondary's orphaned singleton identity gets dropped. + * Generic identity consolidation containment for #6827. The former merge + * success contract must stay disabled until authority/provenance are preserved. + * These fixtures deliberately include both conflicting and exclusive grants. */ - -import { describe, it, expect, beforeAll, afterAll, beforeEach } from 'vitest'; -import { initializeDatabase, closeDatabase, getPool } from '../../src/db/client.js'; -import { runMigrations } from '../../src/db/migrate.js'; -import { mergeUsers } from '../../src/db/user-merge-db.js'; +import { afterAll, beforeAll, beforeEach, describe, expect, it } from 'vitest'; import type { Pool } from 'pg'; - -const PRIMARY_ID = 'user_bind_test_primary'; -const SECONDARY_ID = 'user_bind_test_secondary'; -const ORG_ID = 'org_bind_test'; - -describe('mergeUsers (bind, don\'t delete)', () => { +import { closeDatabase, initializeDatabase } from '../../src/db/client.js'; +import { runMigrations } from '../../src/db/migrate.js'; +import { mergeUsers, previewUserMerge } from '../../src/db/user-merge-db.js'; +import { promoteSecondaryIfPrimaryDeleted } from '../../src/db/identity-db.js'; +import { bumpAuthorizationEpochs } from '../../src/db/authorization-epoch-db.js'; + +const JORDAN = 'user_containment_jordan'; +const SAM = 'user_containment_sam'; +const USERS = [JORDAN, SAM]; +const PINNACLE = 'org_containment_pinnacle'; +const PERSONAL = 'org_containment_sam'; +const ORGS = [PINNACLE, PERSONAL]; +const GROUP_SLUG = 'containment-governance'; +const refusal = { code: 'identity_mutation_disabled' }; + +describe('generic identity mutation containment preserves authority', () => { let pool: Pool; + let groupId: string; + let adminGroupId: string; beforeAll(async () => { pool = initializeDatabase({ connectionString: process.env.DATABASE_URL || 'postgresql://adcp:localdev@localhost:5432/adcp_test', }); await runMigrations(); + const adminGroup = await pool.query<{ id: string }>(`SELECT id FROM working_groups WHERE slug = 'aao-admin'`); + expect(adminGroup.rows).toHaveLength(1); + adminGroupId = adminGroup.rows[0].id; }, 60000); afterAll(async () => { @@ -39,420 +43,166 @@ describe('mergeUsers (bind, don\'t delete)', () => { beforeEach(async () => { await cleanup(); await pool.query( - `INSERT INTO users (workos_user_id, email, first_name, last_name, email_verified, - workos_created_at, workos_updated_at, created_at, updated_at) - VALUES ($1, 'primary@test.example', 'Primary', 'User', true, NOW(), NOW(), NOW(), NOW()), - ($2, 'secondary@test.example', 'Secondary', 'User', true, NOW(), NOW(), NOW(), NOW())`, - [PRIMARY_ID, SECONDARY_ID] + `INSERT INTO organizations (workos_organization_id, name, subscription_status, stripe_subscription_id) + VALUES ($1, 'Pinnacle Agency', 'active', 'sub_containment_pinnacle'), + ($2, 'Sam personal workspace', NULL, NULL)`, ORGS, ); await pool.query( - `INSERT INTO organizations (workos_organization_id, name, created_at, updated_at) - VALUES ($1, 'Test Org', NOW(), NOW()) - ON CONFLICT (workos_organization_id) DO NOTHING`, - [ORG_ID] + `INSERT INTO users (workos_user_id, email, first_name, last_name, email_verified, + primary_organization_id, workos_created_at, workos_updated_at) + VALUES ($1, 'jordan@pinnacle.example', 'Jordan', 'Ochoa', true, $3, NOW(), NOW()), + ($2, 'sam@pinnacle.example', 'Sam', 'Adeyemi', true, $4, NOW(), NOW())`, + [JORDAN, SAM, PINNACLE, PERSONAL], ); - }); - - async function cleanup() { - // addie_threads.person_id has no CASCADE, and person_relationships has no - // FK to users(workos_user_id), so deleting users wouldn't reach either. - // Drop the threads first so the person_relationships DELETE can succeed. await pool.query( - `DELETE FROM addie_threads - WHERE person_id IN ( - SELECT id FROM person_relationships WHERE workos_user_id IN ($1, $2) - )`, - [PRIMARY_ID, SECONDARY_ID] + `INSERT INTO organization_memberships + (workos_user_id, workos_organization_id, workos_membership_id, email, role) + VALUES ($1, $3, 'om_containment_jordan', 'jordan@pinnacle.example', 'owner'), + ($2, $3, 'om_containment_sam_member', 'sam@pinnacle.example', 'member'), + ($2, $4, 'om_containment_sam_personal', 'sam@pinnacle.example', 'owner')`, + [JORDAN, SAM, PINNACLE, PERSONAL], ); - await pool.query( - `DELETE FROM person_relationships WHERE workos_user_id IN ($1, $2)`, - [PRIMARY_ID, SECONDARY_ID] + const group = await pool.query<{ id: string }>( + `INSERT INTO working_groups (name, slug, is_private, committee_type) + VALUES ('Governance containment fixture', $1, true, 'governance') RETURNING id`, [GROUP_SLUG], ); + groupId = group.rows[0].id; + // The active aao-admin membership is the actual platform-admin grant. + // Sam's inactive duplicate must never replace or absorb Jordan's grant. await pool.query( - `DELETE FROM addie_prompt_telemetry WHERE workos_user_id IN ($1, $2)`, - [PRIMARY_ID, SECONDARY_ID] + `INSERT INTO working_group_memberships (working_group_id, workos_user_id, status) + VALUES ($1, $3, 'active'), ($1, $4, 'inactive'), + ($2, $3, 'active'), ($2, $4, 'inactive')`, + [adminGroupId, groupId, JORDAN, SAM], ); await pool.query( - `DELETE FROM admin_module_completions WHERE workos_user_id IN ($1, $2)`, - [PRIMARY_ID, SECONDARY_ID] + `INSERT INTO aao_admin_access_events + (event_type, actor_user_id, target_user_id, mechanism, actor_authorization_mechanism, reason) + VALUES ('granted', $1, $1, 'aao_admin_working_group', 'development', 'Containment fixture grant')`, + [JORDAN], ); + await pool.query(`INSERT INTO working_group_leaders (working_group_id, user_id) VALUES ($1, $2)`, [groupId, JORDAN]); await pool.query( - `DELETE FROM learner_progress WHERE workos_user_id IN ($1, $2)`, - [PRIMARY_ID, SECONDARY_ID] + `INSERT INTO committee_interest (working_group_id, workos_user_id, interest_level) + VALUES ($1, $2, 'leader'), ($1, $3, 'participant')`, [groupId, JORDAN, SAM], ); await pool.query( - `DELETE FROM organization_memberships WHERE workos_user_id IN ($1, $2)`, - [PRIMARY_ID, SECONDARY_ID] - ); - await pool.query(`DELETE FROM users WHERE workos_user_id IN ($1, $2)`, [PRIMARY_ID, SECONDARY_ID]); - await pool.query(`DELETE FROM organizations WHERE workos_organization_id = $1`, [ORG_ID]); - } - - it('keeps the secondary WorkOS user alive after merge', async () => { - await mergeUsers(PRIMARY_ID, SECONDARY_ID, PRIMARY_ID); - - const result = await pool.query( - `SELECT workos_user_id FROM users WHERE workos_user_id IN ($1, $2)`, - [PRIMARY_ID, SECONDARY_ID] + `INSERT INTO working_group_topic_subscriptions (working_group_id, workos_user_id, topic_slugs) + VALUES ($1, $2, ARRAY['governance']), ($1, $3, ARRAY['learning'])`, [groupId, JORDAN, SAM], ); - expect(result.rows.map(r => r.workos_user_id).sort()).toEqual([PRIMARY_ID, SECONDARY_ID].sort()); - }); - - it('binds both WorkOS users to one identity, with the secondary marked non-primary', async () => { - await mergeUsers(PRIMARY_ID, SECONDARY_ID, PRIMARY_ID); - - const result = await pool.query<{ workos_user_id: string; identity_id: string; is_primary: boolean }>( - `SELECT workos_user_id, identity_id, is_primary - FROM identity_workos_users - WHERE workos_user_id IN ($1, $2) - ORDER BY workos_user_id`, - [PRIMARY_ID, SECONDARY_ID] - ); - - expect(result.rows).toHaveLength(2); - const [primary, secondary] = result.rows.sort((a, b) => - a.workos_user_id === PRIMARY_ID ? -1 : 1 - ); - expect(primary.workos_user_id).toBe(PRIMARY_ID); - expect(primary.is_primary).toBe(true); - expect(secondary.workos_user_id).toBe(SECONDARY_ID); - expect(secondary.is_primary).toBe(false); - expect(primary.identity_id).toBe(secondary.identity_id); - }); - - it('drops the secondary user\'s orphaned singleton identity', async () => { - // Capture the secondary's pre-merge identity id, then verify it's gone after. - const before = await pool.query<{ identity_id: string }>( - `SELECT identity_id FROM identity_workos_users WHERE workos_user_id = $1`, - [SECONDARY_ID] - ); - const secondaryIdentityBefore = before.rows[0].identity_id; - - await mergeUsers(PRIMARY_ID, SECONDARY_ID, PRIMARY_ID); - - const after = await pool.query( - `SELECT 1 FROM identities WHERE id = $1`, - [secondaryIdentityBefore] - ); - expect(after.rows).toEqual([]); - }); - - it('moves organization memberships from secondary to primary (existing read paths still work)', async () => { await pool.query( - `INSERT INTO organization_memberships (workos_user_id, workos_organization_id, email, role, created_at, updated_at) - VALUES ($1, $2, 'secondary@test.example', 'member', NOW(), NOW())`, - [SECONDARY_ID, ORG_ID] - ); - - await mergeUsers(PRIMARY_ID, SECONDARY_ID, PRIMARY_ID); - - const result = await pool.query<{ workos_user_id: string }>( - `SELECT workos_user_id FROM organization_memberships WHERE workos_organization_id = $1`, - [ORG_ID] - ); - expect(result.rows).toHaveLength(1); - expect(result.rows[0].workos_user_id).toBe(PRIMARY_ID); - }); - - it('preserves completed certification progress when the primary has a lower-state duplicate', async () => { - const primaryProgress = await pool.query<{ id: string }>( - `INSERT INTO learner_progress - (workos_user_id, module_id, status, started_at, attempts, addie_thread_id) - VALUES ($1, 'B1', 'in_progress', NOW() - INTERVAL '1 day', 1, 'primary-thread') - RETURNING id`, - [PRIMARY_ID] - ); - const secondaryProgress = await pool.query<{ id: string }>( - `INSERT INTO learner_progress - (workos_user_id, module_id, status, started_at, completed_at, score, - attempts, addie_thread_id) - VALUES ($1, 'B1', 'completed', NOW() - INTERVAL '2 days', - NOW() - INTERVAL '1 hour', '{"knowledge": 92}', 2, - 'secondary-thread') - RETURNING id`, - [SECONDARY_ID] + `INSERT INTO subscription_line_items + (workos_organization_id, stripe_subscription_id, stripe_subscription_item_id, price_id, quantity) + VALUES ($1, 'sub_containment_pinnacle', 'si_containment_pinnacle', 'price_containment', 10)`, [PINNACLE], ); await pool.query( - `INSERT INTO learner_progress - (workos_user_id, module_id, status, started_at, attempts) - VALUES ($1, 'B2', 'in_progress', NOW(), 1)`, - [SECONDARY_ID] + `INSERT INTO user_email_aliases (workos_user_id, email) + VALUES ($1, 'jordan-alias@pinnacle.example'), ($2, 'sam-alias@pinnacle.example')`, USERS, ); await pool.query( `INSERT INTO learner_progress (workos_user_id, module_id, status) - VALUES ($1, 'B3', 'not_started'), ($2, 'B3', 'failed')`, - [PRIMARY_ID, SECONDARY_ID] - ); - await pool.query( - `INSERT INTO admin_module_completions - (workos_user_id, module_id, admin_user_id, addie_thread_id, score, - learner_progress_id) - VALUES ($1, 'B1', 'admin_test', 'secondary-thread', - '{"knowledge": 92}', $2)`, - [SECONDARY_ID, secondaryProgress.rows[0].id] - ); - - await mergeUsers(PRIMARY_ID, SECONDARY_ID, PRIMARY_ID); - - const progress = await pool.query<{ - id: string; - module_id: string; - status: string; - score: Record<string, number> | null; - attempts: number; - addie_thread_id: string | null; - }>( - `SELECT id, module_id, status, score, attempts, addie_thread_id - FROM learner_progress - WHERE workos_user_id = $1 - ORDER BY module_id`, - [PRIMARY_ID] - ); - expect(progress.rows).toHaveLength(3); - expect(progress.rows[0]).toMatchObject({ - id: primaryProgress.rows[0].id, - module_id: 'B1', - status: 'completed', - score: { knowledge: 92 }, - attempts: 3, - addie_thread_id: 'secondary-thread', - }); - expect(progress.rows[1]).toMatchObject({ - module_id: 'B2', - status: 'in_progress', - }); - expect(progress.rows[2]).toMatchObject({ - module_id: 'B3', - status: 'failed', - }); - - const secondaryRows = await pool.query( - `SELECT 1 FROM learner_progress WHERE workos_user_id = $1`, - [SECONDARY_ID] - ); - expect(secondaryRows.rows).toHaveLength(0); - - const audit = await pool.query<{ - workos_user_id: string; - learner_progress_id: string; - }>( - `SELECT workos_user_id, learner_progress_id - FROM admin_module_completions - WHERE module_id = 'B1' AND workos_user_id = $1`, - [PRIMARY_ID] - ); - expect(audit.rows).toEqual([{ - workos_user_id: PRIMARY_ID, - learner_progress_id: primaryProgress.rows[0].id, - }]); - }); - - it('records a merge_user audit row', async () => { - await mergeUsers(PRIMARY_ID, SECONDARY_ID, PRIMARY_ID); - - const result = await pool.query<{ resource_id: string; details: { primary_user_id: string; secondary_user_id: string } }>( - `SELECT resource_id, details - FROM registry_audit_log - WHERE action = 'merge_user' AND resource_id = $1 - ORDER BY created_at DESC LIMIT 1`, - [SECONDARY_ID] - ); - - expect(result.rows).toHaveLength(1); - expect(result.rows[0].details.primary_user_id).toBe(PRIMARY_ID); - expect(result.rows[0].details.secondary_user_id).toBe(SECONDARY_ID); - }); - - it('throws if the primary lacks an identity binding (Phase 1 trigger should make this impossible)', async () => { - // Manually break the invariant to confirm we fail loud rather than - // silently producing partial state. The AFTER INSERT trigger normally - // guarantees this can't happen. - await pool.query(`DELETE FROM identity_workos_users WHERE workos_user_id = $1`, [PRIMARY_ID]); - - await expect(mergeUsers(PRIMARY_ID, SECONDARY_ID, PRIMARY_ID)).rejects.toThrow( - /no identity binding/ - ); - - // ROLLBACK should leave the secondary intact (no partial bind). - const secondaryStill = await pool.query( - `SELECT 1 FROM users WHERE workos_user_id = $1`, - [SECONDARY_ID] - ); - expect(secondaryStill.rows).toHaveLength(1); - }); - - it('repoints addie_threads and person_events to the primary\'s person_relationship before deleting the secondary\'s', async () => { - // Both users have a person_relationships row (the path that previously - // failed with FK 23503 on addie_threads_person_id_fkey). - const primaryRel = await pool.query<{ id: string }>( - `INSERT INTO person_relationships (workos_user_id, email, stage) - VALUES ($1, 'primary@test.example', 'exploring') RETURNING id`, - [PRIMARY_ID] - ); - const secondaryRel = await pool.query<{ id: string }>( - `INSERT INTO person_relationships (workos_user_id, email, stage) - VALUES ($1, 'secondary@test.example', 'exploring') RETURNING id`, - [SECONDARY_ID] - ); - const primaryRelId = primaryRel.rows[0].id; - const secondaryRelId = secondaryRel.rows[0].id; - - // Thread keyed to secondary's person_relationship — the row that hit FK - // violations under the old code path. - const thread = await pool.query<{ thread_id: string }>( - `INSERT INTO addie_threads (channel, external_id, user_type, person_id) - VALUES ('slack', 'C_test_merge_pr', 'slack', $1) RETURNING thread_id`, - [secondaryRelId] - ); - await pool.query( - `INSERT INTO person_events (person_id, event_type, data) - VALUES ($1, 'message_received', '{}'::jsonb)`, - [secondaryRelId] + VALUES ($1, 'B1', 'completed'), ($2, 'B1', 'not_started')`, USERS, ); - - await mergeUsers(PRIMARY_ID, SECONDARY_ID, PRIMARY_ID); - - const threadAfter = await pool.query<{ person_id: string }>( - `SELECT person_id FROM addie_threads WHERE thread_id = $1`, - [thread.rows[0].thread_id] - ); - expect(threadAfter.rows[0].person_id).toBe(primaryRelId); - - const eventAfter = await pool.query<{ count: string }>( - `SELECT COUNT(*) as count FROM person_events WHERE person_id = $1 AND event_type = 'message_received'`, - [primaryRelId] - ); - expect(parseInt(eventAfter.rows[0].count, 10)).toBe(1); - - const secondaryRelAfter = await pool.query( - `SELECT 1 FROM person_relationships WHERE id = $1`, - [secondaryRelId] - ); - expect(secondaryRelAfter.rows).toHaveLength(0); + await bumpAuthorizationEpochs(pool, USERS); }); - it('refreshes person_relationships.email from users.email for the surviving primary row (promote case)', async () => { - // Promote shape: primary is the freshly-added credential with no - // person_relationships row of its own; secondary owns the row with the - // old email. After mergeUsers, the row is re-pointed to primary and its - // email column should reflect the primary's current users.email. - await pool.query( - `INSERT INTO person_relationships (workos_user_id, email, stage) - VALUES ($1, 'secondary@test.example', 'exploring')`, - [SECONDARY_ID] - ); + async function cleanup() { + await pool.query(`DELETE FROM aao_admin_access_events WHERE target_user_id = ANY($1)`, [USERS]); + await pool.query(`DELETE FROM working_group_memberships WHERE workos_user_id = ANY($1)`, [USERS]); + await pool.query(`DELETE FROM working_groups WHERE slug = $1`, [GROUP_SLUG]); + await pool.query(`DELETE FROM organization_memberships WHERE workos_user_id = ANY($1)`, [USERS]); + await pool.query(`DELETE FROM users WHERE workos_user_id = ANY($1)`, [USERS]); + await pool.query(`DELETE FROM organizations WHERE workos_organization_id = ANY($1)`, [ORGS]); + } - await mergeUsers(PRIMARY_ID, SECONDARY_ID, PRIMARY_ID); + async function snapshot() { + const tables: Record<string, unknown> = {}; + const selections: [string, string, string[]][] = [ + ['users', 'workos_user_id = ANY($1)', USERS], + ['identities', 'id IN (SELECT identity_id FROM identity_workos_users WHERE workos_user_id = ANY($1))', USERS], + ['identity_workos_users', 'workos_user_id = ANY($1)', USERS], + ['authorization_epochs', 'workos_user_id = ANY($1)', USERS], + ['organization_memberships', 'workos_user_id = ANY($1)', USERS], + ['organizations', 'workos_organization_id = ANY($1)', ORGS], + ['working_group_memberships', 'workos_user_id = ANY($1)', USERS], + ['working_group_leaders', 'user_id = ANY($1)', USERS], + ['committee_interest', 'workos_user_id = ANY($1)', USERS], + ['working_group_topic_subscriptions', 'workos_user_id = ANY($1)', USERS], + ['subscription_line_items', 'workos_organization_id = ANY($1)', ORGS], + ['aao_admin_access_events', 'target_user_id = ANY($1)', USERS], + ['user_email_aliases', 'workos_user_id = ANY($1)', USERS], + ['learner_progress', 'workos_user_id = ANY($1)', USERS], + ]; + // Compare complete rows, including timestamps, ownership, roles and IDs. + for (const [table, where, ids] of selections) { + const result = await pool.query( + `SELECT row_to_json(row) AS row FROM (SELECT * FROM ${table} WHERE ${where}) row + ORDER BY row_to_json(row)::text`, [ids], + ); + tables[table] = result.rows.map(({ row }) => row); + expect(result.rows.length, `${table} fixture must exercise real rows`).toBeGreaterThan(0); + } + return tables; + } - const rel = await pool.query<{ workos_user_id: string; email: string }>( - `SELECT workos_user_id, email FROM person_relationships WHERE workos_user_id = $1`, - [PRIMARY_ID] - ); - expect(rel.rows).toHaveLength(1); - expect(rel.rows[0].email).toBe('primary@test.example'); + it.each([ + ['privilege gain', SAM, JORDAN], + ['privilege loss', JORDAN, SAM], + ])('refuses the %s direction without moving or deleting any authority', async (_direction, primary, secondary) => { + const before = await snapshot(); + await expect(mergeUsers(primary, secondary, JORDAN)).rejects.toMatchObject(refusal); + expect(await snapshot()).toEqual(before); }); - it('refreshes person_relationships.email for primary\'s pre-existing row when its email is stale (merge case)', async () => { - // Both users have a person_relationships row, and primary's row carries a - // stale email that pre-dates the credential swap. After mergeUsers, the - // secondary's row is deleted and the primary's row keeps its id but its - // email column gets re-derived from users.email. - const primaryRel = await pool.query<{ id: string }>( - `INSERT INTO person_relationships (workos_user_id, email, stage) - VALUES ($1, 'stale-primary@test.example', 'exploring') RETURNING id`, - [PRIMARY_ID] + it.each([ + ['privileged to ordinary', JORDAN, SAM], + ['ordinary to privileged', SAM, JORDAN], + ])('refuses automatic promotion from %s with historical bindings', async (_direction, primary, secondary) => { + const original = await pool.query<{ identity_id: string }>( + `SELECT identity_id FROM identity_workos_users WHERE workos_user_id = $1`, [secondary], ); + // Historical bindings are fixture input, not permission to transfer grants. await pool.query( - `INSERT INTO person_relationships (workos_user_id, email, stage) - VALUES ($1, 'secondary@test.example', 'exploring')`, - [SECONDARY_ID] + `UPDATE identity_workos_users + SET identity_id = (SELECT identity_id FROM identity_workos_users WHERE workos_user_id = $1), + is_primary = FALSE + WHERE workos_user_id = $2`, [primary, secondary], ); + await pool.query(`DELETE FROM identities WHERE id = $1`, [original.rows[0].identity_id]); + const before = await snapshot(); - await mergeUsers(PRIMARY_ID, SECONDARY_ID, PRIMARY_ID); + await expect(promoteSecondaryIfPrimaryDeleted(primary)).rejects.toMatchObject(refusal); - const rel = await pool.query<{ id: string; email: string }>( - `SELECT id, email FROM person_relationships WHERE workos_user_id = $1`, - [PRIMARY_ID] - ); - expect(rel.rows).toHaveLength(1); - expect(rel.rows[0].id).toBe(primaryRel.rows[0].id); - expect(rel.rows[0].email).toBe('primary@test.example'); + expect(await snapshot()).toEqual(before); }); - it('refreshes organization_memberships.email from users.email when it has drifted', async () => { - await pool.query( - `INSERT INTO organization_memberships (workos_user_id, workos_organization_id, email, role, created_at, updated_at) - VALUES ($1, $2, 'stale-primary@test.example', 'member', NOW(), NOW())`, - [PRIMARY_ID, ORG_ID] - ); - - await mergeUsers(PRIMARY_ID, SECONDARY_ID, PRIMARY_ID); - - const row = await pool.query<{ email: string }>( - `SELECT email FROM organization_memberships WHERE workos_user_id = $1 AND workos_organization_id = $2`, - [PRIMARY_ID, ORG_ID] - ); - expect(row.rows).toHaveLength(1); - expect(row.rows[0].email).toBe('primary@test.example'); + it('does not permit explicit consolidation or primary-promotion options to bypass refusal', async () => { + const before = await snapshot(); + await expect(mergeUsers(SAM, JORDAN, JORDAN, { + ensurePrimaryFlag: true, + auditContext: { consolidation_confirmed: true, consolidate: true }, + })).rejects.toMatchObject(refusal); + expect(await snapshot()).toEqual(before); }); - it('refreshes email even on the consolidate path when secondary has app-state to move', async () => { - // Consolidate shape: secondary owns app-state (telemetry rows + a stale - // relationship row) and primary is the newly-bound credential. The merge - // moves data forward AND refreshes the denormalized email on the - // surviving primary row. This proves the refresh isn't short-circuited - // by any of the other merge branches. - await pool.query( - `INSERT INTO addie_prompt_telemetry (workos_user_id, rule_id, shown_count, clicked_count) - VALUES ($1, 'secondary_rule', 3, 1)`, - [SECONDARY_ID] - ); - await pool.query( - `INSERT INTO person_relationships (workos_user_id, email, stage) - VALUES ($1, 'secondary@test.example', 'exploring')`, - [SECONDARY_ID] - ); - - await mergeUsers(PRIMARY_ID, SECONDARY_ID, PRIMARY_ID); - - const rel = await pool.query<{ email: string }>( - `SELECT email FROM person_relationships WHERE workos_user_id = $1`, - [PRIMARY_ID] - ); - expect(rel.rows[0].email).toBe('primary@test.example'); - - const telemetry = await pool.query<{ rule_id: string }>( - `SELECT rule_id FROM addie_prompt_telemetry WHERE workos_user_id = $1`, - [PRIMARY_ID] - ); - expect(telemetry.rows.map(r => r.rule_id)).toEqual(['secondary_rule']); + it('preserves all authority under concurrent, reversed and replayed calls', async () => { + const before = await snapshot(); + const attempts = Array.from({ length: 12 }, (_, index) => index % 3 === 0 + ? promoteSecondaryIfPrimaryDeleted(index % 2 ? SAM : JORDAN) + : mergeUsers(index % 2 ? SAM : JORDAN, index % 2 ? JORDAN : SAM, JORDAN, { ensurePrimaryFlag: true })); + for (const result of await Promise.allSettled(attempts)) { + expect(result.status).toBe('rejected'); + if (result.status === 'rejected') expect(result.reason).toMatchObject(refusal); + } + await expect(mergeUsers(SAM, JORDAN, JORDAN)).rejects.toMatchObject(refusal); + expect(await snapshot()).toEqual(before); }); - it('merges addie_prompt_telemetry on (workos_user_id, rule_id), keeping primary\'s row when both have one for the same rule', async () => { - // Same rule on both — primary's counters survive, secondary's are dropped. - await pool.query( - `INSERT INTO addie_prompt_telemetry (workos_user_id, rule_id, shown_count, clicked_count) - VALUES ($1, 'shared_rule', 5, 2), - ($2, 'shared_rule', 9, 4), - ($2, 'unique_rule', 1, 0)`, - [PRIMARY_ID, SECONDARY_ID] - ); - - await mergeUsers(PRIMARY_ID, SECONDARY_ID, PRIMARY_ID); - - const after = await pool.query<{ rule_id: string; shown_count: number; clicked_count: number }>( - `SELECT rule_id, shown_count, clicked_count FROM addie_prompt_telemetry - WHERE workos_user_id = $1 ORDER BY rule_id`, - [PRIMARY_ID] - ); - expect(after.rows).toHaveLength(2); - expect(after.rows[0]).toMatchObject({ rule_id: 'shared_rule', shown_count: 5, clicked_count: 2 }); - expect(after.rows[1]).toMatchObject({ rule_id: 'unique_rule', shown_count: 1, clicked_count: 0 }); - - const secondaryAfter = await pool.query( - `SELECT 1 FROM addie_prompt_telemetry WHERE workos_user_id = $1`, - [SECONDARY_ID] - ); - expect(secondaryAfter.rows).toHaveLength(0); + it('keeps read-only detection available without changing authority', async () => { + const before = await snapshot(); + const preview = await previewUserMerge(SAM, JORDAN); + expect(preview.tables).toContainEqual({ table_name: 'organization_memberships', row_count: 1 }); + expect(preview.tables).toContainEqual({ table_name: 'working_group_memberships', row_count: 2 }); + expect(await snapshot()).toEqual(before); }); }); diff --git a/server/tests/integration/workos-user-deleted-primary.test.ts b/server/tests/integration/workos-user-deleted-primary.test.ts index 2e68d0276f0..5ec601ccae5 100644 --- a/server/tests/integration/workos-user-deleted-primary.test.ts +++ b/server/tests/integration/workos-user-deleted-primary.test.ts @@ -1,28 +1,48 @@ /** - * WorkOS user.deleted webhook — primary-bound user safety. - * - * Issue adcontextprotocol/adcp#3718: deleting the primary binding on a - * multi-credential identity (e.g. GDPR/CCPA erasure via WorkOS dashboard) - * leaves the identity with zero primaries after the CASCADE on - * identity_workos_users.workos_user_id fires. `attachIdentityId` then - * resolves the surviving secondary to a NULL primary and skips the id-swap, - * so they sign in to an empty workspace. - * - * Mitigation: promote the longest-bound surviving secondary to primary in - * the same transaction, BEFORE the CASCADE runs. This test exercises the - * `promoteSecondaryIfPrimaryDeleted` helper end-to-end against a real DB - * and asserts the wiring in the user.deleted handler. + * Automatic promotion containment: historical multi-credential bindings must + * not change primary routing when a provider deletes a credential. Ordinary + * deletion remains valid and must not silently promote a surviving credential. */ -import { describe, it, expect, beforeAll, afterAll, beforeEach } from 'vitest'; +import { describe, it, expect, beforeAll, afterAll, beforeEach, vi } from 'vitest'; + +vi.mock('../../src/addie/error-notifier.js', async (importOriginal) => ({ + ...(await importOriginal<typeof import('../../src/addie/error-notifier.js')>()), + notifySystemError: vi.fn(), +})); + import { initializeDatabase, closeDatabase } from '../../src/db/client.js'; import { runMigrations } from '../../src/db/migrate.js'; -import { promoteSecondaryIfPrimaryDeleted } from '../../src/db/identity-db.js'; +import { + getIdentityRecoveryQuarantine, + IDENTITY_DELETION_SNAPSHOT_MAX_DETAILS_BYTES, + IDENTITY_DELETION_SNAPSHOT_POLICY, + IDENTITY_RECOVERY_STATE, + promoteSecondaryIfPrimaryDeleted, + USER_CASCADE_SNAPSHOT_INVENTORY, + USER_SET_NULL_SNAPSHOT_INVENTORY, +} from '../../src/db/identity-db.js'; +import { deleteIdentityCredential } from '../../src/services/identity-credential-deletion.js'; +import { + bumpAuthorizationEpochs, + getAuthorizationFingerprint, + readCredentialAuthorizationLifecycle, +} from '../../src/db/authorization-epoch-db.js'; +import { notifySystemError } from '../../src/addie/error-notifier.js'; +import { isSlackUserAAOAdmin } from '../../src/addie/mcp/admin-tools.js'; +import { getSlackAdminStatusCache } from '../../src/addie/admin-status-cache.js'; +import { + getUnifiedUsersCache, + setUnifiedUsersCache, +} from '../../src/cache/unified-users.js'; import type { Pool } from 'pg'; const TEST_USER_PREFIX = 'user_wh_deleted_test_'; +const TEST_SLACK_PREFIX = 'slack_wh_deleted_test_'; +const TEST_ORG = 'org_wh_deleted_test_pinnacle'; +const TEST_ORG_PREFIX = 'org_wh_deleted_test_'; -describe('user.deleted: promote secondary before CASCADE (#3718)', () => { +describe('user.deleted: automatic promotion containment (#6827)', () => { let pool: Pool; beforeAll(async () => { @@ -34,14 +54,45 @@ describe('user.deleted: promote secondary before CASCADE (#3718)', () => { }, 60000); afterAll(async () => { - await pool.query(`DELETE FROM users WHERE workos_user_id LIKE $1`, [`${TEST_USER_PREFIX}%`]); + await cleanup(); await closeDatabase(); }); beforeEach(async () => { - await pool.query(`DELETE FROM users WHERE workos_user_id LIKE $1`, [`${TEST_USER_PREFIX}%`]); + await cleanup(); + vi.clearAllMocks(); + notifySystemError.mockReset(); }); + async function cleanup() { + await pool.query(`DELETE FROM aao_admin_access_events WHERE target_user_id LIKE $1`, [`${TEST_USER_PREFIX}%`]); + await pool.query( + `DELETE FROM registry_audit_log + WHERE action IN ('identity_primary_deletion_quarantined', 'identity_credential_deleted') + AND workos_user_id LIKE $1`, + [`${TEST_USER_PREFIX}%`], + ); + await pool.query(`DELETE FROM organization_memberships WHERE workos_user_id LIKE $1`, [`${TEST_USER_PREFIX}%`]); + await pool.query( + `DELETE FROM working_group_memberships + WHERE workos_user_id LIKE $1 OR workos_user_id LIKE $2`, + [`${TEST_USER_PREFIX}%`, `${TEST_SLACK_PREFIX}%`], + ); + await pool.query( + `DELETE FROM working_group_leaders WHERE user_id LIKE $1 OR user_id LIKE $2`, + [`${TEST_USER_PREFIX}%`, `${TEST_SLACK_PREFIX}%`], + ); + await pool.query( + `DELETE FROM working_group_topic_subscriptions + WHERE workos_user_id LIKE $1 OR workos_user_id LIKE $2`, + [`${TEST_USER_PREFIX}%`, `${TEST_SLACK_PREFIX}%`], + ); + await pool.query(`DELETE FROM slack_user_mappings WHERE slack_user_id LIKE $1`, [`${TEST_SLACK_PREFIX}%`]); + await pool.query(`DELETE FROM users WHERE workos_user_id LIKE $1`, [`${TEST_USER_PREFIX}%`]); + await pool.query(`DELETE FROM organizations WHERE workos_organization_id LIKE $1`, [`${TEST_ORG_PREFIX}%`]); + vi.mocked(notifySystemError).mockClear(); + } + async function insertUser(suffix: string, email: string): Promise<string> { const userId = `${TEST_USER_PREFIX}${suffix}`; await pool.query( @@ -90,90 +141,905 @@ describe('user.deleted: promote secondary before CASCADE (#3718)', () => { return identityId; } - it('promotes the longest-bound secondary to primary when the primary is deleted', async () => { - const primary = await insertUser('promote_p', 'primary@test.example'); - const olderSecondary = await insertUser('promote_s1', 'older-secondary@test.example'); - const newerSecondary = await insertUser('promote_s2', 'newer-secondary@test.example'); - - // older binds first (smaller bound_at offset), newer binds later + it('refuses to choose among historical secondary credentials without changing bindings', async () => { + const primary = await insertUser('primary', 'jordan@pinnacle.example'); + const olderSecondary = await insertUser('older', 'sam@pinnacle.example'); + const newerSecondary = await insertUser('newer', 'maya@pinnacle.example'); const identityId = await bindAsSecondary(primary, olderSecondary, 1); await bindAsSecondary(primary, newerSecondary, 2); + const before = await pool.query( + `SELECT * FROM identity_workos_users WHERE identity_id = $1 ORDER BY workos_user_id`, [identityId], + ); + + await expect(promoteSecondaryIfPrimaryDeleted(primary)).rejects.toMatchObject({ + code: 'identity_mutation_disabled', + }); + + const after = await pool.query( + `SELECT * FROM identity_workos_users WHERE identity_id = $1 ORDER BY workos_user_id`, [identityId], + ); + expect(after.rows).toEqual(before.rows); + }); + + it.each(['primary', 'secondary'])('refuses %s promotion calls, including replays', async (target) => { + const primary = await insertUser('replay_primary', 'jordan-replay@pinnacle.example'); + const secondary = await insertUser('replay_secondary', 'sam-replay@pinnacle.example'); + const identityId = await bindAsSecondary(primary, secondary, 1); + const before = await pool.query( + `SELECT * FROM identity_workos_users WHERE identity_id = $1 ORDER BY workos_user_id`, [identityId], + ); + const userId = target === 'primary' ? primary : secondary; + for (const result of await Promise.allSettled(Array.from({ length: 8 }, () => promoteSecondaryIfPrimaryDeleted(userId)))) { + expect(result.status).toBe('rejected'); + if (result.status === 'rejected') expect(result.reason).toMatchObject({ code: 'identity_mutation_disabled' }); + } + await expect(promoteSecondaryIfPrimaryDeleted(userId)).rejects.toMatchObject({ code: 'identity_mutation_disabled' }); + const after = await pool.query( + `SELECT * FROM identity_workos_users WHERE identity_id = $1 ORDER BY workos_user_id`, [identityId], + ); + expect(after.rows).toEqual(before.rows); + }); + + it('allows ordinary deletion without promoting a historical secondary', async () => { + const primary = await insertUser('cascade_primary', 'jordan-cascade@pinnacle.example'); + const secondary = await insertUser('cascade_secondary', 'sam-cascade@pinnacle.example'); + const identityId = await bindAsSecondary(primary, secondary, 1); + await pool.query(`DELETE FROM users WHERE workos_user_id = $1`, [primary]); + + const survivors = await pool.query<{ workos_user_id: string; is_primary: boolean }>( + `SELECT workos_user_id, is_primary FROM identity_workos_users WHERE identity_id = $1`, [identityId], + ); + expect(survivors.rows).toEqual([{ workos_user_id: secondary, is_primary: false }]); + }); + + async function seedDeletionAuthority() { + const primary = await insertUser('revoke_primary', 'jordan-revoke@pinnacle.example'); + const secondary = await insertUser('revoke_secondary', 'sam-revoke@pinnacle.example'); + const bystander = await insertUser('revoke_bystander', 'maya-revoke@pinnacle.example'); + const identityId = await bindAsSecondary(primary, secondary, 1); + await bindAsSecondary(primary, bystander, 2); + const grantOrganizations = [TEST_ORG, `${TEST_ORG_PREFIX}revoked`, `${TEST_ORG_PREFIX}expired`, `${TEST_ORG_PREFIX}future`]; + await pool.query( + `INSERT INTO organizations (workos_organization_id, name) + SELECT id, 'Deletion snapshot ' || id FROM unnest($1::text[]) AS id`, + [grantOrganizations], + ); + await pool.query( + `INSERT INTO organization_memberships (workos_user_id, workos_organization_id, email, role) + VALUES ($1, $4, 'jordan-revoke@pinnacle.example', 'owner'), + ($2, $4, 'sam-revoke@pinnacle.example', 'member'), + ($3, $4, 'maya-revoke@pinnacle.example', 'admin')`, [primary, secondary, bystander, TEST_ORG], + ); + await pool.query( + `UPDATE users SET primary_organization_id = $1 WHERE workos_user_id = ANY($2)`, + [TEST_ORG, [primary, secondary, bystander]], + ); + await pool.query( + `INSERT INTO working_group_memberships (working_group_id, workos_user_id, status) + SELECT id, $1, 'active' FROM working_groups WHERE slug = 'aao-admin' + UNION ALL SELECT id, $2, 'active' FROM working_groups WHERE slug = 'aao-admin' + UNION ALL SELECT id, $3, 'inactive' FROM working_groups WHERE slug = 'aao-admin'`, + [primary, bystander, secondary], + ); + await pool.query( + `INSERT INTO working_group_leaders (working_group_id, user_id) + SELECT id, $1 FROM working_groups WHERE slug = 'aao-admin' + UNION ALL SELECT id, $2 FROM working_groups WHERE slug = 'aao-admin'`, + [primary, bystander], + ); + const slackUserId = `${TEST_SLACK_PREFIX}primary`; + await pool.query( + `INSERT INTO slack_user_mappings ( + slack_user_id, slack_email, workos_user_id, mapping_status, mapping_source, mapped_at + ) VALUES ($1, 'jordan-revoke@pinnacle.example', $2, 'mapped', 'manual_admin', NOW())`, + [slackUserId, primary], + ); + await bumpAuthorizationEpochs(pool, [primary, secondary, bystander]); + await pool.query( + `INSERT INTO organization_credential_grants ( + workos_organization_id, workos_user_id, role, + granted_by_workos_user_id, reason, effective_from, effective_until, + revoked_at, revoked_by_workos_user_id + ) VALUES + ($1, $5, 'owner', 'approver_active', 'active deletion grant', NOW() - INTERVAL '1 day', NULL, NULL, NULL), + ($2, $5, 'admin', 'approver_revoked', 'revoked deletion grant', NOW() - INTERVAL '3 days', NULL, NOW() - INTERVAL '1 day', 'revoker_1'), + ($3, $5, 'member', 'approver_expired', 'expired deletion grant', NOW() - INTERVAL '3 days', NOW() - INTERVAL '1 day', NULL, NULL), + ($4, $5, 'admin', 'approver_future', 'future deletion grant', NOW() + INTERVAL '1 day', NOW() + INTERVAL '3 days', NULL, NULL)`, + [...grantOrganizations, primary], + ); + const aaoAdminGroup = await pool.query<{ id: string }>( + `SELECT id FROM working_groups WHERE slug = 'aao-admin'`, + ); + await pool.query( + `INSERT INTO aao_admin_access_events ( + event_type, actor_user_id, target_user_id, mechanism, + actor_authorization_mechanism, reason + ) VALUES ('granted', 'admin_seed', $1, 'aao_admin_working_group', + 'static_admin_api_key', 'seed active authority before provider deletion')`, + [primary], + ); + return { + primary, + secondary, + bystander, + identityId, + slackUserId, + aaoAdminGroupId: aaoAdminGroup.rows[0].id, + }; + } + + async function survivorAuthority(ids: string[]) { + const tables = [ + ['users', 'workos_user_id'], + ['identity_workos_users', 'workos_user_id'], + ['organization_memberships', 'workos_user_id'], + ['working_group_memberships', 'workos_user_id'], + ['working_group_leaders', 'user_id'], + ['working_group_topic_subscriptions', 'workos_user_id'], + ['slack_user_mappings', 'workos_user_id'], + ]; + const state: Record<string, unknown> = {}; + for (const [table, column] of tables) { + const result = await pool.query( + `SELECT row_to_json(row) AS row FROM (SELECT * FROM ${table} WHERE ${column} = ANY($1)) row + ORDER BY row_to_json(row)::text`, [ids], + ); + state[table] = result.rows.map(({ row }) => row); + } + return state; + } + + async function fullBeforeGraph(identityId: string, ids: string[]) { + const queries = { + identity: [`SELECT to_jsonb(i) AS row FROM identities i WHERE id = $1`, [identityId]], + identity_workos_users: [ + `SELECT to_jsonb(iwu) AS row FROM identity_workos_users iwu + WHERE iwu.identity_id = $1 ORDER BY iwu.workos_user_id`, + [identityId], + ], + users: [ + `SELECT to_jsonb(u) AS row FROM users u + WHERE u.workos_user_id = ANY($1) ORDER BY u.workos_user_id`, + [ids], + ], + organization_memberships: [ + `SELECT to_jsonb(om) AS row FROM organization_memberships om + WHERE om.workos_user_id = ANY($1) + ORDER BY om.workos_user_id, om.workos_organization_id`, + [ids], + ], + working_group_memberships: [ + `SELECT to_jsonb(wgm) AS row FROM working_group_memberships wgm + WHERE wgm.workos_user_id = ANY($1) + ORDER BY wgm.workos_user_id, wgm.working_group_id`, + [ids], + ], + working_group_leaders: [ + `SELECT to_jsonb(wgl) AS row FROM working_group_leaders wgl + WHERE wgl.user_id = ANY($1) ORDER BY wgl.user_id, wgl.working_group_id`, + [ids], + ], + working_group_topic_subscriptions: [ + `SELECT to_jsonb(wgts) AS row FROM working_group_topic_subscriptions wgts + WHERE wgts.workos_user_id = ANY($1) + ORDER BY wgts.workos_user_id, wgts.working_group_id`, + [ids], + ], + slack_user_mappings: [ + `SELECT to_jsonb(sm) AS row FROM slack_user_mappings sm + WHERE sm.workos_user_id = ANY($1) + ORDER BY sm.workos_user_id, sm.slack_user_id`, + [ids], + ], + authorization_epochs: [ + `SELECT to_jsonb(ae) AS row FROM authorization_epochs ae + WHERE ae.workos_user_id = ANY($1) ORDER BY ae.workos_user_id`, + [ids], + ], + } as const; + const entries = await Promise.all(Object.entries(queries).map(async ([key, [sql, params]]) => { + const query = await pool.query(sql, params); + return [key, key === 'identity' ? query.rows[0].row : query.rows.map(({ row }) => row)]; + })); + const snapshotInventory = async ( + specs: readonly { readonly table: string; readonly columns: readonly string[] }[], + ) => Object.fromEntries(await Promise.all(specs.map(async ({ table, columns }) => { + const predicates = columns.map((column) => `child."${column}" = ANY($1)`).join(' OR '); + const rows = await pool.query( + `SELECT to_jsonb(child) AS row FROM "${table}" child + WHERE ${predicates} ORDER BY to_jsonb(child)::text`, + [ids], + ); + return [table, rows.rows.map(({ row }) => row)]; + }))); + const baseGraph = Object.fromEntries(entries); + const cascadeRows = await snapshotInventory(USER_CASCADE_SNAPSHOT_INVENTORY); + cascadeRows.identity_workos_users = baseGraph.identity_workos_users; + cascadeRows.authorization_epochs = baseGraph.authorization_epochs; + return { + ...baseGraph, + user_cascade_rows: cascadeRows, + user_set_null_attributions: await snapshotInventory(USER_SET_NULL_SNAPSHOT_INVENTORY), + }; + } + + it('serializes webhook-vs-backfill deletion, revokes caches, and records the exact recovery graph once', async () => { + const { primary, secondary, bystander, identityId, slackUserId, aaoAdminGroupId } = await seedDeletionAuthority(); + const survivors = [secondary, bystander]; + const allCredentials = [primary, secondary, bystander].sort(); + const authorityBefore = await survivorAuthority(survivors); + const beforeGraph = await fullBeforeGraph(identityId, allCredentials); + const fingerprintsBefore = await Promise.all(survivors.map((id) => getAuthorizationFingerprint([id]))); + expect(await isSlackUserAAOAdmin(slackUserId)).toBe(true); + expect(getSlackAdminStatusCache().has(slackUserId)).toBe(false); + setUnifiedUsersCache(new Map([[TEST_ORG, [{ id: primary, email: 'jordan-revoke@pinnacle.example', firstName: 'Jordan', lastName: 'Test' }]]])); + expect(getUnifiedUsersCache()).not.toBeNull(); + // Alert transport is post-commit and best-effort. Its exact failure path + // must preserve durable quarantine, revocation, and cache invalidation. + vi.mocked(notifySystemError).mockImplementation(() => { + throw new Error('injected operator alert transport failure'); + }); - const result = await promoteSecondaryIfPrimaryDeleted(primary); + const results = await Promise.all([ + deleteIdentityCredential(primary, 'workos_webhook'), + deleteIdentityCredential(primary, 'sync_users_backfill'), + ]); - expect(result).not.toBeNull(); - expect(result!.promotedUserId).toBe(olderSecondary); + expect(results.filter(({ deleted }) => deleted)).toHaveLength(1); + const conflictingConcurrentReplay = results.find(({ replay }) => replay !== null); + expect(conflictingConcurrentReplay?.replay).toMatchObject({ + kind: 'different_source', + requested_deletion_source: expect.stringMatching(/^(workos_webhook|sync_users_backfill)$/), + original_deletion_source: expect.stringMatching(/^(workos_webhook|sync_users_backfill)$/), + }); + expect(conflictingConcurrentReplay?.replay?.requested_deletion_source) + .not.toBe(conflictingConcurrentReplay?.replay?.original_deletion_source); + expect(results.flatMap(({ affectedUserIds }) => affectedUserIds)).toEqual( + expect.arrayContaining([primary, secondary, bystander]), + ); + expect(getUnifiedUsersCache()).toBeNull(); + expect(await survivorAuthority(survivors)).toEqual(authorityBefore); + for (const [index, id] of survivors.entries()) { + expect(await getAuthorizationFingerprint([id])).not.toBe(fingerprintsBefore[index]); + } + expect((await pool.query(`SELECT 1 FROM users WHERE workos_user_id = $1`, [primary])).rows).toEqual([]); + expect((await pool.query(`SELECT 1 FROM organization_memberships WHERE workos_user_id = $1`, [primary])).rows).toEqual([]); + expect((await pool.query( + `SELECT workos_user_id FROM identity_workos_users WHERE identity_id = $1 AND is_primary = TRUE`, [identityId], + )).rows).toEqual([]); - // The deleted user is no longer primary; the older secondary is. - const primaries = await pool.query<{ workos_user_id: string; is_primary: boolean }>( - `SELECT workos_user_id, is_primary FROM identity_workos_users - WHERE identity_id = $1 - ORDER BY workos_user_id`, - [identityId] + const auditRows = await pool.query( + `SELECT id, action, details FROM registry_audit_log + WHERE workos_user_id = $1 + AND action IN ('identity_primary_deletion_quarantined', 'identity_credential_deleted') + ORDER BY created_at, id`, + [primary], ); - const byUser = Object.fromEntries(primaries.rows.map(r => [r.workos_user_id, r.is_primary])); - expect(byUser[primary]).toBe(false); - expect(byUser[olderSecondary]).toBe(true); - expect(byUser[newerSecondary]).toBe(false); + expect(auditRows.rows).toHaveLength(1); + expect(auditRows.rows[0].action).toBe('identity_primary_deletion_quarantined'); + const durable = await getIdentityRecoveryQuarantine(identityId); + expect(durable).toEqual({ + audit_id: auditRows.rows[0].id, + identity_id: identityId, + deleted_workos_user_id: primary, + deletion_source: expect.stringMatching(/^(workos_webhook|sync_users_backfill)$/), + actor: { + type: 'workos_provider', + source: expect.stringMatching(/^(workos_webhook|sync_users_backfill)$/), + workos_user_id: primary, + }, + recovery_state: IDENTITY_RECOVERY_STATE, + snapshot_policy: IDENTITY_DELETION_SNAPSHOT_POLICY, + before_graph: beforeGraph, + aao_admin_revocations: [{ + event_type: 'revoked', + target_user_id: primary, + working_group_id: aaoAdminGroupId, + mechanism: 'aao_admin_working_group', + actor: { + type: 'workos_provider', + source: expect.stringMatching(/^(workos_webhook|sync_users_backfill)$/), + workos_user_id: primary, + }, + evidence_ledger: 'registry_audit_log', + dedicated_ledger_limitation: 'provider_deletion_actor_not_representable', + }], + }); + expect(notifySystemError).toHaveBeenCalledTimes(1); + expect(notifySystemError).toHaveBeenCalledWith({ + source: `identity-primary-deletion-quarantine:${identityId}`, + errorMessage: [ + `Credential ${primary} was deleted without promotion.`, + `Identity ${identityId} is quarantined.`, + `recovery_state=${IDENTITY_RECOVERY_STATE}.`, + `audit_id=${auditRows.rows[0].id}.`, + ].join(' '), + }); + expect((await pool.query( + `SELECT status FROM working_group_memberships WHERE workos_user_id = $1`, + [primary], + )).rows).toEqual([{ status: 'inactive' }]); + expect((await pool.query( + `SELECT 1 FROM working_group_leaders WHERE user_id = $1`, + [primary], + )).rows).toEqual([]); + expect((await pool.query( + `SELECT workos_user_id, mapping_status FROM slack_user_mappings WHERE slack_user_id = $1`, + [slackUserId], + )).rows).toEqual([{ workos_user_id: null, mapping_status: 'unmapped' }]); + expect(await isSlackUserAAOAdmin(slackUserId)).toBe(false); + expect(beforeGraph.user_cascade_rows.organization_credential_grants).toHaveLength(4); + expect(beforeGraph.user_cascade_rows.organization_credential_grants.map((row: any) => ({ + role: row.role, + granted_by: row.granted_by_workos_user_id, + reason: row.reason, + effective_from: row.effective_from, + effective_until: row.effective_until, + revoked_at: row.revoked_at, + revoked_by: row.revoked_by_workos_user_id, + }))).toEqual(expect.arrayContaining([ + expect.objectContaining({ role: 'owner', granted_by: 'approver_active', reason: 'active deletion grant', revoked_at: null }), + expect.objectContaining({ role: 'admin', granted_by: 'approver_revoked', reason: 'revoked deletion grant', revoked_by: 'revoker_1' }), + expect.objectContaining({ role: 'member', granted_by: 'approver_expired', reason: 'expired deletion grant' }), + expect.objectContaining({ role: 'admin', granted_by: 'approver_future', reason: 'future deletion grant' }), + ])); + expect((await pool.query( + `SELECT event_type FROM aao_admin_access_events WHERE target_user_id = $1 ORDER BY created_at`, + [primary], + )).rows).toEqual([{ event_type: 'granted' }]); - // Exactly one primary on the identity — the partial unique index holds. - const primaryCount = primaries.rows.filter(r => r.is_primary).length; - expect(primaryCount).toBe(1); + const fingerprintAfter = await getAuthorizationFingerprint(survivors); + const allAuditRowsBeforeReplay = (await pool.query( + `SELECT * FROM registry_audit_log WHERE workos_user_id = $1 ORDER BY created_at, id`, + [primary], + )).rows; + const epochRowsBeforeReplay = (await pool.query( + `SELECT * FROM authorization_epochs WHERE workos_user_id = ANY($1) ORDER BY workos_user_id`, + [allCredentials], + )).rows; + const replayCache = new Map([[TEST_ORG, [{ + id: primary, + email: 'must-remain-cached@test.example', + firstName: 'Replay', + lastName: 'Sentinel', + }]]]); + setUnifiedUsersCache(replayCache); + const originalSource = durable!.deletion_source; + const sameSourceReplay = await deleteIdentityCredential(primary, originalSource); + expect(sameSourceReplay).toMatchObject({ + deleted: false, + affectedUserIds: allCredentials, + affectedSlackUserIds: [slackUserId], + quarantine: durable, + replay: { + audit_id: auditRows.rows[0].id, + kind: 'same_source', + original_deletion_source: originalSource, + requested_deletion_source: originalSource, + }, + }); + const conflictingSource = originalSource === 'workos_webhook' + ? 'sync_users_backfill' + : 'workos_webhook'; + const conflictingReplay = await deleteIdentityCredential(primary, conflictingSource); + expect(conflictingReplay).toMatchObject({ + deleted: false, + affectedUserIds: allCredentials, + affectedSlackUserIds: [slackUserId], + quarantine: durable, + replay: { + audit_id: auditRows.rows[0].id, + kind: 'different_source', + original_deletion_source: originalSource, + requested_deletion_source: conflictingSource, + }, + }); + expect(await getAuthorizationFingerprint(survivors)).toBe(fingerprintAfter); + expect(await survivorAuthority(survivors)).toEqual(authorityBefore); + expect((await pool.query( + `SELECT * FROM registry_audit_log WHERE workos_user_id = $1 ORDER BY created_at, id`, + [primary], + )).rows).toEqual(allAuditRowsBeforeReplay); + expect((await pool.query( + `SELECT * FROM authorization_epochs WHERE workos_user_id = ANY($1) ORDER BY workos_user_id`, + [allCredentials], + )).rows).toEqual(epochRowsBeforeReplay); + expect(getUnifiedUsersCache()).toBe(replayCache); + expect(notifySystemError).toHaveBeenCalledTimes(1); }); - it('returns null and leaves state untouched when the identity has no secondaries', async () => { - const onlyUser = await insertUser('solo', 'solo@test.example'); + it('deletes a non-primary credential without quarantining or changing the primary', async () => { + const { primary, secondary, identityId } = await seedDeletionAuthority(); - const result = await promoteSecondaryIfPrimaryDeleted(onlyUser); + const deletion = await deleteIdentityCredential(secondary, 'workos_webhook'); - expect(result).toBeNull(); + expect(deletion).toMatchObject({ deleted: true, quarantine: null }); + expect(await getIdentityRecoveryQuarantine(identityId)).toBeNull(); + expect((await pool.query( + `SELECT workos_user_id FROM identity_workos_users + WHERE identity_id = $1 AND is_primary = TRUE`, + [identityId], + )).rows).toEqual([{ workos_user_id: primary }]); + const auditRowsAfterDeletion = (await pool.query( + `SELECT * FROM registry_audit_log WHERE workos_user_id = $1 ORDER BY created_at, id`, + [secondary], + )).rows; + const epochRowsAfterDeletion = (await pool.query( + `SELECT * FROM authorization_epochs + WHERE workos_user_id = ANY($1) ORDER BY workos_user_id`, + [[primary, secondary]], + )).rows; + expect(auditRowsAfterDeletion).toHaveLength(1); + const replay = await deleteIdentityCredential(secondary, 'workos_webhook'); + expect(replay).toMatchObject({ + deleted: false, + quarantine: null, + replay: { + audit_id: auditRowsAfterDeletion[0].id, + kind: 'same_source', + original_deletion_source: 'workos_webhook', + requested_deletion_source: 'workos_webhook', + }, + }); + expect((await pool.query( + `SELECT * FROM registry_audit_log WHERE workos_user_id = $1 ORDER BY created_at, id`, + [secondary], + )).rows).toEqual(auditRowsAfterDeletion); + expect((await pool.query( + `SELECT * FROM authorization_epochs + WHERE workos_user_id = ANY($1) ORDER BY workos_user_id`, + [[primary, secondary]], + )).rows).toEqual(epochRowsAfterDeletion); + expect(notifySystemError).not.toHaveBeenCalled(); + }); - // The original primary binding is unchanged. - const row = await pool.query<{ is_primary: boolean }>( - `SELECT is_primary FROM identity_workos_users WHERE workos_user_id = $1`, - [onlyUser] + it('snapshots and revokes Slack-keyed authority and topic subscriptions for the deleted credential', async () => { + const { primary, identityId, slackUserId, aaoAdminGroupId } = await seedDeletionAuthority(); + await pool.query( + `INSERT INTO working_group_memberships (working_group_id, workos_user_id, status) + VALUES ($1, $2, 'active')`, + [aaoAdminGroupId, slackUserId], + ); + await pool.query( + `INSERT INTO working_group_leaders (working_group_id, user_id) + VALUES ($1, $2)`, + [aaoAdminGroupId, slackUserId], + ); + await pool.query( + `INSERT INTO working_group_topic_subscriptions (working_group_id, workos_user_id, topic_slugs) + VALUES ($1, $2, ARRAY['deleted-topic'])`, + [aaoAdminGroupId, slackUserId], ); - expect(row.rows[0].is_primary).toBe(true); + + await deleteIdentityCredential(primary, 'workos_webhook'); + + expect((await pool.query( + `SELECT status FROM working_group_memberships + WHERE working_group_id = $1 AND workos_user_id = $2`, + [aaoAdminGroupId, slackUserId], + )).rows).toEqual([{ status: 'inactive' }]); + expect((await pool.query( + `SELECT 1 FROM working_group_leaders WHERE working_group_id = $1 AND user_id = $2`, + [aaoAdminGroupId, slackUserId], + )).rows).toEqual([]); + expect((await pool.query( + `SELECT 1 FROM working_group_topic_subscriptions + WHERE working_group_id = $1 AND workos_user_id = $2`, + [aaoAdminGroupId, slackUserId], + )).rows).toEqual([]); + expect(await isSlackUserAAOAdmin(slackUserId)).toBe(false); + + const quarantine = await getIdentityRecoveryQuarantine(identityId); + expect(quarantine?.before_graph.working_group_memberships).toEqual( + expect.arrayContaining([expect.objectContaining({ + workos_user_id: slackUserId, + working_group_id: aaoAdminGroupId, + status: 'active', + })]), + ); + expect(quarantine?.before_graph.working_group_leaders).toEqual( + expect.arrayContaining([expect.objectContaining({ user_id: slackUserId })]), + ); + expect(quarantine?.before_graph.working_group_topic_subscriptions).toEqual( + expect.arrayContaining([expect.objectContaining({ + workos_user_id: slackUserId, + topic_slugs: ['deleted-topic'], + })]), + ); + expect(quarantine?.aao_admin_revocations.filter( + (row) => row.target_user_id === slackUserId, + )).toHaveLength(1); }); - it('returns null when the deleted user is not a primary binding', async () => { - const primary = await insertUser('np_p', 'primary2@test.example'); - const secondary = await insertUser('np_s', 'secondary2@test.example'); - await bindAsSecondary(primary, secondary, 1); + it('quarantines a corrupt zero-primary identity when deleting a secondary leaves survivors', async () => { + const { primary, secondary, bystander, identityId } = await seedDeletionAuthority(); + const credentials = [primary, secondary, bystander].sort(); + await pool.query( + `UPDATE identity_workos_users SET is_primary = FALSE WHERE identity_id = $1`, + [identityId], + ); + await expect(readCredentialAuthorizationLifecycle(secondary)).resolves.toEqual({ + status: 'terminal', + reason: 'missing_primary', + }); + const beforeGraph = await fullBeforeGraph(identityId, credentials); - // Delete the secondary (not the primary). No promotion is needed. - const result = await promoteSecondaryIfPrimaryDeleted(secondary); + const deletion = await deleteIdentityCredential(secondary, 'workos_webhook'); + expect(deletion.deleted).toBe(true); + const auditRows = await pool.query<{ id: string }>( + `SELECT id FROM registry_audit_log + WHERE action = 'identity_primary_deletion_quarantined' AND resource_id = $1`, + [identityId], + ); + expect(auditRows.rows).toHaveLength(1); + expect(await getIdentityRecoveryQuarantine(identityId)).toEqual({ + audit_id: auditRows.rows[0].id, + identity_id: identityId, + deleted_workos_user_id: secondary, + deletion_source: 'workos_webhook', + actor: { type: 'workos_provider', source: 'workos_webhook', workos_user_id: secondary }, + recovery_state: IDENTITY_RECOVERY_STATE, + snapshot_policy: IDENTITY_DELETION_SNAPSHOT_POLICY, + before_graph: beforeGraph, + aao_admin_revocations: [], + }); + expect(notifySystemError).toHaveBeenCalledWith({ + source: `identity-primary-deletion-quarantine:${identityId}`, + errorMessage: [ + `Credential ${secondary} was deleted without promotion.`, + `Identity ${identityId} is quarantined.`, + `recovery_state=${IDENTITY_RECOVERY_STATE}.`, + `audit_id=${auditRows.rows[0].id}.`, + ].join(' '), + }); + expect((await pool.query( + `SELECT workos_user_id FROM identity_workos_users + WHERE identity_id = $1 AND is_primary = TRUE`, + [identityId], + )).rows).toEqual([]); + }); - expect(result).toBeNull(); + it.each(['quarantine-primary', 'non-primary', 'singleton', 'unbound'] as const)( + 'rolls back every side effect when DELETE RETURNING yields no row on the %s path', + async (path) => { + let target: string; + let credentials: string[]; + if (path === 'quarantine-primary' || path === 'non-primary') { + const seeded = await seedDeletionAuthority(); + target = path === 'quarantine-primary' ? seeded.primary : seeded.secondary; + credentials = [seeded.primary, seeded.secondary, seeded.bystander]; + } else { + target = await insertUser(`return_null_${path}`, `${path}@return-null.test`); + credentials = [target]; + await pool.query( + `INSERT INTO organizations (workos_organization_id, name) + VALUES ($1, 'RETURN NULL test')`, + [TEST_ORG], + ); + await pool.query( + `INSERT INTO organization_memberships ( + workos_user_id, workos_organization_id, email, role + ) VALUES ($1, $2, $3, 'admin')`, + [target, TEST_ORG, `${path}@return-null.test`], + ); + if (path === 'unbound') { + const binding = await pool.query<{ identity_id: string }>( + `DELETE FROM identity_workos_users WHERE workos_user_id = $1 RETURNING identity_id`, + [target], + ); + await pool.query(`DELETE FROM identities WHERE id = $1`, [binding.rows[0].identity_id]); + } + await bumpAuthorizationEpochs(pool, [target]); + } + const slackUserId = `${TEST_SLACK_PREFIX}return_null_${path}`; + await pool.query( + `INSERT INTO slack_user_mappings ( + slack_user_id, slack_email, workos_user_id, mapping_status, mapping_source, mapped_at + ) VALUES ($1, $2, $3, 'mapped', 'manual_admin', NOW()) + ON CONFLICT (slack_user_id) DO UPDATE + SET workos_user_id = EXCLUDED.workos_user_id, + mapping_status = 'mapped', mapping_source = 'manual_admin', mapped_at = NOW()`, + [slackUserId, `${path}@return-null.test`, target], + ); + const before = await survivorAuthority(credentials); + const fingerprintBefore = await getAuthorizationFingerprint(credentials); + await pool.query(` + CREATE FUNCTION suppress_confirmed_user_delete() RETURNS trigger AS $$ + BEGIN + IF OLD.workos_user_id = '${target}' THEN + RETURN NULL; + END IF; + RETURN OLD; + END; + $$ LANGUAGE plpgsql; + CREATE TRIGGER suppress_confirmed_user_delete + BEFORE DELETE ON users + FOR EACH ROW EXECUTE FUNCTION suppress_confirmed_user_delete(); + `); + try { + await expect(deleteIdentityCredential(target, 'workos_webhook')) + .rejects.toThrow('Confirmed credential deletion did not delete exactly one locked user'); + expect(await survivorAuthority(credentials)).toEqual(before); + expect(await getAuthorizationFingerprint(credentials)).toBe(fingerprintBefore); + expect((await pool.query( + `SELECT 1 FROM registry_audit_log + WHERE workos_user_id = $1 + AND action IN ('identity_credential_deleted', 'identity_primary_deletion_quarantined')`, + [target], + )).rows).toEqual([]); + expect(notifySystemError).not.toHaveBeenCalled(); + } finally { + await pool.query(`DROP TRIGGER suppress_confirmed_user_delete ON users`); + await pool.query(`DROP FUNCTION suppress_confirmed_user_delete()`); + } + }, + ); - const primaryRow = await pool.query<{ is_primary: boolean }>( - `SELECT is_primary FROM identity_workos_users WHERE workos_user_id = $1`, - [primary] + it('rolls back epoch changes and deletion together when the database refuses deletion', async () => { + const { primary, secondary, bystander } = await seedDeletionAuthority(); + const credentials = [primary, secondary, bystander]; + const before = await survivorAuthority(credentials); + const fingerprintBefore = await getAuthorizationFingerprint(credentials); + // A test-only referencing row forces a failure after the epoch bump. + await pool.query( + `CREATE TABLE identity_containment_delete_guard ( + workos_user_id TEXT REFERENCES users(workos_user_id) ON DELETE RESTRICT + )`, ); - expect(primaryRow.rows[0].is_primary).toBe(true); + try { + await pool.query(`INSERT INTO identity_containment_delete_guard VALUES ($1)`, [primary]); + await expect(deleteIdentityCredential(primary, 'workos_webhook')).rejects.toThrow(); + expect(await survivorAuthority(credentials)).toEqual(before); + expect(await getAuthorizationFingerprint(credentials)).toBe(fingerprintBefore); + expect(await getIdentityRecoveryQuarantine((await pool.query<{ identity_id: string }>( + `SELECT identity_id FROM identity_workos_users WHERE workos_user_id = $1`, + [primary], + )).rows[0].identity_id)).toBeNull(); + expect(notifySystemError).not.toHaveBeenCalled(); + } finally { + await pool.query(`DROP TABLE identity_containment_delete_guard`); + } }); - it('CASCADE on the deleted user removes the binding after promotion', async () => { - const primary = await insertUser('cascade_p', 'cascadeprimary@test.example'); - const secondary = await insertUser('cascade_s', 'cascadesecondary@test.example'); - const identityId = await bindAsSecondary(primary, secondary, 1); + it('rolls back deletion and epochs when the durable quarantine audit write fails', async () => { + const { primary, secondary, bystander, identityId } = await seedDeletionAuthority(); + const credentials = [primary, secondary, bystander]; + const before = await survivorAuthority(credentials); + const fingerprintBefore = await getAuthorizationFingerprint(credentials); + await pool.query(` + CREATE OR REPLACE FUNCTION reject_identity_quarantine_audit() RETURNS trigger AS $$ + BEGIN + IF NEW.workos_user_id = '${primary}' + AND NEW.action = 'identity_primary_deletion_quarantined' THEN + RAISE EXCEPTION 'injected quarantine audit failure'; + END IF; + RETURN NEW; + END; + $$ LANGUAGE plpgsql; + CREATE TRIGGER reject_identity_quarantine_audit + BEFORE INSERT ON registry_audit_log + FOR EACH ROW EXECUTE FUNCTION reject_identity_quarantine_audit(); + `); + try { + await expect(deleteIdentityCredential(primary, 'sync_users_backfill')) + .rejects.toThrow('injected quarantine audit failure'); + expect(await survivorAuthority(credentials)).toEqual(before); + expect(await getAuthorizationFingerprint(credentials)).toBe(fingerprintBefore); + expect(await getIdentityRecoveryQuarantine(identityId)).toBeNull(); + expect(notifySystemError).not.toHaveBeenCalled(); + } finally { + await pool.query(`DROP TRIGGER reject_identity_quarantine_audit ON registry_audit_log`); + await pool.query(`DROP FUNCTION reject_identity_quarantine_audit()`); + } + }); - const result = await promoteSecondaryIfPrimaryDeleted(primary); - expect(result?.promotedUserId).toBe(secondary); + it.each(['insert', 'update'] as const)( + 'rolls back deletion when an authorization epoch %s trigger suppresses rows', + async (operation) => { + const { primary, secondary, bystander, identityId } = await seedDeletionAuthority(); + const credentials = [primary, secondary, bystander]; + if (operation === 'insert') { + await pool.query( + `DELETE FROM authorization_epochs WHERE workos_user_id = ANY($1)`, + [credentials], + ); + } + const before = await survivorAuthority(credentials); + const fingerprintBefore = await getAuthorizationFingerprint(credentials); + const triggerOperation = operation.toUpperCase(); + await pool.query(` + CREATE FUNCTION suppress_authorization_epoch_${operation}() RETURNS trigger AS $$ + BEGIN + IF NEW.workos_user_id = '${primary}' THEN + RETURN NULL; + END IF; + RETURN NEW; + END; + $$ LANGUAGE plpgsql; + CREATE TRIGGER suppress_authorization_epoch_${operation} + BEFORE ${triggerOperation} ON authorization_epochs + FOR EACH ROW EXECUTE FUNCTION suppress_authorization_epoch_${operation}(); + `); + try { + await expect(deleteIdentityCredential(primary, 'workos_webhook')) + .rejects.toThrow('did not bump every live authorization epoch'); + expect(await survivorAuthority(credentials)).toEqual(before); + expect(await getAuthorizationFingerprint(credentials)).toBe(fingerprintBefore); + expect(await getIdentityRecoveryQuarantine(identityId)).toBeNull(); + expect((await pool.query( + `SELECT 1 FROM registry_audit_log + WHERE workos_user_id = $1 + AND action IN ('identity_credential_deleted', 'identity_primary_deletion_quarantined')`, + [primary], + )).rows).toEqual([]); + expect(notifySystemError).not.toHaveBeenCalled(); + } finally { + await pool.query( + `DROP TRIGGER suppress_authorization_epoch_${operation} ON authorization_epochs`, + ); + await pool.query(`DROP FUNCTION suppress_authorization_epoch_${operation}()`); + } + }, + ); - // The webhook handler runs deleteUser() next, which CASCADE-drops the - // (now non-primary) binding. Surviving primary remains. - await pool.query(`DELETE FROM users WHERE workos_user_id = $1`, [primary]); + it('rolls back deletion when the deletion-audit INSERT returns no row', async () => { + const { primary, secondary, bystander, identityId } = await seedDeletionAuthority(); + const credentials = [primary, secondary, bystander]; + const before = await survivorAuthority(credentials); + const fingerprintBefore = await getAuthorizationFingerprint(credentials); + await pool.query(` + CREATE FUNCTION suppress_identity_deletion_audit() RETURNS trigger AS $$ + BEGIN + IF NEW.workos_user_id = '${primary}' + AND NEW.action IN ('identity_credential_deleted', 'identity_primary_deletion_quarantined') THEN + RETURN NULL; + END IF; + RETURN NEW; + END; + $$ LANGUAGE plpgsql; + CREATE TRIGGER suppress_identity_deletion_audit + BEFORE INSERT ON registry_audit_log + FOR EACH ROW EXECUTE FUNCTION suppress_identity_deletion_audit(); + `); + try { + await expect(deleteIdentityCredential(primary, 'workos_webhook')) + .rejects.toThrow('audit was not durably recorded'); + expect(await survivorAuthority(credentials)).toEqual(before); + expect(await getAuthorizationFingerprint(credentials)).toBe(fingerprintBefore); + expect(await getIdentityRecoveryQuarantine(identityId)).toBeNull(); + expect((await pool.query( + `SELECT 1 FROM registry_audit_log + WHERE workos_user_id = $1 + AND action IN ('identity_credential_deleted', 'identity_primary_deletion_quarantined')`, + [primary], + )).rows).toEqual([]); + expect(notifySystemError).not.toHaveBeenCalled(); + } finally { + await pool.query(`DROP TRIGGER suppress_identity_deletion_audit ON registry_audit_log`); + await pool.query(`DROP FUNCTION suppress_identity_deletion_audit()`); + } + }); - const survivors = await pool.query<{ workos_user_id: string; is_primary: boolean }>( - `SELECT workos_user_id, is_primary FROM identity_workos_users WHERE identity_id = $1`, - [identityId] + it('retains row provenance without copying binary PII or email-link bearer secrets', async () => { + const userId = await insertUser('binary_snapshot', 'binary-snapshot@test.example'); + const rawLinkToken = 'raw-email-link-token-must-not-enter-durable-audit'; + await pool.query( + `INSERT INTO user_avatar_uploads (workos_user_id, image_data, content_type) + VALUES ($1, $2, 'image/png')`, + [userId, Buffer.alloc(64, 0xab)], + ); + await pool.query( + `INSERT INTO member_portraits (user_id, image_url, portrait_data, palette, status) + VALUES ($1, 'https://example.test/portrait.png', $2, 'amber', 'approved')`, + [userId, Buffer.alloc(96, 0xcd)], + ); + await pool.query( + `INSERT INTO email_link_tokens ( + token, primary_workos_user_id, target_email, status, expires_at + ) VALUES ($1, $2, 'linked-target@test.example', 'pending', NOW() + INTERVAL '1 hour')`, + [rawLinkToken, userId], + ); + + await expect(deleteIdentityCredential(userId, 'workos_webhook')) + .resolves.toMatchObject({ deleted: true }); + const audit = await pool.query<{ details: any }>( + `SELECT details FROM registry_audit_log + WHERE workos_user_id = $1 AND action = 'identity_credential_deleted'`, + [userId], + ); + expect(audit.rows).toHaveLength(1); + const avatar = audit.rows[0].details.before_graph.user_cascade_rows.user_avatar_uploads[0]; + expect(avatar).toMatchObject({ + workos_user_id: userId, + content_type: 'image/png', + image_data_redacted: true, + image_data_byte_length: 64, + }); + expect(avatar).not.toHaveProperty('image_data'); + const portrait = audit.rows[0].details.before_graph.user_cascade_rows.member_portraits[0]; + expect(portrait).toMatchObject({ + user_id: userId, + image_url: 'https://example.test/portrait.png', + portrait_data_redacted: true, + portrait_data_byte_length: 96, + }); + expect(portrait).not.toHaveProperty('portrait_data'); + const emailLinkToken = + audit.rows[0].details.before_graph.user_cascade_rows.email_link_tokens[0]; + expect(emailLinkToken).toMatchObject({ + primary_workos_user_id: userId, + target_email: 'linked-target@test.example', + status: 'pending', + token_redacted: true, + }); + expect(emailLinkToken).not.toHaveProperty('token'); + expect(JSON.stringify(audit.rows[0].details)).not.toContain(rawLinkToken); + expect(audit.rows[0].details.snapshot_policy).toEqual(IDENTITY_DELETION_SNAPSHOT_POLICY); + expect(Buffer.byteLength(JSON.stringify(audit.rows[0].details), 'utf8')) + .toBeLessThanOrEqual(IDENTITY_DELETION_SNAPSHOT_MAX_DETAILS_BYTES); + }); + + it('fails closed before deletion when the complete durable snapshot exceeds its byte bound', async () => { + const userId = await insertUser('snapshot_bound', 'snapshot-bound@test.example'); + await pool.query( + `INSERT INTO email_link_tokens ( + token, primary_workos_user_id, target_email, status, expires_at, merge_summary + ) VALUES ( + 'snapshot-bound-token', $1, 'bounded-target@test.example', 'pending', + NOW() + INTERVAL '1 hour', jsonb_build_object('payload', repeat('x', $2)) + )`, + [userId, IDENTITY_DELETION_SNAPSHOT_MAX_DETAILS_BYTES], ); - expect(survivors.rows).toHaveLength(1); - expect(survivors.rows[0].workos_user_id).toBe(secondary); - expect(survivors.rows[0].is_primary).toBe(true); + + await expect(deleteIdentityCredential(userId, 'workos_webhook')) + .rejects.toThrow('snapshot exceeds the fail-closed byte limit'); + expect((await pool.query( + `SELECT 1 FROM users WHERE workos_user_id = $1`, + [userId], + )).rows).toHaveLength(1); + expect((await pool.query( + `SELECT 1 FROM email_link_tokens WHERE primary_workos_user_id = $1`, + [userId], + )).rows).toHaveLength(1); + expect((await pool.query( + `SELECT 1 FROM registry_audit_log WHERE workos_user_id = $1`, + [userId], + )).rows).toEqual([]); }); + it('rolls back deletion, audit, and authority when an AAO-admin deactivation returns no row', async () => { + const { primary, secondary, bystander, identityId } = await seedDeletionAuthority(); + const credentials = [primary, secondary, bystander]; + const before = await survivorAuthority(credentials); + const fingerprintBefore = await getAuthorizationFingerprint(credentials); + await pool.query(` + CREATE FUNCTION suppress_aao_admin_deactivation() RETURNS trigger AS $$ + BEGIN + IF OLD.workos_user_id = '${primary}' AND NEW.status = 'inactive' THEN + RETURN NULL; + END IF; + RETURN NEW; + END; + $$ LANGUAGE plpgsql; + CREATE TRIGGER suppress_aao_admin_deactivation + BEFORE UPDATE ON working_group_memberships + FOR EACH ROW EXECUTE FUNCTION suppress_aao_admin_deactivation(); + `); + try { + await expect(deleteIdentityCredential(primary, 'workos_webhook')) + .rejects.toThrow('did not deactivate every working-group membership'); + expect(await survivorAuthority(credentials)).toEqual(before); + expect(await getAuthorizationFingerprint(credentials)).toBe(fingerprintBefore); + expect(await getIdentityRecoveryQuarantine(identityId)).toBeNull(); + expect((await pool.query( + `SELECT event_type FROM aao_admin_access_events WHERE target_user_id = $1`, [primary], + )).rows).toEqual([{ event_type: 'granted' }]); + expect(notifySystemError).not.toHaveBeenCalled(); + } finally { + await pool.query(`DROP TRIGGER suppress_aao_admin_deactivation ON working_group_memberships`); + await pool.query(`DROP FUNCTION suppress_aao_admin_deactivation()`); + } + }); it('deletes an account with accumulated community data', async () => { const userId = await insertUser('community', 'community@test.example'); await pool.query( @@ -205,4 +1071,32 @@ describe('user.deleted: promote secondary before CASCADE (#3718)', () => { expect(restrictive.rows).toEqual([]); }); + + it('keeps the durable snapshot inventory synchronized with every users FK deletion action', async () => { + const readInventory = async (deleteType: 'c' | 'n') => { + const rows = await pool.query<{ table_name: string; column_name: string }>( + `SELECT con.conrelid::regclass::text AS table_name, attr.attname AS column_name + FROM pg_constraint con + JOIN LATERAL unnest(con.conkey) AS key(attnum) ON TRUE + JOIN pg_attribute attr + ON attr.attrelid = con.conrelid AND attr.attnum = key.attnum + WHERE con.contype = 'f' + AND con.confrelid = 'users'::regclass + AND con.confdeltype = $1 + ORDER BY table_name, column_name`, + [deleteType], + ); + const grouped = new Map<string, string[]>(); + for (const row of rows.rows) { + grouped.set(row.table_name, [...(grouped.get(row.table_name) ?? []), row.column_name]); + } + return [...grouped].map(([table, columns]) => ({ table, columns: columns.sort() })); + }; + const normalize = (specs: readonly { readonly table: string; readonly columns: readonly string[] }[]) => + specs.map(({ table, columns }) => ({ table, columns: [...columns].sort() })) + .sort((a, b) => a.table.localeCompare(b.table)); + + expect(await readInventory('c')).toEqual(normalize(USER_CASCADE_SNAPSHOT_INVENTORY)); + expect(await readInventory('n')).toEqual(normalize(USER_SET_NULL_SNAPSHOT_INVENTORY)); + }); }); diff --git a/server/tests/integration/workos-user-deletion-race.test.ts b/server/tests/integration/workos-user-deletion-race.test.ts new file mode 100644 index 00000000000..823b2da15ea --- /dev/null +++ b/server/tests/integration/workos-user-deletion-race.test.ts @@ -0,0 +1,1028 @@ +import express from 'express'; +import request from 'supertest'; +import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest'; +import type { Pool } from 'pg'; + +const mocks = vi.hoisted(() => ({ + constructEvent: vi.fn(), + getUser: vi.fn(), + listUsers: vi.fn(), + invalidateSessionsForUsers: vi.fn(), + notifySystemError: vi.fn(), + tryAutoLinkWebsiteUserToSlack: vi.fn(), + coordinateDeletionCallers: false, + deletionEntrants: [] as Array<{ workosUserId: string; source: string }>, + deletionWaiters: [] as Array<() => void>, +})); + +vi.mock('../../src/auth/workos-client.js', async (importOriginal) => ({ + ...(await importOriginal<typeof import('../../src/auth/workos-client.js')>()), + getWorkos: () => ({ + webhooks: { constructEvent: mocks.constructEvent }, + userManagement: { + getUser: mocks.getUser, + listUsers: mocks.listUsers, + }, + }), +})); + +vi.mock('../../src/middleware/auth.js', async (importOriginal) => ({ + ...(await importOriginal<typeof import('../../src/middleware/auth.js')>()), + invalidateSessionsForUsers: mocks.invalidateSessionsForUsers, +})); + +vi.mock('../../src/addie/error-notifier.js', async (importOriginal) => ({ + ...(await importOriginal<typeof import('../../src/addie/error-notifier.js')>()), + notifySystemError: mocks.notifySystemError, +})); + +vi.mock('../../src/slack/sync.js', async (importOriginal) => ({ + ...(await importOriginal<typeof import('../../src/slack/sync.js')>()), + tryAutoLinkWebsiteUserToSlack: mocks.tryAutoLinkWebsiteUserToSlack, +})); + +vi.mock('../../src/services/identity-credential-deletion.js', async (importOriginal) => { + const actual = await importOriginal<typeof import('../../src/services/identity-credential-deletion.js')>(); + return { + ...actual, + deleteIdentityCredential: async ( + workosUserId: string, + source: 'workos_webhook' | 'sync_users_backfill', + ) => { + if (mocks.coordinateDeletionCallers && workosUserId === 'user_wh_backfill_race_primary') { + mocks.deletionEntrants.push({ workosUserId, source }); + await new Promise<void>((resolve) => { + mocks.deletionWaiters.push(resolve); + if (mocks.deletionWaiters.length === 2) { + for (const release of mocks.deletionWaiters.splice(0)) release(); + } + }); + } + return actual.deleteIdentityCredential(workosUserId, source); + }, + }; +}); + +process.env.WORKOS_WEBHOOK_SECRET = 'test_workos_webhook_secret'; +process.env.WORKOS_PROVIDER_PREFETCH_TIMEOUT_MS = '100'; + +const { initializeDatabase, closeDatabase } = await import('../../src/db/client.js'); +const { runMigrations } = await import('../../src/db/migrate.js'); +const { getAuthorizationFingerprint } = await import('../../src/db/authorization-epoch-db.js'); +const { + getIdentityRecoveryQuarantine, + IDENTITY_DELETION_SNAPSHOT_POLICY, + IDENTITY_RECOVERY_STATE, + withActiveCredentialEventMutation, + withCredentialCreationEventMutation, + upsertWorkosUserInCredentialEvent, +} = await import('../../src/db/identity-db.js'); +const { + createWorkOSWebhooksRouter, + backfillUsers, + shouldAbortBackfillDeletion, +} = await import('../../src/routes/workos-webhooks.js'); +const { getUnifiedUsersCache, setUnifiedUsersCache } = await import('../../src/cache/unified-users.js'); +const { isSlackUserAAOAdmin } = await import('../../src/addie/mcp/admin-tools.js'); +const { deleteIdentityCredential } = await import('../../src/services/identity-credential-deletion.js'); +const { SlackDatabase } = await import('../../src/db/slack-db.js'); +const { WorkingGroupDatabase } = await import('../../src/db/working-group-db.js'); +const { MeetingsDatabase } = await import('../../src/db/meetings-db.js'); +const { + setMembershipRole, + upsertOrganizationMembership, +} = await import('../../src/db/membership-db.js'); + +const PREFIX = 'user_wh_backfill_race_'; +const PRIMARY = `${PREFIX}primary`; +const SECONDARY = `${PREFIX}secondary`; +const BYSTANDER = `${PREFIX}bystander`; +const ORG = 'org_wh_backfill_race_pinnacle'; +const SLACK = 'slack_wh_backfill_race_primary'; +const CHAPTER_SLUG = 'wh-backfill-race-chapter'; + +describe('WorkOS webhook vs sync-users deletion', () => { + let pool: Pool; + + beforeAll(async () => { + pool = initializeDatabase({ + connectionString: + process.env.DATABASE_URL || 'postgresql://adcp:localdev@localhost:5432/adcp_test', + }); + await runMigrations(); + }, 60_000); + + afterAll(async () => { + await cleanup(); + await closeDatabase(); + }); + + beforeEach(async () => { + await cleanup(); + vi.clearAllMocks(); + mocks.deletionEntrants.length = 0; + mocks.deletionWaiters.length = 0; + mocks.coordinateDeletionCallers = false; + mocks.tryAutoLinkWebsiteUserToSlack.mockResolvedValue({ linked: false, reason: 'no_slack_user' }); + }); + + async function cleanup() { + await pool.query(`DROP TRIGGER IF EXISTS delay_racing_user_delete ON users`); + await pool.query(`DROP FUNCTION IF EXISTS delay_racing_user_delete()`); + await pool.query( + `DELETE FROM registry_audit_log + WHERE action IN ('identity_primary_deletion_quarantined', 'identity_credential_deleted') + AND workos_user_id LIKE $1`, + [`${PREFIX}%`], + ); + await pool.query( + `DELETE FROM working_group_memberships WHERE workos_user_id LIKE $1 OR workos_user_id = $2`, + [`${PREFIX}%`, SLACK], + ); + await pool.query( + `DELETE FROM working_group_leaders WHERE user_id LIKE $1 OR user_id = $2`, + [`${PREFIX}%`, SLACK], + ); + await pool.query( + `DELETE FROM working_group_topic_subscriptions + WHERE workos_user_id LIKE $1 OR workos_user_id = $2`, + [`${PREFIX}%`, SLACK], + ); + await pool.query(`DELETE FROM slack_user_mappings WHERE slack_user_id = $1`, [SLACK]); + await pool.query(`DELETE FROM invitation_seat_types WHERE workos_invitation_id LIKE 'invite_wh_%'`); + await pool.query(`DELETE FROM organization_memberships WHERE workos_user_id LIKE $1`, [`${PREFIX}%`]); + await pool.query(`DELETE FROM users WHERE workos_user_id LIKE $1`, [`${PREFIX}%`]); + await pool.query(`DELETE FROM working_groups WHERE slug = $1`, [CHAPTER_SLUG]); + await pool.query(`DELETE FROM organizations WHERE workos_organization_id = $1`, [ORG]); + } + + async function seedIdentity() { + await pool.query( + `INSERT INTO organizations (workos_organization_id, name) VALUES ($1, 'Pinnacle Agency')`, + [ORG], + ); + for (const [id, email] of [ + [PRIMARY, 'jordan-race@pinnacle.example'], + [SECONDARY, 'sam-race@pinnacle.example'], + [BYSTANDER, 'maya-race@pinnacle.example'], + ]) { + await pool.query( + `INSERT INTO users ( + workos_user_id, email, first_name, last_name, email_verified, + workos_created_at, workos_updated_at, created_at, updated_at, + primary_organization_id + ) VALUES ($1, $2, 'Test', 'User', TRUE, NOW(), NOW(), NOW(), NOW(), $3)`, + [id, email, ORG], + ); + await pool.query( + `INSERT INTO organization_memberships ( + workos_user_id, workos_organization_id, email, role + ) VALUES ($1, $2, $3, 'member')`, + [id, ORG, email], + ); + } + + const identities = await pool.query<{ workos_user_id: string; identity_id: string }>( + `SELECT workos_user_id, identity_id FROM identity_workos_users + WHERE workos_user_id = ANY($1)`, + [[PRIMARY, SECONDARY, BYSTANDER]], + ); + const byUser = new Map(identities.rows.map((row) => [row.workos_user_id, row.identity_id])); + const identityId = byUser.get(PRIMARY)!; + await pool.query( + `UPDATE identity_workos_users + SET identity_id = $1, is_primary = FALSE + WHERE workos_user_id = ANY($2)`, + [identityId, [SECONDARY, BYSTANDER]], + ); + await pool.query(`DELETE FROM identities WHERE id = ANY($1)`, [ + [byUser.get(SECONDARY), byUser.get(BYSTANDER)], + ]); + await pool.query( + `INSERT INTO working_group_memberships (working_group_id, workos_user_id, status) + SELECT id, $1, 'active' FROM working_groups WHERE slug = 'aao-admin'`, + [PRIMARY], + ); + await pool.query( + `INSERT INTO working_group_leaders (working_group_id, user_id) + SELECT id, $1 FROM working_groups WHERE slug = 'aao-admin'`, + [PRIMARY], + ); + await pool.query( + `INSERT INTO slack_user_mappings ( + slack_user_id, slack_email, workos_user_id, mapping_status, mapping_source, mapped_at + ) VALUES ($1, 'jordan-race@pinnacle.example', $2, 'mapped', 'manual_admin', NOW())`, + [SLACK, PRIMARY], + ); + return identityId; + } + + function signedWebhook(app: express.Express, event: string, data: Record<string, unknown>) { + return request(app) + .post('/api/webhooks/workos') + .set('Content-Type', 'application/json') + .set('WorkOS-Signature', 'valid-test-signature') + .send({ id: `event_${event}_${Date.now()}`, event, data, created_at: new Date().toISOString() }); + } + + async function within<T>(promise: Promise<T>, timeoutMs = 5_000): Promise<T> { + let timeout: NodeJS.Timeout | undefined; + try { + return await Promise.race([ + promise, + new Promise<never>((_, reject) => { + timeout = setTimeout(() => reject(new Error(`race did not settle within ${timeoutMs}ms`)), timeoutMs); + }), + ]); + } finally { + if (timeout) clearTimeout(timeout); + } + } + + it('fails closed before a suspicious backfill deletion batch can write tombstones', () => { + expect(shouldAbortBackfillDeletion(1_000, 101)).toBe(true); + expect(shouldAbortBackfillDeletion(40, 11)).toBe(true); + expect(shouldAbortBackfillDeletion(40, 10)).toBe(false); + expect(shouldAbortBackfillDeletion(1_000, 25)).toBe(false); + }); + + it('bounds concurrent guarded writers below deletion-reserved pool capacity', async () => { + let activeCallbacks = 0; + let maxActiveCallbacks = 0; + + const results = await within(Promise.all( + Array.from({ length: 10 }, async (_, index) => withCredentialCreationEventMutation( + `${PREFIX}pool_capacity_${index}`, + async (client) => { + await upsertWorkosUserInCredentialEvent(client, { + id: `${PREFIX}pool_capacity_${index}`, + email: `pool-${index}@race.test`, + firstName: 'Pool', + lastName: 'Capacity', + emailVerified: true, + createdAt: new Date().toISOString(), + updatedAt: new Date().toISOString(), + }, 'provider_authoritative'); + activeCallbacks++; + maxActiveCallbacks = Math.max(maxActiveCallbacks, activeCallbacks); + try { + await client.query(`SELECT pg_sleep(0.025)`); + } finally { + activeCallbacks--; + } + }, + )), + ), 3_000); + + expect(results).toHaveLength(10); + expect(results.every((result) => result.applied)).toBe(true); + expect(maxActiveCallbacks).toBeGreaterThan(1); + // One connection stays reserved so confirmed deletion can always enter. + expect(maxActiveCallbacks).toBeLessThanOrEqual(7); + }); + + it('never replays a callback that has started after a retryable database error', async () => { + let callbackCalls = 0; + const retryableAfterMutation = Object.assign(new Error('injected callback deadlock'), { code: '40P01' }); + + await expect(withCredentialCreationEventMutation( + `${PREFIX}no_partial_replay`, + async (client) => { + await upsertWorkosUserInCredentialEvent(client, { + id: `${PREFIX}no_partial_replay`, + email: 'no-partial-replay@race.test', + firstName: 'No', + lastName: 'Replay', + emailVerified: true, + createdAt: new Date().toISOString(), + updatedAt: new Date().toISOString(), + }, 'provider_authoritative'); + callbackCalls++; + throw retryableAfterMutation; + }, + )).rejects.toBe(retryableAfterMutation); + + expect(callbackCalls).toBe(1); + expect((await pool.query( + `SELECT 1 FROM users WHERE workos_user_id = $1`, + [`${PREFIX}no_partial_replay`], + )).rows).toEqual([]); + }); + + it('preserves the distinct provider-authoritative and login-preserving name policies', async () => { + const userId = `${PREFIX}name_policy`; + const timestamp = new Date().toISOString(); + const created = await withCredentialCreationEventMutation(userId, (client) => + upsertWorkosUserInCredentialEvent(client, { + id: userId, + email: 'name-policy@race.test', + firstName: 'Original', + lastName: 'Local', + emailVerified: true, + createdAt: timestamp, + updatedAt: timestamp, + }, 'provider_authoritative'), + ); + expect(created.applied).toBe(true); + + const login = await withActiveCredentialEventMutation(userId, (client) => + upsertWorkosUserInCredentialEvent(client, { + id: userId, + email: 'name-policy@race.test', + firstName: 'Login', + lastName: 'Provider', + emailVerified: true, + createdAt: timestamp, + updatedAt: timestamp, + }, 'preserve_existing'), + ); + expect(login.applied).toBe(true); + expect((await pool.query( + `SELECT first_name, last_name FROM users WHERE workos_user_id = $1`, + [userId], + )).rows).toEqual([{ first_name: 'Original', last_name: 'Local' }]); + + mocks.constructEvent.mockResolvedValue(undefined); + const app = express(); + app.use('/api/webhooks', createWorkOSWebhooksRouter()); + const webhook = await signedWebhook(app, 'user.updated', { + id: userId, + email: 'name-policy@race.test', + first_name: 'Webhook', + last_name: 'Authoritative', + email_verified: true, + created_at: timestamp, + updated_at: timestamp, + }); + expect(webhook.status).toBe(200); + expect((await pool.query( + `SELECT first_name, last_name FROM users WHERE workos_user_id = $1`, + [userId], + )).rows).toEqual([{ first_name: 'Webhook', last_name: 'Authoritative' }]); + }); + + it('durably tombstones a signed deletion received before first local creation', async () => { + const deletedBeforeCreate = `${PREFIX}deleted_before_create`; + mocks.constructEvent.mockResolvedValue(undefined); + const app = express(); + app.use('/api/webhooks', createWorkOSWebhooksRouter()); + + const deletion = await signedWebhook(app, 'user.deleted', { id: deletedBeforeCreate }); + expect(deletion.status).toBe(200); + const audits = await pool.query<{ id: string; action: string; details: Record<string, unknown> }>( + `SELECT id, action, details FROM registry_audit_log + WHERE workos_user_id = $1 + ORDER BY created_at, id`, + [deletedBeforeCreate], + ); + expect(audits.rows).toHaveLength(1); + expect(audits.rows[0].action).toBe('identity_credential_deleted'); + expect(audits.rows[0].details).toMatchObject({ + deleted_workos_user_id: deletedBeforeCreate, + deletion_source: 'workos_webhook', + before_graph: { + users: [], + identity_workos_users: [], + }, + }); + + const replayCache = new Map([[ORG, [{ + id: deletedBeforeCreate, + email: 'must-remain-cached@race.test', + firstName: 'Replay', + lastName: 'Sentinel', + }]]]); + setUnifiedUsersCache(replayCache); + mocks.invalidateSessionsForUsers.mockClear(); + mocks.notifySystemError.mockClear(); + const sameSourceReplay = await signedWebhook(app, 'user.deleted', { id: deletedBeforeCreate }); + expect(sameSourceReplay.status).toBe(200); + const differentSourceReplay = await deleteIdentityCredential( + deletedBeforeCreate, + 'sync_users_backfill', + ); + expect(differentSourceReplay).toMatchObject({ + deleted: false, + affectedUserIds: [deletedBeforeCreate], + affectedSlackUserIds: [], + quarantine: null, + replay: { + audit_id: audits.rows[0].id, + kind: 'different_source', + original_deletion_source: 'workos_webhook', + requested_deletion_source: 'sync_users_backfill', + }, + }); + expect((await pool.query( + `SELECT id, action, details FROM registry_audit_log + WHERE workos_user_id = $1 + ORDER BY created_at, id`, + [deletedBeforeCreate], + )).rows).toEqual(audits.rows); + expect((await pool.query( + `SELECT 1 FROM authorization_epochs WHERE workos_user_id = $1`, + [deletedBeforeCreate], + )).rows).toEqual([]); + expect(getUnifiedUsersCache()).toBe(replayCache); + expect(mocks.invalidateSessionsForUsers).not.toHaveBeenCalled(); + expect(mocks.notifySystemError).not.toHaveBeenCalled(); + + const staleCreate = await signedWebhook(app, 'user.created', { + id: deletedBeforeCreate, + email: 'deleted-before-create@race.test', + first_name: 'Deleted', + last_name: 'BeforeCreate', + email_verified: true, + created_at: new Date().toISOString(), + updated_at: new Date().toISOString(), + }); + expect(staleCreate.status).toBe(200); + expect((await pool.query( + `SELECT 1 FROM users WHERE workos_user_id = $1`, + [deletedBeforeCreate], + )).rows).toEqual([]); + expect((await pool.query( + `SELECT 1 FROM identity_workos_users WHERE workos_user_id = $1`, + [deletedBeforeCreate], + )).rows).toEqual([]); + expect(mocks.tryAutoLinkWebsiteUserToSlack).not.toHaveBeenCalled(); + }); + + it('serializes both real callers and invalidates cached authority exactly through the helper', async () => { + const identityId = await seedIdentity(); + expect((await pool.query<{ workos_user_id: string }>( + `SELECT workos_user_id FROM users WHERE workos_user_id LIKE $1 ORDER BY workos_user_id`, + [`${PREFIX}%`], + )).rows.map((row) => row.workos_user_id)).toEqual([BYSTANDER, PRIMARY, SECONDARY].sort()); + expect(await isSlackUserAAOAdmin(SLACK)).toBe(true); + const survivorFingerprintsBefore = await Promise.all( + [SECONDARY, BYSTANDER].map((id) => getAuthorizationFingerprint([id])), + ); + setUnifiedUsersCache(new Map([[ORG, [{ + id: PRIMARY, + email: 'jordan-race@pinnacle.example', + firstName: 'Jordan', + lastName: 'Test', + }]]])); + + mocks.listUsers.mockImplementation(async (input: { organizationId?: string }) => ({ + data: input.organizationId + ? [] + : [ + { + id: SECONDARY, + email: 'sam-race@pinnacle.example', + firstName: 'Sam', + lastName: 'Test', + emailVerified: true, + createdAt: new Date().toISOString(), + updatedAt: new Date().toISOString(), + }, + { + id: BYSTANDER, + email: 'maya-race@pinnacle.example', + firstName: 'Maya', + lastName: 'Test', + emailVerified: true, + createdAt: new Date().toISOString(), + updatedAt: new Date().toISOString(), + }, + ], + })); + mocks.getUser.mockImplementation(async (workosUserId: string) => { + if (workosUserId === PRIMARY) { + throw Object.assign(new Error('not found'), { status: 404 }); + } + return { id: workosUserId }; + }); + mocks.constructEvent.mockResolvedValue(undefined); + mocks.coordinateDeletionCallers = true; + + // Keep the winning transaction open briefly so the other real caller + // reaches the identity lock and exercises PostgreSQL serialization. + await pool.query(` + CREATE FUNCTION delay_racing_user_delete() RETURNS trigger AS $$ + BEGIN + IF OLD.workos_user_id = '${PRIMARY}' THEN + PERFORM pg_sleep(0.15); + END IF; + RETURN OLD; + END; + $$ LANGUAGE plpgsql; + CREATE TRIGGER delay_racing_user_delete + BEFORE DELETE ON users + FOR EACH ROW EXECUTE FUNCTION delay_racing_user_delete(); + `); + + const app = express(); + app.use('/api/webhooks', createWorkOSWebhooksRouter()); + const webhookEvent = { + id: 'event_wh_backfill_race', + event: 'user.deleted', + data: { id: PRIMARY }, + created_at: new Date().toISOString(), + }; + + const [webhookResponse, backfill] = await Promise.all([ + request(app) + .post('/api/webhooks/workos') + .set('Content-Type', 'application/json') + .set('WorkOS-Signature', 'valid-test-signature') + .send(webhookEvent), + backfillUsers(), + ]); + + expect(webhookResponse.status).toBe(200); + expect(mocks.deletionEntrants).toEqual(expect.arrayContaining([ + { workosUserId: PRIMARY, source: 'workos_webhook' }, + { workosUserId: PRIMARY, source: 'sync_users_backfill' }, + ])); + expect(mocks.deletionEntrants).toHaveLength(2); + expect(backfill.usersRemoved + backfill.usersSkipped).toBeGreaterThanOrEqual(1); + expect((await pool.query(`SELECT 1 FROM users WHERE workos_user_id = $1`, [PRIMARY])).rows).toEqual([]); + expect((await pool.query( + `SELECT 1 FROM organization_memberships WHERE workos_user_id = $1`, + [PRIMARY], + )).rows).toEqual([]); + expect((await pool.query( + `SELECT workos_user_id FROM identity_workos_users + WHERE identity_id = $1 AND is_primary = TRUE`, + [identityId], + )).rows).toEqual([]); + expect(await getIdentityRecoveryQuarantine(identityId)).toMatchObject({ + identity_id: identityId, + deleted_workos_user_id: PRIMARY, + deletion_source: expect.stringMatching(/^(workos_webhook|sync_users_backfill)$/), + recovery_state: IDENTITY_RECOVERY_STATE, + snapshot_policy: IDENTITY_DELETION_SNAPSHOT_POLICY, + }); + const quarantineAudits = (await pool.query<{ id: string }>( + `SELECT id FROM registry_audit_log + WHERE action = 'identity_primary_deletion_quarantined' AND resource_id = $1`, + [identityId], + )).rows; + expect(quarantineAudits).toHaveLength(1); + for (const [index, id] of [SECONDARY, BYSTANDER].entries()) { + expect(await getAuthorizationFingerprint([id])).not.toBe(survivorFingerprintsBefore[index]); + } + expect(mocks.invalidateSessionsForUsers).toHaveBeenCalledWith( + expect.arrayContaining([PRIMARY, SECONDARY, BYSTANDER]), + ); + expect(getUnifiedUsersCache()).toBeNull(); + expect(mocks.notifySystemError).toHaveBeenCalledTimes(1); + expect(mocks.notifySystemError).toHaveBeenCalledWith({ + source: `identity-primary-deletion-quarantine:${identityId}`, + errorMessage: [ + `Credential ${PRIMARY} was deleted without promotion.`, + `Identity ${identityId} is quarantined.`, + `recovery_state=${IDENTITY_RECOVERY_STATE}.`, + `audit_id=${quarantineAudits[0].id}.`, + ].join(' '), + }); + expect((await pool.query( + `SELECT status FROM working_group_memberships WHERE workos_user_id = $1`, + [PRIMARY], + )).rows).toEqual([{ status: 'inactive' }]); + expect((await pool.query( + `SELECT 1 FROM working_group_leaders WHERE user_id = $1`, + [PRIMARY], + )).rows).toEqual([]); + expect((await pool.query( + `SELECT workos_user_id, mapping_status FROM slack_user_mappings WHERE slack_user_id = $1`, + [SLACK], + )).rows).toEqual([{ workos_user_id: null, mapping_status: 'unmapped' }]); + expect(await isSlackUserAAOAdmin(SLACK)).toBe(false); + + // A stale listUsers page after the signed deletion must not resurrect the + // users row, automatic identity binding, or any authority. + mocks.listUsers.mockImplementation(async (input: { organizationId?: string }) => ({ + data: input.organizationId + ? [] + : [ + { + id: PRIMARY, + email: 'jordan-race@pinnacle.example', + firstName: 'Jordan', + lastName: 'Test', + emailVerified: true, + createdAt: new Date().toISOString(), + updatedAt: new Date().toISOString(), + }, + { + id: SECONDARY, + email: 'sam-race@pinnacle.example', + firstName: 'Sam', + lastName: 'Test', + emailVerified: true, + createdAt: new Date().toISOString(), + updatedAt: new Date().toISOString(), + }, + { + id: BYSTANDER, + email: 'maya-race@pinnacle.example', + firstName: 'Maya', + lastName: 'Test', + emailVerified: true, + createdAt: new Date().toISOString(), + updatedAt: new Date().toISOString(), + }, + ], + })); + const staleReplay = await backfillUsers(); + expect(staleReplay.usersSkipped).toBeGreaterThanOrEqual(1); + expect((await pool.query(`SELECT 1 FROM users WHERE workos_user_id = $1`, [PRIMARY])).rows).toEqual([]); + expect((await pool.query( + `SELECT 1 FROM identity_workos_users WHERE workos_user_id = $1`, + [PRIMARY], + )).rows).toEqual([]); + expect((await pool.query( + `SELECT 1 FROM organization_memberships WHERE workos_user_id = $1`, + [PRIMARY], + )).rows).toEqual([]); + expect(await isSlackUserAAOAdmin(SLACK)).toBe(false); + }); + + it('lets deletion finish while post-commit work waits and fences stale Slack/chapter writes', async () => { + await seedIdentity(); + const chapter = await pool.query<{ id: string }>( + `INSERT INTO working_groups (name, slug) VALUES ('Race chapter', $1) RETURNING id`, + [CHAPTER_SLUG], + ); + await pool.query( + `UPDATE slack_user_mappings + SET workos_user_id = NULL, mapping_status = 'unmapped', mapping_source = NULL + WHERE slack_user_id = $1`, + [SLACK], + ); + mocks.constructEvent.mockResolvedValue(undefined); + let releaseSideEffects!: () => void; + let sideEffectsEntered!: () => void; + const entered = new Promise<void>((resolve) => { sideEffectsEntered = resolve; }); + const release = new Promise<void>((resolve) => { releaseSideEffects = resolve; }); + mocks.tryAutoLinkWebsiteUserToSlack.mockImplementation(async () => { + sideEffectsEntered(); + await release; + const mapping = await new SlackDatabase().mapUser({ + slack_user_id: SLACK, + workos_user_id: PRIMARY, + mapping_source: 'email_auto', + }); + if (mapping) { + await new WorkingGroupDatabase().addMembership({ + working_group_id: chapter.rows[0].id, + workos_user_id: PRIMARY, + }); + } + return { linked: Boolean(mapping), slack_user_id: SLACK, workos_user_id: PRIMARY, chapters_joined: 0 }; + }); + + const app = express(); + app.use('/api/webhooks', createWorkOSWebhooksRouter()); + const staleUserEvent = Promise.resolve(signedWebhook(app, 'user.created', { + id: PRIMARY, + email: 'jordan-race@gmail.com', + first_name: 'Jordan', + last_name: 'Race', + email_verified: true, + created_at: new Date().toISOString(), + updated_at: new Date().toISOString(), + })); + await entered; + let deletionSettled = false; + const deletion = deleteIdentityCredential(PRIMARY, 'workos_webhook') + .finally(() => { deletionSettled = true; }); + await new Promise((resolve) => setTimeout(resolve, 50)); + const deletionWasBlocked = !deletionSettled; + releaseSideEffects(); + const [eventResponse, deletionResult] = await within(Promise.all([staleUserEvent, deletion])); + + expect(deletionWasBlocked).toBe(false); + expect(eventResponse.status).toBe(200); + expect(deletionResult.deleted).toBe(true); + expect((await pool.query(`SELECT 1 FROM users WHERE workos_user_id = $1`, [PRIMARY])).rows).toEqual([]); + expect((await pool.query( + `SELECT 1 FROM organization_memberships WHERE workos_user_id = $1`, [PRIMARY], + )).rows).toEqual([]); + expect((await pool.query( + `SELECT status FROM working_group_memberships + WHERE workos_user_id = $1 AND working_group_id = $2`, + [PRIMARY, chapter.rows[0].id], + )).rows).toEqual([]); + expect((await pool.query( + `SELECT workos_user_id, mapping_status FROM slack_user_mappings WHERE slack_user_id = $1`, [SLACK], + )).rows).toEqual([{ workos_user_id: null, mapping_status: 'unmapped' }]); + const replay = await signedWebhook(app, 'user.created', { + id: PRIMARY, + email: 'jordan-race@gmail.com', + first_name: 'Jordan', + last_name: 'Race', + email_verified: true, + created_at: new Date().toISOString(), + updated_at: new Date().toISOString(), + }); + expect(replay.status).toBe(200); + expect(mocks.tryAutoLinkWebsiteUserToSlack).toHaveBeenCalledTimes(1); + expect((await pool.query(`SELECT 1 FROM users WHERE workos_user_id = $1`, [PRIMARY])).rows).toEqual([]); + expect((await pool.query( + `SELECT 1 FROM working_group_memberships + WHERE workos_user_id = $1 AND working_group_id = $2 AND status = 'active'`, + [PRIMARY, chapter.rows[0].id], + )).rows).toEqual([]); + }); + + it('commits signed membership locally before post-commit work and lets deletion revoke it immediately', async () => { + await seedIdentity(); + await pool.query( + `DELETE FROM organization_memberships WHERE workos_user_id = $1 AND workos_organization_id = $2`, + [PRIMARY, ORG], + ); + await pool.query( + `INSERT INTO invitation_seat_types ( + workos_invitation_id, workos_organization_id, email, seat_type + ) VALUES ('invite_wh_membership_race', $1, 'jordan-race@pinnacle.example', 'community_only')`, + [ORG], + ); + mocks.constructEvent.mockResolvedValue(undefined); + mocks.getUser.mockResolvedValue({ + id: PRIMARY, + email: 'jordan-race@pinnacle.example', + firstName: 'Jordan', + lastName: 'Race', + emailVerified: true, + createdAt: new Date().toISOString(), + updatedAt: new Date().toISOString(), + }); + let releasePostWrite!: () => void; + let postWriteEntered!: () => void; + const entered = new Promise<void>((resolve) => { postWriteEntered = resolve; }); + const release = new Promise<void>((resolve) => { releasePostWrite = resolve; }); + mocks.tryAutoLinkWebsiteUserToSlack.mockImplementation(async () => { + postWriteEntered(); + await release; + return { linked: false, reason: 'already_linked' }; + }); + + const app = express(); + app.use('/api/webhooks', createWorkOSWebhooksRouter()); + const staleMembershipEvent = Promise.resolve(signedWebhook(app, 'organization_membership.created', { + id: 'om_race_restore', + user_id: PRIMARY, + organization_id: ORG, + status: 'active', + role: { slug: 'admin' }, + created_at: new Date().toISOString(), + updated_at: new Date().toISOString(), + })); + await entered; + const membershipWhileLocked = (await pool.query( + `SELECT role FROM organization_memberships + WHERE workos_user_id = $1 AND workos_organization_id = $2`, + [PRIMARY, ORG], + )).rows; + let deletionSettled = false; + const deletion = deleteIdentityCredential(PRIMARY, 'workos_webhook') + .finally(() => { deletionSettled = true; }); + await new Promise((resolve) => setTimeout(resolve, 50)); + const deletionWasBlocked = !deletionSettled; + releasePostWrite(); + const [eventResponse, deletionResult] = await within(Promise.all([staleMembershipEvent, deletion])); + + expect(deletionWasBlocked).toBe(false); + expect(membershipWhileLocked).toEqual([{ role: 'admin' }]); + expect(eventResponse.status).toBe(200); + expect(deletionResult.deleted).toBe(true); + expect((await pool.query( + `SELECT 1 FROM organization_memberships WHERE workos_user_id = $1`, [PRIMARY], + )).rows).toEqual([]); + expect((await pool.query( + `SELECT workos_user_id, mapping_status FROM slack_user_mappings WHERE slack_user_id = $1`, [SLACK], + )).rows).toEqual([{ workos_user_id: null, mapping_status: 'unmapped' }]); + expect((await pool.query( + `SELECT 1 FROM working_group_memberships WHERE workos_user_id = $1 AND status = 'active'`, [PRIMARY], + )).rows).toEqual([]); + const replay = await signedWebhook(app, 'organization_membership.created', { + id: 'om_race_restore_replay', + user_id: PRIMARY, + organization_id: ORG, + status: 'active', + role: { slug: 'admin' }, + created_at: new Date().toISOString(), + updated_at: new Date().toISOString(), + }); + expect(replay.status).toBe(200); + expect(mocks.tryAutoLinkWebsiteUserToSlack).toHaveBeenCalledTimes(1); + expect((await pool.query( + `SELECT 1 FROM organization_memberships WHERE workos_user_id = $1`, [PRIMARY], + )).rows).toEqual([]); + }); + + it('serializes a distinct Slack/WG/leader/topic authority writer with deletion and leaves no alias authority', async () => { + await seedIdentity(); + const chapter = await pool.query<{ id: string }>( + `INSERT INTO working_groups (name, slug) VALUES ('Authority writer race', $1) RETURNING id`, + [CHAPTER_SLUG], + ); + await pool.query( + `UPDATE slack_user_mappings + SET workos_user_id = NULL, mapping_status = 'unmapped', mapping_source = NULL + WHERE slack_user_id = $1`, + [SLACK], + ); + let writerEntered!: () => void; + let releaseWriter!: () => void; + const entered = new Promise<void>((resolve) => { writerEntered = resolve; }); + const release = new Promise<void>((resolve) => { releaseWriter = resolve; }); + + const writer = withActiveCredentialEventMutation(PRIMARY, async (client) => { + await new SlackDatabase().mapUser({ + slack_user_id: SLACK, + workos_user_id: PRIMARY, + mapping_source: 'email_auto', + }, client); + await new WorkingGroupDatabase().addLeader(chapter.rows[0].id, PRIMARY, client); + await client.query( + `INSERT INTO working_group_memberships (working_group_id, workos_user_id, status) + VALUES ($1, $2, 'active')`, + [chapter.rows[0].id, SLACK], + ); + await client.query( + `INSERT INTO working_group_leaders (working_group_id, user_id) + VALUES ($1, $2)`, + [chapter.rows[0].id, SLACK], + ); + await client.query( + `INSERT INTO working_group_topic_subscriptions (working_group_id, workos_user_id, topic_slugs) + VALUES ($1, $2, ARRAY['race-topic'])`, + [chapter.rows[0].id, SLACK], + ); + writerEntered(); + await release; + }); + await entered; + + let deletionSettled = false; + const deletion = deleteIdentityCredential(PRIMARY, 'workos_webhook') + .finally(() => { deletionSettled = true; }); + await new Promise((resolve) => setTimeout(resolve, 50)); + expect(deletionSettled).toBe(false); + releaseWriter(); + const [writerResult, deletionResult] = await within(Promise.all([writer, deletion])); + + expect(writerResult.applied).toBe(true); + expect(deletionResult.deleted).toBe(true); + expect((await pool.query( + `SELECT workos_user_id, mapping_status FROM slack_user_mappings WHERE slack_user_id = $1`, + [SLACK], + )).rows).toEqual([{ workos_user_id: null, mapping_status: 'unmapped' }]); + expect((await pool.query( + `SELECT 1 FROM working_group_memberships + WHERE working_group_id = $1 AND workos_user_id = ANY($2) AND status = 'active'`, + [chapter.rows[0].id, [PRIMARY, SLACK]], + )).rows).toEqual([]); + expect((await pool.query( + `SELECT 1 FROM working_group_leaders + WHERE working_group_id = $1 AND user_id = ANY($2)`, + [chapter.rows[0].id, [PRIMARY, SLACK]], + )).rows).toEqual([]); + expect((await pool.query( + `SELECT 1 FROM working_group_topic_subscriptions + WHERE working_group_id = $1 AND workos_user_id = ANY($2)`, + [chapter.rows[0].id, [PRIMARY, SLACK]], + )).rows).toEqual([]); + expect(await isSlackUserAAOAdmin(SLACK)).toBe(false); + }); + + it('rejects every authority-writer fence after the credential is durably deleted', async () => { + await seedIdentity(); + const chapter = await pool.query<{ id: string }>( + `INSERT INTO working_groups (name, slug) VALUES ('Deleted writer fence', $1) RETURNING id`, + [CHAPTER_SLUG], + ); + expect((await deleteIdentityCredential(PRIMARY, 'workos_webhook')).deleted).toBe(true); + + const workingGroups = new WorkingGroupDatabase(); + await expect(workingGroups.grantAAOAdminMembership({ + targetUserId: PRIMARY, + actorUserId: BYSTANDER, + actorAuthorizationMechanism: 'aao_admin_working_group', + reason: 'must remain deleted', + })).rejects.toThrow('Cannot grant AAO admin authority to an inactive credential'); + await expect(workingGroups.addMembership({ + working_group_id: chapter.rows[0].id, + workos_user_id: PRIMARY, + })).rejects.toThrow('Cannot add working-group authority to an inactive credential'); + await expect(workingGroups.addLeader( + chapter.rows[0].id, + PRIMARY, + )).rejects.toThrow('Cannot assign leadership to an inactive credential'); + await expect(upsertOrganizationMembership({ + user_id: PRIMARY, + organization_id: ORG, + membership_id: 'om_deleted_writer_fence', + email: 'deleted-writer@race.test', + first_name: 'Deleted', + last_name: 'Writer', + role: 'admin', + seat_type: 'community_only', + has_explicit_seat_type: true, + })).rejects.toThrow('Cannot write organization authority for an inactive credential'); + await expect(setMembershipRole( + PRIMARY, + ORG, + 'admin', + )).rejects.toThrow('Cannot update organization authority for an inactive credential'); + await expect(new MeetingsDatabase().updateTopicSubscription({ + working_group_id: chapter.rows[0].id, + workos_user_id: PRIMARY, + topic_slugs: ['must-remain-deleted'], + })).rejects.toThrow('Cannot update topic subscriptions for an inactive credential'); + + expect((await pool.query( + `SELECT 1 FROM working_group_memberships + WHERE working_group_id = $1 AND workos_user_id = $2 AND status = 'active'`, + [chapter.rows[0].id, PRIMARY], + )).rows).toEqual([]); + expect((await pool.query( + `SELECT 1 FROM working_group_leaders WHERE working_group_id = $1 AND user_id = $2`, + [chapter.rows[0].id, PRIMARY], + )).rows).toEqual([]); + expect((await pool.query( + `SELECT 1 FROM working_group_topic_subscriptions + WHERE working_group_id = $1 AND workos_user_id = $2`, + [chapter.rows[0].id, PRIMARY], + )).rows).toEqual([]); + expect((await pool.query( + `SELECT 1 FROM organization_memberships WHERE workos_user_id = $1`, + [PRIMARY], + )).rows).toEqual([]); + expect((await pool.query( + `SELECT 1 FROM aao_admin_access_events + WHERE target_user_id = $1 AND event_type = 'granted'`, + [PRIMARY], + )).rows).toEqual([]); + }); + + it('does not hold the credential lock while membership provider prefetch is hung', async () => { + await seedIdentity(); + mocks.constructEvent.mockResolvedValue(undefined); + let providerEntered!: () => void; + const entered = new Promise<void>((resolve) => { providerEntered = resolve; }); + mocks.getUser.mockImplementation(async () => { + providerEntered(); + return new Promise(() => undefined); + }); + const app = express(); + app.use('/api/webhooks', createWorkOSWebhooksRouter()); + const started = Date.now(); + const event = Promise.resolve(signedWebhook(app, 'organization_membership.created', { + id: 'om_hung_provider', + user_id: PRIMARY, + organization_id: ORG, + status: 'active', + role: { slug: 'admin' }, + created_at: new Date().toISOString(), + updated_at: new Date().toISOString(), + })); + await entered; + + const deletion = await within(deleteIdentityCredential(PRIMARY, 'workos_webhook'), 1_000); + const eventResponse = await within(event, 1_000); + + expect(deletion.deleted).toBe(true); + expect(eventResponse.status).toBe(500); + expect(Date.now() - started).toBeLessThan(1_000); + expect((await pool.query(`SELECT 1 FROM users WHERE workos_user_id = $1`, [PRIMARY])).rows) + .toEqual([]); + }); + + it('fails a signed stale writer closed after bounded credential-lock retries', async () => { + await seedIdentity(); + mocks.constructEvent.mockResolvedValue(undefined); + const lockHolder = await pool.connect(); + await lockHolder.query('BEGIN'); + await lockHolder.query(`SELECT pg_advisory_xact_lock(hashtextextended($1, 6827))`, [PRIMARY]); + try { + const app = express(); + app.use('/api/webhooks', createWorkOSWebhooksRouter()); + const started = Date.now(); + const response = await within(signedWebhook(app, 'user.updated', { + id: PRIMARY, + email: 'stale-update@pinnacle.example', + first_name: 'Stale', + last_name: 'Update', + email_verified: true, + created_at: new Date().toISOString(), + updated_at: new Date().toISOString(), + }), 5_000); + expect(response.status).toBe(500); + expect(Date.now() - started).toBeLessThan(4_000); + expect((await pool.query<{ email: string }>( + `SELECT email FROM users WHERE workos_user_id = $1`, [PRIMARY], + )).rows).toEqual([{ email: 'jordan-race@pinnacle.example' }]); + expect(mocks.tryAutoLinkWebsiteUserToSlack).not.toHaveBeenCalled(); + } finally { + await lockHolder.query('ROLLBACK'); + lockHolder.release(); + } + }); +}); diff --git a/server/tests/unit/aao-admin-membership-audit.test.ts b/server/tests/unit/aao-admin-membership-audit.test.ts index 479f2e23494..7c0f3db5903 100644 --- a/server/tests/unit/aao-admin-membership-audit.test.ts +++ b/server/tests/unit/aao-admin-membership-audit.test.ts @@ -1,12 +1,13 @@ import { beforeEach, describe, expect, it, vi } from 'vitest'; const mocks = vi.hoisted(() => ({ + poolQuery: vi.fn(), clientQuery: vi.fn(), release: vi.fn(), })); vi.mock('../../src/db/client.js', () => ({ - query: vi.fn(), + query: mocks.poolQuery, getPool: () => ({ connect: vi.fn().mockResolvedValue({ query: mocks.clientQuery, release: mocks.release }), }), @@ -16,6 +17,27 @@ vi.mock('../../src/addie/services/journey-computation.js', () => ({ computeJourneyStage: vi.fn(), })); +vi.mock('../../src/db/identity-db.js', async (importOriginal) => ({ + ...(await importOriginal<typeof import('../../src/db/identity-db.js')>()), + withActiveCredentialEventMutation: async ( + _workosUserId: string, + mutation: (client: { query: typeof mocks.clientQuery }) => Promise<unknown>, + ) => { + const client = { query: mocks.clientQuery }; + await client.query('BEGIN'); + try { + const value = await mutation(client); + await client.query('COMMIT'); + mocks.release(); + return { applied: true, value }; + } catch (error) { + await client.query('ROLLBACK'); + mocks.release(); + throw error; + } + }, +})); + import { WorkingGroupDatabase } from '../../src/db/working-group-db.js'; function queryResult(rows: unknown[] = [], rowCount = rows.length) { @@ -25,6 +47,12 @@ function queryResult(rows: unknown[] = [], rowCount = rows.length) { describe('AAO site-admin membership audit transaction', () => { beforeEach(() => { vi.clearAllMocks(); + mocks.poolQuery.mockImplementation((sql: string) => { + if (sql.includes('FROM slack_user_mappings')) { + return Promise.resolve(queryResult([{ workos_user_id: 'user_canonical' }])); + } + throw new Error(`Unexpected pool SQL: ${sql}`); + }); mocks.clientQuery.mockImplementation((sql: string) => { if (sql === 'BEGIN' || sql === 'COMMIT' || sql === 'ROLLBACK') return Promise.resolve(queryResult()); if (sql.includes('SELECT id FROM working_groups')) return Promise.resolve(queryResult([{ id: 'wg_aao_admin' }])); @@ -74,6 +102,7 @@ describe('AAO site-admin membership audit transaction', () => { }); it('rolls the membership write back when the audit insert fails', async () => { + mocks.poolQuery.mockResolvedValue(queryResult()); mocks.clientQuery.mockImplementation((sql: string) => { if (sql.includes('INSERT INTO aao_admin_access_events')) return Promise.reject(new Error('audit storage unavailable')); if (sql === 'BEGIN' || sql === 'ROLLBACK') return Promise.resolve(queryResult()); @@ -95,6 +124,7 @@ describe('AAO site-admin membership audit transaction', () => { }); it('rolls an active-membership delete back when its revoke audit insert fails', async () => { + mocks.poolQuery.mockResolvedValue(queryResult()); mocks.clientQuery.mockImplementation((sql: string) => { if (sql.includes('INSERT INTO aao_admin_access_events')) return Promise.reject(new Error('audit storage unavailable')); if (sql === 'BEGIN' || sql === 'ROLLBACK') return Promise.resolve(queryResult()); diff --git a/server/tests/unit/aao-admin-status.test.ts b/server/tests/unit/aao-admin-status.test.ts index 5a10031722d..eaec2feefad 100644 --- a/server/tests/unit/aao-admin-status.test.ts +++ b/server/tests/unit/aao-admin-status.test.ts @@ -13,7 +13,6 @@ vi.mock('../../src/db/working-group-db.js', () => ({ })); import { - AAO_ADMIN_POSITIVE_CACHE_TTL_MS, isWebUserAAOAdmin, resolveWebUserAAOAdminAccess, } from '../../src/addie/admin-status-lookup.js'; @@ -45,12 +44,9 @@ describe('site-admin access decisions', () => { else process.env.ADMIN_EMAILS = originalAdminEmails; }); - it('bounds cached positive membership decisions to one minute', async () => { + it('does not cache positive membership decisions across requests', async () => { await expect(isWebUserAAOAdmin('user_admin')).resolves.toBe(true); - const cached = getWebAdminStatusCache().get('user_admin'); - - expect(cached?.expiresAt).toBe(Date.now() + AAO_ADMIN_POSITIVE_CACHE_TTL_MS); - vi.advanceTimersByTime(AAO_ADMIN_POSITIVE_CACHE_TTL_MS + 1); + expect(getWebAdminStatusCache().get('user_admin')).toBeUndefined(); await isWebUserAAOAdmin('user_admin'); expect(mocks.isMember).toHaveBeenCalledTimes(2); }); diff --git a/server/tests/unit/addie-admin-global-auth.test.ts b/server/tests/unit/addie-admin-global-auth.test.ts index 38fa4c89bee..530a793bdd4 100644 --- a/server/tests/unit/addie-admin-global-auth.test.ts +++ b/server/tests/unit/addie-admin-global-auth.test.ts @@ -17,6 +17,7 @@ const mocks = vi.hoisted(() => ({ getModelExecutionReadiness: vi.fn(), getRouterShadowSummary: vi.fn(), serveHtmlWithConfig: vi.fn(), + readCredentialAuthorizationLifecycle: vi.fn(), })); vi.hoisted(() => { @@ -53,6 +54,11 @@ vi.mock('../../src/db/client.js', async (importOriginal) => ({ getPool: () => ({ query: mocks.poolQuery }), })); +vi.mock('../../src/db/authorization-epoch-db.js', async (importOriginal) => ({ + ...(await importOriginal<typeof import('../../src/db/authorization-epoch-db.js')>()), + readCredentialAuthorizationLifecycle: mocks.readCredentialAuthorizationLifecycle, +})); + vi.mock('../../src/db/working-group-db.js', () => ({ WorkingGroupDatabase: class WorkingGroupDatabase { getWorkingGroupBySlug = mocks.getAdminWorkingGroupBySlug; @@ -124,6 +130,20 @@ describe('Addie real global-admin boundary', () => { mocks.resolveEffectiveMembership.mockResolvedValue({ is_member: true }); mocks.checkPlatformBanForApiKey.mockResolvedValue({ banned: false }); mocks.checkPlatformBan.mockResolvedValue({ banned: false }); + mocks.readCredentialAuthorizationLifecycle.mockImplementation((workosUserId: string) => + Promise.resolve({ + status: 'active', + snapshot: { + workos_user_id: workosUserId, + email: 'sso-admin@example.test', + first_name: 'SSO', + last_name: 'Admin', + identity_id: '00000000-0000-4000-8000-000000000001', + primary_workos_user_id: workosUserId, + fingerprint: '', + }, + }), + ); mocks.getAdminWorkingGroupBySlug.mockResolvedValue({ id: 'wg_aao_admin', slug: 'aao-admin', diff --git a/server/tests/unit/authorization-epoch-session.test.ts b/server/tests/unit/authorization-epoch-session.test.ts index 0ad33ee6e2a..b167fdad474 100644 --- a/server/tests/unit/authorization-epoch-session.test.ts +++ b/server/tests/unit/authorization-epoch-session.test.ts @@ -16,11 +16,14 @@ const mocks = vi.hoisted(() => ({ loadSealedSession: vi.fn(), checkPlatformBan: vi.fn(), checkPlatformBanForApiKey: vi.fn(), - getAuthorizationFingerprint: vi.fn(), + readCredentialAuthorizationLifecycle: vi.fn(), poolQuery: vi.fn(), + verifyWorkOSJWT: vi.fn(), })); vi.hoisted(() => { + delete process.env.DEV_USER_EMAIL; + delete process.env.DEV_USER_ID; process.env.WORKOS_API_KEY = process.env.WORKOS_API_KEY ?? 'sk_test'; process.env.WORKOS_CLIENT_ID = process.env.WORKOS_CLIENT_ID ?? 'client_test'; process.env.WORKOS_COOKIE_PASSWORD = @@ -46,7 +49,7 @@ vi.mock('../../src/db/bans-db.js', () => ({ })); vi.mock('../../src/db/authorization-epoch-db.js', () => ({ - getAuthorizationFingerprint: mocks.getAuthorizationFingerprint, + readCredentialAuthorizationLifecycle: mocks.readCredentialAuthorizationLifecycle, bumpAuthorizationEpochs: vi.fn(), })); @@ -56,7 +59,12 @@ vi.mock('../../src/db/client.js', () => ({ isDatabaseInitialized: () => true, })); -import { requireAuth } from '../../src/middleware/auth.js'; +vi.mock('../../src/auth/workos-jwt.js', () => ({ + looksLikeJWT: vi.fn(() => true), + verifyWorkOSJWT: mocks.verifyWorkOSJWT, +})); + +import { optionalAuth, requireAuth, validateWorkOSBearerJWT } from '../../src/middleware/auth.js'; const SESSION_USER = { id: 'user_epoch_primary', @@ -87,6 +95,27 @@ function makeResponse(): Response { } as unknown as Response; } +function makeBearerRequest(token: string): Request { + return { + headers: { authorization: `Bearer ${token}` }, + } as unknown as Request; +} + +function activeLifecycle(fingerprint: string) { + return { + status: 'active' as const, + snapshot: { + workos_user_id: SESSION_USER.id, + email: SESSION_USER.email, + first_name: SESSION_USER.firstName, + last_name: SESSION_USER.lastName, + identity_id: 'identity_epoch', + primary_workos_user_id: SESSION_USER.id, + fingerprint, + }, + }; +} + describe('persisted authorization epoch gates the session cache', () => { beforeEach(() => { vi.clearAllMocks(); @@ -113,7 +142,15 @@ describe('persisted authorization epoch gates the session cache', () => { authenticate: mocks.authenticate, refresh: vi.fn(), }); - mocks.getAuthorizationFingerprint.mockResolvedValue('user_epoch_primary:1'); + mocks.readCredentialAuthorizationLifecycle.mockResolvedValue( + activeLifecycle('user_epoch_primary:1'), + ); + mocks.verifyWorkOSJWT.mockResolvedValue({ + sub: SESSION_USER.id, + email: SESSION_USER.email, + expiresAt: Math.floor(Date.now() / 1000) + 3600, + isM2M: false, + }); }); it('serves the cached session while the fingerprint is unchanged', async () => { @@ -136,7 +173,9 @@ describe('persisted authorization epoch gates the session cache', () => { expect(mocks.authenticate).toHaveBeenCalledTimes(1); // A binding mutation bumped the credential's epoch on another instance. - mocks.getAuthorizationFingerprint.mockResolvedValue('user_epoch_primary:2'); + mocks.readCredentialAuthorizationLifecycle.mockResolvedValue( + activeLifecycle('user_epoch_primary:2'), + ); await requireAuth(makeRequest(cookie), makeResponse(), next); @@ -151,11 +190,134 @@ describe('persisted authorization epoch gates the session cache', () => { await requireAuth(makeRequest(cookie), makeResponse(), next); expect(mocks.authenticate).toHaveBeenCalledTimes(1); - mocks.getAuthorizationFingerprint.mockRejectedValueOnce(new Error('db down')); + mocks.readCredentialAuthorizationLifecycle.mockResolvedValueOnce({ status: 'unavailable' }); await requireAuth(makeRequest(cookie), makeResponse(), next); expect(next).toHaveBeenCalledTimes(2); expect(mocks.authenticate).toHaveBeenCalledTimes(2); }); + + it('denies an epoch-0 credential on replica B immediately after replica A records deletion', async () => { + const cookie = `sealed-epoch-zero-deleted-${Date.now()}`; + const next = vi.fn() as NextFunction; + mocks.readCredentialAuthorizationLifecycle.mockResolvedValue(activeLifecycle('')); + + await requireAuth(makeRequest(cookie), makeResponse(), next); + expect(next).toHaveBeenCalledTimes(1); + expect(mocks.authenticate).toHaveBeenCalledTimes(1); + + // Replica A committed a non-cascading deletion tombstone. Replica B still + // has its own cold epoch-0 session entry, but the next hit sees the marker. + mocks.readCredentialAuthorizationLifecycle.mockResolvedValue({ + status: 'terminal', reason: 'deleted_or_quarantined', + }); + const response = makeResponse(); + await requireAuth(makeRequest(cookie), response, next); + + expect(next).toHaveBeenCalledTimes(1); + expect(mocks.authenticate).toHaveBeenCalledTimes(2); + expect(response.status).toHaveBeenCalledWith(401); + }); + + it('does not serve replica-B cached authority when deletion-marker lookup is unavailable', async () => { + const cookie = `sealed-deletion-marker-outage-${Date.now()}`; + const next = vi.fn() as NextFunction; + mocks.readCredentialAuthorizationLifecycle.mockResolvedValue(activeLifecycle('')); + + await requireAuth(makeRequest(cookie), makeResponse(), next); + mocks.readCredentialAuthorizationLifecycle.mockResolvedValue({ status: 'unavailable' }); + const response = makeResponse(); + await requireAuth(makeRequest(cookie), response, next); + + expect(next).toHaveBeenCalledTimes(1); + expect(mocks.authenticate).toHaveBeenCalledTimes(2); + expect(response.status).toHaveBeenCalledWith(503); + }); + + it('invalidates replica B through persistence when only replica A receives local eviction', async () => { + vi.resetModules(); + const replicaA = await import('../../src/middleware/auth.js'); + vi.resetModules(); + const replicaB = await import('../../src/middleware/auth.js'); + const cookieA = `sealed-replica-a-${Date.now()}`; + const cookieB = `sealed-replica-b-${Date.now()}`; + const nextA = vi.fn() as NextFunction; + const nextB = vi.fn() as NextFunction; + mocks.readCredentialAuthorizationLifecycle.mockResolvedValue(activeLifecycle('')); + + await replicaA.requireAuth(makeRequest(cookieA), makeResponse(), nextA); + await replicaB.requireAuth(makeRequest(cookieB), makeResponse(), nextB); + expect(mocks.authenticate).toHaveBeenCalledTimes(2); + + // The deletion process only has access to replica A's in-memory cache. + // Replica B retains its independent entry and must observe the durable + // deletion marker on its immediate next request, without waiting for TTL. + replicaA.invalidateSessionsForUsers([SESSION_USER.id]); + mocks.readCredentialAuthorizationLifecycle.mockResolvedValue({ + status: 'terminal', reason: 'deleted_or_quarantined', + }); + const responseB = makeResponse(); + await replicaB.requireAuth(makeRequest(cookieB), responseB, nextB); + + expect(nextB).toHaveBeenCalledTimes(1); + expect(mocks.authenticate).toHaveBeenCalledTimes(3); + expect(responseB.status).toHaveBeenCalledWith(401); + }); + + it('denies an epoch-0 cached bearer JWT on the immediate hit after deletion', async () => { + const token = `header.${Date.now()}.signature`; + mocks.readCredentialAuthorizationLifecycle.mockResolvedValue(activeLifecycle('')); + + await expect(validateWorkOSBearerJWT(makeBearerRequest(token))).resolves.toMatchObject({ + user: { id: SESSION_USER.id }, + }); + expect(mocks.verifyWorkOSJWT).toHaveBeenCalledTimes(1); + + mocks.readCredentialAuthorizationLifecycle.mockResolvedValue({ + status: 'terminal', reason: 'deleted_or_quarantined', + }); + await expect(validateWorkOSBearerJWT(makeBearerRequest(token))).resolves.toBeNull(); + + expect(mocks.verifyWorkOSJWT).toHaveBeenCalledTimes(2); + }); + + it('cannot poison requireAuth from optionalAuth with a deleted sealed credential', async () => { + const cookie = `sealed-optional-poison-${Date.now()}`; + const optionalRequest = makeRequest(cookie); + const optionalNext = vi.fn() as NextFunction; + mocks.readCredentialAuthorizationLifecycle.mockResolvedValue({ + status: 'terminal', reason: 'deleted_or_quarantined', + }); + + await optionalAuth(optionalRequest, makeResponse(), optionalNext); + expect(optionalNext).toHaveBeenCalledTimes(1); + expect(optionalRequest.user).toBeUndefined(); + + const requiredResponse = makeResponse(); + const requiredNext = vi.fn() as NextFunction; + await requireAuth(makeRequest(cookie), requiredResponse, requiredNext); + + expect(mocks.authenticate).toHaveBeenCalledTimes(2); + expect(requiredNext).not.toHaveBeenCalled(); + expect(requiredResponse.status).toHaveBeenCalledWith(401); + }); + + it('cannot serve optionalAuth from a requireAuth cache after deletion', async () => { + const cookie = `sealed-required-poison-${Date.now()}`; + const requiredNext = vi.fn() as NextFunction; + await requireAuth(makeRequest(cookie), makeResponse(), requiredNext); + expect(requiredNext).toHaveBeenCalledTimes(1); + + mocks.readCredentialAuthorizationLifecycle.mockResolvedValue({ + status: 'terminal', reason: 'deleted_or_quarantined', + }); + const optionalRequest = makeRequest(cookie); + const optionalNext = vi.fn() as NextFunction; + await optionalAuth(optionalRequest, makeResponse(), optionalNext); + + expect(mocks.authenticate).toHaveBeenCalledTimes(2); + expect(optionalNext).toHaveBeenCalledTimes(1); + expect(optionalRequest.user).toBeUndefined(); + }); }); diff --git a/server/tests/unit/credential-detach-db.test.ts b/server/tests/unit/credential-detach-db.test.ts new file mode 100644 index 00000000000..b956f83801e --- /dev/null +++ b/server/tests/unit/credential-detach-db.test.ts @@ -0,0 +1,196 @@ +import { beforeEach, describe, expect, it, vi } from 'vitest'; + +const mocks = vi.hoisted(() => ({ connect: vi.fn() })); +vi.mock('../../src/db/client.js', () => ({ getPool: () => ({ connect: mocks.connect }) })); + +import { CredentialDetachConflict, detachCredential, type DetachCredentialInput } from '../../src/db/credential-detach-db.js'; + +const input: DetachCredentialInput = { + hostUserId: 'user_host', credentialId: 'user_target', + expectedIdentityId: 'bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb', expectedAuthorizationEpoch: '7', + actorUserId: 'user_actor', actorCredentialId: 'user_actor', + actorIdentityId: 'aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa', +}; +const result = (rows: unknown[], rowCount: number | null = rows.length) => ({ rows, rowCount }); +type Fault = (params: unknown[]) => unknown; + +function clientWithFault(fragment = '', fault?: Fault) { + let auditId: unknown; + const client = { + release: vi.fn(), + query: vi.fn(async (sql: string, params: unknown[] = []) => { + if (fault && sql.includes(fragment)) return fault(params); + if (['BEGIN', 'COMMIT', 'ROLLBACK', 'SET CONSTRAINTS ALL IMMEDIATE'].includes(sql)) return result([]); + if (sql.includes('pg_try_advisory_lock')) return result([{ acquired: true }]); + if (sql.includes('pg_try_advisory_xact_lock')) return result([{ acquired: true }]); + if (sql.includes('pg_advisory_unlock')) return result([{ unlocked: true }]); + if (sql.includes('FROM admin_credential_bind_operations')) return result([]); + if (sql.includes('AS blocked')) return result([{ blocked: false }]); + if (sql.includes('SELECT id FROM identities')) return result([ + { id: input.actorIdentityId }, { id: input.expectedIdentityId }, + ]); + if (sql.includes('FROM identity_workos_users')) return result([ + { workos_user_id: input.actorCredentialId, identity_id: input.actorIdentityId, is_primary: true }, + { workos_user_id: input.hostUserId, identity_id: input.expectedIdentityId, is_primary: true }, + { workos_user_id: input.credentialId, identity_id: input.expectedIdentityId, is_primary: false }, + ]); + if (sql.includes('SELECT workos_user_id FROM users')) return result( + (params[0] as string[]).map((workos_user_id) => ({ workos_user_id })), + ); + if (sql.includes('SELECT workos_user_id, epoch::text FROM authorization_epochs')) return sql.includes('FOR UPDATE') + ? result([{ workos_user_id: input.credentialId, epoch: '7' }]) + : result([{ workos_user_id: input.hostUserId, epoch: '1' }, { workos_user_id: input.credentialId, epoch: '8' }]); + if (sql.includes('INSERT INTO identities')) return result([{ id: params[0] }]); + if (sql.includes('UPDATE identity_workos_users')) return result([ + { workos_user_id: input.credentialId, identity_id: params[0], is_primary: true }, + ]); + if (sql.includes('INSERT INTO authorization_epochs')) return result([ + { workos_user_id: input.hostUserId, epoch: '1' }, { workos_user_id: input.credentialId, epoch: '8' }, + ]); + if (sql.includes('INSERT INTO registry_audit_log')) { + auditId = params[0]; + return result([{ id: auditId }]); + } + if (sql.includes('FROM registry_audit_log')) return result([{ id: auditId, matches: true }]); + throw new Error(`Unexpected SQL: ${sql}`); + }), + }; + mocks.connect.mockResolvedValue(client); + return client; +} + +function expectRollback(client: ReturnType<typeof clientWithFault>) { + expect(client.query).toHaveBeenCalledWith('ROLLBACK'); + expect(client.query).not.toHaveBeenCalledWith('COMMIT'); + // A failed or uncertain session unlock cannot be returned to the pool. + expect(client.release).toHaveBeenCalledExactlyOnceWith(true); +} + +describe('credential detach transaction integrity', () => { + beforeEach(() => vi.clearAllMocks()); + + it('orders identity locks deterministically and verifies every unlock before commit', async () => { + const client = clientWithFault(); + const detached = await detachCredential(input); + expect(detached.affectedCredentialIds).toEqual([input.hostUserId, input.credentialId]); + const calls = client.query.mock.calls; + const locks = calls.filter(([sql]) => sql.includes('pg_try_advisory_lock')); + const unlocks = calls.filter(([sql]) => sql.includes('pg_advisory_unlock')); + expect(locks.map(([, params]) => params)).toEqual([ + [`identity-binding:${input.actorIdentityId}`], [`identity-binding:${input.expectedIdentityId}`], + ]); + expect(unlocks.map(([, params]) => params)).toEqual(locks.map(([, params]) => params).reverse()); + const lifecycleLocks = calls.filter(([sql]) => sql.includes('pg_try_advisory_xact_lock')); + expect(lifecycleLocks.map(([, params]) => params)).toEqual([ + [input.actorCredentialId], [input.hostUserId], [input.credentialId], + ]); + expect(calls.indexOf(lifecycleLocks[0])).toBeGreaterThan(calls.findIndex(([sql]) => sql.includes('FROM identity_workos_users'))); + expect(calls.indexOf(lifecycleLocks.at(-1)!)).toBeLessThan(calls.findIndex(([sql]) => sql.includes('FROM users'))); + expect(calls.at(-1)?.[0]).toBe('COMMIT'); + expect(calls.indexOf(unlocks.at(-1)!)).toBeLessThan(calls.findIndex(([sql]) => sql === 'COMMIT')); + expect(client.release).toHaveBeenCalledExactlyOnceWith(false); + }); + + const invalidAuditResults: [string, Fault][] = [ + ['rowCount 0 and no row (suppressed INSERT)', () => result([])], + ['rowCount 0 despite a returned row', ([id]) => result([{ id }], 0)], + ['rowCount greater than one', ([id]) => result([{ id }, { id }], 2)], + ['rowCount greater than one despite one returned row', ([id]) => result([{ id }], 2)], + ['NULL rowCount', ([id]) => result([{ id }], null)], + ['rowCount 1 with no row', () => result([], 1)], + ['NULL returned row', () => result([null])], + ['NULL returned id', () => result([{ id: null }])], + ['missing returned id', () => result([{}])], + ['different returned id', () => result([{ id: 'wrong-id' }])], + ['thrown INSERT error', () => { throw new Error('injected audit failure'); }], + ]; + it.each(invalidAuditResults)('rolls back and discards the connection for audit %s', async (_name, fault) => { + const client = clientWithFault('INSERT INTO registry_audit_log', fault); + await expect(detachCredential(input)).rejects.toThrow(); + expectRollback(client); + }); + + it.each(['pg_try_advisory_lock', 'pg_try_advisory_xact_lock', 'pg_advisory_unlock'])('rejects every invalid %s outcome', async (operation) => { + const column = operation === 'pg_advisory_unlock' ? 'unlocked' : 'acquired'; + for (const badResult of [result([{ [column]: false }]), result([{ [column]: null }]), + result([]), result([{ [column]: true }], 0), result([{ [column]: true }], null), + result([{ [column]: true }, { [column]: true }], 2)]) { + const client = clientWithFault(operation, () => badResult); + await expect(detachCredential(input)).rejects.toThrow(); + expectRollback(client); + } + const client = clientWithFault(operation, () => { throw new Error('injected lock failure'); }); + await expect(detachCredential(input)).rejects.toThrow('injected lock failure'); + expectRollback(client); + }); + + it.each([result([]), result([{ id: 'duplicate' }, { id: 'duplicate' }]), result([{ id: null }])])( + 'rejects an audit removed or duplicated after INSERT: %j', async (stored) => { + const client = clientWithFault("WHERE details->>'operation_id'", () => stored); + await expect(detachCredential(input)).rejects.toThrow(); + expectRollback(client); + }, + ); + + it('keeps the original failure and destroys the connection when rollback also fails', async () => { + const client = clientWithFault('INSERT INTO registry_audit_log', () => { throw new Error('audit unavailable'); }); + const query = client.query.getMockImplementation()!; + client.query.mockImplementation(async (sql, params) => { + if (sql === 'ROLLBACK') throw new Error('connection lost'); + return query(sql, params); + }); + await expect(detachCredential(input)).rejects.toThrow('audit unavailable'); + expectRollback(client); + }); + + it('maps NOWAIT contention to a conflict and rolls back', async () => { + const client = clientWithFault('SELECT id FROM identities', () => { + throw Object.assign(new Error('could not obtain lock'), { code: '55P03' }); + }); + await expect(detachCredential(input)).rejects.toBeInstanceOf(CredentialDetachConflict); + expectRollback(client); + }); + + it('requires every affected authorization epoch to advance', async () => { + const client = clientWithFault('INSERT INTO authorization_epochs', () => result([ + { workos_user_id: input.credentialId, epoch: '8' }, + ])); + await expect(detachCredential(input)).rejects.toThrow('authorization epochs'); + expectRollback(client); + expect(client.query.mock.calls.some(([sql]) => sql.includes('INSERT INTO registry_audit_log'))).toBe(false); + }); + + it.each([result([{ blocked: true }]), result([{ blocked: null }]), result([{}]), result([]), + result([{ blocked: false }], 0), result([{ blocked: false }], null), + result([{ blocked: false }, { blocked: false }], 2)])('fails closed on unavailable/terminal lifecycle evidence: %j', async (marker) => { + const client = clientWithFault('AS blocked', () => marker); + await expect(detachCredential(input)).rejects.toThrow(); + expectRollback(client); + expect(client.query.mock.calls.some(([sql]) => sql.includes('UPDATE identity_workos_users'))).toBe(false); + }); + + it('rechecks lifecycle evidence after deferred triggers and rolls back late quarantine', async () => { + let checks = 0; + const client = clientWithFault('AS blocked', () => result([{ blocked: ++checks > 1 }])); + await expect(detachCredential(input)).rejects.toBeInstanceOf(CredentialDetachConflict); + expectRollback(client); + expect(client.query.mock.calls.some(([sql]) => sql.includes('INSERT INTO registry_audit_log'))).toBe(true); + }); + + it.each([result([], null), result([], 1), result([{ id: 'operation', provider_user_id: input.credentialId, status: 'reconciliation_required' }]), + result([{ id: 'operation', provider_user_id: 'unexpected', status: 'committed' }])])( + 'fails closed on uncertain operation evidence: %j', async (operations) => { + const client = clientWithFault('FROM admin_credential_bind_operations', () => operations); + await expect(detachCredential(input)).rejects.toThrow(); + expectRollback(client); + }, + ); + + it('fails closed when the canonical actor user row is missing', async () => { + const client = clientWithFault('SELECT workos_user_id FROM users', () => result([ + { workos_user_id: input.hostUserId }, { workos_user_id: input.credentialId }, + ])); + await expect(detachCredential(input)).rejects.toBeInstanceOf(CredentialDetachConflict); + expectRollback(client); + }); +}); diff --git a/server/tests/unit/identity-mutation-containment.test.ts b/server/tests/unit/identity-mutation-containment.test.ts new file mode 100644 index 00000000000..39a825ef937 --- /dev/null +++ b/server/tests/unit/identity-mutation-containment.test.ts @@ -0,0 +1,86 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; + +const { getPool } = vi.hoisted(() => ({ getPool: vi.fn() })); +vi.mock('../../src/db/client.js', () => ({ getPool })); +vi.mock('../../src/logger.js', () => ({ + createLogger: () => ({ info: vi.fn(), warn: vi.fn(), error: vi.fn() }), +})); +vi.mock('../../src/addie/error-notifier.js', () => ({ notifySystemError: vi.fn() })); + +import { mergeUsers, previewUserMerge } from '../../src/db/user-merge-db.js'; +import { promoteSecondaryIfPrimaryDeleted } from '../../src/db/identity-db.js'; +import { IdentityMutationDisabledError } from '../../src/db/identity-mutation-policy.js'; + +const refusal = { + code: 'identity_mutation_disabled', + message: 'Identity consolidation is disabled until authority and provenance can be preserved.', +}; + +describe('identity mutation containment at the database seam', () => { + beforeEach(() => { + getPool.mockReset(); + getPool.mockImplementation(() => { + throw new Error('The containment guard must refuse before accessing the database'); + }); + }); + + afterEach(() => { + expect(getPool).not.toHaveBeenCalled(); + }); + + it.each([ + ['automatic', 'user_sam', 'user_jordan', 'auto'], + ['member', 'user_sam', 'user_jordan', 'user_sam'], + ['admin', 'user_jordan', 'user_sam', 'user_admin'], + ['same credential', 'user_sam', 'user_sam', 'user_sam'], + ['missing credential', '', 'user_jordan', 'user_admin'], + ])('refuses %s merges before opening a connection', async (_caller, primary, secondary, actor) => { + await expect(mergeUsers(primary, secondary, actor)).rejects.toMatchObject(refusal); + }); + + it('rejects promotion and operator confirmation options without a bypass', async () => { + await expect(mergeUsers('user_sam', 'user_jordan', 'user_admin', { + ensurePrimaryFlag: true, + auditContext: { consolidation_confirmed: true, consolidate: true }, + })).rejects.toBeInstanceOf(IdentityMutationDisabledError); + }); + + it.each(['user_sam', 'user_jordan', ''])('refuses automatic primary promotion for %j', async (userId) => { + await expect(promoteSecondaryIfPrimaryDeleted(userId)).rejects.toMatchObject(refusal); + }); + + it('keeps simultaneous, reversed, and replayed operations harmless', async () => { + const attempts = Array.from({ length: 12 }, (_, index) => index % 3 === 0 + ? promoteSecondaryIfPrimaryDeleted('user_jordan') + : mergeUsers( + index % 2 ? 'user_sam' : 'user_jordan', + index % 2 ? 'user_jordan' : 'user_sam', + 'user_admin', + { ensurePrimaryFlag: true, auditContext: { consolidation_confirmed: true } }, + )); + for (const result of await Promise.allSettled(attempts)) { + expect(result.status).toBe('rejected'); + if (result.status === 'rejected') expect(result.reason).toMatchObject(refusal); + } + await expect(mergeUsers('user_sam', 'user_jordan', 'user_admin')).rejects.toMatchObject(refusal); + }); +}); + +describe('read-only merge detection', () => { + it('keeps preview available and issues only reads', async () => { + const query = vi.fn().mockResolvedValue({ rows: [{ count: '1' }] }); + const release = vi.fn(); + getPool.mockReset(); + getPool.mockReturnValue({ connect: vi.fn().mockResolvedValue({ query, release }) }); + + const preview = await previewUserMerge('user_sam', 'user_jordan'); + + expect(preview.tables).toContainEqual({ table_name: 'organization_memberships', row_count: 1 }); + expect(query).toHaveBeenCalled(); + for (const [sql, parameters] of query.mock.calls) { + expect(sql).toMatch(/^SELECT COUNT\(\*\)/); + expect(parameters).toEqual(['user_jordan']); + } + expect(release).toHaveBeenCalledOnce(); + }); +}); diff --git a/server/tests/unit/identity-mutation-route-containment.test.ts b/server/tests/unit/identity-mutation-route-containment.test.ts new file mode 100644 index 00000000000..3d38bc962ce --- /dev/null +++ b/server/tests/unit/identity-mutation-route-containment.test.ts @@ -0,0 +1,191 @@ +import { afterAll, beforeEach, describe, expect, it, vi } from 'vitest'; +import express, { type NextFunction, type Request, type Response } from 'express'; +import request from 'supertest'; + +const binding = vi.hoisted(() => ({ createAndBindAdminCredential: vi.fn() })); +const mocks = vi.hoisted(() => ({ + getPool: vi.fn(), + getWorkos: vi.fn(), + query: vi.fn(), + createUser: vi.fn(), + updateUser: vi.fn(), + deleteUser: vi.fn(), + getUser: vi.fn(), + sendEmailLinkVerification: vi.fn(), +})); + +vi.hoisted(() => { + process.env.WORKOS_API_KEY ??= 'sk_test_identity_containment'; + process.env.WORKOS_CLIENT_ID ??= 'client_test_identity_containment'; + process.env.WORKOS_COOKIE_PASSWORD ??= 'test-cookie-password-at-least-32-chars-long'; +}); + +// Exercise the registered routers after the authentication boundary, without a +// database. Any handler that reaches a provider or local data seam fails here. +vi.mock('../../src/middleware/auth.js', async (importOriginal) => { + const original = await importOriginal<typeof import('../../src/middleware/auth.js')>(); + const authenticate = (req: Request, res: Response, next: NextFunction) => { + if (req.headers['x-test-unauthenticated']) return res.sendStatus(401); + req.user = { + id: String(req.headers['x-test-user'] || 'user_member'), + authWorkosUserId: 'user_signed_in_credential', + identityId: '00000000-0000-4000-8000-000000000682', + email: 'member@example.test', + emailVerified: true, + createdAt: new Date(0).toISOString(), + updatedAt: new Date(0).toISOString(), + }; + req.adminAccessMechanism = req.headers['x-test-admin-access-mechanism'] as typeof req.adminAccessMechanism; + next(); + }; + return { ...original, requireAuth: authenticate, requireGlobalAdmin: [authenticate] }; +}); + +vi.mock('../../src/services/admin-credential-bind.js', () => binding); + +vi.mock('../../src/db/client.js', async (importOriginal) => ({ + ...(await importOriginal<typeof import('../../src/db/client.js')>()), + getPool: mocks.getPool, + query: mocks.query, +})); +vi.mock('../../src/auth/workos-client.js', () => { + const workos = { userManagement: { + createUser: mocks.createUser, + updateUser: mocks.updateUser, + deleteUser: mocks.deleteUser, + getUser: mocks.getUser, + } }; + return { workos, getWorkos: () => { mocks.getWorkos(); return workos; } }; +}); +vi.mock('../../src/routes/workos-webhooks.js', () => ({ + backfillOrganizationMemberships: vi.fn(), + backfillUsers: vi.fn(), + backfillOrganizationDomains: vi.fn(), +})); +vi.mock('../../src/notifications/email.js', () => ({ + sendEmailLinkVerification: mocks.sendEmailLinkVerification, + sendSlackInviteEmail: vi.fn(), + hasSlackInviteBeenSent: vi.fn(), +})); + +import { createAdminUsersRouter } from '../../src/routes/admin/users.js'; +import { createAccountLinkingRouter } from '../../src/routes/account-linking.js'; +import { stopAuthTimers } from '../../src/middleware/auth.js'; + +const app = express(); +app.use(express.json()); +app.use('/api/admin/users', createAdminUsersRouter()); +app.use('/api/me/linked-emails', createAccountLinkingRouter()); + +const refusal = { + error: 'identity_mutation_disabled', + message: 'Identity consolidation is disabled until authority and provenance can be preserved.', +}; +const operations = [ + { method: 'post', path: '/api/admin/users/user_member/credentials', body: { workos_user_id: 'user_admin' } }, + { method: 'post', path: '/api/admin/users/user_admin/credentials', body: { workos_user_id: 'user_member' } }, + { method: 'post', path: '/api/admin/users/user_member/credentials/user_admin/promote', body: {} }, + { method: 'post', path: '/api/admin/users/user_admin/credentials/user_member/promote', body: {} }, + { method: 'post', path: '/api/admin/users/user_missing/credentials/user_missing/promote', body: {} }, + { method: 'put', path: '/api/me/linked-emails/primary', body: { email: 'admin@example.test' } }, +] as const; + +function expectNoDataAccess() { + for (const mock of Object.values(mocks)) expect(mock).not.toHaveBeenCalled(); +} + +describe('identity mutation route containment', () => { + beforeEach(() => { + vi.resetAllMocks(); + mocks.getPool.mockImplementation(() => { throw new Error('Unexpected database pool access'); }); + mocks.query.mockImplementation(() => { throw new Error('Unexpected database query'); }); + }); + + it.each(operations)('refuses $method $path before provider or local access', async ({ method, path, body }) => { + for (const consolidate of [false, true]) { + for (const mechanism of ['sso', 'static_admin_api_key']) { + const response = await request(app)[method](path) + .set('X-Test-Admin-Access-Mechanism', mechanism) + .send({ ...body, consolidate }) + .expect(409); + expect(response.body).toEqual(refusal); + expectNoDataAccess(); + } + } + }); + + it.each(operations)('still authenticates $method $path', async ({ method, path, body }) => { + await request(app)[method](path) + .set('X-Test-Unauthenticated', 'true') + .send(body) + .expect(401); + expectNoDataAccess(); + }); + + it('refuses concurrent and replayed calls in both identity directions', async () => { + const invoke = ({ method, path, body }: typeof operations[number]) => request(app)[method](path) + .send({ ...body, consolidate: true }) + .expect(409) + .expect(({ body }) => expect(body).toEqual(refusal)); + await Promise.all([...operations, ...operations].map(invoke)); + for (const operation of operations) await invoke(operation); + expectNoDataAccess(); + }); + + it.each([{}, { email: null }, { email: 'member@example.test' }, { workos_user_id: 'invalid' }])( + 'cannot bypass refusal through missing, same-identity, or invalid input: %j', async (body) => { + for (const { method, path } of operations) { + await request(app)[method](path).send(body).expect(409) + .expect(({ body }) => expect(body).toEqual(refusal)); + } + expectNoDataAccess(); + }, + ); + + it('routes fresh admin creation to the compensating binding service with the exact actor', async () => { + binding.createAndBindAdminCredential.mockResolvedValue({ status: 201, body: { bound: true } }); + await request(app).post('/api/admin/users/user_member/linked-emails') + .send({ email: 'NEW@example.test' }).expect(201); + expect(binding.createAndBindAdminCredential).toHaveBeenCalledExactlyOnceWith({ + hostUserId: 'user_member', email: 'new@example.test', + actorUserId: 'user_signed_in_credential', + actorIdentityId: '00000000-0000-4000-8000-000000000682', + }); + expectNoDataAccess(); + }); + + it.each([{ consolidate: true }, { promote: true }])('contains destructive create-and-bind options: %j', async (body) => { + await request(app).post('/api/admin/users/user_member/linked-emails') + .send({ email: 'new@example.test', ...body }).expect(409); + expect(binding.createAndBindAdminCredential).not.toHaveBeenCalled(); + expectNoDataAccess(); + }); + + it('keeps fresh admin creation behind authentication and an identity-bearing actor', async () => { + await request(app).post('/api/admin/users/user_member/linked-emails') + .set('X-Test-Unauthenticated', 'true').send({ email: 'new@example.test' }).expect(401); + await request(app).post('/api/admin/users/user_member/linked-emails') + .set('X-Test-Admin-Access-Mechanism', 'static_admin_api_key') + .send({ email: 'new@example.test' }).expect(403); + expect(binding.createAndBindAdminCredential).not.toHaveBeenCalled(); + expectNoDataAccess(); + }); + + it('refuses a member attempt to combine existing accounts without issuing a token or email', async () => { + mocks.query.mockImplementation(async (sql: string) => { + if (sql.includes('COUNT(*)')) return { rows: [{ count: '0' }] }; + if (sql.includes('FROM users')) return { rows: [{ workos_user_id: 'user_existing', email: 'existing@example.test' }] }; + if (sql.trimStart().startsWith('SELECT')) return { rows: [] }; + throw new Error(`Unexpected mutation: ${sql}`); + }); + const response = await request(app).post('/api/me/linked-emails') + .send({ email: 'existing@example.test', consolidate: true }).expect(409); + expect(response.body.error).toBe('This email already has an AAO account'); + expect(mocks.query.mock.calls.every(([sql]) => sql.trimStart().startsWith('SELECT'))).toBe(true); + expect(mocks.sendEmailLinkVerification).not.toHaveBeenCalled(); + expect(mocks.getPool).not.toHaveBeenCalled(); + for (const method of [mocks.createUser, mocks.updateUser, mocks.deleteUser]) expect(method).not.toHaveBeenCalled(); + }); +}); + +afterAll(() => stopAuthTimers()); diff --git a/server/tests/unit/login-name-preservation.test.ts b/server/tests/unit/login-name-preservation.test.ts index 81de4fa45c5..5c3e77c3c28 100644 --- a/server/tests/unit/login-name-preservation.test.ts +++ b/server/tests/unit/login-name-preservation.test.ts @@ -18,6 +18,9 @@ import * as path from 'path'; const SRC_DIR = path.resolve(__dirname, '../../src'); const HTTP_FILE = path.join(SRC_DIR, 'http.ts'); +const IDENTITY_DB_FILE = path.join(SRC_DIR, 'db/identity-db.ts'); +const WEBHOOK_FILE = path.join(SRC_DIR, 'routes/workos-webhooks.ts'); +const MCP_OAUTH_FILE = path.join(SRC_DIR, 'mcp/oauth-provider.ts'); const MEMBERSHIP_DB_FILE = path.join(SRC_DIR, 'db/membership-db.ts'); const ADMIN_ORGS_FILE = path.join(SRC_DIR, 'routes/admin/organizations.ts'); @@ -55,7 +58,14 @@ function findUnconditionalNameOverwrites(filePath: string): string[] { describe('auth callback name preservation', () => { const httpContent = fs.readFileSync(HTTP_FILE, 'utf-8'); - const authCallbackSection = extractAuthCallbackUpsert(httpContent); + const identityDbContent = fs.readFileSync(IDENTITY_DB_FILE, 'utf-8'); + const authCallbackSection = extractCredentialEventUpsert(identityDbContent); + + it('routes the auth callback through the credential-event upsert', () => { + const callback = extractAuthCallback(httpContent); + expect(callback).toContain('upsertWorkosUserInCredentialEvent'); + expect(callback).toContain("'preserve_existing'"); + }); it('should not unconditionally overwrite first_name on login', () => { expect(authCallbackSection).not.toContain('first_name = EXCLUDED.first_name'); @@ -69,6 +79,19 @@ describe('auth callback name preservation', () => { expect(authCallbackSection).toMatch(/COALESCE.*users\.first_name.*EXCLUDED\.first_name/s); expect(authCallbackSection).toMatch(/COALESCE.*users\.last_name.*EXCLUDED\.last_name/s); }); + + it('keeps login and provider-event name precedence explicit at every shared-helper caller', () => { + const webhookContent = fs.readFileSync(WEBHOOK_FILE, 'utf-8'); + const mcpOauthContent = fs.readFileSync(MCP_OAUTH_FILE, 'utf-8'); + expect(webhookContent).toMatch( + /upsertWorkosUserInCredentialEvent\([\s\S]*?'provider_authoritative'/, + ); + expect(mcpOauthContent).toMatch( + /upsertWorkosUserInCredentialEvent\([\s\S]*?'preserve_existing'/, + ); + expect(identityDbContent).toContain("'provider_authoritative'"); + expect(identityDbContent).toContain("'preserve_existing'"); + }); }); describe('/api/me name source', () => { @@ -109,12 +132,22 @@ describe('no unconditional name overwrites anywhere in runtime code', () => { // --- helpers --- -function extractAuthCallbackUpsert(content: string): string { +function extractAuthCallback(content: string): string { const callbackStart = content.indexOf("'/auth/callback'"); if (callbackStart === -1) throw new Error('Could not find /auth/callback handler'); - const insertStart = content.indexOf('INSERT INTO users', callbackStart); - if (insertStart === -1) throw new Error('Could not find INSERT INTO users in auth callback'); + const callbackEnd = content.indexOf("// GET /auth/logout", callbackStart); + if (callbackEnd === -1) throw new Error('Could not find end of /auth/callback handler'); + + return content.slice(callbackStart, callbackEnd); +} + +function extractCredentialEventUpsert(content: string): string { + const helperStart = content.indexOf('export async function upsertWorkosUserInCredentialEvent'); + if (helperStart === -1) throw new Error('Could not find credential-event user upsert'); + + const insertStart = content.indexOf('INSERT INTO users', helperStart); + if (insertStart === -1) throw new Error('Could not find INSERT INTO users in credential-event upsert'); const onConflict = content.indexOf('ON CONFLICT', insertStart); if (onConflict === -1) throw new Error('Could not find ON CONFLICT in auth callback upsert'); diff --git a/server/tests/unit/native-auth-http-wiring.test.ts b/server/tests/unit/native-auth-http-wiring.test.ts index c1f4b4b55d5..456d4e8150a 100644 --- a/server/tests/unit/native-auth-http-wiring.test.ts +++ b/server/tests/unit/native-auth-http-wiring.test.ts @@ -2,6 +2,14 @@ import { afterEach, describe, expect, it, vi } from 'vitest'; import request from 'supertest'; const mocks = vi.hoisted(() => ({ + query: vi.fn().mockResolvedValue({ rows: [], rowCount: 0 }), + listUsers: vi.fn(), + createUser: vi.fn(), + updateUser: vi.fn(), + deleteUser: vi.fn(), + createOrganizationMembership: vi.fn(), + updateOrganizationMembership: vi.fn(), + deleteOrganizationMembership: vi.fn(), getAuthorizationUrl: vi.fn(), authenticateWithCode: vi.fn(), listOrganizationMemberships: vi.fn(), @@ -27,6 +35,13 @@ vi.mock('@workos-inc/node', () => ({ DomainDataState: { Verified: 'verified' }, WorkOS: class WorkOS { userManagement = { + listUsers: mocks.listUsers, + createUser: mocks.createUser, + updateUser: mocks.updateUser, + deleteUser: mocks.deleteUser, + createOrganizationMembership: mocks.createOrganizationMembership, + updateOrganizationMembership: mocks.updateOrganizationMembership, + deleteOrganizationMembership: mocks.deleteOrganizationMembership, getAuthorizationUrl: mocks.getAuthorizationUrl, authenticateWithCode: mocks.authenticateWithCode, listOrganizationMemberships: mocks.listOrganizationMemberships, @@ -49,14 +64,35 @@ vi.mock('../../src/config.js', async () => { vi.mock('../../src/db/client.js', () => ({ initializeDatabase: vi.fn(), getPool: vi.fn().mockReturnValue({ - query: vi.fn().mockResolvedValue({ rows: [], rowCount: 0 }), + query: mocks.query, }), isDatabaseInitialized: vi.fn().mockReturnValue(false), closeDatabase: vi.fn(), healthCheck: vi.fn().mockResolvedValue(undefined), - query: vi.fn().mockResolvedValue({ rows: [], rowCount: 0 }), + query: mocks.query, })); +vi.mock('../../src/db/identity-db.js', async (importOriginal) => { + const actual = await importOriginal<typeof import('../../src/db/identity-db.js')>(); + return { + ...actual, + withCredentialCreationEventMutation: vi.fn(async ( + _userId: string, + mutation: (client: { query: typeof mocks.query }) => Promise<unknown>, + ) => ({ + applied: true, + value: await mutation({ query: mocks.query }), + })), + upsertWorkosUserInCredentialEvent: vi.fn(async ( + client: { query: typeof mocks.query }, + user: { id: string }, + ) => client.query( + 'INSERT INTO users (workos_user_id) VALUES ($1) ON CONFLICT DO NOTHING', + [user.id], + )), + }; +}); + vi.mock('../../src/db/migrate.js', () => ({ runMigrations: vi.fn().mockResolvedValue(undefined), })); @@ -340,7 +376,10 @@ describe('native OAuth HTTPServer wiring', () => { slack_user_id: 'U123', workos_user_id: null, } as never); - const mapUser = vi.spyOn(SlackDatabase.prototype, 'mapUser').mockResolvedValueOnce(undefined); + const mapUser = vi.spyOn(SlackDatabase.prototype, 'mapUser').mockResolvedValueOnce({ + slack_user_id: 'U123', + workos_user_id: 'user_1', + } as never); server = new HTTPServer(); const response = await request(appFor(server)).get('/auth/callback').query({ @@ -389,7 +428,10 @@ describe('native OAuth HTTPServer wiring', () => { slack_user_id: 'U123', workos_user_id: null, } as never); - vi.spyOn(SlackDatabase.prototype, 'mapUser').mockResolvedValueOnce(undefined); + vi.spyOn(SlackDatabase.prototype, 'mapUser').mockResolvedValueOnce({ + slack_user_id: 'U123', + workos_user_id: 'user_1', + } as never); mocks.sendAccountLinkedMessage.mockRejectedValueOnce(new Error('synthetic Slack rejection')); server = new HTTPServer(); @@ -404,4 +446,45 @@ describe('native OAuth HTTPServer wiring', () => { expect(response.status).toBe(302); expect(mocks.sendAccountLinkedMessage).toHaveBeenCalledWith(origin, 'Person'); }); + + it.each([ + ['local', 'user_sam', 'sam.adeyemi@gmail.com', 'user_jordan', 'sam.adeyemi@googlemail.com'], + ['provider', 'user_jordan', 'sam.adeyemi@googlemail.com', 'user_sam', 'sam.adeyemi@gmail.com'], + ])('only detects aliases during concurrent/replayed OAuth callbacks (%s)', async (location, id, email, otherId, otherEmail) => { + mocks.authenticateWithCode.mockResolvedValue({ + sealedSession: 'test-sealed-session', + user: { id, email, firstName: 'Sam', lastName: 'Adeyemi', emailVerified: true, + createdAt: '2026-01-01T00:00:00.000Z', updatedAt: '2026-01-01T00:00:00.000Z' }, + }); + mocks.listOrganizationMemberships.mockResolvedValue({ data: [ + { organizationId: 'org_pinnacle', userId: id, status: 'active' }, + ] }); + mocks.listUsers.mockResolvedValue({ data: [{ id: otherId, email: otherEmail }] }); + mocks.query.mockImplementation(async (sql: string) => ({ + rows: sql.includes('FROM users u') && sql.includes('user_email_aliases') && location === 'local' + ? [{ workos_user_id: otherId, email: otherEmail }] : [], + rowCount: 0, + })); + server = new HTTPServer(); + const callback = () => request(appFor(server!)).get('/auth/callback').query({ + code: 'test-code', state: JSON.stringify({ return_to: '/member-hub', consolidate: true }), + }); + const responses = await Promise.all([callback(), callback(), callback()]); + responses.push(await callback()); + for (const response of responses) { + expect(response.status).toBe(302); + expect(response.headers.location).toBe(`/member-hub?duplicate_email=${encodeURIComponent(otherEmail)}`); + } + for (const write of [mocks.createUser, mocks.updateUser, mocks.deleteUser, + mocks.createOrganizationMembership, mocks.updateOrganizationMembership, mocks.deleteOrganizationMembership]) { + expect(write).not.toHaveBeenCalled(); + } + expect(mocks.listOrganizationMemberships.mock.calls.every(([input]) => input.userId === id)).toBe(true); + const writes = mocks.query.mock.calls.filter(([sql]) => /^\s*(INSERT|UPDATE|DELETE)\b/i.test(sql)); + // Ordinary login may upsert the authenticating user. Alias detection must + // never synthesize the duplicate or mutate either credential's authority. + expect(writes.every(([sql, params]) => sql.trimStart().startsWith('INSERT INTO users ') && params[0] === id)).toBe(true); + expect(mocks.query.mock.calls.some(([sql]) => sql.includes('FROM users u') && sql.includes('user_email_aliases'))).toBe(true); + }); + }); diff --git a/server/tests/unit/network-health-security.test.ts b/server/tests/unit/network-health-security.test.ts index 94f35fa58c4..309ef8f76af 100644 --- a/server/tests/unit/network-health-security.test.ts +++ b/server/tests/unit/network-health-security.test.ts @@ -20,6 +20,7 @@ const mocks = vi.hoisted(() => ({ upsertAlertRule: vi.fn(), resolveAlert: vi.fn(), serveHtmlWithConfig: vi.fn(), + readCredentialAuthorizationLifecycle: vi.fn(), })); vi.hoisted(() => { @@ -56,6 +57,11 @@ vi.mock('../../src/db/client.js', async (importOriginal) => ({ getPool: () => ({ query: mocks.poolQuery }), })); +vi.mock('../../src/db/authorization-epoch-db.js', async (importOriginal) => ({ + ...(await importOriginal<typeof import('../../src/db/authorization-epoch-db.js')>()), + readCredentialAuthorizationLifecycle: mocks.readCredentialAuthorizationLifecycle, +})); + vi.mock('../../src/addie/mcp/admin-tools.js', () => ({ isWebUserAAOAdmin: mocks.isWebUserAAOAdmin, })); @@ -181,6 +187,20 @@ describe('network-health global authorization boundary', () => { mocks.resolveEffectiveMembership.mockResolvedValue({ is_member: true }); mocks.checkPlatformBanForApiKey.mockResolvedValue({ banned: false }); mocks.checkPlatformBan.mockResolvedValue({ banned: false }); + mocks.readCredentialAuthorizationLifecycle.mockImplementation((workosUserId: string) => + Promise.resolve({ + status: 'active', + snapshot: { + workos_user_id: workosUserId, + email: 'user@example.test', + first_name: 'Regular', + last_name: 'User', + identity_id: '00000000-0000-4000-8000-000000000001', + primary_workos_user_id: workosUserId, + fingerprint: '', + }, + }), + ); mocks.isWebUserAAOAdmin.mockResolvedValue(false); mocks.poolQuery.mockImplementation((sql: string) => { if (sql.includes('FROM users')) { diff --git a/server/tests/unit/set-primary-email.test.ts b/server/tests/unit/set-primary-email.test.ts index ec783dfc3b5..e25d5016c68 100644 --- a/server/tests/unit/set-primary-email.test.ts +++ b/server/tests/unit/set-primary-email.test.ts @@ -3,79 +3,13 @@ import * as fs from 'fs'; import * as path from 'path'; import { isEmailUnavailable } from '../../src/routes/account-linking-errors.js'; -/** - * Static analysis test for the set-primary-email endpoint. - * Verifies the route handler exists and follows the correct swap pattern. - */ - -const ACCOUNT_LINKING_FILE = path.resolve( - __dirname, - '../../src/routes/account-linking.ts' -); - const DASHBOARD_SETTINGS_FILE = path.resolve( __dirname, '../../public/dashboard-settings.html' ); -describe('Set primary email endpoint', () => { - const source = fs.readFileSync(ACCOUNT_LINKING_FILE, 'utf-8'); - - it('registers a PUT /primary route on the router', () => { - expect(source).toMatch(/router\.put\(\s*['"]\/primary['"]/); - }); - - it('requires authentication and rate limiting', () => { - expect(source).toMatch(/router\.put\(\s*['"]\/primary['"],\s*requireAuth,\s*verifyExecuteLimiter/); - }); - - it('validates email is a verified alias before swapping', () => { - expect(source).toMatch(/user_email_aliases/); - expect(source).toMatch(/workos_user_id = \$1 AND LOWER\(email\) = \$2 AND verified_at IS NOT NULL/); - }); - - it('updates WorkOS as source of truth', () => { - expect(source).toMatch(/getWorkos\(\)\.userManagement\.updateUser/); - }); - - it('swaps emails in a transaction', () => { - // Must use BEGIN/COMMIT for atomicity - expect(source).toMatch(/BEGIN/); - expect(source).toMatch(/COMMIT/); - // Must update users table - expect(source).toMatch(/UPDATE users SET email/); - // Must delete alias for new primary - expect(source).toMatch(/DELETE FROM user_email_aliases/); - // Must insert old primary as alias - expect(source).toMatch(/INSERT INTO user_email_aliases[\s\S]*?VALUES/); - }); - - it('locks alias row with FOR UPDATE to prevent races', () => { - expect(source).toMatch(/FOR UPDATE/); - }); - - it('updates WorkOS before the DB swap so a WorkOS rejection leaves DB state untouched', () => { - const beginIdx = source.indexOf("'BEGIN'"); - const workosIdx = source.indexOf('getWorkos().userManagement.updateUser'); - expect(workosIdx).toBeGreaterThan(-1); - expect(beginIdx).toBeGreaterThan(-1); - expect(workosIdx).toBeLessThan(beginIdx); - }); - - it('classifies WorkOS rejections via isEmailUnavailable and returns a friendly 409', () => { - expect(source).toMatch(/isEmailUnavailable\(workosError\)/); - expect(source).toMatch(/already associated with another account/); - }); - - it('rolls back on error', () => { - expect(source).toMatch(/ROLLBACK/); - }); - - it('refreshes denormalized email on organization_memberships and person_relationships', () => { - expect(source).toMatch(/UPDATE organization_memberships SET email/); - expect(source).toMatch(/UPDATE person_relationships SET email/); - }); -}); +// The executable identity-mutation-route-containment suite covers this route's +// refusal before provider and database access. Keep helper/UI coverage here. describe('isEmailUnavailable', () => { it('matches WorkOS GenericServerException with "This email is not available" message', () => { diff --git a/server/tests/unit/working-group-admin-global-auth.test.ts b/server/tests/unit/working-group-admin-global-auth.test.ts index 7c1623375bc..3612580f6a7 100644 --- a/server/tests/unit/working-group-admin-global-auth.test.ts +++ b/server/tests/unit/working-group-admin-global-auth.test.ts @@ -22,6 +22,7 @@ const mocks = vi.hoisted(() => ({ removeMembership: vi.fn(), invalidateMemberContextCache: vi.fn(), invalidateWebAdminStatusCache: vi.fn(), + readCredentialAuthorizationLifecycle: vi.fn(), })); vi.hoisted(() => { @@ -58,6 +59,11 @@ vi.mock('../../src/db/client.js', async (importOriginal) => ({ getPool: () => ({ query: mocks.poolQuery }), })); +vi.mock('../../src/db/authorization-epoch-db.js', async (importOriginal) => ({ + ...(await importOriginal<typeof import('../../src/db/authorization-epoch-db.js')>()), + readCredentialAuthorizationLifecycle: mocks.readCredentialAuthorizationLifecycle, +})); + vi.mock('../../src/addie/mcp/admin-tools.js', async () => { const { isWebUserAAOAdmin } = await import( '../../src/addie/admin-status-lookup.js' @@ -129,6 +135,20 @@ describe('working-group real global-admin boundary', () => { mocks.resolveEffectiveMembership.mockResolvedValue({ is_member: true }); mocks.checkPlatformBanForApiKey.mockResolvedValue({ banned: false }); mocks.checkPlatformBan.mockResolvedValue({ banned: false }); + mocks.readCredentialAuthorizationLifecycle.mockImplementation((workosUserId: string) => + Promise.resolve({ + status: 'active', + snapshot: { + workos_user_id: workosUserId, + email: 'sso-admin@example.test', + first_name: 'SSO', + last_name: 'Admin', + identity_id: '00000000-0000-4000-8000-000000000001', + primary_workos_user_id: workosUserId, + fingerprint: '', + }, + }), + ); mocks.getWorkingGroupBySlug.mockImplementation((slug: string) => Promise.resolve(slug === 'aao-admin' ? { id: 'wg_aao_admin', slug: 'aao-admin' } diff --git a/server/tests/unit/workos-client-timeout.test.ts b/server/tests/unit/workos-client-timeout.test.ts index d11502cfecc..e8ec2f70443 100644 --- a/server/tests/unit/workos-client-timeout.test.ts +++ b/server/tests/unit/workos-client-timeout.test.ts @@ -48,6 +48,16 @@ describe('WorkOS client request budgets', () => { }); }); + it('bounds admin credential mutations to one attempt', async () => { + const { getAdminCredentialMutationWorkos } = await import('../../src/auth/workos-client.js'); + getAdminCredentialMutationWorkos(); + getAdminCredentialMutationWorkos(); + expect(mocks.constructWorkOS).toHaveBeenCalledOnce(); + expect(mocks.constructWorkOS).toHaveBeenCalledWith('sk_test_timeout', { + clientId: 'client_test_timeout', timeout: 10_000, maxRetries: 0, + }); + }); + it('makes one SDK attempt when retries are disabled', async () => { const { WorkOS } = await vi.importActual<typeof import('@workos-inc/node')>( '@workos-inc/node', diff --git a/server/tests/unit/workos-webhook-inactive-membership.test.ts b/server/tests/unit/workos-webhook-inactive-membership.test.ts index 766db18f53f..bcdc5e20ba9 100644 --- a/server/tests/unit/workos-webhook-inactive-membership.test.ts +++ b/server/tests/unit/workos-webhook-inactive-membership.test.ts @@ -15,7 +15,7 @@ describe('non-active organization_membership webhook handling', () => { )?.[0] ?? ''; it('routes non-active membership updates through local cache deletion only', () => { - expect(nonActiveBlock).toContain('deleteInactiveMembershipCache(membership)'); + expect(nonActiveBlock).toContain('deleteInactiveMembershipCache(membership, client)'); expect(nonActiveBlock).not.toContain('deleteMembership(membership)'); }); diff --git a/server/tests/unit/workos-webhook-user-deleted-wiring.test.ts b/server/tests/unit/workos-webhook-user-deleted-wiring.test.ts index f2cb6659273..93cd6bd099a 100644 --- a/server/tests/unit/workos-webhook-user-deleted-wiring.test.ts +++ b/server/tests/unit/workos-webhook-user-deleted-wiring.test.ts @@ -3,24 +3,18 @@ import * as fs from 'fs'; import * as path from 'path'; /** - * Static-analysis wiring assertions on the user.deleted webhook handler. - * - * Issue adcontextprotocol/adcp#3718: WorkOS-side deletion of a primary - * credential leaves multi-credential identities with zero primaries. The - * handler must promote a surviving secondary before the CASCADE fires, - * invalidate the 60-second session cache, and never return a 5xx on - * promotion failure (which would trigger WorkOS retry storms). - * - * These checks are static so a refactor that re-orders the calls or drops - * the invalidation fails fast without spinning up the full HTTP stack. + * Keep provider revocation wired without choosing a successor identity. + * Runtime DB-seam tests separately prove that promotion refuses before writes. */ const WEBHOOK_FILE = path.resolve(__dirname, '../../src/routes/workos-webhooks.ts'); -const IDENTITY_DB_FILE = path.resolve(__dirname, '../../src/db/identity-db.ts'); const webhookSource = fs.readFileSync(WEBHOOK_FILE, 'utf-8'); -const identitySource = fs.readFileSync(IDENTITY_DB_FILE, 'utf-8'); +const DELETION_SERVICE_FILE = path.resolve(__dirname, '../../src/services/identity-credential-deletion.ts'); +const deletionServiceSource = fs.readFileSync(DELETION_SERVICE_FILE, 'utf-8'); +const IDENTITY_DB_FILE = path.resolve(__dirname, '../../src/db/identity-db.ts'); +const identityDbSource = fs.readFileSync(IDENTITY_DB_FILE, 'utf-8'); -describe('user.deleted handler wiring (#3718)', () => { +describe('user.deleted containment wiring (#6827)', () => { const userDeletedBlockMatch = webhookSource.match( /case 'user\.deleted':\s*\{([\s\S]*?)\n\s{10}\}/, ); @@ -30,41 +24,73 @@ describe('user.deleted handler wiring (#3718)', () => { expect(block).not.toEqual(''); }); - it('calls promoteSecondaryIfPrimaryDeleted before deleteUser', () => { - const promoteIdx = block.indexOf('promoteSecondaryIfPrimaryDeleted'); - const deleteUserIdx = block.indexOf('deleteUser('); - expect(promoteIdx).toBeGreaterThanOrEqual(0); - expect(deleteUserIdx).toBeGreaterThan(promoteIdx); + it('preserves credential revocation without promoting a successor', () => { + expect(block).not.toContain('promoteSecondaryIfPrimaryDeleted'); + expect(block).toContain("await deleteIdentityCredential(user.id, 'workos_webhook')"); }); - it('invalidates session caches for the deleted user', () => { - expect(block).toMatch(/invalidateSessionsForUsers\s*\(/); + it('keeps session and unified cache invalidation inside the deletion helper', () => { + expect(block).not.toContain('invalidateSessionsForUsers'); + expect(block).not.toContain('invalidateUnifiedUsersCache'); + expect(deletionServiceSource).toContain('invalidateSessionsForUsers(result.affectedUserIds)'); + expect(deletionServiceSource).toContain('invalidateUnifiedUsersCache()'); + expect(deletionServiceSource).toContain('invalidateSlackAdminStatusCache(slackUserId)'); + expect(deletionServiceSource).toContain('invalidateWebAdminStatusCache(workosUserId)'); + expect(deletionServiceSource).toContain('invalidateMemberContextCache(slackUserId)'); }); -}); -describe('promoteSecondaryIfPrimaryDeleted error containment (#3718)', () => { - // The handler awaits the helper directly without a try/catch — a thrown - // error would land in the outer catch and produce a 500, triggering WorkOS - // retry storms. The helper must catch its own errors and return null so - // the handler falls through to the 200 path. - const helperSource = identitySource.match( - /export async function promoteSecondaryIfPrimaryDeleted[\s\S]*?\n\}\n/, - )?.[0]; + it('routes sync-users confirmed deletion through the same helper', () => { + const backfillBlock = webhookSource.slice( + webhookSource.indexOf('export async function backfillUsers'), + webhookSource.indexOf('export async function backfillOrganizationDomains'), + ); + expect(backfillBlock).toContain( + "deleteIdentityCredential(row.workos_user_id, 'sync_users_backfill')", + ); + expect(backfillBlock).toContain('upsertWorkosUserUnlessConfirmedDeleted(user)'); + expect(backfillBlock).toContain('shouldAbortBackfillDeletion(localUsers.rows.length, candidates.length)'); + expect(backfillBlock).toContain('deletion circuit breaker refused an unsafe batch'); + expect(backfillBlock).not.toMatch(/DELETE FROM organization_memberships/); + expect(backfillBlock).not.toMatch(/DELETE FROM users/); + }); - it('matched the helper source', () => { - expect(helperSource).toBeDefined(); + it('guards every authority-bearing provider writer with the deletion lock seam', () => { + for (const eventType of [ + 'organization_membership.created', + 'organization_membership.updated', + 'user.created', + 'user.updated', + ]) { + const eventStart = webhookSource.indexOf(`case '${eventType}'`); + const eventEnd = webhookSource.indexOf('\n case ', eventStart + 1); + const eventBlock = webhookSource.slice(eventStart, eventEnd); + expect(eventStart).toBeGreaterThan(-1); + expect(eventBlock).toContain(eventType === 'user.created' + ? 'withCredentialCreationEventMutation(' + : 'withActiveCredentialEventMutation('); + } }); - it('swallows DB errors (catch + return null) so the webhook returns 200', () => { - expect(helperSource).toMatch(/catch\s*\(/); - expect(helperSource).toMatch(/return null/); + it('keeps guarded user and membership helpers on the supplied transaction client', () => { + expect(webhookSource).toContain( + 'credentialClient ?? pool, user.id, user.first_name, user.last_name', + ); + expect(webhookSource).toContain( + 'canAddSeat(membership.organization_id, seatType, client)', + ); }); - it('emits an explicit ops alert on promotion failure', () => { - // logger.warn auto-posts to #admin-errors (posthog.ts:201-205); plus an - // explicit notifySystemError so the alert doesn't get lost in the warn - // stream during noisy hours. - expect(helperSource).toMatch(/logger\.warn/); - expect(helperSource).toMatch(/notifySystemError/); + it('uses one credential-first lock order and fails closed after bounded retries', () => { + expect(identityDbSource).toContain('await lockCredentialThenIdentityMutation(client, workosUserId)'); + const sharedLock = identityDbSource.indexOf('async function lockCredentialThenIdentityMutation'); + const credentialLock = identityDbSource.indexOf('await lockCredentialMutations(client, [workosUserId])', sharedLock); + const bindingLock = identityDbSource.indexOf('FOR UPDATE', credentialLock); + const identityLock = identityDbSource.indexOf('FOR UPDATE OF i', bindingLock); + expect(credentialLock).toBeGreaterThan(sharedLock); + expect(bindingLock).toBeGreaterThan(credentialLock); + expect(identityLock).toBeGreaterThan(bindingLock); + expect(identityDbSource).toContain('CREDENTIAL_MUTATION_MAX_ATTEMPTS = 3'); + expect(identityDbSource).toContain('if (await hasTerminalCredentialLifecycleMarker(client, workosUserId))'); + expect(identityDbSource).not.toContain('Credential event mutation retry bound exhausted; applying without lock'); }); });