Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
182 changes: 182 additions & 0 deletions docs/runbooks/admin-credential-bind-reconciliation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,182 @@
---
description: "Reconcile admin WorkOS credential binding failures, provider compensation, lifecycle quarantine, and uncertain outcomes."
"og:title": "AdCP Admin Credential Binding Reconciliation"
---

# Admin credential binding reconciliation

This is the admin containment and compensation slice of #6827. It does not
close the issue or enable organization authorization enforcement.

`POST /api/admin/users/:userId/linked-emails` creates a fresh WorkOS credential
and binds it without transferring memberships or other app state. An
identity-bearing admin is required. The journal records the exact authenticated
credential and its observed identity; both actor and host bindings are checked
again inside the local transaction. The host must remain primary.

`POST /api/admin/users/:userId/credentials/:credentialId/promote` returns 409,
including requests with `consolidate: true` and orphan-primary repairs, because
its previous consolidation rewrote authority and membership provenance.

## Mutation and replay rules

The dedicated WorkOS client makes one attempt with a ten-second timeout.
WorkOS does not promise general create-user deduplication from an idempotency
header; see the [WorkOS SDK guidance](https://workos.com/docs/sdks/go).

The `admin_credential_bind_operations` journal and its audit event commit before
the provider request. Its normalized-email hash uniquely reserves the address
across hosts while an operation is unresolved or committed. Session advisory
locks serialize concurrent requests for the address and host; uncertain lock
acquisition or release destroys the database connection.

| Status | Meaning and operator action |
| --- | --- |
| `creating` | Intent exists; creation may be in flight or its result may have been lost. Keep replay blocked. |
| `provider_created` | The exact returned WorkOS user ID is recorded; local binding may be incomplete. Keep replay blocked. |
| `committed` | User, binding, authorization epochs and audit committed together. A retry returns success only if actor, host and credential remain live and the original binding and email still match under lifecycle locks. |
| `compensating` | Local rollback was acknowledged and a durable admission quarantine blocks callbacks. Deletion of the exact newly created WorkOS user may be in flight. Keep replay blocked. |
| `compensated` | Provider deletion returned success after a confirmed local rollback, and its terminal marker and audit committed under the lifecycle lock. A fresh attempt with a new provider ID is permitted. |
| `provider_rejected` | A non-retryable provider rejection was received without SDK retries. A fresh attempt is permitted. |
| `reconciliation_required` | Provider, commit, compensation or lock outcome is uncertain. Keep replay blocked until adjudicated. |

A local write failure before COMMIT, followed by an acknowledged ROLLBACK,
permits compensation. Any COMMIT exception requires reconciliation: deleting
the credential could destroy a binding that actually committed. A local row
that appeared independently, including through a webhook, also requires
reconciliation instead of deleting potentially adopted state.

The local bind and committed replay use #7459's seed-6827 credential lifecycle
locks on the same transaction client, followed by binding and identity row
locks. A deletion or quarantine marker blocks all local binding success writes.
Unknown lock/query/cardinality results require reconciliation. The exact actor
identity and active host primary are rechecked while these locks are held.

Compensation first rechecks that the returned provider ID has no local user,
binding or terminal marker under that same fence. It commits
`identity_credential_admin_compensation_quarantined` together with the
`compensating` journal transition and checked audit cardinality. This admission
quarantine explicitly records `provider_delete_confirmed: false`; it does not
claim that WorkOS deleted the user. An unconfirmed quarantine COMMIT prevents
the upstream delete entirely.

A second transaction reacquires the lifecycle lock and requires exactly that
operation's quarantine, matching the provider ID and exact admin credential
and identity, without other markers or local artifacts. It holds the lock
through the provider response and local COMMIT. Only a confirmed provider delete
records the separate `identity_credential_admin_compensation_deleted` marker
with `provider_delete_confirmed: true`. This evidence is distinct from a signed
provider-deletion event. Both lifecycle actions block delayed creation callbacks.
The durable quarantine covers the gap between transactions and survives failed
or ambiguous provider deletion and subsequent marker/audit/COMMIT failures.
Those outcomes remain `reconciliation_required`, with no invented confirmed
deletion claim. `confirmed_delete_evidence_failed` means the provider
acknowledged deletion but its local deletion marker/audit could not commit;
`compensation_commit_ambiguous` means that commit's acknowledgment is uncertain.
Neither result claims successful compensation. A later signed deletion still
records its own provider source and follows #7459's normal replay behavior.

A provider timeout, transport failure, retryable status or failed compensation
never triggers an immediate GET to infer success or rollback. Delayed completion
can invalidate that read. There is no automatic stale-operation expiry. If a
later journal update fails, the previously committed intent remains blocking.

## Adjudication

Use the response's `operation_id` to inspect the journal and audit trail:

```sql
SELECT id, host_user_id, host_identity_id, actor_user_id, actor_identity_id,
provider_user_id, status, failure_code, created_at, updated_at
FROM admin_credential_bind_operations
WHERE id = :operation_id;

SELECT created_at, workos_user_id, details
FROM registry_audit_log
WHERE action IN ('admin_credential_bind', 'identity_credential_admin_compensation_deleted',
'identity_credential_admin_compensation_quarantined')
AND resource_id = :operation_id
ORDER BY created_at;
```

Keep provider bodies, passwords, tokens and email addresses out of logs and
incident comments. The journal stores only identifiers, an email hash, fixed
status/failure codes and timestamps. It has no foreign keys so later deletion
cannot erase the evidence.

Engineering must establish the final outcome of the original request through
provider request/event history and, where necessary, WorkOS support. A single
current GET, a 404, or elapsed time is insufficient. Match the recorded user ID
and original operation; never delete an account found merely by email.

Inspect the local user, identity binding, epochs and audit in a transaction.
Determine whether a webhook or another flow adopted the created account and
whether it now has independent authority. Preserve those memberships and their
provenance. Repair only the reviewed operation, with an individually attributed
audit event recording the evidence and final disposition in the same transaction
as the journal transition. Mark `committed` only after confirming the expected
binding; release the email reservation as `compensated` only after establishing
that the original creation/deletion request has completed, the created account
is absent, and no local binding remains. Leave unresolved evidence blocking.
For a quarantined credential, retain its quarantine and record confirmed
deletion evidence under the shared credential lifecycle lock before committing
`compensated`. A quarantine alone is never proof of provider deletion.
There is intentionally no public retry-unblock or reconciliation endpoint.

## Route inventory and scope

The existing-credential `POST /api/admin/users/:userId/credentials` remains
contained by #7459 before its legacy WorkOS GET and generic merge handler.
The following admin provider-write routes were traced and do not subsequently
bind or promote a person identity:

- `POST /api/admin/organizations/:orgId/add-users`
- `POST /api/admin/organizations/audit-admins` with `fix=true` (organization role promotion)
- `POST /api/admin/accounts/:id/migrate-members`
- `PUT /api/admin/accounts/:orgId/members/:userId/role`
- `POST /api/admin/domain-users/backfill-members`

No membership mutation was added to fresh credential binding. Member primary
email mutation/reconciliation, automatic alias detection, generic merge
containment and detach integrity remain separate work.

## Merge order with containment #7459

This PR is stacked on the exact [#7459](https://github.com/adcontextprotocol/adcp/pull/7459)
head `9a7cd5362787a0badbfccd0f1cb1dd002654c22f`. Land #7459 first, then retarget
and rebase this PR onto main. Its database, member, automatic alias,
existing-credential and promotion guards remain intact. Fresh binding does not
call or add an exception to `mergeUsers` or the consolidation policy.

The only predecessor is corrected #7459. The admin saga has no API or schema
dependency on #7460 or migration 594; it uses existing identity, epoch and audit
tables plus its independent journal 593. Keep the corrected predecessor's
post-deletion missing-primary signal intact when rebasing.

The shared `server/src/routes/admin/users.ts` hunk is resolved as follows:

- Keep both the `refuseIdentityConsolidation` and `createAndBindAdminCredential`
imports.
- Keep `refuseIdentityConsolidation` on `POST /:userId/credentials`.
- Keep `POST /:userId/credentials/:credentialId/promote` as an unconditional
refusal using that middleware, with no legacy mutation body or orphan repair.
- Replace only `POST /:userId/linked-emails`'s blanket refusal and old merge body
with this PR's fresh-credential handler and durable saga. Requests to
consolidate or promote are still refused before any provider operation.

The overlapping tests retain the fresh-bind/fault tests and #7459's
existing-credential and broader promotion containment assertions. Promotion
fixtures bind directly in SQL. Only fresh `/linked-emails` is removed from the
blanket-refusal matrix; new route tests cover identity-bearing authentication,
destructive intent rejection, exact actor attribution and provider/local failure
boundaries. Both PRs' focused suites and typecheck must pass after rebasing.

Migration 593 was assigned after a read-only live ledger audit on 2026-09-13:
the highest applied version was 587; versions 588–592 and 593 were absent.
Main subsequently added unrelated migration 589; this change adds or modifies
none of 588–592. Migration 592 remains owned by member compensation
[#7463](https://github.com/adcontextprotocol/adcp/pull/7463). The local/origin-main
union and open-PR migration audit have no collision for 593. Recheck migration
numbering before merging parallel security PRs. If #7463 lands first, retain
both independent WorkOS client factories and both timeout tests when resolving
their shared insertion points; the admin journal does not depend on 592.
10 changes: 7 additions & 3 deletions server/public/admin-people.html
Original file line number Diff line number Diff line change
Expand Up @@ -593,6 +593,8 @@ <h3 style="margin: 0 0 var(--space-2); font-size: var(--text-base);">AAO site-ad
data-cred-id="${escapeHtml(c.workos_user_id)}"
data-cred-email="${escapeHtml(c.email || '')}"
data-host-id="${escapeHtml(r.workos_user_id)}"
data-identity-id="${escapeHtml(data.identity_id)}"
data-authorization-epoch="${escapeHtml(c.authorization_epoch)}"
data-target-label="${escapeHtml(targetLabel)}">Remove</button>
</div>`;
return `
Expand Down Expand Up @@ -639,7 +641,7 @@ <h3 style="margin: 0 0 var(--space-2); font-size: var(--text-base);">Sign-in cre
container.querySelectorAll('.unbind-cred-btn').forEach(btn => {
btn.addEventListener('click', (ev) => {
const b = ev.currentTarget;
unbindCredential(b.dataset.hostId, b.dataset.credId, b.dataset.credEmail, b.dataset.targetLabel);
unbindCredential(b.dataset.hostId, b.dataset.credId, b.dataset.credEmail, b.dataset.targetLabel, b.dataset.identityId, b.dataset.authorizationEpoch);
});
});
container.querySelectorAll('.promote-cred-btn').forEach(btn => {
Expand Down Expand Up @@ -751,10 +753,10 @@ <h3 style="margin: 0 0 var(--space-2); font-size: var(--text-base);">Sign-in cre
}
}

async function unbindCredential(hostId, credId, credEmail, targetLabel) {
async function unbindCredential(hostId, credId, credEmail, targetLabel, identityId, authorizationEpoch) {
const credLabel = credEmail || credId;
if (credEmail) {
const typed = prompt(`Remove sign-in credential ${credLabel} from ${targetLabel}?\n\nThe WorkOS user stays alive in WorkOS but will no longer be associated with this account. Sign-ins via that email will reach an empty workspace.\n\nTo confirm, type the credential's email exactly:\n ${credEmail}`);
const typed = prompt(`Remove sign-in credential ${credLabel} from ${targetLabel}?\n\nThe WorkOS user stays alive in WorkOS but will no longer be associated with this account. Existing memberships and permissions stay with their current credential. This does not undo past account consolidation.\n\nTo confirm, type the credential's email exactly:\n ${credEmail}`);
if (typed === null) return;
if (typed.trim().toLowerCase() !== credEmail.trim().toLowerCase()) {
alert('Confirmation email did not match. Cancelled.');
Expand All @@ -766,6 +768,8 @@ <h3 style="margin: 0 0 var(--space-2); font-size: var(--text-base);">Sign-in cre
try {
const res = await AdminSidebar.fetch(`/api/admin/users/${encodeURIComponent(hostId)}/credentials/${encodeURIComponent(credId)}`, {
method: 'DELETE',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ expected_identity_id: identityId, expected_authorization_epoch: authorizationEpoch }),
});
const data = await res.json();
if (!res.ok) {
Expand Down
13 changes: 8 additions & 5 deletions server/src/addie/admin-status-lookup.ts
Original file line number Diff line number Diff line change
Expand Up @@ -43,9 +43,10 @@ const wgDb = new WorkingGroupDatabase();
export async function isWebUserAAOAdmin(workosUserId: string): Promise<boolean> {
const cache = getWebAdminStatusCache();
const cached = cache.get(workosUserId);
if (cached && cached.expiresAt > Date.now()) {
if (cached && !cached.isAdmin && cached.expiresAt > Date.now()) {
return cached.isAdmin;
}
if (cached?.isAdmin) cache.delete(workosUserId);

try {
const adminGroup = await wgDb.getWorkingGroupBySlug(AAO_ADMIN_WORKING_GROUP_SLUG);
Expand All @@ -58,10 +59,12 @@ export async function isWebUserAAOAdmin(workosUserId: string): Promise<boolean>
}

const isAdmin = await wgDb.isMember(adminGroup.id, workosUserId);
cache.set(workosUserId, {
isAdmin,
expiresAt: Date.now() + (isAdmin ? AAO_ADMIN_POSITIVE_CACHE_TTL_MS : AAO_ADMIN_NEGATIVE_CACHE_TTL_MS),
});
if (!isAdmin) {
cache.set(workosUserId, {
isAdmin: false,
expiresAt: Date.now() + AAO_ADMIN_NEGATIVE_CACHE_TTL_MS,
});
}
logger.debug({ workosUserId, isAdmin }, 'Checked web user admin status');
return isAdmin;
} catch (error) {
Expand Down
13 changes: 8 additions & 5 deletions server/src/addie/mcp/admin-tools.ts
Original file line number Diff line number Diff line change
Expand Up @@ -220,9 +220,10 @@ export async function isSlackUserAAOAdmin(
): Promise<boolean> {
// Check cache first
const cached = adminStatusCache.get(slackUserId);
if (cached && cached.expiresAt > Date.now()) {
if (cached && !cached.isAdmin && cached.expiresAt > Date.now()) {
return cached.isAdmin;
}
if (cached?.isAdmin) adminStatusCache.delete(slackUserId);

try {
// Look up the Slack user mapping to get their WorkOS user ID
Expand Down Expand Up @@ -259,10 +260,12 @@ export async function isSlackUserAAOAdmin(
const isAdmin = await wgDb.isMember(adminGroup.id, mapping.workos_user_id);

// Cache the result
adminStatusCache.set(slackUserId, {
isAdmin,
expiresAt: Date.now() + (isAdmin ? ADMIN_POSITIVE_CACHE_TTL_MS : ADMIN_NEGATIVE_CACHE_TTL_MS),
});
if (!isAdmin) {
adminStatusCache.set(slackUserId, {
isAdmin: false,
expiresAt: Date.now() + ADMIN_NEGATIVE_CACHE_TTL_MS,
});
}

logger.info(
{
Expand Down
16 changes: 16 additions & 0 deletions server/src/auth/workos-client.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ let _ownerlessPromotionWorkos: WorkOS | null = null;
let _pipesWorkos: WorkOS | null = null;
let _authorizationObserverWorkos: WorkOS | null = null;
let _authorizationEnforcementWorkos: WorkOS | null = null;
let _adminCredentialMutationWorkos: WorkOS | null = null;
let _clientId = '';

/** Returns the shared WorkOS client. Constructed on first call; WORKOS_API_KEY and WORKOS_CLIENT_ID must be set by then. */
Expand All @@ -26,6 +27,21 @@ export function getWorkos(): WorkOS {
return _workos;
}

/** One attempt only: a timed-out admin mutation requires durable adjudication. */
export function getAdminCredentialMutationWorkos(): WorkOS {
if (!_adminCredentialMutationWorkos) {
if (!process.env.WORKOS_API_KEY || !process.env.WORKOS_CLIENT_ID) {
throw new Error('WorkOS configuration is required');
}
_adminCredentialMutationWorkos = new WorkOS(process.env.WORKOS_API_KEY, {
clientId: process.env.WORKOS_CLIENT_ID,
timeout: 10_000,
maxRetries: 0,
});
}
return _adminCredentialMutationWorkos;
}

/** Returns a WorkOS client with a shorter timeout for DB-lock-held promotion flows. */
export function getOwnerlessPromotionWorkos(): WorkOS {
if (!_ownerlessPromotionWorkos) {
Expand Down
Loading
Loading