Skip to content

feat(idempotency): let PgBackend write the replay record in the handler's own transaction #1294

Closed

Description

@bokelley

Summary

On 8.0.0, IdempotencyStore.wrap runs the handler, then calls backend.put() while it holds a per-key advisory lock (adcp/server/idempotency/store.py:288-311). PgBackend.hold takes that lock on a connection from a separate lock_pool (backends.py:572-586). put writes on that connection's transaction, which commits only when hold exits (backends.py:513-537). The seller's business writes commit on its own connection inside the handler. If the process dies after the business commit and before the lock transaction commits, no replay record exists, and the buyer's retry executes the handler again.

The PgBackend docstring (backends.py:324-335) acknowledges this: "Co-tx — handler passes its own connection so the cache write commits atomically with side effects — is planned as a follow-on enhancement." #548 asked for the same contract. No API for it exists in 8.0.0.

Reproduction (adcp 8.0.0)

A put that fails stands in for a crash between the two commits:

import asyncio
import inspect
import logging

from adcp.server.idempotency import IdempotencyStore, MemoryBackend, PgBackend

logging.disable(logging.WARNING)
business_rows = []  # stands in for the seller's own committed table


class CrashBeforeCacheCommit(MemoryBackend):
    crashed = False

    async def put(self, scope_key, key, entry):
        if not self.crashed:
            self.crashed = True
            raise ConnectionError("process died before idempotency put")
        await super().put(scope_key, key, entry)


store = IdempotencyStore(backend=CrashBeforeCacheCommit())


class Seller:
    @store.wrap
    async def create_media_buy(self, params, context=None):
        business_rows.append(params["idempotency_key"])  # committed by the seller's own transaction
        return {"media_buy_id": f"mb_{len(business_rows)}"}


async def main():
    seller, ctx = Seller(), {"caller_identity": "buyer_1"}
    req = {"idempotency_key": "k-0001-abcdef-0123456789", "total_budget": 100}
    print("attempt 1:", await seller.create_media_buy(req, ctx))
    print("retry    :", await seller.create_media_buy(req, ctx))
    print("business rows:", business_rows)
    print("PgBackend.put params:", list(inspect.signature(PgBackend.put).parameters))
    print("PgBackend.hold params:", list(inspect.signature(PgBackend.hold).parameters))

asyncio.run(main())
attempt 1: {'media_buy_id': 'mb_1'}
retry    : {'media_buy_id': 'mb_2'}
business rows: ['k-0001-abcdef-0123456789', 'k-0001-abcdef-0123456789']
PgBackend.put params: ['self', 'scope_key', 'key', 'entry']
PgBackend.hold params: ['self', 'scope_key', 'key']

Neither put nor hold accepts a caller connection, so there's no way to place the record in the business transaction.

Why an adopter hits this

Embedded Sales Agent records the gap in the core/idempotency.py module docstring ("Atomicity caveat"). It then works around it per tool:

  • media_buys has a partial unique index on (tenant_id, principal_id, idempotency_key) (src/core/database/models.py, idx_media_buys_idempotency_key).
  • update_media_buy keeps its own replay store. _update_media_buy_impl (src/core/tools/media_buy_update.py) looks up a completed workflow step with WorkflowRepository.find_by_idempotency_key (src/core/database/repositories/workflow.py), applies its own replay-window cutoff, and commits the response with the state change in one unit of work.

That amounts to a second idempotency cache, with its own TTL logic, kept in sync with the SDK's by hand. Tools that lack such a guard can double-execute after a crash in that window. For example, a retried cancel or creative sync can be re-sent to the ad server.

Ask

One of:

  1. Co-transaction put. Accept a caller connection, e.g. PgBackend.put(..., conn=...) or a context-local store.bind_connection(conn). wrap would then write the replay record on the handler's transaction, which commits atomically with the business rows. The advisory lock can stay on lock_pool.
  2. Two-phase API. async with store.reserve(scope_key, key, payload_hash) as slot: returns a reservation, and the handler calls await slot.record(response, conn=conn) inside its own transaction before committing. A row reserved but never recorded is then unambiguous on retry: it means "in flight or crashed".

Activity

  1. added
    enhancementNew feature or request
    claude-triagingTriage routine is actively working on this issue (1-3 min)
    on Oct 2, 2026
  2. bokelley commented on Oct 2, 2026

    @bokelley
    ContributorAuthor

    Triage

    Classification: Feature request
    Bucket(s): middleware (no middleware label exists in this repo — flagged as gap)
    Status: ready-for-human

    What the experts said:

    • ad-tech-protocol-expert: The atomicity gap is real and protocol-significant per AdCP's idempotency guarantee; Option 1 (conn=) is simpler, but both options share an advisory-lock release race — the lock-pool transaction commits when hold() exits, before the handler-owned transaction can; TTL precision (clock skew between lock_pool and business-conn) also needs documentation.
    • code-reviewer: Option 1's back-compat claim is overstated — existing IdempotencyBackend subclasses break at runtime when store.py passes conn=handler_conn to a put() that doesn't declare it; Option 2 (reserve()/slot.record()) is safer (no ABC churn, lifecycle recorded flag), but shares the same lock-release race without an additional mechanism to keep the advisory lock held across both transactions.
    • security-reviewer: Architectural blocker — pg_advisory_xact_lock releases when hold() exits, before handler-conn commits; a concurrent retry acquires the lock, sees no record, and re-executes — the race is wider than without the feature; also flags a medium risk that caller-controlled connection objects bypass the write silently (replay-attack surface on create_media_buy / accept_proposal).

    My take: All three experts converge on the same architectural blocker: the lock lifetime doesn't span the handler-owned transaction, so the co-tx window still has a concurrent-retry race. Neither proposed option resolves this without a lock-strategy redesign.

    @bokelley — the design question to settle before implementation: how should hold() keep the advisory lock held until after the caller-owned transaction commits? Candidates include using a session-level pg_advisory_lock (released explicitly after handler-conn commits), a commit-callback hook into hold(), or moving the advisory lock inside the business connection's transaction. Option 2 (reserve()/slot.record()) is the cleaner API once the lock issue is resolved — it avoids IdempotencyBackend ABC churn and gives a lifecycle gate (slot.recorded flag); recommend leading with that shape.


    Triaged by Claude Code. Session: https://claude.ai/code/session_01FMmQhb6JsRjzzyTP9LXxHW


    Generated by Claude Code

  3. added and removed
    claude-triagingTriage routine is actively working on this issue (1-3 min)
    on Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions