Skip to content

bug(reporting): producer retries a failed or unanswered slice every worker turn with no backoff or bound #1255

Closed

Description

@bokelley

Summary

On 8.0.0b18, ReportingProducer retries an unanswered or failed slice on every worker turn. The delay doesn't grow after repeated failures, and there is no bound. Retries continue after the obligation has passed automated_recovery_deadline_at (action_required). Every other stuck period is retried on every turn too, so the load grows with outage length × configurations × stuck periods. max_periods_per_turn=64 caps periods per turn, not upstream requests.

For a source whose slice is an ad server report job (GAM, for example), this turns an upstream outage into a steady flood of report requests. That flood can exhaust the network's report quota, which delays recovery and also starves the adopter's non-reporting traffic to the same API.

Reproduction (ESA, adcp 8.0.0b18, memory store)

  • Daily UTC snapshot configuration, delivery_sla=PT1H, 6h recovery window, worker_interval=60s.
  • fetch_slice returns None (the upstream did not answer).
  • run_worker(now=...) at the day's close and after:
now (after day D closes) fetch_slice calls in that turn
+0h 1 (day D)
+1m 1 (day D)
+7h (past recovery deadline, action_required) 1 (day D)
+30h (day D+1 also unanswered) 2 (D and D+1)

With a 60s interval, one stuck configuration makes about 360 upstream calls in a 6h outage, and 50 configurations make about 18,000. turn.escalated reports the obligation once it is past the deadline, but the producer keeps acquiring it.

The same holds for a typed failure. ReportingSourceErrorV1.retry_after_seconds is accepted for RATE_LIMITED / QUOTA_EXHAUSTED / PROVIDER_TRANSIENT, and the materializer workers honour it (materializer/pg.py, materializer/memory.py). ReportingProducer never reads it, though: a source that says "back off 900s" is re-fetched on the next turn.

Expected

A per-obligation retry schedule owned by the producer, persisted with the obligation so it survives restarts and is shared across replicas:

  1. Backoff. Exponential with jitter after consecutive failures or None results, for example 1m → 2m → 4m … capped (configurable, say 30m).
  2. Honour retry_after_seconds from source errors as a floor on the next attempt.
  3. A bound. Past automated_recovery_deadline_at, drop to a slow cadence (for example hourly), or stop after a configurable horizon and leave the period action_required until an operator replays it (manual_replay already exists as a request cause).
  4. Account/source scope. A failure with scope="account" or scope="source" should back off every obligation sharing that scope, not each one independently.
  5. Observability. Expose the next attempt time on the turn, so adopters can log and alert without keeping their own clock.

Adopters should not need to build this. The producer is the only component that sees every obligation's attempts, and an adopter-side cache would lose that state across replicas and restarts.

Refs #1240.

Activity

  1. added
    claude-triagingTriage routine is actively working on this issue (1-3 min)
    on Sep 30, 2026
  2. bokelley commented on Sep 30, 2026

    @bokelley
    ContributorAuthor

    Triage

    Classification: Bug
    Bucket(s): client (reporting)
    Status: ready-to-implement

    What the experts said:

    • ad-tech-protocol-expert: retry_after_seconds on RATE_LIMITED/QUOTA_EXHAUSTED/PROVIDER_TRANSIENT errors is a mandatory scheduling directive — the materializer workers already honour it, so the producer ignoring it is a spec violation; item 4 (scope-level backoff) is also not optional, because scope: Literal["slice", "account", "source"] already exists on ReportingSourceErrorV1 and an account-scoped error explicitly signals that no slice for that account should be attempted during the window.
    • code-reviewer: The fix should follow the RestatementCheckpointStore pattern — add an optional @runtime_checkable RetryScheduleStore(Protocol) with get_retry_schedule / record_retry_schedule (additive, guarded by isinstance); the authoritative retry guard belongs at the top of acquire_obligation, covering both the _acquire_pending and _acquire_progress paths; earliest_retry_at: datetime | None = None on WorkerTurn is non-breaking.

    My take: Both experts converge: the core bug (items 1-3) plus scope-level backoff (item 4) share the same state machine and should ship together; item 5 (observability) is a trivial additive field that can ride the same PR. The only design question to settle before coding is whether item 3's post-deadline slow-cadence is configurable (to avoid breaking a source in deliberate hold-state) — the reviewer recommends a configurable cap rather than a hard stop.

    Implementation scope:

    • src/adcp/reporting/ledger/store.py — add frozen RetryScheduleEntry (fields: scope_key, retry_not_before, attempt) and RetryScheduleStore(Protocol) with get_retry_schedule / record_retry_schedule; mirrors RestatementCheckpointStore.
    • src/adcp/reporting/ledger/producer.py — guard top of acquire_obligation with a RetryScheduleStore isinstance check; on failure at line ~979, write retry state using max(error.retry.retry_after_seconds, backoff_for_attempt) as the delay, widened to scope_key = f"{account_id}:{source_scope}" for account/source errors; add earliest_retry_at: datetime | None = None to WorkerTurn; honour automated_recovery_deadline_at with a configurable post-deadline cadence (default hourly) passed as a new optional constructor param (non-breaking with a default).
    • src/adcp/reporting/ledger/producer_progress.py — optionally thread retry-schedule awareness into next_producer_obligations as a pre-filter optimisation (not the authoritative guard).
    • Tests — tests/reporting/ledger/test_producer.py: slice skipped within retry window, scope-level suppression across obligations, retry_after_seconds used as floor, post-deadline slow-cadence, earliest_retry_at populated on WorkerTurn.
    • Non-breaking rationale — RetryScheduleStore is optional and guarded by isinstance; adopters without it get in-turn-memory-only behaviour (acceptable degradation); all new WorkerTurn fields have defaults; new constructor params have defaults.

    Note: no matching reporting label exists in this repo — flagging the gap here for the label audit.


    Triaged by Claude Code. Session: https://claude.ai/code/session_016yHug1tAeq1ut6PHLv3RUR


    Generated by Claude Code

  3. added
    bugSomething isn't working
    and removed
    claude-triagingTriage routine is actively working on this issue (1-3 min)
    on Sep 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions