You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
bug(reporting): producer retries a failed or unanswered slice every worker turn with no backoff or bound #1255
On 8.0.0b18, ReportingProducer retries an unanswered or failed slice on every worker turn. The delay doesn't grow after repeated failures, and there is no bound. Retries continue after the obligation has passed automated_recovery_deadline_at (action_required). Every other stuck period is retried on every turn too, so the load grows with outage length × configurations × stuck periods. max_periods_per_turn=64 caps periods per turn, not upstream requests.
For a source whose slice is an ad server report job (GAM, for example), this turns an upstream outage into a steady flood of report requests. That flood can exhaust the network's report quota, which delays recovery and also starves the adopter's non-reporting traffic to the same API.
Reproduction (ESA, adcp 8.0.0b18, memory store)
Daily UTC snapshot configuration, delivery_sla=PT1H, 6h recovery window, worker_interval=60s.
fetch_slice returns None (the upstream did not answer).
run_worker(now=...) at the day's close and after:
now (after day D closes)
fetch_slice calls in that turn
+0h
1 (day D)
+1m
1 (day D)
+7h (past recovery deadline, action_required)
1 (day D)
+30h (day D+1 also unanswered)
2 (D and D+1)
With a 60s interval, one stuck configuration makes about 360 upstream calls in a 6h outage, and 50 configurations make about 18,000. turn.escalated reports the obligation once it is past the deadline, but the producer keeps acquiring it.
The same holds for a typed failure. ReportingSourceErrorV1.retry_after_seconds is accepted for RATE_LIMITED / QUOTA_EXHAUSTED / PROVIDER_TRANSIENT, and the materializer workers honour it (materializer/pg.py, materializer/memory.py). ReportingProducer never reads it, though: a source that says "back off 900s" is re-fetched on the next turn.
Expected
A per-obligation retry schedule owned by the producer, persisted with the obligation so it survives restarts and is shared across replicas:
Backoff. Exponential with jitter after consecutive failures or None results, for example 1m → 2m → 4m … capped (configurable, say 30m).
Honour retry_after_seconds from source errors as a floor on the next attempt.
A bound. Past automated_recovery_deadline_at, drop to a slow cadence (for example hourly), or stop after a configurable horizon and leave the period action_required until an operator replays it (manual_replay already exists as a request cause).
Account/source scope. A failure with scope="account" or scope="source" should back off every obligation sharing that scope, not each one independently.
Observability. Expose the next attempt time on the turn, so adopters can log and alert without keeping their own clock.
Adopters should not need to build this. The producer is the only component that sees every obligation's attempts, and an adopter-side cache would lose that state across replicas and restarts.
ad-tech-protocol-expert: retry_after_seconds on RATE_LIMITED/QUOTA_EXHAUSTED/PROVIDER_TRANSIENT errors is a mandatory scheduling directive — the materializer workers already honour it, so the producer ignoring it is a spec violation; item 4 (scope-level backoff) is also not optional, because scope: Literal["slice", "account", "source"] already exists on ReportingSourceErrorV1 and an account-scoped error explicitly signals that no slice for that account should be attempted during the window.
code-reviewer: The fix should follow the RestatementCheckpointStore pattern — add an optional @runtime_checkable RetryScheduleStore(Protocol) with get_retry_schedule / record_retry_schedule (additive, guarded by isinstance); the authoritative retry guard belongs at the top of acquire_obligation, covering both the _acquire_pending and _acquire_progress paths; earliest_retry_at: datetime | None = None on WorkerTurn is non-breaking.
My take: Both experts converge: the core bug (items 1-3) plus scope-level backoff (item 4) share the same state machine and should ship together; item 5 (observability) is a trivial additive field that can ride the same PR. The only design question to settle before coding is whether item 3's post-deadline slow-cadence is configurable (to avoid breaking a source in deliberate hold-state) — the reviewer recommends a configurable cap rather than a hard stop.
Implementation scope:
src/adcp/reporting/ledger/store.py — add frozen RetryScheduleEntry (fields: scope_key, retry_not_before, attempt) and RetryScheduleStore(Protocol) with get_retry_schedule / record_retry_schedule; mirrors RestatementCheckpointStore.
src/adcp/reporting/ledger/producer.py — guard top of acquire_obligation with a RetryScheduleStore isinstance check; on failure at line ~979, write retry state using max(error.retry.retry_after_seconds, backoff_for_attempt) as the delay, widened to scope_key = f"{account_id}:{source_scope}" for account/source errors; add earliest_retry_at: datetime | None = None to WorkerTurn; honour automated_recovery_deadline_at with a configurable post-deadline cadence (default hourly) passed as a new optional constructor param (non-breaking with a default).
src/adcp/reporting/ledger/producer_progress.py — optionally thread retry-schedule awareness into next_producer_obligations as a pre-filter optimisation (not the authoritative guard).
Tests — tests/reporting/ledger/test_producer.py: slice skipped within retry window, scope-level suppression across obligations, retry_after_seconds used as floor, post-deadline slow-cadence, earliest_retry_at populated on WorkerTurn.
Non-breaking rationale — RetryScheduleStore is optional and guarded by isinstance; adopters without it get in-turn-memory-only behaviour (acceptable degradation); all new WorkerTurn fields have defaults; new constructor params have defaults.
Note: no matching reporting label exists in this repo — flagging the gap here for the label audit.
Activity
bokelley commentedon Sep 30, 2026
Triage
Classification: Bug
Bucket(s): client (reporting)
Status: ready-to-implement
What the experts said:
retry_after_secondsonRATE_LIMITED/QUOTA_EXHAUSTED/PROVIDER_TRANSIENTerrors is a mandatory scheduling directive — the materializer workers already honour it, so the producer ignoring it is a spec violation; item 4 (scope-level backoff) is also not optional, becausescope: Literal["slice", "account", "source"]already exists onReportingSourceErrorV1and an account-scoped error explicitly signals that no slice for that account should be attempted during the window.RestatementCheckpointStorepattern — add an optional@runtime_checkable RetryScheduleStore(Protocol)withget_retry_schedule/record_retry_schedule(additive, guarded byisinstance); the authoritative retry guard belongs at the top ofacquire_obligation, covering both the_acquire_pendingand_acquire_progresspaths;earliest_retry_at: datetime | None = NoneonWorkerTurnis non-breaking.My take: Both experts converge: the core bug (items 1-3) plus scope-level backoff (item 4) share the same state machine and should ship together; item 5 (observability) is a trivial additive field that can ride the same PR. The only design question to settle before coding is whether item 3's post-deadline slow-cadence is configurable (to avoid breaking a source in deliberate hold-state) — the reviewer recommends a configurable cap rather than a hard stop.
Implementation scope:
src/adcp/reporting/ledger/store.py— add frozenRetryScheduleEntry(fields:scope_key,retry_not_before,attempt) andRetryScheduleStore(Protocol)withget_retry_schedule/record_retry_schedule; mirrorsRestatementCheckpointStore.src/adcp/reporting/ledger/producer.py— guard top ofacquire_obligationwith aRetryScheduleStoreisinstance check; on failure at line ~979, write retry state usingmax(error.retry.retry_after_seconds, backoff_for_attempt)as the delay, widened toscope_key = f"{account_id}:{source_scope}"for account/source errors; addearliest_retry_at: datetime | None = NonetoWorkerTurn; honourautomated_recovery_deadline_atwith a configurable post-deadline cadence (default hourly) passed as a new optional constructor param (non-breaking with a default).src/adcp/reporting/ledger/producer_progress.py— optionally thread retry-schedule awareness intonext_producer_obligationsas a pre-filter optimisation (not the authoritative guard).tests/reporting/ledger/test_producer.py: slice skipped within retry window, scope-level suppression across obligations,retry_after_secondsused as floor, post-deadline slow-cadence,earliest_retry_atpopulated onWorkerTurn.RetryScheduleStoreis optional and guarded byisinstance; adopters without it get in-turn-memory-only behaviour (acceptable degradation); all newWorkerTurnfields have defaults; new constructor params have defaults.Note: no matching
reportinglabel exists in this repo — flagging the gap here for the label audit.Triaged by Claude Code. Session: https://claude.ai/code/session_016yHug1tAeq1ut6PHLv3RUR
Generated by Claude Code