Skip to content

Commit ac5e432

Browse files
authored
feat(reporting): add public failure and lifecycle test helpers (#1232)
1 parent c567099 commit ac5e432

8 files changed

Lines changed: 1011 additions & 17 deletions

File tree

‎docs/reliable-reporting-service.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -186,5 +186,9 @@ checks replay identity plus every staged byte. `DeterministicReportingClock`
186186
and `ScriptedReportingAdapter` support settling-window and failure-injection
187187
tests without sleeping.
188188

189+
The [adapter testing guide](reporting-adapter-testing.md) adds named failure
190+
points, sync/async barriers, injectable PostgreSQL staging/seals, and reusable
191+
exact-read assertions after service recreation or durable recovery.
192+
189193
See [`examples/reliable_reporting_adapters.py`](../examples/reliable_reporting_adapters.py)
190194
for compact GAM-like and FreeWheel-like adapter definitions.

‎docs/reporting-adapter-testing.md‎

Lines changed: 161 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,161 @@
1+
# Testing reporting adapters and service recovery
2+
3+
`adcp.reporting.testing` supplies a deterministic clock, scripted sync/async
4+
adapters, named failure points, barriers, and assertions for exact reads after
5+
service recovery. These helpers use the same public contracts as adopter
6+
implementations. They also have lazy exports from `adcp.reporting`.
7+
8+
## Start with adapter conformance
9+
10+
Pass an adapter and a frozen request to `run_reporting_adapter_conformance`.
11+
It wraps `fetch_slice` in `InlineReportingSource`, executes the same source key
12+
twice, and verifies identical manifests and every staged byte. By default it
13+
uses fresh `InMemoryStagingStore` and `InMemorySealStore` instances. An optional
14+
`clock` drives both execution timestamps and conformance deadline checks.
15+
16+
```python
17+
from adcp.reporting.fixtures import redacted_snapshot_request
18+
from adcp.reporting.testing import (
19+
DeterministicReportingClock,
20+
run_reporting_adapter_conformance,
21+
)
22+
23+
24+
async def test_adapter(adapter):
25+
request = redacted_snapshot_request()
26+
clock = DeterministicReportingClock(request.period.source_read_cutoff_at)
27+
await run_reporting_adapter_conformance(adapter, request, clock=clock)
28+
```
29+
30+
Use a request/capability fixture that matches the adapter's declared offering,
31+
scope, currency, metrics, and source-local period. The SDK's `redacted_*`
32+
fixtures describe a sample provider, not every provider's contract.
33+
34+
To test source replay through PostgreSQL, install `adcp[pg]`, use an isolated
35+
test database/schema, and inject the public stores. Initialize the schema before
36+
wrapping a store; failure wrappers borrow it and do not own DDL or the pool.
37+
38+
```python
39+
from adcp.reporting.inline_storage import PgReportingSealStore, PgReportingStagingStore
40+
from adcp.reporting.testing import run_reporting_adapter_conformance
41+
42+
43+
async def test_pg_replay(pool, adapter, fresh_adapter, request, clock):
44+
staging = PgReportingStagingStore(pool=pool)
45+
await staging.create_schema() # Includes the replay-seal schema.
46+
first = await run_reporting_adapter_conformance(
47+
adapter, request, clock=clock,
48+
staging=staging, seals=PgReportingSealStore(pool=pool),
49+
)
50+
replay = await run_reporting_adapter_conformance(
51+
fresh_adapter, request, clock=clock,
52+
staging=PgReportingStagingStore(pool=pool),
53+
seals=PgReportingSealStore(pool=pool),
54+
)
55+
assert replay == first
56+
```
57+
58+
## Inject failures at commit boundaries
59+
60+
`ReportingFailurePlan.at(point, *steps)` queues an exception, a
61+
`ReportingTestBarrier`, or `None` for a successful visit. Steps are consumed in
62+
order; visits after the queue is exhausted proceed normally. Call
63+
`assert_consumed()` to detect an unreached or misspelled point. The `hits` tuple
64+
records all visits. Use synthetic errors without real provider bodies or
65+
credentials.
66+
67+
| Component | Points | Effect of an `after` failure |
68+
| --- | --- | --- |
69+
| `ScriptedReportingAdapter(..., failures=plan)` | `fetch.before`, `fetch.after` | The queued answer has been consumed. |
70+
| `FaultInjectingReportingStagingStore(store, plan)` | `stage.before`, `stage.after` | The underlying object write remains. |
71+
| Same staging wrapper | `read.before`, `read.after` | The read completed before the injected error. |
72+
| `FaultInjectingReportingSealStore(store, plan)` | `seal.get.before`, `seal.get.after` | Seal lookup completed. |
73+
| Same seal wrapper | `seal.before`, `seal.after` | The durable seal remains despite the lost acknowledgement. |
74+
75+
For example, `plan.at("seal.after", OSError("lost acknowledgement"))` exercises
76+
the case where a retry must reuse an existing sealed publication. Register the
77+
wrappers through `service.sources.register(..., staging=..., seals=...)`.
78+
They accept memory, PostgreSQL, or adopter stores and never copy or repair
79+
private state. Source fetch errors appear as failed slices; unexpected storage
80+
errors appear in the service turn's `configuration_errors`. Both leave the
81+
service available to retry other configurations.
82+
83+
Adopter wrappers can add boundaries around their own operations with
84+
`await plan.hit("destination.publish.after")` or
85+
`plan.hit_sync("notification.send.before")`. These names are chosen by the
86+
test; they do not install hooks in SDK production workers. Queue access is
87+
thread-safe. Use a barrier when a test needs a particular order between
88+
concurrent calls.
89+
90+
## Hold an in-flight call during shutdown
91+
92+
Create a `ReportingTestBarrier` inside the test's event loop. It supports async
93+
operations and synchronous provider SDKs running in worker threads. Await
94+
`wait()` to observe the boundary and always `release()` it in cleanup. Its
95+
finite wall-clock timeout is a deadlock watchdog, separate from the reporting
96+
clock. Calling `pause_sync()` on the event-loop thread raises instead of
97+
blocking the loop.
98+
99+
```python
100+
import asyncio
101+
102+
from adcp.reporting import ReliableReportingShutdownTimeoutError, ReliableReportingState
103+
from adcp.reporting.testing import ReportingFailurePlan, ReportingTestBarrier
104+
105+
106+
async def assert_shutdown_drains(service, plan):
107+
# Configure the service and give its ScriptedReportingAdapter this plan.
108+
barrier = ReportingTestBarrier()
109+
plan.at("fetch.after", barrier)
110+
work = asyncio.create_task(service.run_worker())
111+
try:
112+
await barrier.wait()
113+
try:
114+
await service.close(timeout=0)
115+
except ReliableReportingShutdownTimeoutError:
116+
assert service.state is ReliableReportingState.STOPPING
117+
else:
118+
raise AssertionError("shutdown completed while a fetch was held")
119+
finally:
120+
barrier.release()
121+
try:
122+
await work
123+
finally:
124+
await service.close()
125+
```
126+
127+
An owned resource must remain open while that call is held and close exactly
128+
once after settlement. Extend this pattern to test revocation during a fetch,
129+
failed callbacks, or cancellation of a caller. Use explicit events rather than
130+
sleeping to guess whether work has started.
131+
132+
## Assert stable publication after recovery
133+
134+
`capture_reporting_lifecycle(store, account_id=..., reporting_obligation_id=...)`
135+
captures a published obligation and all its revisions. It checks each revision's
136+
exact read, walks every page with identity/count/cursor bounds, recomputes the
137+
Core content hash (including managed typed totals), and retains canonical row
138+
bytes. Set `page_size=1` to exercise pagination with small fixtures.
139+
140+
After closing the first service, construct a fresh service and adapters, retry
141+
the same configuration, and call
142+
`await assert_reporting_lifecycle_replay(snapshot, restarted.store)`. It fails
143+
on missing, changed, or extra publications. Keep workers quiescent during both
144+
captures, and hold or advance the test clock deliberately so replay targets the
145+
same work rather than a newly due period.
146+
147+
For in-memory tests, retain the same `InMemoryReportingLedgerStore`, staging,
148+
and seal instances across services. This tests service recreation within one
149+
process. For PostgreSQL, construct fresh ledger/staging/seal instances over the
150+
retained test database and run with both pool transaction modes. The assertions
151+
also work in a separate process; the kit does not label memory as durable or
152+
certify a reporting tier.
153+
154+
Managed pipeline tests can reuse `InMemoryReportingReconciliationStore`,
155+
`InMemoryReportingOutbox`, `ReferenceReportingDestinationWriter`, and
156+
`ReferenceReportingResolver` from the existing reporting packages. Reference
157+
destination implementations are test fixtures and are not production-eligible.
158+
Use the failure plan in adopter wrappers to test destination, notification, or
159+
receipt retry boundaries, then assert the corresponding public records as well
160+
as the source lifecycle snapshot. This snapshot covers obligations/revisions
161+
and exact rows, not the entire notification or receipt state machine.

‎src/adcp/reporting/__init__.py‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -110,9 +110,30 @@
110110
from adcp.reporting.testing import (
111111
DeterministicReportingClock as DeterministicReportingClock,
112112
)
113+
from adcp.reporting.testing import (
114+
FaultInjectingReportingSealStore as FaultInjectingReportingSealStore,
115+
)
116+
from adcp.reporting.testing import (
117+
FaultInjectingReportingStagingStore as FaultInjectingReportingStagingStore,
118+
)
119+
from adcp.reporting.testing import (
120+
ReportingFailurePlan as ReportingFailurePlan,
121+
)
122+
from adcp.reporting.testing import (
123+
ReportingLifecycleSnapshot as ReportingLifecycleSnapshot,
124+
)
125+
from adcp.reporting.testing import (
126+
ReportingTestBarrier as ReportingTestBarrier,
127+
)
113128
from adcp.reporting.testing import (
114129
ScriptedReportingAdapter as ScriptedReportingAdapter,
115130
)
131+
from adcp.reporting.testing import (
132+
assert_reporting_lifecycle_replay as assert_reporting_lifecycle_replay,
133+
)
134+
from adcp.reporting.testing import (
135+
capture_reporting_lifecycle as capture_reporting_lifecycle,
136+
)
116137
from adcp.reporting.testing import (
117138
run_reporting_adapter_conformance as run_reporting_adapter_conformance,
118139
)
@@ -147,7 +168,14 @@
147168
"ReportingAccountContext": ("service", "ReportingAccountContext"),
148169
"ReportingAdapter": ("service", "ReportingAdapter"),
149170
"DeterministicReportingClock": ("testing", "DeterministicReportingClock"),
171+
"FaultInjectingReportingSealStore": ("testing", "FaultInjectingReportingSealStore"),
172+
"FaultInjectingReportingStagingStore": ("testing", "FaultInjectingReportingStagingStore"),
173+
"ReportingFailurePlan": ("testing", "ReportingFailurePlan"),
174+
"ReportingLifecycleSnapshot": ("testing", "ReportingLifecycleSnapshot"),
175+
"ReportingTestBarrier": ("testing", "ReportingTestBarrier"),
150176
"ScriptedReportingAdapter": ("testing", "ScriptedReportingAdapter"),
177+
"assert_reporting_lifecycle_replay": ("testing", "assert_reporting_lifecycle_replay"),
178+
"capture_reporting_lifecycle": ("testing", "capture_reporting_lifecycle"),
151179
"run_reporting_adapter_conformance": ("testing", "run_reporting_adapter_conformance"),
152180
}
153181

@@ -192,7 +220,14 @@ def __dir__() -> list[str]:
192220
"ReportingAccountContext",
193221
"ReportingAdapter",
194222
"DeterministicReportingClock",
223+
"FaultInjectingReportingSealStore",
224+
"FaultInjectingReportingStagingStore",
225+
"ReportingFailurePlan",
226+
"ReportingLifecycleSnapshot",
227+
"ReportingTestBarrier",
195228
"ScriptedReportingAdapter",
229+
"assert_reporting_lifecycle_replay",
230+
"capture_reporting_lifecycle",
196231
"run_reporting_adapter_conformance",
197232
"build_reporting_receipt",
198233
"evaluate_reporting_ledger",

0 commit comments

Comments
 (0)