|
| 1 | +# Testing reporting adapters and service recovery |
| 2 | + |
| 3 | +`adcp.reporting.testing` supplies a deterministic clock, scripted sync/async |
| 4 | +adapters, named failure points, barriers, and assertions for exact reads after |
| 5 | +service recovery. These helpers use the same public contracts as adopter |
| 6 | +implementations. They also have lazy exports from `adcp.reporting`. |
| 7 | + |
| 8 | +## Start with adapter conformance |
| 9 | + |
| 10 | +Pass an adapter and a frozen request to `run_reporting_adapter_conformance`. |
| 11 | +It wraps `fetch_slice` in `InlineReportingSource`, executes the same source key |
| 12 | +twice, and verifies identical manifests and every staged byte. By default it |
| 13 | +uses fresh `InMemoryStagingStore` and `InMemorySealStore` instances. An optional |
| 14 | +`clock` drives both execution timestamps and conformance deadline checks. |
| 15 | + |
| 16 | +```python |
| 17 | +from adcp.reporting.fixtures import redacted_snapshot_request |
| 18 | +from adcp.reporting.testing import ( |
| 19 | + DeterministicReportingClock, |
| 20 | + run_reporting_adapter_conformance, |
| 21 | +) |
| 22 | + |
| 23 | + |
| 24 | +async def test_adapter(adapter): |
| 25 | + request = redacted_snapshot_request() |
| 26 | + clock = DeterministicReportingClock(request.period.source_read_cutoff_at) |
| 27 | + await run_reporting_adapter_conformance(adapter, request, clock=clock) |
| 28 | +``` |
| 29 | + |
| 30 | +Use a request/capability fixture that matches the adapter's declared offering, |
| 31 | +scope, currency, metrics, and source-local period. The SDK's `redacted_*` |
| 32 | +fixtures describe a sample provider, not every provider's contract. |
| 33 | + |
| 34 | +To test source replay through PostgreSQL, install `adcp[pg]`, use an isolated |
| 35 | +test database/schema, and inject the public stores. Initialize the schema before |
| 36 | +wrapping a store; failure wrappers borrow it and do not own DDL or the pool. |
| 37 | + |
| 38 | +```python |
| 39 | +from adcp.reporting.inline_storage import PgReportingSealStore, PgReportingStagingStore |
| 40 | +from adcp.reporting.testing import run_reporting_adapter_conformance |
| 41 | + |
| 42 | + |
| 43 | +async def test_pg_replay(pool, adapter, fresh_adapter, request, clock): |
| 44 | + staging = PgReportingStagingStore(pool=pool) |
| 45 | + await staging.create_schema() # Includes the replay-seal schema. |
| 46 | + first = await run_reporting_adapter_conformance( |
| 47 | + adapter, request, clock=clock, |
| 48 | + staging=staging, seals=PgReportingSealStore(pool=pool), |
| 49 | + ) |
| 50 | + replay = await run_reporting_adapter_conformance( |
| 51 | + fresh_adapter, request, clock=clock, |
| 52 | + staging=PgReportingStagingStore(pool=pool), |
| 53 | + seals=PgReportingSealStore(pool=pool), |
| 54 | + ) |
| 55 | + assert replay == first |
| 56 | +``` |
| 57 | + |
| 58 | +## Inject failures at commit boundaries |
| 59 | + |
| 60 | +`ReportingFailurePlan.at(point, *steps)` queues an exception, a |
| 61 | +`ReportingTestBarrier`, or `None` for a successful visit. Steps are consumed in |
| 62 | +order; visits after the queue is exhausted proceed normally. Call |
| 63 | +`assert_consumed()` to detect an unreached or misspelled point. The `hits` tuple |
| 64 | +records all visits. Use synthetic errors without real provider bodies or |
| 65 | +credentials. |
| 66 | + |
| 67 | +| Component | Points | Effect of an `after` failure | |
| 68 | +| --- | --- | --- | |
| 69 | +| `ScriptedReportingAdapter(..., failures=plan)` | `fetch.before`, `fetch.after` | The queued answer has been consumed. | |
| 70 | +| `FaultInjectingReportingStagingStore(store, plan)` | `stage.before`, `stage.after` | The underlying object write remains. | |
| 71 | +| Same staging wrapper | `read.before`, `read.after` | The read completed before the injected error. | |
| 72 | +| `FaultInjectingReportingSealStore(store, plan)` | `seal.get.before`, `seal.get.after` | Seal lookup completed. | |
| 73 | +| Same seal wrapper | `seal.before`, `seal.after` | The durable seal remains despite the lost acknowledgement. | |
| 74 | + |
| 75 | +For example, `plan.at("seal.after", OSError("lost acknowledgement"))` exercises |
| 76 | +the case where a retry must reuse an existing sealed publication. Register the |
| 77 | +wrappers through `service.sources.register(..., staging=..., seals=...)`. |
| 78 | +They accept memory, PostgreSQL, or adopter stores and never copy or repair |
| 79 | +private state. Source fetch errors appear as failed slices; unexpected storage |
| 80 | +errors appear in the service turn's `configuration_errors`. Both leave the |
| 81 | +service available to retry other configurations. |
| 82 | + |
| 83 | +Adopter wrappers can add boundaries around their own operations with |
| 84 | +`await plan.hit("destination.publish.after")` or |
| 85 | +`plan.hit_sync("notification.send.before")`. These names are chosen by the |
| 86 | +test; they do not install hooks in SDK production workers. Queue access is |
| 87 | +thread-safe. Use a barrier when a test needs a particular order between |
| 88 | +concurrent calls. |
| 89 | + |
| 90 | +## Hold an in-flight call during shutdown |
| 91 | + |
| 92 | +Create a `ReportingTestBarrier` inside the test's event loop. It supports async |
| 93 | +operations and synchronous provider SDKs running in worker threads. Await |
| 94 | +`wait()` to observe the boundary and always `release()` it in cleanup. Its |
| 95 | +finite wall-clock timeout is a deadlock watchdog, separate from the reporting |
| 96 | +clock. Calling `pause_sync()` on the event-loop thread raises instead of |
| 97 | +blocking the loop. |
| 98 | + |
| 99 | +```python |
| 100 | +import asyncio |
| 101 | + |
| 102 | +from adcp.reporting import ReliableReportingShutdownTimeoutError, ReliableReportingState |
| 103 | +from adcp.reporting.testing import ReportingFailurePlan, ReportingTestBarrier |
| 104 | + |
| 105 | + |
| 106 | +async def assert_shutdown_drains(service, plan): |
| 107 | + # Configure the service and give its ScriptedReportingAdapter this plan. |
| 108 | + barrier = ReportingTestBarrier() |
| 109 | + plan.at("fetch.after", barrier) |
| 110 | + work = asyncio.create_task(service.run_worker()) |
| 111 | + try: |
| 112 | + await barrier.wait() |
| 113 | + try: |
| 114 | + await service.close(timeout=0) |
| 115 | + except ReliableReportingShutdownTimeoutError: |
| 116 | + assert service.state is ReliableReportingState.STOPPING |
| 117 | + else: |
| 118 | + raise AssertionError("shutdown completed while a fetch was held") |
| 119 | + finally: |
| 120 | + barrier.release() |
| 121 | + try: |
| 122 | + await work |
| 123 | + finally: |
| 124 | + await service.close() |
| 125 | +``` |
| 126 | + |
| 127 | +An owned resource must remain open while that call is held and close exactly |
| 128 | +once after settlement. Extend this pattern to test revocation during a fetch, |
| 129 | +failed callbacks, or cancellation of a caller. Use explicit events rather than |
| 130 | +sleeping to guess whether work has started. |
| 131 | + |
| 132 | +## Assert stable publication after recovery |
| 133 | + |
| 134 | +`capture_reporting_lifecycle(store, account_id=..., reporting_obligation_id=...)` |
| 135 | +captures a published obligation and all its revisions. It checks each revision's |
| 136 | +exact read, walks every page with identity/count/cursor bounds, recomputes the |
| 137 | +Core content hash (including managed typed totals), and retains canonical row |
| 138 | +bytes. Set `page_size=1` to exercise pagination with small fixtures. |
| 139 | + |
| 140 | +After closing the first service, construct a fresh service and adapters, retry |
| 141 | +the same configuration, and call |
| 142 | +`await assert_reporting_lifecycle_replay(snapshot, restarted.store)`. It fails |
| 143 | +on missing, changed, or extra publications. Keep workers quiescent during both |
| 144 | +captures, and hold or advance the test clock deliberately so replay targets the |
| 145 | +same work rather than a newly due period. |
| 146 | + |
| 147 | +For in-memory tests, retain the same `InMemoryReportingLedgerStore`, staging, |
| 148 | +and seal instances across services. This tests service recreation within one |
| 149 | +process. For PostgreSQL, construct fresh ledger/staging/seal instances over the |
| 150 | +retained test database and run with both pool transaction modes. The assertions |
| 151 | +also work in a separate process; the kit does not label memory as durable or |
| 152 | +certify a reporting tier. |
| 153 | + |
| 154 | +Managed pipeline tests can reuse `InMemoryReportingReconciliationStore`, |
| 155 | +`InMemoryReportingOutbox`, `ReferenceReportingDestinationWriter`, and |
| 156 | +`ReferenceReportingResolver` from the existing reporting packages. Reference |
| 157 | +destination implementations are test fixtures and are not production-eligible. |
| 158 | +Use the failure plan in adopter wrappers to test destination, notification, or |
| 159 | +receipt retry boundaries, then assert the corresponding public records as well |
| 160 | +as the source lifecycle snapshot. This snapshot covers obligations/revisions |
| 161 | +and exact rows, not the entire notification or receipt state machine. |
0 commit comments