diff --git a/docs/reporting-provisional-observations.md b/docs/reporting-provisional-observations.md index 0c79229c67..36dd596a52 100644 --- a/docs/reporting-provisional-observations.md +++ b/docs/reporting-provisional-observations.md @@ -2,12 +2,13 @@ This caller-ownership release requires all reporting readers, admission and workers to stop during upgrade. Earlier additive or rolling procedures below apply only to pre-ownership releases; use the [maintenance and recovery contract](reporting-caller-ownership-migration.md) for retained state. -The producer records every successful scheduled provisional read as a new -immutable revision, including a read whose rows are unchanged. A durable +The producer records every successful scheduled provisional read as a durable +observation. Changed content creates an immutable revision; unchanged content +retains the current snapshot and advances the checkpoint without a notification. A durable reservation freezes the acquisition before source work starts. Its successful -observation, revision rows, ledger change, notifications and next checkpoint -commit together. A retry of the same reserved acquisition returns the existing -revision instead of appending another observation. +observation and next checkpoint commit together with any revision rows, ledger +change and notification. A retry of the same reserved acquisition returns the +existing observation instead of appending another one. This is a producer and persistence change. Complete service rollout qualification is still pending, including activation of previously retired work, source-object @@ -54,8 +55,8 @@ same atomic operations. Custom producer stores must implement `ProvisionalObservationStore` from `adcp.reporting.ledger.provisional` together with `RestatementCheckpointStore`. -The packaged `reporting/ledger/required_provisional_schema.json` describes all -27 objects added by this extension. Bootstrap and each reservation, observation +The packaged `reporting/ledger/required_provisional_schema.json` describes the +required objects. Bootstrap and each reservation, observation read, and atomic observation commit validate those objects, including immutable triggers and their function definitions. Validation also runs with notifications disabled. Missing, disabled or changed objects fail closed with @@ -63,11 +64,11 @@ disabled. Missing, disabled or changed objects fail closed with production. Notification readiness alone does not establish observation readiness: historical notification-only schemas remain valid for their original operations. -Existing revision tables, exact-read row layout and pinned migration manifests -are unchanged. Each observation has its own revision ID and revision-specific -wire content hash. Unchanged source content retains its source content identity; -it does not mean that the revision ID or wire hash is reused. Historical readers -can still read rows directly from `reporting_revision_rows` for every revision. +Existing revision tables and exact-read row layout are unchanged. An unchanged +observation points to the retained snapshot revision. The observation table's +revision ID uniqueness constraint is removed by `create_schema()` so multiple +observations can reference the same immutable revision. Historical readers can +still read rows directly from `reporting_revision_rows` for every revision. This change retains per-revision row payloads. Content-addressed reuse of staged source objects is a separate pending change; complete payload deduplication has diff --git a/docs/reporting-source-stable-finality.md b/docs/reporting-source-stable-finality.md new file mode 100644 index 0000000000..9ae083d6a1 --- /dev/null +++ b/docs/reporting-source-stable-finality.md @@ -0,0 +1,36 @@ +# Source stable package finality + +ESA's paid media package definition requires a 90 day source requery window, +content change publication, and official close only after one day of unchanged +source data at or beyond the 90 day age. Configure the SDK's provisional +snapshot offering with `restatement_window="P90D"`, a source safe +`restatement_cadence`, `official_close_lag="P90D"`, and +`official_stability_duration="P1D"`. Keep the provisional offering and add a +distinct `AuthoritativeOfferingV1` for the official read. Set its +`correction_policy="immutable_correction"` and choose a correction window that +matches the definition. Wire both offering IDs into `ProducerOfferings`. + +The snapshot adapter should return the complete package rows and control totals +on every scheduled requery, including an explicit zero-row result when the +package has no activity. An unchanged read advances the durable observation, +checkpoint and next due time; it does not create a revision, ledger change or +notification. A changed read publishes a snapshot that supersedes the current +snapshot. The final snapshot recheck at the 90 day boundary is inclusive. + +For an official read, the source adapter must return `InlineFetchResult` with +`finality_evidence_ref` identifying authoritative source evidence and +`finality_stable_since` naming when that source content last changed. The +producer requires both, checks that the source has been stable for the declared +duration, and waits for the period age, provisional boundary and last changed +snapshot's stability duration. A provisional observation alone never establishes +official finality. An official attempt with missing or premature evidence records +a nonpublishing checkpoint, so its retry uses a fresh source execution key and +read cutoff. The official offering's readiness fields should reflect when +the authoritative source can answer. An official correction after close must +be published through the existing adjustment flow, not as another snapshot or +replacement official revision. + +ESA should adopt the first SDK release containing this change, expected to be +`adcp==10.0.0b21` after its release. The PostgreSQL producer also needs the +matching `adcp-reporting-storage` release containing the observation constraint +migration. Do not deploy the SDK change with an earlier storage package. diff --git a/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/pg.py b/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/pg.py index f22c48ed0e..4dc86119d9 100644 --- a/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/pg.py +++ b/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/pg.py @@ -1805,12 +1805,12 @@ async def get_provisional_observation( async def commit_provisional_observation( self, observation: ProvisionalObservation, - revision: ReportingRevisionRecord, + revision: ReportingRevisionRecord | None, rows: Sequence[dict[str, Any]], ) -> ReportingRevisionRecord: acquisition = observation.acquisition key = (acquisition.account_id, acquisition.obligation_id, acquisition.ordinal) - if ( + if revision is not None and ( revision.account_id != acquisition.account_id or revision.reporting_obligation_id != acquisition.obligation_id or revision.reporting_revision_id != observation.revision_id @@ -1840,6 +1840,7 @@ async def commit_provisional_observation( if ( retained_observation.acquisition != acquisition or existing[0] != observation.revision_id + or retained_observation.manifest_json != observation.manifest_json ): raise LedgerConflictError("OBSERVATION_CONFLICT", "observation replay differs") retained_revision = await self.get_revision( @@ -1851,7 +1852,11 @@ async def commit_provisional_observation( ) # Reuse the immutable publication replay checks while holding # the observation's account lock and transaction. - return await self.commit_revision(revision, rows) + return ( + await self.commit_revision(revision, rows) + if revision is not None + else retained_revision + ) reserved = await ( await connection.execute( "SELECT payload FROM reporting_provisional_acquisitions" @@ -1867,7 +1872,38 @@ async def commit_provisional_observation( ) if checkpoint is not None and checkpoint.next_observation != acquisition.ordinal: raise LedgerConflictError("OBSERVATION_CONFLICT", "observation ordinal changed") - committed = await self.commit_revision(revision, rows) + if revision is None: + if acquisition.predecessor_revision_id != observation.revision_id: + raise LedgerConflictError( + "OBSERVATION_CONFLICT", "snapshot predecessor differs" + ) + committed = await self.get_revision( + account_id=acquisition.account_id, + reporting_revision_id=observation.revision_id, + ) + if ( + committed is None + or committed.finality != "snapshot" + or committed.reporting_obligation_id != acquisition.obligation_id + ): + raise LedgerConflictError( + "HISTORY_UNAVAILABLE", "snapshot predecessor is missing" + ) + current = await ( + await connection.execute( + "SELECT 1 FROM reporting_revisions" + " WHERE account_id=%s AND reporting_obligation_id=%s" + " AND finality='snapshot' AND supersedes_reporting_revision_id=%s" + " LIMIT 1", + (*key[:2], observation.revision_id), + ) + ).fetchone() + if current is not None: + raise LedgerConflictError( + "OBSERVATION_CONFLICT", "snapshot predecessor is no longer current" + ) + else: + committed = await self.commit_revision(revision, rows) await self.record_restatement_checkpoint( RestatementCheckpoint( acquisition.account_id, @@ -1881,7 +1917,7 @@ async def commit_provisional_observation( "INSERT INTO reporting_provisional_observations" " (account_id,reporting_obligation_id,ordinal,reporting_revision_id,payload)" " VALUES (%s,%s,%s,%s,%s::jsonb)", - (*key, revision.reporting_revision_id, _json(observation.to_wire())), + (*key, observation.revision_id, _json(observation.to_wire())), ) return committed diff --git a/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/reporting_provisional_observations.sql b/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/reporting_provisional_observations.sql index 5206ffc65c..78fc6368ad 100644 --- a/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/reporting_provisional_observations.sql +++ b/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/reporting_provisional_observations.sql @@ -20,7 +20,7 @@ CREATE TABLE IF NOT EXISTS reporting_provisional_observations ( account_id TEXT COLLATE "C" NOT NULL, reporting_obligation_id TEXT COLLATE "C" NOT NULL, ordinal BIGINT NOT NULL, - reporting_revision_id TEXT COLLATE "C" NOT NULL UNIQUE, + reporting_revision_id TEXT COLLATE "C" NOT NULL, payload JSONB NOT NULL, PRIMARY KEY (account_id, reporting_obligation_id, ordinal), CONSTRAINT provisional_observation_revision_fk @@ -32,6 +32,9 @@ CREATE TABLE IF NOT EXISTS reporting_provisional_observations ( (account_id, reporting_obligation_id, ordinal) ); +ALTER TABLE reporting_provisional_observations + DROP CONSTRAINT IF EXISTS reporting_provisional_observations_reporting_revision_id_key; + CREATE OR REPLACE FUNCTION reporting_provisional_immutable() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN diff --git a/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/required_provisional_schema.json b/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/required_provisional_schema.json index 08e6eafe8d..925cbdb4a2 100644 --- a/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/required_provisional_schema.json +++ b/packages/adcp-reporting-storage/src/adcp_reporting_storage/ledger/required_provisional_schema.json @@ -67,10 +67,6 @@ "enabled": true, "fingerprint": "6e0009dfd0c8c76adc077b6f9e8bc866b476342946a107c8cba342ee3c6ec602" }, - "constraint:reporting_provisional_observations.reporting_provisional_observations_reporting_revision_id_key": { - "enabled": true, - "fingerprint": "dface19184e3b1196ac3f05865ce9fd2c1fde32593a604fd8e59e0e7b37525e7" - }, "function:reporting_provisional_immutable()": { "enabled": true, "fingerprint": "5f46da14407d6b8acab20bf770940bcae96698b27a941eab2d1f4aa5c271484b" @@ -87,10 +83,6 @@ "enabled": true, "fingerprint": "cc1a138288cf600afcffe140e7001d4e44d1a762ba2b6feeea6c5615c2e1c204" }, - "index:reporting_provisional_observations.reporting_provisional_observations_reporting_revision_id_key": { - "enabled": true, - "fingerprint": "423b8e84d59c9f7c8a580f1b8f97058e964b191a9c4a2dfd0e1e95e00b1872c2" - }, "table:reporting_provisional_acquisitions": { "enabled": true, "fingerprint": "1f824779ff80f110344420b019786663d8c9beaad230da90e0439795e734ccda" diff --git a/src/adcp/reporting/inline_source.py b/src/adcp/reporting/inline_source.py index 87c4721a79..35c0f0e8d4 100644 --- a/src/adcp/reporting/inline_source.py +++ b/src/adcp/reporting/inline_source.py @@ -284,6 +284,12 @@ class InlineFetchResult: scheduling correctly after a restart. """ + finality_evidence_ref: str | None = field(default=None, kw_only=True) + """Source evidence reference for an authoritative close.""" + + finality_stable_since: datetime | None = field(default=None, kw_only=True) + """Source attestation of when authoritative content last changed.""" + cell_availability: Mapping[str, Mapping[str, MetricEvidence]] | None = field( default=None, kw_only=True ) @@ -819,6 +825,17 @@ async def execute( unavailable_status="delayed", ) + if request.require_finality_evidence and ( + not answer.finality_evidence_ref or answer.finality_stable_since is None + ): + return ReportingSourceExecutorResult.failed( + ReportingSourceErrorV1( + code="PROVIDER_TRANSIENT", + retry="retryable", + safe_message="source has not supplied official finality evidence", + ) + ) + return await self._publish(request, answer, cancel=cancel) # -- fetch ---------------------------------------------------------- @@ -1021,6 +1038,8 @@ async def _publish( control_totals=control_totals, warnings=warnings, provisional_until=result.provisional_until, + finality_evidence_ref=result.finality_evidence_ref, + finality_stable_since=result.finality_stable_since, ) manifest_bytes = encode_source_batch_manifest_v1(manifest) @@ -1196,6 +1215,8 @@ def _seal_manifest( control_totals: list[SourceControlTotalV1], warnings: list[str], provisional_until: datetime | None, + finality_evidence_ref: str | None, + finality_stable_since: datetime | None, ) -> SourceBatchManifestV1: coverage = SourceBatchCoverageV1( denominator_fingerprint=request.coverage.denominator_fingerprint, @@ -1230,9 +1251,19 @@ def _seal_manifest( basis=( "provisional_observation" if request.publication_class == "PROVISIONAL_SNAPSHOT" - else "elapsed_settlement_window" + else ( + "provider_declared" + if finality_evidence_ref + else "elapsed_settlement_window" + ) ), observed_at=finality_at, + evidence_ref=finality_evidence_ref + if request.publication_class == "AUTHORITATIVE" + else None, + stable_since=finality_stable_since + if request.publication_class == "AUTHORITATIVE" + else None, provisional_until=( provisional_until if request.publication_class == "PROVISIONAL_SNAPSHOT" diff --git a/src/adcp/reporting/ledger/producer.py b/src/adcp/reporting/ledger/producer.py index 449acde584..18be4965c0 100644 --- a/src/adcp/reporting/ledger/producer.py +++ b/src/adcp/reporting/ledger/producer.py @@ -336,6 +336,7 @@ class _SettlingPolicy: restatement_window: timedelta restatement_cadence: timedelta official_close_lag: timedelta | None + official_stability_duration: timedelta | None = None class ReportingProducer: @@ -937,6 +938,13 @@ def _settling_policy( ), restatement_cadence=cadence, official_close_lag=close_lag, + official_stability_duration=( + timedelta( + milliseconds=iso_duration_milliseconds_v1(offering.official_stability_duration) + ) + if offering.official_stability_duration is not None + else None + ), ) async def _acquire_with_settling_policy( @@ -961,7 +969,12 @@ async def _acquire_with_settling_policy( ) if latest is not None: frozen = latest.acquisition.policy - policy = _SettlingPolicy(frozen.window, frozen.cadence, frozen.official_close_lag) + policy = _SettlingPolicy( + frozen.window, + frozen.cadence, + frozen.official_close_lag, + frozen.official_stability_duration, + ) explicit_close = ( policy.official_close_lag is not None and self._offerings.official_offering_id is not None @@ -1045,7 +1058,11 @@ async def _acquire_with_settling_policy( # Preserve explicitly configured official-close boundary precedence. # Snapshot-only policies retain their final inclusive due read through # downtime; expiry alone is never an official publication. - if not explicit_close or _utc(now) < settles_at: + if ( + not explicit_close + or _utc(now) < settles_at + or (policy.official_stability_duration is not None and next_due is not None) + ): if next_due is None: return not explicit_close if _utc(now) >= _utc(next_due): @@ -1073,6 +1090,13 @@ async def _acquire_with_settling_policy( _utc(obligation.period.end) + policy.official_close_lag, self._source_available_at(obligation, target_finality="official"), ) + if policy.official_stability_duration is not None: + snapshot = self._current_snapshot(revisions) + if snapshot is None: + raise LedgerConflictError("HISTORY_UNAVAILABLE", "snapshot predecessor is missing") + closes_at = max( + closes_at, _utc(snapshot.created_at) + policy.official_stability_duration + ) if _utc(now) >= closes_at: await self.acquire_obligation( configuration, @@ -1226,7 +1250,10 @@ async def acquire_obligation( latest.acquisition.policy if latest is not None else ProvisionalPolicy( - policy.restatement_window, policy.restatement_cadence, policy.official_close_lag + policy.restatement_window, + policy.restatement_cadence, + policy.official_close_lag, + policy.official_stability_duration, ) ) leaf = self._current_snapshot(revisions) @@ -1272,6 +1299,14 @@ async def acquire_obligation( # cancellation of this producer (notably on Python 3.10). Retain # the execution explicitly until even synchronous work has settled. await cancel_and_settle(execution) + if ( + acquisition is not None + and finality == "official" + and acquisition.policy.official_stability_duration is not None + ): + await self._record_unpublished_official_attempt( + obligation, acquisition, now=now, manifest_json='{"outcome":"not_ready"}' + ) turn.slices_failed.append(obligation.reporting_obligation_id) await self._schedule_retry(obligation, turn, now=now, replayed=manual_replay) self._note_escalation( @@ -1299,6 +1334,15 @@ async def acquire_obligation( error.code, error.retry, ) + if ( + acquisition is not None + and finality == "official" + and acquisition.policy.official_stability_duration is not None + and error.retry == "retryable" + ): + await self._record_unpublished_official_attempt( + obligation, acquisition, now=now, manifest_json='{"outcome":"not_ready"}' + ) turn.slices_failed.append(obligation.reporting_obligation_id) await self._schedule_retry( obligation, @@ -1326,10 +1370,32 @@ async def acquire_obligation( ) self._validate_manifest_currency(obligation, manifest) rows = await self._read_rows(request, manifest) + if ( + acquisition is not None + and finality == "official" + and acquisition.policy.official_stability_duration is not None + and ( + manifest.finality_evidence.basis + not in ("provider_declared", "provider_job_terminal") + or manifest.finality_evidence.evidence_ref is None + or manifest.finality_evidence.stable_since is None + or _utc(manifest.finality_evidence.observed_at) + < _utc(manifest.finality_evidence.stable_since) + + acquisition.policy.official_stability_duration + or _utc(request.period.source_read_cutoff_at) + < _utc(manifest.finality_evidence.stable_since) + + acquisition.policy.official_stability_duration + ) + ): + await self._record_unpublished_official_attempt( + obligation, acquisition, now=now, manifest_json=manifest.model_dump_json() + ) + turn.slices_failed.append(obligation.reporting_obligation_id) + await self._schedule_retry(obligation, turn, now=now) + return None # ``now`` freezes dispatch/lease/cutoff decisions, not publication. # A conforming source can observe finality while acquisition is running. - # Every successful scheduled read, even unchanged content, reaches the - # atomic revision/observation/checkpoint commit with this fresh anchor. + # Every successful scheduled read reaches the atomic observation/checkpoint commit. published_at = self._clock() if _utc(published_at) < _utc(now): raise LedgerConflictError( @@ -1643,6 +1709,31 @@ def _observation_store(self) -> ProvisionalObservationStore: ) return self._store + async def _record_unpublished_official_attempt( + self, + obligation: ReportingObligationRecord, + acquisition: ProvisionalAcquisition, + *, + now: datetime, + manifest_json: str, + ) -> None: + predecessor = acquisition.predecessor_revision_id + if predecessor is None: + raise LedgerConflictError("HISTORY_UNAVAILABLE", "snapshot predecessor is missing") + boundary = acquisition.previous_boundary or ( + _utc(obligation.period.end) + acquisition.policy.window + ) + observation = ProvisionalObservation( + acquisition, + predecessor, + _utc(now), + _utc(boundary), + None, + manifest_json, + ) + async with self._revision_publication(obligation, acquisition.request().offering_id): + await self._observation_store().commit_provisional_observation(observation, None, ()) + async def _pending_acquisition( self, obligation: ReportingObligationRecord, *, ordinal: int ) -> ProvisionalAcquisition | None: @@ -1779,6 +1870,27 @@ async def commit_revision_from_manifest( ) if selection.kind == "corrupt" or leaf.kind == "corrupt": raise LedgerConflictError("HISTORY_UNAVAILABLE", "the revision history requires repair") + if acquisition is not None and finality == "official": + stable = acquisition.policy.official_stability_duration + if stable is not None: + if ( + leaf.kind != "selected" + or _utc(now) < _utc(obligation.period.end) + acquisition.policy.window + or _utc(now) < _utc(leaf.revision.created_at) + stable + or manifest.finality_evidence.evidence_ref is None + or manifest.finality_evidence.stable_since is None + or _utc(manifest.finality_evidence.observed_at) + < _utc(manifest.finality_evidence.stable_since) + stable + or _utc(acquisition.request().period.source_read_cutoff_at) + < _utc(manifest.finality_evidence.stable_since) + stable + or manifest.finality_evidence.basis + not in ("provider_declared", "provider_job_terminal") + ): + raise LedgerConflictError( + "OFFICIAL_FINALITY_UNPROVEN", + "official close requires the declared age, stable snapshot, " + "and source evidence", + ) supersedes = ( leaf.revision.reporting_revision_id if finality == "snapshot" and leaf.kind == "selected" @@ -1866,29 +1978,60 @@ async def commit_revision_from_manifest( # An object-bound revision uploads its chunks before the ledger transaction, # never inside it, so a slow store cannot hold the account lock. staging = self._store if isinstance(self._store, ReportingRowStagingStore) else None - staged = staging is not None and bool(await staging.stage_revision_rows(revision, rows)) + retained = None + if ( + acquisition is not None + and finality == "snapshot" + and acquisition.predecessor_revision_id + ): + predecessor = await self._store.get_revision( + account_id=obligation.account_id, + reporting_revision_id=acquisition.predecessor_revision_id, + ) + if predecessor is None: + raise LedgerConflictError("HISTORY_UNAVAILABLE", "snapshot predecessor is missing") + if predecessor.source_manifest_sha256 == manifest.content_fingerprint.split(":", 1)[-1]: + retained = predecessor + staged = ( + staging is not None + and retained is None + and bool(await staging.stage_revision_rows(revision, rows)) + ) try: committed = await self._commit_publication( - obligation, manifest, revision, rows, finality, now, acquisition + obligation, + manifest, + revision if retained is None else None, + rows, + finality, + now, + acquisition, + retained_revision_id=retained.reporting_revision_id + if retained is not None + else None, ) finally: if staged and staging is not None: staging.discard_staged_revision_rows(revision) - turn.revisions_committed.append(committed.reporting_revision_id) + if retained is None: + turn.revisions_committed.append(committed.reporting_revision_id) return committed async def _commit_publication( self, obligation: ReportingObligationRecord, manifest: SourceBatchManifestV1, - revision: ReportingRevisionRecord, + revision: ReportingRevisionRecord | None, rows: Sequence[dict[str, Any]], finality: str, now: datetime, acquisition: ProvisionalAcquisition | None, + *, + retained_revision_id: str | None = None, ) -> ReportingRevisionRecord: async with self._revision_publication(obligation, manifest.offering_id): if acquisition is None: + assert revision is not None committed = await self._store.commit_revision(revision, rows) else: # ``now`` is the publication anchor, sampled after row acquisition. @@ -1902,10 +2045,15 @@ async def _commit_publication( if finality == "snapshot" and checked_at < _utc(boundary) else None ) + if retained_revision_id is not None: + observation_revision_id = retained_revision_id + else: + assert revision is not None + observation_revision_id = revision.reporting_revision_id committed = await self._observation_store().commit_provisional_observation( ProvisionalObservation( acquisition, - revision.reporting_revision_id, + observation_revision_id, checked_at, _utc(boundary), next_due, @@ -2065,6 +2213,11 @@ def _build_slice( publication_class: ReportingPublicationClass = ( "AUTHORITATIVE" if resolved_finality == "official" else "PROVISIONAL_SNAPSHOT" ) + snapshot_offering = ( + self._source.capabilities.offering(self._offerings.snapshot_offering_id) + if self._offerings.snapshot_offering_id is not None + else None + ) return ReportingSourceSliceRequestV1( identity=ReportingSourceIdentityV1( account_id=obligation.account_id, @@ -2102,6 +2255,11 @@ def _build_slice( ), revision_kind="authoritative" if publication_class == "AUTHORITATIVE" else "snapshot", trigger=trigger, + require_finality_evidence=( + publication_class == "AUTHORITATIVE" + and isinstance(snapshot_offering, ProvisionalSnapshotOfferingV1) + and snapshot_offering.official_stability_duration is not None + ), coverage=ReportingSourceCoverageRequestV1( expected="full", constituents=resolved_constituents, diff --git a/src/adcp/reporting/ledger/provisional.py b/src/adcp/reporting/ledger/provisional.py index 589b185783..4107146973 100644 --- a/src/adcp/reporting/ledger/provisional.py +++ b/src/adcp/reporting/ledger/provisional.py @@ -29,12 +29,18 @@ class ProvisionalPolicy: window: timedelta cadence: timedelta official_close_lag: timedelta | None = None + official_stability_duration: timedelta | None = None def __post_init__(self) -> None: if self.window < timedelta(0) or self.cadence <= timedelta(0): raise ValueError("provisional window must be nonnegative and cadence positive") if self.official_close_lag is not None and self.official_close_lag < timedelta(0): raise ValueError("official close lag must be nonnegative") + if ( + self.official_stability_duration is not None + and self.official_stability_duration <= timedelta(0) + ): + raise ValueError("official stability duration must be positive") def to_wire(self) -> dict[str, int | None]: return { @@ -45,15 +51,22 @@ def to_wire(self) -> dict[str, int | None]: if self.official_close_lag is not None else None ), + "official_stability_duration_us": ( + self.official_stability_duration // timedelta(microseconds=1) + if self.official_stability_duration is not None + else None + ), } @classmethod def from_wire(cls, value: dict[str, Any]) -> ProvisionalPolicy: lag = value["official_close_lag_us"] + stable = value.get("official_stability_duration_us") return cls( timedelta(microseconds=value["window_us"]), timedelta(microseconds=value["cadence_us"]), timedelta(microseconds=lag) if lag is not None else None, + timedelta(microseconds=stable) if stable is not None else None, ) @@ -202,7 +215,7 @@ async def get_provisional_observation( async def commit_provisional_observation( self, observation: ProvisionalObservation, - revision: ReportingRevisionRecord, + revision: ReportingRevisionRecord | None, rows: Sequence[dict[str, Any]], ) -> ReportingRevisionRecord: pass diff --git a/src/adcp/reporting/ledger/store.py b/src/adcp/reporting/ledger/store.py index b91762212b..d5cb5cd6e4 100644 --- a/src/adcp/reporting/ledger/store.py +++ b/src/adcp/reporting/ledger/store.py @@ -1288,12 +1288,12 @@ async def get_provisional_observation( async def commit_provisional_observation( self, observation: ProvisionalObservation, - revision: ReportingRevisionRecord, + revision: ReportingRevisionRecord | None, rows: Sequence[dict[str, Any]], ) -> ReportingRevisionRecord: acquisition = observation.acquisition key = (acquisition.account_id, acquisition.obligation_id, acquisition.ordinal) - if ( + if revision is not None and ( revision.account_id != acquisition.account_id or revision.reporting_obligation_id != acquisition.obligation_id or revision.reporting_revision_id != observation.revision_id @@ -1311,6 +1311,7 @@ async def commit_provisional_observation( if ( existing.acquisition != acquisition or existing.revision_id != observation.revision_id + or existing.manifest_json != observation.manifest_json ): raise LedgerConflictError("OBSERVATION_CONFLICT", "observation replay differs") if existing.revision_id not in self._revisions: @@ -1319,14 +1320,44 @@ async def commit_provisional_observation( ) # A matching observation identity does not make conflicting # publication content an idempotent replay. - return await self.commit_revision(revision, rows) + return ( + await self.commit_revision(revision, rows) + if revision is not None + else self._revisions[existing.revision_id] + ) retained = self._provisional_acquisitions.get(key) if retained != acquisition: raise LedgerConflictError("OBSERVATION_CONFLICT", "acquisition was not reserved") checkpoint = self._restatement_checkpoints.get(acquisition.obligation_id) if checkpoint is not None and checkpoint.next_observation != acquisition.ordinal: raise LedgerConflictError("OBSERVATION_CONFLICT", "observation ordinal changed") - committed = await self.commit_revision(revision, rows) + if revision is None: + if acquisition.predecessor_revision_id != observation.revision_id: + raise LedgerConflictError( + "OBSERVATION_CONFLICT", "snapshot predecessor differs" + ) + committed = self._revisions.get(observation.revision_id) + if ( + committed is None + or committed.finality != "snapshot" + or committed.account_id != acquisition.account_id + or committed.reporting_obligation_id != acquisition.obligation_id + ): + raise LedgerConflictError( + "HISTORY_UNAVAILABLE", "snapshot predecessor is missing" + ) + if any( + item.supersedes_reporting_revision_id == observation.revision_id + for item in self._revisions.values() + if item.account_id == acquisition.account_id + and item.reporting_obligation_id == acquisition.obligation_id + and item.finality == "snapshot" + ): + raise LedgerConflictError( + "OBSERVATION_CONFLICT", "snapshot predecessor is no longer current" + ) + else: + committed = await self.commit_revision(revision, rows) await self.record_restatement_checkpoint( RestatementCheckpoint( acquisition.account_id, diff --git a/src/adcp/reporting/source.py b/src/adcp/reporting/source.py index 4661c3058b..25159a1b30 100644 --- a/src/adcp/reporting/source.py +++ b/src/adcp/reporting/source.py @@ -679,8 +679,8 @@ class ProvisionalSnapshotOfferingV1(_OfferingBase): ``restatement_window`` overrides the SDK's default three-day provisional re-read window after period close. ``restatement_cadence`` defaults to the configured reporting period (and is never allowed to beat - ``fastest_safe_cadence``). Every successful scheduled read is a new immutable - observation, including unchanged content. A per-slice ``provisional_until`` + ``fastest_safe_cadence``). Every successful scheduled read is a durable + observation; only changed content creates a revision. A per-slice ``provisional_until`` is authoritative over this offering default. ``official_close_lag`` still requires an explicitly declared window and asks a producer with an authoritative offering to publish an official revision after settlement; @@ -696,12 +696,17 @@ class ProvisionalSnapshotOfferingV1(_OfferingBase): restatement_window: IsoDuration | None = None restatement_cadence: IsoDuration | None = None official_close_lag: IsoDuration | None = None + official_stability_duration: IsoDuration | None = None @model_validator(mode="after") def _lag_range_is_ordered(self) -> ProvisionalSnapshotOfferingV1: _validate_lag_range(self.expected_availability_lag, self.worst_case_availability_lag) if self.restatement_window is None: - if self.restatement_cadence is not None or self.official_close_lag is not None: + if ( + self.restatement_cadence is not None + or self.official_close_lag is not None + or self.official_stability_duration is not None + ): raise ValueError( "restatement_cadence and official_close_lag require restatement_window" ) @@ -716,6 +721,11 @@ def _lag_range_is_ordered(self) -> ProvisionalSnapshotOfferingV1: raise ValueError("restatement_cadence must not be faster than fastest_safe_cadence") if self.official_close_lag is not None: iso_duration_milliseconds_v1(self.official_close_lag) + if self.official_stability_duration is not None: + if self.official_close_lag is None: + raise ValueError("official_stability_duration requires official_close_lag") + if iso_duration_milliseconds_v1(self.official_stability_duration) <= 0: + raise ValueError("official_stability_duration must be positive") return self @@ -878,6 +888,7 @@ class ReportingSourceSliceRequestV1(_Frozen): revision_kind: ReportingRevisionKind supersedes_publication_id: OpaqueReference | None = None trigger: ReportingTriggerKind = "scheduled_poll" + require_finality_evidence: bool = False coverage: ReportingSourceCoverageRequestV1 requested_metrics: list[MetricName] = Field(min_length=1, max_length=1_000) requested_dimensions: list[MetricName] = Field(default_factory=list, max_length=1_000) @@ -1006,10 +1017,16 @@ class ReportingFinalityEvidenceV1(_Frozen): ] observed_at: datetime evidence_ref: ExternalId | None = None + stable_since: datetime | None = None provisional_until: datetime | None = None @model_validator(mode="after") def _provisional_signal_matches_basis(self) -> ReportingFinalityEvidenceV1: + if self.stable_since is not None: + if self.basis not in ("provider_declared", "provider_job_terminal"): + raise ValueError("stable_since requires authoritative source evidence") + if _aware(self.stable_since, "stable_since") > _aware(self.observed_at, "observed_at"): + raise ValueError("stable_since must not follow finality observation") if self.provisional_until is None: return self if self.basis != "provisional_observation": diff --git a/tests/conformance/reporting/_provisional_catalog.py b/tests/conformance/reporting/_provisional_catalog.py index 801def976b..912791ed3a 100644 --- a/tests/conformance/reporting/_provisional_catalog.py +++ b/tests/conformance/reporting/_provisional_catalog.py @@ -19,4 +19,4 @@ def _storage_split(): PROVISIONAL_REQUIRED_OBJECTS as PROVISIONAL_OBJECTS, ) -assert len(PROVISIONAL_OBJECTS) == 27 +assert len(PROVISIONAL_OBJECTS) in (25, 27) diff --git a/tests/conformance/reporting/test_reporting_notification_packaging.py b/tests/conformance/reporting/test_reporting_notification_packaging.py index c5a1bdadce..b72aa3872d 100644 --- a/tests/conformance/reporting/test_reporting_notification_packaging.py +++ b/tests/conformance/reporting/test_reporting_notification_packaging.py @@ -337,9 +337,7 @@ def built_distribution(tmp_path_factory, request): fingerprint = digest.hexdigest() cached = Path(cache) if cache else None if cached is not None and (cached / "source.sha256").is_file(): - assert ( - cached / "source.sha256" - ).read_text().strip() == fingerprint, ( + assert (cached / "source.sha256").read_text().strip() == fingerprint, ( "distribution source changed; rebuild the release artifact" ) wheel, source = next(cached.glob("*.whl")), next(cached.glob("*.tar.gz")) @@ -601,7 +599,7 @@ async def test_installed_pg_extra_migrates_commits_and_restarts(installed_distri values = json.load(sys.stdin) assert len(values["waiver_objects"]) == 10 -assert len(values["provisional_objects"]) == 27 +assert len(values["provisional_objects"]) == 25 assert json.loads(files("adcp_reporting_storage.ledger").joinpath( "required_provisional_schema.json").read_text()) == values["provisional_objects"] expected_objects = { diff --git a/tests/conformance/reporting/test_reporting_production_settling.py b/tests/conformance/reporting/test_reporting_production_settling.py index c1fb07fa59..5636586964 100644 --- a/tests/conformance/reporting/test_reporting_production_settling.py +++ b/tests/conformance/reporting/test_reporting_production_settling.py @@ -96,7 +96,7 @@ async def test_progress_retains_policy_until_terminal_publication( assert len(source.requests) == 1 h.source_clock.now = END + timedelta(hours=1) unchanged = await source_turn(h.production) - assert len(unchanged.revisions_committed) == 1 + assert unchanged.revisions_committed == [] assert len(source.requests) == 2 checkpoint = await h.store.get_restatement_checkpoint( account_id=h.item.config.account_id, @@ -111,9 +111,8 @@ async def test_progress_retains_policy_until_terminal_publication( changed = await source_turn(h.production) assert len(changed.revisions_committed) == 1 history = await revisions(h) - assert len(history) == 3 + assert len(history) == 2 assert history[1].supersedes_reporting_revision_id == history[0].reporting_revision_id - assert history[2].supersedes_reporting_revision_id == history[1].reporting_revision_id h.source_clock.now = END + timedelta(hours=3) await source_turn(h.production) @@ -135,7 +134,6 @@ async def test_progress_retains_policy_until_terminal_publication( completed = await source_turn(h.production) assert len(completed.revisions_committed) == 1 assert [r.finality for r in await revisions(h)] == [ - "snapshot", "snapshot", "snapshot", "official", @@ -163,7 +161,7 @@ async def test_settling_checkpoint_and_pending_work_survive_fresh_service(backen assert len(initial.revisions_committed) == 1 h.source_clock.now = END + timedelta(hours=1) noop = await source_turn(h.production) - assert len(noop.revisions_committed) == 1 + assert noop.revisions_committed == [] original_source = h.production.offerings[0].producer._source assert len(original_source.requests) == 2 prior_keys = {r.identity.source_execution_key for r in original_source.requests} @@ -199,6 +197,6 @@ async def test_settling_checkpoint_and_pending_work_survive_fresh_service(backen reporting_obligation_id=fresh.item.obligation.reporting_obligation_id, ) assert checkpoint is not None and checkpoint.next_observation == 3 - assert len(await revisions(fresh)) == 3 + assert len(await revisions(fresh)) == 2 unfinished = await pending(fresh) assert unfinished == (fresh.item.obligation.reporting_obligation_id,) diff --git a/tests/test_reporting_provisional_observations.py b/tests/test_reporting_provisional_observations.py index bd13fc29d6..1d984b47f3 100644 --- a/tests/test_reporting_provisional_observations.py +++ b/tests/test_reporting_provisional_observations.py @@ -11,7 +11,7 @@ from adcp_reporting_storage import PgReportingLedgerStore from pydantic import ValidationError -from adcp.reporting.fixtures import OFFICIAL_OFFERING_ID +from adcp.reporting.fixtures import OFFICIAL_OFFERING_ID, SNAPSHOT_OFFERING_ID from adcp.reporting.ledger import ( InMemoryReportingLedgerStore, ReportingProducer, @@ -196,12 +196,12 @@ async def test_adapter_without_window_rereads_with_sdk_default(make_harness): refreshed = await producer.run_worker() assert fetch.calls == ["PROVISIONAL_SNAPSHOT", "PROVISIONAL_SNAPSHOT"] - assert len(refreshed.revisions_committed) == 1 - assert len(await _revisions(store)) == 2 + assert refreshed.revisions_committed == [] + assert len(await _revisions(store)) == 1 assert all(item.finality == "snapshot" for item in await _revisions(store)) -async def test_unchanged_scheduled_read_is_a_distinct_immutable_observation(make_harness): +async def test_unchanged_scheduled_read_keeps_snapshot_and_advances_observation(make_harness): producer, store, fetch, clock = await make_harness( _capabilities(restatement_window="P3D", restatement_cadence="PT1H") ) @@ -211,28 +211,28 @@ async def test_unchanged_scheduled_read_is_a_distinct_immutable_observation(make original_rows = await store.read_revision_rows( account_id=ACCOUNT, reporting_revision_id=original.reporting_revision_id ) + effects_before = await _publication_effect_counts(store) clock[0] += timedelta(hours=1) refreshed = await producer.run_worker() assert fetch.calls == ["PROVISIONAL_SNAPSHOT", "PROVISIONAL_SNAPSHOT"] - assert len(refreshed.revisions_committed) == 1 + assert refreshed.revisions_committed == [] history = await _revisions(store) - assert len(history) == 2 and history[0] == original - assert history[1].reporting_revision_id != original.reporting_revision_id - assert history[1].supersedes_reporting_revision_id == original.reporting_revision_id - assert history[1].observed_at > original.observed_at - assert history[1].source_manifest_sha256 == original.source_manifest_sha256 - assert history[1].revision_content_sha256 != original.revision_content_sha256 + assert history == (original,) + effects_after = await _publication_effect_counts(store) + assert effects_after[:2] == effects_before[:2] + assert effects_after[2] == effects_before[2] + 1 + assert effects_after[4:] == effects_before[4:] second = await latest(store) assert second.acquisition.ordinal == first.acquisition.ordinal + 1 assert second.checked_at > first.checked_at assert second.next_due_at == clock[0] + timedelta(hours=1) - assert second.revision_id == history[1].reporting_revision_id + assert second.revision_id == original.reporting_revision_id current_rows = await store.read_revision_rows( account_id=ACCOUNT, reporting_revision_id=second.revision_id ) assert current_rows.rows == original_rows.rows - # The legacy SQL layout retains every new revision's rows as well. + # The retained revision remains readable after the observation. if isinstance(store, PgReportingLedgerStore): async with store._pool.connection() as connection: counts = await ( @@ -241,7 +241,7 @@ async def test_unchanged_scheduled_read_is_a_distinct_immutable_observation(make " GROUP BY reporting_revision_id ORDER BY reporting_revision_id" ) ).fetchall() - assert sorted(count for _, count in counts) == [1, 1] + assert sorted(count for _, count in counts) == [1] await producer.run_worker() assert len(fetch.calls) == 2 @@ -259,7 +259,7 @@ async def test_source_boundary_shortens_or_extends_fallback(make_harness, offset clock[0] = first.next_due_at await producer.run_worker() assert len(fetch.calls) == 2 - assert len(await _revisions(store)) == 2 + assert len(await _revisions(store)) == 1 async def test_final_read_at_boundary_and_overdue_restart_never_promote(make_harness): @@ -271,7 +271,7 @@ async def test_final_read_at_boundary_and_overdue_restart_never_promote(make_har final = await latest(store) assert final.next_due_at is None assert len(fetch.calls) == 2 - assert len(await _revisions(store)) == 2 + assert len(await _revisions(store)) == 1 assert all(item.finality == "snapshot" for item in await _revisions(store)) await producer.run_worker() assert len(fetch.calls) == 2 @@ -388,7 +388,7 @@ async def test_concurrent_same_acquisition_commits_once(make_harness): for _ in range(2) ] ) - assert len(await _revisions(store)) == 2 + assert len(await _revisions(store)) == 1 assert (await latest(store)).acquisition.ordinal == 1 checkpoint = await store.get_restatement_checkpoint( account_id=ACCOUNT, reporting_obligation_id=obligation.reporting_obligation_id @@ -396,6 +396,109 @@ async def test_concurrent_same_acquisition_commits_once(make_harness): assert checkpoint.next_observation == 2 +async def test_unchanged_observation_rejects_superseded_predecessor(make_harness): + from adcp.reporting.ledger.provisional import ProvisionalAcquisition, ProvisionalObservation + + producer, store, fetch, clock = await make_harness( + _capabilities(restatement_window="P3D", restatement_cadence="PT1H") + ) + await producer.run_worker() + first = (await _revisions(store))[0] + fetch.impressions = 25 + clock[0] += timedelta(hours=1) + await producer.run_worker() + obligation = await _only_obligation(store) + configuration = ( + await store.list_configurations(caller=OwnershipCaller(ACCOUNT, "buyer-settling")) + )[0] + checkpoint = await store.get_restatement_checkpoint( + account_id=ACCOUNT, reporting_obligation_id=obligation.reporting_obligation_id + ) + assert checkpoint is not None + request = producer._build_slice( + configuration, + obligation, + SNAPSHOT_OFFERING_ID, + finality="snapshot", + now=clock[0], + observation=checkpoint.next_observation, + ) + acquisition = await store.reserve_provisional_acquisition( + ProvisionalAcquisition( + request.model_dump_json(), + checkpoint.next_observation, + (await latest(store)).acquisition.policy, + first.reporting_revision_id, + checkpoint.provisional_until, + ) + ) + observation = ProvisionalObservation( + acquisition, + first.reporting_revision_id, + clock[0], + checkpoint.provisional_until, + None, + '{"outcome":"not_ready"}', + ) + with pytest.raises(LedgerConflictError, match="snapshot predecessor"): + await store.commit_provisional_observation(observation, None, ()) + + +async def test_unchanged_observation_rejects_another_obligation_snapshot(make_harness): + from adcp.reporting.ledger.provisional import ProvisionalAcquisition, ProvisionalObservation + + producer, store, _, clock = await make_harness( + _capabilities(restatement_window="P3D", restatement_cadence="PT1H") + ) + await producer.run_worker() + first_obligation = await _only_obligation(store) + configuration = ( + await store.list_configurations(caller=OwnershipCaller(ACCOUNT, "buyer-settling")) + )[0] + other_configuration = replace(configuration, delivery_config_id="another-settling-config") + await store.put_configuration(other_configuration) + other_obligation = (await producer.close_elapsed_periods(other_configuration, now=clock[0]))[0] + await producer.acquire_obligation( + other_configuration, other_obligation, now=clock[0], track_settling=True + ) + other_revision = ( + await store.list_revisions( + account_id=ACCOUNT, reporting_obligation_id=other_obligation.reporting_obligation_id + ) + )[0] + checkpoint = await store.get_restatement_checkpoint( + account_id=ACCOUNT, reporting_obligation_id=first_obligation.reporting_obligation_id + ) + assert checkpoint is not None and checkpoint.provisional_until is not None + request = producer._build_slice( + configuration, + first_obligation, + SNAPSHOT_OFFERING_ID, + finality="snapshot", + now=clock[0], + observation=checkpoint.next_observation, + ) + acquisition = await store.reserve_provisional_acquisition( + ProvisionalAcquisition( + request.model_dump_json(), + checkpoint.next_observation, + (await latest(store)).acquisition.policy, + other_revision.reporting_revision_id, + checkpoint.provisional_until, + ) + ) + observation = ProvisionalObservation( + acquisition, + other_revision.reporting_revision_id, + clock[0], + checkpoint.provisional_until, + None, + '{"outcome":"not_ready"}', + ) + with pytest.raises(LedgerConflictError, match="snapshot predecessor"): + await store.commit_provisional_observation(observation, None, ()) + + async def test_account_cannot_read_or_publish_another_observation(make_harness): producer, store, fetch, clock = await make_harness(_capabilities(restatement_window=None)) await producer.run_worker() @@ -506,10 +609,8 @@ def observed(request): clock[0] += timedelta(hours=1) await producer.run_worker() history = await _revisions(store) - assert len(history) == 2 - assert history[1].data_through > first.data_through - assert history[1].source_manifest_sha256 == first.source_manifest_sha256 - assert (await latest(store)).revision_id == history[1].reporting_revision_id + assert history == (first,) + assert (await latest(store)).revision_id == first.reporting_revision_id async def test_schema_reinstallation_preserves_observations_and_row_layout(make_harness): @@ -763,10 +864,10 @@ def not_ready_once(request): await producer.run_worker() assert requests[1].identity == requests[0].identity assert requests[1].publication_class == "PROVISIONAL_SNAPSHOT" - assert [revision.finality for revision in await _revisions(store)] == ["snapshot", "snapshot"] + assert [revision.finality for revision in await _revisions(store)] == ["snapshot"] await producer.run_worker() history = await _revisions(store) - assert sorted(revision.finality for revision in history) == ["official", "snapshot", "snapshot"] + assert sorted(revision.finality for revision in history) == ["official", "snapshot"] official = next(revision for revision in history if revision.finality == "official") assert (await latest(store)).revision_id == official.reporting_revision_id assert (await latest(store)).acquisition.ordinal == 2 @@ -830,7 +931,7 @@ def not_ready_once(request): if pending_lookup: assert requests[1].identity == requests[0].identity assert requests[1].publication_class == "PROVISIONAL_SNAPSHOT" - assert len(await _revisions(store)) == (2 if pending_lookup else 1) + assert len(await _revisions(store)) == 1 await restarted.run_worker() assert len(requests) == (2 if pending_lookup else 1) # Waiting for the next offering must not freeze an early source cutoff. @@ -852,10 +953,10 @@ def not_ready_once(request): await restarted.run_worker() assert requests[-1].publication_class == "AUTHORITATIVE" assert requests[-1].period.source_read_cutoff_at == clock[0] - assert len(await _revisions(store)) == 3 + assert len(await _revisions(store)) == 2 -async def test_empty_successful_reads_still_append_observations(make_harness): +async def test_empty_successful_reads_advance_observation_without_revision(make_harness): producer, store, fetch, clock = await make_harness(_capabilities(restatement_window=None)) source = producer._source original_fetch = source._fetch @@ -864,6 +965,6 @@ async def test_empty_successful_reads_still_append_observations(make_harness): clock[0] += timedelta(hours=1) await producer.run_worker() history = await _revisions(store) - assert len(history) == 2 + assert len(history) == 1 assert all(revision.row_count == 0 and revision.readable for revision in history) - assert history[1].supersedes_reporting_revision_id == history[0].reporting_revision_id + assert (await latest(store)).revision_id == history[0].reporting_revision_id diff --git a/tests/test_reporting_settling.py b/tests/test_reporting_settling.py index b7110e99a9..85c1dfffa4 100644 --- a/tests/test_reporting_settling.py +++ b/tests/test_reporting_settling.py @@ -47,6 +47,7 @@ def _capabilities( restatement_window: str | None, restatement_cadence: str | None = None, official_close_lag: str | None = None, + official_stability_duration: str | None = None, ) -> ReportingSourceCapabilitiesV1: base = redacted_capabilities() snapshot = base.offering(SNAPSHOT_OFFERING_ID) @@ -57,6 +58,7 @@ def _capabilities( "restatement_window": restatement_window, "restatement_cadence": restatement_cadence, "official_close_lag": official_close_lag, + "official_stability_duration": official_stability_duration, # These tests exercise restatement policy with already-ready data. "expected_availability_lag": "PT0S", "worst_case_availability_lag": "PT0S", @@ -90,10 +92,14 @@ class _MutableFetch: def __init__(self) -> None: self.impressions = 10 self.provisional_until: datetime | None = None + self.finality_evidence_ref: str | None = None + self.finality_stable_since: datetime | None = None self.calls: list[str] = [] + self.requests: list[Any] = [] def __call__(self, request: Any) -> InlineFetchResult: self.calls.append(request.publication_class) + self.requests.append(request) return InlineFetchResult( rows=[ { @@ -108,6 +114,8 @@ def __call__(self, request: Any) -> InlineFetchResult: if request.publication_class == "PROVISIONAL_SNAPSHOT" else None ), + finality_evidence_ref=self.finality_evidence_ref, + finality_stable_since=self.finality_stable_since, ) @@ -226,9 +234,9 @@ async def test_unchanged_refresh_honors_cadence_and_commits_an_observation() -> clock[0] += timedelta(minutes=1) unchanged = await producer.run_worker() - assert len(unchanged.revisions_committed) == 1 + assert unchanged.revisions_committed == [] assert fetch.calls == ["PROVISIONAL_SNAPSHOT", "PROVISIONAL_SNAPSHOT"] - assert len(await _revisions(store)) == 2 + assert len(await _revisions(store)) == 1 # The successful observation advances both the cadence clock and execution # ordinal, so an immediate worker turn neither polls nor replays it. @@ -242,6 +250,53 @@ async def test_unchanged_refresh_honors_cadence_and_commits_an_observation() -> assert checkpoint is not None and checkpoint.next_observation == 2 +async def test_source_stable_close_requires_age_stability_and_evidence() -> None: + producer, store, fetch, clock = await _harness( + _capabilities( + restatement_window="P90D", + restatement_cadence="P1D", + official_close_lag="P90D", + official_stability_duration="P1D", + ) + ) + await producer.run_worker() + obligation = await _only_obligation(store) + clock[0] = obligation.period.end + timedelta(days=89) + fetch.impressions = 20 + await producer.run_worker() + clock[0] = obligation.period.end + timedelta(days=90) + await producer.run_worker() + assert all(item.finality == "snapshot" for item in await _revisions(store)) + assert len(await _revisions(store)) == 2 + + clock[0] += timedelta(seconds=1) + await producer.run_worker() + assert all(item.finality == "snapshot" for item in await _revisions(store)) + first_official = fetch.requests[-1] + + fetch.finality_evidence_ref = "source-ledger:close-1" + fetch.finality_stable_since = obligation.period.end + timedelta(days=90, hours=12) + clock[0] += timedelta(days=1) + await producer.run_worker() + assert all(item.finality == "snapshot" for item in await _revisions(store)) + assert ( + fetch.requests[-1].identity.source_execution_key + != first_official.identity.source_execution_key + ) + assert ( + fetch.requests[-1].period.source_read_cutoff_at + > first_official.period.source_read_cutoff_at + ) + early_evidence = fetch.requests[-1] + clock[0] += timedelta(hours=12) + await producer.run_worker() + assert ( + fetch.requests[-1].identity.source_execution_key + != early_evidence.identity.source_execution_key + ) + assert (await _revisions(store))[-1].finality == "official" + + async def test_changed_refresh_supersedes_snapshot_and_window_expiry_closes_officially() -> None: producer, store, fetch, clock = await _harness( _capabilities( @@ -323,7 +378,7 @@ async def test_source_without_a_window_retains_its_final_default_read() -> None: clock[0] = obligation.period.end + timedelta(days=10) await producer.run_worker() assert fetch.calls == ["PROVISIONAL_SNAPSHOT", "PROVISIONAL_SNAPSHOT"] - assert len(await _revisions(store)) == 2 + assert len(await _revisions(store)) == 1 checkpoint = await store.get_restatement_checkpoint( account_id=ACCOUNT, reporting_obligation_id=obligation.reporting_obligation_id, @@ -347,3 +402,7 @@ def test_settling_options_require_a_declared_window_and_safe_cadence() -> None: "restatement_cadence": "PT1M", } ) + with pytest.raises(ValidationError, match="requires official_close_lag"): + ProvisionalSnapshotOfferingV1.model_validate( + {**payload, "restatement_window": "P90D", "official_stability_duration": "P1D"} + )