Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
183 changes: 51 additions & 132 deletions .agents/shortcuts/verification-profile-shadow-rollout.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,35 +6,50 @@ heartbeats. It adds a bounded database read/write after public processing, but
does not make another agent request or change public grading, badges, or
notifications.

## 1. Deploy disabled
## 1. Deploy read-only comparison

Migration `572_verification_profile_shadow_rollout.sql` creates the bounded
shadow ledger and inserts the audited setting as
`{ "enabled": false, "expires_at": null }`.
After deploy, confirm the admin settings response reports it disabled.
Migration `591_enable_verification_profile_comparisons.sql` turns the existing
audited switch on persistently as `{ "enabled": true, "expires_at": null }`.
It records the old and new values in `system_settings_audit`. The worker reuses
completed authoritative heartbeat evidence; it sends no additional request to
an agent and changes no public status, badge, token, or notification.

Before enabling, collect a full 48-hour disabled baseline from the deployed
worker's structured `Compliance heartbeat shadow flush completed after public
processing` records. Every scheduled heartbeat, including an empty queue, must
emit this record with:
The owner dashboard initially presents the result as an **agent-wide preview**.
It must not be described as an exact `(agent, role, AdCP release)` result. Role
selection remains disabled until role-scoped immutable assessments exist. A
stale, absent, malformed, or unreadable comparison is shown as unavailable and
never as passing.

- `publicProcessingDurationMs`;
- `queue.eligibleBacklog`, `queue.selectedCount`, and `queue.batchLimit`;
- policy, SDK, selected target, and compliance-cache versions in `inputs`;
- aggregate `outcomes` and `skipReasons` (never endpoint URLs); and
- shadow candidates, attempts, writes, disables, setting/write errors, and
write latency.
Do not deploy profile selection or enforcement during this phase.

Do not infer a zero backlog from missing logs. Missing baseline samples make
the health comparison unproven. Record the deployed image SHA, policy version,
SDK version, and compliance-cache/catalog version at baseline start. Freeze
those worker/compliance inputs through the baseline and observation window. A
change to any of them, or a machine-convergence gap, invalidates the comparison
and requires a fresh disabled baseline.
## 2. Observe ordinary production health

Do not deploy the enforcement patch during this phase.
Every scheduled heartbeat, including an empty queue, emits the structured
`Compliance heartbeat shadow flush completed after public processing` record
with aggregate queue, outcome, skip-reason, policy/SDK/target, write-count, and
latency fields. Endpoint URLs are excluded.

## 2. Enable collection through the authenticated admin API
Monitor normal deployments and traffic. A deployment or cache-version change
does not restart a frozen evidence window; provenance on each immutable row
identifies the source run and evaluator version. Check:

- comparison write errors and disabled writes;
- p95 public processing duration and eligible queue backlog;
- the distribution of current, stale, pending, and unavailable owner previews;
- Spec partial/failing causes and all unresolved Sandbox bundles; and
- whether evidence explanations match the source run.

The audit command remains available for aggregate and restricted diagnosis:

```bash
npm run audit:verification-profile-shadow -- --hours=48
```

Add `--include-agents` only for a restricted operator review. That output can
contain endpoint URLs and must not be pasted into public issues or pull
requests.

## 3. Disable or re-enable collection

From a signed-in global-admin browser session, use the same-origin console:

Expand All @@ -46,118 +61,22 @@ await fetch('/api/admin/settings/verification-profile-shadow-rollout', {
}).then((response) => response.json())
```

This write is recorded in `system_settings_audit` with the authenticated
credential identifier and returns an `expires_at` value 72 hours in the future.
The lease disables itself at expiry; that transition is also recorded in the
audit table with the system auto-expiry actor. Enable it again explicitly if a
new observation window is approved. It only affects future completed heartbeat
runs.

## 3. Observe for at least 48 hours

Production and Testing endpoints become due after their configured check
intervals (defaults: 12 and 24 hours respectively). The scheduler polls hourly
in batches of 10, so those intervals are eligibility targets, not completion
SLOs: queue depth, timeouts, and inconclusive target selection can delay a
completed assessment. Wait 48 hours, then verify actual repeat-run coverage
rather than assuming the cadence delivered it. From a source checkout:

```bash
npm run audit:verification-profile-shadow -- --hours=48
```

Against the compiled production image:

```bash
fly ssh console -a adcp-docs -C \
'node /app/dist/scripts/audit-verification-profile-shadow.js --hours=48'
```

For a restricted endpoint-by-endpoint review, add `--include-agents`. That
output includes endpoint URLs, candidate transition counts, exact badge
identity (`role` plus `adcp_version`), a separately flagged legacy-badge cohort,
per-endpoint blocking reasons, and review reasons. Do not paste it into public
issues, pull requests, or other unrestricted systems.

Candidate transitions compare only the version, proposed Spec/Sandbox statuses,
and recommendation. The repeat gate counts only complete, bundle-backed
observations with no unattributed failures or Sandbox-applicable unresolved
bundles. Evidence-count changes are reported separately for manual review;
they do not by themselves mean that the candidate outcome flapped.
Endpoints with no assessment are labeled `not_assessed` instead of being
misreported as incomplete runs with missing bundle evidence.
The aggregate report also separates unassessed endpoints by absent, recent, or
stale latest public runs. Restricted output identifies whether the latest
public run had no track evidence. These diagnostics explain coverage failure;
they never remove an endpoint from the denominator or turn it into a pass.

Before proceeding, require:

- at least the requested observation window since the first assessment written
by the current policy version (the audit fails closed before that age);
- at least 95% coverage, with both numerator and denominator limited to the
currently eligible monitored endpoints;
- stable repeat observations for at least 95% of eligible endpoints, with no
candidate-outcome transitions; never use an endpoint without two stable
observations to make an individual migration decision;
- no incomplete latest runs, missing bundle evidence, unresolved
Sandbox-applicable bundle gaps, or failures that cannot be attributed to
their originating evidence;
- manual review of valid partial/failing candidate bundles as impact outcomes,
rather than treating those outcomes as corrupt shadow data;
- manual review of every public-passing endpoint that is not candidate-passing
under Spec;
- manual review of every active/degraded badge reported as affected;
- manual review of every ambiguous mixed controller/failure phase;
- an explicit owner decision on legacy Live/multi-mode badges and badges on
unmonitored lifecycle stages.

These are decision gates, not automatic migration rules.

Any change to evaluator semantics must advance
`VERIFICATION_PROFILE_SHADOW_POLICY_VERSION`. Re-enable the audited lease after
deploying that change and start a new 48-hour window; never combine observations
from two policy versions to satisfy coverage or stability gates.

When a prior window failed and a replacement evidence stream is intentionally
started, advance the policy version even if candidate grading is unchanged.
This makes the first write the unambiguous start of the replacement window and
prevents old policy history from satisfying the minimum-age gate.

The shadow evaluator intentionally leaves mixed executed partial bundles
unresolved, even when their visible steps include a controller omission. The
SDK result currently does not preserve every cause that can make such a bundle
partial, so promoting it would risk hiding unrelated missing evidence. Phase 2
must not proceed until those endpoints either produce explicit controller-only
`storyboards_missing_tools` evidence or the SDK exposes complete causal bundle
data and the evaluator is reviewed again.

Also inspect the structured heartbeat health logs over the window. Account for
every scheduled tick with either a health record or an explicit same-job
overlap skip, require zero setting/write errors, a shadow write success rate of
100% for attempted rows, and no more than a 10% increase in p95 public heartbeat
processing duration or p95 eligible queue backlog versus the frozen disabled
baseline. Treat missing ticks, missing fields, or a changed worker/compliance
input as unproven. Disable the lease and investigate if any health threshold is
missed, even when the assessment coverage gate passes.

## 4. Roll back collection at any time

Use the same endpoint with `{ "enabled": false }`; do not rely on waiting for
the 72-hour lease when an immediate stop is needed. The disable is audited and
is rechecked before each shadow write in an in-flight heartbeat batch. Rows are
scheduled for deletion after the fixed 90-day retention window.
Public behavior is unchanged whether collection is on or off.
Use `{ "enabled": false }` to stop future writes and `{ "enabled": true }` to
resume them. The persistent switch has no automatic expiry. Each change is
audited with the authenticated credential identifier, and the setting is
checked atomically with every insert. Existing rows expire under the fixed
90-day retention policy. Public behavior is unchanged either way.

Retention cleanup continues through the scheduled heartbeat maintenance seam
while collection is disabled. Do not mutate rollout state or prune rows through
a direct database session during an audited rollout.

## 5. Phase 2 only after review
## 4. Phase 2 only after working-group review

The enforcement change must remain a separate deployment. Rebase it onto the
then-current main branch, renumber its migration after this phase's current migration,
rerun its migration and concurrency tests, and stage it behind its own
canary/rollback control. Do not infer an endpoint's selected profile solely
from the shadow recommendation; owner choice and communication are required
before any badge revocation or public status change.
The enforcement change must remain a separate deployment. It is blocked until
the working group decides the JWT grading-profile claim, Sandbox launch gate,
and Legacy lifetime. Sandbox also requires reviewed causal-bundle evidence and
a versioned published exception catalog. The selection data model must be keyed
to exact `(agent_url, role, adcp_version)` identity with immutable source-run
provenance, authenticated optimistic concurrency, and append-only audit history.
Never infer an owner's selection from the evaluator recommendation.
19 changes: 12 additions & 7 deletions docs/registry/maintaining-your-agent.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -33,14 +33,19 @@ AAO continuously re-evaluates your agent's conformance on an approximately **1-h

The heartbeat is not a registry crawl. A crawl re-reads your `adagents.json` or capability snapshot and updates registry metadata. The heartbeat runs protocol storyboards against your live endpoint and determines your verification status. The two are independent operations.

### Verified (Spec) vs Verified (Sandbox)
### Grading profiles and verification modes

Both qualifiers run the same storyboards on the same ~1h heartbeat. The difference is where the runner targets:
The agent dashboard can compare three grading profiles from the same completed heartbeat evidence:

| Qualifier | Runner targets | What it attests |
|---|---|---|
| **(Spec)** | Any endpoint you register — test deployment, local dev, sandbox-only stack | AdCP wire format and protocol semantics are correct |
| **(Sandbox)** | Your registered **production** `agent_url`, using natural-key references with `sandbox: true` or preverified sandbox `account_id` references | Your production code path keeps sandbox accounts isolated with zero real-world side effects |
| Grading profile | Meaning |
|---|---|
| **Legacy** | The current production grading and badge behavior. This remains authoritative during the read-only comparison rollout. |
| **Spec** | Strict v3 grading. Required evidence must be complete; missing or ambiguous evidence never counts as a pass. |
| **Sandbox** | Production-lifecycle grading that excludes only catalog-proven compliance-controller gaps. All ambiguous gaps and failures observable on the production endpoint still fail closed. |

This comparison is passive. Opening the dashboard does not contact your agent, enqueue another compliance run, or change its badge. The first rollout phase presents an **agent-wide preview** from one source run, including the concrete compliance bundle, evaluator policy, assessment time, and stale or incomplete evidence. It is not an exact badge-role grade and cannot be selected. Legacy continues to control every public registry and badge result while the role-specific data model and Sandbox exception catalog are reviewed.

Grading profiles are separate from badge `verification_modes`. Modes such as `spec` and `live` describe what kind of verification evidence a badge carries; `live` remains an independent signal. The Legacy grading profile is a compatibility policy, not a verification mode and not a replacement for `live`.

See [AAO Verified](/docs/building/verification/aao-verified) for complete eligibility and attestation details.

Expand All @@ -52,7 +57,7 @@ The agent dashboard at [agenticadvertising.org/dashboard/agents](https://agentic
|---|---|---|
| **Active** | All storyboards are passing on the current heartbeat. Your AAO Verified badge is live and auto-renewing. | No action needed. Monitor for regressions after deploys. |
| **Degraded** | At least one storyboard started failing. The badge **continues to render** while a 48-hour grace period runs. | Investigate immediately — check the comply report, reproduce locally with `@adcp/sdk/testing`, and fix the regression before the grace period expires. |
| **Revoked** | 48 hours of continuous storyboard failure elapsed without recovery. The qualifier has dropped from the badge. (Sandbox), if separately held, is unaffected — the two axes are independent. | Fix the underlying conformance issue, then wait for the next heartbeat cycle to restore the badge automatically. |
| **Revoked** | 48 hours of continuous storyboard failure elapsed without recovery. The verification qualifier has dropped from the badge. | Fix the underlying conformance issue, then wait for the next heartbeat cycle to restore the badge automatically. |
| **Recovery** | Storyboards are passing again after a prior failure. The badge qualifier reissues automatically. | No action needed beyond confirming the fix is stable. |

<Note>
Expand Down
Loading
Loading