diff --git a/.agents/shortcuts/verification-profile-shadow-rollout.md b/.agents/shortcuts/verification-profile-shadow-rollout.md index 89ced5e48c8..683c2ed74e5 100644 --- a/.agents/shortcuts/verification-profile-shadow-rollout.md +++ b/.agents/shortcuts/verification-profile-shadow-rollout.md @@ -6,35 +6,50 @@ heartbeats. It adds a bounded database read/write after public processing, but does not make another agent request or change public grading, badges, or notifications. -## 1. Deploy disabled +## 1. Deploy read-only comparison -Migration `572_verification_profile_shadow_rollout.sql` creates the bounded -shadow ledger and inserts the audited setting as -`{ "enabled": false, "expires_at": null }`. -After deploy, confirm the admin settings response reports it disabled. +Migration `591_enable_verification_profile_comparisons.sql` turns the existing +audited switch on persistently as `{ "enabled": true, "expires_at": null }`. +It records the old and new values in `system_settings_audit`. The worker reuses +completed authoritative heartbeat evidence; it sends no additional request to +an agent and changes no public status, badge, token, or notification. -Before enabling, collect a full 48-hour disabled baseline from the deployed -worker's structured `Compliance heartbeat shadow flush completed after public -processing` records. Every scheduled heartbeat, including an empty queue, must -emit this record with: +The owner dashboard initially presents the result as an **agent-wide preview**. +It must not be described as an exact `(agent, role, AdCP release)` result. Role +selection remains disabled until role-scoped immutable assessments exist. A +stale, absent, malformed, or unreadable comparison is shown as unavailable and +never as passing. -- `publicProcessingDurationMs`; -- `queue.eligibleBacklog`, `queue.selectedCount`, and `queue.batchLimit`; -- policy, SDK, selected target, and compliance-cache versions in `inputs`; -- aggregate `outcomes` and `skipReasons` (never endpoint URLs); and -- shadow candidates, attempts, writes, disables, setting/write errors, and - write latency. +Do not deploy profile selection or enforcement during this phase. -Do not infer a zero backlog from missing logs. Missing baseline samples make -the health comparison unproven. Record the deployed image SHA, policy version, -SDK version, and compliance-cache/catalog version at baseline start. Freeze -those worker/compliance inputs through the baseline and observation window. A -change to any of them, or a machine-convergence gap, invalidates the comparison -and requires a fresh disabled baseline. +## 2. Observe ordinary production health -Do not deploy the enforcement patch during this phase. +Every scheduled heartbeat, including an empty queue, emits the structured +`Compliance heartbeat shadow flush completed after public processing` record +with aggregate queue, outcome, skip-reason, policy/SDK/target, write-count, and +latency fields. Endpoint URLs are excluded. -## 2. Enable collection through the authenticated admin API +Monitor normal deployments and traffic. A deployment or cache-version change +does not restart a frozen evidence window; provenance on each immutable row +identifies the source run and evaluator version. Check: + +- comparison write errors and disabled writes; +- p95 public processing duration and eligible queue backlog; +- the distribution of current, stale, pending, and unavailable owner previews; +- Spec partial/failing causes and all unresolved Sandbox bundles; and +- whether evidence explanations match the source run. + +The audit command remains available for aggregate and restricted diagnosis: + +```bash +npm run audit:verification-profile-shadow -- --hours=48 +``` + +Add `--include-agents` only for a restricted operator review. That output can +contain endpoint URLs and must not be pasted into public issues or pull +requests. + +## 3. Disable or re-enable collection From a signed-in global-admin browser session, use the same-origin console: @@ -46,118 +61,22 @@ await fetch('/api/admin/settings/verification-profile-shadow-rollout', { }).then((response) => response.json()) ``` -This write is recorded in `system_settings_audit` with the authenticated -credential identifier and returns an `expires_at` value 72 hours in the future. -The lease disables itself at expiry; that transition is also recorded in the -audit table with the system auto-expiry actor. Enable it again explicitly if a -new observation window is approved. It only affects future completed heartbeat -runs. - -## 3. Observe for at least 48 hours - -Production and Testing endpoints become due after their configured check -intervals (defaults: 12 and 24 hours respectively). The scheduler polls hourly -in batches of 10, so those intervals are eligibility targets, not completion -SLOs: queue depth, timeouts, and inconclusive target selection can delay a -completed assessment. Wait 48 hours, then verify actual repeat-run coverage -rather than assuming the cadence delivered it. From a source checkout: - -```bash -npm run audit:verification-profile-shadow -- --hours=48 -``` - -Against the compiled production image: - -```bash -fly ssh console -a adcp-docs -C \ - 'node /app/dist/scripts/audit-verification-profile-shadow.js --hours=48' -``` - -For a restricted endpoint-by-endpoint review, add `--include-agents`. That -output includes endpoint URLs, candidate transition counts, exact badge -identity (`role` plus `adcp_version`), a separately flagged legacy-badge cohort, -per-endpoint blocking reasons, and review reasons. Do not paste it into public -issues, pull requests, or other unrestricted systems. - -Candidate transitions compare only the version, proposed Spec/Sandbox statuses, -and recommendation. The repeat gate counts only complete, bundle-backed -observations with no unattributed failures or Sandbox-applicable unresolved -bundles. Evidence-count changes are reported separately for manual review; -they do not by themselves mean that the candidate outcome flapped. -Endpoints with no assessment are labeled `not_assessed` instead of being -misreported as incomplete runs with missing bundle evidence. -The aggregate report also separates unassessed endpoints by absent, recent, or -stale latest public runs. Restricted output identifies whether the latest -public run had no track evidence. These diagnostics explain coverage failure; -they never remove an endpoint from the denominator or turn it into a pass. - -Before proceeding, require: - -- at least the requested observation window since the first assessment written - by the current policy version (the audit fails closed before that age); -- at least 95% coverage, with both numerator and denominator limited to the - currently eligible monitored endpoints; -- stable repeat observations for at least 95% of eligible endpoints, with no - candidate-outcome transitions; never use an endpoint without two stable - observations to make an individual migration decision; -- no incomplete latest runs, missing bundle evidence, unresolved - Sandbox-applicable bundle gaps, or failures that cannot be attributed to - their originating evidence; -- manual review of valid partial/failing candidate bundles as impact outcomes, - rather than treating those outcomes as corrupt shadow data; -- manual review of every public-passing endpoint that is not candidate-passing - under Spec; -- manual review of every active/degraded badge reported as affected; -- manual review of every ambiguous mixed controller/failure phase; -- an explicit owner decision on legacy Live/multi-mode badges and badges on - unmonitored lifecycle stages. - -These are decision gates, not automatic migration rules. - -Any change to evaluator semantics must advance -`VERIFICATION_PROFILE_SHADOW_POLICY_VERSION`. Re-enable the audited lease after -deploying that change and start a new 48-hour window; never combine observations -from two policy versions to satisfy coverage or stability gates. - -When a prior window failed and a replacement evidence stream is intentionally -started, advance the policy version even if candidate grading is unchanged. -This makes the first write the unambiguous start of the replacement window and -prevents old policy history from satisfying the minimum-age gate. - -The shadow evaluator intentionally leaves mixed executed partial bundles -unresolved, even when their visible steps include a controller omission. The -SDK result currently does not preserve every cause that can make such a bundle -partial, so promoting it would risk hiding unrelated missing evidence. Phase 2 -must not proceed until those endpoints either produce explicit controller-only -`storyboards_missing_tools` evidence or the SDK exposes complete causal bundle -data and the evaluator is reviewed again. - -Also inspect the structured heartbeat health logs over the window. Account for -every scheduled tick with either a health record or an explicit same-job -overlap skip, require zero setting/write errors, a shadow write success rate of -100% for attempted rows, and no more than a 10% increase in p95 public heartbeat -processing duration or p95 eligible queue backlog versus the frozen disabled -baseline. Treat missing ticks, missing fields, or a changed worker/compliance -input as unproven. Disable the lease and investigate if any health threshold is -missed, even when the assessment coverage gate passes. - -## 4. Roll back collection at any time - -Use the same endpoint with `{ "enabled": false }`; do not rely on waiting for -the 72-hour lease when an immediate stop is needed. The disable is audited and -is rechecked before each shadow write in an in-flight heartbeat batch. Rows are -scheduled for deletion after the fixed 90-day retention window. -Public behavior is unchanged whether collection is on or off. +Use `{ "enabled": false }` to stop future writes and `{ "enabled": true }` to +resume them. The persistent switch has no automatic expiry. Each change is +audited with the authenticated credential identifier, and the setting is +checked atomically with every insert. Existing rows expire under the fixed +90-day retention policy. Public behavior is unchanged either way. Retention cleanup continues through the scheduled heartbeat maintenance seam while collection is disabled. Do not mutate rollout state or prune rows through a direct database session during an audited rollout. -## 5. Phase 2 only after review +## 4. Phase 2 only after working-group review -The enforcement change must remain a separate deployment. Rebase it onto the -then-current main branch, renumber its migration after this phase's current migration, -rerun its migration and concurrency tests, and stage it behind its own -canary/rollback control. Do not infer an endpoint's selected profile solely -from the shadow recommendation; owner choice and communication are required -before any badge revocation or public status change. +The enforcement change must remain a separate deployment. It is blocked until +the working group decides the JWT grading-profile claim, Sandbox launch gate, +and Legacy lifetime. Sandbox also requires reviewed causal-bundle evidence and +a versioned published exception catalog. The selection data model must be keyed +to exact `(agent_url, role, adcp_version)` identity with immutable source-run +provenance, authenticated optimistic concurrency, and append-only audit history. +Never infer an owner's selection from the evaluator recommendation. diff --git a/docs/registry/maintaining-your-agent.mdx b/docs/registry/maintaining-your-agent.mdx index f530dfd56bb..b024785bd6e 100644 --- a/docs/registry/maintaining-your-agent.mdx +++ b/docs/registry/maintaining-your-agent.mdx @@ -33,14 +33,19 @@ AAO continuously re-evaluates your agent's conformance on an approximately **1-h The heartbeat is not a registry crawl. A crawl re-reads your `adagents.json` or capability snapshot and updates registry metadata. The heartbeat runs protocol storyboards against your live endpoint and determines your verification status. The two are independent operations. -### Verified (Spec) vs Verified (Sandbox) +### Grading profiles and verification modes -Both qualifiers run the same storyboards on the same ~1h heartbeat. The difference is where the runner targets: +The agent dashboard can compare three grading profiles from the same completed heartbeat evidence: -| Qualifier | Runner targets | What it attests | -|---|---|---| -| **(Spec)** | Any endpoint you register — test deployment, local dev, sandbox-only stack | AdCP wire format and protocol semantics are correct | -| **(Sandbox)** | Your registered **production** `agent_url`, using natural-key references with `sandbox: true` or preverified sandbox `account_id` references | Your production code path keeps sandbox accounts isolated with zero real-world side effects | +| Grading profile | Meaning | +|---|---| +| **Legacy** | The current production grading and badge behavior. This remains authoritative during the read-only comparison rollout. | +| **Spec** | Strict v3 grading. Required evidence must be complete; missing or ambiguous evidence never counts as a pass. | +| **Sandbox** | Production-lifecycle grading that excludes only catalog-proven compliance-controller gaps. All ambiguous gaps and failures observable on the production endpoint still fail closed. | + +This comparison is passive. Opening the dashboard does not contact your agent, enqueue another compliance run, or change its badge. The first rollout phase presents an **agent-wide preview** from one source run, including the concrete compliance bundle, evaluator policy, assessment time, and stale or incomplete evidence. It is not an exact badge-role grade and cannot be selected. Legacy continues to control every public registry and badge result while the role-specific data model and Sandbox exception catalog are reviewed. + +Grading profiles are separate from badge `verification_modes`. Modes such as `spec` and `live` describe what kind of verification evidence a badge carries; `live` remains an independent signal. The Legacy grading profile is a compatibility policy, not a verification mode and not a replacement for `live`. See [AAO Verified](/docs/building/verification/aao-verified) for complete eligibility and attestation details. @@ -52,7 +57,7 @@ The agent dashboard at [agenticadvertising.org/dashboard/agents](https://agentic |---|---|---| | **Active** | All storyboards are passing on the current heartbeat. Your AAO Verified badge is live and auto-renewing. | No action needed. Monitor for regressions after deploys. | | **Degraded** | At least one storyboard started failing. The badge **continues to render** while a 48-hour grace period runs. | Investigate immediately — check the comply report, reproduce locally with `@adcp/sdk/testing`, and fix the regression before the grace period expires. | -| **Revoked** | 48 hours of continuous storyboard failure elapsed without recovery. The qualifier has dropped from the badge. (Sandbox), if separately held, is unaffected — the two axes are independent. | Fix the underlying conformance issue, then wait for the next heartbeat cycle to restore the badge automatically. | +| **Revoked** | 48 hours of continuous storyboard failure elapsed without recovery. The verification qualifier has dropped from the badge. | Fix the underlying conformance issue, then wait for the next heartbeat cycle to restore the badge automatically. | | **Recovery** | Storyboards are passing again after a prior failure. The badge qualifier reissues automatically. | No action needed beyond confirming the fix is stable. | diff --git a/server/public/dashboard-agents.html b/server/public/dashboard-agents.html index 1cc4b3ba598..d24ae050f83 100644 --- a/server/public/dashboard-agents.html +++ b/server/public/dashboard-agents.html @@ -272,6 +272,48 @@ text-decoration: underline; } .verification-embed-toggle:hover { color: var(--color-primary-700); } + .grading-profile-comparison { + margin-top: var(--space-3); + padding-top: var(--space-3); + border-top: var(--border-1) solid var(--color-border); + } + .grading-profile-comparison-header { + display: flex; + justify-content: space-between; + gap: var(--space-2); + flex-wrap: wrap; + margin-bottom: var(--space-2); + font-size: var(--text-xs); + color: var(--color-text-secondary); + } + .grading-profile-comparison-header strong { color: var(--color-text-primary); } + .grading-profile-grid { + display: grid; + grid-template-columns: repeat(3, minmax(0, 1fr)); + gap: var(--space-2); + } + .grading-profile-card { + padding: var(--space-2); + border: var(--border-1) solid var(--color-border); + border-radius: var(--radius-md); + background: var(--color-bg-card); + font-size: var(--text-xs); + } + .grading-profile-card--selected { border-color: var(--color-primary-600); } + .grading-profile-name { font-weight: var(--font-semibold); color: var(--color-text-primary); } + .grading-profile-status { display: block; margin: var(--space-1) 0; font-weight: var(--font-semibold); } + .grading-profile-status--passing { color: var(--color-success-fg); } + .grading-profile-status--failing { color: var(--color-error-600); } + .grading-profile-status--partial, + .grading-profile-status--unavailable { color: var(--color-text-muted); } + .grading-profile-note { margin-top: var(--space-2); color: var(--color-text-muted); font-size: var(--text-xs); } + .grading-profile-provenance { margin-top: var(--space-2); font-size: var(--text-xs); color: var(--color-text-secondary); } + .grading-profile-provenance dl { display: grid; grid-template-columns: max-content 1fr; gap: var(--space-1) var(--space-2); } + .grading-profile-provenance dt { font-weight: var(--font-semibold); } + .grading-profile-provenance dd { margin: 0; overflow-wrap: anywhere; } + @media (max-width: 700px) { + .grading-profile-grid { grid-template-columns: 1fr; } + } .verification-embed-drawer { display: none; flex-direction: column; @@ -2224,6 +2266,81 @@

Agents

`; } + function renderGradingProfileComparisons(cs) { + const comparisons = Array.isArray(cs?.grading_profile_comparisons) + ? cs.grading_profile_comparisons + : []; + if (comparisons.length === 0) return ''; + + const profileNames = ['legacy', 'spec', 'sandbox']; + return comparisons.map(comparison => { + const version = typeof comparison.compliance_bundle_version === 'string' + ? comparison.compliance_bundle_version + : 'version unavailable'; + if (!comparison.profiles || comparison.availability === 'pending' || comparison.availability === 'temporarily_unavailable') { + return `
+
Agent-wide grading preview
+
${escapeHtml(comparison.unavailable_reason || 'Comparison evidence is unavailable. Legacy remains authoritative.')}
+
`; + } + const assessedAt = comparison.assessed_at ? new Date(comparison.assessed_at) : null; + const freshness = assessedAt && !Number.isNaN(assessedAt.getTime()) + ? `${comparison.stale ? 'Stale · ' : ''}${timeAgo(assessedAt)}` + : 'Assessment time unavailable'; + const selected = comparison.selected_profile || 'legacy'; + const profiles = comparison.profiles && typeof comparison.profiles === 'object' + ? comparison.profiles + : {}; + const cards = profileNames.map(name => { + const profile = profiles[name] || {}; + // Freshness applies to the complete comparison, not an individual + // profile. Fail closed if an older or inconsistent API response + // happens to retain a passing per-profile status on a stale row. + const available = comparison.availability !== 'stale' + && profile.available !== false + && typeof profile.status === 'string'; + const status = available ? profile.status : 'unavailable'; + const statusLabel = status === 'passing' ? 'Passed' + : status === 'failing' ? 'Failed' + : status === 'partial' ? 'Incomplete' + : 'Unavailable'; + const selectedClass = selected === name ? ' grading-profile-card--selected' : ''; + return `
+ ${escapeHtml(name.charAt(0).toUpperCase() + name.slice(1))}${selected === name ? ' · Current' : ''} + ${escapeHtml(statusLabel)} + ${escapeHtml(profile.explanation || 'No explanation available.')} + ${!available && profile.observed_status ? `Historical observation: ${escapeHtml(profile.observed_status)}` : ''} +
`; + }).join(''); + const evidence = comparison.evidence && typeof comparison.evidence === 'object' ? comparison.evidence : {}; + const sourceRun = comparison.source_run_id || 'Unavailable'; + const policy = comparison.evaluator_policy_version || 'Unavailable'; + const target = comparison.requested_compliance_target || 'Unavailable'; + const sourceTestedAt = comparison.source_tested_at ? new Date(comparison.source_tested_at) : null; + const testedLabel = sourceTestedAt && !Number.isNaN(sourceTestedAt.getTime()) + ? sourceTestedAt.toLocaleString() + : 'Unavailable'; + return `
+
+ Agent-wide grading preview · ${escapeHtml(version)} + ${escapeHtml(freshness)} +
+
${cards}
+
+ Evidence and provenance +
+
Source run
${escapeHtml(sourceRun)}
+
Evaluator policy
${escapeHtml(policy)}
+
Requested target
${escapeHtml(target)}
+
Source tested
${escapeHtml(testedLabel)}
+
Evidence
${escapeHtml(String(evidence.selected_storyboard_count ?? 0))} storyboards; ${escapeHtml(String(evidence.observed_failure_count ?? 0))} observed failures; ${escapeHtml(String(evidence.flat_failure_count ?? 0))} run-level failure records; ${escapeHtml(String(evidence.controller_gap_phase_count ?? 0))} controller-gap phases; ${escapeHtml(String(evidence.sandbox_unresolved_bundle_count ?? 0))} unresolved Sandbox bundles
+
+
+
Agent-wide preview only—not an exact badge-role grade. Legacy still controls the public registry and badge. Comparing profiles does not contact the agent.
+
`; + }).join(''); + } + function renderVerificationPanel(cs, agentUrl, hasAuth) { const badges = Array.isArray(cs && cs.verified_badges) ? cs.verified_badges : []; // Audit-comment from #3603: values MUST NOT end in "Agent" or @@ -2361,6 +2478,7 @@

Agents

${declaredHtml}
${hint}
+ ${typeof renderGradingProfileComparisons === 'function' ? renderGradingProfileComparisons(cs) : ''} ${buildNoticesSectionHtml(cs)} `; } @@ -2492,6 +2610,7 @@

Agents

AAO Verified ${badgeCountSummary} + ${typeof renderGradingProfileComparisons === 'function' ? renderGradingProfileComparisons(cs) : ''} ${rowsHtml} ${buildNoticesSectionHtml(cs)} `; diff --git a/server/src/addie/jobs/compliance-heartbeat.ts b/server/src/addie/jobs/compliance-heartbeat.ts index 47fc64ca6ac..fdcb8d797d6 100644 --- a/server/src/addie/jobs/compliance-heartbeat.ts +++ b/server/src/addie/jobs/compliance-heartbeat.ts @@ -23,7 +23,7 @@ import { type ComplianceResult, } from '../services/compliance-testing.js'; import { ComplianceDatabase, type LifecycleStage } from '../../db/compliance-db.js'; -import { query, withDatabaseDeadline } from '../../db/client.js'; +import { query } from '../../db/client.js'; import { ComplianceRefreshRequestsDatabase } from '../../db/compliance-refresh-requests-db.js'; import { notifyComplianceChange, notifyVerificationChange } from '../../notifications/compliance.js'; import { notifySystemError } from '../error-notifier.js'; @@ -32,7 +32,6 @@ import { logOutboundRequest } from '../../db/outbound-log-db.js'; import { AAO_UA_COMPLIANCE } from '../../config/user-agents.js'; import { revokeUnsupportedPublicBadges, runBadgeFanOut } from '../../services/badge-issuance.js'; import { adaptAuthForSdk } from '../../services/sdk-auth-adapter.js'; -import { getVerificationProfileShadowRollout } from '../../db/system-settings-db.js'; import { pruneVerificationProfileShadowAssessments, recordVerificationProfileShadowAssessment, @@ -496,9 +495,8 @@ export async function runComplianceHeartbeatJob(options: HeartbeatOptions = {}): } } - // Shadow persistence is deliberately outside the public heartbeat loop. - // Re-check the audited switch with a short deadline before every write so a - // disable or automatic expiry takes effect within an already-running batch. + // Comparison persistence is deliberately outside the public heartbeat loop. + // It reuses the completed run and has no agent traffic or badge side effects. const publicProcessingDurationMs = Date.now() - batchStartedAt; const shadowFlushStartedAt = Date.now(); const shadowStats = { @@ -507,34 +505,10 @@ export async function runComplianceHeartbeatJob(options: HeartbeatOptions = {}): recorded: 0, disabled: 0, errors: 0, - setting_errors: 0, total_write_latency_ms: 0, max_write_latency_ms: 0, }; for (const pending of pendingShadowAssessments) { - let enabled = false; - try { - enabled = (await withDatabaseDeadline( - Date.now() + 500, - () => getVerificationProfileShadowRollout(), - // Expiry is an audited compare-and-set write when the 72-hour window - // elapses, so this bounded operation cannot use a read-only transaction. - { readOnly: false }, - )).enabled; - } catch (settingError) { - shadowStats.setting_errors++; - shadowStats.errors++; - logger.error( - { settingError, agentUrl: pending.agentUrl }, - 'Verification profile shadow setting could not be read; collection remains disabled', - ); - continue; - } - if (!enabled) { - shadowStats.disabled++; - continue; - } - shadowStats.attempted++; const writeStartedAt = Date.now(); try { @@ -569,8 +543,7 @@ export async function runComplianceHeartbeatJob(options: HeartbeatOptions = {}): } } // Emit one aggregate health record for every scheduled heartbeat, including - // empty queues and disabled collection. That makes a frozen pre-rollout - // baseline and the collection window comparable without exposing endpoints. + // empty queues, without exposing endpoint URLs. logger.info( { publicProcessingDurationMs, diff --git a/server/src/db/migrations/591_enable_verification_profile_comparisons.sql b/server/src/db/migrations/591_enable_verification_profile_comparisons.sql new file mode 100644 index 00000000000..2b4453aebd6 --- /dev/null +++ b/server/src/db/migrations/591_enable_verification_profile_comparisons.sql @@ -0,0 +1,58 @@ +-- Phase 1 of owner-selectable grading profiles: keep comparison collection on +-- until an administrator deliberately disables it. This is observation-only; +-- it reuses completed heartbeat evidence and cannot change public trust state. + +-- Snapshot the source fields needed to explain a comparison. Existing shadow +-- rows are backfilled from their source run before collection becomes durable; +-- new rows populate these columns in the same INSERT as the assessment. +ALTER TABLE verification_profile_shadow_assessments + ADD COLUMN IF NOT EXISTS source_tested_at TIMESTAMPTZ, + ADD COLUMN IF NOT EXISTS requested_compliance_target TEXT, + ADD COLUMN IF NOT EXISTS flat_failure_count INTEGER NOT NULL DEFAULT 0 + CHECK (flat_failure_count >= 0); + +-- Older v3 rows only distinguished malformed flat failures. Preserve that +-- known lower bound; new writes store the complete flat-failure count. +UPDATE verification_profile_shadow_assessments +SET flat_failure_count = unattributed_flat_failure_count +WHERE flat_failure_count < unattributed_flat_failure_count; + +UPDATE verification_profile_shadow_assessments assessment +SET source_tested_at = source.tested_at, + requested_compliance_target = source.requested_compliance_target +FROM agent_compliance_runs source +WHERE source.id = assessment.source_run_id + AND assessment.source_tested_at IS NULL; + +ALTER TABLE verification_profile_shadow_assessments + ALTER COLUMN source_tested_at SET NOT NULL; + +COMMENT ON COLUMN verification_profile_shadow_assessments.source_tested_at IS + 'Immutable snapshot of the source compliance run tested_at timestamp'; +COMMENT ON COLUMN verification_profile_shadow_assessments.requested_compliance_target IS + 'Immutable snapshot of the compliance target used by the source run'; +COMMENT ON COLUMN verification_profile_shadow_assessments.flat_failure_count IS + 'Total SDK run-level failures, including entries with storyboard and step identity'; + +WITH previous AS MATERIALIZED ( + SELECT value + FROM system_settings + WHERE key = 'verification_profile_shadow_rollout' + FOR UPDATE +), updated AS ( + UPDATE system_settings + SET value = '{"enabled": true, "expires_at": null}'::jsonb, + description = 'Collect read-only Legacy, Spec, and Sandbox comparisons from completed compliance heartbeats', + updated_at = NOW(), + updated_by = 'migration:591_enable_verification_profile_comparisons' + WHERE key = 'verification_profile_shadow_rollout' + RETURNING value +) +INSERT INTO system_settings_audit (key, old_value, new_value, changed_by, changed_at) +SELECT + 'verification_profile_shadow_rollout', + previous.value, + updated.value, + 'migration:591_enable_verification_profile_comparisons', + NOW() +FROM previous CROSS JOIN updated; diff --git a/server/src/db/system-settings-db.ts b/server/src/db/system-settings-db.ts index 2f79f3b6867..8b5aead7b2b 100644 --- a/server/src/db/system-settings-db.ts +++ b/server/src/db/system-settings-db.ts @@ -81,9 +81,6 @@ export interface VerificationProfileShadowRolloutSetting { expires_at: string | null; } -const VERIFICATION_PROFILE_SHADOW_ROLLOUT_TTL_HOURS = 72; -const VERIFICATION_PROFILE_SHADOW_AUTO_EXPIRY_ACTOR = 'system:verification-profile-shadow-auto-expiry'; - // ============== Setting Keys ============== export const SETTING_KEYS = { @@ -245,49 +242,11 @@ export async function setOrganizationAuthorizationEnforcement( } /** - * Read the observation-only verification-profile collection switch. Missing or - * malformed values are disabled so a rollback cannot accidentally collect. + * Read the observation-only verification-profile comparison switch. Missing + * or malformed values are disabled so a rollback cannot accidentally collect. */ export async function getVerificationProfileShadowRollout(): Promise { - const result = await query<{ value: unknown }>( - `WITH current_setting AS MATERIALIZED ( - SELECT value AS old_value - FROM system_settings - WHERE key = $1 - FOR UPDATE - ), - expired AS ( - UPDATE system_settings setting - SET value = '{"enabled": false, "expires_at": null}'::jsonb, - updated_at = NOW(), - updated_by = $2 - FROM current_setting current - WHERE setting.key = $1 - AND jsonb_typeof(current.old_value) = 'object' - AND current.old_value->>'enabled' = 'true' - AND CASE - WHEN COALESCE(current.old_value->>'expires_at', '') ~ - '^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}(?:[.][0-9]{3})?Z$' - THEN (current.old_value->>'expires_at')::timestamptz <= NOW() - ELSE FALSE - END - RETURNING current.old_value, setting.value AS new_value - ), - audit AS ( - INSERT INTO system_settings_audit ( - key, old_value, new_value, changed_by, changed_at - ) - SELECT $1, old_value, new_value, $2, NOW() - FROM expired - ) - SELECT new_value AS value FROM expired - UNION ALL - SELECT old_value AS value - FROM current_setting - WHERE NOT EXISTS (SELECT 1 FROM expired)`, - [SETTING_KEYS.VERIFICATION_PROFILE_SHADOW_ROLLOUT, VERIFICATION_PROFILE_SHADOW_AUTO_EXPIRY_ACTOR], - ); - const setting = result.rows[0]?.value ?? null; + const setting = await getSetting(SETTING_KEYS.VERIFICATION_PROFILE_SHADOW_ROLLOUT); if ( setting === null || typeof setting !== 'object' || @@ -303,24 +262,17 @@ export async function getVerificationProfileShadowRollout(): Promise, updatedBy?: string, ): Promise { - const expiresAt = setting.enabled - ? new Date(Date.now() + VERIFICATION_PROFILE_SHADOW_ROLLOUT_TTL_HOURS * 60 * 60 * 1_000).toISOString() - : null; - const persisted = { enabled: setting.enabled, expires_at: expiresAt }; + const persisted = { enabled: setting.enabled, expires_at: null }; await setSetting( SETTING_KEYS.VERIFICATION_PROFILE_SHADOW_ROLLOUT, persisted, diff --git a/server/src/db/verification-profile-shadow-db.ts b/server/src/db/verification-profile-shadow-db.ts index 276c524199b..35f0632f01d 100644 --- a/server/src/db/verification-profile-shadow-db.ts +++ b/server/src/db/verification-profile-shadow-db.ts @@ -3,6 +3,18 @@ import type { LifecycleStage } from './compliance-db.js'; import type { VerificationProfileShadowAssessment } from '../services/verification-profile-shadow.js'; import { SETTING_KEYS } from './system-settings-db.js'; +export interface StoredVerificationProfileAssessment extends VerificationProfileShadowAssessment { + id: string; + source_run_id: string; + agent_url: string; + lifecycle_stage: LifecycleStage; + adcp_version: string; + evaluated_at: Date; + source_tested_at: Date; + requested_compliance_target: string | null; + source_provenance: Record | null; +} + export async function recordVerificationProfileShadowAssessment(input: { sourceRunId: string; agentUrl: string; @@ -12,15 +24,22 @@ export async function recordVerificationProfileShadowAssessment(input: { }): Promise { const { assessment } = input; const result = await withDatabaseDeadline(Date.now() + 2_000, () => query( - `WITH rollout AS MATERIALIZED ( + `WITH collection_enabled AS MATERIALIZED ( SELECT 1 FROM system_settings - WHERE key = $33 - AND value->>'enabled' = 'true' - AND COALESCE(value->>'expires_at', '') ~ - '^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}(?:[.][0-9]{3})?Z$' - AND (value->>'expires_at')::timestamptz > NOW() + WHERE key = $34 + AND value = '{"enabled": true, "expires_at": null}'::jsonb FOR SHARE + ), source_run AS MATERIALIZED ( + SELECT tested_at, requested_compliance_target + FROM agent_compliance_runs + WHERE id = $1 + AND agent_url = $2 + AND lifecycle_stage = $3 + AND adcp_version IS NOT DISTINCT FROM $4 + AND dry_run = FALSE + AND is_authoritative = TRUE + AND completeness = 'complete' ) INSERT INTO verification_profile_shadow_assessments ( source_run_id, agent_url, lifecycle_stage, adcp_version, policy_version, @@ -35,11 +54,12 @@ export async function recordVerificationProfileShadowAssessment(input: { sandbox_observable_failure_count, non_controller_gap_step_count, controller_missing_storyboard_count, other_missing_storyboard_count, mixed_controller_failure_phase_count, - unattributed_flat_failure_count, unexplained_phase_failure_count, + flat_failure_count, unattributed_flat_failure_count, + unexplained_phase_failure_count, sandbox_unresolved_executed_bundle_count, sandbox_unresolved_missing_tools_bundle_count, sandbox_unresolved_unknown_bundle_count, - evaluated_at + evaluated_at, source_tested_at, requested_compliance_target ) SELECT $1, $2, $3, $4, $5, $6, $7, $8, @@ -52,42 +72,11 @@ export async function recordVerificationProfileShadowAssessment(input: { $22, $23, $24, $25, $26, $27, - $28, $29, - $30, $31, $32, - NOW() - FROM rollout - ON CONFLICT (source_run_id) DO UPDATE SET - lifecycle_stage = EXCLUDED.lifecycle_stage, - adcp_version = EXCLUDED.adcp_version, - policy_version = EXCLUDED.policy_version, - current_public_status = EXCLUDED.current_public_status, - proposed_spec_status = EXCLUDED.proposed_spec_status, - proposed_sandbox_status = EXCLUDED.proposed_sandbox_status, - sandbox_eligible = EXCLUDED.sandbox_eligible, - recommended_profile = EXCLUDED.recommended_profile, - run_complete = EXCLUDED.run_complete, - bundle_evidence_present = EXCLUDED.bundle_evidence_present, - failing_bundle_count = EXCLUDED.failing_bundle_count, - incomplete_bundle_count = EXCLUDED.incomplete_bundle_count, - sandbox_unresolved_bundle_count = EXCLUDED.sandbox_unresolved_bundle_count, - unattributed_failure_count = EXCLUDED.unattributed_failure_count, - selected_storyboard_count = EXCLUDED.selected_storyboard_count, - applicable_phase_count = EXCLUDED.applicable_phase_count, - controller_gap_phase_count = EXCLUDED.controller_gap_phase_count, - controller_gap_step_count = EXCLUDED.controller_gap_step_count, - controller_cascade_step_count = EXCLUDED.controller_cascade_step_count, - observed_failure_count = EXCLUDED.observed_failure_count, - sandbox_observable_failure_count = EXCLUDED.sandbox_observable_failure_count, - non_controller_gap_step_count = EXCLUDED.non_controller_gap_step_count, - controller_missing_storyboard_count = EXCLUDED.controller_missing_storyboard_count, - other_missing_storyboard_count = EXCLUDED.other_missing_storyboard_count, - mixed_controller_failure_phase_count = EXCLUDED.mixed_controller_failure_phase_count, - unattributed_flat_failure_count = EXCLUDED.unattributed_flat_failure_count, - unexplained_phase_failure_count = EXCLUDED.unexplained_phase_failure_count, - sandbox_unresolved_executed_bundle_count = EXCLUDED.sandbox_unresolved_executed_bundle_count, - sandbox_unresolved_missing_tools_bundle_count = EXCLUDED.sandbox_unresolved_missing_tools_bundle_count, - sandbox_unresolved_unknown_bundle_count = EXCLUDED.sandbox_unresolved_unknown_bundle_count, - evaluated_at = NOW() + $28, $29, $30, + $31, $32, $33, + NOW(), source_run.tested_at, source_run.requested_compliance_target + FROM collection_enabled CROSS JOIN source_run + ON CONFLICT (source_run_id) DO NOTHING RETURNING source_run_id`, [ input.sourceRunId, @@ -117,6 +106,7 @@ export async function recordVerificationProfileShadowAssessment(input: { assessment.controller_missing_storyboard_count, assessment.other_missing_storyboard_count, assessment.mixed_controller_failure_phase_count, + assessment.flat_failure_count, assessment.unattributed_flat_failure_count, assessment.unexplained_phase_failure_count, assessment.sandbox_unresolved_executed_bundle_count, @@ -129,8 +119,37 @@ export async function recordVerificationProfileShadowAssessment(input: { } /** - * Delete rows beyond the fixed 90-day retention window. This remains callable - * while collection is disabled, so expiry does not depend on new heartbeats. + * Return the newest agent-wide comparison. Assessments remain read-only + * evidence: callers may display them, but they do not drive badges. + */ +export async function getLatestVerificationProfileAssessment( + agentUrl: string, + policyVersion: string, +): Promise { + const result = await withDatabaseDeadline(Date.now() + 2_000, () => query( + `SELECT s.*, r.provenance_json AS source_provenance + FROM verification_profile_shadow_assessments s + JOIN agent_compliance_runs r + ON r.id = s.source_run_id + AND r.agent_url = s.agent_url + AND r.lifecycle_stage = s.lifecycle_stage + AND r.adcp_version IS NOT DISTINCT FROM s.adcp_version + WHERE s.agent_url = $1 + AND s.policy_version = $2 + AND s.adcp_version IS NOT NULL + AND s.source_tested_at IS NOT NULL + AND r.dry_run = FALSE + AND r.is_authoritative = TRUE + AND r.completeness = 'complete' + ORDER BY s.evaluated_at DESC + LIMIT 1`, + [agentUrl, policyVersion], + ), { readOnly: true }); + return result.rows[0] ?? null; +} + +/** + * Delete rows beyond the fixed 90-day retention window. */ export async function pruneVerificationProfileShadowAssessments(): Promise { const result = await withDatabaseDeadline(Date.now() + 2_000, () => query<{ pruned_count: string | number }>( diff --git a/server/src/routes/registry-api.ts b/server/src/routes/registry-api.ts index 06d2b7396ae..cb5aff4f415 100644 --- a/server/src/routes/registry-api.ts +++ b/server/src/routes/registry-api.ts @@ -55,6 +55,11 @@ import { selectedComplianceTargetMatchesObservedProfile, UNRESOLVED_COMPLIANCE_TARGET_MESSAGE, } from "../addie/services/compliance-testing.js"; +import { getLatestVerificationProfileAssessment } from "../db/verification-profile-shadow-db.js"; +import { + sandboxProfileAssessmentReasons, + VERIFICATION_PROFILE_SHADOW_POLICY_VERSION, +} from "../services/verification-profile-shadow.js"; import { getPublicJwks } from "../services/verification-token.js"; import { renderBadgeSvg, VALID_BADGE_ROLES } from "../services/badge-svg.js"; import { revokeUnsupportedPublicBadges, runBadgeFanOut } from "../services/badge-issuance.js"; @@ -3560,7 +3565,7 @@ registry.registerPath({ operationId: "getAgentCompliance", summary: "Get agent compliance detail", description: - "Returns detailed compliance status for a single agent, including track-level results, storyboard counts, and timestamps.\n\nIf the agent has opted out of compliance monitoring, returns a minimal response with `status: opted_out`.", + "Returns detailed compliance status for a single agent, including track-level results, storyboard counts, timestamps, and an owner-only read-only agent-wide grading-profile comparison. The comparison is not an exact badge-role grade. Reading it never contacts the agent or changes public badge state.\n\nIf the agent has opted out of compliance monitoring, returns a minimal response with `status: opted_out`.", tags: ["Agent Compliance"], request: { params: z.object({ @@ -6949,6 +6954,141 @@ export function createRegistryApiRouters(config: RegistryApiConfig): { serializeStoryboardStatus(s, { includeDiagnostics }), ); + // Phase 1 of owner-selectable grading profiles is comparison-only. + // Reuse the same heartbeat evidence and keep Legacy authoritative; no + // read on this path contacts the agent or changes public trust state. + let gradingProfileComparisons: Array> = []; + if (includeDiagnostics) { + try { + const assessment = await getLatestVerificationProfileAssessment( + agentUrl, + VERIFICATION_PROFILE_SHADOW_POLICY_VERSION, + ); + if (!assessment) { + gradingProfileComparisons = [{ + scope: 'agent', + availability: 'pending', + unavailable_reason: 'No current-policy comparison is available yet. The next completed heartbeat will create one.', + selected_profile: 'legacy', + selection_enabled: false, + compliance_bundle_version: status.adcp_version ?? null, + profiles: null, + }]; + } else { + const ageMs = Date.now() - new Date(assessment.evaluated_at).getTime(); + const staleAfterMs = Math.max(24, (metadata?.check_interval_hours ?? 12) * 2) * 60 * 60 * 1000; + const assessedAt = new Date(assessment.evaluated_at); + const sourceTestedAt = new Date(assessment.source_tested_at); + const sourceIsLatest = status.last_run_id !== null + && assessment.source_run_id === status.last_run_id; + const datesInvalid = !Number.isFinite(ageMs) + || Number.isNaN(assessedAt.getTime()) + || Number.isNaN(sourceTestedAt.getTime()); + const stale = !sourceIsLatest || datesInvalid || ageMs > staleAfterMs; + const specReasons = [ + assessment.observed_failure_count > 0 + ? `${assessment.observed_failure_count} observed failure${assessment.observed_failure_count === 1 ? '' : 's'}` + : null, + assessment.failing_bundle_count > 0 + ? `${assessment.failing_bundle_count} failing evidence bundle${assessment.failing_bundle_count === 1 ? '' : 's'}` + : null, + assessment.flat_failure_count > 0 + ? `${assessment.flat_failure_count} run-level failure record${assessment.flat_failure_count === 1 ? '' : 's'}` + : null, + assessment.incomplete_bundle_count > 0 + ? `${assessment.incomplete_bundle_count} incomplete evidence bundle${assessment.incomplete_bundle_count === 1 ? '' : 's'}` + : null, + assessment.controller_gap_phase_count > 0 + ? `${assessment.controller_gap_phase_count} controller-gap phase${assessment.controller_gap_phase_count === 1 ? '' : 's'}` + : null, + assessment.unattributed_failure_count > 0 + ? `${assessment.unattributed_failure_count} unattributed failure${assessment.unattributed_failure_count === 1 ? '' : 's'}` + : null, + !assessment.bundle_evidence_present ? 'bundle evidence is missing' : null, + !assessment.run_complete ? 'the source run is incomplete' : null, + ].filter((reason): reason is string => reason !== null); + const sandboxReasons = sandboxProfileAssessmentReasons(assessment); + const staleExplanation = !sourceIsLatest + ? 'A newer authoritative compliance run exists without a matching comparison. This older result is historical evidence and cannot count as a pass.' + : datesInvalid + ? 'The comparison has invalid freshness timestamps, so its observed result is historical evidence and cannot count as a pass.' + : 'The latest comparison is stale, so its observed result is shown only as historical evidence and cannot count as a pass.'; + const outcome = ( + observedStatus: string | null, + eligible: boolean, + explanation: string, + ) => ({ + available: eligible && !stale, + status: eligible && !stale ? observedStatus : null, + observed_status: observedStatus, + explanation: !eligible ? explanation : stale ? staleExplanation : explanation, + }); + gradingProfileComparisons = [{ + scope: 'agent', + availability: stale ? 'stale' : 'current', + unavailable_reason: stale ? staleExplanation : null, + selected_profile: 'legacy', + selection_enabled: false, + source_run_id: assessment.source_run_id, + evaluator_policy_version: assessment.policy_version, + requested_compliance_target: assessment.requested_compliance_target, + compliance_bundle_version: assessment.adcp_version, + assessed_at: Number.isNaN(assessedAt.getTime()) ? null : assessedAt.toISOString(), + source_tested_at: Number.isNaN(sourceTestedAt.getTime()) ? null : sourceTestedAt.toISOString(), + stale, + incomplete: assessment.proposed_spec_status === 'partial' + || assessment.proposed_sandbox_status === 'partial' + || (assessment.sandbox_eligible && assessment.proposed_sandbox_status === null), + public_impact: 'None. Legacy remains authoritative during read-only comparison.', + profiles: { + legacy: outcome( + assessment.current_public_status, + true, + 'Current production grading and badge behavior.', + ), + spec: outcome( + assessment.proposed_spec_status, + true, + assessment.proposed_spec_status === 'passing' + ? 'The run is complete and all required evidence passes strict grading.' + : `${assessment.proposed_spec_status === 'failing' ? 'Strict grading failed' : 'Strict grading is incomplete'}: ${specReasons.join('; ') || 'required evidence is unresolved'}.`, + ), + sandbox: outcome( + assessment.proposed_sandbox_status, + assessment.sandbox_eligible, + !assessment.sandbox_eligible + ? 'Sandbox grading is available only for production lifecycle agents.' + : assessment.proposed_sandbox_status === 'passing' + ? 'Observable production behavior passes; only catalog-proven controller gaps are excluded.' + : `Sandbox grading ${assessment.proposed_sandbox_status === 'failing' ? 'failed' : 'is incomplete'}: ${sandboxReasons.join('; ') || 'required evidence is unresolved'}.`, + ), + }, + evidence: { + run_complete: assessment.run_complete, + bundle_evidence_present: assessment.bundle_evidence_present, + selected_storyboard_count: assessment.selected_storyboard_count, + controller_gap_phase_count: assessment.controller_gap_phase_count, + observed_failure_count: assessment.observed_failure_count, + flat_failure_count: assessment.flat_failure_count, + unattributed_failure_count: assessment.unattributed_failure_count, + sandbox_unresolved_bundle_count: assessment.sandbox_unresolved_bundle_count, + }, + }]; + } + } catch (err) { + logger.warn({ err, agentUrl }, "Verification profile comparison query failed"); + gradingProfileComparisons = [{ + scope: 'agent', + availability: 'temporarily_unavailable', + unavailable_reason: 'Comparison evidence is temporarily unavailable. Legacy remains authoritative.', + selected_profile: 'legacy', + selection_enabled: false, + compliance_bundle_version: status.adcp_version ?? null, + profiles: null, + }]; + } + } + res.json({ agent_url: agentUrl, requested_compliance_target: status.requested_compliance_target ?? null, @@ -7022,6 +7162,10 @@ export function createRegistryApiRouters(config: RegistryApiConfig): { verified_protocol_version: b.verified_protocol_version, badge_url: `/api/registry/agents/${encodedUrl}/badge/${b.role}.svg`, })), + // Owner/admin-only comparisons. Anonymous and cross-org callers get + // the same key with an empty list so ownership cannot be inferred + // from response shape. + grading_profile_comparisons: gradingProfileComparisons, }); } catch (error) { logger.error({ err: error, path: req.path }, "Failed to get compliance status"); diff --git a/server/src/schemas/registry.ts b/server/src/schemas/registry.ts index f7cb1722194..9a215755719 100644 --- a/server/src/schemas/registry.ts +++ b/server/src/schemas/registry.ts @@ -828,6 +828,47 @@ export const VerificationBadgeSchema = z }) .openapi("VerificationBadge"); +const GradingProfileOutcomeSchema = z.object({ + available: z.boolean(), + status: z.enum(['passing', 'partial', 'failing']).nullable(), + observed_status: z.enum(['passing', 'partial', 'failing']).nullable(), + explanation: z.string(), +}); + +export const GradingProfileComparisonSchema = z.object({ + scope: z.literal('agent').openapi({ + description: 'The current evaluator produces an agent-wide result. It must not be interpreted as an exact badge-role grade.', + }), + availability: z.enum(['current', 'stale', 'pending', 'temporarily_unavailable']), + unavailable_reason: z.string().nullable(), + selected_profile: z.literal('legacy'), + selection_enabled: z.literal(false), + source_run_id: z.string().uuid().optional(), + evaluator_policy_version: z.string().optional(), + requested_compliance_target: z.string().nullable().optional(), + compliance_bundle_version: z.string().nullable(), + assessed_at: z.string().datetime().nullable().optional(), + source_tested_at: z.string().datetime().nullable().optional(), + stale: z.boolean().optional(), + incomplete: z.boolean().optional(), + public_impact: z.string().optional(), + profiles: z.object({ + legacy: GradingProfileOutcomeSchema, + spec: GradingProfileOutcomeSchema, + sandbox: GradingProfileOutcomeSchema, + }).nullable(), + evidence: z.object({ + run_complete: z.boolean(), + bundle_evidence_present: z.boolean(), + selected_storyboard_count: z.number().int().nonnegative(), + controller_gap_phase_count: z.number().int().nonnegative(), + observed_failure_count: z.number().int().nonnegative(), + flat_failure_count: z.number().int().nonnegative(), + unattributed_failure_count: z.number().int().nonnegative(), + sandbox_unresolved_bundle_count: z.number().int().nonnegative(), + }).optional(), +}).openapi('GradingProfileComparison'); + export const AgentComplianceDetailSchema = z .object({ provenance: ComplianceRunProvenanceSchema.nullable().optional(), @@ -894,6 +935,9 @@ export const AgentComplianceDetailSchema = z .openapi({ description: "Owner-scoped: triggered_by value of the most recent non-dry-run compliance check. Null for non-owners and when no run has been recorded. Operators use this as a UX cue ('did this verdict come from my recent test or the system heartbeat?')." }), verified: z.boolean().optional(), verified_badges: z.array(VerificationBadgeSchema).optional(), + grading_profile_comparisons: z.array(GradingProfileComparisonSchema).optional().openapi({ + description: "Owner/admin-only read-only agent-wide comparison of Legacy, Spec, and Sandbox grading from one source run. Empty for other viewers. This is not an exact badge-role result. Legacy remains authoritative while selection_enabled is false; reading this field never contacts the agent.", + }), }) .openapi("AgentComplianceDetail"); diff --git a/server/src/services/verification-profile-shadow.ts b/server/src/services/verification-profile-shadow.ts index c85b48aa2ee..7572b30136b 100644 --- a/server/src/services/verification-profile-shadow.ts +++ b/server/src/services/verification-profile-shadow.ts @@ -31,6 +31,7 @@ export interface VerificationProfileShadowAssessment { incomplete_bundle_count: number; sandbox_unresolved_bundle_count: number; unattributed_failure_count: number; + flat_failure_count: number; unattributed_flat_failure_count: number; unexplained_phase_failure_count: number; sandbox_unresolved_executed_bundle_count: number; @@ -38,6 +39,43 @@ export interface VerificationProfileShadowAssessment { sandbox_unresolved_unknown_bundle_count: number; } +/** + * Explain every fail-closed Sandbox gate using the same assessment fields as + * the evaluator below. Keeping this next to the evaluator makes UI copy drift + * visible in unit tests when a new gate is added. + */ +export function sandboxProfileAssessmentReasons( + assessment: VerificationProfileShadowAssessment, +): string[] { + const reasons: string[] = []; + const addCount = (count: number, singular: string, plural = `${singular}s`) => { + if (count > 0) reasons.push(`${count} ${count === 1 ? singular : plural}`); + }; + + addCount(assessment.sandbox_observable_failure_count, 'production-observable failure'); + addCount(assessment.flat_failure_count, 'run-level failure record'); + addCount(assessment.unattributed_flat_failure_count, 'unattributed run-level failure'); + addCount(assessment.failing_bundle_count, 'failing evidence bundle'); + if (!assessment.run_complete) reasons.push('the source run is incomplete'); + if (!assessment.bundle_evidence_present) reasons.push('bundle evidence is missing'); + if (assessment.applicable_phase_count === 0) reasons.push('no applicable phases were observed'); + addCount(assessment.sandbox_unresolved_bundle_count, 'unresolved evidence bundle'); + addCount(assessment.non_controller_gap_step_count, 'non-controller evidence gap'); + addCount( + assessment.other_missing_storyboard_count, + 'storyboard missing for reasons other than the controller', + 'storyboards missing for reasons other than the controller', + ); + addCount(assessment.mixed_controller_failure_phase_count, 'mixed controller/failure phase'); + + const controllerExplainsPublicGap = assessment.controller_gap_phase_count > 0 + || assessment.controller_missing_storyboard_count > 0; + if (assessment.current_public_status !== 'passing' && !controllerExplainsPublicGap) { + reasons.push(`the public outcome is ${assessment.current_public_status} without a controller-only explanation`); + } + return reasons; +} + type ShadowStep = TestStepResult & { requirement?: string; skip?: { detail?: unknown }; @@ -366,6 +404,7 @@ export function deriveVerificationProfileShadowAssessment( incomplete_bundle_count: incompleteBundles.length, sandbox_unresolved_bundle_count: sandboxEligible ? sandboxIncompleteBundleCount : 0, unattributed_failure_count: unattributedFailureCount, + flat_failure_count: flatFailures.length, unattributed_flat_failure_count: unattributedFlatFailureCount, unexplained_phase_failure_count: unexplainedPhaseFailureCount, sandbox_unresolved_executed_bundle_count: sandboxEligible diff --git a/server/tests/integration/registry-api-compliance-verdict-source.test.ts b/server/tests/integration/registry-api-compliance-verdict-source.test.ts index cc2e3b77a0b..0f872303212 100644 --- a/server/tests/integration/registry-api-compliance-verdict-source.test.ts +++ b/server/tests/integration/registry-api-compliance-verdict-source.test.ts @@ -164,9 +164,9 @@ describe('GET /api/registry/agents/:encodedUrl/compliance — owner-scope gate ( `INSERT INTO agent_compliance_runs ( agent_url, lifecycle_stage, overall_status, headline, tracks_json, tracks_passed, tracks_failed, tracks_skipped, tracks_partial, - triggered_by, dry_run, tested_at + triggered_by, dry_run, tested_at, adcp_version, requested_compliance_target ) VALUES ($1, 'production', 'passing', 'all clear', - '[]'::jsonb, 0, 0, 0, 0, 'owner_test', false, NOW()) + '[]'::jsonb, 0, 0, 0, 0, 'owner_test', false, NOW(), '3.1.20', '3.1') RETURNING id`, [AGENT_URL], ); @@ -174,17 +174,51 @@ describe('GET /api/registry/agents/:encodedUrl/compliance — owner-scope gate ( await pool.query( `INSERT INTO agent_compliance_status ( agent_url, status, last_checked_at, last_passed_at, - tracks_summary_json, headline, status_changed_at, updated_at + tracks_summary_json, headline, status_changed_at, updated_at, + adcp_version, requested_compliance_target ) VALUES ($1, 'passing', NOW(), NOW(), - '{}'::jsonb, 'all clear', NOW(), NOW()) + '{}'::jsonb, 'all clear', NOW(), NOW(), '3.1.20', '3.1') ON CONFLICT (agent_url) DO UPDATE SET status = EXCLUDED.status, last_checked_at = NOW(), last_passed_at = NOW(), headline = EXCLUDED.headline, + adcp_version = EXCLUDED.adcp_version, + requested_compliance_target = EXCLUDED.requested_compliance_target, updated_at = NOW()`, [AGENT_URL], ); + await pool.query( + `INSERT INTO verification_profile_shadow_assessments ( + source_run_id, agent_url, lifecycle_stage, adcp_version, policy_version, + current_public_status, proposed_spec_status, proposed_sandbox_status, + sandbox_eligible, recommended_profile, run_complete, + bundle_evidence_present, failing_bundle_count, + incomplete_bundle_count, sandbox_unresolved_bundle_count, + unattributed_failure_count, + selected_storyboard_count, applicable_phase_count, + controller_gap_phase_count, controller_gap_step_count, + controller_cascade_step_count, observed_failure_count, + sandbox_observable_failure_count, non_controller_gap_step_count, + controller_missing_storyboard_count, other_missing_storyboard_count, + mixed_controller_failure_phase_count, + unattributed_flat_failure_count, unexplained_phase_failure_count, + sandbox_unresolved_executed_bundle_count, + sandbox_unresolved_missing_tools_bundle_count, + sandbox_unresolved_unknown_bundle_count, + source_tested_at, requested_compliance_target + ) VALUES ( + $1, $2, 'production', '3.1.20', 'verification-profiles-v3', + 'passing', 'partial', 'passing', + TRUE, 'sandbox', TRUE, + TRUE, 0, 1, 0, 0, + 12, 10, 1, 1, 1, 0, + 0, 0, 1, 0, 0, + 0, 0, 0, 0, 0, + NOW(), '3.1' + )`, + [complianceRunId, AGENT_URL], + ); await pool.query( `INSERT INTO agent_storyboard_status ( agent_url, storyboard_id, status, last_tested_at, run_id, @@ -300,6 +334,7 @@ describe('GET /api/registry/agents/:encodedUrl/compliance — owner-scope gate ( expect(res.body.membership_tier_label).toBeNull(); expect(res.body.subscription_status).toBeNull(); expect(res.body.is_api_access_tier).toBe(false); + expect(res.body.grading_profile_comparisons).toEqual([]); expectPublicStoryboardStatus(res.body); }); @@ -315,6 +350,7 @@ describe('GET /api/registry/agents/:encodedUrl/compliance — owner-scope gate ( expect(res.body.membership_tier_label).toBeNull(); expect(res.body.subscription_status).toBeNull(); expect(res.body.is_api_access_tier).toBe(false); + expect(res.body.grading_profile_comparisons).toEqual([]); expectPublicStoryboardStatus(res.body); }); @@ -326,9 +362,75 @@ describe('GET /api/registry/agents/:encodedUrl/compliance — owner-scope gate ( expect(res.body.membership_tier).toBe('company_standard'); expect(res.body.subscription_status).toBe('active'); expect(res.body.is_api_access_tier).toBe(true); + expect(res.body.grading_profile_comparisons).toEqual([ + expect.objectContaining({ + scope: 'agent', + availability: 'current', + selected_profile: 'legacy', + selection_enabled: false, + source_run_id: complianceRunId, + evaluator_policy_version: 'verification-profiles-v3', + requested_compliance_target: '3.1', + compliance_bundle_version: '3.1.20', + stale: false, + evidence: expect.objectContaining({ flat_failure_count: 0 }), + profiles: { + legacy: expect.objectContaining({ available: true, status: 'passing' }), + spec: expect.objectContaining({ available: true, status: 'partial' }), + sandbox: expect.objectContaining({ available: true, status: 'passing' }), + }, + }), + ]); expectPublicStoryboardStatus(res.body, { includeDiagnostics: true }); }); + it('static admin API key sees the same read-only agent-wide comparison', async () => { + currentUserId = STATIC_ADMIN_USER_ID; + const res = await request(app).get(endpoint); + expect(res.status).toBe(200); + expect(res.body.grading_profile_comparisons).toEqual([ + expect.objectContaining({ + scope: 'agent', + availability: 'current', + source_run_id: complianceRunId, + selection_enabled: false, + }), + ]); + }); + + it('fails closed when the latest public heartbeat has no matching comparison', async () => { + const newerRun = await pool.query<{ id: string }>( + `INSERT INTO agent_compliance_runs ( + agent_url, lifecycle_stage, overall_status, headline, tracks_json, + triggered_by, dry_run, tested_at, adcp_version, requested_compliance_target + ) VALUES ( + $1, 'production', 'passing', 'newer heartbeat', '[]'::jsonb, + 'heartbeat', FALSE, NOW() + INTERVAL '1 minute', '3.1.20', '3.1' + ) RETURNING id`, + [AGENT_URL], + ); + try { + currentUserId = OWNER_USER_ID; + const res = await request(app).get(endpoint); + expect(res.status).toBe(200); + expect(res.body.grading_profile_comparisons).toEqual([ + expect.objectContaining({ + availability: 'stale', + source_run_id: complianceRunId, + stale: true, + unavailable_reason: expect.stringContaining('newer authoritative compliance run'), + profiles: { + legacy: expect.objectContaining({ available: false, status: null, observed_status: 'passing' }), + spec: expect.objectContaining({ available: false, status: null, observed_status: 'partial' }), + sandbox: expect.objectContaining({ available: false, status: null, observed_status: 'passing' }), + }, + }), + ]); + } finally { + await pool.query('DELETE FROM agent_compliance_runs WHERE id = $1', [newerRun.rows[0].id]); + } + }); + it('owner of a free-tier org still sees verdict_source (is_owner is broader than is_api_access_tier)', async () => { // Drop the owner org's membership tier to null so is_api_access_tier // computes false. The verdict_source gate is on is_owner, which is diff --git a/server/tests/integration/verification-profile-shadow-migration.test.ts b/server/tests/integration/verification-profile-shadow-migration.test.ts index 8d66857b9c9..e8182033593 100644 --- a/server/tests/integration/verification-profile-shadow-migration.test.ts +++ b/server/tests/integration/verification-profile-shadow-migration.test.ts @@ -13,6 +13,10 @@ const DIAGNOSTICS_MIGRATION = readFileSync( resolve(__dirname, '../../src/db/migrations/576_verification_profile_shadow_diagnostics.sql'), 'utf8', ); +const ENABLE_COMPARISONS_MIGRATION = readFileSync( + resolve(__dirname, '../../src/db/migrations/591_enable_verification_profile_comparisons.sql'), + 'utf8', +); describe.skipIf(!process.env.DATABASE_URL)('verification profile shadow migrations', () => { let pool: Pool; @@ -33,6 +37,14 @@ describe.skipIf(!process.env.DATABASE_URL)('verification profile shadow migratio updated_at TIMESTAMPTZ DEFAULT NOW(), updated_by VARCHAR(255) ); + CREATE TABLE system_settings_audit ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + key VARCHAR(100) NOT NULL, + old_value JSONB, + new_value JSONB NOT NULL, + changed_by VARCHAR(255), + changed_at TIMESTAMPTZ NOT NULL DEFAULT NOW() + ); CREATE TABLE agent_compliance_runs ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), agent_url TEXT, @@ -40,7 +52,9 @@ describe.skipIf(!process.env.DATABASE_URL)('verification profile shadow migratio overall_status TEXT NOT NULL DEFAULT 'unknown', tracks_json JSONB NOT NULL DEFAULT '[]'::jsonb, dry_run BOOLEAN NOT NULL DEFAULT FALSE, - is_authoritative BOOLEAN NOT NULL DEFAULT TRUE + is_authoritative BOOLEAN NOT NULL DEFAULT TRUE, + adcp_version TEXT, + requested_compliance_target TEXT ); CREATE TABLE discovered_agents (agent_url TEXT PRIMARY KEY); CREATE TABLE agent_registry_metadata ( @@ -64,6 +78,7 @@ describe.skipIf(!process.env.DATABASE_URL)('verification profile shadow migratio sourceRunId = run.rows[0].id; await client.query(MIGRATION); await client.query(DIAGNOSTICS_MIGRATION); + await client.query(ENABLE_COMPARISONS_MIGRATION); }); afterAll(async () => { @@ -75,12 +90,49 @@ describe.skipIf(!process.env.DATABASE_URL)('verification profile shadow migratio await pool?.end(); }); - it('starts collection disabled and remains idempotent', async () => { + it('enables persistent comparison collection with an audit record', async () => { await client.query(MIGRATION); - const setting = await client.query<{ value: { enabled: boolean; expires_at: string | null } }>( - `SELECT value FROM system_settings WHERE key = 'verification_profile_shadow_rollout'`, + const setting = await client.query<{ + value: { enabled: boolean; expires_at: string | null }; + updated_by: string; + }>( + `SELECT value, updated_by + FROM system_settings + WHERE key = 'verification_profile_shadow_rollout'`, + ); + expect(setting.rows).toEqual([{ + value: { enabled: true, expires_at: null }, + updated_by: 'migration:591_enable_verification_profile_comparisons', + }]); + + const audit = await client.query<{ + old_value: { enabled: boolean; expires_at: string | null }; + new_value: { enabled: boolean; expires_at: string | null }; + changed_by: string; + }>( + `SELECT old_value, new_value, changed_by + FROM system_settings_audit + WHERE key = 'verification_profile_shadow_rollout'`, ); - expect(setting.rows).toEqual([{ value: { enabled: false, expires_at: null } }]); + expect(audit.rows).toEqual([{ + old_value: { enabled: false, expires_at: null }, + new_value: { enabled: true, expires_at: null }, + changed_by: 'migration:591_enable_verification_profile_comparisons', + }]); + }); + + it('backfills immutable source provenance before enabling collection', async () => { + const columns = await client.query<{ column_name: string }>( + `SELECT column_name + FROM information_schema.columns + WHERE table_schema = $1 + AND table_name = 'verification_profile_shadow_assessments'`, + [TEST_SCHEMA], + ); + expect(columns.rows.map((row) => row.column_name)).toEqual(expect.arrayContaining([ + 'source_tested_at', + 'requested_compliance_target', + ])); }); it('accepts a bounded production assessment without raw request or response columns', async () => { @@ -97,12 +149,12 @@ describe.skipIf(!process.env.DATABASE_URL)('verification profile shadow migratio controller_cascade_step_count, observed_failure_count, sandbox_observable_failure_count, non_controller_gap_step_count, controller_missing_storyboard_count, other_missing_storyboard_count, - mixed_controller_failure_phase_count + mixed_controller_failure_phase_count, source_tested_at ) VALUES ( $1, 'https://seller.example.test/mcp', 'production', '3.1', 'verification-profiles-v1', 'passing', 'partial', 'passing', TRUE, 'sandbox', TRUE, TRUE, 0, 0, 0, 0, - 10, 9, 1, 1, 2, 0, 0, 0, 1, 0, 0 + 10, 9, 1, 1, 2, 0, 0, 0, 1, 0, 0, NOW() )`, [sourceRunId], ); @@ -137,12 +189,12 @@ describe.skipIf(!process.env.DATABASE_URL)('verification profile shadow migratio controller_cascade_step_count, observed_failure_count, sandbox_observable_failure_count, non_controller_gap_step_count, controller_missing_storyboard_count, other_missing_storyboard_count, - mixed_controller_failure_phase_count + mixed_controller_failure_phase_count, source_tested_at ) VALUES ( $1, 'https://seller.example.test/partial', 'production', 'verification-profiles-v1', 'partial', 'partial', 'partial', TRUE, NULL, TRUE, TRUE, 2, 1, 0, 0, - 10, 9, 1, 1, 2, 2, 1, 0, 1, 0, 0 + 10, 9, 1, 1, 2, 2, 1, 0, 1, 0, 0, NOW() )`, [run.rows[0].id], )).resolves.toBeDefined(); @@ -165,12 +217,12 @@ describe.skipIf(!process.env.DATABASE_URL)('verification profile shadow migratio controller_cascade_step_count, observed_failure_count, sandbox_observable_failure_count, non_controller_gap_step_count, controller_missing_storyboard_count, other_missing_storyboard_count, - mixed_controller_failure_phase_count + mixed_controller_failure_phase_count, source_tested_at ) VALUES ( $1, 'https://seller.example.test/testing', 'testing', 'verification-profiles-v1', 'passing', 'passing', 'passing', TRUE, 'sandbox', TRUE, TRUE, 0, 0, 0, 0, - 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0 + 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, NOW() )`, [anotherRun.rows[0].id], )).rejects.toThrow(); @@ -232,16 +284,16 @@ describe.skipIf(!process.env.DATABASE_URL)('verification profile shadow migratio controller_cascade_step_count, observed_failure_count, sandbox_observable_failure_count, non_controller_gap_step_count, controller_missing_storyboard_count, other_missing_storyboard_count, - mixed_controller_failure_phase_count + mixed_controller_failure_phase_count, source_tested_at ) VALUES ($1, 'https://audit.example.test/mcp', 'production', '3.1', 'verification-profiles-v3', 'partial', 'partial', 'partial', TRUE, NULL, FALSE, TRUE, 0, 1, 0, 0, - 10, 9, 0, 0, 0, 0, 0, 0, 0, 0, 0), + 10, 9, 0, 0, 0, 0, 0, 0, 0, 0, 0, NOW()), ($2, 'https://audit.example.test/mcp', 'production', '3.1', 'verification-profiles-v3', 'partial', 'partial', 'partial', TRUE, NULL, TRUE, TRUE, 0, 1, 0, 0, - 10, 9, 0, 0, 0, 0, 0, 0, 0, 0, 0)`, + 10, 9, 0, 0, 0, 0, 0, 0, 0, 0, 0, NOW())`, [firstRun.rows[0].id, secondRun.rows[0].id], ); diff --git a/server/tests/unit/admin-organization-authorization-setting.test.ts b/server/tests/unit/admin-organization-authorization-setting.test.ts index f77b47833cc..0fb61a83bbb 100644 --- a/server/tests/unit/admin-organization-authorization-setting.test.ts +++ b/server/tests/unit/admin-organization-authorization-setting.test.ts @@ -85,7 +85,7 @@ describe("organization authorization runtime admin setting", () => { getShadowSettingMock.mockResolvedValue({ enabled: false, expires_at: null }); setShadowSettingMock.mockResolvedValue({ enabled: true, - expires_at: '2026-09-02T10:00:00.000Z', + expires_at: null, }); }); @@ -108,7 +108,7 @@ describe("organization authorization runtime admin setting", () => { ); expect(putResponse.body.verification_profile_shadow_rollout).toEqual({ enabled: true, - expires_at: '2026-09-02T10:00:00.000Z', + expires_at: null, }); }); diff --git a/server/tests/unit/compliance-heartbeat.test.ts b/server/tests/unit/compliance-heartbeat.test.ts index 346a9d55da4..9d0a4681de5 100644 --- a/server/tests/unit/compliance-heartbeat.test.ts +++ b/server/tests/unit/compliance-heartbeat.test.ts @@ -11,7 +11,6 @@ const mocks = vi.hoisted(() => ({ revokeBadge: vi.fn(), getRegistryMetadata: vi.fn(), query: vi.fn(), - withDatabaseDeadline: vi.fn(), comply: vi.fn(), complianceResultToDbInput: vi.fn(), classifyCapabilityResolutionError: vi.fn(), @@ -23,7 +22,6 @@ const mocks = vi.hoisted(() => ({ adaptAuthForSdk: vi.fn(), revokeUnsupportedPublicBadges: vi.fn(), runBadgeFanOut: vi.fn(), - getVerificationProfileShadowRollout: vi.fn(), recordVerificationProfileShadowAssessment: vi.fn(), pruneVerificationProfileShadowAssessments: vi.fn(), deriveVerificationProfileShadowAssessment: vi.fn(), @@ -62,7 +60,6 @@ vi.mock('../../src/db/compliance-db.js', () => ({ vi.mock('../../src/db/client.js', () => ({ query: mocks.query, - withDatabaseDeadline: mocks.withDatabaseDeadline, })); vi.mock('../../src/db/compliance-refresh-requests-db.js', () => ({ @@ -105,10 +102,6 @@ vi.mock('../../src/services/badge-issuance.js', () => ({ runBadgeFanOut: mocks.runBadgeFanOut, })); -vi.mock('../../src/db/system-settings-db.js', () => ({ - getVerificationProfileShadowRollout: mocks.getVerificationProfileShadowRollout, -})); - vi.mock('../../src/db/verification-profile-shadow-db.js', () => ({ recordVerificationProfileShadowAssessment: mocks.recordVerificationProfileShadowAssessment, pruneVerificationProfileShadowAssessments: mocks.pruneVerificationProfileShadowAssessments, @@ -138,7 +131,6 @@ describe('runComplianceHeartbeatJob', () => { { agent_url: 'https://agent.example.com/mcp', lifecycle_stage: 'testing', last_checked_at: null }, ]); mocks.query.mockResolvedValue({ rows: [], rowCount: 0 }); - mocks.withDatabaseDeadline.mockImplementation(async (_deadline, operation) => operation()); mocks.resolveOwnerAuth.mockResolvedValue(undefined); mocks.getLastKnownSupportedVersions.mockResolvedValue(['3.1']); mocks.countComplianceRuns.mockResolvedValue(4); @@ -158,7 +150,6 @@ describe('runComplianceHeartbeatJob', () => { mocks.recordComplianceRun.mockResolvedValue({}); mocks.getBadgesForAgent.mockResolvedValue([]); mocks.getRegistryMetadata.mockResolvedValue(null); - mocks.getVerificationProfileShadowRollout.mockResolvedValue({ enabled: false }); mocks.recordVerificationProfileShadowAssessment.mockResolvedValue(true); mocks.pruneVerificationProfileShadowAssessments.mockResolvedValue(0); mocks.revokeUnsupportedPublicBadges.mockResolvedValue({ issued: [], revoked: [], degraded: [], unchanged: [] }); @@ -195,7 +186,7 @@ describe('runComplianceHeartbeatJob', () => { queue: { eligibleBacklog: 0, selectedCount: 0, batchLimit: 1 }, inputs: expect.objectContaining({ policyVersion: 'verification-profiles-v3' }), outcomes: { checked: 0, passed: 0, failed: 0, skipped: 0 }, - shadow: expect.objectContaining({ candidates: 0, errors: 0, setting_errors: 0 }), + shadow: expect.objectContaining({ candidates: 0, attempted: 0, errors: 0 }), }), 'Compliance heartbeat shadow flush completed after public processing', ); @@ -409,13 +400,12 @@ describe('runComplianceHeartbeatJob', () => { expect(mocks.releaseExecutionFence).toHaveBeenCalledOnce(); }); - it('records shadow evidence only when the audited collection switch is enabled', async () => { + it('records comparison evidence from every authoritative heartbeat', async () => { const complianceResult = { overall_status: 'passing', summary: { headline: 'All good' }, agent_profile: { specialisms: [], adcp_supported_versions: ['3.1'] }, }; - mocks.getVerificationProfileShadowRollout.mockResolvedValueOnce({ enabled: true }); mocks.comply.mockResolvedValueOnce(complianceResult); mocks.recordComplianceRun.mockResolvedValueOnce({ run: { id: 'run-shadow' }, @@ -438,15 +428,9 @@ describe('runComplianceHeartbeatJob', () => { lifecycleStage: 'testing', }), ); - expect(mocks.withDatabaseDeadline).toHaveBeenCalledWith( - expect.any(Number), - expect.any(Function), - { readOnly: false }, - ); }); it('keeps public compliance successful when shadow persistence fails', async () => { - mocks.getVerificationProfileShadowRollout.mockResolvedValueOnce({ enabled: true }); mocks.comply.mockResolvedValueOnce({ overall_status: 'passing', summary: { headline: 'All good' }, @@ -468,7 +452,7 @@ describe('runComplianceHeartbeatJob', () => { }); }); - it('flushes shadow writes after public processing and rechecks disable before every write', async () => { + it('flushes every comparison write after public processing', async () => { const complianceResult = { overall_status: 'passing', summary: { headline: 'All good' }, @@ -482,9 +466,6 @@ describe('runComplianceHeartbeatJob', () => { mocks.recordComplianceRun .mockResolvedValueOnce({ run: { id: 'run-one' }, statusTransition: null, storyboardStatuses: [] }) .mockResolvedValueOnce({ run: { id: 'run-two' }, statusTransition: null, storyboardStatuses: [] }); - mocks.getVerificationProfileShadowRollout - .mockResolvedValueOnce({ enabled: true }) - .mockResolvedValueOnce({ enabled: false }); mocks.recordVerificationProfileShadowAssessment.mockImplementationOnce(async () => { expect(mocks.recordComplianceRun).toHaveBeenCalledTimes(2); return true; @@ -498,39 +479,12 @@ describe('runComplianceHeartbeatJob', () => { skipped: 0, }); - expect(mocks.getVerificationProfileShadowRollout).toHaveBeenCalledTimes(2); - expect(mocks.recordVerificationProfileShadowAssessment).toHaveBeenCalledTimes(1); + expect(mocks.recordVerificationProfileShadowAssessment).toHaveBeenCalledTimes(2); expect(mocks.recordVerificationProfileShadowAssessment).toHaveBeenCalledWith( expect.objectContaining({ sourceRunId: 'run-one' }), ); }); - it('fails shadow collection closed when its setting cannot be read', async () => { - mocks.getVerificationProfileShadowRollout.mockRejectedValueOnce(new Error('settings unavailable')); - mocks.comply.mockResolvedValueOnce({ - overall_status: 'passing', - summary: { headline: 'All good' }, - agent_profile: { specialisms: [], adcp_supported_versions: ['3.1'] }, - }); - mocks.recordComplianceRun.mockResolvedValueOnce({ - run: { id: 'run-setting-failure' }, - statusTransition: null, - storyboardStatuses: [], - }); - - const { runComplianceHeartbeatJob } = await import('../../src/addie/jobs/compliance-heartbeat.js'); - await expect(runComplianceHeartbeatJob({ limit: 1 })).resolves.toEqual({ - checked: 1, - passed: 1, - failed: 0, - skipped: 0, - }); - // Pure derivation may be queued after public processing, but a failed - // bounded setting read must prevent any shadow persistence. - expect(mocks.deriveVerificationProfileShadowAssessment).toHaveBeenCalledOnce(); - expect(mocks.recordVerificationProfileShadowAssessment).not.toHaveBeenCalled(); - }); - it('keeps public compliance successful when retention cleanup fails', async () => { mocks.comply.mockResolvedValueOnce({ overall_status: 'passing', diff --git a/server/tests/unit/dashboard-track-coverage-gap.test.ts b/server/tests/unit/dashboard-track-coverage-gap.test.ts index c317aa5b423..d6c7dae2877 100644 --- a/server/tests/unit/dashboard-track-coverage-gap.test.ts +++ b/server/tests/unit/dashboard-track-coverage-gap.test.ts @@ -21,11 +21,13 @@ const buildTrackCoverageGapNote = context.buildTrackCoverageGapNote as ( const blockerHelperStart = dashboardSource.indexOf('function pickStoryboardBlockingReason'); const blockerHelperEnd = dashboardSource.indexOf('function finiteCount', blockerHelperStart); +const gradingProfileStart = dashboardSource.indexOf('function renderGradingProfileComparisons'); const verificationPanelStart = dashboardSource.indexOf('function renderVerificationPanel'); const verificationPanelEnd = dashboardSource.indexOf('function timeAgo', verificationPanelStart); if ( blockerHelperStart < 0 || blockerHelperEnd < 0 || + gradingProfileStart < 0 || verificationPanelStart < 0 || verificationPanelEnd < 0 ) { @@ -35,8 +37,10 @@ if ( const verificationContext = vm.createContext({ buildNoticesSectionHtml: () => '', escapeHtml: (value: unknown) => String(value), + timeAgo: () => '2h ago', }); vm.runInContext(dashboardSource.slice(blockerHelperStart, blockerHelperEnd), verificationContext); +vm.runInContext(dashboardSource.slice(gradingProfileStart, verificationPanelStart), verificationContext); vm.runInContext(dashboardSource.slice(verificationPanelStart, verificationPanelEnd), verificationContext); type StoryboardStatus = { status?: string | null }; @@ -48,6 +52,9 @@ const renderVerificationPanel = verificationContext.renderVerificationPanel as ( agentUrl: string, hasAuth: boolean, ) => string; +const renderGradingProfileComparisons = verificationContext.renderGradingProfileComparisons as ( + complianceStatus: Record | null, +) => string; describe('dashboard track coverage-gap guidance', () => { it('explains partial tracks when coverage-gap skips block storyboard eligibility', () => { @@ -245,3 +252,67 @@ describe('dashboard verification blocker guidance', () => { expect(html).not.toContain('before it can earn AAO Verified (Spec)'); }); }); + +describe('dashboard grading profile comparison', () => { + const currentComparison = { + scope: 'agent', + availability: 'current', + selected_profile: 'legacy', + compliance_bundle_version: '3.1.20', + assessed_at: '2026-09-15T10:00:00.000Z', + source_tested_at: '2026-09-15T09:59:00.000Z', + source_run_id: '11111111-1111-4111-8111-111111111111', + evaluator_policy_version: 'verification-profiles-v3', + requested_compliance_target: '3.1', + profiles: { + legacy: { available: true, status: 'passing', observed_status: 'passing', explanation: 'Legacy passed.' }, + spec: { available: true, status: 'partial', observed_status: 'partial', explanation: 'One bundle is incomplete.' }, + sandbox: { available: true, status: 'passing', observed_status: 'passing', explanation: 'Observable behavior passed.' }, + }, + evidence: { + selected_storyboard_count: 12, + observed_failure_count: 0, + flat_failure_count: 0, + controller_gap_phase_count: 2, + sandbox_unresolved_bundle_count: 0, + }, + }; + + it('renders agent-wide scope, all profiles, and source provenance', () => { + const html = renderGradingProfileComparisons({ grading_profile_comparisons: [currentComparison] }); + expect(html).toContain('Agent-wide grading preview · 3.1.20'); + expect(html).toContain('Legacy · Current'); + expect(html).toContain('One bundle is incomplete.'); + expect(html).toContain('11111111-1111-4111-8111-111111111111'); + expect(html).toContain('verification-profiles-v3'); + expect(html).toContain('0 run-level failure records'); + expect(html).toContain('not an exact badge-role grade'); + }); + + it('never renders a stale observed pass as Passed', () => { + const stale = structuredClone(currentComparison); + stale.availability = 'stale'; + stale.profiles.legacy = { + available: false, + status: null, + observed_status: 'passing', + explanation: 'The latest comparison is stale.', + }; + const html = renderGradingProfileComparisons({ grading_profile_comparisons: [stale] }); + expect(html).toContain('Unavailable'); + expect(html).toContain('Historical observation: passing'); + expect(html).not.toContain('>Passed<'); + }); + + it('renders missing evidence as explicitly pending', () => { + const html = renderGradingProfileComparisons({ + grading_profile_comparisons: [{ + availability: 'pending', + profiles: null, + unavailable_reason: 'No current-policy comparison is available yet.', + }], + }); + expect(html).toContain('Agent-wide grading preview'); + expect(html).toContain('No current-policy comparison is available yet.'); + }); +}); diff --git a/server/tests/unit/verification-profile-shadow-db.test.ts b/server/tests/unit/verification-profile-shadow-db.test.ts index 8df2b655264..61c50308e55 100644 --- a/server/tests/unit/verification-profile-shadow-db.test.ts +++ b/server/tests/unit/verification-profile-shadow-db.test.ts @@ -10,6 +10,7 @@ vi.mock('../../src/db/client.js', () => ({ })); import { + getLatestVerificationProfileAssessment, pruneVerificationProfileShadowAssessments, recordVerificationProfileShadowAssessment, } from '../../src/db/verification-profile-shadow-db.js'; @@ -20,7 +21,7 @@ describe('recordVerificationProfileShadowAssessment', () => { deadlineMock.mockClear(); }); - it('writes an idempotent bounded assessment bound to its source run', async () => { + it('writes an immutable bounded assessment bound to its authoritative source run', async () => { await expect(recordVerificationProfileShadowAssessment({ sourceRunId: 'run-1', agentUrl: 'https://seller.example.test/mcp', @@ -39,6 +40,7 @@ describe('recordVerificationProfileShadowAssessment', () => { incomplete_bundle_count: 0, sandbox_unresolved_bundle_count: 0, unattributed_failure_count: 0, + flat_failure_count: 0, unattributed_flat_failure_count: 0, unexplained_phase_failure_count: 0, sandbox_unresolved_executed_bundle_count: 0, @@ -60,9 +62,14 @@ describe('recordVerificationProfileShadowAssessment', () => { expect(queryMock).toHaveBeenCalledOnce(); expect(deadlineMock).toHaveBeenCalledWith(expect.any(Number), expect.any(Function), { readOnly: false }); - expect(queryMock.mock.calls[0][0]).toContain('ON CONFLICT (source_run_id) DO UPDATE'); - expect(queryMock.mock.calls[0][0]).toContain('FOR SHARE'); - expect(queryMock.mock.calls[0][0]).toContain("value->>'enabled' = 'true'"); + expect(queryMock.mock.calls[0][0]).toContain('ON CONFLICT (source_run_id) DO NOTHING'); + expect(queryMock.mock.calls[0][0]).toContain('AND agent_url = $2'); + expect(queryMock.mock.calls[0][0]).toContain('AND lifecycle_stage = $3'); + expect(queryMock.mock.calls[0][0]).toContain('AND adcp_version IS NOT DISTINCT FROM $4'); + expect(queryMock.mock.calls[0][0]).toContain('FROM system_settings'); + expect(queryMock.mock.calls[0][0]).toContain( + `value = '{"enabled": true, "expires_at": null}'::jsonb`, + ); expect(queryMock.mock.calls[0][1]).toEqual([ 'run-1', 'https://seller.example.test/mcp', @@ -96,48 +103,39 @@ describe('recordVerificationProfileShadowAssessment', () => { 0, 0, 0, + 0, 'verification_profile_shadow_rollout', ]); }); - it('does not write when the rollout lease was disabled before the atomic insert', async () => { - queryMock.mockResolvedValueOnce({ rows: [], rowCount: 0 }); + it('returns the latest current-policy authoritative complete comparison', async () => { + const row = { source_run_id: 'run-2', adcp_version: '3.1' }; + queryMock.mockResolvedValueOnce({ rows: [row], rowCount: 1 }); - await expect(recordVerificationProfileShadowAssessment({ - sourceRunId: 'run-disabled', - agentUrl: 'https://seller.example.test/mcp', - lifecycleStage: 'production', - assessment: { - policy_version: 'verification-profiles-v1', - current_public_status: 'passing', - proposed_spec_status: 'passing', - proposed_sandbox_status: 'passing', - sandbox_eligible: true, - recommended_profile: 'spec', - run_complete: true, - bundle_evidence_present: true, - failing_bundle_count: 0, - incomplete_bundle_count: 0, - sandbox_unresolved_bundle_count: 0, - unattributed_failure_count: 0, - unattributed_flat_failure_count: 0, - unexplained_phase_failure_count: 0, - sandbox_unresolved_executed_bundle_count: 0, - sandbox_unresolved_missing_tools_bundle_count: 0, - sandbox_unresolved_unknown_bundle_count: 0, - selected_storyboard_count: 1, - applicable_phase_count: 1, - controller_gap_phase_count: 0, - controller_gap_step_count: 0, - controller_cascade_step_count: 0, - observed_failure_count: 0, - sandbox_observable_failure_count: 0, - non_controller_gap_step_count: 0, - controller_missing_storyboard_count: 0, - other_missing_storyboard_count: 0, - mixed_controller_failure_phase_count: 0, - }, - })).resolves.toBe(false); + await expect(getLatestVerificationProfileAssessment( + 'https://seller.example.test/mcp', + 'verification-profiles-v3', + )).resolves.toEqual(row); + + expect(queryMock.mock.calls[0][0]).toContain('r.is_authoritative = TRUE'); + expect(queryMock.mock.calls[0][0]).toContain("r.completeness = 'complete'"); + expect(queryMock.mock.calls[0][0]).toContain('r.agent_url = s.agent_url'); + expect(queryMock.mock.calls[0][0]).toContain('r.lifecycle_stage = s.lifecycle_stage'); + expect(queryMock.mock.calls[0][0]).toContain('r.adcp_version IS NOT DISTINCT FROM s.adcp_version'); + expect(queryMock.mock.calls[0][0]).toContain('LIMIT 1'); + expect(queryMock.mock.calls[0][1]).toEqual([ + 'https://seller.example.test/mcp', + 'verification-profiles-v3', + ]); + expect(deadlineMock).toHaveBeenCalledWith(expect.any(Number), expect.any(Function), { readOnly: true }); + }); + + it('returns null when no comparison exists', async () => { + queryMock.mockResolvedValueOnce({ rows: [], rowCount: 0 }); + await expect(getLatestVerificationProfileAssessment( + 'https://seller.example.test/mcp', + 'verification-profiles-v3', + )).resolves.toBeNull(); }); it('exposes the fixed-retention pruning procedure', async () => { diff --git a/server/tests/unit/verification-profile-shadow-setting.test.ts b/server/tests/unit/verification-profile-shadow-setting.test.ts index 248b9181623..02308287366 100644 --- a/server/tests/unit/verification-profile-shadow-setting.test.ts +++ b/server/tests/unit/verification-profile-shadow-setting.test.ts @@ -1,4 +1,4 @@ -import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { beforeEach, describe, expect, it, vi } from 'vitest'; const queryMock = vi.hoisted(() => vi.fn()); @@ -14,14 +14,13 @@ import { describe('verification profile shadow rollout setting', () => { beforeEach(() => queryMock.mockReset()); - afterEach(() => vi.useRealTimers()); it.each([ { rows: [] }, { rows: [{ value: true }] }, { rows: [{ value: { enabled: 'true', expires_at: null } }] }, + { rows: [{ value: { enabled: true, expires_at: null, unexpected: 'ignored' } }] }, { rows: [{ value: { enabled: true, expires_at: '2026-09-02T00:00:00.000Z', unexpected: 'ignored' } }] }, - { rows: [{ value: { enabled: true, expires_at: null } }] }, { rows: [{ value: { enabled: false, expires_at: '2026-09-02T00:00:00.000Z' } }] }, ])('defaults absent or malformed persisted values off %#', async ({ rows }) => { queryMock.mockResolvedValueOnce({ rows }); @@ -30,44 +29,35 @@ describe('verification profile shadow rollout setting', () => { it('reads a valid enabled value', async () => { queryMock.mockResolvedValueOnce({ - rows: [{ value: { enabled: true, expires_at: '2026-09-02T00:00:00.000Z' } }], + rows: [{ value: { enabled: true, expires_at: null } }], }); await expect(getVerificationProfileShadowRollout()).resolves.toEqual({ enabled: true, - expires_at: '2026-09-02T00:00:00.000Z', + expires_at: null, }); - expect(queryMock.mock.calls[0][0]).toContain('INSERT INTO system_settings_audit'); - expect(queryMock.mock.calls[0][1]).toEqual([ - 'verification_profile_shadow_rollout', - 'system:verification-profile-shadow-auto-expiry', - ]); + expect(queryMock.mock.calls[0][1]).toEqual(['verification_profile_shadow_rollout']); }); - it('returns the atomically disabled value after an audited expiry', async () => { + it('fails closed for legacy expiring leases', async () => { queryMock.mockResolvedValueOnce({ - rows: [{ value: { enabled: false, expires_at: null } }], + rows: [{ value: { enabled: true, expires_at: '2026-09-02T00:00:00.000Z' } }], }); await expect(getVerificationProfileShadowRollout()).resolves.toEqual({ enabled: false, expires_at: null, }); - expect(queryMock.mock.calls[0][0]).toContain("current.old_value->>'expires_at'"); - expect(queryMock.mock.calls[0][0]).toContain('UPDATE system_settings setting'); - expect(queryMock.mock.calls[0][0]).toContain('INSERT INTO system_settings_audit'); }); - it('writes an audited 72-hour lease when enabling', async () => { - vi.useFakeTimers(); - vi.setSystemTime(new Date('2026-08-30T10:00:00.000Z')); + it('writes an audited persistent switch when enabling', async () => { queryMock.mockResolvedValueOnce({ rows: [] }); await expect( setVerificationProfileShadowRollout({ enabled: true }, 'credential_admin'), - ).resolves.toEqual({ enabled: true, expires_at: '2026-09-02T10:00:00.000Z' }); + ).resolves.toEqual({ enabled: true, expires_at: null }); expect(queryMock.mock.calls[0][1]).toEqual([ 'verification_profile_shadow_rollout', - JSON.stringify({ enabled: true, expires_at: '2026-09-02T10:00:00.000Z' }), + JSON.stringify({ enabled: true, expires_at: null }), 'credential_admin', ]); }); diff --git a/server/tests/unit/verification-profile-shadow.test.ts b/server/tests/unit/verification-profile-shadow.test.ts index c5124eb76cc..7dc767eb370 100644 --- a/server/tests/unit/verification-profile-shadow.test.ts +++ b/server/tests/unit/verification-profile-shadow.test.ts @@ -1,6 +1,10 @@ import { describe, expect, it } from 'vitest'; import type { ComplianceResult } from '@adcp/sdk/testing'; -import { deriveVerificationProfileShadowAssessment } from '../../src/services/verification-profile-shadow.js'; +import { + deriveVerificationProfileShadowAssessment, + sandboxProfileAssessmentReasons, + type VerificationProfileShadowAssessment, +} from '../../src/services/verification-profile-shadow.js'; function resultWith( scenarios: Array<{ @@ -520,6 +524,10 @@ describe('deriveVerificationProfileShadowAssessment', () => { const assessment = deriveVerificationProfileShadowAssessment(result, 'production', 'passing'); expect(assessment.unattributed_failure_count).toBe(0); + expect(assessment.flat_failure_count).toBe(1); + expect(sandboxProfileAssessmentReasons(assessment)).toContain( + '1 run-level failure record', + ); expect(assessment.proposed_spec_status).toBe('failing'); expect(assessment.proposed_sandbox_status).toBe('failing'); }); @@ -544,6 +552,7 @@ describe('deriveVerificationProfileShadowAssessment', () => { const assessment = deriveVerificationProfileShadowAssessment(result, 'production', 'passing'); expect(assessment.unattributed_failure_count).toBe(1); + expect(assessment.flat_failure_count).toBe(1); expect(assessment.unattributed_flat_failure_count).toBe(1); expect(assessment.unexplained_phase_failure_count).toBe(0); expect(assessment.proposed_spec_status).toBe('failing'); @@ -560,6 +569,7 @@ describe('deriveVerificationProfileShadowAssessment', () => { const assessment = deriveVerificationProfileShadowAssessment(result, 'production', 'failing'); expect(assessment.unattributed_failure_count).toBe(1); + expect(assessment.flat_failure_count).toBe(0); expect(assessment.unattributed_flat_failure_count).toBe(0); expect(assessment.unexplained_phase_failure_count).toBe(1); }); @@ -606,3 +616,65 @@ describe('deriveVerificationProfileShadowAssessment', () => { expect(assessment.selected_storyboard_count).toBe(1); }); }); + +describe('sandboxProfileAssessmentReasons', () => { + const passingAssessment = (): VerificationProfileShadowAssessment => ({ + policy_version: 'verification-profiles-v3', + current_public_status: 'passing', + proposed_spec_status: 'passing', + proposed_sandbox_status: 'passing', + sandbox_eligible: true, + recommended_profile: 'spec', + run_complete: true, + bundle_evidence_present: true, + selected_storyboard_count: 1, + applicable_phase_count: 1, + controller_gap_phase_count: 0, + controller_gap_step_count: 0, + controller_cascade_step_count: 0, + observed_failure_count: 0, + sandbox_observable_failure_count: 0, + non_controller_gap_step_count: 0, + controller_missing_storyboard_count: 0, + other_missing_storyboard_count: 0, + mixed_controller_failure_phase_count: 0, + failing_bundle_count: 0, + incomplete_bundle_count: 0, + sandbox_unresolved_bundle_count: 0, + unattributed_failure_count: 0, + flat_failure_count: 0, + unattributed_flat_failure_count: 0, + unexplained_phase_failure_count: 0, + sandbox_unresolved_executed_bundle_count: 0, + sandbox_unresolved_missing_tools_bundle_count: 0, + sandbox_unresolved_unknown_bundle_count: 0, + }); + + it.each([ + ['sandbox_observable_failure_count', 1, '1 production-observable failure'], + ['flat_failure_count', 1, '1 run-level failure record'], + ['unattributed_flat_failure_count', 1, '1 unattributed run-level failure'], + ['failing_bundle_count', 1, '1 failing evidence bundle'], + ['run_complete', false, 'the source run is incomplete'], + ['bundle_evidence_present', false, 'bundle evidence is missing'], + ['applicable_phase_count', 0, 'no applicable phases were observed'], + ['sandbox_unresolved_bundle_count', 1, '1 unresolved evidence bundle'], + ['non_controller_gap_step_count', 1, '1 non-controller evidence gap'], + ['other_missing_storyboard_count', 1, '1 storyboard missing for reasons other than the controller'], + ['mixed_controller_failure_phase_count', 1, '1 mixed controller/failure phase'], + ['current_public_status', 'partial', 'the public outcome is partial without a controller-only explanation'], + ] as const)('explains the %s evaluator gate', (field, value, expected) => { + const assessment = passingAssessment(); + Object.assign(assessment, { [field]: value }); + expect(sandboxProfileAssessmentReasons(assessment)).toContain(expected); + }); + + it('does not call a public partial unexplained when a controller-only gap explains it', () => { + const assessment = passingAssessment(); + assessment.current_public_status = 'partial'; + assessment.controller_gap_phase_count = 1; + expect(sandboxProfileAssessmentReasons(assessment)).not.toContain( + 'the public outcome is partial without a controller-only explanation', + ); + }); +}); diff --git a/static/openapi/registry.yaml b/static/openapi/registry.yaml index a8e47a531bc..80a3c5ec72e 100644 --- a/static/openapi/registry.yaml +++ b/static/openapi/registry.yaml @@ -4509,6 +4509,11 @@ components: type: array items: $ref: "#/components/schemas/VerificationBadge" + grading_profile_comparisons: + type: array + items: + $ref: "#/components/schemas/GradingProfileComparison" + description: Owner/admin-only read-only agent-wide comparison of Legacy, Spec, and Sandbox grading from one source run. Empty for other viewers. This is not an exact badge-role result. Legacy remains authoritative while selection_enabled is false; reading this field never contacts the agent. required: - agent_url - status @@ -4623,6 +4628,203 @@ components: - verified_specialisms - verification_modes - verified_protocol_version + GradingProfileComparison: + type: object + properties: + scope: + type: string + enum: + - agent + description: The current evaluator produces an agent-wide result. It must not be interpreted as an exact badge-role grade. + availability: + type: string + enum: + - current + - stale + - pending + - temporarily_unavailable + unavailable_reason: + type: + - string + - "null" + selected_profile: + type: string + enum: + - legacy + selection_enabled: + type: boolean + enum: + - false + source_run_id: + type: string + format: uuid + evaluator_policy_version: + type: string + requested_compliance_target: + type: + - string + - "null" + compliance_bundle_version: + type: + - string + - "null" + assessed_at: + type: + - string + - "null" + format: date-time + source_tested_at: + type: + - string + - "null" + format: date-time + stale: + type: boolean + incomplete: + type: boolean + public_impact: + type: string + profiles: + type: + - object + - "null" + properties: + legacy: + type: object + properties: + available: + type: boolean + status: + type: + - string + - "null" + enum: + - passing + - partial + - failing + - null + observed_status: + type: + - string + - "null" + enum: + - passing + - partial + - failing + - null + explanation: + type: string + required: + - available + - status + - observed_status + - explanation + spec: + type: object + properties: + available: + type: boolean + status: + type: + - string + - "null" + enum: + - passing + - partial + - failing + - null + observed_status: + type: + - string + - "null" + enum: + - passing + - partial + - failing + - null + explanation: + type: string + required: + - available + - status + - observed_status + - explanation + sandbox: + type: object + properties: + available: + type: boolean + status: + type: + - string + - "null" + enum: + - passing + - partial + - failing + - null + observed_status: + type: + - string + - "null" + enum: + - passing + - partial + - failing + - null + explanation: + type: string + required: + - available + - status + - observed_status + - explanation + required: + - legacy + - spec + - sandbox + evidence: + type: object + properties: + run_complete: + type: boolean + bundle_evidence_present: + type: boolean + selected_storyboard_count: + type: integer + minimum: 0 + controller_gap_phase_count: + type: integer + minimum: 0 + observed_failure_count: + type: integer + minimum: 0 + flat_failure_count: + type: integer + minimum: 0 + unattributed_failure_count: + type: integer + minimum: 0 + sandbox_unresolved_bundle_count: + type: integer + minimum: 0 + required: + - run_complete + - bundle_evidence_present + - selected_storyboard_count + - controller_gap_phase_count + - observed_failure_count + - flat_failure_count + - unattributed_failure_count + - sandbox_unresolved_bundle_count + required: + - scope + - availability + - unavailable_reason + - selected_profile + - selection_enabled + - compliance_bundle_version + - profiles AgentVerification: type: object properties: @@ -11060,7 +11262,7 @@ paths: operationId: getAgentCompliance summary: Get agent compliance detail description: |- - Returns detailed compliance status for a single agent, including track-level results, storyboard counts, and timestamps. + Returns detailed compliance status for a single agent, including track-level results, storyboard counts, timestamps, and an owner-only read-only agent-wide grading-profile comparison. The comparison is not an exact badge-role grade. Reading it never contacts the agent or changes public badge state. If the agent has opted out of compliance monitoring, returns a minimal response with `status: opted_out`. tags: