Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .changeset/capabilities-echoed-release-external-root.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
---
'@adcp/sdk': patch
---

Validate `get_adcp_capabilities` against the release the agent echoed even when an external schema root is active for the pin (#3320). Hosted compliance always runs with one, so a 3.0 target graded a conformant 3.2 capabilities response against 3.0: every storyboard `response_schema` check on that task failed, and discovery raised spurious `capabilities_response_schema_invalid` notices.

The storyboard `response_schema` check, discovery's capability validation and `getAdcpCapabilities()` now use the echoed `adcp_version` when it is the pin's major, is listed in the response's own `adcp.supported_versions`, and has a schema root: one supplied for that release, an external root registered for it, or the SDK's bundled schemas. Otherwise they validate strictly against the pin. Requiring the `supported_versions` listing is new. Other tasks, storyboard selection, the run's `adcp_version` and grading stay on the pin.

New `servedReleaseSchemaRoots` option on `comply()`, `runStoryboard()` and `runStoryboardStep()`: a map of AdCP version to schema root (same shape as `schemaRoot`) for releases an agent may echo. Those roots are used only for this capabilities validation. `withServedReleaseSchemaRoots(roots, fn)` from `@adcp/sdk/testing` scopes the same map for other callers.
34 changes: 34 additions & 0 deletions docs/guides/VALIDATE-YOUR-AGENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -187,6 +187,40 @@ await comply(agentUrl, {
});
```

**Capabilities served at a newer release.** A 3.0 pin sends only
`adcp_major_version: 3`, so a conformant agent may serve and echo its highest
3.x release in `get_adcp_capabilities` (versioning spec, "Server echoes"). The
runner validates that one response against the echoed release when the echo is
the pin's major, is listed in the response's own `adcp.supported_versions`, and
has a schema root. That covers discovery's `capabilities_response_schema_invalid`
notices, `getAdcpCapabilities()` and every storyboard `response_schema` check on
`get_adcp_capabilities`, even with `schemaRoot` set for the pin. Otherwise it
validates strictly against the pin. Every other task's response, storyboard
selection, the run's `adcp_version` and grading stay on the pin.

The echoed release's schemas come from the SDK's bundle by default. A grader
that ships its own roots per release can supply them with
`servedReleaseSchemaRoots` (on `comply()`, `runStoryboard()` and
`runStoryboardStep()`), keyed by AdCP version:

```ts
await comply(agentUrl, {
version: '3.0.27',
complianceDir: '/app/dist/compliance/3.0.27',
schemaRoot: '/app/dist/schemas/3.0.27',
servedReleaseSchemaRoots: {
'3.1.29': '/app/dist/schemas/3.1.29',
'3.2.5': '/app/dist/schemas/3.2.5',
},
});
```

Those roots are used only for that capabilities validation. Each one is checked
against its key once per process, and a mismatched or missing root throws
before the run starts. Callers that validate outside `comply()` and the runner
can scope the same map with `withServedReleaseSchemaRoots(roots, fn)` from
`@adcp/sdk/testing`.

**Multi-agent storyboards under `comply()`.** `comply()` grades one agent, so
storyboards that declare `requires: [multi_agent]` (for example the
governance-aware seller scenarios) skip with `requirement_unmet` and cap their
Expand Down
126 changes: 74 additions & 52 deletions src/lib/core/TaskExecutor.ts
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ import {
type ValidationMode,
} from '../validation/client-hooks';
import { formatIssues } from '../validation/schema-validator';
import { resolveServedValidationVersion } from '../validation/schema-loader';
import { withServedValidationVersion } from '../validation/schema-loader';
import { ADCP_VERSION } from '../version';
import { unwrapProtocolResponse, isAdcpOperationSuccess } from '../utils/response-unwrapper';
import { extractAdcpErrorInfo, extractCorrelationId } from '../utils/error-extraction';
Expand Down Expand Up @@ -6125,64 +6125,31 @@ export class TaskExecutor {
// Normal dispatch uses lastKnownServerVersion; restart recovery passes
// the persisted continuation version explicitly so concurrent or fresh
// executors cannot validate the seller's response against another wire.
let validationVersion =
const pinnedValidationVersion =
(serverVersion ?? this.lastKnownServerVersion) === 'v2'
? 'v2.5'
: (this.responseAdcpVersionForValidation() ?? this.config.adcpVersion ?? ADCP_VERSION);
// Capabilities are negotiated before any pin applies: a 3.0 pin (or a
// major-only envelope) is served the seller's highest release in the
// major, echoed in `adcp_version`. Validate against that release, as the
// versioning spec requires ("Server echoes"), falling back to the pin
// when there is no usable echo. Only the validator's version changes;
// the unwrapper keeps `effectiveResponseAdcpVersion()`.
if (taskName === 'get_adcp_capabilities' && validationVersion !== 'v2.5') {
validationVersion = resolveServedValidationVersion(response, validationVersion);
}
const normalizedResponse = this.normalizeResponseForValidation(response, taskName, validationVersion === 'v2.5');
// TaskExecutor owns the user-facing log entry below. Passing debugLogs
// into the lower-level hook would emit a second, unversioned duplicate.
const outcome =
(this.config.experimentalIdentity33?.() ||
(this.config.identity33SchemaExperiment &&
normalizedResponse &&
typeof normalizedResponse === 'object' &&
isAdcpVersionAtLeast((normalizedResponse as { adcp_version?: string }).adcp_version, '3.3'))) &&
mode !== 'off'
? validateIdentity33Payload(taskName, 'response', normalizedResponse, validationVersion)
: validateIncomingResponse(taskName, normalizedResponse, mode, undefined, validationVersion);
if (outcome.valid) return { valid: true, errors: [] };

const errorStrings = outcome.issues.map(i => `${i.pointer}: ${i.message}`);

if (logViolations) {
debugLogs.push({
timestamp: new Date().toISOString(),
type: mode === 'strict' ? 'error' : 'warning',
message: `Schema validation ${mode === 'strict' ? 'failed' : 'warning'} for ${taskName}: ${formatIssues(
outcome.issues
)}`,
errors: errorStrings,
issues: outcome.issues,
schemaVariant: outcome.variant,
schemaVersion: validationVersion,
mode,
});
}

if (mode === 'warn') {
if (logViolations) {
console.warn(
`Schema validation failed for ${taskName} against AdCP ${validationVersion} (non-blocking):`,
errorStrings
);
}
return { valid: true, errors: [] };
}

if (logViolations) {
console.error(`Schema validation failed for ${taskName} against AdCP ${validationVersion}:`, errorStrings);
// versioning spec requires ("Server echoes"), when it is same-major,
// listed in the response's `supported_versions` and has a schema root;
// otherwise against the pin. This holds even when an external schema
// root is active for the pin. Only the validator's version changes; the
// unwrapper keeps `effectiveResponseAdcpVersion()`.
if (taskName === 'get_adcp_capabilities' && pinnedValidationVersion !== 'v2.5') {
return withServedValidationVersion(response, pinnedValidationVersion, validationVersion =>
this.validateResponseSchemaAgainst(response, taskName, debugLogs, mode, logViolations, validationVersion)
);
}
return { valid: false, errors: errorStrings };
return this.validateResponseSchemaAgainst(
response,
taskName,
debugLogs,
mode,
logViolations,
pinnedValidationVersion
);
} catch (error) {
const errorMessage = error instanceof Error ? error.message : 'unknown error';
console.error('Error during schema validation:', errorMessage);
Expand All @@ -6193,6 +6160,61 @@ export class TaskExecutor {
}
}

private validateResponseSchemaAgainst(
response: any,
taskName: string,
debugLogs: any[],
mode: ValidationMode,
logViolations: boolean,
validationVersion: string
): { valid: boolean; errors: string[] } {
const normalizedResponse = this.normalizeResponseForValidation(response, taskName, validationVersion === 'v2.5');
// TaskExecutor owns the user-facing log entry below. Passing debugLogs
// into the lower-level hook would emit a second, unversioned duplicate.
const outcome =
(this.config.experimentalIdentity33?.() ||
(this.config.identity33SchemaExperiment &&
normalizedResponse &&
typeof normalizedResponse === 'object' &&
isAdcpVersionAtLeast((normalizedResponse as { adcp_version?: string }).adcp_version, '3.3'))) &&
mode !== 'off'
? validateIdentity33Payload(taskName, 'response', normalizedResponse, validationVersion)
: validateIncomingResponse(taskName, normalizedResponse, mode, undefined, validationVersion);
if (outcome.valid) return { valid: true, errors: [] };

const errorStrings = outcome.issues.map(i => `${i.pointer}: ${i.message}`);

if (logViolations) {
debugLogs.push({
timestamp: new Date().toISOString(),
type: mode === 'strict' ? 'error' : 'warning',
message: `Schema validation ${mode === 'strict' ? 'failed' : 'warning'} for ${taskName}: ${formatIssues(
outcome.issues
)}`,
errors: errorStrings,
issues: outcome.issues,
schemaVariant: outcome.variant,
schemaVersion: validationVersion,
mode,
});
}

if (mode === 'warn') {
if (logViolations) {
console.warn(
`Schema validation failed for ${taskName} against AdCP ${validationVersion} (non-blocking):`,
errorStrings
);
}
return { valid: true, errors: [] };
}

if (logViolations) {
console.error(`Schema validation failed for ${taskName} against AdCP ${validationVersion}:`, errorStrings);
}
return { valid: false, errors: errorStrings };
}

private responseAdcpVersionForValidation(): string | undefined {
if (this.config.versionEnvelope === 'major-only') return '3.0';
return undefined;
Expand Down
15 changes: 8 additions & 7 deletions src/lib/testing/client.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ import type { AgentRequestSigningConfig } from '../types/adcp';
import { createHmac, hkdfSync, randomBytes } from 'node:crypto';
import { AsyncLocalStorage } from 'node:async_hooks';
import { validateIncomingResponse } from '../validation/client-hooks';
import { resolveServedValidationVersion } from '../validation/schema-loader';
import { withServedValidationVersion } from '../validation/schema-loader';
import { extractVersionUnsupportedDetails } from '../utils/error-extraction';
import { buildAgentSigningContext } from '../signing/client';
import type { VerifierCapability } from '../signing/types';
Expand Down Expand Up @@ -848,13 +848,14 @@ export async function discoverAgentProfile(
// Validate against the release the seller served (its echoed
// `adcp_version`), not the pin. A 3.0 pin negotiates major-only, so
// the seller serves its highest 3.x release (versioning spec,
// "Server echoes"). Falls back to the pin when there is no usable echo.
const validation = validateIncomingResponse(
'get_adcp_capabilities',
// "Server echoes"). The echo is used when it is same-major, listed in
// the response's `supported_versions` and has a schema root, even
// with an external root active for the pin; otherwise the pin.
const validation = withServedValidationVersion(
caps.data,
'strict',
undefined,
resolveServedValidationVersion(caps.data, schemaAdcpVersion ?? client.getAdcpVersion())
schemaAdcpVersion ?? client.getAdcpVersion(),
validationVersion =>
validateIncomingResponse('get_adcp_capabilities', caps.data, 'strict', undefined, validationVersion)
);
if (!validation.valid) {
profile.capabilities_schema_issues = validation.issues.map(issue => ({
Expand Down
12 changes: 9 additions & 3 deletions src/lib/testing/compliance/comply.ts
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ import { detectController, hasTestController } from '../test-controller';
import type { ControllerDetection } from '../test-controller';
import { randomBytes } from 'crypto';
import { isPre31AdcpVersion } from '../../utils/adcp-version-config';
import { withExternalSchemaRoot } from '../../validation/schema-loader';
import { withExternalSchemaRoot, withServedReleaseSchemaRoots } from '../../validation/schema-loader';
import { redactOAuthUrlForOutput, redactOAuthUrlsInText } from '../storyboard/oauth-metadata-graph';
import { LIBRARY_VERSION } from '../../version';
import { validationFailsStep } from '../storyboard/validations';
Expand Down Expand Up @@ -2287,7 +2287,13 @@ async function complyImpl(agentUrl: string, options: ComplyOptions): Promise<Com
// not abort discovery or the storyboard currently in flight.
const signal = externalSignal;

return await withExternalSchemaRoot(complianceIndex.adcp_version, scopedSchemaRoot, async () => {
// `servedReleaseSchemaRoots` stays in `testOptions` so routed and nested
// storyboard runs see it too; scoping it here also covers discovery.
return await withExternalSchemaRoot(complianceIndex.adcp_version, scopedSchemaRoot, () =>
withServedReleaseSchemaRoots(testOptions.servedReleaseSchemaRoots, () => complyInSchemaScope())
);

async function complyInSchemaScope(): Promise<ComplianceResult> {
let effectiveOptions: TestOptions = applyAdcpVersionRunOptions(complianceIndex.adcp_version, {
...testOptions,
// `comply()` has always defaulted an unset transport to MCP when it built
Expand Down Expand Up @@ -2725,7 +2731,7 @@ async function complyImpl(agentUrl: string, options: ComplyOptions): Promise<Com
total_duration_ms: Date.now() - start,
notices: noticesDedup,
};
});
}
}

function hasComplyTimeoutBudgetExpired(start: number, timeout_ms: number | undefined, now = Date.now()): boolean {
Expand Down
7 changes: 6 additions & 1 deletion src/lib/testing/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -248,7 +248,12 @@ export { runAgainstLocalAgent } from './local-agent-runner';
export type { LocalAgentRunResult, PerStoryboardOverride, RunAgainstLocalAgentOptions } from './local-agent-runner';

// External schema bundles for hosted compliance/certification runs.
export { registerExternalSchemaRoot, unregisterExternalSchemaRoot, withExternalSchemaRoot } from '../validation';
export {
registerExternalSchemaRoot,
unregisterExternalSchemaRoot,
withExternalSchemaRoot,
withServedReleaseSchemaRoots,
} from '../validation';

// Storyboard-driven testing
export {
Expand Down
17 changes: 11 additions & 6 deletions src/lib/testing/storyboard/runner.ts
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ import {
resolveBundleKey,
schemaAllowsTopLevelField,
withExternalSchemaRoot,
withServedReleaseSchemaRoots,
} from '../../validation/schema-loader';
import { ADCP_VERSION, parseAdcpMajorVersion } from '../../version';
import {
Expand Down Expand Up @@ -1486,12 +1487,14 @@ export async function runStoryboard(
};
}
const schemaRoot = getRunSchemaRoot(options);
if (schemaRoot) {
return await withExternalSchemaRoot(schemaRoot.adcpVersion, schemaRoot.schemaRoot, () =>
const runBody = () =>
withServedReleaseSchemaRoots(options.servedReleaseSchemaRoots, () =>
runStoryboardBody(agentUrlOrUrls, storyboard, options)
);
if (schemaRoot) {
return await withExternalSchemaRoot(schemaRoot.adcpVersion, schemaRoot.schemaRoot, runBody);
}
return await runStoryboardBody(agentUrlOrUrls, storyboard, options);
return await runBody();
},
{ isolate: !hasTestClientScope() }
);
Expand Down Expand Up @@ -5489,12 +5492,14 @@ export async function runStoryboardStep(
options = resolveDeclaredTestKit(storyboard, options);
options = applyNativeA2AComplianceTransportOptions(options);
const schemaRoot = getRunSchemaRoot(options);
if (schemaRoot) {
return await withExternalSchemaRoot(schemaRoot.adcpVersion, schemaRoot.schemaRoot, () =>
const runBody = () =>
withServedReleaseSchemaRoots(options.servedReleaseSchemaRoots, () =>
runStoryboardStepBody(agentUrl, storyboard, stepId, options)
);
if (schemaRoot) {
return await withExternalSchemaRoot(schemaRoot.adcpVersion, schemaRoot.schemaRoot, runBody);
}
return await runStoryboardStepBody(agentUrl, storyboard, stepId, options);
return await runBody();
},
{ isolate: !hasTestClientScope() }
);
Expand Down
47 changes: 44 additions & 3 deletions src/lib/testing/storyboard/validations.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,11 @@ import { prepareResponseForSchemaValidation, TOOL_RESPONSE_SCHEMAS } from '../..
import { injectLegacyEnvelopeStatus } from '../../utils/envelope-status-compat';
import { TRANSPORT_SUFFIX_REGEX } from '../../utils/a2a-discovery';
import { validateResponse, type ValidationIssue } from '../../validation/schema-validator';
import { hasSchemaBundle, isExternalSchemaRootActive } from '../../validation/schema-loader';
import {
hasSchemaBundle,
isExternalSchemaRootActive,
withServedValidationVersion,
} from '../../validation/schema-loader';
import { ADCP_VERSION, LIBRARY_VERSION } from '../../version';
import { isPre31AdcpVersion } from '../../utils/adcp-version-config';
import { gte as semverGte, valid as validSemver } from 'semver';
Expand Down Expand Up @@ -536,6 +540,32 @@ function validateResponseSchema(
validation: StoryboardValidation,
ctx: ValidationContext,
taskResult: TaskResult
): ValidationResult {
// Capabilities are negotiated before any pin applies: a 3.0 pin is served
// the agent's highest release in the major, echoed in `adcp_version`.
// Validate against that release (versioning spec, "Server echoes") when it
// is same-major, listed in the response's `supported_versions` and has a
// schema root, even when an external root is active for the pin. Otherwise
// keep the pin. Every other task stays on the pin.
const pinnedVersion = ctx.adcpVersion;
if (ctx.taskName === 'get_adcp_capabilities' && pinnedVersion !== undefined) {
return withServedValidationVersion(taskResult.data, pinnedVersion, validationVersion =>
validateResponseSchemaForVersion(
validation,
validationVersion === pinnedVersion
? ctx
: { ...ctx, adcpVersion: validationVersion, responseAdcpVersion: validationVersion },
taskResult
)
);
}
return validateResponseSchemaForVersion(validation, ctx, taskResult);
}

function validateResponseSchemaForVersion(
validation: StoryboardValidation,
ctx: ValidationContext,
taskResult: TaskResult
): ValidationResult {
const taskName = ctx.taskName;
const schemaIdentityVersion = ctx.adcpVersion ?? ctx.responseAdcpVersion ?? ADCP_VERSION;
Expand Down Expand Up @@ -689,10 +719,21 @@ function validateResponseSchema(

/** @internal Shared with the storyboard runner's Zod-rejection recovery path. */
export function isExternalResponseSchemaAuthoritative(
ctx: Pick<ValidationContext, 'adcpVersion' | 'responseAdcpVersion'>
ctx: Pick<ValidationContext, 'adcpVersion' | 'responseAdcpVersion'> &
Partial<Pick<ValidationContext, 'taskName' | 'taskResult'>>
): boolean {
const validationVersion = responseSchemaValidationVersion(ctx.adcpVersion, ctx.responseAdcpVersion);
return validationVersion !== undefined && isExternalSchemaRootActive(validationVersion);
if (validationVersion !== undefined && isExternalSchemaRootActive(validationVersion)) return true;
// A capabilities response validated against its echoed release is
// authoritative when that release's root is external (registered, or
// supplied through servedReleaseSchemaRoots), same as a pinned external root.
const pinnedVersion = ctx.adcpVersion;
if (ctx.taskName !== 'get_adcp_capabilities' || pinnedVersion === undefined || !ctx.taskResult) return false;
return withServedValidationVersion(
ctx.taskResult.data,
pinnedVersion,
version => version !== pinnedVersion && isExternalSchemaRootActive(version)
);
}

function responseSchemaValidationVersion(
Expand Down
Loading
Loading