Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .changeset/fix-3385-failed-preparation-evidence.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
---
'@adcp/sdk': patch
---

Preserve producer evidence when prepared fixture discovery fails (#3385).

Authorized full and standalone reuse now retain fixture-handle resolutions and a `fixture_preparation` diagnostic phase with the producer's errors and actual exchanges. The historical phase is outside graded `phases`, preserving consumer and JUnit failure counts. Protected captures are stripped from its public contexts; denied callers still receive only `PreparedFixturesDeniedError`, while the private audit sink retains failed producer evidence.

Discovery failures identify the attempted task and configured transport. SDK connection errors are never represented as seller responses, protocol rejections remain protocol failures, and malformed seller pages retain a schema diagnostic against the actual received payload before SDK projection. The terminal failed page is retained even beyond the bounded successful-page evidence history. Captured text payloads redact credential fields.
11 changes: 11 additions & 0 deletions docs/guides/COLLECTION-FIXTURE-FOUNDATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -717,6 +717,17 @@ Until a companion `adcontextprotocol/adcp` PR lands all six, no *published*
storyboard can legally use any of this. That is a merge dependency, not an
implementation gap — and it is why the downgrade story below matters.

When a prepared fixture graph failed discovery, authorized reuse retains its
producer phase as `fixture_preparation` and its handle records as
`fixture_resolutions`. The producer phase is diagnostic history outside graded
`phases`, so it does not add JUnit failures or change consumer grading counts.
Protected capture keys are stripped from its public step contexts; a private
audit sink receives the complete failed producer record after a denial. Failed
discovery responses retain the official client's protocol response when available;
successful-page evidence retains parsed SDK data. HTTP-only A2A evidence includes
its status and an extraction note when the bounded capture was truncated. SDK
connection diagnostics never stand in for a seller response.

### Output inventory for the amendment

Everything a report consumer can newly observe, with where it is emitted and
Expand Down
2 changes: 2 additions & 0 deletions src/lib/protocols/mcp.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1450,6 +1450,7 @@ export async function callMCPToolWithOAuth(options: MCPCallOptions): Promise<unk
debugLogs.push({
type: modernAttempt.response?.isError ? 'error' : 'success',
message: `MCP: Tool ${toolName} response received`,
response: modernAttempt.response,
timestamp: new Date().toISOString(),
});
return modernAttempt.response;
Expand Down Expand Up @@ -1480,6 +1481,7 @@ export async function callMCPToolWithOAuth(options: MCPCallOptions): Promise<unk
debugLogs.push({
type: response?.isError ? 'error' : 'success',
message: `MCP: Tool ${toolName} response received`,
response,
timestamp: new Date().toISOString(),
});

Expand Down
26 changes: 21 additions & 5 deletions src/lib/protocols/rawResponseCapture.ts
Original file line number Diff line number Diff line change
Expand Up @@ -280,12 +280,28 @@ async function readBodyBounded(
response: Response,
maxBodyBytes: number
): Promise<{ body: string; bodyTruncated: boolean }> {
let text: string;
if (!response.body) return { body: '', bodyTruncated: false };
const reader = response.body.getReader();
const decoder = new TextDecoder();
let text = '';
let complete = false;
try {
text = await response.text();
while (true) {
const chunk = await reader.read();
if (chunk.done) {
text += decoder.decode();
complete = true;
return { body: text.slice(0, maxBodyBytes), bodyTruncated: text.length > maxBodyBytes };
}
text += decoder.decode(chunk.value, { stream: true });
if (text.length > maxBodyBytes) return { body: text.slice(0, maxBodyBytes), bodyTruncated: true };
}
} catch {
return { body: '', bodyTruncated: false };
return { body: text, bodyTruncated: true };
} finally {
// A cloned response is a tee. Waiting for cancellation here can deadlock
// until the original body is consumed, which happens only after fetch returns.
if (!complete) void reader.cancel().catch(() => {});
reader.releaseLock();
}
if (text.length <= maxBodyBytes) return { body: text, bodyTruncated: false };
return { body: text.slice(0, maxBodyBytes), bodyTruncated: true };
}
139 changes: 130 additions & 9 deletions src/lib/testing/storyboard/runner.ts
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@ import {
preparedBindingRegistryFor,
PreparedStoryboardFixtures,
resolvePreparedFixtures,
isTrustedPreparedFixtures,
type PreparedFixtureBinding,
type PreparedFixturesDisposition,
type PreparedFixturesRefusal,
Expand Down Expand Up @@ -4816,12 +4817,7 @@ async function executeStoryboardPass(
const disposition = prepared.state.disposition;
if (disposition.status === 'failed') {
if (!callerOwnsClients) await closeScopedConnections(options.protocol);
return buildDiscoveryFailedResult(agentUrls, storyboard, {
step: 'Reuse prepared fixture state',
passed: false,
duration_ms: 0,
error: disposition.detail,
});
return buildPreparationFailedRunResult(agentUrls, storyboard, prepared.state, disposition.detail);
}
if (disposition.status === 'unsatisfied') {
fixtureUnsatisfied = true;
Expand Down Expand Up @@ -6956,7 +6952,16 @@ async function emitPreparedFixturesAudit(
const sink = options.prepared_fixtures_audit_sink;
if (!sink) return;
try {
await sink(record);
const state = options.prepared_fixtures;
const auditRecord =
isTrustedPreparedFixtures(state) && state.disposition.status === 'failed'
? {
...record,
fixture_resolutions: state.resolutions,
...(state.phase && { fixture_preparation: state.phase }),
}
: record;
await sink(auditRecord);
} catch {
// Intentionally swallowed; see above.
}
Expand Down Expand Up @@ -7060,13 +7065,111 @@ function buildPreparedRefusalStepResult(
};
}

/** Historical producer evidence does not add graded steps to the consumer run. */
function preparedPhaseForOutput(phase: StoryboardPhaseResult): StoryboardPhaseResult {
return { ...phase, steps: phase.steps.map(step => ({ ...step, context: withoutProtectedCaptures(step.context) })) };
}

/**
* Result for a full run reusing a prepared state that FAILED preparation.
*
* Not `buildDiscoveryFailedResult`: that reports agent capability
* discovery, and reusing a failed preparation is a different failure with
* different evidence. Saying `discovery_failed` with task `''` and no
* validations sent an implementor looking at their transport and auth
* when the actual failure was product discovery, and it dropped the
* producer records the preparation had already collected.
*
* The producer phase is retained separately as diagnostic evidence, with
* protected captures stripped from context. Its errors and exchanges survive,
* and `fixture_resolutions`
* carries the same records `prepared.resolutions` holds. Counts and
* failure meaning are unchanged: this is a failure, never a skip, and
* never `fixture_unsatisfied`.
*/
function buildPreparationFailedRunResult(
agentUrls: string[],
storyboard: Storyboard,
state: PreparedStoryboardFixtures,
detail: string
): StoryboardResult {
const resolutions = state.resolutions;
const producerPhase = state.phase;
// The task that actually failed, read off the producer's own record
// rather than guessed. Absent only when preparation failed before any
// step was recorded.
const failingStep = producerPhase?.steps.find(step => step.passed === false);
const syntheticStep: StoryboardStepResult = {
storyboard_id: storyboard.id,
step_id: 'prepared_fixtures_preparation_failed',
phase_id: 'prepared_fixtures_preparation_failed',
title: 'Storyboard failed: fixture preparation did not resolve',
task: failingStep?.task ?? '',
passed: false,
skipped: false,
duration_ms: 0,
validations: [
{
check: 'fixture_capture_unbuildable',
passed: false,
description: 'Fixture preparation must resolve before a run can reuse it',
error: detail,
json_pointer: null,
expected: 'a fixture graph that resolved during preparation',
actual: 'preparation_failed',
schema_id: null,
schema_url: null,
// Pre-wire for THIS run: the consumer never dispatched. The
// producer's own exchange is retained in fixture_preparation.
request: null,
response: null,
},
],
context: {},
extraction: { path: 'none' },
error: detail,
};
const phases: StoryboardPhaseResult[] = [
{
phase_id: 'prepared_fixtures_preparation_failed',
phase_title: 'Prepared fixture preparation failed',
passed: false,
steps: [syntheticStep],
duration_ms: 0,
},
];
return {
storyboard_id: storyboard.id,
storyboard_title: storyboard.title,
agent_url: redactOAuthUrlForOutput(agentUrls[0]!),
agent_urls: agentUrls.map(redactOAuthUrlForOutput),
overall_passed: false,
...(resolutions.length > 0 && { fixture_resolutions: resolutions }),
...(state.coverageGaps.length > 0 && { coverage_gaps: state.coverageGaps }),
phases,
...(producerPhase && { fixture_preparation: preparedPhaseForOutput(producerPhase) }),
context: {},
total_duration_ms: 0,
passed_count: 0,
failed_count: 1,
skipped_count: 0,
runner_capability_version: RUNNER_CAPABILITY_VERSION,
tested_at: new Date().toISOString(),
strict_validation_summary: summarizeStrictValidation(phases),
notices: [],
};
}

/** Terminal step result carrying a non-resolved preparation disposition. */
function buildPreparedDispositionStepResult(
storyboard: Storyboard,
found: { step: StoryboardStep; phaseId: string },
context: StoryboardContext,
disposition: PreparedFixturesDisposition,
contributions: Iterable<string>
contributions: Iterable<string>,
/** Producer records to republish; a standalone result is the whole run. */
resolutions: readonly FixtureResolutionRecord[] = [],
producerPhase?: StoryboardPhaseResult
): StoryboardStepResult {
const detail = 'detail' in disposition ? disposition.detail : 'fixture preparation did not resolve';
if (disposition.status === 'failed') {
Expand Down Expand Up @@ -7096,8 +7199,18 @@ function buildPreparedDispositionStepResult(
actual: 'preparation_failed',
schema_id: null,
schema_url: null,
// Pre-wire for THIS step: the consumer never dispatched. The
// producer's own exchange rides on its records below.
request: null,
response: null,
},
],
// The producer's own records, so a standalone consumer sees the same
// fixture-handle evidence a full run carries. It had none before:
// the caller was told preparation failed with no way to see which
// handle, which strategy, or what the seller actually returned.
...(resolutions.length > 0 && { fixture_resolutions: [...resolutions] }),
...(producerPhase && { fixture_preparation: preparedPhaseForOutput(producerPhase) }),
context: withoutProtectedCaptures(context),
extraction: { path: 'none' },
contributions: Array.from(contributions),
Expand Down Expand Up @@ -7462,7 +7575,15 @@ async function runStoryboardStepBody(
// A terminal preparation outcome is carried forward verbatim. A
// retry must not be able to launder a failed graph into a freshly
// discovered alternative.
return buildPreparedDispositionStepResult(storyboard, found, context, disposition, contributions);
return buildPreparedDispositionStepResult(
storyboard,
found,
context,
disposition,
contributions,
prepared.state.resolutions,
prepared.state.phase
);
}
const hydrationViolation = hydratePreparedCaptures(prepared.state, context, protectedWriter, contextProvenance);
if (hydrationViolation) {
Expand Down
Loading
Loading