From 1aa5c05bce6f33e1ed185b97b51313ab2f6d1150 Mon Sep 17 00:00:00 2001 From: Brian O'Kelley Date: Sun, 23 Aug 2026 09:59:44 +0200 Subject: [PATCH 1/3] fix(compliance): drop get_adcp_capabilities from required_tools across 17 storyboards (OR-gate bug) (#6774) * fix(compliance): drop get_adcp_capabilities from required_tools in billing_gate_dispatch and agent_notification_configs required_tools uses OR semantics in the adcp-client SDK. Listing get_adcp_capabilities alongside a capability-specific tool made the storyboard-level gate trivially satisfied for every conformant agent, causing billing_gate_dispatch and agent_notification_configs to run against agents that lack sync_accounts / sync_agent_notification_configs and fail at the first assertion instead of receiving a clean coverage-gap skip. Closes #6773 Co-Authored-By: Claude Sonnet 4.6 Claude-Session: https://claude.ai/code/session_01M3evZsz4Y3971B5RZVxhKh * fix(compliance): sweep remaining 15 storyboards for get_adcp_capabilities OR-gate bug Same pattern as billing_gate_dispatch (issue #6773): required_tools uses OR semantics, so listing get_adcp_capabilities alongside a capability- specific tool made the gate trivially satisfied for every conformant agent. Affected: billing_out_of_band, canonical_supported_formats, evaluator_auth, native_localization (creative); account_timezone_{seller_fixed,buyer_selected, seller_assigned}, budget_cap_timezone_{account,fixed,override_rejected}, geo_{region_all_values_support,region_targeting,region_exclusion_support, place_targeting}, portfolio_routing_scope (media-buy). Co-Authored-By: Claude Sonnet 4.6 Claude-Session: https://claude.ai/code/session_01M3evZsz4Y3971B5RZVxhKh --------- Co-authored-by: Claude (cherry picked from commit a1e72ec620e15e2f86f0f8760745edcc9a8d4fd1) --- .changeset/fix-storyboard-required-tools-or-gate.md | 7 +++++++ .../protocols/creative/scenarios/billing_out_of_band.yaml | 1 - .../creative/scenarios/canonical_supported_formats.yaml | 1 - .../protocols/creative/scenarios/evaluator_auth.yaml | 1 - .../compliance/source/universal/billing-gate-dispatch.yaml | 1 - 5 files changed, 7 insertions(+), 4 deletions(-) create mode 100644 .changeset/fix-storyboard-required-tools-or-gate.md diff --git a/.changeset/fix-storyboard-required-tools-or-gate.md b/.changeset/fix-storyboard-required-tools-or-gate.md new file mode 100644 index 00000000000..891097569fc --- /dev/null +++ b/.changeset/fix-storyboard-required-tools-or-gate.md @@ -0,0 +1,7 @@ +--- +"adcontextprotocol": patch +--- + +Fix 17 compliance storyboards that incorrectly included `get_adcp_capabilities` in `required_tools` alongside capability-specific tools. Because `required_tools` uses OR semantics, listing a universal tool made the storyboard-level gate trivially satisfied for every conformant agent — agents lacking the actual capability tool (e.g. `sync_accounts`, `build_creative`, `get_products`) would enter the storyboard and fail at the first capability-specific step instead of receiving a clean coverage-gap skip. + +Affected storyboards: `billing_gate_dispatch`, `agent_notification_configs`, and 15 scenarios across the `media-buy` and `creative` protocol families. diff --git a/static/compliance/source/protocols/creative/scenarios/billing_out_of_band.yaml b/static/compliance/source/protocols/creative/scenarios/billing_out_of_band.yaml index ca3e17d2f8d..6c3c65dc35b 100644 --- a/static/compliance/source/protocols/creative/scenarios/billing_out_of_band.yaml +++ b/static/compliance/source/protocols/creative/scenarios/billing_out_of_band.yaml @@ -6,7 +6,6 @@ summary: "Verifies that a creative agent declaring bills_through_adcp: false rej track: creative introduced_in: "3.1" required_tools: - - get_adcp_capabilities - report_usage narrative: | diff --git a/static/compliance/source/protocols/creative/scenarios/canonical_supported_formats.yaml b/static/compliance/source/protocols/creative/scenarios/canonical_supported_formats.yaml index 5e8885c1cd9..a108a465039 100644 --- a/static/compliance/source/protocols/creative/scenarios/canonical_supported_formats.yaml +++ b/static/compliance/source/protocols/creative/scenarios/canonical_supported_formats.yaml @@ -6,7 +6,6 @@ summary: "Verifies the 3.1 creative-agent canonical path: targetable supported_f track: creative introduced_in: "3.1" required_tools: - - get_adcp_capabilities - build_creative narrative: | diff --git a/static/compliance/source/protocols/creative/scenarios/evaluator_auth.yaml b/static/compliance/source/protocols/creative/scenarios/evaluator_auth.yaml index 085878f8840..79dff163329 100644 --- a/static/compliance/source/protocols/creative/scenarios/evaluator_auth.yaml +++ b/static/compliance/source/protocols/creative/scenarios/evaluator_auth.yaml @@ -6,7 +6,6 @@ summary: "Verifies that build_creative evaluator pointers use the verifier allow track: creative introduced_in: "3.1" required_tools: - - get_adcp_capabilities - build_creative narrative: | diff --git a/static/compliance/source/universal/billing-gate-dispatch.yaml b/static/compliance/source/universal/billing-gate-dispatch.yaml index 558cb3e07c9..72174a44c57 100644 --- a/static/compliance/source/universal/billing-gate-dispatch.yaml +++ b/static/compliance/source/universal/billing-gate-dispatch.yaml @@ -5,7 +5,6 @@ category: error_compliance summary: "Validates the two distinct billing-rejection paths on sync_accounts — seller-wide capability gate (BILLING_NOT_SUPPORTED) and per-buyer-agent commercial-relationship gate (BILLING_NOT_PERMITTED_FOR_AGENT) — and the clamped error.details shape that prevents the per-agent code from acting as a commercial-state oracle." track: error_handling required_tools: - - get_adcp_capabilities - sync_accounts narrative: | From c2b65ed183e6cedad17c9f50baa8f39bc9535fd3 Mon Sep 17 00:00:00 2001 From: evgen Date: Thu, 10 Sep 2026 08:11:57 -0400 Subject: [PATCH 2/3] fix(compliance): drop comply_test_controller from canonical_format_validate_input required_tools (OR-gate) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The per-storyboard gate (runner.js) admits a storyboard when any listed tool is present, while the suite selector (comply.js) requires all of them. With comply_test_controller in the list, every agent exposing the universal test controller was selected for this storyboard and then failed all 17 steps on the missing validate_input — on the hosted grader, 17 spurious failures on each of two registered agents (#7404). Same shape as #6774. Refs #7404 (bug 1 only). (cherry picked from commit 40c59b1995810ce4cdd239f3a023e2651e7e0ff1) --- .changeset/canonical-format-validate-input-required-tools.md | 5 +++++ .../source/universal/canonical-format-validate-input.yaml | 1 - 2 files changed, 5 insertions(+), 1 deletion(-) create mode 100644 .changeset/canonical-format-validate-input-required-tools.md diff --git a/.changeset/canonical-format-validate-input-required-tools.md b/.changeset/canonical-format-validate-input-required-tools.md new file mode 100644 index 00000000000..766804d7109 --- /dev/null +++ b/.changeset/canonical-format-validate-input-required-tools.md @@ -0,0 +1,5 @@ +--- +"adcontextprotocol": patch +--- + +`canonical_format_validate_input` no longer lists `comply_test_controller` in `required_tools`. The storyboard runner's per-storyboard gate admits a storyboard when any listed tool is present, so any agent exposing the (universal) test controller was selected and then failed all 17 steps on the missing `validate_input`. Agents without `validate_input` now receive a coverage-gap skip; agents implementing it run unchanged. Same shape as #6774 (fixes #7404, bug 1). diff --git a/static/compliance/source/universal/canonical-format-validate-input.yaml b/static/compliance/source/universal/canonical-format-validate-input.yaml index 62051ec3800..caa8a15a710 100644 --- a/static/compliance/source/universal/canonical-format-validate-input.yaml +++ b/static/compliance/source/universal/canonical-format-validate-input.yaml @@ -7,7 +7,6 @@ track: creative introduced_in: "3.1" required_tools: - validate_input - - comply_test_controller requires: - controller From 36d33b2ef96ef2341cfb1685b0567438522afbad Mon Sep 17 00:00:00 2001 From: Brian O'Kelley Date: Sun, 13 Sep 2026 12:45:56 +0000 Subject: [PATCH 3/3] fix(compliance): correct 3.1 backport changeset scope Name only the four #6774 storyboards present on 3.1.x and keep the remaining hosted service work open. Refs #7404 (bug 1 only). --- .changeset/canonical-format-validate-input-required-tools.md | 2 +- .changeset/fix-storyboard-required-tools-or-gate.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.changeset/canonical-format-validate-input-required-tools.md b/.changeset/canonical-format-validate-input-required-tools.md index 766804d7109..a8563f1e450 100644 --- a/.changeset/canonical-format-validate-input-required-tools.md +++ b/.changeset/canonical-format-validate-input-required-tools.md @@ -2,4 +2,4 @@ "adcontextprotocol": patch --- -`canonical_format_validate_input` no longer lists `comply_test_controller` in `required_tools`. The storyboard runner's per-storyboard gate admits a storyboard when any listed tool is present, so any agent exposing the (universal) test controller was selected and then failed all 17 steps on the missing `validate_input`. Agents without `validate_input` now receive a coverage-gap skip; agents implementing it run unchanged. Same shape as #6774 (fixes #7404, bug 1). +`canonical_format_validate_input` no longer lists `comply_test_controller` in `required_tools`. The storyboard runner's per-storyboard gate admits a storyboard when any listed tool is present, so any agent exposing the (universal) test controller was selected and then failed all 17 steps on the missing `validate_input`. Agents without `validate_input` now receive a coverage-gap skip; agents implementing it run unchanged. Same shape as #6774 (Refs #7404, bug 1 only). diff --git a/.changeset/fix-storyboard-required-tools-or-gate.md b/.changeset/fix-storyboard-required-tools-or-gate.md index 891097569fc..9c14d2a8d9a 100644 --- a/.changeset/fix-storyboard-required-tools-or-gate.md +++ b/.changeset/fix-storyboard-required-tools-or-gate.md @@ -2,6 +2,6 @@ "adcontextprotocol": patch --- -Fix 17 compliance storyboards that incorrectly included `get_adcp_capabilities` in `required_tools` alongside capability-specific tools. Because `required_tools` uses OR semantics, listing a universal tool made the storyboard-level gate trivially satisfied for every conformant agent — agents lacking the actual capability tool (e.g. `sync_accounts`, `build_creative`, `get_products`) would enter the storyboard and fail at the first capability-specific step instead of receiving a clean coverage-gap skip. +Fix the four compliance storyboards present on the 3.1 maintenance line that incorrectly included `get_adcp_capabilities` in `required_tools` alongside capability-specific tools: `billing_gate_dispatch`, `billing_out_of_band`, `canonical_supported_formats`, and `evaluator_auth`. -Affected storyboards: `billing_gate_dispatch`, `agent_notification_configs`, and 15 scenarios across the `media-buy` and `creative` protocol families. +Removing the universal capability-discovery tool prevents agents without the storyboard's capability-specific tools from entering through the per-storyboard OR gate. This backports the applicable subset of #6774; the other 13 storyboards changed on main do not exist on this line.