Repository navigation
Description
Summary
Strict Spec grading (strict-complete-bundles) is unreachable for a media-buy seller that passes every storyboard it is graded on. It counts a bundle as incomplete whenever the bundle is partial or untested. For our agent, all 19 incomplete bundles are storyboards the runner itself skipped as not applicable to this agent: tools from protocols we don't declare, capabilities we don't advertise, a runtime the runner doesn't have, or storyboards never selected. None of them comes from a failing check.
This is the seller-side counterpart of #8527, which covers signals-only agents.
Evidence
Agent https://sales-agent.elementaltv.io/mcp: media-buy 3.1, specialism sales-non-guaranteed, supported_protocols: ["media_buy"].
Hosted heartbeat 5b0d5c9f-fa8c-474e-84ba-b7d2c42b9b97 (2026-10-27T11:54Z, bundle 3.1.29, SDK 17.0.0-beta.0):
overall_status: passing, "All 6 track(s) pass", 189/189 scenarios, 0 failing or partial storyboards.- Legacy: passing. Strict Spec:
partial("Required evidence is incomplete"). Evidence:relevant_bundle_count: 37,incomplete_bundle_count: 19,failing_bundle_count: 0,relevant_failure_count: 0, no missing bundles.
A local adcp storyboard run --json at the same pins (SDK 17.0.0-beta.0, --compliance-version 3.1.29, sandbox principal) shows why each bundle is incomplete. The step-level skip reasons come from the runner's bundle_results and scenario results:
| Bundle(s) | Runner's reason |
|---|---|
error-compliance-signals, schema-validation-signals, get-signals-pagination-integrity, wholesale-feed-signals |
listed in storyboards_missing_tools / step not_applicable: signals tools; the agent does not declare signals |
collection-lists-pagination-integrity, property-lists-pagination-integrity, content-standards-pagination-integrity |
storyboards_missing_tools: tools of protocols the agent does not declare |
wholesale-feed-products, wholesale-feed-product-webhooks, wholesale-feed-bulk-webhooks, wholesale-feed-signal-webhooks |
step skip capability_unsupported: the agent advertises no wholesale feed |
signed-requests, webhook-receiver-envelope |
step skip not_applicable: request signing and webhooks are not advertised |
webhook-emission |
step skip requirement_unmet: webhook_receiver: "storyboard requires a runtime that is not available on this run" (a runner limitation, not the agent) |
idempotency |
9 steps pass; rate_limit_replay_invariant skips with missing_test_kit_contract (the test kit, not the agent) |
protocol:media-buy |
16 storyboards pass; 31 are untested because they were never selected for this agent (e.g. audience_buy_flow, clicks_buy_flow, reach_buy_flow, frequency_cap_enforcement, inventory_list_*, proposal_finalize*, refine_products, vendor_metric_*) |
buildComplianceBundleResults (@adcp/sdk testing/compliance/comply.js) maps missing-tools storyboards to partial and treats any skip outside NEUTRAL_BUNDLE_SKIP_REASONS as a coverage gap. A storyboard with no result is untested. deriveVerificationProfileRoleAssessments (server/src/services/verification-profile-assessment.ts) then marks Strict partial for any partial or untested relevant bundle. Together, these make every optional capability a seller doesn't offer count against it.
The only way to clear this today would be to implement signals, collection and property lists, content standards, wholesale feeds, webhooks and request signing, plus every optional media-buy flow, in a non-guaranteed CTV seller that has no use for them.
Ask
Make Strict grade evidence "complete" when every storyboard that applies to the agent passes, as Legacy already does for the badge. For example:
- Treat a storyboard as
not_applicable(neutral) when the runner skipped it withmissing_toolfor a protocol the agent does not declare insupported_protocols,capability_unsupported,not_applicable, or a runner-siderequirement_unmet/missing_test_kit_contract. - Exclude protocol-bundle storyboards that were not selected for the agent's declared capabilities and specialisms, instead of counting them as
untested.
Related: #8527 (same gap for signals-only agents), #8506 and #7573 (earlier instances of the same class).
Activity
bokelley commentedon Oct 28, 2026
Pre-RC triage: prioritizing this and #8527 as P1 operational bugs in the active bug queue. #8506 is the related missing-tool prerequisite cascade; #8503 is the owned-signals storyboard defect. #8535 tracks the separate dashboard wording problem.
Implementation brief:
Acceptance: exercise a signals-only agent and a media-buy seller without a creative library, plus controls with a missing required tool and a genuine failed assertion. Verify role/version bundle completeness and the owner/public presentation against the same saved run. The goal is reachable, accurate grading without lowering the conformance floor.
Synaeve commentedon Nov 12, 2026
Two PRs for the gaps that remain on #8534:
buildComplianceBundleResultsapplies the declined-capability rollup to 3.1.24+ bundles, which carry the fix(compliance): gate 3.1 governance scenarios by capability #7625 backport of the "Applicability order" rule, instead of only from 3.2. This clears the declinedmedia-buyscenarios (except those whose gate also involvescompliance_testing.*, tracked in Capability rollup: don't let an AND-composedcompliance_testing.*predicate cap a scenario the agent declined on product grounds adcp-client#3701) and thewholesale_feed_*bundles for 3.1 sellers, once hosted pins a release that includes it.signed_requestsas not applicable when the agent doesn't advertiserequest_signing.supported, and lists incomplete bundle ids in the evidence. It changes no Strict verdict on its own today.The rows for tools of undeclared protocols were already covered by #8824 and #8601. Still open: the hosted runner gaps (webhook receiver, replay receiver, rate-limit test kit), which keep Strict unreachable until the runner supports them or a maintainer scopes
webhook_receiver_envelopeout for sellers.test(compliance): 3.1.29 declined capability gates grade not applicab…
chore(deps): bump @adcp/sdk to 17.0.0-beta.27 and pin hosted prerelea…
bokelley commentedon Nov 15, 2026
Hosted grading now runs
@adcp/sdk17.0.0-beta.27 (deployed 2026-11-15 05:38 UTC). That includes adcontextprotocol/adcp-client#3699, which applies the declined-capability rollup to 3.1.24+ bundles, plus #8952 and #8953.I re-ran your agent locally with beta.27 against the 3.1.33 bundle (unauthenticated, like the heartbeat):
wholesale_feed_*bundles now gradenot_applicableinstead of counting as incomplete.media-buy,sales-non-guaranteed,deterministic-testing,idempotency) come from one seat issue.create_media_buyreturnsaccount-ambiguous: brand domain 'acmeoutdoor.example' matches 2 advertisers in this seat; the operator must deduplicate advertiser domains in Vista, which cascades through 12 storyboards.comply-controller-mode-gatefails discovery withunknown-bearer-token.compliance_testing.*predicate cap a scenario the agent declined on product grounds adcp-client#3701.Once the advertiser duplicate is resolved, your next heartbeat (every 6 hours) should show the Strict picture. If any bundle still counts as incomplete for a reason other than the ones above, please post the run ID here.