Skip to content

Strict Spec grading unreachable for media-buy sellers: runner-skipped storyboards count as incomplete bundles #8534

Open

Description

@idemidenko

Summary

Strict Spec grading (strict-complete-bundles) is unreachable for a media-buy seller that passes every storyboard it is graded on. It counts a bundle as incomplete whenever the bundle is partial or untested. For our agent, all 19 incomplete bundles are storyboards the runner itself skipped as not applicable to this agent: tools from protocols we don't declare, capabilities we don't advertise, a runtime the runner doesn't have, or storyboards never selected. None of them comes from a failing check.

This is the seller-side counterpart of #8527, which covers signals-only agents.

Evidence

Agent https://sales-agent.elementaltv.io/mcp: media-buy 3.1, specialism sales-non-guaranteed, supported_protocols: ["media_buy"].

Hosted heartbeat 5b0d5c9f-fa8c-474e-84ba-b7d2c42b9b97 (2026-10-27T11:54Z, bundle 3.1.29, SDK 17.0.0-beta.0):

  • overall_status: passing, "All 6 track(s) pass", 189/189 scenarios, 0 failing or partial storyboards.
  • Legacy: passing. Strict Spec: partial ("Required evidence is incomplete"). Evidence: relevant_bundle_count: 37, incomplete_bundle_count: 19, failing_bundle_count: 0, relevant_failure_count: 0, no missing bundles.

A local adcp storyboard run --json at the same pins (SDK 17.0.0-beta.0, --compliance-version 3.1.29, sandbox principal) shows why each bundle is incomplete. The step-level skip reasons come from the runner's bundle_results and scenario results:

Bundle(s) Runner's reason
error-compliance-signals, schema-validation-signals, get-signals-pagination-integrity, wholesale-feed-signals listed in storyboards_missing_tools / step not_applicable: signals tools; the agent does not declare signals
collection-lists-pagination-integrity, property-lists-pagination-integrity, content-standards-pagination-integrity storyboards_missing_tools: tools of protocols the agent does not declare
wholesale-feed-products, wholesale-feed-product-webhooks, wholesale-feed-bulk-webhooks, wholesale-feed-signal-webhooks step skip capability_unsupported: the agent advertises no wholesale feed
signed-requests, webhook-receiver-envelope step skip not_applicable: request signing and webhooks are not advertised
webhook-emission step skip requirement_unmet: webhook_receiver: "storyboard requires a runtime that is not available on this run" (a runner limitation, not the agent)
idempotency 9 steps pass; rate_limit_replay_invariant skips with missing_test_kit_contract (the test kit, not the agent)
protocol:media-buy 16 storyboards pass; 31 are untested because they were never selected for this agent (e.g. audience_buy_flow, clicks_buy_flow, reach_buy_flow, frequency_cap_enforcement, inventory_list_*, proposal_finalize*, refine_products, vendor_metric_*)

buildComplianceBundleResults (@adcp/sdk testing/compliance/comply.js) maps missing-tools storyboards to partial and treats any skip outside NEUTRAL_BUNDLE_SKIP_REASONS as a coverage gap. A storyboard with no result is untested. deriveVerificationProfileRoleAssessments (server/src/services/verification-profile-assessment.ts) then marks Strict partial for any partial or untested relevant bundle. Together, these make every optional capability a seller doesn't offer count against it.

The only way to clear this today would be to implement signals, collection and property lists, content standards, wholesale feeds, webhooks and request signing, plus every optional media-buy flow, in a non-guaranteed CTV seller that has no use for them.

Ask

Make Strict grade evidence "complete" when every storyboard that applies to the agent passes, as Legacy already does for the badge. For example:

  1. Treat a storyboard as not_applicable (neutral) when the runner skipped it with missing_tool for a protocol the agent does not declare in supported_protocols, capability_unsupported, not_applicable, or a runner-side requirement_unmet / missing_test_kit_contract.
  2. Exclude protocol-bundle storyboards that were not selected for the agent's declared capabilities and specialisms, instead of counting them as untested.

Related: #8527 (same gap for signals-only agents), #8506 and #7573 (earlier instances of the same class).

Activity

  1. added
    bugSomething isn't working
    priority:P1High priority: important bug or follow-up, not immediate P0
    on Oct 28, 2026
  2. added this to the P0 Bugs milestone on Oct 28, 2026
  3. bokelley commented on Oct 28, 2026

    @bokelley
    Contributor

    Pre-RC triage: prioritizing this and #8527 as P1 operational bugs in the active bug queue. #8506 is the related missing-tool prerequisite cascade; #8503 is the owned-signals storyboard defect. #8535 tracks the separate dashboard wording problem.

    Implementation brief:

    • Determine applicability from declared protocols/capabilities and the storyboards actually selected for the claimed role and version. A missing tool for an unrelated protocol must not make that protocol required.
    • Carry legitimate not-applicable results through the SDK bundle assessment and the hosted Strict Spec assessment consistently.
    • Preserve the distinction between an inapplicable check, missing required evidence and a real failure. Runner limitations and missing test-kit support must not silently turn required but unexecuted checks into passes.
    • Keep actual assertion/schema failures blocking. A declared protocol that lacks a required tool must still be incomplete or failing.

    Acceptance: exercise a signals-only agent and a media-buy seller without a creative library, plus controls with a missing required tool and a genuine failed assertion. Verify role/version bundle completeness and the owner/public presentation against the same saved run. The goal is reachable, accurate grading without lowering the conformance floor.

  4. Synaeve commented on Nov 12, 2026

    @Synaeve
    Contributor

    Two PRs for the gaps that remain on #8534:

    The rows for tools of undeclared protocols were already covered by #8824 and #8601. Still open: the hosted runner gaps (webhook receiver, replay receiver, rate-limit test kit), which keep Strict unreachable until the runner supports them or a maintainer scopes webhook_receiver_envelope out for sellers.

  5. bokelley commented on Nov 15, 2026

    @bokelley
    Contributor

    Hosted grading now runs @adcp/sdk 17.0.0-beta.27 (deployed 2026-11-15 05:38 UTC). That includes adcontextprotocol/adcp-client#3699, which applies the declined-capability rollup to 3.1.24+ bundles, plus #8952 and #8953.

    I re-ran your agent locally with beta.27 against the 3.1.33 bundle (unauthenticated, like the heartbeat):

    • Fixed: the five wholesale_feed_* bundles now grade not_applicable instead of counting as incomplete.
    • On your side: most of the five failing bundles (media-buy, sales-non-guaranteed, deterministic-testing, idempotency) come from one seat issue. create_media_buy returns account-ambiguous: brand domain 'acmeoutdoor.example' matches 2 advertisers in this seat; the operator must deduplicate advertiser domains in Vista, which cascades through 12 storyboards. comply-controller-mode-gate fails discovery with unknown-bearer-token.
    • Still open on our side: webhook emission and webhook receiver are runner gaps. The remaining capability-gate case is Capability rollup: don't let an AND-composed compliance_testing.* predicate cap a scenario the agent declined on product grounds adcp-client#3701.

    Once the advertiser duplicate is resolved, your next heartbeat (every 6 hours) should show the Strict picture. If any bundle still counts as incomplete for a reason other than the ones above, please post the run ID here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingcompliance-suitepriority:P1High priority: important bug or follow-up, not immediate P0

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions