Skip to content

billing_gate_dispatch 1.1.1: all phases optional since #7581, so the runner's vacuous-pass guard fails every seller that passes it #8105

Closed

Description

@LukasGoTom

A storyboard whose phases are all optional can never grade as passing, and billing_gate_dispatch became one in #7581. Example: a seller that advertises supported_billing: ["operator"] passes the two sync_accounts rejection steps and the capability read with every validation green, and still gets billing_gate_dispatch/storyboard: storyboard-level verdict failed.

Summary

#7581 (fixes #7573) marked capability_discovery optional: true so a signals agent without an account block skips it. That was the last required phase. universal/billing-gate-dispatch.yaml 1.1.1 now has six phases and all six are optional.

The storyboard runner computes overall_passed from the required phases only (src/lib/testing/storyboard/runner.ts, same on main at 15.6.0):

const requiredPhaseHasExecutedPass = phaseResults.some((p, idx) => {
  const phaseDef = storyboard.phases[idx];
  if (!phaseDef || phaseDef.optional || !p.passed) return false;   // optional never counts
  return p.steps.some(s => !s.skipped && s.passed);
});
const requiredPhaseDefs = storyboard.phases.filter(phaseDef => !phaseDef.optional);   // []
const requiredPhasesNonFailingSkipped = requiredPhaseDefs.length > 0 && …;            // false
const requiredPhasesCoveredByCapabilityGates = … && requiredPhaseDefs.length > 0 && …; // false
const requiredPhasesPassed =
  !hasExecutableSteps ||
  (requiredPhaseDefs.every(…) && (requiredPhaseHasExecutedPass || requiredPhasesNonFailingSkipped || …)); // false

The comment on that block says the guard exists so "a storyboard where every phase is optional … would [not] pass vacuously". With every phase optional it is unpassable instead.

Why it matters

Hosted grader, bundle 3.2.3, @adcp/sdk 14.2.0, https://dev-demo-mcp.gotom.io/mcp: billing_gate_dispatch 3/4 with one failure and no failing step, which keeps the error_handling track at partial (17/18). Local run against the same bundle with 14.0.0:

phase step result
capability_discovery Read supported_billing passed, 6/6 validations
capability_gate_operator — skipped not_applicable (seller advertises operator)
capability_gate_agent Submit omitted agent billing passed, 8/8 (BILLING_NOT_SUPPORTED, recovery: correctable, details.scope: capability, context echoed)
capability_gate_advertiser Submit omitted advertiser billing passed, 8/8
per_agent_gate_reject — skipped not_applicable (test kit is not passthrough_only)
per_agent_gate_recover — skipped not_applicable
storyboard — failed: "storyboard-level verdict failed"

Steps: 3 passed, 0 failed, 3 skipped. The hosted 3/4 is those three steps plus the synthetic storyboard scenario. Any seller that omits a BillingParty value from supported_billing lands here; a seller that advertises all three skips every gate phase and lands here too.

Ask

Make capability_discovery required again and keep its requires_capability: { path: account, present: true } gate. The runner already passes a required phase that a capability gate skipped (requiredPhasesCoveredByCapabilityGates), so the #7573 case still skips cleanly, and the executed sync_accounts phases count as the storyboard's pass. One-line change plus a patch changeset, 3.1.x backport alongside #7406.

Alternatively, in adcp-client: when requiredPhaseDefs.length === 0, accept an executed pass in any phase instead of failing. Happy to open either PR.

Related: #7573, #7581, #6773, #6907.

Activity

  1. added
    claude-triagingTriage routine is actively working on this issue (1-3 min)
    on Oct 9, 2026
  2. added this to the P0 Bugs milestone on Oct 9, 2026
  3. bokelley commented on Oct 9, 2026

    @bokelley
    Contributor

    Thanks for filing!

    Triage

    Classification: Conformance failure
    Bucket(s): compliance-suite
    Status: ready-to-implement
    Milestone: P0 Bugs

    What the experts said:

    • ad-tech-protocol-expert: Option A is correct — removing optional: true from capability_discovery restores scoring without changing wire behavior or step semantics; the existing requires_capability gate handles agents without account block via requiredPhasesCoveredByCapabilityGates; Option B (runner fix) is unsafe and would silently pass future misconfigured all-optional storyboards; patch-eligible for 3.1.x.
    • code-reviewer: Safe fix; correctness depends on requiredPhasesCoveredByCapabilityGates in adcp-client (unverifiable from this repo but supported by the schema contract); Ready-to-implement not Execute — this reverses a deliberate change from fix(compliance): gate billing capability discovery #7581 which requires explicit judgment.

    My take: Bug confirmed: all 6 phases in billing-gate-dispatch.yaml 1.1.1 have optional: true after #7581, making the storyboard always-fail. Fix is narrow and patch-eligible for both main and 3.1.x; confirm the requiredPhasesCoveredByCapabilityGates path in adcp-client before merging.

    Implementation:

    • File: static/compliance/source/universal/billing-gate-dispatch.yaml line 142
    • Change: remove optional: true from capability_discovery; leave requires_capability: { path: account, present: true } intact
    • The requiredPhasesCoveredByCapabilityGates runner path covers agents without an account block — billing_gate_dispatch: capability_discovery has no required_tools guard, failing signals agents on an account capability they cannot declare #7573 does not re-emerge
    • Add adcontextprotocol: patch changeset; land on main, cherry-pick to 3.1.x
    • Pre-merge: verify requiredPhasesCoveredByCapabilityGates in adcp-client handles a required phase skipped by capability gate
    • Do not pursue Option B — it would mask future all-optional storyboard misconfigurations

    Triaged by Claude Code. Session: https://claude.ai/code/session_01EQpUrgG1RRn8U5SP2L5EQX


    Generated by Claude Code

  4. added
    bugSomething isn't working
    claude-triagedIssue has been triaged by the Claude Code triage routine. Remove to re-triage.
    priority:P0Immediate priority: user/revenue/security/reliability bug to work now
    and removed
    claude-triagingTriage routine is actively working on this issue (1-3 min)
    on Oct 9, 2026
  5. bokelley commented on Oct 16, 2026

    @bokelley
    Contributor

    Verified the released v3.2.4 artifact: universal/billing-gate-dispatch.yaml is storyboard version 1.1.2 and capability_discovery is required (the optional flag is absent), with the account capability gate preserved. The SDK runner retains requiredPhasesCoveredByCapabilityGates for agents without an account block. This ships the fix from #8129, so closing the all-optional storyboard defect. Evidence: https://github.com/adcontextprotocol/adcp/blob/v3.2.4/dist/compliance/3.2.4/universal/billing-gate-dispatch.yaml and #8129.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingclaude-triagedIssue has been triaged by the Claude Code triage routine. Remove to re-triage.community-reportedcompliance-suitepriority:P0Immediate priority: user/revenue/security/reliability bug to work now

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions