Skip to content

Hosted grader: 11 failing steps remain on two cards after #7454/#7405/#7406 (sales 10, signals 1); budget-truncated runs still published as complete grades #7404

Closed

Description

@EvgenyAndroid

I co-lead the Signals and Measurement working group. Filed as an operator of two registered agents reading the grader's output; no WG position attaches.

Two registered agents, both graded on line 3.1.20, both showing "degraded" for reasons that do not reproduce with the published runner against the same production endpoints. Registry feeds (public, no auth):

  • https://agenticadvertising.org/api/registry/agents/https%3A%2F%2Fsell.nofluffadvisory.com%2Fmcp/compliance — media-buy seller, heartbeat 2026-09-10T00:38:51Z, 159/193 scenarios, 13/35 storyboards
  • https://agenticadvertising.org/api/registry/agents/https%3A%2F%2Fadcp.signal-stack.io%2Fmcp/compliance — signals agent, heartbeat 2026-09-10T00:35:07Z, 66/68 scenarios, 9/35 storyboards, verified

Summing failure_count over storyboard_statuses: 25 failing steps on the sales card, 37 on the signals card, 62 total. 53 of them fall into the two classes below. None of the 53 can be made to pass by any change to either agent.

1. required_tools is an OR gate in the per-storyboard runner and an AND gate in the suite selector

universal/canonical-format-validate-input.yaml (3.1.20 and 3.2.0-rc.1 alike):

required_tools:
  - validate_input
  - comply_test_controller

requires:
  - controller

Neither agent exposes validate_input (anonymous tools/list returns 11 tools on each; the runner's own discovery observation on both cards lists none). Both expose comply_test_controller. Result on the cards: signals 0/17 failing, sales 1/18 with 17 failing steps. Every one of those steps calls a tool that does not exist.

The published runner disagrees with itself about what required_tools means. @adcp/sdk@13.0.2, dist/lib/testing/compliance/comply.js:896 (suite selection, what storyboard run / comply use):

const missing = (sb.required_tools ?? []).filter((t) => !discoveredToolNames.has(t));
if (missing.length > 0) {
  missingToolStoryboards.push({ ..., reason: `missing required_tools: ${missing.join(", ")}` });

Any missing tool excludes the storyboard. Locally that is what happens: the storyboard is absent from the 61-storyboard selection for the sales agent and listed under storyboards.missing_tools for the signals agent.

dist/lib/testing/storyboard/runner.js:1421 (per-storyboard gate):

const hasAnyRequired = storyboard.required_tools.some((t) => options.agentTools.includes(t));

Any present tool admits it. The hosted grader evidently runs the second path: it selected 75 storyboards for the sales agent (its own budget message says so, see 3) against the selector's 61, and canonical_format_validate_input is among the extra 14.

This is the #6773 shape. PR #6774 (a1e72ec, 3.2.0-beta.6) fixed it for get_adcp_capabilities by removing that tool from 17 storyboards' required_tools. It did not touch this storyboard, whose gate is still satisfiable by comply_test_controller alone, and it is not on 3.1.x: billing-gate-dispatch.yaml there still lists get_adcp_capabilities, which is why that storyboard reads 1/2 on the signals card (its sync_accounts step fails on an agent with no accounts protocol).

2. Steps the SDK marks not_applicable are graded as failures on the hosted runner

Signals card: deterministic_testing 0/7 failing, idempotency 0/11 failing. Neither is a gate case. deterministic-testing.yaml requires only comply_test_controller, which the agent has; idempotency.yaml has no required_tools at all. Both are legitimately selected. Their media-buy steps (accounts, media buys, creatives, delivery, budget) cannot apply to a signals-only agent.

Locally, @adcp/sdk@13.0.2 storyboard run against the same production endpoint on 2026-09-10 marks those steps not_applicable and the storyboards pass: 57/57 scenarios, 32/108 steps passed, 0 failed, 76 skipped, both storyboards in the executed set. The hosted runner fails every step in both.

I can see the difference in outcome but not the cause. Candidates: a different runner build on the hosted side, or a configuration that disables step-level applicability. Which one it is decides whether this is a bug or a policy.

3. The 600 s budget truncated a run and the card published the truncated result

Sales card, heartbeat 2026-09-09T15:46:21Z: total_duration_ms 638171 against the neighbouring 3.1.20 heartbeats at 150.5, 221.1, 182.0 and 219.7 s. The run's observations carried:

Compliance timeout budget of 600000ms was reached. Stopped starting new storyboards after 50/75 selected storyboard(s).

For that cycle the card read 2/13 storyboards and 130 scenarios: every storyboard the run never reached dropped out of the denominator, and 11 that had been passing at 01:41Z vanished with them. The next heartbeat (00:38Z) restored 13/35. Nothing changed on the agent between those runs. Normal runs sit at 25–37% of the budget, so this is one slow cycle, but the card presented it as a grade.

Asks

  1. Make the two gates agree. Either the per-storyboard runner requires every listed tool, as the selector does, or required_tools is documented as OR and storyboards that need a specific tool declare it another way. Whichever way, canonical_format_validate_input should not be selectable for an agent without validate_input.
  2. Backport, or say that 3.1.x will not get, fix(compliance): drop get_adcp_capabilities from required_tools across 17 storyboards (OR-gate bug) #6774 and whatever fixes 1. Both cards grade on 3.1.20 today.
  3. State what the hosted runner does with step-level not_applicable, and if it differs from the published SDK, which build or setting produces the difference.
  4. When a run hits the budget, keep the previous complete result on the card (or mark the run partial) instead of publishing the truncated denominator.

Reproduce

# published runner, same line as the cards
npx @adcp/sdk@13.0.2 storyboard run https://sell.nofluffadvisory.com/mcp --compliance-version 3.1.20 --timeout 300
npx @adcp/sdk@13.0.2 storyboard run https://adcp.signal-stack.io/mcp --compliance-version 3.1.20 --timeout 300

Sales: 228 passed / 1 failed / 76 skipped, the 1 being a deliberate provenance decline; canonical_format_validate_input not selected. Signals: 57/57 scenarios, 0 failed; deterministic_testing and idempotency pass with their media-buy steps not_applicable. Not @adcp/sdk@13.0.3: it fails every comply_test_controller response-schema check under storyboard run (adcp-client#2867), which would mask the second class.

The 9 failing steps on the two cards that are not in these classes (8 on sales across deterministic_testing, idempotency, get_media_buys_pagination_integrity, stale_response_advisory; 1 on signals in error_compliance_signals) also do not reproduce locally, but I have no step text for them: the feed returns first_failed_step_id and first_failure_message as null on all 35 storyboards of both cards. Separate question; happy to take it elsewhere if the feed is not meant to expose them.

Activity

  1. added
    claude-triagingTriage routine is actively working on this issue (1-3 min)
    on Sep 10, 2026
  2. bokelley commented on Sep 10, 2026

    @bokelley
    Contributor

    Triage

    Classification: Conformance failure
    Bucket(s): compliance-suite
    Status: ready-for-human

    What the experts said:

    • ad-tech-protocol-expert: All three active bugs are confirmed defects, not protocol policy questions. Bug 1 is non-breaking and patch-eligible from source; Bugs 2–3 require access to the hosted runner build or config — not_applicable is defined harness behavior and the published SDK 13.0.2 is the reference. P0: 53/62 failing steps on public registry cards reflect harness failures that appear as agent defects, directly undermining compliance program trust.
    • code-reviewer: Bug 1 is a one-line source edit (static/compliance/source/universal/canonical-format-validate-input.yaml), patch changeset, backport to 3.1.x. Also confirms billing_gate_dispatch source is already at v1.1.0 (correct: required_tools: [sync_accounts] only) but no 3.1.x dist carries it — both fixes ship in the same next 3.1.x patch cut. Bugs 2–3 are service-level defects outside storyboard YAML scope.

    My take: Two surfaces — storyboard content (Bug 1, ready to implement from source) and hosted grader service (Bugs 2–3, need someone with runner access to investigate). Bug 4 (first_failed_step_id: null) is best filed separately once the grader investigation starts; its root cause (grader never writing the field vs. feed publisher not reading it) is independent.


    Bug 1 — ready to implement

    canonical_format_validate_input is selected by the hosted grader's OR gate for any agent with comply_test_controller (universal), then fails all 17 steps because the agent has no validate_input. Fix is the same pattern as #6774:

     required_tools:
    -  - validate_input
    -  - comply_test_controller
    +  - validate_input
    • File: static/compliance/source/universal/canonical-format-validate-input.yaml
    • Changeset: patch on adcontextprotocol
    • Base: main, then cherry-pick to backport/3.1.x-required-tools-gate-fixes
    • Bundle in the same 3.1.x cut: billing_gate_dispatch — source v1.1.0 is already correct but no dist/compliance/3.1.x/ artifact carries it yet; no additional source edit needed, just include in the backport
    • Non-breaking: agents implementing validate_input run identically; agents without it receive a clean coverage_gap skip instead of 17 spurious failures

    Bug 2 — hosted runner grades not_applicable steps as failures (flag for human)

    deterministic_testing and idempotency correctly select on the signals agent; the SDK marks their media-buy steps not_applicable (0 failed, 76 skipped). The hosted grader counts every one as a failure. not_applicable is defined harness behavior — no protocol decision required, but someone with grader access needs to confirm whether the service runs a stale build or a config flag disables step-level applicability (COMPLIANCE_HARNESS_SKIP_NOT_APPLICABLE or equivalent).

    Bug 3 — budget-truncated run overwrites complete grade (flag for human)

    A 638 s run hit the 600 s budget and published 2/13 storyboards, dropping 11 previously-passing storyboards from the denominator. The grader should gate on a run_complete signal (or budget_exhausted field in runner output) before committing results. If runner output lacks this field, adding it is patch-eligible. The card should retain the prior complete result until a new complete run is available.


    Triaged by Claude Code. Session: https://claude.ai/code/session_01PsYtFTvHPQZesBaaswFhzW


    Generated by Claude Code

  3. added
    bugSomething isn't working
    claude-triagedIssue has been triaged by the Claude Code triage routine. Remove to re-triage.
    priority:P0Immediate priority: user/revenue/security/reliability bug to work now
    and removed
    claude-triagingTriage routine is actively working on this issue (1-3 min)
    on Sep 10, 2026
  4. EvgenyAndroid commented on Sep 10, 2026

    @EvgenyAndroid
    ContributorAuthor

    Bug 1 PRs: #7405 (main) and #7406 (3.1.x backport, bundling the four #6774 storyboards that exist on that line — the other 13 are 3.2-only).

  5. EvgenyAndroid commented on Sep 10, 2026

    @EvgenyAndroid
    ContributorAuthor

    Bug 3 recurred, and there is new data on both bug 3 and the "slow agent" reading.

    Second truncation, 15 hours after the first. Sales card, heartbeat 2026-09-10T15:56:03Z: total_duration_ms 702418, observation verbatim:

    Compliance timeout budget of 600000ms was reached. Stopped starting new storyboards after 48/75 selected storyboard(s).

    The card again published the truncated result — 13 storyboard_statuses entries, 2 passing, and media_buy 52/65 where the untruncated 00:38:51Z run read 62/78. Recent durations on the same line and the same agent: 219.7 s (00:38Z today), 638.2 s (15:46Z yesterday, truncated), 702.4 s (15:56Z today, truncated). Two of the last three scheduled runs published a partial grade.

    The two steps that run flagged are not slow at the agent. Same observation block:

    Step "Accept legacy fixed MREC selector after canonical normalization" took 10.3s.
    Step "Bind the creative to the package via update_media_buy" took 12.6s.

    Measured against the same production endpoint just now, from an ordinary client, sandbox principal, one call each:

    tool ms
    list_creative_formats 67
    get_products (static) 79
    update_media_buy (pause) 169
    update_media_buy (cancel) 171
    create_media_buy 240
    sync_creatives (live image, bound to package) 244
    get_products (brief → curation, the heaviest path) 974

    Nothing on the agent is within an order of magnitude of 10 s. One thing I can rule out as a cause: the storyboard fixture host test-assets.adcontextprotocol.org is NXDOMAIN from every resolver I tried (Google DNS returns Status 3; adcontextprotocol.org itself resolves fine). Our creative gate HEAD-probes image URLs with a 4 s budget, so a hanging host would have shown up — but DNS failure returns immediately, and sync_creatives with three such creatives measured 248 ms total. If the hosted runner resolves that host and gets a slow answer rather than NXDOMAIN, its per-step latency would be its own network, not ours.

    That leaves the budget interacting with something on the runner side. Whatever the cause, the card-facing half of bug 3 stands on its own: a run that stops at 48 of 75 storyboards should not overwrite a complete grade.

  6. EvgenyAndroid commented on Sep 11, 2026

    @EvgenyAndroid
    ContributorAuthor

    Three additions to the hosted-grader-service half (bugs 2 and 3), which hasn't moved since triage. One is new, one puts numbers on an existing bug, one is a question that decides whether bug 4 is worth filing.

    1. New: the hosted results move without an agent change

    sell.nofluffadvisory.com deployed at 2026-09-09T22:24Z and not again until 2026-09-11T10:54Z. Two heartbeats fall inside that window, so both graded the identical Worker build, on the same line, with the same storyboard selection:

    heartbeat duration scenarios core deterministic_testing
    2026-09-10T00:38:51Z 220 s 159/193 51/55 19/24, 3 failing
    2026-09-11T06:47:42Z 152 s 158/193 50/55 18/24, 4 failing

    One passing scenario left core between them with nothing changed underneath it. Running that storyboard against the same production endpoint on the same line and SDK (@adcp/sdk@13.0.2, --compliance-version 3.1.20) reads 21 passed, 0 failed both before and after, so I can see that a step flipped but not which one.

    This is a different claim from "these failures don't reproduce locally". Two runs of the same grader against the same build disagreeing with each other is a property of the grader, not of the agent, and it is measurable from the public feed without any runner access.

    2. Bug 3: the truncation publishes a shrunken denominator, not a partial result

    heartbeat duration published
    2026-09-09T15:46Z 638 s 101/130 scenarios, 13 storyboards, core 12/14
    2026-09-10T15:56Z 702 s 98/124 scenarios, 13 storyboards, core 12/14
    2026-09-11T06:47Z 152 s 158/193 scenarios, 35 storyboards, core 50/55

    Both slow runs carried Compliance timeout budget of 600000ms was reached. Stopped starting new storyboards after 48/75 selected storyboard(s) (50/75 on the earlier one). Every storyboard the run never reached dropped out of the denominator rather than being reported as unrun, which is why the card read 2/13 for a cycle and media_buy fell from 62/78 to 52/65. It has self-recovered twice, so this is intermittent rather than a trend.

    3. Question that decides bug 4

    Nine failing steps across the two cards are outside the required_tools class and don't reproduce locally: eight on the sales card (deterministic_testing, idempotency, get_media_buys_pagination_integrity, stale_response_advisory) and one on the signals card (error_compliance_signals 6/7). The public feed returns first_failed_step_id and first_failure_message as null on all 35 storyboards of both agents.

    Does the grader write those fields at all? If it does and the feed isn't publishing them, that's bug 4 as triaged. If it never writes them, the fields are decorative and I'll stop asking for step detail through that route. Either answer is useful; I can't tell which from outside.

    Unrelated, for the record

    Chasing item 1 surfaced a real conformance defect of my own, so this isn't all pointing outward. core/brand-ref.json is additionalProperties: false, and my list_accounts was echoing buyer-supplied keys from account.brand straight into the response — a buyer sending brand.name made the entire response schema-invalid, not just that item. Fixed and deployed 2026-09-11T10:54Z. It surfaced as an advisory-level Schema validation failed for list_accounts (non-blocking) line in a local storyboard run, never on the card.

  7. 3 remaining items

  8. EvgenyAndroid commented on Sep 13, 2026

    @EvgenyAndroid
    ContributorAuthor

    Update from the operator side, with one structural question at the end.

    I spent today fixing everything on my two agents that was genuinely mine, so the cards would stop conflating my defects with this issue's. On the signals agent that work is finished, and what remains is a clean measurement.

    A card where these two bugs are the only thing left

    https://adcp.signal-stack.io/mcp, heartbeat 2026-09-13T14:26:11.417Z, 67/68 scenarios, 10/35 storyboards. Every failing step, classified against each storyboard's own required_tools in the 3.1.20 cache:

    storyboard failing steps why
    canonical_format_validate_input 17 selected although the agent has no validate_input
    billing_gate_dispatch 1 selected although the agent has no sync_accounts
    deterministic_testing 7 runner's own message says it grades not_applicable
    idempotency 11 same

    36 failing steps. 18 from bug 1, 18 from bug 2, nothing else. The residue is zero, not small.

    The one real defect that was on this card is gone: activate_signal returned INTERNAL_ERROR where error_compliance_signals asserts INVALID_REQUEST or VALIDATION_ERROR. Fixed this morning, and the grader confirmed it — that storyboard went 6/7 to 7/7 and the card 9 to 10. So the agent side of this card is now empty, and the 36 are not mine to fix.

    For bug 2 it is worth quoting what the runner says while failing the step, because it states the correct grade itself:

    Skipped: agent advertised comply_test_controller, but does not implement a seed_* scenario
    required by this storyboard (`fixture_seed_unsupported`). Storyboard grades not_applicable —
    the buyer-side flow depends on pre-seeded state the agent has no way to accept.
    

    It then counts all 18 of those steps as failures across the two storyboards. A signals agent has no products or media buys to seed; seed_product is not a gap it can close.

    The media-buy card corroborates it

    Same exercise on https://sell.nofluffadvisory.com/mcp, where I shipped five conformance fixes today (total_budget on get_media_buys items, commitment timestamps on forced and seeded buys, the compliance_testing.scenarios closed enum, and the per-package pricing trio on get_media_buy_delivery). Run 2026-09-13T14:46:01.285Z against 2026-09-13T04:26:43.320Z this morning:

    core          51/55 -> 55/55    partial -> passing
    media_buy     62/78 -> 66/78
    reporting     10/17 -> 14/17
    scenarios    159/193 -> 171/193
    storyboards   13/35 -> 15/35
    headline     "1 passing, 4 partial, 1 silent" -> "2 passing, 3 partial, 1 silent"
    

    Twelve scenarios and two storyboards, entirely from fixing my own schema defects. What is left on that card is 20 failing steps: 17 are canonical_format_validate_input — bug 1 again, no validate_input — and the other 3 are two separately-known upstream items, a runner test-kit contract that is not configured and an advisory-severity step. None of the 20 is an agent defect.

    I mention the movement only to make the point that the numbers on both cards now move when a real defect is fixed, and the remainder does not.

    billing_gate_dispatch confirms the class rather than extending it

    billing_gate_dispatch declares required_tools: [get_adcp_capabilities, sync_accounts]. The agent has the first and not the second, so the OR gate admits it and the capability-discovery step then fails on an account block a signals agent has no reason to carry. Its own prerequisites say the quiet part: "The agent under test exposes sync_accounts and advertises an account capability block."

    I am not filing this separately, because #7406 already patches that exact file. I raise it only as a second independent instance of the same gate behaviour, on a storyboard the original report did not cover.

    The structural question

    The remediation in this repository is data, not logic. #7406's changeset says so plainly:

    Removing the universal capability-discovery tool prevents agents without the storyboard's capability-specific tools from entering through the per-storyboard OR gate.

    That works, and it is the right immediate move — stripping get_adcp_capabilities from required_tools denies the OR gate the one tool every agent has. But the gate still reads .some(...) while the suite selector requires all, so the property being relied on is "no storyboard lists two capability-specific tools where an agent might hold exactly one". That has to be maintained by hand on every future storyboard, forever, and nothing tests it.

    Which raises the question I cannot answer from outside: where does the gate itself get reconciled? comply.js and runner.js ship in @adcp/sdk, whose repository.url is git+https://github.com/adcontextprotocol/adcp-client.git, so a fix to the divergence cannot land in this repository at all. I looked for runner source here and found none.

    If there is an adcp-client issue tracking the .some(...) versus all-of divergence, I would like to follow it and will stop restating the point here. If there is not, I am happy to file one there with the same reproduction, on a word from whoever owns that split.

    Separately, and with no expectation attached: #7406 is approved and mergeable, #7405 is currently blocked. Both would retire the 18 mis-selection steps above.

  9. bokelley commented on Sep 13, 2026

    @bokelley
    Contributor

    Structural question — cross-repo escalation

    The .some() vs all-of gate divergence (runner.js:1421 vs comply.js:896) lives in adcontextprotocol/adcp-client, not in this repository. There is no adcp-client issue on record here for that split. Filing one there is the right call.

    Sibling-repo-fix-needed: adcp-client — dist/lib/testing/storyboard/runner.js:1421 uses .some() (OR) where dist/lib/testing/compliance/comply.js:896 uses .filter() (AND). The per-storyboard gate therefore admits storyboards the suite selector would skip. The storyboard-YAML fix in #7406 is the correct near-term patch; reconciling the two paths in the SDK eliminates the invariant you described ("no storyboard lists two capability-specific tools where an agent might hold exactly one") so it no longer has to be maintained by hand on every future storyboard.

    For the adcp-client issue: the reproduction steps from the original report (runner.js:1421 exact line, the two affected call paths, the @adcp/sdk@13.0.2 test command) are sufficient. The fix candidate is changing .some(...) to .every(...) or aligning it with the .filter() shape — but that's for the SDK maintainer to decide on the correct semantics.


    Triaged by Claude Code. Session: https://claude.ai/code/session_011Jt9GMuxWo5VknTzFUQNoD


    Generated by Claude Code

  10. EvgenyAndroid commented on Sep 13, 2026

    @EvgenyAndroid
    ContributorAuthor

    Filed as adcontextprotocol/adcp-client#2882.

    It leads with the recurrence rather than the divergence, because that is the part that is new. adcp-client#2652 reported this exact billing_gate_dispatch mis-selection and was closed 2026-08-25 on the strength of #6774, with the explicit note that the loader/helper work was "no longer required to prevent this failure." That was correct for the line #6774 landed on; billing_gate_dispatch then mis-selected again on 3.1.20 last week, because the maintenance backport is #7406 and it is still open. So the issue asks for the divergence to be tracked rather than patched a particular way — and says plainly that if the any-of gate turns out to be deliberate and the suite selector is the side that should move, that answers it equally well.

    I offered to test a candidate against both registered agents.

    Nothing outstanding for you here. Both .some(...) instances on my cards are covered by #7405 and #7406, and I would still like to see #7406 land — it retires 18 of the 36 failing steps on the signals card on its own.

  11. EvgenyAndroid commented on Sep 15, 2026

    @EvgenyAndroid
    ContributorAuthor

    Two observations on bug 2, both from the public feed only. I tried to build a stronger local-vs-hosted reproduction and could not, so I am posting what survived rather than what I hoped to show.

    failure_count in the public feed is a derived residual. Across all 35 storyboards on sell.nofluffadvisory.com/mcp and all 38 on adcp.signal-stack.io/mcp, failure_count is exactly steps_total - steps_passed — every storyboard, no exception — and skipped_count is 0 on all 73. first_failed_step_id, first_failure_message and first_failure_validations are null or empty on all 73 as well.

    So in the public projection there is no representation of a skipped step at all: anything that is not passed is reported as a failure, and an owner cannot tell which class it was. That is worth separating from the grading question in this issue. Even where the grader internally classifies a step as not_applicable, a reader of the card sees a failure — so part of what bug 2 looks like from outside may be a reporting artifact rather than a grading one. Those need different fixes, and the reporting half looks cheap.

    Concretely on my sales card (run 2026-09-13T14:46:01.285Z): sum(failure_count) is 20, of which canonical_format_validate_input contributes 17. 17 of the 35 storyboards are untested at 0/0 steps and contribute nothing either way.

    The 3.1.20 cache line is still what runs post-fix. The required_tools OR-gate fix merged in #7405/#7406 on 2026-09-14 (12:15:41Z and 12:16:01Z) and shipped in v3.1.23. My signals card re-ran at 2026-09-14T20:03:40.370Z — 7h48m after that merge — and still reports adcp_version: 3.1.20. I am not claiming the grader should have moved by then; only recording that a post-merge run is still on the 3.1.20 line, so cards graded on it will keep carrying the pre-fix canonical_format_validate_input numbers until the line advances. (My sales card is useless as evidence here: it last ran 21.5 hours before the merge.)

    What I could not establish, in case someone else tries the same route. I tried to show that steps the SDK skips locally are the same steps the hosted grader counts as failures, using idempotency (hosted 9/11, failure_count 2) against a local run on @adcp/sdk 13.0.4 with --compliance-version 3.1.20 (STORYBOARD-OK, 9 passed / 0 failed / 2 skipped). The arithmetic lines up, but the inference does not hold up:

    • Both local skips come from requires_contract gates in universal/idempotency.yaml — parallel_dispatch_runner (line 590) and rate_limit_trip_runner (line 769). Those kits are runner capability contracts (harness_mode: black_box, burst_step_contract), not credentials, so the local skip means the local CLI does not implement that harness. Loading the kits explicitly changes nothing. Whether the hosted runner implements them I cannot see, so its 2 failures may be real.
    • rate_limit_trip_runner declares exhausted_outcome: not_applicable after up to 500 attempts. This agent implements no rate limiting at all, so a runner that does drive the burst would exhaust it and produce not_applicable — which, per the residual above, would surface on the card as a failure regardless.
    • stale_response_advisory (hosted 3/4, failure_count 1) is not a candidate either: this agent does not implement force_upstream_unavailable, and the local run is also STORYBOARD-PARTIAL with 4 advisory validation failures. Local and hosted agree it is not clean.

    One housekeeping note: this issue's title still says "53 of 62 failing steps on two cards". Across the two cards today that is 21 (20 + 1), mostly because #7405/#7406 fixed the selector half. Might be worth a retitle so the remaining scope reads as bug 2 only.

  12. EvgenyAndroid commented on Sep 16, 2026

    @EvgenyAndroid
    ContributorAuthor

    Correction to my 2026-09-15 comment. The numbers in it were right for the two runs I sampled. The conclusions I drew from them were wrong, and #7454, linked in this thread on 2026-09-13, already answered most of it.

    1. The feed does represent skips, and bug 2 was never a reporting artifact. Before #7454 the serializer already passed skipped_count through, but only prerequisite_failed cascades were counted there. Neutral skips and their cascades were dropped from steps_total. Every other skip counted as a failure. My 73 rows had no counted cascade, so failure_count matched the residual everywhere. That is grading, as filed. first_failed_* is also null because the public feed is redacted ("public failure observations/headlines remain redacted", #7454). So my line that an owner cannot tell which class a step was is wrong too.

    2. Bug 2 moved, and I missed it. #7454 (merged 2026-09-13 14:16 UTC) classifies fixture_seed_unsupported as not_applicable. That is the reason I quoted for these steps on 2026-09-13; before #7454 it fell through to a failure. On the signals card, deterministic_testing (0/7) and idempotency (0/11) have read untested 0/0 on every run since 2026-09-14 20:03 UTC, the first run to carry #7454's provenance fields and the run I cited. Those 18 steps are no longer counted as failures, though the card doesn't show them as passing either, as the local SDK does. Across the same line, the whole signals card went from the 36 failing steps I counted on 2026-09-13 to 1 (billing_gate_dispatch). All of those runs also report sdk_version 14.0.0-rc.35, so I can't attribute the whole change to #7454.

    3. My 3.1.20 point was contradicted by its own evidence. That same 09-14 run already showed canonical_format_validate_input as untested 0/0.

    For the retitle note: the two cards now sum to 11 failing steps (sales 10, signals 1), not 21. On sales, canonical_format_validate_input passes 16/16 because this agent now implements validate_input, not because of a runner change. All four runs dated 2026-09-16 still report compliance_bundle_version: 3.1.20.

  13. changed the title [-]Hosted grader (3.1.20): the storyboard gate admits on any one required tool, and steps the SDK marks not_applicable are graded as failures — 53 of 62 failing steps on two cards are ungradable[/-] [+]Hosted grader: 11 failing steps remain on two cards after #7454/#7405/#7406 (sales 10, signals 1); budget-truncated runs still published as complete grades[/+] on Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingclaude-triagedIssue has been triaged by the Claude Code triage routine. Remove to re-triage.compliance-suitepriority:P0Immediate priority: user/revenue/security/reliability bug to work now

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions