Skip to content

Hosted storyboard history: represent incomplete fixture coverage separately from executed passes #8416

Open

Description

@bokelley

Observed behavior

A full reporting run against a native catalog without the optional feature returns an explicit coverage_gaps[].reason: fixture_unsatisfied with zero authored reporting steps. The SDK's raw overall_passed boolean can still be true because no executed assertion failed. Addie's existing history writer stores that boolean and counts without a separate incomplete-coverage state; the manual matrix's clean count similarly means no failed rows, not tested feature support.

#8336 fixes Addie's visible report to show Incomplete — coverage gap and Not applicable for this run, while preserving actual SDK results, existing history fields and floor accounting. Training CLI now prints the real coverage-gap detail. No missing native fixture is treated as proof of nonsupport or as an executed reporting pass.

Follow-up: give hosted run history/aggregate presentation a reviewed representation of incomplete fixture coverage, retaining original failure evidence and distinguishing a completed no-match from discovery/schema/protocol errors. Preserve immutable historical records; do not backfill invented support, globally skip mandatory failures or silently lower grading floors. A failed selected implementation must stay failed even when another scope is unexercised.

This host accounting/presentation follow-up is separate from catalog selection and the per-invocation reporting adapter; no evaluator/registry policy implementation or release is assigned by this issue.

Activity

  1. added this to the Spec Backlog milestone on Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    claude-triagedIssue has been triaged by the Claude Code triage routine. Remove to re-triage.compliance-suite

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions