Skip to content

feat(framework): link tests to specification rules with spec_assert and a requirements catalog - #124

Open
segiodongo wants to merge 3 commits into
Universal-Commerce-Protocol:mainfrom
segiodongo:upstream/2-requirements-markers
Open

segiodongo wants to merge 3 commits into
Universal-Commerce-Protocol:mainfrom
segiodongo:upstream/2-requirements-markers

Conversation

@segiodongo

@segiodongo segiodongo commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Lets each shopping test say which rule of the specification it checks, by a stable ID, and checks those citations before any test runs. This is the foundation for a requirements coverage report (next PR) and for the new shopping tests that follow, each of which cites the rules it covers.

Three commits

  1. requirements/: the shopping requirements catalog. Every RFC 2119 rule on the shopping pages of ucp at f2626c28 (510 extracted, 507 after review), one JSON file per page, generated with the requirements extractor from the .github repository plus two preview options (capability from bullet or file path; upper-case keywords counted when not bold). requirements.lock.json gives each rule a stable ID such as REQ-SHOPPING-CART-0007 that survives edits to the page; history.json / releases.json record which release first stated each rule; review.json holds 204 human decisions on actor and testability. tools/build_catalog.py --check fails if any output would change, so the catalog can be verified in CI. 211 rules are testable by this suite (REST, against a business), 98 of them MUST / MUST NOT. Details in requirements/README.md.
  2. framework/: markers and the check. @spec_assert("REQ-…", …) (with partly(id, gap) for a rule checked only in part), @no_rule, @scenario, @outside_catalog and @older_version. framework/marker_check.py runs before any test — from ucp-conformance (including --dry-run) and from pytest via conftest.py — and stops on a test with no marker or a citation of an unknown / retired / not-a-rule ID; citing an untestable rule or one outside the test's capabilities is a warning. --requirements_check=warn|off relaxes it.
  3. shopping/: the existing 77 tests get their markers. Decorators and imports only, no test-body changes: 24 cite rules (21 distinct, 3 in part), 26 no_rule, 16 scenario, 8 outside_catalog, 3 older_version.

The IDs are stable within this repository, not official. The lock records each rule's text digest so they can be mapped to the IDs the ucp repository issues later. The catalog's added fields sit under a demo key in each record; happy to rename that in this PR if reviewers prefer (e.g. suite).

Verification: 80 unit tests + 55 subtests (uv run pytest tests); strict check passes with 0 unmarked; --suite=all --dry-run lists 83 tests (77 shopping + 6 core/common); against the Flower Shop sample server (2026-04-08): 83 run, 81 passed, 2 skipped, 0 failed; build_catalog.py --check reports every output up to date; pre-commit and cspell clean.

Type of change

  • New feature (non-breaking change which adds functionality)

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules

@segiodongo
segiodongo force-pushed the upstream/2-requirements-markers branch 5 times, most recently from 3017b87 to 0e9ffd4 Compare November 13, 2026 20:52
… rule IDs

Add a catalog of every RFC 2119 rule on the shopping specification pages,
so that the shopping tests can cite the rule they check by a stable ID.

- requirements/catalog/: one file per page (510 rules at ucp f2626c28, 507
  after review), capabilities.json, the extractor's report, and
  requirements.lock.json, which issues each rule its ID. A rule keeps its ID
  when the page around it changes or its wording is lightly edited; a
  removed rule's ID is retired, never reused.
- requirements/catalog/history.json and requirements/releases.json: which
  releases (2026-01-11, 2026-01-23, 2026-04-08, 2026-08-25) state each rule,
  so a server can later be measured against the release it implements.
- requirements/review.json: a person's decisions on 204 rules the automatic
  pass couldn't settle (actor, testability, not-a-rule, splits).
- requirements/tools/build_catalog.py: regenerates the catalog from a
  specification checkout and the extractor in the .github repository
  (org-tools/requirements-extractor, 1687b56); --check fails if any output
  would change. Unit tests in tests/build_catalog_test.py.
- .pre-commit-config.yaml keeps the generated catalog out of prettier and
  codespell; .gitignore ignores requirements/reports/.

211 rules are testable by this suite, 98 of them MUST or MUST NOT. The IDs
are stable within this repository; official IDs will be issued in the ucp
repository, and every lock entry records its text digest for mapping.
… the marks

Let each shopping test say which rule of the requirements catalog it checks,
and check those citations before any test runs.

- framework/decorators.py: spec_assert(*ids) cites one or more stable rule
  IDs, with partly(id, gap) for a rule a test checks only in part;
  no_rule(reason), scenario(reason), outside_catalog(reason) and
  older_version(version, reason) mark tests that cite none. A test carries
  one marker. The previous spec_assert(section, id) stub had no callers.
- framework/requirements.py loads the catalog, its lock and its history once
  and says what an ID names: a rule, not a rule, retired, removed or unknown.
- framework/marker_check.py checks every shopping test's marker, from the
  runner (including --dry-run) and from pytest (conftest.py). In strict
  mode, the default, a test with no marker, or a citation of an unknown,
  retired or not-a-rule ID, stops the run; citing a rule this suite cannot
  test, or one outside the test's capabilities, is a warning.
  --requirements_check=warn (--requirements-check=warn with pytest) reports
  without stopping; off skips the check.

Unit tests in tests/decorators_test.py, tests/marker_check_test.py and
tests/requirements_test.py.
Give each of the 77 shopping tests its marker, so the strict check passes: 24 cite rules of the catalog (21 distinct rules, 3 of them only in part), 26 are no_rule (behavior the spec describes without a keyword), 16 are scenario (a store's own data), 8 are outside_catalog (core and field-ownership rules outside the shopping pages) and 3 are older_version (behavior a later release changed). Decorators and imports only; no test body changes.
@segiodongo
segiodongo force-pushed the upstream/2-requirements-markers branch from 0e9ffd4 to 4c962ba Compare November 18, 2026 20:08
@segiodongo
segiodongo marked this pull request as ready for review November 18, 2026 20:09
@segiodongo
segiodongo requested a review from damaz91 November 18, 2026 20:23
@damaz91 damaz91 added the status:needs-triage Signal that the PR is ready for human triage label Nov 18, 2026
@segiodongo segiodongo self-assigned this Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

status:needs-triage Signal that the PR is ready for human triage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants