Repository navigation
feat(framework): link tests to specification rules with spec_assert and a requirements catalog - #124
Open
segiodongo wants to merge 3 commits into
Conversation
8 of 9 tasks
segiodongo
force-pushed
the
upstream/2-requirements-markers
branch
5 times, most recently
from
November 13, 2026 20:52
3017b87 to
0e9ffd4
Compare
… rule IDs Add a catalog of every RFC 2119 rule on the shopping specification pages, so that the shopping tests can cite the rule they check by a stable ID. - requirements/catalog/: one file per page (510 rules at ucp f2626c28, 507 after review), capabilities.json, the extractor's report, and requirements.lock.json, which issues each rule its ID. A rule keeps its ID when the page around it changes or its wording is lightly edited; a removed rule's ID is retired, never reused. - requirements/catalog/history.json and requirements/releases.json: which releases (2026-01-11, 2026-01-23, 2026-04-08, 2026-08-25) state each rule, so a server can later be measured against the release it implements. - requirements/review.json: a person's decisions on 204 rules the automatic pass couldn't settle (actor, testability, not-a-rule, splits). - requirements/tools/build_catalog.py: regenerates the catalog from a specification checkout and the extractor in the .github repository (org-tools/requirements-extractor, 1687b56); --check fails if any output would change. Unit tests in tests/build_catalog_test.py. - .pre-commit-config.yaml keeps the generated catalog out of prettier and codespell; .gitignore ignores requirements/reports/. 211 rules are testable by this suite, 98 of them MUST or MUST NOT. The IDs are stable within this repository; official IDs will be issued in the ucp repository, and every lock entry records its text digest for mapping.
… the marks Let each shopping test say which rule of the requirements catalog it checks, and check those citations before any test runs. - framework/decorators.py: spec_assert(*ids) cites one or more stable rule IDs, with partly(id, gap) for a rule a test checks only in part; no_rule(reason), scenario(reason), outside_catalog(reason) and older_version(version, reason) mark tests that cite none. A test carries one marker. The previous spec_assert(section, id) stub had no callers. - framework/requirements.py loads the catalog, its lock and its history once and says what an ID names: a rule, not a rule, retired, removed or unknown. - framework/marker_check.py checks every shopping test's marker, from the runner (including --dry-run) and from pytest (conftest.py). In strict mode, the default, a test with no marker, or a citation of an unknown, retired or not-a-rule ID, stops the run; citing a rule this suite cannot test, or one outside the test's capabilities, is a warning. --requirements_check=warn (--requirements-check=warn with pytest) reports without stopping; off skips the check. Unit tests in tests/decorators_test.py, tests/marker_check_test.py and tests/requirements_test.py.
Give each of the 77 shopping tests its marker, so the strict check passes: 24 cite rules of the catalog (21 distinct rules, 3 of them only in part), 26 are no_rule (behavior the spec describes without a keyword), 16 are scenario (a store's own data), 8 are outside_catalog (core and field-ownership rules outside the shopping pages) and 3 are older_version (behavior a later release changed). Decorators and imports only; no test body changes.
segiodongo
force-pushed
the
upstream/2-requirements-markers
branch
from
November 18, 2026 20:08
0e9ffd4 to
4c962ba
Compare
segiodongo
marked this pull request as ready for review
November 18, 2026 20:09
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Lets each shopping test say which rule of the specification it checks, by a stable ID, and checks those citations before any test runs. This is the foundation for a requirements coverage report (next PR) and for the new shopping tests that follow, each of which cites the rules it covers.
Three commits
requirements/: the shopping requirements catalog. Every RFC 2119 rule on the shopping pages ofucpatf2626c28(510 extracted, 507 after review), one JSON file per page, generated with the requirements extractor from the.githubrepository plus two preview options (capability from bullet or file path; upper-case keywords counted when not bold).requirements.lock.jsongives each rule a stable ID such asREQ-SHOPPING-CART-0007that survives edits to the page;history.json/releases.jsonrecord which release first stated each rule;review.jsonholds 204 human decisions on actor and testability.tools/build_catalog.py --checkfails if any output would change, so the catalog can be verified in CI. 211 rules are testable by this suite (REST, against a business), 98 of them MUST / MUST NOT. Details inrequirements/README.md.framework/: markers and the check.@spec_assert("REQ-…", …)(withpartly(id, gap)for a rule checked only in part),@no_rule,@scenario,@outside_catalogand@older_version.framework/marker_check.pyruns before any test — fromucp-conformance(including--dry-run) and frompytestviaconftest.py— and stops on a test with no marker or a citation of an unknown / retired / not-a-rule ID; citing an untestable rule or one outside the test's capabilities is a warning.--requirements_check=warn|offrelaxes it.shopping/: the existing 77 tests get their markers. Decorators and imports only, no test-body changes: 24 cite rules (21 distinct, 3 in part), 26no_rule, 16scenario, 8outside_catalog, 3older_version.Verification: 80 unit tests + 55 subtests (
uv run pytest tests); strict check passes with 0 unmarked;--suite=all --dry-runlists 83 tests (77 shopping + 6 core/common); against the Flower Shop sample server (2026-04-08): 83 run, 81 passed, 2 skipped, 0 failed;build_catalog.py --checkreports every output up to date; pre-commit and cspell clean.Type of change
Checklist