Repository navigation
feat: refactor conformance suite into vertical-agnostic architecture (RFC #520) - #107
segiodongo wants to merge 7 commits into
Conversation
…(RFC #520) Restructure conformance test harness into a 4-tier architecture (framework, core, common, shopping) per RFC #520. - framework/: BaseIntegrationTest, dynamic mock servers, discovery profile parsing, and decorators (@requires_capability, @spec_assert). - core/: Protocol-level tests without retail domain dependencies (protocol discovery, binding, security, idempotency). - common/: Cross-cutting extensions (webhooks, AP2, card credentials). - shopping/: Retail domain tests (checkout lifecycle, business logic, discount, fulfillment, order, validation) and fixtures. - platforms/google.yaml: Platform certification profile verifying required capabilities and payment handlers. - runner.py: Dynamic CLI test orchestrator (ucp-conformance entrypoint) supporting suite filtering, platform evaluation, and capability checks. - integration_test_utils.py: Backward-compatible alias facade emitting DeprecationWarning.
6273dd1 to
be7ecbb
Compare
unittest counts a skipped test in testsRun, so the summary's Passed line (testsRun minus failures and errors) also counted every skip as a pass. With one passing, one failing and one skipped test it printed Passed: 2. Passed now leaves out skipped tests; the Skipped line is unchanged.
|
Hi all, a quick update and two questions to help move this forward. Update: I pushed a small fix to the runner's summary, which counted skipped tests as passed. unittest includes skipped tests in 1. Size. This PR is large (+3.5k/−2.3k across 58 files), which makes it hard to review. If it would help, I can split it into smaller PRs:
I'm also happy to keep it as one PR if you prefer. 2. Order with the open test PRs. 14 open PRs (#94–#118) change files that this PR moves or edits, so whichever lands first means the other side rebases. I'm glad to do that work either way:
This PR is also the base for follow-up work that links each test to the spec requirements it checks and reports coverage. I'd send that as a series of small PRs once the layout settles. Which would you prefer? Thanks for taking the time to review. |
damaz91
left a comment
There was a problem hiding this comment.
Thanks for putting together this refactoring toward the multi-vertical architecture in RFC #520, @segiodongo!
On your two questions:
- Splitting into smaller PRs: Yes, splitting this into smaller PRs will make review much easier and avoid painful rebase conflicts:
- PR 1: Add
framework/,runner.py, andplatforms/while keeping existing tests at their current paths (addressing the flag/env-var/capability issues noted inline). - PR 2: Pure file move (
git mv) of shopping tests and fixtures intoshopping/with minimal import-only edits so Git rename detection stays at ~100%. - PR 3: Extract vertical-agnostic
core/andcommon/suites.
- PR 1: Add
- Ordering vs. open test PRs (
#89–#118): We should land the ready open test PRs (or land PR 1 first since it doesn't move existing test files) before doing the directory rename in PR 2, so contributors don't have to manually resolve combined file-move + logic diffs.
I left inline comments on a few functional regressions and coverage gaps to address (in particular, 5 tests in common/ are currently skipped due to non-existent capability names, TokenBindingTest was dropped, and --conformance_input / SERVER_URL are shadowed or ignored).
| checkout.Checkout.model_rebuild(_types_namespace={"Payment": Payment}) | ||
|
|
||
|
|
||
| @requires_capability("dev.ucp.common.payments.ap2") |
There was a problem hiding this comment.
dev.ucp.common.payments.ap2 is not a valid UCP capability name (the schema defines dev.ucp.common.payment.ap2_mandate, or dev.ucp.shopping.ap2_mandate in older server profiles). Because no server advertises dev.ucp.common.payments.ap2, Ap2MandateTest is silently skipped on every run.
There was a problem hiding this comment.
You're right, thanks. Fixed in e4bc51a: the test requires dev.ucp.common.payment.ap2_mandate. Because releases before 2026-08-25 advertised it as dev.ucp.shopping.ap2_mandate, framework.discovery records that rename (FORMER_NAMES) and supports_capability() accepts either, so the test runs against both generations of servers. It still skips on the sample server, which advertises neither.
| checkout.Checkout.model_rebuild(_types_namespace={"Payment": Payment}) | ||
|
|
||
|
|
||
| @requires_capability("dev.ucp.common.payments.cards") |
There was a problem hiding this comment.
dev.ucp.common.payments.cards does not exist in ucp.capabilities (card_credential.json is a data type schema under common/types/, not an advertised capability). Decorating CardCredentialTest with @requires_capability("dev.ucp.common.payments.cards") causes this test to be permanently skipped.
There was a problem hiding this comment.
Agreed, there is no such capability. Fixed in e4bc51a: the test is gated on dev.ucp.shopping.checkout only (the endpoint the credential is submitted to), which matches how it behaved on main. It runs and passes against the sample server now.
| checkout.Checkout.model_rebuild(_types_namespace={"Payment": Payment}) | ||
|
|
||
|
|
||
| @requires_capability("dev.ucp.common.webhooks") |
There was a problem hiding this comment.
Two issues here:
dev.ucp.common.webhooksis not a capability inucp.capabilities(order webhooks are configured underdev.ucp.shopping.order), causing all 3 tests inWebhookTestto be silently skipped.WebhookTestdrivescreate_checkout_session(),complete_checkout_session(), and/testing/simulate-shipping/{order_id}and asserts onorder.fulfillment.expectations. For the same reasonwebhook_structure_test.pywas placed undershopping/order/, this test is shopping-order specific and belongs undershopping/order/.
There was a problem hiding this comment.
Both points taken. In e4bc51a the test moves to shopping/order/webhook_test.py next to webhook_structure_test.py and requires dev.ucp.shopping.order, whose config carries the webhook_url. All three tests run and pass against the sample server; common/webhooks/ is removed.
| checkout.Checkout.model_rebuild(_types_namespace={"Payment": Payment}) | ||
|
|
||
|
|
||
| class TokenBindingTest(integration_test_utils.IntegrationTestBase): |
There was a problem hiding this comment.
TokenBindingTest (test_token_binding_completion, which tests credential.binding with checkout_id and identity on POST /checkout-sessions/{id}/complete) was deleted here and not migrated. core/binding_test.py replaces it with ProtocolBindingTest on /.well-known/ucp, dropping payment token binding test coverage. Please restore TokenBindingTest under shopping/ (or common/payments/).
There was a problem hiding this comment.
Thanks for catching this. TokenBindingTest.test_token_binding_completion is restored in e4bc51a as common/payments/token_binding_test.py, next to the other payment-credential tests, on ShoppingIntegrationTestBase, and registered in the common suite.
| config_path = ( | ||
| getattr(FLAGS, "shopping_conformance_input", None) | ||
| or getattr(FLAGS, "conformance_input", None) | ||
| or str(Path(DEFAULT_SHOPPING_FIXTURES) / "conformance_input.json") |
There was a problem hiding this comment.
Because --shopping_conformance_input is defined with a non-empty default (.../shopping/fixtures/flower_shop/conformance_input.json), getattr(FLAGS, "shopping_conformance_input", None) is always truthy and short-circuits before getattr(FLAGS, "conformance_input", None). Passing --conformance_input on the CLI or via runner.py is ignored for all shopping tests. The same short-circuit bug occurs on lines 532–534 for shopping_fixtures_dir vs. test_data_dir.
Additionally, super().setUp() on line 510 starts AgentProfileServer before self.conformance_config is loaded here, so when --conformance_input is not explicitly passed, AgentProfileServer starts with an empty conformance_config.
There was a problem hiding this comment.
Correct on both counts. Fixed in 0c4671f: the shopping_* flags are gone. conformance_input, fixture_config and test_data_dir are defined once in framework.base_test with None defaults; ShoppingIntegrationTestBase sets default_conformance_input to the flower-shop input and falls back to the bundled fixtures only when the flags are unset. The base class resolves the input before it starts the agent profile server, so the agent advertises the merchant's ucp_version. I checked that --conformance_input is read by passing an input with "ucp_version": "2099-01-01", which makes test_discovery fail on the version as it should.
| response = self.client.post( | ||
| f"/testing/simulate-shipping/{order_id}", | ||
| headers=self.get_headers(), # Standard headers only | ||
| "/testing/simulate-shipping/test-order-security", |
There was a problem hiding this comment.
/testing/simulate-shipping/{id} is a sample-server-specific shopping helper endpoint, not part of the core UCP specification. Non-shopping or custom servers that do not implement this route will return 404 instead of 403 and fail --suite=core. This test belongs under shopping/.
There was a problem hiding this comment.
Agreed. In e4bc51a the tests move to shopping/order/simulation_url_security_test.py in main's form (a real completed order, and 200 with the correct secret rather than 200/404), gated on dev.ucp.shopping.order.
| self.assertIn("ucp", data, "Discovery profile must be wrapped in 'ucp' key") | ||
| ucp_data = data["ucp"] | ||
| # The discovery profile may be returned at the root or wrapped in `ucp`. | ||
| ucp_data = data.get("ucp", data) |
There was a problem hiding this comment.
Per source/schemas/profile.json ("required": ["ucp"]), discovery profiles must be wrapped in a top-level "ucp" object. Falling back to data when "ucp" is missing allows non-conformant unwrapped profiles to pass. Please keep self.assertIn("ucp", data, ...).
There was a problem hiding this comment.
Restored in e4bc51a: self.assertIn("ucp", data, …) is back and the profile is read from data["ucp"].
| ) | ||
| self.assert_response_status(response, 422) | ||
| resp = self.client.get("/.well-known/ucp", headers=headers) | ||
| self.assertIn(resp.status_code, [200, 400]) |
There was a problem hiding this comment.
Sending an incompatible UCP-Agent version to GET /.well-known/ucp and asserting [200, 400] does not verify version negotiation rejection (422), because discovery is a public GET endpoint that returns 200 regardless of agent version. Please retain the POST /checkout-sessions version-negotiation test (asserting 422 on "2099-01-01") in shopping/.
There was a problem hiding this comment.
You're right that a GET cannot show the rejection. The POST /checkout-sessions test asserting 422 on 2099-01-01 was already retained as ShoppingProtocolTest.test_version_negotiation in shopping/checkout/protocol_test.py; in e4bc51a I removed the GET-based copy from core/ and left a note pointing to the shopping test.
| headers=headers, | ||
| ) | ||
| # Server should return 405 (Method Not Allowed), 415, or 400 | ||
| self.assertIn(response.status_code, [400, 404, 405, 415]) |
There was a problem hiding this comment.
Because /.well-known/ucp is a GET-only endpoint, servers return 405 Method Not Allowed before inspecting Content-Type, so POST /.well-known/ucp with Content-Type: text/plain does not actually verify unsupported Content-Type rejection.
There was a problem hiding this comment.
Agreed, removed in e4bc51a. A Content-Type check needs an endpoint that reads a body, so it belongs with a vertical's own endpoints rather than in core/.
| "shopping", | ||
| "platforms", | ||
| "runner.py", | ||
| ] |
There was a problem hiding this comment.
integration_test_utils.py (still imported by core/idempotency_test.py and common/*) and shopping-agent-test.json (loaded from the repo root by framework/agent_profile_server.py, whose fallback DEFAULT_PROFILE lacks required ucp.version and services fields) are excluded from only-include, which will break wheel installations.
There was a problem hiding this comment.
Fixed in da0a398: both files are in only-include. I also removed the DEFAULT_PROFILE fallback — with the template packaged there is no case where serving an invalid profile beats a clear FileNotFoundError.
damaz91
left a comment
There was a problem hiding this comment.
Thank you for driving the RFC #520 4-tier architecture refactor and fixing the skipped-test summary calculation in a03b68c, @segiodongo!
On a detailed pass through the shopping/base.py extraction, CLI flag wiring in runner.py, and the main merge (56ad0ca), there are a few functional issues and merge regressions that need to be addressed before we can merge:
--conformance_inputand--test_data_dirare shadowed by non-empty defaults (shopping/base.py):shopping_conformance_inputandshopping_fixtures_dirare registered with non-empty default paths pointing toflower_shop. InShoppingIntegrationTestBase.setUp(),getattr(FLAGS, "shopping_conformance_input", None) or getattr(FLAGS, "conformance_input", None)always short-circuits on the non-empty default, so passing--conformance_inputor--test_data_diris silently ignored.--fixture_configonucp-conformance(runner.py/README.md): TheREADME.mdexample passes--fixture_configtouv run ucp-conformance, which fails withunrecognized arguments: --fixture_config=...becauseparse_args()does not accept it (and_sync_absl_flags()parsesFLAGSbefore_load_tests()importsshopping.base).- Regressions of #15 and #58 in
shopping/base.py:update_checkout_session()dropped the fallback preservation ofbuyer,fulfillment,discounts, andplatformfromcheckout_objwhen not passed (#15 /88527b2), causingPUT /checkout-sessions/{id}updates to wipe out previously set fields.create_checkout_payload()revertedpayment_handler.Base.versionandcheckout_req.ucp["version"]to hardcoded"2026-04-08"instead ofself.conformance_config.get("ucp_version", DEFAULT_UCP_VERSION)(#58 /7422310).
- Dropped
TokenBindingTest(binding_test.py): Rootbinding_test.py(TokenBindingTest.test_token_binding_completion, testing payment credential token binding onPOST /checkout-sessions/{id}/complete) was deleted whencore/binding_test.py(ProtocolBindingTest) was introduced, removingtest_token_binding_completionfrom the suite. integration_test_utils.get_headers()omitsidempotency-keyby default: Whenidempotency_keyisNone,integration_test_utils.get_headers()omits theidempotency-keyheader (unlikeBaseIntegrationTest.get_headers()), affectingcommon/payments/ap2_test.pyandcommon/payments/card_credential_test.py(which also triggerDeprecationWarningby importingintegration_test_utils).--dry-runcontacts the server (runner.py):fetch_server_profile(args.server_url)runs beforeif args.dry_run:, contacting/.well-known/ucpeven during--dry-run.
Left detailed inline comments on each location below.
| getattr(FLAGS, "shopping_conformance_input", None) | ||
| or getattr(FLAGS, "conformance_input", None) | ||
| or str(Path(DEFAULT_SHOPPING_FIXTURES) / "conformance_input.json") | ||
| ) |
There was a problem hiding this comment.
Because shopping_conformance_input (line 92) and shopping_fixtures_dir (line 86) are defined with non-empty default strings pointing to DEFAULT_SHOPPING_FIXTURES, getattr(FLAGS, "shopping_conformance_input", None) and getattr(FLAGS, "shopping_fixtures_dir", None) (line 546) are always truthy. As a result, getattr(FLAGS, "conformance_input", None) and getattr(FLAGS, "test_data_dir", None) are never reached, and passing --conformance_input or --test_data_dir on the CLI is silently ignored.
Additionally, super().setUp() at line 510 starts AgentProfileServer using self.conformance_config before ShoppingIntegrationTestBase.setUp() loads the shopping config here.
Consider setting the default of shopping_conformance_input and shopping_fixtures_dir (as well as test_data_dir) to None so explicit --conformance_input / --test_data_dir flags take precedence over the DEFAULT_SHOPPING_FIXTURES fallback, and overriding _load_conformance_config() so BaseIntegrationTest.setUp() receives the resolved config before starting AgentProfileServer.
There was a problem hiding this comment.
Done as you suggest in 0c4671f (see the reply on line 525): None defaults, a single set of input flags in the framework, and the input resolved by the base class before the agent profile server starts.
| headers: dict[str, str] | None = None, | ||
| ) -> Any: | ||
| """Update a checkout session.""" | ||
| currency = currency if currency is not None else checkout_obj.currency |
There was a problem hiding this comment.
Regression from #15 (88527b2): update_checkout_session() dropped the fallback preservation of buyer, fulfillment, discounts, and platform from checkout_obj when those optional arguments are None:
def to_json(obj):
if obj is not None and hasattr(obj, "model_dump"):
return obj.model_dump(mode="json", exclude_none=True)
return obj
if buyer is None:
buyer = to_json(getattr(checkout_obj, "buyer", None))
if fulfillment is None:
fulfillment = to_json(getattr(checkout_obj, "fulfillment", None))
if discounts is None:
discounts = to_json(getattr(checkout_obj, "discounts", None))
if platform is None:
platform = to_json(getattr(checkout_obj, "platform", None))Without this, calling self.update_checkout_session(checkout_obj, discounts=...) or self.update_checkout_session(checkout_obj, fulfillment=...) sends a full-replacement PUT that wipes out existing buyer or fulfillment state on the checkout.
There was a problem hiding this comment.
Restored in 09e70cb — see the reply on line 977.
| "--conformance_input", | ||
| default=None, | ||
| help="Optional path to custom conformance input JSON configuration.", | ||
| ) |
There was a problem hiding this comment.
README.md (line 204) documents running uv run ucp-conformance ... --fixture_config=path/to/your/test_fixtures.json, but parse_args() does not define --fixture_config (or --test_data_dir / --shopping_fixtures_dir), causing argparse to fail with unrecognized arguments: --fixture_config=....
Also note that _sync_absl_flags() (line 147) calls flags.FLAGS(flag_args, known_only=True) before _load_tests() imports shopping.base, so any absl flags defined in shopping.base are not yet registered when FLAGS is parsed. Please add --fixture_config (and --test_data_dir / --shopping_fixtures_dir) to parse_args() and ensure FLAGS is synced after test modules are imported (or via FLAGS / environment variables).
There was a problem hiding this comment.
Both fixed in 0c4671f: the flags are accepted, and FLAGS is synced after the test modules are imported (see the replies on lines 147 and 179).
|
|
||
| server_profile: UcpProfile | None = None | ||
| try: | ||
| server_profile = fetch_server_profile(args.server_url) |
There was a problem hiding this comment.
README.md (line 224) notes that --dry-run lists selected tests "without contacting the server", but fetch_server_profile(args.server_url) runs unconditionally here before if args.dry_run: is checked at line 279. Skipping fetch_server_profile when args.dry_run is set (and resolving Path(__file__).resolve().parent / "platforms" / f"{args.platform}.yaml" at line 243 so --platform works outside the repo root) will make --dry-run and --platform behave as documented.
There was a problem hiding this comment.
Fixed in 0c4671f: the dry-run listing happens before discovery is fetched, so --dry-run makes no requests (checked against an unreachable URL), and --platform=<name> resolves against platforms/ next to runner.py (checked from another directory).
| Validated Paths: | ||
| - GET /.well-known/ucp | ||
| """ | ||
|
|
There was a problem hiding this comment.
Root binding_test.py on main contained TokenBindingTest.test_token_binding_completion, which tested payment credential token binding (credential.binding per schemas/common/types/binding.json) on POST /checkout-sessions/{id}/complete. Deleting root binding_test.py when adding ProtocolBindingTest here dropped test_token_binding_completion from the conformance suite. Please preserve TokenBindingTest (e.g., under common/payments/ alongside card_credential_test.py and ap2_test.py, or under shopping/checkout/) and register it in runner.py's SUITE_REGISTRY.
There was a problem hiding this comment.
Restored in e4bc51a as common/payments/token_binding_test.py and registered in SUITE_REGISTRY under common — see the reply on the deleted binding_test.py.
Review of Universal-Commerce-Protocol#107 found that the inputs a merchant passes were silently ignored or rejected: - `--conformance_input` and `--test_data_dir` were shadowed by the `shopping_*` flags, whose non-empty flower-shop defaults always won. The duplicate flags are gone; `conformance_input`, `fixture_config` and `test_data_dir` now live in `framework.base_test` with `None` defaults, and the shopping base class supplies its bundled fixtures only when nothing is given. - The conformance input was read after `BaseIntegrationTest.setUp()` had already started the agent profile server, so the agent advertised the default release instead of the merchant's. The base class now resolves the vertical's `default_conformance_input` before starting it. - `ucp-conformance` did not accept `--fixture_config`, `--test_data_dir` or `--verbose_http`, and parsed the absl flags before the test modules that define them were imported. The runner now loads the suites first and forwards every flag. - `--dry-run` fetched `/.well-known/ucp` before listing tests; it now returns before any request is made. `--platform=<name>` resolves against the `platforms/` directory next to `runner.py`, so it works outside the repository root. - `server_url` and `simulation_secret` default to `SERVER_URL` and `SIMULATION_SECRET` again, as on main, so the documented `uv run pytest` invocation works without flags.
Extracting `shopping/base.py` from `integration_test_utils.py` dropped two behaviours that main had: - `update_checkout_session()` no longer carried `buyer`, `fulfillment`, `discounts` and `platform` over from the current checkout when they were not passed (Universal-Commerce-Protocol#15). Because the update is a full-replacement PUT, a call such as `update_checkout_session(obj, discounts=...)` wiped the buyer and fulfillment state a test had set up. The fallbacks are back. - `create_checkout_payload()` hardcoded `"2026-04-08"` for the payment handler and request versions instead of the `ucp_version` declared in the conformance input (Universal-Commerce-Protocol#58). Both now use the base class's `ucp_version`, the same value the agent profile server advertises.
Five tests under `common/` were gated on capability names no server advertises (`dev.ucp.common.payments.ap2`, `dev.ucp.common.payments.cards`, `dev.ucp.common.webhooks`), so they skipped on every run, and several tests placed in `core/` depended on the shopping vertical. This commit puts each test where its real precondition is: - `Ap2MandateTest` requires `dev.ucp.common.payment.ap2_mandate`, the name the schema defines. Releases before 2026-08-25 advertised it as `dev.ucp.shopping.ap2_mandate`; `framework.discovery` records that rename so the test runs against either. - `CardCredentialTest` and the restored `TokenBindingTest` (dropped from main's `binding_test.py` when `core/binding_test.py` was added) are gated only on the checkout endpoint they drive: card credentials and token binding are payment data types, not advertised capabilities. - `WebhookTest` moves to `shopping/order/` and requires `dev.ucp.shopping.order`, whose config carries the `webhook_url`. - The checkout idempotency tests move to `shopping/checkout/`; the simulation-endpoint security tests move to `shopping/order/` in main's form (a real completed order, 200 with the correct secret). The GET-only variants left in `core/` could not fail and are removed. - `core/protocol_test.py` requires the top-level `ucp` object again (profile.json lists it as required) and no longer "negotiates" versions against the public discovery GET, which answers 200 for any `UCP-Agent` version; the 422 check stays with the shopping suite. - `core/binding_test.py` drops the Content-Type test: a POST to the GET-only discovery route is answered 405 before the header is read. The `common/` tests import `shopping.base` directly instead of the deprecated `integration_test_utils` facade.
`integration_test_utils.py` (the facade tests written before the tiered layout still import) and `shopping-agent-test.json` (the profile `framework/agent_profile_server.py` serves) were missing from the wheel's `only-include`, so an installed `ucp-conformance` could not find them. Both are listed now. With the template guaranteed to ship, the server's silent fallback to a hand-written profile that lacked the required `ucp.version` is gone; a missing template raises instead of advertising an invalid agent. The facade's `get_headers()` also sends an `Idempotency-Key` by default again, as main and `BaseIntegrationTest.get_headers()` do.
|
Thanks for the thorough review, @damaz91 — every point was valid. I've pushed the fixes as four commits so they can be checked one at a time:
Against the sample server the run goes from 81 tests / 75 passed / 6 skipped to 79 / 77 / 2: the five tests that skipped on non-existent capability names now run and pass, On the split: agreed on the three-PR shape and on landing the ready test PRs (or PR 1) before the directory move. I'll cut PR 1 from this branch once these fixes look right to you, so the split inherits them rather than forking the branch twice. Does that order work for you? |
|
Following up on the split: PR 1 is open as #120 and the other two are prepared on top of it, so the whole path is visible now:
Two small departures from the shape sketched above, both to keep rebases trivial for the open test PRs: the payment tests move straight to I'll open 2 and 3 as their bases merge, and I'm converting this PR to a draft so the review happens on the smaller ones. The review comments here are all resolved in the stack; happy to point at the specific commit for any of them. |
Third step of the RFC #520 split of Universal-Commerce-Protocol#107. Every shopping test now imports `ShoppingIntegrationTestBase` from `shopping.base` instead of the `integration_test_utils` facade, calls `self.get_headers()`, tolerates the SDK's older and newer locations for the payment models, and declares the capability it exercises with `@requires_capability`, so a server that does not advertise, say, `dev.ucp.shopping.discount` skips those tests instead of failing them. Test bodies are unchanged. `conftest.py` takes `FLAGS` from the framework for the same reason.
Third step of the RFC #520 split of Universal-Commerce-Protocol#107. Every shopping test now imports `ShoppingIntegrationTestBase` from `shopping.base` instead of the `integration_test_utils` facade, calls `self.get_headers()`, tolerates the SDK's older and newer locations for the payment models, and declares the capability it exercises with `@requires_capability`, so a server that does not advertise, say, `dev.ucp.shopping.discount` skips those tests instead of failing them. Test bodies are unchanged. `conftest.py` takes `FLAGS` from the framework for the same reason.
… #520, 1/3) (#120) * feat(framework): add the vertical-agnostic test framework First step of the RFC #520 split of #107. `framework/` holds the pieces of the conformance harness that do not depend on a business vertical: - `base_test.BaseIntegrationTest`: HTTP client, merchant inputs (`--conformance_input`, `--fixture_config`, `--test_data_dir`, with `SERVER_URL` / `SIMULATION_SECRET` honoured as environment variables), discovery profile caching, capability gating and the agent profile server lifecycle. A vertical's base class points `default_conformance_input` at its bundled fixture. - `discovery`: parses `/.well-known/ucp` into `UcpProfile`. - `decorators.requires_capability`: skips a test class or method when the server does not advertise a capability. - `platform`: platform certification profiles (required capabilities and payment handlers) loaded from YAML. - `agent_profile_server`, `mock_webhook_server`, `constants`: moved out of `integration_test_utils` unchanged in behaviour; the agent profile template is required rather than silently replaced by a fallback. Nothing uses the package yet; the next commits wire the runner and the existing helpers to it. Existing tests are not touched. * feat(runner): add the ucp-conformance runner and platform profiles `runner.py` (installed as `ucp-conformance`) selects suites or individual modules, fetches the server's discovery profile once and shares it with the tests, optionally validates a platform certification profile from `platforms/`, lists tests with `--dry-run` without contacting the server, and forwards every merchant input flag to the suites after importing them, so suite-defined flags are known to absl. The registry lists today's root test modules under a single `shopping` suite; `core` and `common` follow once those tiers exist. CI runs the suite through the runner instead of invoking each module in a loop. The wheel now lists its top-level modules and data explicitly so the console script works from an installed package. * refactor: build integration_test_utils on the framework The shopping helpers keep their module, class names and functions, so no test file changes, but the pieces that are not shopping-specific now come from `framework/`: the command-line flags, `AgentProfileServer`, `MockWebhookServer`, the HTTP client and discovery handling in `BaseIntegrationTest`. `IntegrationTestBase` is the shopping subclass, which also gates every test on the `dev.ucp.shopping.checkout` capability and points the framework at the flower-shop input when none is given. Compared with the previous module the helpers behave the same, with two small improvements carried over from review: `update_checkout_session` defaults a destination's `type` to `shipping_address`, and missing fixture files are logged instead of silently ignored. * docs: describe the ucp-conformance runner * fix(framework): release the client and agent server when setUp() skips unittest does not run tearDown() when setUp() raises, and SkipTest is an exception, so a test skipped from setUp() left its AgentProfileServer listening on port 8285 and its HTTP client open. Every later test in the process then failed to bind the port (or, with the leaked server still answering /healthz, silently talked to the stale one). setUp() now checks everything that can skip the test before creating any resource, and registers each resource with addCleanup(), which unittest runs in every outcome. The tearDown() override is no longer needed. Verified against a discovery-only server that advertises no capabilities: all 77 tests skip in a few milliseconds with no bind errors (previously 19 of 20 tests in a three-module run hit "address already in use"). * fix(framework): keep inherited capability requirements per class requires_capability() tested hasattr(), which walks the MRO, and then mutated the set it found. Decorating a subclass whose base class was already decorated therefore added the subclass's capability to the base class's set, and with it to every sibling. The decorator now builds a new set on the decorated object from whatever it inherits, so requirements accumulate down the hierarchy without touching the parent. With that in place the shopping base class declares its own requirement, @requires_capability("dev.ucp.shopping.checkout"), instead of checking the profile after super().setUp() had already started the agent profile server. BaseIntegrationTest evaluates it before any resource is created, so a non-shopping server skips the whole suite without spinning up and tearing down the mock agent for each test. * fix(runner): exit non-zero when a specifier or filter selects no tests An empty TestSuite reports wasSuccessful(), so `--suite=typo`, a dotted path whose class does not exist, or a `-k` filter that matches nothing all exited 0. _load_tests() now returns the specifiers it could not load, and main() exits 1 for those and for a selection of zero tests. A dotted path whose attribute is missing or is not a TestCase subclass logs an error instead of falling through silently. * fix(framework): spell out required_caps and services_data for cspell * chore(platforms): add the license header to google.yaml
Third step of the RFC #520 split of Universal-Commerce-Protocol#107. Every shopping test now imports `ShoppingIntegrationTestBase` from `shopping.base` instead of the `integration_test_utils` facade, calls `self.get_headers()`, tolerates the SDK's older and newer locations for the payment models, and declares the capability it exercises with `@requires_capability`, so a server that does not advertise, say, `dev.ucp.shopping.discount` skips those tests instead of failing them. Test bodies are unchanged. `conftest.py` takes `FLAGS` from the framework for the same reason.
Fix: harden these four workflows together: pin every referenced Action to its current tag's full commit SHA, add permissions: contents: read to the pre-commit job, disable credential persistence on the read-only checkouts, and give the pure signal job permissions: {}. linter.yaml keeps its packages/statuses permissions and spellcheck.yaml keeps its job-level contents/actions reads, both unchanged; triggers and job commands are untouched. conformance-tests.yml is already pinned and is excluded because open PR #107 modifies it.
…oves only) Second step of the RFC #520 split of Universal-Commerce-Protocol#107. Every test module moves to the package that owns it, with no content changes, so git tracks each file as a 100% rename and open pull requests rebase onto the new paths: - checkout, discount, fulfillment, order and validation tests go under `shopping/<domain>/`; `test_data/flower_shop` becomes `shopping/fixtures/flower_shop` (a symlink keeps the old path working). - the payment credential tests (`ap2_test`, `card_credential_test`, `binding_test` as `token_binding_test`) go under `common/payments/`. The tests still import `integration_test_utils`; the runner registry, CI and README follow the files.
Third step of the RFC #520 split of Universal-Commerce-Protocol#107. Every shopping test now imports `ShoppingIntegrationTestBase` from `shopping.base` instead of the `integration_test_utils` facade, calls `self.get_headers()`, tolerates the SDK's older and newer locations for the payment models, and declares the capability it exercises with `@requires_capability`, so a server that does not advertise, say, `dev.ucp.shopping.discount` skips those tests instead of failing them. Test bodies are unchanged. `conftest.py` takes `FLAGS` from the framework for the same reason.
…oves only) (RFC #520, 2/3) (#121) * refactor: move the shopping tests and fixtures into shopping/ (file moves only) Second step of the RFC #520 split of #107. Every test module moves to the package that owns it, with no content changes, so git tracks each file as a 100% rename and open pull requests rebase onto the new paths: - checkout, discount, fulfillment, order and validation tests go under `shopping/<domain>/`; `test_data/flower_shop` becomes `shopping/fixtures/flower_shop` (a symlink keeps the old path working). - the payment credential tests (`ap2_test`, `card_credential_test`, `binding_test` as `token_binding_test`) go under `common/payments/`. The tests still import `integration_test_utils`; the runner registry, CI and README follow the files. * refactor: move the shopping helpers to shopping/base.py behind a facade `integration_test_utils.py` becomes `shopping/base.py` (git shows it as a 94% copy because a facade keeps the old path). The facade re-exports every name tests import from `integration_test_utils`, so the moved tests and any open pull request keep working unchanged; it is not yet marked deprecated because the in-tree tests still use it. The shopping fixtures default to `shopping/fixtures/flower_shop`, and the wheel lists the `common` and `shopping` packages instead of the former top-level test modules. * chore: re-trigger CI after the GitHub Actions incident of 2026-11-09
Third step of the RFC #520 split of Universal-Commerce-Protocol#107. Every shopping test now imports `ShoppingIntegrationTestBase` from `shopping.base` instead of the `integration_test_utils` facade, calls `self.get_headers()`, tolerates the SDK's older and newer locations for the payment models, and declares the capability it exercises with `@requires_capability`, so a server that does not advertise, say, `dev.ucp.shopping.discount` skips those tests instead of failing them. Test bodies are unchanged. `conftest.py` takes `FLAGS` from the framework for the same reason.
…FC #520, 3/3) (#122) * refactor(shopping): use shopping.base directly and declare capabilities Third step of the RFC #520 split of #107. Every shopping test now imports `ShoppingIntegrationTestBase` from `shopping.base` instead of the `integration_test_utils` facade, calls `self.get_headers()`, tolerates the SDK's older and newer locations for the payment models, and declares the capability it exercises with `@requires_capability`, so a server that does not advertise, say, `dev.ucp.shopping.discount` skips those tests instead of failing them. Test bodies are unchanged. `conftest.py` takes `FLAGS` from the framework for the same reason. * fix(common): gate the payment tests on capabilities that exist The three payment credential tests under `common/payments/` were gated on capability names no server advertises, so they always skipped. Now: - `Ap2MandateTest` requires `dev.ucp.common.payment.ap2_mandate`, the name the schema defines. Releases before 2026-08-25 advertised it as `dev.ucp.shopping.ap2_mandate`; `framework.discovery` records that rename (`FORMER_NAMES`) so the test runs against either generation of server. - `CardCredentialTest` and `TokenBindingTest` are gated on the checkout endpoint they drive: card credentials and token binding are payment data types, not advertised capabilities. They import `shopping.base` directly, which leaves `integration_test_utils` without in-tree users; it now emits a DeprecationWarning for external ones. * feat(core): extract the vertical-agnostic protocol tests `core/` holds the tests any UCP server must pass regardless of the verticals it implements, which is everything observable from the discovery document alone: - `core/protocol_test.py`: the discovery checks from the former shopping protocol test (required `ucp` wrapper, date-based version matching the conformance input, reverse-DNS capability and handler names, payment handler structure, published spec/schema URLs). - `core/binding_test.py`: the standard transport headers are accepted and discovery answers with `application/json`. `shopping/checkout/protocol_test.py` keeps what needs the shopping vertical: the service advertisement checks and version negotiation, which can only be observed on `POST /checkout-sessions` (422 for an unsupported version). The runner gains the `core` suite and the README documents the tiers. * refactor(shopping): group the discount tests under shopping/discount/ The discount flow tests (`test_discount_flow`, `test_multiple_discounts_*`, `test_fixed_amount_discount`) move from the checkout business-logic module to `shopping/discount/discount_test.py`, and `test_unknown_discount_code` joins them from the validation suite, so every discount test lives with the capability it exercises and is gated on `dev.ucp.shopping.discount`. `shopping/discount/semantics_test.py` (the former `discount_test.py`) is unchanged. Test bodies are unchanged; only their module and gate move. * refactor(shopping): gate two extension tests and refresh two docs (review) Review feedback on #122: - test_complete_without_fulfillment requires dev.ucp.shopping.fulfillment: on a server without the extension, completing without fulfillment is a legitimate 200, so the test now skips there instead of failing. - test_discount_entry_is_negative requires dev.ucp.shopping.discount, so it no longer sends an unnegotiated discounts field to a server that does not advertise the extension and validates strictly. - core/__init__.py: the idempotency and simulation-URL tests live under shopping/, so the docstring names only transport binding and discovery. - README and --suite help: a test module, class or file path is also accepted, with examples.
Summary
Refactors the conformance suite from a flat, shopping-specific layout into a vertical-agnostic, tiered architecture, so that verticals beyond shopping can be certified without forking the framework (RFC #520).
This branch has been rebased onto the latest
mainand integratesmain's newer and rewritten tests into the new tier layout.Architecture
Tests are organised into tiers:
framework/core/common/shopping/Suites are selected declaratively via
SUITE_REGISTRYinrunner.pyand exposed through aucp-conformanceconsole script:Capability gating is now declarative. Tests annotate their requirements with
@requires_capability("dev.ucp.shopping.order")instead of hand-rolling a discovery probe insetUp, so a business that does not advertise a capability sees a skip rather than a failure.Integrating
main's testsThree of the four commits originally on this branch had already been squash-merged into
main, so only the refactor commit is replayed here.main's newer tests were moved into the tier layout:totals_test.pyshopping/checkout/totals_test.pyfulfillment_structure_test.pyshopping/fulfillment/structure_test.pywebhook_structure_test.pyshopping/order/webhook_structure_test.pydiscount_test.pyshopping/discount/semantics_test.pyprotocol_test.pycore/protocol_test.pyandshopping/checkout/protocol_test.pybusiness_logic_test.pyshopping/checkout/business_logic_test.py(discount cases moved to the discount tier)These appear as renames rather than deletions in the diff.
A note on one placement:
webhook_structure_test.pylanded undershopping/order/rather than a sharedcommon/webhooks/module. Although it exercises webhook delivery mechanics, every case drivescreate_checkout_session/complete_checkout_sessionand gates ondev.ucp.shopping.order, so it is not vertical-agnostic in its current form. The genuinely shared webhook assertions live incommon/webhooks/.integration_test_utils.pyis retained as a thin deprecation facade that re-exports the relocated symbols, so any out-of-tree tests importing it continue to work.SDK compatibility
Several model types moved from
ucp_sdk.models.schemas.shoppingtoucp_sdk.models.common.typesbetween SDK 0.4.6 and 0.5.0. Imports for the affected types are now wrapped in atry/except ImportErrorfallback, so the suite imports and runs cleanly against both. Onmainthe suite cannot currently be collected under 0.5.0.CI workflow
Two changes to
.github/workflows/conformance-tests.yml:for test_file in *_test.pyglob loop is replaced by a singleuv run ucp-conformance --suite=allinvocation.--platform=googleis replaced with--conformance_input=shopping/fixtures/flower_shop/conformance_input.json.On (2): enforcing a single vendor's certification profile in the upstream repository's own CI would cause unrelated PRs to fail whenever that profile's mandates move ahead of the reference server. Running against the reference server's own declared conformance input keeps the signal attributable to the change under review. Vendor profiles remain fully supported via
--platformfor downstream certification runs.Verification
Run against the reference server under the exact CI configuration (python-sdk
v2026-04-08-6, samples atmain):main(baseline)Both branches fail the same two tests —
order_test.py::test_order_fulfillment_retrievalandorder_test.py::test_order_update— with a500fromPUT /checkout-sessions/{id}. These are pre-existing onmainand are not introduced by this PR.pre-commit run --all-filespasses.PR description covering the tier architecture, how main's tests were remapped, SDK compatibility shims, CI changes with rationale, and a verification table showing no regressions.