Repository navigation
feat: add the vertical-agnostic conformance framework and runner (RFC #520, 1/3) - #120
Conversation
First step of the RFC #520 split of Universal-Commerce-Protocol#107. `framework/` holds the pieces of the conformance harness that do not depend on a business vertical: - `base_test.BaseIntegrationTest`: HTTP client, merchant inputs (`--conformance_input`, `--fixture_config`, `--test_data_dir`, with `SERVER_URL` / `SIMULATION_SECRET` honoured as environment variables), discovery profile caching, capability gating and the agent profile server lifecycle. A vertical's base class points `default_conformance_input` at its bundled fixture. - `discovery`: parses `/.well-known/ucp` into `UcpProfile`. - `decorators.requires_capability`: skips a test class or method when the server does not advertise a capability. - `platform`: platform certification profiles (required capabilities and payment handlers) loaded from YAML. - `agent_profile_server`, `mock_webhook_server`, `constants`: moved out of `integration_test_utils` unchanged in behaviour; the agent profile template is required rather than silently replaced by a fallback. Nothing uses the package yet; the next commits wire the runner and the existing helpers to it. Existing tests are not touched.
`runner.py` (installed as `ucp-conformance`) selects suites or individual modules, fetches the server's discovery profile once and shares it with the tests, optionally validates a platform certification profile from `platforms/`, lists tests with `--dry-run` without contacting the server, and forwards every merchant input flag to the suites after importing them, so suite-defined flags are known to absl. The registry lists today's root test modules under a single `shopping` suite; `core` and `common` follow once those tiers exist. CI runs the suite through the runner instead of invoking each module in a loop. The wheel now lists its top-level modules and data explicitly so the console script works from an installed package.
The shopping helpers keep their module, class names and functions, so no test file changes, but the pieces that are not shopping-specific now come from `framework/`: the command-line flags, `AgentProfileServer`, `MockWebhookServer`, the HTTP client and discovery handling in `BaseIntegrationTest`. `IntegrationTestBase` is the shopping subclass, which also gates every test on the `dev.ucp.shopping.checkout` capability and points the framework at the flower-shop input when none is given. Compared with the previous module the helpers behave the same, with two small improvements carried over from review: `update_checkout_session` defaults a destination's `type` to `shipping_address`, and missing fixture files are logged instead of silently ignored.
damaz91
left a comment
There was a problem hiding this comment.
Thanks @segiodongo for splitting #107 into focused PRs and addressing all of the earlier feedback on flag forwarding, setUp() ordering, --dry-run, update_checkout_session() field preservation, and wheel packaging!
I verified the full suite locally (ucp-conformance --platform=google --suite=all) against the reference Python sample server (76 passed, 1 skipped, 0 failures).
There are 3 small issues remaining around resource cleanup when setUp() skips, @requires_capability class inheritance, and runner.py exit status on invalid --suite specifiers — see the inline comments below. Once those are resolved, this looks ready to merge!
unittest does not run tearDown() when setUp() raises, and SkipTest is an exception, so a test skipped from setUp() left its AgentProfileServer listening on port 8285 and its HTTP client open. Every later test in the process then failed to bind the port (or, with the leaked server still answering /healthz, silently talked to the stale one). setUp() now checks everything that can skip the test before creating any resource, and registers each resource with addCleanup(), which unittest runs in every outcome. The tearDown() override is no longer needed. Verified against a discovery-only server that advertises no capabilities: all 77 tests skip in a few milliseconds with no bind errors (previously 19 of 20 tests in a three-module run hit "address already in use").
requires_capability() tested hasattr(), which walks the MRO, and then
mutated the set it found. Decorating a subclass whose base class was
already decorated therefore added the subclass's capability to the base
class's set, and with it to every sibling. The decorator now builds a new
set on the decorated object from whatever it inherits, so requirements
accumulate down the hierarchy without touching the parent.
With that in place the shopping base class declares its own requirement,
@requires_capability("dev.ucp.shopping.checkout"), instead of checking the
profile after super().setUp() had already started the agent profile
server. BaseIntegrationTest evaluates it before any resource is created,
so a non-shopping server skips the whole suite without spinning up and
tearing down the mock agent for each test.
An empty TestSuite reports wasSuccessful(), so `--suite=typo`, a dotted path whose class does not exist, or a `-k` filter that matches nothing all exited 0. _load_tests() now returns the specifiers it could not load, and main() exits 1 for those and for a selection of zero tests. A dotted path whose attribute is missing or is not a TestCase subclass logs an error instead of falling through silently.
|
Thanks for the careful pass, @damaz91 — all three points are addressed in |
damaz91
left a comment
There was a problem hiding this comment.
Thanks @segiodongo for the clean fixes in 1fa2024, 3bc67a4, and 752baae and for keeping the stacked branches synced!
Re-verified locally against both the sample server (76 passed, 1 skipped) and a capability-less discovery server (77 skipped with zero agent server binds or port leaks), as well as the invalid/empty --suite and -k exit codes.
Left one minor, non-blocking comment inline on the license header for platforms/google.yaml that can be picked up in one of the follow-up PRs. LGTM!
| @@ -0,0 +1,12 @@ | |||
| platform: "google" | |||
There was a problem hiding this comment.
Nit (non-blocking — feel free to pick this up in PR 2 or PR 3 so we don't reset approvals here): let's include the standard # Copyright 2026 UCP Authors Apache 2.0 license header at the top of platforms/google.yaml.
Asked for in the review of Universal-Commerce-Protocol#120 (non-blocking), folded in here so that PR's approval stands.
|
Hi @segiodongo, thanks for submitting this PR. spellcheck is failing on 752baae. There are 7 cspell hits: reqs in framework/base_test.py (lines 186–194) and svcs in framework/discovery.py (lines 110–112). Could you please rename them (e.g. required_caps, services) or add them to .cspell/custom-words.txt so it passes this check. Thanks |
|
Thanks @carolinerg1! Renamed |
|
Once this framework lands, we would like to add catalog and identity-linking tests using |
f33ae54
into
Universal-Commerce-Protocol:main
Rebased onto the framework split (Universal-Commerce-Protocol#120). The AP2 Mandates profile route moves from the framework AgentProfileServer into a shopping-layer subclass, Ap2AgentProfileServer, which Ap2MandateTest starts in place of the default server. Headers use the base class custom_profile_url. Test bodies and the AP2 helpers are unchanged from 9e2f030. Signed-off-by: Tymofii Pidlisnyi <tima@aeoess.com>
Universal-Commerce-Protocol#120 replaced the CI loop over `*_test.py` with `ucp-conformance --suite=all`, which resolves through `SUITE_REGISTRY`. A module the registry does not name contributes no cases to that run, so this file would have merged and never executed: `--suite=all` resolved 77 cases on this branch with the module present and selected none of them. Main resolves 78; this head predates Universal-Commerce-Protocol#117. With the module registered the same command resolves 80 and selects its three cases. Removing the line again returns both to 77 and zero.
Universal-Commerce-Protocol#120 replaced the CI loop over `*_test.py` with `ucp-conformance --suite=all`, which resolves through `SUITE_REGISTRY`. A module the registry does not name contributes no cases to that run, so this file would have merged and never executed. Rebased onto main and registered. Measured on this head: `--suite=all` selects the module's cases, and removing the line again selects none.
Universal-Commerce-Protocol#120 replaced the CI loop over `*_test.py` with `ucp-conformance --suite=all`, which resolves through `SUITE_REGISTRY`. A module the registry does not name contributes no cases to that run, so this file would have merged and never executed. Rebased onto main and registered. Measured on this head: `--suite=all` selects the module's cases, and removing the line again selects none.
Universal-Commerce-Protocol#120 replaced the CI loop over `*_test.py` with `ucp-conformance --suite=all`, which resolves through `SUITE_REGISTRY`. A module the registry does not name contributes no cases to that run, so this file would have merged and never executed. Rebased onto main and registered. Measured on this head: `--suite=all` selects the module's cases, and removing the line again selects none.
Universal-Commerce-Protocol#120 replaced the CI loop over `*_test.py` with `ucp-conformance --suite=all`, which resolves through `SUITE_REGISTRY`. A module the registry does not name contributes no cases to that run, so this file would have merged and never executed. Rebased onto main and registered. Measured on this head: `--suite=all` selects the module's cases, and removing the line again selects none.
Three cases asserting that a create does not adopt or persist members the request schema marks `omit`, and that a create without them is unaffected. Placed as `shopping/checkout/field_ownership_test.py` and registered as `shopping.checkout.field_ownership_test`, following the layout Universal-Commerce-Protocol#121 landed and dropping the redundant `checkout_` prefix the way `checkout_lifecycle_test` became `shopping.checkout.lifecycle_test`. The path and the module name are a suggestion and easy to change. Universal-Commerce-Protocol#120 made `SUITE_REGISTRY` the only path `--suite=all` runs, so the entry is what makes the module execute at all: without it `--suite=all` resolves 78 cases and selects none of these three, and with it 81. Run against the sample server from samples main: 3 passed on their own, and 81 passed for the whole suite with one skip elsewhere.
Four cases over cart conversion: conversion uses cart contents over the payload, a second conversion returns the existing checkout, cancel returns the cart state before deletion, and cart discount codes carry forward. Placed as `shopping/cart/cart_test.py` in a new `shopping/cart/` package, following the layout Universal-Commerce-Protocol#121 landed, and registered as `shopping.cart.cart_test`. The earlier description proposed this path once the relocation merged. Universal-Commerce-Protocol#120 made `SUITE_REGISTRY` the only path `--suite=all` runs, so the entry is what makes the module execute: without it `--suite=all` resolves 78 cases and selects none of these four, and with it 82. Run against the sample server from samples main: 4 passed.
Seven cases over the payment terms extension, asserted against the raw wire payload: schedule sums, term id uniqueness, selected_term_id resolution, selection handling, and accepted_term on the order. Placed as `common/payments/terms_test.py` and registered as `common.payments.terms_test`, following the layout Universal-Commerce-Protocol#121 landed. Payment terms is a common payment extension, which is why it sits beside the other payment modules rather than under `shopping/`. Universal-Commerce-Protocol#120 made `SUITE_REGISTRY` the only path `--suite=all` runs, so the entry is what makes the module reachable: without it `--suite=all` selects none of these seven. Run against the sample server from samples main: 7 skipped, for the reason the description gives, since nothing in the reference advertises `dev.ucp.common.payment.terms`.
Three cases asserting that a create does not adopt or persist members the request schema marks `omit`, and that a create without them is unaffected. Placed as `shopping/checkout/field_ownership_test.py` and registered as `shopping.checkout.field_ownership_test`, following the layout #121 landed and dropping the redundant `checkout_` prefix the way `checkout_lifecycle_test` became `shopping.checkout.lifecycle_test`. The path and the module name are a suggestion and easy to change. #120 made `SUITE_REGISTRY` the only path `--suite=all` runs, so the entry is what makes the module execute at all: without it `--suite=all` resolves 78 cases and selects none of these three, and with it 81. Run against the sample server from samples main: 3 passed on their own, and 81 passed for the whole suite with one skip elsewhere.
First of three PRs that replace #107, as discussed there. This one adds the framework and the runner and leaves every test file where it is, so the open test PRs rebase onto it without conflicts.
What's in it
framework/:BaseIntegrationTest(HTTP client, merchant input flags, discovery caching, capability gating, agent profile server lifecycle),discovery.UcpProfile,requires_capability, platform certification profiles, andAgentProfileServer/MockWebhookServermoved out ofintegration_test_utilsunchanged.runner.py, installed asucp-conformance: suite or module selection, one discovery fetch shared with the tests, optional--platformvalidation,--dry-runthat makes no requests, and every merchant input flag forwarded to the suites after they are imported.integration_test_utils.pyis rebuilt on the framework with its public names intact (IntegrationTestBase,get_headers,test_data,FLAGS, …). The root test modules are untouched and run both through the runner and throughpytest.platforms/google.yaml, the console script and wheel contents inpyproject.toml, CI switched to the runner, README section for the runner.Review feedback from #107 addressed here
--conformance_input,--fixture_config,--test_data_dirand--verbose_httpare accepted by the runner and reach the tests; the input flags are defined once inframework.base_testwithNonedefaults and the shopping helpers fall back to the flower-shop fixtures only when nothing is given.ucp_version.SERVER_URL/SIMULATION_SECRETwork as environment variables again;--dry-runis offline;--platform=<name>resolves against the packagedplatforms/directory.update_checkout_session()keeps carryingbuyer/fulfillment/discounts/platformover, and the payload versions followucp_versionfrom the input, as onmain(fix: update_checkout_session to handle all optional args correctly. #15, feat: drive protocol_test version and capability asserts from conformance_input #58).integration_test_utils.pyandshopping-agent-test.jsonare in the wheel; the invalid agent-profile fallback is gone.Verification (sample server, 2026-04-08):
ucp-conformance --suite=allruns main's 77 tests: 76 passed, 1 skipped (test_discovery_urls, already marked skip), 0 failed — same asmain.SERVER_URL=… uv run pytestruns them too. Wheel built and inspected; each commit runs on its own.Next: PR 2 moves the test files into
shopping/andcommon/with no content changes; PR 3 extractscore/and applies the capability gating. Both are prepared on top of this branch (see the compare links in #107).