Skip to content

feat: add the vertical-agnostic conformance framework and runner (RFC #520, 1/3) - #120

Merged
carolinerg1 merged 9 commits into
Universal-Commerce-Protocol:mainfrom
segiodongo:split/1-framework-runner
Nov 5, 2026
Merged

carolinerg1 merged 9 commits into
Universal-Commerce-Protocol:mainfrom
segiodongo:split/1-framework-runner

Conversation

@segiodongo

Copy link
Copy Markdown
Collaborator

First of three PRs that replace #107, as discussed there. This one adds the framework and the runner and leaves every test file where it is, so the open test PRs rebase onto it without conflicts.

What's in it

  • framework/: BaseIntegrationTest (HTTP client, merchant input flags, discovery caching, capability gating, agent profile server lifecycle), discovery.UcpProfile, requires_capability, platform certification profiles, and AgentProfileServer / MockWebhookServer moved out of integration_test_utils unchanged.
  • runner.py, installed as ucp-conformance: suite or module selection, one discovery fetch shared with the tests, optional --platform validation, --dry-run that makes no requests, and every merchant input flag forwarded to the suites after they are imported.
  • integration_test_utils.py is rebuilt on the framework with its public names intact (IntegrationTestBase, get_headers, test_data, FLAGS, …). The root test modules are untouched and run both through the runner and through pytest.
  • platforms/google.yaml, the console script and wheel contents in pyproject.toml, CI switched to the runner, README section for the runner.

Review feedback from #107 addressed here

  • --conformance_input, --fixture_config, --test_data_dir and --verbose_http are accepted by the runner and reach the tests; the input flags are defined once in framework.base_test with None defaults and the shopping helpers fall back to the flower-shop fixtures only when nothing is given.
  • The conformance input is resolved before the agent profile server starts, so the agent advertises the merchant's ucp_version.
  • SERVER_URL / SIMULATION_SECRET work as environment variables again; --dry-run is offline; --platform=<name> resolves against the packaged platforms/ directory.
  • update_checkout_session() keeps carrying buyer / fulfillment / discounts / platform over, and the payload versions follow ucp_version from the input, as on main (fix: update_checkout_session to handle all optional args correctly. #15, feat: drive protocol_test version and capability asserts from conformance_input #58).
  • integration_test_utils.py and shopping-agent-test.json are in the wheel; the invalid agent-profile fallback is gone.

Verification (sample server, 2026-04-08): ucp-conformance --suite=all runs main's 77 tests: 76 passed, 1 skipped (test_discovery_urls, already marked skip), 0 failed — same as main. SERVER_URL=… uv run pytest runs them too. Wheel built and inspected; each commit runs on its own.

Next: PR 2 moves the test files into shopping/ and common/ with no content changes; PR 3 extracts core/ and applies the capability gating. Both are prepared on top of this branch (see the compare links in #107).

First step of the RFC #520 split of Universal-Commerce-Protocol#107. `framework/` holds the pieces
of the conformance harness that do not depend on a business vertical:

- `base_test.BaseIntegrationTest`: HTTP client, merchant inputs
  (`--conformance_input`, `--fixture_config`, `--test_data_dir`, with
  `SERVER_URL` / `SIMULATION_SECRET` honoured as environment variables),
  discovery profile caching, capability gating and the agent profile
  server lifecycle. A vertical's base class points
  `default_conformance_input` at its bundled fixture.
- `discovery`: parses `/.well-known/ucp` into `UcpProfile`.
- `decorators.requires_capability`: skips a test class or method when the
  server does not advertise a capability.
- `platform`: platform certification profiles (required capabilities and
  payment handlers) loaded from YAML.
- `agent_profile_server`, `mock_webhook_server`, `constants`: moved out of
  `integration_test_utils` unchanged in behaviour; the agent profile
  template is required rather than silently replaced by a fallback.

Nothing uses the package yet; the next commits wire the runner and the
existing helpers to it. Existing tests are not touched.
`runner.py` (installed as `ucp-conformance`) selects suites or individual
modules, fetches the server's discovery profile once and shares it with
the tests, optionally validates a platform certification profile from
`platforms/`, lists tests with `--dry-run` without contacting the server,
and forwards every merchant input flag to the suites after importing
them, so suite-defined flags are known to absl.

The registry lists today's root test modules under a single `shopping`
suite; `core` and `common` follow once those tiers exist. CI runs the
suite through the runner instead of invoking each module in a loop.

The wheel now lists its top-level modules and data explicitly so the
console script works from an installed package.
The shopping helpers keep their module, class names and functions, so no
test file changes, but the pieces that are not shopping-specific now come
from `framework/`: the command-line flags, `AgentProfileServer`,
`MockWebhookServer`, the HTTP client and discovery handling in
`BaseIntegrationTest`. `IntegrationTestBase` is the shopping subclass,
which also gates every test on the `dev.ucp.shopping.checkout` capability
and points the framework at the flower-shop input when none is given.

Compared with the previous module the helpers behave the same, with two
small improvements carried over from review: `update_checkout_session`
defaults a destination's `type` to `shipping_address`, and missing fixture
files are logged instead of silently ignored.

@damaz91 damaz91 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @segiodongo for splitting #107 into focused PRs and addressing all of the earlier feedback on flag forwarding, setUp() ordering, --dry-run, update_checkout_session() field preservation, and wheel packaging!

I verified the full suite locally (ucp-conformance --platform=google --suite=all) against the reference Python sample server (76 passed, 1 skipped, 0 failures).

There are 3 small issues remaining around resource cleanup when setUp() skips, @requires_capability class inheritance, and runner.py exit status on invalid --suite specifiers — see the inline comments below. Once those are resolved, this looks ready to merge!

Comment thread framework/base_test.py Resolved
Comment thread integration_test_utils.py Resolved
Comment thread framework/decorators.py Resolved
Comment thread runner.py Resolved
unittest does not run tearDown() when setUp() raises, and SkipTest is an
exception, so a test skipped from setUp() left its AgentProfileServer
listening on port 8285 and its HTTP client open. Every later test in the
process then failed to bind the port (or, with the leaked server still
answering /healthz, silently talked to the stale one).

setUp() now checks everything that can skip the test before creating any
resource, and registers each resource with addCleanup(), which unittest
runs in every outcome. The tearDown() override is no longer needed.

Verified against a discovery-only server that advertises no capabilities:
all 77 tests skip in a few milliseconds with no bind errors (previously 19
of 20 tests in a three-module run hit "address already in use").
requires_capability() tested hasattr(), which walks the MRO, and then
mutated the set it found. Decorating a subclass whose base class was
already decorated therefore added the subclass's capability to the base
class's set, and with it to every sibling. The decorator now builds a new
set on the decorated object from whatever it inherits, so requirements
accumulate down the hierarchy without touching the parent.

With that in place the shopping base class declares its own requirement,
@requires_capability("dev.ucp.shopping.checkout"), instead of checking the
profile after super().setUp() had already started the agent profile
server. BaseIntegrationTest evaluates it before any resource is created,
so a non-shopping server skips the whole suite without spinning up and
tearing down the mock agent for each test.
An empty TestSuite reports wasSuccessful(), so `--suite=typo`, a dotted
path whose class does not exist, or a `-k` filter that matches nothing all
exited 0. _load_tests() now returns the specifiers it could not load, and
main() exits 1 for those and for a selection of zero tests. A dotted path
whose attribute is missing or is not a TestCase subclass logs an error
instead of falling through silently.
@segiodongo

Copy link
Copy Markdown
Collaborator Author

Thanks for the careful pass, @damaz91 — all three points are addressed in 1fa2024, 3bc67a4 and 752baae (one commit per theme; details in the inline replies). #121 and the PR 3 branch are rebased on top so the stack stays consistent. Full suite against the sample server is unchanged at 76 passed / 1 skipped, and against a server with no capabilities every test now skips cleanly without the port leak.

@segiodongo
segiodongo requested a review from damaz91 November 2, 2026 18:07

@damaz91 damaz91 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @segiodongo for the clean fixes in 1fa2024, 3bc67a4, and 752baae and for keeping the stacked branches synced!

Re-verified locally against both the sample server (76 passed, 1 skipped) and a capability-less discovery server (77 skipped with zero agent server binds or port leaks), as well as the invalid/empty --suite and -k exit codes.

Left one minor, non-blocking comment inline on the license header for platforms/google.yaml that can be picked up in one of the follow-up PRs. LGTM!

Comment thread platforms/google.yaml
@@ -0,0 +1,12 @@
platform: "google"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit (non-blocking — feel free to pick this up in PR 2 or PR 3 so we don't reset approvals here): let's include the standard # Copyright 2026 UCP Authors Apache 2.0 license header at the top of platforms/google.yaml.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Added in #121 as 78f413a so this approval stands; it will reach main with that PR.

@damaz91 damaz91 added status:under-review and removed status:needs-triage Signal that the PR is ready for human triage labels Nov 3, 2026
segiodongo added a commit to segiodongo/conformance that referenced this pull request Nov 3, 2026
Asked for in the review of Universal-Commerce-Protocol#120 (non-blocking), folded in here so that PR's approval stands.
@carolinerg1

Copy link
Copy Markdown

Hi @segiodongo, thanks for submitting this PR.

spellcheck is failing on 752baae. There are 7 cspell hits: reqs in framework/base_test.py (lines 186–194) and svcs in framework/discovery.py (lines 110–112). Could you please rename them (e.g. required_caps, services) or add them to .cspell/custom-words.txt so it passes this check. Thanks

@segiodongo

Copy link
Copy Markdown
Collaborator Author

Thanks @carolinerg1! Renamed reqs to required_caps in framework/base_test.py and svcs to services_data in framework/discovery.py in fcd75de, and also included the Apache 2.0 license header on platforms/google.yaml (66571c5) from @damaz91's inline nit. Restacked #121 on top as well.

@evoleinik

Copy link
Copy Markdown

Once this framework lands, we would like to add catalog and identity-linking tests using requires_capability. They would cover anonymous search and lookup, the identity_required 401 with and without error="invalid_token", and the 403 path. They would also cover the account-status cases proposed in Universal-Commerce-Protocol/ucp#1047. The flower-shop sample server has no catalog, so these would skip in CI here. We can run them against a live B2B seller and share the results. Would you rather we wait for parts 2 and 3?

@carolinerg1
carolinerg1 merged commit f33ae54 into Universal-Commerce-Protocol:main Nov 5, 2026
16 checks passed
aeoess added a commit to aeoess/conformance that referenced this pull request Nov 9, 2026
Rebased onto the framework split (Universal-Commerce-Protocol#120). The AP2 Mandates profile route
moves from the framework AgentProfileServer into a shopping-layer
subclass, Ap2AgentProfileServer, which Ap2MandateTest starts in place of
the default server. Headers use the base class custom_profile_url.
Test bodies and the AP2 helpers are unchanged from 9e2f030.

Signed-off-by: Tymofii Pidlisnyi <tima@aeoess.com>
vishkaty added a commit to vishkaty/conformance that referenced this pull request Nov 10, 2026
Universal-Commerce-Protocol#120 replaced the CI loop over `*_test.py` with `ucp-conformance --suite=all`,
which resolves through `SUITE_REGISTRY`. A module the registry does not name
contributes no cases to that run, so this file would have merged and never
executed: `--suite=all` resolved 77 cases on this branch with the module
present and selected none of them. Main resolves 78; this head predates Universal-Commerce-Protocol#117.

With the module registered the same command resolves 80 and selects its three
cases. Removing the line again returns both to 77 and zero.
vishkaty added a commit to vishkaty/conformance that referenced this pull request Nov 10, 2026
Universal-Commerce-Protocol#120 replaced the CI loop over `*_test.py` with `ucp-conformance --suite=all`,
which resolves through `SUITE_REGISTRY`. A module the registry does not name
contributes no cases to that run, so this file would have merged and never
executed.

Rebased onto main and registered. Measured on this head: `--suite=all` selects
the module's cases, and removing the line again selects none.
vishkaty added a commit to vishkaty/conformance that referenced this pull request Nov 10, 2026
Universal-Commerce-Protocol#120 replaced the CI loop over `*_test.py` with `ucp-conformance --suite=all`,
which resolves through `SUITE_REGISTRY`. A module the registry does not name
contributes no cases to that run, so this file would have merged and never
executed.

Rebased onto main and registered. Measured on this head: `--suite=all` selects
the module's cases, and removing the line again selects none.
vishkaty added a commit to vishkaty/conformance that referenced this pull request Nov 10, 2026
Universal-Commerce-Protocol#120 replaced the CI loop over `*_test.py` with `ucp-conformance --suite=all`,
which resolves through `SUITE_REGISTRY`. A module the registry does not name
contributes no cases to that run, so this file would have merged and never
executed.

Rebased onto main and registered. Measured on this head: `--suite=all` selects
the module's cases, and removing the line again selects none.
vishkaty added a commit to vishkaty/conformance that referenced this pull request Nov 10, 2026
Universal-Commerce-Protocol#120 replaced the CI loop over `*_test.py` with `ucp-conformance --suite=all`,
which resolves through `SUITE_REGISTRY`. A module the registry does not name
contributes no cases to that run, so this file would have merged and never
executed.

Rebased onto main and registered. Measured on this head: `--suite=all` selects
the module's cases, and removing the line again selects none.
vishkaty added a commit to vishkaty/conformance that referenced this pull request Nov 11, 2026
Three cases asserting that a create does not adopt or persist members the
request schema marks `omit`, and that a create without them is unaffected.

Placed as `shopping/checkout/field_ownership_test.py` and registered as
`shopping.checkout.field_ownership_test`, following the layout Universal-Commerce-Protocol#121 landed and
dropping the redundant `checkout_` prefix the way `checkout_lifecycle_test`
became `shopping.checkout.lifecycle_test`. The path and the module name are a
suggestion and easy to change.

Universal-Commerce-Protocol#120 made `SUITE_REGISTRY` the only path `--suite=all` runs, so the entry is
what makes the module execute at all: without it `--suite=all` resolves 78
cases and selects none of these three, and with it 81.

Run against the sample server from samples main: 3 passed on their own, and
81 passed for the whole suite with one skip elsewhere.
vishkaty added a commit to vishkaty/conformance that referenced this pull request Nov 11, 2026
Four cases over cart conversion: conversion uses cart contents over the
payload, a second conversion returns the existing checkout, cancel returns the
cart state before deletion, and cart discount codes carry forward.

Placed as `shopping/cart/cart_test.py` in a new `shopping/cart/` package,
following the layout Universal-Commerce-Protocol#121 landed, and registered as `shopping.cart.cart_test`.
The earlier description proposed this path once the relocation merged.

Universal-Commerce-Protocol#120 made `SUITE_REGISTRY` the only path `--suite=all` runs, so the entry is
what makes the module execute: without it `--suite=all` resolves 78 cases and
selects none of these four, and with it 82.

Run against the sample server from samples main: 4 passed.
vishkaty added a commit to vishkaty/conformance that referenced this pull request Nov 11, 2026
Seven cases over the payment terms extension, asserted against the raw wire
payload: schedule sums, term id uniqueness, selected_term_id resolution,
selection handling, and accepted_term on the order.

Placed as `common/payments/terms_test.py` and registered as
`common.payments.terms_test`, following the layout Universal-Commerce-Protocol#121 landed. Payment terms
is a common payment extension, which is why it sits beside the other payment
modules rather than under `shopping/`.

Universal-Commerce-Protocol#120 made `SUITE_REGISTRY` the only path `--suite=all` runs, so the entry is
what makes the module reachable: without it `--suite=all` selects none of
these seven.

Run against the sample server from samples main: 7 skipped, for the reason the
description gives, since nothing in the reference advertises
`dev.ucp.common.payment.terms`.
damaz91 pushed a commit that referenced this pull request Nov 12, 2026
Three cases asserting that a create does not adopt or persist members the
request schema marks `omit`, and that a create without them is unaffected.

Placed as `shopping/checkout/field_ownership_test.py` and registered as
`shopping.checkout.field_ownership_test`, following the layout #121 landed and
dropping the redundant `checkout_` prefix the way `checkout_lifecycle_test`
became `shopping.checkout.lifecycle_test`. The path and the module name are a
suggestion and easy to change.

#120 made `SUITE_REGISTRY` the only path `--suite=all` runs, so the entry is
what makes the module execute at all: without it `--suite=all` resolves 78
cases and selects none of these three, and with it 81.

Run against the sample server from samples main: 3 passed on their own, and
81 passed for the whole suite with one skip elsewhere.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants