Skip to content

feat: refactor conformance suite into vertical-agnostic architecture (RFC #520) - #107

Closed
segiodongo wants to merge 7 commits into
Universal-Commerce-Protocol:mainfrom
segiodongo:feat/vertical-agnostic-conformance
Closed

segiodongo wants to merge 7 commits into
Universal-Commerce-Protocol:mainfrom
segiodongo:feat/vertical-agnostic-conformance

Conversation

@segiodongo

@segiodongo segiodongo commented Sep 10, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Refactors the conformance suite from a flat, shopping-specific layout into a vertical-agnostic, tiered architecture, so that verticals beyond shopping can be certified without forking the framework (RFC #520).

This branch has been rebased onto the latest main and integrates main's newer and rewritten tests into the new tier layout.

Architecture

Tests are organised into tiers:

Tier Scope Contents
framework/ Vertical-agnostic plumbing base test, discovery, capability gating, mock agent profile server, mock webhook receiver
core/ Protocol-level, vertical-agnostic discovery, version negotiation, transport headers
common/ Cross-vertical concerns webhooks, idempotency, errors
shopping/ Shopping vertical checkout, cart, order, fulfillment, discount, validation, payment

Suites are selected declaratively via SUITE_REGISTRY in runner.py and exposed through a ucp-conformance console script:

uv run ucp-conformance --server_url=<url> \
  --conformance_input=shopping/fixtures/flower_shop/conformance_input.json \
  --suite=all

Capability gating is now declarative. Tests annotate their requirements with @requires_capability("dev.ucp.shopping.order") instead of hand-rolling a discovery probe in setUp, so a business that does not advertise a capability sees a skip rather than a failure.

Integrating main's tests

Three of the four commits originally on this branch had already been squash-merged into main, so only the refactor commit is replayed here. main's newer tests were moved into the tier layout:

Previously (root) Now
totals_test.py shopping/checkout/totals_test.py
fulfillment_structure_test.py shopping/fulfillment/structure_test.py
webhook_structure_test.py shopping/order/webhook_structure_test.py
discount_test.py shopping/discount/semantics_test.py
protocol_test.py split across core/protocol_test.py and shopping/checkout/protocol_test.py
business_logic_test.py shopping/checkout/business_logic_test.py (discount cases moved to the discount tier)

These appear as renames rather than deletions in the diff.

A note on one placement: webhook_structure_test.py landed under shopping/order/ rather than a shared common/webhooks/ module. Although it exercises webhook delivery mechanics, every case drives create_checkout_session / complete_checkout_session and gates on dev.ucp.shopping.order, so it is not vertical-agnostic in its current form. The genuinely shared webhook assertions live in common/webhooks/.

integration_test_utils.py is retained as a thin deprecation facade that re-exports the relocated symbols, so any out-of-tree tests importing it continue to work.

SDK compatibility

Several model types moved from ucp_sdk.models.schemas.shopping to ucp_sdk.models.common.types between SDK 0.4.6 and 0.5.0. Imports for the affected types are now wrapped in a try/except ImportError fallback, so the suite imports and runs cleanly against both. On main the suite cannot currently be collected under 0.5.0.

CI workflow

Two changes to .github/workflows/conformance-tests.yml:

  1. The for test_file in *_test.py glob loop is replaced by a single uv run ucp-conformance --suite=all invocation.
  2. --platform=google is replaced with --conformance_input=shopping/fixtures/flower_shop/conformance_input.json.

On (2): enforcing a single vendor's certification profile in the upstream repository's own CI would cause unrelated PRs to fail whenever that profile's mandates move ahead of the reference server. Running against the reference server's own declared conformance input keeps the signal attributable to the change under review. Vendor profiles remain fully supported via --platform for downstream certification runs.

Verification

Run against the reference server under the exact CI configuration (python-sdk v2026-04-08-6, samples at main):

Tests Passed Skipped Failed
main (baseline) 77 74 1 2
This branch 81 79 6 2

Both branches fail the same two tests — order_test.py::test_order_fulfillment_retrieval and order_test.py::test_order_update — with a 500 from PUT /checkout-sessions/{id}. These are pre-existing on main and are not introduced by this PR.

pre-commit run --all-files passes.

PR description covering the tier architecture, how main's tests were remapped, SDK compatibility shims, CI changes with rationale, and a verification table showing no regressions.

@damaz91 damaz91 added the status:needs-triage Signal that the PR is ready for human triage label Sep 10, 2026
…(RFC #520)

Restructure conformance test harness into a 4-tier architecture (framework, core, common, shopping) per RFC #520.

- framework/: BaseIntegrationTest, dynamic mock servers, discovery profile parsing, and decorators (@requires_capability, @spec_assert).
- core/: Protocol-level tests without retail domain dependencies (protocol discovery, binding, security, idempotency).
- common/: Cross-cutting extensions (webhooks, AP2, card credentials).
- shopping/: Retail domain tests (checkout lifecycle, business logic, discount, fulfillment, order, validation) and fixtures.
- platforms/google.yaml: Platform certification profile verifying required capabilities and payment handlers.
- runner.py: Dynamic CLI test orchestrator (ucp-conformance entrypoint) supporting suite filtering, platform evaluation, and capability checks.
- integration_test_utils.py: Backward-compatible alias facade emitting DeprecationWarning.
segiodongo and others added 2 commits October 29, 2026 11:27
unittest counts a skipped test in testsRun, so the summary's Passed line
(testsRun minus failures and errors) also counted every skip as a pass.
With one passing, one failing and one skipped test it printed Passed: 2.

Passed now leaves out skipped tests; the Skipped line is unchanged.
@segiodongo

Copy link
Copy Markdown
Collaborator Author

Hi all, a quick update and two questions to help move this forward.

Update: I pushed a small fix to the runner's summary, which counted skipped tests as passed. unittest includes skipped tests in testsRun, so Passed now leaves them out.

1. Size. This PR is large (+3.5k/−2.3k across 58 files), which makes it hard to review. If it would help, I can split it into smaller PRs:

  1. add framework/ and runner.py, leaving the existing tests where they are;
  2. move the tests into shopping/ as a rename-only change;
  3. add the core and common suites.

I'm also happy to keep it as one PR if you prefer.

2. Order with the open test PRs. 14 open PRs (#94–#118) change files that this PR moves or edits, so whichever lands first means the other side rebases. I'm glad to do that work either way:

  • land those first, and I'll rebase this PR onto them; or
  • land this first (or just its rename-only part), and I'll help each author update their branch. With the move in its own rename-only commit, git follows the renames, so those rebases should be straightforward.

This PR is also the base for follow-up work that links each test to the spec requirements it checks and reports coverage. I'd send that as a series of small PRs once the layout settles.

Which would you prefer? Thanks for taking the time to review.

@damaz91 damaz91 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for putting together this refactoring toward the multi-vertical architecture in RFC #520, @segiodongo!

On your two questions:

  1. Splitting into smaller PRs: Yes, splitting this into smaller PRs will make review much easier and avoid painful rebase conflicts:
    • PR 1: Add framework/, runner.py, and platforms/ while keeping existing tests at their current paths (addressing the flag/env-var/capability issues noted inline).
    • PR 2: Pure file move (git mv) of shopping tests and fixtures into shopping/ with minimal import-only edits so Git rename detection stays at ~100%.
    • PR 3: Extract vertical-agnostic core/ and common/ suites.
  2. Ordering vs. open test PRs (#89–#118): We should land the ready open test PRs (or land PR 1 first since it doesn't move existing test files) before doing the directory rename in PR 2, so contributors don't have to manually resolve combined file-move + logic diffs.

I left inline comments on a few functional regressions and coverage gaps to address (in particular, 5 tests in common/ are currently skipped due to non-existent capability names, TokenBindingTest was dropped, and --conformance_input / SERVER_URL are shadowed or ignored).

Comment thread common/payments/ap2_test.py Outdated
checkout.Checkout.model_rebuild(_types_namespace={"Payment": Payment})


@requires_capability("dev.ucp.common.payments.ap2")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dev.ucp.common.payments.ap2 is not a valid UCP capability name (the schema defines dev.ucp.common.payment.ap2_mandate, or dev.ucp.shopping.ap2_mandate in older server profiles). Because no server advertises dev.ucp.common.payments.ap2, Ap2MandateTest is silently skipped on every run.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, thanks. Fixed in e4bc51a: the test requires dev.ucp.common.payment.ap2_mandate. Because releases before 2026-08-25 advertised it as dev.ucp.shopping.ap2_mandate, framework.discovery records that rename (FORMER_NAMES) and supports_capability() accepts either, so the test runs against both generations of servers. It still skips on the sample server, which advertises neither.

Comment thread common/payments/card_credential_test.py Outdated
checkout.Checkout.model_rebuild(_types_namespace={"Payment": Payment})


@requires_capability("dev.ucp.common.payments.cards")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dev.ucp.common.payments.cards does not exist in ucp.capabilities (card_credential.json is a data type schema under common/types/, not an advertised capability). Decorating CardCredentialTest with @requires_capability("dev.ucp.common.payments.cards") causes this test to be permanently skipped.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, there is no such capability. Fixed in e4bc51a: the test is gated on dev.ucp.shopping.checkout only (the endpoint the credential is submitted to), which matches how it behaved on main. It runs and passes against the sample server now.

Comment thread common/webhooks/webhook_test.py Outdated
checkout.Checkout.model_rebuild(_types_namespace={"Payment": Payment})


@requires_capability("dev.ucp.common.webhooks")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two issues here:

  1. dev.ucp.common.webhooks is not a capability in ucp.capabilities (order webhooks are configured under dev.ucp.shopping.order), causing all 3 tests in WebhookTest to be silently skipped.
  2. WebhookTest drives create_checkout_session(), complete_checkout_session(), and /testing/simulate-shipping/{order_id} and asserts on order.fulfillment.expectations. For the same reason webhook_structure_test.py was placed under shopping/order/, this test is shopping-order specific and belongs under shopping/order/.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both points taken. In e4bc51a the test moves to shopping/order/webhook_test.py next to webhook_structure_test.py and requires dev.ucp.shopping.order, whose config carries the webhook_url. All three tests run and pass against the sample server; common/webhooks/ is removed.

checkout.Checkout.model_rebuild(_types_namespace={"Payment": Payment})


class TokenBindingTest(integration_test_utils.IntegrationTestBase):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TokenBindingTest (test_token_binding_completion, which tests credential.binding with checkout_id and identity on POST /checkout-sessions/{id}/complete) was deleted here and not migrated. core/binding_test.py replaces it with ProtocolBindingTest on /.well-known/ucp, dropping payment token binding test coverage. Please restore TokenBindingTest under shopping/ (or common/payments/).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching this. TokenBindingTest.test_token_binding_completion is restored in e4bc51a as common/payments/token_binding_test.py, next to the other payment-credential tests, on ShoppingIntegrationTestBase, and registered in the common suite.

Comment thread shopping/base.py Outdated
config_path = (
getattr(FLAGS, "shopping_conformance_input", None)
or getattr(FLAGS, "conformance_input", None)
or str(Path(DEFAULT_SHOPPING_FIXTURES) / "conformance_input.json")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because --shopping_conformance_input is defined with a non-empty default (.../shopping/fixtures/flower_shop/conformance_input.json), getattr(FLAGS, "shopping_conformance_input", None) is always truthy and short-circuits before getattr(FLAGS, "conformance_input", None). Passing --conformance_input on the CLI or via runner.py is ignored for all shopping tests. The same short-circuit bug occurs on lines 532–534 for shopping_fixtures_dir vs. test_data_dir.

Additionally, super().setUp() on line 510 starts AgentProfileServer before self.conformance_config is loaded here, so when --conformance_input is not explicitly passed, AgentProfileServer starts with an empty conformance_config.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct on both counts. Fixed in 0c4671f: the shopping_* flags are gone. conformance_input, fixture_config and test_data_dir are defined once in framework.base_test with None defaults; ShoppingIntegrationTestBase sets default_conformance_input to the flower-shop input and falls back to the bundled fixtures only when the flags are unset. The base class resolves the input before it starts the agent profile server, so the agent advertises the merchant's ucp_version. I checked that --conformance_input is read by passing an input with "ucp_version": "2099-01-01", which makes test_discovery fail on the version as it should.

Comment thread core/security_test.py Outdated
response = self.client.post(
f"/testing/simulate-shipping/{order_id}",
headers=self.get_headers(), # Standard headers only
"/testing/simulate-shipping/test-order-security",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/testing/simulate-shipping/{id} is a sample-server-specific shopping helper endpoint, not part of the core UCP specification. Non-shopping or custom servers that do not implement this route will return 404 instead of 403 and fail --suite=core. This test belongs under shopping/.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. In e4bc51a the tests move to shopping/order/simulation_url_security_test.py in main's form (a real completed order, and 200 with the correct secret rather than 200/404), gated on dev.ucp.shopping.order.

Comment thread core/protocol_test.py Outdated
self.assertIn("ucp", data, "Discovery profile must be wrapped in 'ucp' key")
ucp_data = data["ucp"]
# The discovery profile may be returned at the root or wrapped in `ucp`.
ucp_data = data.get("ucp", data)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Per source/schemas/profile.json ("required": ["ucp"]), discovery profiles must be wrapped in a top-level "ucp" object. Falling back to data when "ucp" is missing allows non-conformant unwrapped profiles to pass. Please keep self.assertIn("ucp", data, ...).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Restored in e4bc51a: self.assertIn("ucp", data, …) is back and the profile is read from data["ucp"].

Comment thread core/protocol_test.py Outdated
)
self.assert_response_status(response, 422)
resp = self.client.get("/.well-known/ucp", headers=headers)
self.assertIn(resp.status_code, [200, 400])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sending an incompatible UCP-Agent version to GET /.well-known/ucp and asserting [200, 400] does not verify version negotiation rejection (422), because discovery is a public GET endpoint that returns 200 regardless of agent version. Please retain the POST /checkout-sessions version-negotiation test (asserting 422 on "2099-01-01") in shopping/.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right that a GET cannot show the rejection. The POST /checkout-sessions test asserting 422 on 2099-01-01 was already retained as ShoppingProtocolTest.test_version_negotiation in shopping/checkout/protocol_test.py; in e4bc51a I removed the GET-based copy from core/ and left a note pointing to the shopping test.

Comment thread core/binding_test.py Outdated
headers=headers,
)
# Server should return 405 (Method Not Allowed), 415, or 400
self.assertIn(response.status_code, [400, 404, 405, 415])

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because /.well-known/ucp is a GET-only endpoint, servers return 405 Method Not Allowed before inspecting Content-Type, so POST /.well-known/ucp with Content-Type: text/plain does not actually verify unsupported Content-Type rejection.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, removed in e4bc51a. A Content-Type check needs an endpoint that reads a body, so it belongs with a vertical's own endpoints rather than in core/.

Comment thread pyproject.toml
"shopping",
"platforms",
"runner.py",
]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

integration_test_utils.py (still imported by core/idempotency_test.py and common/*) and shopping-agent-test.json (loaded from the repo root by framework/agent_profile_server.py, whose fallback DEFAULT_PROFILE lacks required ucp.version and services fields) are excluded from only-include, which will break wheel installations.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in da0a398: both files are in only-include. I also removed the DEFAULT_PROFILE fallback — with the template packaged there is no case where serving an invalid profile beats a clear FileNotFoundError.

@damaz91 damaz91 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for driving the RFC #520 4-tier architecture refactor and fixing the skipped-test summary calculation in a03b68c, @segiodongo!

On a detailed pass through the shopping/base.py extraction, CLI flag wiring in runner.py, and the main merge (56ad0ca), there are a few functional issues and merge regressions that need to be addressed before we can merge:

  1. --conformance_input and --test_data_dir are shadowed by non-empty defaults (shopping/base.py): shopping_conformance_input and shopping_fixtures_dir are registered with non-empty default paths pointing to flower_shop. In ShoppingIntegrationTestBase.setUp(), getattr(FLAGS, "shopping_conformance_input", None) or getattr(FLAGS, "conformance_input", None) always short-circuits on the non-empty default, so passing --conformance_input or --test_data_dir is silently ignored.
  2. --fixture_config on ucp-conformance (runner.py / README.md): The README.md example passes --fixture_config to uv run ucp-conformance, which fails with unrecognized arguments: --fixture_config=... because parse_args() does not accept it (and _sync_absl_flags() parses FLAGS before _load_tests() imports shopping.base).
  3. Regressions of #15 and #58 in shopping/base.py:
    • update_checkout_session() dropped the fallback preservation of buyer, fulfillment, discounts, and platform from checkout_obj when not passed (#15 / 88527b2), causing PUT /checkout-sessions/{id} updates to wipe out previously set fields.
    • create_checkout_payload() reverted payment_handler.Base.version and checkout_req.ucp["version"] to hardcoded "2026-04-08" instead of self.conformance_config.get("ucp_version", DEFAULT_UCP_VERSION) (#58 / 7422310).
  4. Dropped TokenBindingTest (binding_test.py): Root binding_test.py (TokenBindingTest.test_token_binding_completion, testing payment credential token binding on POST /checkout-sessions/{id}/complete) was deleted when core/binding_test.py (ProtocolBindingTest) was introduced, removing test_token_binding_completion from the suite.
  5. integration_test_utils.get_headers() omits idempotency-key by default: When idempotency_key is None, integration_test_utils.get_headers() omits the idempotency-key header (unlike BaseIntegrationTest.get_headers()), affecting common/payments/ap2_test.py and common/payments/card_credential_test.py (which also trigger DeprecationWarning by importing integration_test_utils).
  6. --dry-run contacts the server (runner.py): fetch_server_profile(args.server_url) runs before if args.dry_run:, contacting /.well-known/ucp even during --dry-run.

Left detailed inline comments on each location below.

Comment thread shopping/base.py Outdated
getattr(FLAGS, "shopping_conformance_input", None)
or getattr(FLAGS, "conformance_input", None)
or str(Path(DEFAULT_SHOPPING_FIXTURES) / "conformance_input.json")
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because shopping_conformance_input (line 92) and shopping_fixtures_dir (line 86) are defined with non-empty default strings pointing to DEFAULT_SHOPPING_FIXTURES, getattr(FLAGS, "shopping_conformance_input", None) and getattr(FLAGS, "shopping_fixtures_dir", None) (line 546) are always truthy. As a result, getattr(FLAGS, "conformance_input", None) and getattr(FLAGS, "test_data_dir", None) are never reached, and passing --conformance_input or --test_data_dir on the CLI is silently ignored.

Additionally, super().setUp() at line 510 starts AgentProfileServer using self.conformance_config before ShoppingIntegrationTestBase.setUp() loads the shopping config here.

Consider setting the default of shopping_conformance_input and shopping_fixtures_dir (as well as test_data_dir) to None so explicit --conformance_input / --test_data_dir flags take precedence over the DEFAULT_SHOPPING_FIXTURES fallback, and overriding _load_conformance_config() so BaseIntegrationTest.setUp() receives the resolved config before starting AgentProfileServer.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done as you suggest in 0c4671f (see the reply on line 525): None defaults, a single set of input flags in the framework, and the input resolved by the base class before the agent profile server starts.

Comment thread shopping/base.py
headers: dict[str, str] | None = None,
) -> Any:
"""Update a checkout session."""
currency = currency if currency is not None else checkout_obj.currency

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regression from #15 (88527b2): update_checkout_session() dropped the fallback preservation of buyer, fulfillment, discounts, and platform from checkout_obj when those optional arguments are None:

    def to_json(obj):
      if obj is not None and hasattr(obj, "model_dump"):
        return obj.model_dump(mode="json", exclude_none=True)
      return obj

    if buyer is None:
      buyer = to_json(getattr(checkout_obj, "buyer", None))
    if fulfillment is None:
      fulfillment = to_json(getattr(checkout_obj, "fulfillment", None))
    if discounts is None:
      discounts = to_json(getattr(checkout_obj, "discounts", None))
    if platform is None:
      platform = to_json(getattr(checkout_obj, "platform", None))

Without this, calling self.update_checkout_session(checkout_obj, discounts=...) or self.update_checkout_session(checkout_obj, fulfillment=...) sends a full-replacement PUT that wipes out existing buyer or fulfillment state on the checkout.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Restored in 09e70cb — see the reply on line 977.

Comment thread runner.py
"--conformance_input",
default=None,
help="Optional path to custom conformance input JSON configuration.",
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

README.md (line 204) documents running uv run ucp-conformance ... --fixture_config=path/to/your/test_fixtures.json, but parse_args() does not define --fixture_config (or --test_data_dir / --shopping_fixtures_dir), causing argparse to fail with unrecognized arguments: --fixture_config=....

Also note that _sync_absl_flags() (line 147) calls flags.FLAGS(flag_args, known_only=True) before _load_tests() imports shopping.base, so any absl flags defined in shopping.base are not yet registered when FLAGS is parsed. Please add --fixture_config (and --test_data_dir / --shopping_fixtures_dir) to parse_args() and ensure FLAGS is synced after test modules are imported (or via FLAGS / environment variables).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both fixed in 0c4671f: the flags are accepted, and FLAGS is synced after the test modules are imported (see the replies on lines 147 and 179).

Comment thread runner.py

server_profile: UcpProfile | None = None
try:
server_profile = fetch_server_profile(args.server_url)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

README.md (line 224) notes that --dry-run lists selected tests "without contacting the server", but fetch_server_profile(args.server_url) runs unconditionally here before if args.dry_run: is checked at line 279. Skipping fetch_server_profile when args.dry_run is set (and resolving Path(__file__).resolve().parent / "platforms" / f"{args.platform}.yaml" at line 243 so --platform works outside the repo root) will make --dry-run and --platform behave as documented.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 0c4671f: the dry-run listing happens before discovery is fetched, so --dry-run makes no requests (checked against an unreachable URL), and --platform=<name> resolves against platforms/ next to runner.py (checked from another directory).

Comment thread core/binding_test.py
Validated Paths:
- GET /.well-known/ucp
"""

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Root binding_test.py on main contained TokenBindingTest.test_token_binding_completion, which tested payment credential token binding (credential.binding per schemas/common/types/binding.json) on POST /checkout-sessions/{id}/complete. Deleting root binding_test.py when adding ProtocolBindingTest here dropped test_token_binding_completion from the conformance suite. Please preserve TokenBindingTest (e.g., under common/payments/ alongside card_credential_test.py and ap2_test.py, or under shopping/checkout/) and register it in runner.py's SUITE_REGISTRY.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Restored in e4bc51a as common/payments/token_binding_test.py and registered in SUITE_REGISTRY under common — see the reply on the deleted binding_test.py.

@damaz91 damaz91 added status:under-review and removed status:needs-triage Signal that the PR is ready for human triage labels Oct 30, 2026
Review of Universal-Commerce-Protocol#107 found that the inputs a merchant passes were silently
ignored or rejected:

- `--conformance_input` and `--test_data_dir` were shadowed by the
  `shopping_*` flags, whose non-empty flower-shop defaults always won.
  The duplicate flags are gone; `conformance_input`, `fixture_config`
  and `test_data_dir` now live in `framework.base_test` with `None`
  defaults, and the shopping base class supplies its bundled fixtures
  only when nothing is given.
- The conformance input was read after `BaseIntegrationTest.setUp()` had
  already started the agent profile server, so the agent advertised the
  default release instead of the merchant's. The base class now resolves
  the vertical's `default_conformance_input` before starting it.
- `ucp-conformance` did not accept `--fixture_config`, `--test_data_dir`
  or `--verbose_http`, and parsed the absl flags before the test modules
  that define them were imported. The runner now loads the suites first
  and forwards every flag.
- `--dry-run` fetched `/.well-known/ucp` before listing tests; it now
  returns before any request is made. `--platform=<name>` resolves
  against the `platforms/` directory next to `runner.py`, so it works
  outside the repository root.
- `server_url` and `simulation_secret` default to `SERVER_URL` and
  `SIMULATION_SECRET` again, as on main, so the documented
  `uv run pytest` invocation works without flags.
Extracting `shopping/base.py` from `integration_test_utils.py` dropped
two behaviours that main had:

- `update_checkout_session()` no longer carried `buyer`, `fulfillment`,
  `discounts` and `platform` over from the current checkout when they
  were not passed (Universal-Commerce-Protocol#15). Because the update is a full-replacement PUT, a
  call such as `update_checkout_session(obj, discounts=...)` wiped the
  buyer and fulfillment state a test had set up. The fallbacks are back.
- `create_checkout_payload()` hardcoded `"2026-04-08"` for the payment
  handler and request versions instead of the `ucp_version` declared in
  the conformance input (Universal-Commerce-Protocol#58). Both now use the base class's
  `ucp_version`, the same value the agent profile server advertises.
Five tests under `common/` were gated on capability names no server
advertises (`dev.ucp.common.payments.ap2`, `dev.ucp.common.payments.cards`,
`dev.ucp.common.webhooks`), so they skipped on every run, and several
tests placed in `core/` depended on the shopping vertical. This commit
puts each test where its real precondition is:

- `Ap2MandateTest` requires `dev.ucp.common.payment.ap2_mandate`, the
  name the schema defines. Releases before 2026-08-25 advertised it as
  `dev.ucp.shopping.ap2_mandate`; `framework.discovery` records that
  rename so the test runs against either.
- `CardCredentialTest` and the restored `TokenBindingTest` (dropped from
  main's `binding_test.py` when `core/binding_test.py` was added) are
  gated only on the checkout endpoint they drive: card credentials and
  token binding are payment data types, not advertised capabilities.
- `WebhookTest` moves to `shopping/order/` and requires
  `dev.ucp.shopping.order`, whose config carries the `webhook_url`.
- The checkout idempotency tests move to `shopping/checkout/`; the
  simulation-endpoint security tests move to `shopping/order/` in main's
  form (a real completed order, 200 with the correct secret). The
  GET-only variants left in `core/` could not fail and are removed.
- `core/protocol_test.py` requires the top-level `ucp` object again
  (profile.json lists it as required) and no longer "negotiates"
  versions against the public discovery GET, which answers 200 for any
  `UCP-Agent` version; the 422 check stays with the shopping suite.
- `core/binding_test.py` drops the Content-Type test: a POST to the
  GET-only discovery route is answered 405 before the header is read.

The `common/` tests import `shopping.base` directly instead of the
deprecated `integration_test_utils` facade.
`integration_test_utils.py` (the facade tests written before the tiered
layout still import) and `shopping-agent-test.json` (the profile
`framework/agent_profile_server.py` serves) were missing from the wheel's
`only-include`, so an installed `ucp-conformance` could not find them.
Both are listed now.

With the template guaranteed to ship, the server's silent fallback to a
hand-written profile that lacked the required `ucp.version` is gone; a
missing template raises instead of advertising an invalid agent.

The facade's `get_headers()` also sends an `Idempotency-Key` by default
again, as main and `BaseIntegrationTest.get_headers()` do.
@segiodongo

Copy link
Copy Markdown
Collaborator Author

Thanks for the thorough review, @damaz91 — every point was valid. I've pushed the fixes as four commits so they can be checked one at a time:

  • 0c4671f runner and flags. ucp-conformance accepts --fixture_config, --test_data_dir and --verbose_http, and the values actually reach the tests: the shopping_* flags that shadowed --conformance_input / --test_data_dir are gone, the three input flags are defined once in framework.base_test with None defaults, and the shopping base class uses the flower-shop fixtures only as a fallback. The conformance input is resolved before the agent profile server starts, so the agent advertises the merchant's ucp_version. The runner imports the suites before syncing FLAGS. --dry-run no longer contacts the server, --platform=<name> resolves against the packaged platforms/ directory, and SERVER_URL / SIMULATION_SECRET work as environment variables again.
  • 09e70cb shopping/base.py regressions. update_checkout_session() carries buyer, fulfillment, discounts and platform over again (fix: update_checkout_session to handle all optional args correctly. #15), and the payload versions come from the conformance input's ucp_version (feat: drive protocol_test version and capability asserts from conformance_input #58).
  • e4bc51a capability gating and placement. Ap2MandateTest requires dev.ucp.common.payment.ap2_mandate, with the pre-2026-08-25 name dev.ucp.shopping.ap2_mandate recognised by framework.discovery; CardCredentialTest and the restored TokenBindingTest are gated on the checkout endpoint only; WebhookTest, the checkout idempotency tests and the simulation-endpoint security tests move under shopping/; core/protocol_test.py requires the ucp wrapper again; the GET-based version-negotiation and Content-Type tests are removed because they couldn't observe what they claimed. Nothing in common/ imports the deprecated facade any more.
  • da0a398 packaging. integration_test_utils.py and shopping-agent-test.json are in the wheel, the invalid DEFAULT_PROFILE fallback is gone, and the facade's get_headers() sends an Idempotency-Key by default.

Against the sample server the run goes from 81 tests / 75 passed / 6 skipped to 79 / 77 / 2: the five tests that skipped on non-existent capability names now run and pass, TokenBindingTest is back, and the three tests that could not fail are gone. The two remaining skips are test_discovery_urls (already marked skip) and AP2, which the sample server doesn't advertise under either name.

On the split: agreed on the three-PR shape and on landing the ready test PRs (or PR 1) before the directory move. I'll cut PR 1 from this branch once these fixes look right to you, so the split inherits them rather than forking the branch twice. Does that order work for you?

@segiodongo

Copy link
Copy Markdown
Collaborator Author

Following up on the split: PR 1 is open as #120 and the other two are prepared on top of it, so the whole path is visible now:

  1. feat: add the vertical-agnostic conformance framework and runner (RFC #520, 1/3) #120 — framework/, runner.py, platforms/; no test file touched.
  2. split/2-move-shopping-tests — pure git mv into shopping/ and common/, every test a 100% rename, tests keep importing the facade: segiodongo/conformance@split/1-framework-runner...split/2-move-shopping-tests
  3. split/3-core-common-suites — core/ extraction, capability gating, import switch, discount tests grouped: segiodongo/conformance@split/2-move-shopping-tests...split/3-core-common-suites

Two small departures from the shape sketched above, both to keep rebases trivial for the open test PRs: the payment tests move straight to common/payments/ in step 2 rather than twice, and steps 1 and 2 make no edits at all inside test files (the import and decorator changes that would collide with open PRs all land in step 3). The end state of step 3 is the content of this PR's current head.

I'll open 2 and 3 as their bases merge, and I'm converting this PR to a draft so the review happens on the smaller ones. The review comments here are all resolved in the stack; happy to point at the specific commit for any of them.

@segiodongo
segiodongo marked this pull request as draft October 30, 2026 19:22
segiodongo added a commit to segiodongo/conformance that referenced this pull request Nov 2, 2026
Third step of the RFC #520 split of Universal-Commerce-Protocol#107. Every shopping test now imports
`ShoppingIntegrationTestBase` from `shopping.base` instead of the
`integration_test_utils` facade, calls `self.get_headers()`, tolerates the
SDK's older and newer locations for the payment models, and declares the
capability it exercises with `@requires_capability`, so a server that does
not advertise, say, `dev.ucp.shopping.discount` skips those tests instead
of failing them. Test bodies are unchanged.

`conftest.py` takes `FLAGS` from the framework for the same reason.
segiodongo added a commit to segiodongo/conformance that referenced this pull request Nov 3, 2026
Third step of the RFC #520 split of Universal-Commerce-Protocol#107. Every shopping test now imports
`ShoppingIntegrationTestBase` from `shopping.base` instead of the
`integration_test_utils` facade, calls `self.get_headers()`, tolerates the
SDK's older and newer locations for the payment models, and declares the
capability it exercises with `@requires_capability`, so a server that does
not advertise, say, `dev.ucp.shopping.discount` skips those tests instead
of failing them. Test bodies are unchanged.

`conftest.py` takes `FLAGS` from the framework for the same reason.
carolinerg1 pushed a commit that referenced this pull request Nov 5, 2026
… #520, 1/3) (#120)

* feat(framework): add the vertical-agnostic test framework

First step of the RFC #520 split of #107. `framework/` holds the pieces
of the conformance harness that do not depend on a business vertical:

- `base_test.BaseIntegrationTest`: HTTP client, merchant inputs
  (`--conformance_input`, `--fixture_config`, `--test_data_dir`, with
  `SERVER_URL` / `SIMULATION_SECRET` honoured as environment variables),
  discovery profile caching, capability gating and the agent profile
  server lifecycle. A vertical's base class points
  `default_conformance_input` at its bundled fixture.
- `discovery`: parses `/.well-known/ucp` into `UcpProfile`.
- `decorators.requires_capability`: skips a test class or method when the
  server does not advertise a capability.
- `platform`: platform certification profiles (required capabilities and
  payment handlers) loaded from YAML.
- `agent_profile_server`, `mock_webhook_server`, `constants`: moved out of
  `integration_test_utils` unchanged in behaviour; the agent profile
  template is required rather than silently replaced by a fallback.

Nothing uses the package yet; the next commits wire the runner and the
existing helpers to it. Existing tests are not touched.

* feat(runner): add the ucp-conformance runner and platform profiles

`runner.py` (installed as `ucp-conformance`) selects suites or individual
modules, fetches the server's discovery profile once and shares it with
the tests, optionally validates a platform certification profile from
`platforms/`, lists tests with `--dry-run` without contacting the server,
and forwards every merchant input flag to the suites after importing
them, so suite-defined flags are known to absl.

The registry lists today's root test modules under a single `shopping`
suite; `core` and `common` follow once those tiers exist. CI runs the
suite through the runner instead of invoking each module in a loop.

The wheel now lists its top-level modules and data explicitly so the
console script works from an installed package.

* refactor: build integration_test_utils on the framework

The shopping helpers keep their module, class names and functions, so no
test file changes, but the pieces that are not shopping-specific now come
from `framework/`: the command-line flags, `AgentProfileServer`,
`MockWebhookServer`, the HTTP client and discovery handling in
`BaseIntegrationTest`. `IntegrationTestBase` is the shopping subclass,
which also gates every test on the `dev.ucp.shopping.checkout` capability
and points the framework at the flower-shop input when none is given.

Compared with the previous module the helpers behave the same, with two
small improvements carried over from review: `update_checkout_session`
defaults a destination's `type` to `shipping_address`, and missing fixture
files are logged instead of silently ignored.

* docs: describe the ucp-conformance runner

* fix(framework): release the client and agent server when setUp() skips

unittest does not run tearDown() when setUp() raises, and SkipTest is an
exception, so a test skipped from setUp() left its AgentProfileServer
listening on port 8285 and its HTTP client open. Every later test in the
process then failed to bind the port (or, with the leaked server still
answering /healthz, silently talked to the stale one).

setUp() now checks everything that can skip the test before creating any
resource, and registers each resource with addCleanup(), which unittest
runs in every outcome. The tearDown() override is no longer needed.

Verified against a discovery-only server that advertises no capabilities:
all 77 tests skip in a few milliseconds with no bind errors (previously 19
of 20 tests in a three-module run hit "address already in use").

* fix(framework): keep inherited capability requirements per class

requires_capability() tested hasattr(), which walks the MRO, and then
mutated the set it found. Decorating a subclass whose base class was
already decorated therefore added the subclass's capability to the base
class's set, and with it to every sibling. The decorator now builds a new
set on the decorated object from whatever it inherits, so requirements
accumulate down the hierarchy without touching the parent.

With that in place the shopping base class declares its own requirement,
@requires_capability("dev.ucp.shopping.checkout"), instead of checking the
profile after super().setUp() had already started the agent profile
server. BaseIntegrationTest evaluates it before any resource is created,
so a non-shopping server skips the whole suite without spinning up and
tearing down the mock agent for each test.

* fix(runner): exit non-zero when a specifier or filter selects no tests

An empty TestSuite reports wasSuccessful(), so `--suite=typo`, a dotted
path whose class does not exist, or a `-k` filter that matches nothing all
exited 0. _load_tests() now returns the specifiers it could not load, and
main() exits 1 for those and for a selection of zero tests. A dotted path
whose attribute is missing or is not a TestCase subclass logs an error
instead of falling through silently.

* fix(framework): spell out required_caps and services_data for cspell

* chore(platforms): add the license header to google.yaml
segiodongo added a commit to segiodongo/conformance that referenced this pull request Nov 5, 2026
Third step of the RFC #520 split of Universal-Commerce-Protocol#107. Every shopping test now imports
`ShoppingIntegrationTestBase` from `shopping.base` instead of the
`integration_test_utils` facade, calls `self.get_headers()`, tolerates the
SDK's older and newer locations for the payment models, and declares the
capability it exercises with `@requires_capability`, so a server that does
not advertise, say, `dev.ucp.shopping.discount` skips those tests instead
of failing them. Test bodies are unchanged.

`conftest.py` takes `FLAGS` from the framework for the same reason.
carolinerg1 pushed a commit that referenced this pull request Nov 6, 2026
Fix: harden these four workflows together: pin every referenced Action to its current tag's full commit SHA, add permissions: contents: read to the pre-commit job, disable credential persistence on the read-only checkouts, and give the pure signal job permissions: {}. linter.yaml keeps its packages/statuses permissions and spellcheck.yaml keeps its job-level contents/actions reads, both unchanged; triggers and job commands are untouched. conformance-tests.yml is already pinned and is excluded because open PR #107 modifies it.
segiodongo added a commit to segiodongo/conformance that referenced this pull request Nov 9, 2026
…oves only)

Second step of the RFC #520 split of Universal-Commerce-Protocol#107. Every test module moves to the
package that owns it, with no content changes, so git tracks each file as
a 100% rename and open pull requests rebase onto the new paths:

- checkout, discount, fulfillment, order and validation tests go under
  `shopping/<domain>/`; `test_data/flower_shop` becomes
  `shopping/fixtures/flower_shop` (a symlink keeps the old path working).
- the payment credential tests (`ap2_test`, `card_credential_test`,
  `binding_test` as `token_binding_test`) go under `common/payments/`.

The tests still import `integration_test_utils`; the runner registry, CI
and README follow the files.
segiodongo added a commit to segiodongo/conformance that referenced this pull request Nov 9, 2026
Third step of the RFC #520 split of Universal-Commerce-Protocol#107. Every shopping test now imports
`ShoppingIntegrationTestBase` from `shopping.base` instead of the
`integration_test_utils` facade, calls `self.get_headers()`, tolerates the
SDK's older and newer locations for the payment models, and declares the
capability it exercises with `@requires_capability`, so a server that does
not advertise, say, `dev.ucp.shopping.discount` skips those tests instead
of failing them. Test bodies are unchanged.

`conftest.py` takes `FLAGS` from the framework for the same reason.
@segiodongo segiodongo closed this Nov 9, 2026
damaz91 pushed a commit that referenced this pull request Nov 10, 2026
…oves only) (RFC #520, 2/3) (#121)

* refactor: move the shopping tests and fixtures into shopping/ (file moves only)

Second step of the RFC #520 split of #107. Every test module moves to the
package that owns it, with no content changes, so git tracks each file as
a 100% rename and open pull requests rebase onto the new paths:

- checkout, discount, fulfillment, order and validation tests go under
  `shopping/<domain>/`; `test_data/flower_shop` becomes
  `shopping/fixtures/flower_shop` (a symlink keeps the old path working).
- the payment credential tests (`ap2_test`, `card_credential_test`,
  `binding_test` as `token_binding_test`) go under `common/payments/`.

The tests still import `integration_test_utils`; the runner registry, CI
and README follow the files.

* refactor: move the shopping helpers to shopping/base.py behind a facade

`integration_test_utils.py` becomes `shopping/base.py` (git shows it as a
94% copy because a facade keeps the old path). The facade re-exports every
name tests import from `integration_test_utils`, so the moved tests and
any open pull request keep working unchanged; it is not yet marked
deprecated because the in-tree tests still use it.

The shopping fixtures default to `shopping/fixtures/flower_shop`, and the
wheel lists the `common` and `shopping` packages instead of the former
top-level test modules.

* chore: re-trigger CI after the GitHub Actions incident of 2026-11-09
segiodongo added a commit to segiodongo/conformance that referenced this pull request Nov 12, 2026
Third step of the RFC #520 split of Universal-Commerce-Protocol#107. Every shopping test now imports
`ShoppingIntegrationTestBase` from `shopping.base` instead of the
`integration_test_utils` facade, calls `self.get_headers()`, tolerates the
SDK's older and newer locations for the payment models, and declares the
capability it exercises with `@requires_capability`, so a server that does
not advertise, say, `dev.ucp.shopping.discount` skips those tests instead
of failing them. Test bodies are unchanged.

`conftest.py` takes `FLAGS` from the framework for the same reason.
carolinerg1 pushed a commit that referenced this pull request Nov 13, 2026
…FC #520, 3/3) (#122)

* refactor(shopping): use shopping.base directly and declare capabilities

Third step of the RFC #520 split of #107. Every shopping test now imports
`ShoppingIntegrationTestBase` from `shopping.base` instead of the
`integration_test_utils` facade, calls `self.get_headers()`, tolerates the
SDK's older and newer locations for the payment models, and declares the
capability it exercises with `@requires_capability`, so a server that does
not advertise, say, `dev.ucp.shopping.discount` skips those tests instead
of failing them. Test bodies are unchanged.

`conftest.py` takes `FLAGS` from the framework for the same reason.

* fix(common): gate the payment tests on capabilities that exist

The three payment credential tests under `common/payments/` were gated on
capability names no server advertises, so they always skipped. Now:

- `Ap2MandateTest` requires `dev.ucp.common.payment.ap2_mandate`, the name
  the schema defines. Releases before 2026-08-25 advertised it as
  `dev.ucp.shopping.ap2_mandate`; `framework.discovery` records that rename
  (`FORMER_NAMES`) so the test runs against either generation of server.
- `CardCredentialTest` and `TokenBindingTest` are gated on the checkout
  endpoint they drive: card credentials and token binding are payment data
  types, not advertised capabilities.

They import `shopping.base` directly, which leaves `integration_test_utils`
without in-tree users; it now emits a DeprecationWarning for external ones.

* feat(core): extract the vertical-agnostic protocol tests

`core/` holds the tests any UCP server must pass regardless of the
verticals it implements, which is everything observable from the
discovery document alone:

- `core/protocol_test.py`: the discovery checks from the former shopping
  protocol test (required `ucp` wrapper, date-based version matching the
  conformance input, reverse-DNS capability and handler names, payment
  handler structure, published spec/schema URLs).
- `core/binding_test.py`: the standard transport headers are accepted and
  discovery answers with `application/json`.

`shopping/checkout/protocol_test.py` keeps what needs the shopping vertical:
the service advertisement checks and version negotiation, which can only
be observed on `POST /checkout-sessions` (422 for an unsupported version).

The runner gains the `core` suite and the README documents the tiers.

* refactor(shopping): group the discount tests under shopping/discount/

The discount flow tests (`test_discount_flow`, `test_multiple_discounts_*`,
`test_fixed_amount_discount`) move from the checkout business-logic module
to `shopping/discount/discount_test.py`, and `test_unknown_discount_code`
joins them from the validation suite, so every discount test lives with
the capability it exercises and is gated on `dev.ucp.shopping.discount`.
`shopping/discount/semantics_test.py` (the former `discount_test.py`) is
unchanged. Test bodies are unchanged; only their module and gate move.

* refactor(shopping): gate two extension tests and refresh two docs (review)

Review feedback on #122:

- test_complete_without_fulfillment requires dev.ucp.shopping.fulfillment:
  on a server without the extension, completing without fulfillment is a
  legitimate 200, so the test now skips there instead of failing.
- test_discount_entry_is_negative requires dev.ucp.shopping.discount, so it
  no longer sends an unnegotiated discounts field to a server that does not
  advertise the extension and validates strictly.
- core/__init__.py: the idempotency and simulation-URL tests live under
  shopping/, so the docstring names only transport binding and discovery.
- README and --suite help: a test module, class or file path is also
  accepted, with examples.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants