Skip to content

test: negotiate AP2 before mandate conformance checks - #115

Open
aeoess wants to merge 2 commits into
Universal-Commerce-Protocol:mainfrom
aeoess:test/ap2-negotiation
Open

aeoess wants to merge 2 commits into
Universal-Commerce-Protocol:mainfrom
aeoess:test/ap2-negotiation

Conversation

@aeoess

@aeoess aeoess commented Sep 29, 2026 •

Copy link
Copy Markdown

Description

Refs #114.

test_ap2_mandate_completion sent the placeholder mandate header.payload.signature~kb_signature and expected the checkout to complete. The mock Platform profile never declared AP2, so AP2 was never negotiated, and a Business that ignores mandates passed the only AP2 test.

This PR makes the AP2 tests run only when AP2 is actually negotiated, and adds the missing negative case.

  • Opt-in AP2 Platform profile. The mock agent server serves a second profile, /profiles/shopping-agent-ap2.json, that declares checkout, the AP2 Mandates extension extending checkout, and a public P-256 signing key for the trusted platform provider flow. The default profile is unchanged, because a Business that supports AP2 must refuse every completion without a mandate once AP2 is negotiated, which would affect every other test. Any existing keys in the profile are preserved.
  • Negotiation computed, not assumed. capability_intersection() implements the UCP intersection algorithm: name match, highest mutual version, and repeated pruning of extensions whose parent is absent. The AP2 tests skip unless AP2 is in that intersection. A Business that merely lists AP2 in a shape that does not negotiate is skipped, not tested.
  • New negative case. test_complete_without_mandate_is_rejected requires that completion without ap2.checkout_mandate does not complete and carries mandate_required.
  • Positive case skipped. test_ap2_mandate_completion now skips with an explicit reason. Once AP2 is negotiated, a Business that verifies the mandate must reject the placeholder, so the test cannot stay a passing positive. A real positive needs a signed SD-JWT+kb mandate fixture and a signing model, which this PR does not attempt.

Release scope

AP2 names changed between UCP releases. In 2026-04-08 the capability is dev.ucp.shopping.ap2_mandate, with /specification/ap2-mandates, /specification/checkout and top-level signing_keys. In 2026-08-25 the capability is dev.ucp.common.payment.ap2_mandate, with the payment extension paths, /specification/shopping/checkout and keys. #114 cited the newer name from the spec at ucp@4a3aa10f. ap2_release() describes both releases and returns None for others, in which case the tests skip.

Not in this PR

  • No AP2 cryptographic verification (signature, key binding, expiry, terms match).
  • No change to samples. The current sample server does not declare AP2, so both AP2 tests skip against it.

Error form for a missing mandate

Updated after review. The test accepts mandate_required as an error message on HTTP 200 or 4xx, or as a top-level code on 4xx. A resourceless 200 must be the UCP error envelope (ucp.status = error). 5xx fails. Since 9e2f030 the test also refuses complete_in_progress and completed, which mean the Business accepted or finished completion, and treats a 200 body as a checkout only when it has a string id and a non-completing status.

How it was tested

Against the Python sample server from samples@c2e04403, with python-sdk@v2026-04-08-6 as pinned in CI and the flower shop data. V1 to V5 used local, unsubmitted modifications of the sample to exercise both outcomes. Rerun at 9e2f030: V2 was checked with three forms (HTTP 200 error envelope, 422 with messages, 422 with a top-level code), all passing.

Variant positive new negative
V0: unmodified sample skipped (not negotiated) skipped (not negotiated)
V1: sample declares AP2, does not enforce it skipped fails, checkout completes without a mandate
V2: sample negotiates AP2 and returns mandate_required skipped passes
V5: sample negotiates AP2 and returns complete_in_progress with a mandate_required message skipped fails (it passed before 9e2f030)
V3a: Platform profile without the checkout parent skipped (AP2 pruned) skipped (AP2 pruned)
V4: Business declares AP2 at a version the Platform lacks skipped (no mutual version) skipped (no mutual version)

Full suite against the unmodified sample: 73 passed, 3 skipped, 2 failed, compared with 74 passed, 1 skipped, 2 failed without this change. The two failures are order_test.py::OrderTest::test_order_fulfillment_retrieval and test_order_update, which return HTTP 500 with and without this change. The difference is the two AP2 tests now skipping. ruff check and ruff format --check pass.

Tested the source tree now published as 7ad5f54, with samples@52319df and python-sdk@v2026-04-08-6. The runs preceded the sign-off amendment, which changed the commit ID without changing any files.

Against the unmodified sample, the full suite reported 81 passed, 3 skipped, 0 failures and 0 errors. Both AP2 tests skipped because AP2 was not negotiated.

With local sample changes for UCP 2026-04-08, the new refusal test failed when the sample declared AP2 but completed without a mandate. It passed when the sample returned HTTP 422 with mandate_required. The positive mandate test remained skipped.

Separate profile and capability-intersection probes covered both supported releases. The patched-server runs covered only 2026-04-08. The other server variants listed above were not rerun after these review changes.

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation. N/A, the change is test code only.
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes. In the earlier run described above, two pre-existing order_test failures occurred with and without this change. The run of the tree published as 7ad5f54 reported 0 failures.
  • Any dependent changes have been merged and published in downstream modules. N/A, no dependent changes.

@google-cla

google-cla Bot commented Sep 29, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@damaz91 damaz91 added the status:needs-triage Signal that the PR is ready for human triage label Sep 29, 2026
@aeoess
aeoess force-pushed the test/ap2-negotiation branch from 5516246 to 18f1c96 Compare October 9, 2026 22:52

@damaz91 damaz91 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this thoughtful PR, @aeoess!

The core architectural change here is spot-on:

  1. Opt-in AP2 Platform Profile (/profiles/shopping-agent-ap2.json): Because negotiating the AP2 extension security-locks the checkout session (complete_checkout requests without ap2.checkout_mandate must be rejected), serving a dedicated AP2 platform profile for ap2_test.py while keeping the default /profiles/shopping-agent.json profile unchanged prevents breaking standard non-AP2 checkout completion tests against AP2-capable servers.
  2. Discovery & capability_intersection(): Implementing the 4-step capability intersection algorithm (including highest mutual version selection and transitive extends pruning across both array and object-map capability formats) and skipping when the server does not negotiate AP2 is clean and spec-accurate.
  3. Skipping test_ap2_mandate_completion until a real SD-JWT+kb fixture exists: Agreed—sending the literal placeholder "header.payload.signature~kb_signature" previously produced a false pass only against servers that ignored AP2 validation.

Answer to your question on mandate_required error envelope vs HTTP status

In UCP, servers rejecting a request with a 4xx status (such as 400 or 422) commonly return the standard UCP error envelope (source/schemas/shopping/types/error_response.json, i.e., {"ucp": {"version": "...", "status": "error"}, "messages": [{"type": "error", "code": "mandate_required", ...}]})—which is also what the Python and Node.js reference servers in samples emit on 4xx errors—while some REST implementations return a checkout resource or top-level {"code": "mandate_required"}.

Right now, the else: branch in ap2_test.py (lines 150–156) only checks body.get("code") == "mandate_required", so a server returning HTTP 400/422 with messages: [{"type": "error", "code": "mandate_required", ...}] will fail at line 152. Accepting mandate_required inside messages[] on both 200 and 4xx responses (while also allowing top-level body.get("code") == "mandate_required" on 4xx and rejecting 5xx) will cover both styles cleanly. See the inline suggestion on ap2_test.py.

One small administrative note: please also sign the Google CLA (see the @google-cla comment above) so CI can clear the PR for merge.

Comment thread ap2_test.py Outdated
codes,
msg=f"Expected error message 'mandate_required'. Resp: {response.text}",
)
else:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When a server rejects the request with a 4xx HTTP status (e.g. 400 or 422), standard UCP error responses (error_response.json, as well as the Python and Node.js reference servers in samples) still use the messages array ({"ucp": {"version": "...", "status": "error"}, "messages": [{"type": "error", "code": "mandate_required", ...}]}) rather than a flat top-level {"code": "mandate_required"}. Also, on HTTP 200, if a server returns the checkout resource ("id" in body with status != "completed") alongside messages, ucp.status == "error" is only required for resourceless envelopes, and 5xx responses should not pass.

Suggested unification across 200 and 4xx:

    self.assertTrue(
        response.status_code == 200 or 400 <= response.status_code < 500,
        f"Expected HTTP 200 or 4xx for missing mandate, got {response.status_code}: {response.text}",
    )
    messages = body.get("messages", [])
    has_message_error = any(
        m.get("type") == "error" and m.get("code") == "mandate_required"
        for m in messages
    )
    has_top_level_code = (
        400 <= response.status_code < 500
        and body.get("code") == "mandate_required"
    )
    if response.status_code == 200 and "id" not in body:
      self.assertEqual(
          body.get("ucp", {}).get("status"),
          "error",
          f"Expected resourceless 200 UCP error envelope, got: {body}",
      )
    self.assertTrue(
        has_message_error or has_top_level_code,
        f"Expected error code 'mandate_required' in messages or body, got: {body}",
    )

Comment thread integration_test_utils.py Outdated
"checkout_schema": f"{base}/schemas/shopping/checkout.json",
"keys_field": "signing_keys",
}
return None

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Optional enhancement: since the 2026-08-25 release is now published on ucp.dev (where AP2 moved to dev.ucp.common.payment.ap2_mandate and top-level keys), adding the "2026-08-25" entry here would make ap2_release() ready for both releases:

      "2026-08-25": {
          "capability": "dev.ucp.common.payment.ap2_mandate",
          "spec": f"{base}/specification/payment/extensions/ap2-mandates",
          "schema": f"{base}/schemas/common/payment_ap2_mandate.json",
          "checkout_spec": f"{base}/specification/shopping/checkout",
          "checkout_schema": f"{base}/schemas/shopping/checkout.json",
          "keys_field": "keys",
      },

@damaz91 damaz91 added status:under-review and removed status:needs-triage Signal that the PR is ready for human triage labels Oct 30, 2026
aeoess added a commit to aeoess/conformance that referenced this pull request Oct 30, 2026
…25 AP2 names

Review on Universal-Commerce-Protocol#115: servers that reject with 4xx usually return the UCP error
envelope with mandate_required in messages[], not a top-level code. The
check now accepts either form on 4xx, the messages form on 200, requires
the error envelope only for a resourceless 200, and fails on 5xx.

ap2_release() now also covers 2026-08-25 (dev.ucp.common.payment.ap2_mandate,
payment extension paths, top-level keys).
@aeoess

aeoess commented Oct 30, 2026

Copy link
Copy Markdown
Author

Thanks for the review. Both are in 1994583.

The missing-mandate test now accepts mandate_required in messages[] on HTTP 200 or 4xx, or as a top-level code on 4xx. It still rejects completed checkouts and 5xx responses, and requires the UCP error envelope for a resourceless 200. A messages value that is not a list is treated as empty, so a malformed body fails the assertion instead of raising.

ap2_release() now also covers 2026-08-25, with the new capability name, URLs and top-level keys.

Ruff passes, and the assertions were checked against 13 response shapes. I have not rerun the full sample-server matrix on this commit. The CLA check is passing.

@aeoess

aeoess commented Oct 30, 2026

Copy link
Copy Markdown
Author

@damaz91 thanks for the review on 18f1c96. Since then we found one more false pass in the missing-mandate test. A Business that answered with status: complete_in_progress and a mandate_required message passed, even though complete_in_progress means it accepted completion without a mandate.

9e2f030 refuses complete_in_progress and completed, requires a JSON object, and treats a 200 body as a checkout only when it has a string id and a non-completing status. Any other 200 still needs the UCP error envelope. The forms accepted before still pass: mandate_required in messages on 200 or 4xx, and a top-level code on 4xx.

Rerun against the Python sample (samples@c2e04403, python-sdk@v2026-04-08-6) with a local, unsubmitted launcher that switches the sample's AP2 behaviour. The column is the result of test_complete_without_mandate_is_rejected:

Business behaviour at 1994583 at 9e2f030
AP2 not declared skipped skipped
declares AP2, completes without a mandate fails fails
HTTP 200 error envelope with mandate_required passes passes
HTTP 422 with mandate_required in messages passes passes
HTTP 422 with top-level code passes passes
HTTP 200 complete_in_progress with mandate_required passes fails
AP2 at a version the Platform lacks skipped skipped

The full suite against the unmodified sample is the same at both heads: 73 passed, 3 skipped, 2 failed, the two order_test cases that return HTTP 500 without this PR. ruff check and ruff format --check pass. The PR description is updated to match.

Could you take another look at the updated head?

@aeoess
aeoess force-pushed the test/ap2-negotiation branch from 9e2f030 to afcc958 Compare November 9, 2026 00:25
@aeoess

aeoess commented Nov 9, 2026

Copy link
Copy Markdown
Author

Rebased onto main after #120.

The AP2 profile route now lives in a shopping-layer subclass, Ap2AgentProfileServer, so the new generic framework server stays unchanged. The AP2 test assertions and negotiation helpers are unchanged from 9e2f030.

I reran the AP2 scenarios against both versions. All results match, including rejection of completion without a mandate and the complete_in_progress false pass.

Locally, the full Flower Shop suite reports 76 passed, 3 skipped and no failures. Both AP2 tests skip against that sample because it does not negotiate AP2. The positive test also remains disabled until a valid signed mandate fixture is available.

Rebased onto the tiered layout (Universal-Commerce-Protocol#121, Universal-Commerce-Protocol#122). The AP2 helpers and the mock
agent profile server that declares AP2 Mandates move out of the deprecated
integration_test_utils facade into common/payments/ap2_support.py, next to
the test. Ap2MandateTest keeps the requires_capability gate that main added
and still skips unless AP2 is in the negotiated capability intersection.
Test bodies and the AP2 helpers are unchanged from afcc958.

Signed-off-by: Tymofii Pidlisnyi <tima@aeoess.com>
@aeoess
aeoess force-pushed the test/ap2-negotiation branch from afcc958 to 6a56aaa Compare November 14, 2026 05:58
@aeoess

aeoess commented Nov 14, 2026

Copy link
Copy Markdown
Author

I rebased this onto main after #121 and #122. The AP2 helpers and mock profile server now live beside the test in common/payments/ap2_support.py. Both test bodies are unchanged, and the tests still skip unless AP2 is negotiated.

The local Flower Shop run had 81 passed, 3 skipped and 0 failures. In a separate run with AP2 enabled, the negative test caught a checkout that completed without a mandate and passed when the sample returned 422 mandate_required. The positive completion test still skips.

If you prefer a different location for the helper module, I can move it.

@damaz91 damaz91 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing the review feedback, catching the complete_in_progress edge case, and rebasing cleanly onto the new package layout in common/payments/ap2_support.py, @aeoess! Everything looks solid—approving.

Two small non-blocking notes that could be cleaned up in a follow-up:

  • Fixture-driven readiness in test_complete_without_mandate_is_rejected (common/payments/ap2_test.py): Passing a buyer payload to self.create_checkout_session(buyer=...) (matching CheckoutLifecycleTest.test_complete_checkout) and using get_valid_payment_payload() from shopping.base instead of self._payment_instrument() will ensure strict or custom servers don't reject the completion request on missing buyer details or custom payment handler IDs before reaching AP2 mandate validation.
  • Docstring nit on Ap2AgentProfileServer (common/payments/ap2_support.py): The class docstring still refers to AP2 Mandates living in the shopping layer from the earlier rebase, whereas the helper now lives under common/payments/.

@damaz91
damaz91 requested a review from gswetha November 16, 2026 09:17
@gswetha

gswetha commented Nov 17, 2026

Copy link
Copy Markdown

Thanks for this PR @aeoess! The opt-in AP2 agent profile, per-release metadata, and capability intersection check are great additions :)

I know you've done a lot of work on this PR already, but I'd like to provide a few suggestions to make it better:

  1. Ensure 2026-08-25 uses ap2["checkout_spec"] (common/payments/ap2_support.py):
    Because shopping-agent-test.json now already defines dev.ucp.shopping.checkout (with /specification/checkout/), capabilities.setdefault("dev.ucp.shopping.checkout", ...) is a no-op with the default template. So for 2026-08-25, the served profile keeps the 2026-04-08 spec path instead of ap2["checkout_spec"] (/specification/shopping/checkout). Assigning capabilities["dev.ucp.shopping.checkout"] = [...] directly (and using content_dict["ucp"].setdefault("capabilities", {})) fixes this.

  2. Make checkout ready to complete before testing mandate refusal (common/payments/ap2_test.py):
    Seconding @damaz91's note: passing buyer=self.DEFAULT_BUYER to self.create_checkout_session(...) and using get_valid_payment_payload() instead of the hardcoded _payment_instrument() avoids false failures against compliant servers that require buyer details or use custom payment handler IDs before checking AP2 mandates.

Good opportunities to fix here but non-blocking to merge:
3. Handle non-JSON responses cleanly (common/payments/ap2_test.py):
body = response.json() if response.content else {} raises an unhandled JSONDecodeError if a server returns plain text/HTML on an error response. Catching ValueError and failing with the HTTP status and response text makes failures much clearer. Also, when response.status_code == 200 with "id" in body, asserting body.get("order") is None helps ensure no order was created despite the error.

  1. Tighten edge cases in capability_intersection() (common/payments/ap2_support.py):
    • Guard against {"ucp": None} (e.g., ucp = profile.get("ucp") if isinstance(profile.get("ucp"), dict) else profile).
    • Filter versions to non-empty strings (or unwrap {"root": ...} as in framework/discovery.py) so non-string/empty versions can't raise TypeError or match "".
    • Distinguish absent extends (None) from an explicit empty extends: [], which currently evaluates falsy and is kept as a root capability rather than pruned.

Minor non-blocking nits:

  • Update the Ap2AgentProfileServer class docstring to reflect common/payments/ instead of the shopping layer.
  • Consider @unittest.skip(...) on test_ap2_mandate_completion instead of self.skipTest(...) inside the body so setUp() doesn't spin up the mock server for an unconditionally skipped test.

Thank you so much!

Serve the release's own checkout spec in the AP2 profile, make the
checkout ready to complete before the mandate refusal check, fail with
the HTTP status on a non-JSON response, assert no order on a 200 with a
checkout, and tighten capability_intersection (ucp null, version
wrapper and empty versions, explicit empty extends).

Signed-off-by: Tymofii Pidlisnyi <tima@aeoess.com>
@aeoess

aeoess commented Nov 17, 2026

Copy link
Copy Markdown
Author

@gswetha Applied both requested fixes, the optional robustness changes and both nits in 7ad5f54. The AP2 profile now uses the checkout specification for its declared release, and the refusal test supplies buyer details and uses the configured payment fixture. DEFAULT_BUYER belongs to the lifecycle test class, so the AP2 test has its own copy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants