Skip to content

test: make the checkout state machine assertions able to fail - #111

Merged
carolinerg1 merged 2 commits into
Universal-Commerce-Protocol:mainfrom
vishkaty:test/state-machine-assertions-can-fail
Nov 6, 2026
Merged

carolinerg1 merged 2 commits into
Universal-Commerce-Protocol:mainfrom
vishkaty:test/state-machine-assertions-can-fail

Conversation

@vishkaty

Copy link
Copy Markdown
Contributor

Category (Required)

  • Core Protocol: Changes to the base communication layer, global context, or breaking refactors. (Requires Technical Council approval)
  • Governance/Contributing: Updates to GOVERNANCE.md, CONTRIBUTING.md, or CODEOWNERS. (Requires Governance Council approval)
  • Capability: New schemas (Discovery, Cart, etc.) or extensions. (Requires Maintainer approval)
  • Documentation: Updates to README, or documentations regarding schema or capabilities. (Requires Maintainer approval)
  • Infrastructure: CI/CD, Linters, or build scripts. (Requires DevOps Maintainer approval)
  • Maintenance: Version bumps, lockfile updates, or minor bug fixes. (Requires DevOps Maintainer approval)
  • SDK: Language-specific SDK updates and releases. (Requires DevOps Maintainer approval)
  • Samples / Conformance: Maintaining samples and the conformance suite. (Requires Maintainer approval)
  • UCP Schema: Changes to the ucp-schema tool (resolver, linter, validator). (Requires Maintainer approval)
  • Community Health (.github): Updates to templates, workflows, or org-level configs. (Requires DevOps Maintainer approval)

Description

Two checks in checkout_lifecycle_test.py pass against servers they are meant
to reject. Both were found by mutating the reference server rather than by
reading, and each mutation is reproduced below.

Five state transition tests accept a server error. They assert only
assertNotEqual(status, 200), which accepts any non 200, including a 500.
Replacing the samples state guard with a raise so it answers 500 rather than a
4xx, all five still pass, with five matching 500s in the server log.

test_repeated_cancel in the same file already does this correctly, with
assertGreaterEqual(400) plus assertLess(500). #57 introduced that pair
here, replacing the same assertNotEqual shape on the same guard, so this
follows an established local pattern rather than inventing one. It is also the
reason the file as a whole goes red under that mutant. The five now carry their
own verdict, and their docstrings are updated to describe the band they
enforce.

test_complete_checkout has an escape hatch. It returns early when
completion answers 409 with stock in the body. Making the complete path
always raise OutOfStockError with status 409, the test still passes in
isolation, so a business that can never complete a checkout passes the
completion test. The hatch also keyed on the reference response text, which is
the kind of coupling a conformance suite should avoid.

That case is already owned elsewhere. validation_test.test_out_of_stock uses
the dedicated out_of_stock_item from conformance_input.json, gardenias,
seeded at quantity 0, while this test uses items[0], bouquet_roses, seeded at
1000. A vendor whose configured item is genuinely out of stock fails earlier,
at create, so the hatch never protected them. CI also wipes and reimports the
database before every run.

Verification

Against the reference server at samples main, with python-sdk pinned to
v2026-04-08-6 exactly as the CI in this repo pins it:

condition before after
clean server 11 tests OK 11 tests OK
always 409 complete, this test alone OK FAILED
500 on the state guard, the five all pass all FAIL

Under the always 409 mutant the file is red in both states, because the
_complete_checkout helper requires a 200 for three other tests. The change is
visible in that test on its own.

Full suite: 16 of 17 files pass, unchanged before and after. order_test.py
fails on the two pre existing 500s from samples#218, which this does not touch.

Scope

These five do not address the in band posture, where a business answers 200
with an error in messages[]. The previous assertion did not accept that
either, so this change does not narrow what a conformant server may do.

Sequencing

#107 renames this file and rewrites the header lines adjacent to every site
touched here, while leaving the hatch and the five assertions byte identical.
There is no semantic overlap, but whichever lands second needs a textual
rebase. Happy to do that either way round.

Two checks in checkout_lifecycle_test.py pass against servers they are meant
to reject.

Five state transition tests assert only assertNotEqual(status, 200), which
accepts any non 200, including a 500. Replacing the samples state guard with
a raise so that it answers 500 rather than a 4xx, all five still pass:

  Ran 5 tests ... OK      with 5 matching 500s in the server log

test_repeated_cancel in the same file already does this correctly, with
assertGreaterEqual(400) plus assertLess(500). PR Universal-Commerce-Protocol#57 introduced that pair in
this file, replacing the same assertNotEqual shape on the same guard, so this
follows an established local pattern. It is also why the file as a whole goes
red under that mutant. The five now carry their own verdict rather than
relying on a sibling, and their docstrings are updated to describe the band
they enforce.

test_complete_checkout returns early when completion answers 409 with stock
in the body. Making the samples complete path always raise OutOfStockError
with status 409, that test still passes in isolation, so a business that can
never complete a checkout passes the completion test. The escape hatch is
removed. The hatch keyed on the reference response text, which is the kind of
coupling a conformance suite should avoid.

That case is already owned elsewhere: validation_test.test_out_of_stock uses
the dedicated out_of_stock_item from conformance_input.json, gardenias, which
is seeded at quantity 0, while this test uses items[0], bouquet_roses, seeded
at 1000. A vendor whose configured item is out of stock fails earlier, at
create, so the hatch never protected them. CI also wipes and reimports the
database before every run.

Verified against the reference server, python-sdk pinned to v2026-04-08-6 as
conformance CI pins it:

  clean server, before and after       11 tests OK
  always 409 complete, this test only  before OK, after FAILED
  500 on the state guard, the five     before all pass, after all FAIL

Under the always 409 mutant the file is red in both states, because the
_complete_checkout helper requires a 200 for three other tests. The change is
visible in that test on its own.

Full suite: 16 of 17 files pass, unchanged before and after. order_test.py
fails on the two pre existing 500s from samples#218, which this does not touch.

Out of scope: these five do not address the in band posture, where a business
answers 200 with an error in messages[]. The previous assertion did not accept
it either, so this change does not narrow what a conformant server may do.

@damaz91 damaz91 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — great catch and thorough mutation testing, @vishkaty!

Both changes close real false-positive gaps in checkout_lifecycle_test.py:

  1. Removing the 409 "stock" early-return escape hatch from test_complete_checkout ensures the primary completion test actually requires 200 OK and a completed checkout on the seeded in-stock fixture (while validation_test.py::ValidationTest::test_out_of_stock continues to own the out-of-stock check).
  2. Replacing assertNotEqual(response.status_code, 200) with assertGreaterEqual(..., 400) and assertLess(..., 500) across test_cannot_update_canceled_checkout, test_cannot_complete_canceled_checkout, test_complete_is_idempotent, test_cannot_update_completed_checkout, and test_cannot_cancel_completed_checkout ensures 5xx server errors fail the suite, consistent with test_repeated_cancel.

@damaz91 damaz91 added status:under-review and removed status:needs-triage Signal that the PR is ready for human triage labels Oct 30, 2026
@carolinerg1
carolinerg1 merged commit 65b22cd into Universal-Commerce-Protocol:main Nov 6, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants