Repository navigation
test: make the checkout state machine assertions able to fail - #111
Merged
carolinerg1 merged 2 commits intoNov 6, 2026
Conversation
Two checks in checkout_lifecycle_test.py pass against servers they are meant to reject. Five state transition tests assert only assertNotEqual(status, 200), which accepts any non 200, including a 500. Replacing the samples state guard with a raise so that it answers 500 rather than a 4xx, all five still pass: Ran 5 tests ... OK with 5 matching 500s in the server log test_repeated_cancel in the same file already does this correctly, with assertGreaterEqual(400) plus assertLess(500). PR Universal-Commerce-Protocol#57 introduced that pair in this file, replacing the same assertNotEqual shape on the same guard, so this follows an established local pattern. It is also why the file as a whole goes red under that mutant. The five now carry their own verdict rather than relying on a sibling, and their docstrings are updated to describe the band they enforce. test_complete_checkout returns early when completion answers 409 with stock in the body. Making the samples complete path always raise OutOfStockError with status 409, that test still passes in isolation, so a business that can never complete a checkout passes the completion test. The escape hatch is removed. The hatch keyed on the reference response text, which is the kind of coupling a conformance suite should avoid. That case is already owned elsewhere: validation_test.test_out_of_stock uses the dedicated out_of_stock_item from conformance_input.json, gardenias, which is seeded at quantity 0, while this test uses items[0], bouquet_roses, seeded at 1000. A vendor whose configured item is out of stock fails earlier, at create, so the hatch never protected them. CI also wipes and reimports the database before every run. Verified against the reference server, python-sdk pinned to v2026-04-08-6 as conformance CI pins it: clean server, before and after 11 tests OK always 409 complete, this test only before OK, after FAILED 500 on the state guard, the five before all pass, after all FAIL Under the always 409 mutant the file is red in both states, because the _complete_checkout helper requires a 200 for three other tests. The change is visible in that test on its own. Full suite: 16 of 17 files pass, unchanged before and after. order_test.py fails on the two pre existing 500s from samples#218, which this does not touch. Out of scope: these five do not address the in band posture, where a business answers 200 with an error in messages[]. The previous assertion did not accept it either, so this change does not narrow what a conformant server may do.
damaz91
approved these changes
Oct 30, 2026
damaz91
left a comment
Contributor
There was a problem hiding this comment.
LGTM — great catch and thorough mutation testing, @vishkaty!
Both changes close real false-positive gaps in checkout_lifecycle_test.py:
- Removing the
409"stock"early-return escape hatch fromtest_complete_checkoutensures the primary completion test actually requires200 OKand acompletedcheckout on the seeded in-stock fixture (whilevalidation_test.py::ValidationTest::test_out_of_stockcontinues to own the out-of-stock check). - Replacing
assertNotEqual(response.status_code, 200)withassertGreaterEqual(..., 400)andassertLess(..., 500)acrosstest_cannot_update_canceled_checkout,test_cannot_complete_canceled_checkout,test_complete_is_idempotent,test_cannot_update_completed_checkout, andtest_cannot_cancel_completed_checkoutensures5xxserver errors fail the suite, consistent withtest_repeated_cancel.
carolinerg1
approved these changes
Nov 6, 2026
carolinerg1
merged commit Nov 6, 2026
65b22cd
into
Universal-Commerce-Protocol:main
14 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Category (Required)
ucp-schematool (resolver, linter, validator). (Requires Maintainer approval)Description
Two checks in
checkout_lifecycle_test.pypass against servers they are meantto reject. Both were found by mutating the reference server rather than by
reading, and each mutation is reproduced below.
Five state transition tests accept a server error. They assert only
assertNotEqual(status, 200), which accepts any non 200, including a 500.Replacing the samples state guard with a raise so it answers 500 rather than a
4xx, all five still pass, with five matching 500s in the server log.
test_repeated_cancelin the same file already does this correctly, withassertGreaterEqual(400)plusassertLess(500). #57 introduced that pairhere, replacing the same
assertNotEqualshape on the same guard, so thisfollows an established local pattern rather than inventing one. It is also the
reason the file as a whole goes red under that mutant. The five now carry their
own verdict, and their docstrings are updated to describe the band they
enforce.
test_complete_checkouthas an escape hatch. It returns early whencompletion answers 409 with
stockin the body. Making the complete pathalways raise
OutOfStockErrorwith status 409, the test still passes inisolation, so a business that can never complete a checkout passes the
completion test. The hatch also keyed on the reference response text, which is
the kind of coupling a conformance suite should avoid.
That case is already owned elsewhere.
validation_test.test_out_of_stockusesthe dedicated
out_of_stock_itemfromconformance_input.json, gardenias,seeded at quantity 0, while this test uses
items[0], bouquet_roses, seeded at1000. A vendor whose configured item is genuinely out of stock fails earlier,
at create, so the hatch never protected them. CI also wipes and reimports the
database before every run.
Verification
Against the reference server at samples main, with python-sdk pinned to
v2026-04-08-6exactly as the CI in this repo pins it:Under the always 409 mutant the file is red in both states, because the
_complete_checkouthelper requires a 200 for three other tests. The change isvisible in that test on its own.
Full suite: 16 of 17 files pass, unchanged before and after.
order_test.pyfails on the two pre existing 500s from samples#218, which this does not touch.
Scope
These five do not address the in band posture, where a business answers 200
with an error in
messages[]. The previous assertion did not accept thateither, so this change does not narrow what a conformant server may do.
Sequencing
#107 renames this file and rewrites the header lines adjacent to every site
touched here, while leaving the hatch and the five assertions byte identical.
There is no semantic overlap, but whichever lands second needs a textual
rebase. Happy to do that either way round.