Repository navigation
feat(runner): support servers on another machine (--discovery_url, --target_dir, --agent_profile_url, --webhook_base_url) - #123
Conversation
64c0539 to
165b34b
Compare
A server can serve a different profile at /.well-known/ucp than the one under test, such as a profile for an older release. --discovery_url points the runner's capability check, the shopping endpoint lookup and the shopping protocol tests at another profile: a full URL, or a path on --server_url. The core tests still check /.well-known/ucp, since serving the current profile there is itself a requirement. The value reaches the tests as an absl flag and as FLAGS_discovery_url. A new flag_value helper reads the flag and falls back to the environment, which is how pytest runs set flags, as FLAGS_server_url already does. The runner's warning now names the URL it failed to fetch. (cherry picked from commit e85882a)
A profile lists the shopping service once per transport, and the order is not fixed. The endpoint lookup and the shopping protocol tests took the first entry, so a profile that listed MCP before REST had no usable endpoint. Both now pick the entry whose transport is rest. The protocol tests fall back to the first entry when there is none, so their transport check still reports that case. The lookup moves into a function, shopping_rest_endpoint, with unit tests. (cherry picked from commit ab72069)
…ke effect The shopping fixture flags defaulted to the flower shop's files, and the shopping tests read them before the general flags, so a merchant's own --conformance_input never reached the shopping tests. The runner had no --fixture_config or --test_data_dir at all, although the README shows --fixture_config. The flags now default to unset, and a new fixture_paths function picks the configured files, then the flower shop's. The runner gains the two options and passes them as flags and as FLAGS_<name> environment variables; the environment is what reaches the shopping tests, since their module defines its flags after the runner parses its own. The runner stops before the run when one of these paths does not exist, since the tests would otherwise quietly use their defaults. (cherry picked from commit 6361c84)
…where The tests name the local mock servers by localhost addresses: the agent profile in the UCP-Agent header, and the webhook address in that profile. A server on another machine can reach neither. --agent_profile_url sets the profile URL the tests send, for example a tunnel to the mock agent port. --webhook_base_url replaces http://localhost:<webhook port> in the served profile with a public address that reaches the mock webhook server. Both reach the tests as flags and FLAGS_<name> environment variables, and the base class and the legacy integration_test_utils helper now share one agent_profile_url function. When the server is not on this machine and no webhook base URL is given, the webhook tests skip with a reason instead of waiting for deliveries that cannot arrive, and the runner says so before the run. (cherry picked from commit 8e3fdc7)
Testing a server elsewhere takes several options. --target_dir names a folder whose target.json gives them: server_url, discovery_url, conformance_input, fixture_config, test_data_dir, agent_profile_url and webhook_base_url. Other keys are ignored, so the file can carry notes. Paths are relative to the folder, and options on the command line win. --server_url now defaults to unset so a target file can supply it; with neither, it falls back as before to $UCP_SERVER_URL, then http://localhost:8182. An unusable target file is a usage error. (cherry picked from commit 06e8c62)
Covers --discovery_url, the fixture options, --agent_profile_url, --webhook_base_url and --target_dir, with an example target.json, and notes that request signing is not covered. (cherry picked from commit c30cc3b)
165b34b to
8a72935
Compare
|
Rebased onto Ready for review @damaz91 @carolinerg1 @niranjanmanjunath — thanks! |
damaz91
left a comment
There was a problem hiding this comment.
Thanks for putting this together and rebasing onto main, @segiodongo! The remote-target options (--discovery_url, --agent_profile_url, --webhook_base_url, --target_dir), rest_entry(), and _missing_paths() look very clean and well-tested.
Left a few inline notes, with one test fix worth making before merge:
SERVER_URLisolation inTargetDirTest.test_server_url_defaults: ClearingSERVER_URLalongsideUCP_SERVER_URLsoSERVER_URL=... uv run pytest(theREADME.mdwalkthrough command) does not fail whenpytestcollectstests/remote_target_test.py(one-click suggestion inline).- CI coverage &
pytestcollection:.github/workflows/conformance-tests.ymlcurrently only invokesuv run ucp-conformance --suite=all, sotests/remote_target_test.pyisn't executed in CI yet. Consider adding auv run pytest testsstep there, and optionally settingtestpaths = ["core", "common", "shopping"]under[tool.pytest.ini_options]inpyproject.tomlso bareuv run pyteststays scoped to live conformance suites againstSERVER_URL. - PR description nit: After rebasing onto #120,
_sync_absl_flags()onmainalready forwards--conformance_input,--fixture_config, and--test_data_dir; on top ofmain, this PR adds fail-fast missing-path validation (_missing_paths()) andFLAGS_<name>environment fallback (flag_value()).
(Heads-up on the red Governance / Approvals check: the new top-level tests/ directory wasn't matched in conformance-rules.yml, causing it to fall back to governance-council. Opened Universal-Commerce-Protocol/.github#92 to add tests/** under infra_and_tooling.)
| def test_server_url_defaults(self): | ||
| """Test the server URL's fallbacks when no target gives one.""" | ||
| with mock.patch.dict(os.environ): | ||
| os.environ.pop("UCP_SERVER_URL", None) | ||
| self.assertEqual( | ||
| runner.parse_args([]).server_url, runner.DEFAULT_SERVER_URL | ||
| ) | ||
| os.environ["UCP_SERVER_URL"] = "https://env.example" | ||
| self.assertEqual(runner.parse_args([]).server_url, "https://env.example") |
There was a problem hiding this comment.
apply_target() checks SERVER_URL before UCP_SERVER_URL (os.environ.get("SERVER_URL") or os.environ.get("UCP_SERVER_URL") or DEFAULT_SERVER_URL), while this test only clears UCP_SERVER_URL.
Because bare uv run pytest (the command in README.md) collects tests/remote_target_test.py alongside the conformance suites, running SERVER_URL=http://localhost:8182 SIMULATION_SECRET=... uv run pytest fails on line 274 (AssertionError: 'http://localhost:8182' != 'https://env.example'), and any other SERVER_URL fails on line 270.
| def test_server_url_defaults(self): | |
| """Test the server URL's fallbacks when no target gives one.""" | |
| with mock.patch.dict(os.environ): | |
| os.environ.pop("UCP_SERVER_URL", None) | |
| self.assertEqual( | |
| runner.parse_args([]).server_url, runner.DEFAULT_SERVER_URL | |
| ) | |
| os.environ["UCP_SERVER_URL"] = "https://env.example" | |
| self.assertEqual(runner.parse_args([]).server_url, "https://env.example") | |
| def test_server_url_defaults(self): | |
| """Test the server URL's fallbacks when no target gives one.""" | |
| with mock.patch.dict(os.environ): | |
| os.environ.pop("SERVER_URL", None) | |
| os.environ.pop("UCP_SERVER_URL", None) | |
| self.assertEqual( | |
| runner.parse_args([]).server_url, runner.DEFAULT_SERVER_URL | |
| ) | |
| os.environ["UCP_SERVER_URL"] = "https://env.example" | |
| self.assertEqual(runner.parse_args([]).server_url, "https://env.example") | |
| os.environ["SERVER_URL"] = "https://primary-env.example" | |
| self.assertEqual( | |
| runner.parse_args([]).server_url, "https://primary-env.example" | |
| ) |
There was a problem hiding this comment.
Good catch — applied your suggestion as is: the test clears SERVER_URL too, and now also checks that SERVER_URL wins over UCP_SERVER_URL. Verified SERVER_URL=http://localhost:8182 SIMULATION_SECRET=… pytest tests passes.
| class _Probe(unittest.TestCase): | ||
| """A test that needs no server, so the runner has something to run.""" | ||
|
|
||
| def test_nothing(self): | ||
| """Pass.""" |
There was a problem hiding this comment.
Nit: because _Probe subclasses unittest.TestCase at module level with a test_* method, pytest collects and runs _Probe::test_nothing as an extra standalone test case (31 collected items instead of 30). Defining _Probe inside run_main() (or naming its method runTest instead of test_nothing) avoids exposing the dummy probe to test collection.
There was a problem hiding this comment.
Done — the probe TestCase is now defined inside run_main(), so pytest tests collects 30 items (the probe is no longer one of them). I went with the local class rather than runTest, since pytest's unittest collector would still pick up a module-level runTest.
| def fixture_paths() -> FixturePaths: | ||
| """Return the fixture files to use: those configured, else the defaults. | ||
|
|
||
| Each value comes from a flag or its FLAGS_<name> environment variable | ||
| (see flag_value); the flower shop's files are the fallback. | ||
| """ | ||
| return FixturePaths( | ||
| conformance_input=flag_value("conformance_input") | ||
| or DEFAULT_CONFORMANCE_INPUT, | ||
| fixture_config=flag_value("fixture_config") or DEFAULT_FIXTURE_CONFIG, | ||
| data_dir=flag_value("test_data_dir") or DEFAULT_SHOPPING_FIXTURES, | ||
| ) |
There was a problem hiding this comment.
Minor nit: ShoppingIntegrationTestBase.setUp() only reads paths["fixture_config"] and paths["data_dir"] (since conformance_input is already loaded earlier by BaseIntegrationTest.setUp() via _load_conformance_config()). Keeping conformance_input here for completeness is fine, just noting in case FixturePaths was intended to mirror only what ShoppingIntegrationTestBase loads.
There was a problem hiding this comment.
Right — conformance_input is read earlier by BaseIntegrationTest.setUp(). I kept it in FixturePaths on purpose: fixture_paths() is the one place (and the one unit test) that states the flag → FLAGS_<name> → default order for all three options the runner forwards, so the README's table has a single source. Happy to trim it if you'd rather the type mirror only what ShoppingIntegrationTestBase loads.
…I (review) Review feedback on Universal-Commerce-Protocol#123: - test_server_url_defaults clears SERVER_URL as well as UCP_SERVER_URL, so `SERVER_URL=... uv run pytest` no longer fails on it, and it now also checks that SERVER_URL wins over UCP_SERVER_URL. - The probe TestCase that run_main feeds the runner is defined inside run_main, so pytest no longer collects it as a 31st test. - pyproject.toml sets testpaths to core, common and shopping: a bare `pytest` runs the conformance suites against SERVER_URL, and the framework's unit tests run with `pytest tests`. - The conformance workflow runs `uv run pytest tests` before starting the sample server.
|
Thanks @damaz91 — all addressed in
And thanks for opening .github#92 for the |
damaz91
left a comment
There was a problem hiding this comment.
Thanks for the quick follow-up, @segiodongo! All review comments are cleanly addressed (SERVER_URL isolation, local Probe scoping, testpaths separation, and running uv run pytest tests in CI). LGTM!
6e0ac6a
into
Universal-Commerce-Protocol:main
Follow-up to the RFC #520 split (#120, #121, #122): options the runner needs when the server under test is not on this machine, plus two fixes found while running against a remote store.
Options added (
runner.py, documented in the README under Testing a Server on Another Machine):--discovery_url/.well-known/ucp(full URL, or a path on--server_url). The core tests still check/.well-known/ucp.--agent_profile_urlUCP-Agent; give a public URL that reaches the suite's agent profile server (--mock_agent_port).--webhook_base_url--mock_webhook_port). Without it, the webhook tests skip when the server is not local instead of waiting for deliveries that cannot arrive.--target_dirtarget.json(paths relative to the folder; command-line options win).Fixes
--conformance_input,--fixture_configand--test_data_dir: on top ofmain's_sync_absl_flags()forwarding, the runner now stops before the run when one of these paths does not exist (_missing_paths()), since the tests would otherwise quietly use their defaults, and the shopping tests resolve each one from its flag or itsFLAGS_<name>environment variable (flag_value(),shopping.base.fixture_paths()).shopping.base.rest_entry).Tests:
tests/remote_target_test.py(30 unit tests, no server needed:python -m pytest tests; the conformance workflow now runs them, andtestpathskeeps a barepyteston the conformance suites) covers URL resolution, flag forwarding,target.jsonloading and precedence, and the local/remote address checks..cspell/custom-words.txtgainsflagsaver(absl's test helper).Verification (sample server, 2026-04-08):
ucp-conformance --suite=all— 83 tests, 81 passed, 2 skipped, 0 failed, the same result asmain;--dry-runlists 83 tests; pre-commit and cspell clean.