Skip to content

feat(runner): support servers on another machine (--discovery_url, --target_dir, --agent_profile_url, --webhook_base_url) - #123

Merged
carolinerg1 merged 7 commits into
Universal-Commerce-Protocol:mainfrom
segiodongo:upstream/1-remote-target
Nov 18, 2026
Merged

carolinerg1 merged 7 commits into
Universal-Commerce-Protocol:mainfrom
segiodongo:upstream/1-remote-target

Conversation

@segiodongo

@segiodongo segiodongo commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

Follow-up to the RFC #520 split (#120, #121, #122): options the runner needs when the server under test is not on this machine, plus two fixes found while running against a remote store.

Options added (runner.py, documented in the README under Testing a Server on Another Machine):

Option Use it when
--discovery_url The profile under test is not at /.well-known/ucp (full URL, or a path on --server_url). The core tests still check /.well-known/ucp.
--agent_profile_url The server fetches the agent profile named in UCP-Agent; give a public URL that reaches the suite's agent profile server (--mock_agent_port).
--webhook_base_url The server sends order webhooks; give a public URL that reaches the suite's webhook receiver (--mock_webhook_port). Without it, the webhook tests skip when the server is not local instead of waiting for deliveries that cannot arrive.
--target_dir Keep the options for one server together in a target.json (paths relative to the folder; command-line options win).

Fixes

  • --conformance_input, --fixture_config and --test_data_dir: on top of main's _sync_absl_flags() forwarding, the runner now stops before the run when one of these paths does not exist (_missing_paths()), since the tests would otherwise quietly use their defaults, and the shopping tests resolve each one from its flag or its FLAGS_<name> environment variable (flag_value(), shopping.base.fixture_paths()).
  • The shopping helpers use the shopping service's REST entry from discovery rather than its first entry, so a profile that lists an embedded or MCP binding first still resolves the REST endpoint (shopping.base.rest_entry).

Tests: tests/remote_target_test.py (30 unit tests, no server needed: python -m pytest tests; the conformance workflow now runs them, and testpaths keeps a bare pytest on the conformance suites) covers URL resolution, flag forwarding, target.json loading and precedence, and the local/remote address checks. .cspell/custom-words.txt gains flagsaver (absl's test helper).

Verification (sample server, 2026-04-08): ucp-conformance --suite=all — 83 tests, 81 passed, 2 skipped, 0 failed, the same result as main; --dry-run lists 83 tests; pre-commit and cspell clean.

A server can serve a different profile at /.well-known/ucp than the one
under test, such as a profile for an older release. --discovery_url
points the runner's capability check, the shopping endpoint lookup and
the shopping protocol tests at another profile: a full URL, or a path on
--server_url. The core tests still check /.well-known/ucp, since serving
the current profile there is itself a requirement.

The value reaches the tests as an absl flag and as FLAGS_discovery_url.
A new flag_value helper reads the flag and falls back to the
environment, which is how pytest runs set flags, as FLAGS_server_url
already does. The runner's warning now names the URL it failed to fetch.

(cherry picked from commit e85882a)
A profile lists the shopping service once per transport, and the order
is not fixed. The endpoint lookup and the shopping protocol tests took
the first entry, so a profile that listed MCP before REST had no usable
endpoint. Both now pick the entry whose transport is rest. The protocol
tests fall back to the first entry when there is none, so their
transport check still reports that case.

The lookup moves into a function, shopping_rest_endpoint, with unit
tests.

(cherry picked from commit ab72069)
…ke effect

The shopping fixture flags defaulted to the flower shop's files, and the
shopping tests read them before the general flags, so a merchant's own
--conformance_input never reached the shopping tests. The runner had no
--fixture_config or --test_data_dir at all, although the README shows
--fixture_config.

The flags now default to unset, and a new fixture_paths function picks
the configured files, then the flower shop's. The runner gains the two
options and passes them as flags and as FLAGS_<name> environment
variables; the environment is what reaches the shopping tests, since
their module defines its flags after the runner parses its own. The
runner stops before the run when one of these paths does not exist,
since the tests would otherwise quietly use their defaults.

(cherry picked from commit 6361c84)
…where

The tests name the local mock servers by localhost addresses: the
agent profile in the UCP-Agent header, and the webhook address in that
profile. A server on another machine can reach neither.

--agent_profile_url sets the profile URL the tests send, for example a
tunnel to the mock agent port. --webhook_base_url replaces
http://localhost:<webhook port> in the served profile with a public
address that reaches the mock webhook server. Both reach the tests as
flags and FLAGS_<name> environment variables, and the base class and
the legacy integration_test_utils helper now share one agent_profile_url
function.

When the server is not on this machine and no webhook base URL is
given, the webhook tests skip with a reason instead of waiting for
deliveries that cannot arrive, and the runner says so before the run.

(cherry picked from commit 8e3fdc7)
Testing a server elsewhere takes several options. --target_dir names a
folder whose target.json gives them: server_url, discovery_url,
conformance_input, fixture_config, test_data_dir, agent_profile_url and
webhook_base_url. Other keys are ignored, so the file can carry notes.
Paths are relative to the folder, and options on the command line win.

--server_url now defaults to unset so a target file can supply it; with
neither, it falls back as before to $UCP_SERVER_URL, then
http://localhost:8182. An unusable target file is a usage error.

(cherry picked from commit 06e8c62)
Covers --discovery_url, the fixture options, --agent_profile_url,
--webhook_base_url and --target_dir, with an example target.json, and
notes that request signing is not covered.

(cherry picked from commit c30cc3b)
@segiodongo
segiodongo force-pushed the upstream/1-remote-target branch from 165b34b to 8a72935 Compare November 13, 2026 19:05
@segiodongo
segiodongo marked this pull request as ready for review November 13, 2026 19:05
@segiodongo

Copy link
Copy Markdown
Collaborator Author

Rebased onto main now that #122 has merged — the PR is down to its own six commits (runner options for a server on another machine, the flag-forwarding fix and the REST-entry fix; 83 tests, 81 passed, 2 skipped against the sample server, same as main).

Ready for review @damaz91 @carolinerg1 @niranjanmanjunath — thanks!

@damaz91 damaz91 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for putting this together and rebasing onto main, @segiodongo! The remote-target options (--discovery_url, --agent_profile_url, --webhook_base_url, --target_dir), rest_entry(), and _missing_paths() look very clean and well-tested.

Left a few inline notes, with one test fix worth making before merge:

  1. SERVER_URL isolation in TargetDirTest.test_server_url_defaults: Clearing SERVER_URL alongside UCP_SERVER_URL so SERVER_URL=... uv run pytest (the README.md walkthrough command) does not fail when pytest collects tests/remote_target_test.py (one-click suggestion inline).
  2. CI coverage & pytest collection: .github/workflows/conformance-tests.yml currently only invokes uv run ucp-conformance --suite=all, so tests/remote_target_test.py isn't executed in CI yet. Consider adding a uv run pytest tests step there, and optionally setting testpaths = ["core", "common", "shopping"] under [tool.pytest.ini_options] in pyproject.toml so bare uv run pytest stays scoped to live conformance suites against SERVER_URL.
  3. PR description nit: After rebasing onto #120, _sync_absl_flags() on main already forwards --conformance_input, --fixture_config, and --test_data_dir; on top of main, this PR adds fail-fast missing-path validation (_missing_paths()) and FLAGS_<name> environment fallback (flag_value()).

(Heads-up on the red Governance / Approvals check: the new top-level tests/ directory wasn't matched in conformance-rules.yml, causing it to fall back to governance-council. Opened Universal-Commerce-Protocol/.github#92 to add tests/** under infra_and_tooling.)

Comment on lines +266 to +274
def test_server_url_defaults(self):
"""Test the server URL's fallbacks when no target gives one."""
with mock.patch.dict(os.environ):
os.environ.pop("UCP_SERVER_URL", None)
self.assertEqual(
runner.parse_args([]).server_url, runner.DEFAULT_SERVER_URL
)
os.environ["UCP_SERVER_URL"] = "https://env.example"
self.assertEqual(runner.parse_args([]).server_url, "https://env.example")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

apply_target() checks SERVER_URL before UCP_SERVER_URL (os.environ.get("SERVER_URL") or os.environ.get("UCP_SERVER_URL") or DEFAULT_SERVER_URL), while this test only clears UCP_SERVER_URL.

Because bare uv run pytest (the command in README.md) collects tests/remote_target_test.py alongside the conformance suites, running SERVER_URL=http://localhost:8182 SIMULATION_SECRET=... uv run pytest fails on line 274 (AssertionError: 'http://localhost:8182' != 'https://env.example'), and any other SERVER_URL fails on line 270.

Suggested change
def test_server_url_defaults(self):
"""Test the server URL's fallbacks when no target gives one."""
with mock.patch.dict(os.environ):
os.environ.pop("UCP_SERVER_URL", None)
self.assertEqual(
runner.parse_args([]).server_url, runner.DEFAULT_SERVER_URL
)
os.environ["UCP_SERVER_URL"] = "https://env.example"
self.assertEqual(runner.parse_args([]).server_url, "https://env.example")
def test_server_url_defaults(self):
"""Test the server URL's fallbacks when no target gives one."""
with mock.patch.dict(os.environ):
os.environ.pop("SERVER_URL", None)
os.environ.pop("UCP_SERVER_URL", None)
self.assertEqual(
runner.parse_args([]).server_url, runner.DEFAULT_SERVER_URL
)
os.environ["UCP_SERVER_URL"] = "https://env.example"
self.assertEqual(runner.parse_args([]).server_url, "https://env.example")
os.environ["SERVER_URL"] = "https://primary-env.example"
self.assertEqual(
runner.parse_args([]).server_url, "https://primary-env.example"
)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — applied your suggestion as is: the test clears SERVER_URL too, and now also checks that SERVER_URL wins over UCP_SERVER_URL. Verified SERVER_URL=http://localhost:8182 SIMULATION_SECRET=… pytest tests passes.

Comment thread tests/remote_target_test.py Outdated
Comment on lines +38 to +42
class _Probe(unittest.TestCase):
"""A test that needs no server, so the runner has something to run."""

def test_nothing(self):
"""Pass."""

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: because _Probe subclasses unittest.TestCase at module level with a test_* method, pytest collects and runs _Probe::test_nothing as an extra standalone test case (31 collected items instead of 30). Defining _Probe inside run_main() (or naming its method runTest instead of test_nothing) avoids exposing the dummy probe to test collection.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — the probe TestCase is now defined inside run_main(), so pytest tests collects 30 items (the probe is no longer one of them). I went with the local class rather than runTest, since pytest's unittest collector would still pick up a module-level runTest.

Comment thread shopping/base.py
Comment on lines +109 to +120
def fixture_paths() -> FixturePaths:
"""Return the fixture files to use: those configured, else the defaults.

Each value comes from a flag or its FLAGS_<name> environment variable
(see flag_value); the flower shop's files are the fallback.
"""
return FixturePaths(
conformance_input=flag_value("conformance_input")
or DEFAULT_CONFORMANCE_INPUT,
fixture_config=flag_value("fixture_config") or DEFAULT_FIXTURE_CONFIG,
data_dir=flag_value("test_data_dir") or DEFAULT_SHOPPING_FIXTURES,
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor nit: ShoppingIntegrationTestBase.setUp() only reads paths["fixture_config"] and paths["data_dir"] (since conformance_input is already loaded earlier by BaseIntegrationTest.setUp() via _load_conformance_config()). Keeping conformance_input here for completeness is fine, just noting in case FixturePaths was intended to mirror only what ShoppingIntegrationTestBase loads.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right — conformance_input is read earlier by BaseIntegrationTest.setUp(). I kept it in FixturePaths on purpose: fixture_paths() is the one place (and the one unit test) that states the flag → FLAGS_<name> → default order for all three options the runner forwards, so the README's table has a single source. Happy to trim it if you'd rather the type mirror only what ShoppingIntegrationTestBase loads.

…I (review)

Review feedback on Universal-Commerce-Protocol#123:

- test_server_url_defaults clears SERVER_URL as well as UCP_SERVER_URL, so
  `SERVER_URL=... uv run pytest` no longer fails on it, and it now also
  checks that SERVER_URL wins over UCP_SERVER_URL.
- The probe TestCase that run_main feeds the runner is defined inside
  run_main, so pytest no longer collects it as a 31st test.
- pyproject.toml sets testpaths to core, common and shopping: a bare
  `pytest` runs the conformance suites against SERVER_URL, and the
  framework's unit tests run with `pytest tests`.
- The conformance workflow runs `uv run pytest tests` before starting the
  sample server.
@segiodongo

Copy link
Copy Markdown
Collaborator Author

Thanks @damaz91 — all addressed in a21a1f1, replies inline:

  1. SERVER_URL isolation — suggestion applied; SERVER_URL=… uv run pytest tests passes.
  2. CI & collection — the conformance workflow now has a Run framework unit tests step (uv run pytest tests) before the sample server starts, and pyproject.toml sets testpaths = ["core", "common", "shopping"], so a bare uv run pytest stays on the conformance suites (collects 83: 3 core, 3 common, 77 shopping) while pytest tests runs the 30 unit tests. The probe TestCase moved inside run_main(), so it is no longer collected.
  3. Description — reworded the forwarding bullet: on top of main's _sync_absl_flags(), this PR adds the fail-fast _missing_paths() check and the FLAGS_<name> fallback via flag_value() / fixture_paths().

And thanks for opening .github#92 for the tests/** governance mapping.

@damaz91 damaz91 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick follow-up, @segiodongo! All review comments are cleanly addressed (SERVER_URL isolation, local Probe scoping, testpaths separation, and running uv run pytest tests in CI). LGTM!

@damaz91
damaz91 requested a review from gsmith85 November 18, 2026 10:11
@segiodongo segiodongo self-assigned this Nov 18, 2026
@carolinerg1
carolinerg1 merged commit 6e0ac6a into Universal-Commerce-Protocol:main Nov 18, 2026
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants