Skip to content

test(conformance): handle gateways whose sandboxes can't reach the test machine #5409

Open

Description

@matthewgrossman

User Story

I run openshell-conformance, or the Kubernetes E2E with OPENSHELL_E2E_KUBERNETES_FEATURES=e2e,e2e-kubernetes, against a remote Kubernetes or OpenShift gateway. Its sandboxes can't reach the machine running the tests. I want conformance to tell me that this environment requirement isn't met, instead of failing tests that say nothing about whether my gateway meets the contract.

Description

The credential_gating conformance tests (#5318) start servers on the machine that runs the tests (tests/support/host-server) and reach them from sandboxes as host.openshell.internal. When sandboxes can't reach that machine, for example on a remote cluster, these tests fail.

Before #5318, the same scenarios lived in e2e/rust behind the e2e-host-gateway feature, so OPENSHELL_E2E_KUBERNETES_FEATURES=e2e,e2e-kubernetes excluded them on a remote cluster. openshell-conformance has no feature selection, so it now runs them unconditionally. The compatibility review on #5318 flagged this, and the review accepted it as a tracked follow-up.

Options:

  1. Keep an opt-in. Gate the host-dependent conformance tests behind a declared environment capability, such as an env var or a openshell-conformance flag, as e2e-host-gateway did.
  2. Move them to a feature suite. Keep conformance free of tests that need the test machine, and run credential_gating from tests/suites/features/ on lanes that provide reachability.
  3. Check reachability up front. Before the host-dependent tests run, start a host server and have a sandbox connect to it through host.openshell.internal. If that fails, skip those tests with a clear message that names the missing requirement.

Context

  • tests/suites/conformance/src/credential_gating.rs, tests/support/host-server/.
  • TESTING.md (remote cluster section) and tests/suites/conformance/README.md state the requirement.
  • e2e/rust/e2e-kubernetes.sh: OPENSHELL_E2E_KUBERNETES_FEATURES, e2e-host-gateway.
  • tests/suites/features/host-gateway checks the same reachability on its own.
  • Other tests ported later (test(e2e): decide which e2e/rust tests to port and in what order #5297) that use the test machine as an upstream will have the same requirement.

Definition of Done

  • One option is chosen, with the reasoning recorded here.
  • openshell-conformance against a gateway whose sandboxes can't reach the test machine doesn't report the host-dependent tests as failures of the gateway, and its output names the missing requirement.
  • Lanes that provide reachability still run the host-dependent tests.
  • TESTING.md and tests/suites/conformance/README.md describe how to run conformance against a remote cluster.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions