Repository navigation
Description
User Story
I run openshell-conformance, or the Kubernetes E2E with OPENSHELL_E2E_KUBERNETES_FEATURES=e2e,e2e-kubernetes, against a remote Kubernetes or OpenShift gateway. Its sandboxes can't reach the machine running the tests. I want conformance to tell me that this environment requirement isn't met, instead of failing tests that say nothing about whether my gateway meets the contract.
Description
The credential_gating conformance tests (#5318) start servers on the machine that runs the tests (tests/support/host-server) and reach them from sandboxes as host.openshell.internal. When sandboxes can't reach that machine, for example on a remote cluster, these tests fail.
Before #5318, the same scenarios lived in e2e/rust behind the e2e-host-gateway feature, so OPENSHELL_E2E_KUBERNETES_FEATURES=e2e,e2e-kubernetes excluded them on a remote cluster. openshell-conformance has no feature selection, so it now runs them unconditionally. The compatibility review on #5318 flagged this, and the review accepted it as a tracked follow-up.
Options:
- Keep an opt-in. Gate the host-dependent conformance tests behind a declared environment capability, such as an env var or a
openshell-conformanceflag, ase2e-host-gatewaydid. - Move them to a feature suite. Keep conformance free of tests that need the test machine, and run
credential_gatingfromtests/suites/features/on lanes that provide reachability. - Check reachability up front. Before the host-dependent tests run, start a host server and have a sandbox connect to it through
host.openshell.internal. If that fails, skip those tests with a clear message that names the missing requirement.
Context
tests/suites/conformance/src/credential_gating.rs,tests/support/host-server/.TESTING.md(remote cluster section) andtests/suites/conformance/README.mdstate the requirement.e2e/rust/e2e-kubernetes.sh:OPENSHELL_E2E_KUBERNETES_FEATURES,e2e-host-gateway.tests/suites/features/host-gatewaychecks the same reachability on its own.- Other tests ported later (test(e2e): decide which e2e/rust tests to port and in what order #5297) that use the test machine as an upstream will have the same requirement.
Definition of Done
- One option is chosen, with the reasoning recorded here.
-
openshell-conformanceagainst a gateway whose sandboxes can't reach the test machine doesn't report the host-dependent tests as failures of the gateway, and its output names the missing requirement. - Lanes that provide reachability still run the host-dependent tests.
-
TESTING.mdandtests/suites/conformance/README.mddescribe how to run conformance against a remote cluster.
Activity