Skip to content

test(testkit): run conformance against any gateway with one pullable test image #5352

Open

Description

@matthewgrossman

User Story

I'm moving e2e/rust tests onto testkit, and I want driver authors and operators to run openshell-conformance against their own gateway as is. Today that needs OPENSHELL_TEST_PYTHON_IMAGE, because the default test image isn't published. I need one test image that any gateway can pull, with the tools the ported tests use, so conformance runs against any gateway without an override.

Description

Sandbox tests use two images today:

  • openshell/e2e-python:dev, built from e2e/python/Dockerfile.workload, is the default for testkit::python_image(). It has system Python, a uv Python and virtualenv, curl, git, openssl, requests, google-auth, cloudpickle, and a non-root sandbox user. It isn't published, so it exists only where a lane builds it (the Docker and Podman e2e/rust scripts, mise run e2e:workload:build) or loads it (the tmachine installers). Its name has no registry, so a gateway that doesn't have it pulls from Docker Hub and fails.
  • ghcr.io/astral-sh/uv:0.12.17-python3.12-trixie-slim@sha256:…, a public image pinned by digest. The Kubernetes and VM runners pass it through OPENSHELL_TEST_PYTHON_IMAGE. It has the Python standard library only: no curl, git or openssl, and no non-root user.

#5318 tried making the public uv image the default. It passed CI on every lane, but it was reverted in review because:

  • Nothing pre-pulls it, so each host or node pulls it during its first sandbox test, inside that test's timeout.
  • Every lane then needs outbound registry access.
  • It drops curl, git and openssl, which many tests still to be ported (test(e2e): decide which e2e/rust tests to port and in what order #5297) use.
  • It can't satisfy a policy that names the sandbox user, which the base test policy does. Podman rejects a named user the image doesn't have.

Options

  1. A trusted public image with more tools. It should come from a known publisher, be pinned by digest, and include Python, curl, git, openssl and a non-root user. That leaves us nothing to publish or maintain.
  2. Publish our own test image. CI publishes e2e/python/Dockerfile.workload (or a slimmer version) to ghcr, and testkit pins it by digest. We control its contents, at the cost of a publish step and keeping its packages up to date.

Either way, the follow-up also has to decide:

  • Pulling during tests: whether the tmachine installers and lane scripts pre-load the pinned image, so tests don't pull it inside their timeouts.
  • Gateways that can't pull it: whether conformance checks up front that the image can be pulled, and fails or skips with a clear message instead of a sandbox timeout.

Context

  • tests/testkit/testkit/src/image.rs: python_image(), PYTHON_BINARIES.
  • tests/testkit/testkit/src/base-policy.yaml: names run_as_user: sandbox.
  • e2e/support/conformance.sh (E2E_PULLABLE_PYTHON_IMAGE), e2e/rust/e2e-{docker,podman,kubernetes,vm}.sh.
  • The legacy harness has the same split: E2E_WORKLOAD_IMAGE in e2e/rust/src/harness/sandbox.rs.
  • tmachine builds and loads e2e-python:dev (tests/tmachine/artifacts.nix, config.nix, the openshell_gateway role, and the openshell-k3s.yaml playbook). The Podman driver e2e playbook (drivers/podman/e2e.yaml) uses it directly.

Definition of Done

  • One image is chosen, with the reasoning recorded here.
  • testkit::python_image() defaults to it, pinned by registry and digest, and PYTHON_BINARIES covers its Python paths.
  • Lanes don't pull it inside a test's timeout, or the pull time is measured and shown to be acceptable.
  • openshell-conformance passes against every conformance row and host lane with no OPENSHELL_TEST_PYTHON_IMAGE set.
  • The lane scripts and tmachine no longer build the workload image or override the image for testkit suites, or a remaining use is documented.
  • If we publish our own image, CI publishes it and something keeps its base and packages updated.
  • tests/testkit/README.md, tests/suites/conformance/README.md and TESTING.md describe the image and the override.

Activity

  1. changed the title [-]test(testkit): use one pullable test image with the tools ported tests need[/-] [+]test(testkit): run conformance against any gateway with one pullable test image[/+] on Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions