Repository navigation
test: run an E2E suite against the published gateway of the pinned core version #6
Description
Part of the automation phase of NVIDIA/OpenShell#5287. It depends on #2 and #3 being on main, and core bump PRs (#5) will call it.
Problem
CI runs only unit tests. Nothing checks the SDK against a real gateway, so a bump PR from #5 can regenerate the bindings and pass CI while the curated API is broken against the new gateway.
Proposal
Port the sdk-go E2E lane (e2e.yml, start-gateway.sh, stop-gateway.sh) and seed the suite from core's e2e/python.
Gateway:
start-gateway.sh <core-version>runs the published gateway for that core version, using core's owndeploy/docker/docker-compose.ymlandgateway.tomlat the tag.- It uses the Docker compute driver and plaintext gRPC on
127.0.0.1:8080, pins the supervisor and sandbox images to the same version, and registers the gateway ase2e. - It waits up to 15 minutes for the gateway image of a fresh tag to be published.
stop-gateway.shcleans up. The gateway log is uploaded when a run fails.- The scripts should stay identical to sdk-go's apart from the pin lookup, which reads
OPENSHELL_CORE_REFinmise.tomlhere.
Workflow:
e2e.ymlis reusable (workflow_call) and can also be started by hand (workflow_dispatch). It takes acore-versioninput that defaults to the pin.- It runs on PRs and on
main, and ci: propose bindings updates for new core releases and track drift against core main #5's bump PRs call it with the proposed tag. mise run e2eruns the suite againstOPENSHELL_GATEWAY=e2e.- The suite lives in
tests/e2e/. The defaultmise run testexcludes it, so unit tests need no gateway.
Workload image. Sandbox.exec_python pickles the callable with cloudpickle and runs it inside the sandbox. The sandbox therefore needs the same Python minor version as the client, plus cloudpickle. Core builds that image from e2e/python/Dockerfile.workload (openshell/e2e-python:dev):
- Vendor that Dockerfile.
- Build it in the job and point the gateway's sandbox image at it.
- Add a drift check against core at the pin, like the OAuth fixture's.
Which tests move
Only tests whose subject is the SDK move here. Tests whose subject is gateway or sandbox enforcement stay in core and are ported to the Rust testkit there; NVIDIA/OpenShell#5287 tracks that work.
| Core file | Here? | Why |
|---|---|---|
test_sandbox_api.py |
Yes | CRUD, exec, interactive exec, listing, labels and selectors, all through the curated API |
test_workspace_api.py |
Yes | Workspace client, request ID replay |
test_sandbox_exec_python.py |
Yes | exec_python, cloudpickle round trip |
test_exec_admission.py |
Yes | Exec request IDs, timeouts and cancellation as the SDK sees them |
oidc/oidc_auth_test.py |
Later | The SDK's OIDC token flow; needs Keycloak, so a separate follow-up |
test_sandbox_landlock.py, test_sandbox_policy.py, test_policy_validation.py, test_sandbox_venv.py |
No | Sandbox policy enforcement |
test_security_tls.py, test_sandbox_providers.py, oidc/workspace_authz_test.py |
No | Gateway TLS, credential injection and authorization |
Some of these tests call private client._stub.<RPC> methods. Keep those calls, because real users depend on _stub too, and note which RPCs have no curated wrapper.
Acceptance criteria
-
mise run e2eruns locally against a gateway started bystart-gateway.sh. -
e2e.ymlpasses against the gateway of the pinned core version (v0.1.7). - The four test files above are ported and pass.
- The workload Dockerfile is vendored with a drift check.
- ci: propose bindings updates for new core releases and track drift against core main #5's bump PRs run E2E against the proposed core version.
- A follow-up issue exists for the OIDC lane.
Activity