Repository navigation
test(e2e): port CLI exec, attach, and SSH host identity tests to testkit - #5442
Draft
matthewgrossman wants to merge 29 commits into
Draft
matthewgrossman wants to merge 29 commits into
matthewgrossman wants to merge 29 commits into
Conversation
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Port the six credential gating scenarios from e2e/rust/tests/credential_gating.rs into a conformance suite. Each test imports its own provider profile and creates a provider under a run-unique name, creates sandboxes from the Python test image through the CLI, and reaches probe servers on the host through host.openshell.internal. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
The conformance suite now creates sandboxes from testkit::python_image(). Docker and Podman runners build openshell/e2e-python:dev also in e2e-conformance-only mode, and the Kubernetes and VM runners default OPENSHELL_TEST_PYTHON_IMAGE to the pullable uv Python image. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Add testkit::policy (a base policy from base-policy.yaml, allow(), and Endpoint settings), install_provider(), YamlFile, ExecOutput::success(), and PYTHON_BINARIES, so suites build policies and providers the same way. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Add host-server-support with HttpRecorder and WebSocketRecorder, servers on the test host that sandboxes reach through host.openshell.internal. Move the host-gateway feature test onto them and onto testkit's base policy. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Use testkit's policy helpers and install_provider and host-server-support's recorders, and assert on the requests that reached the host server instead of verdicts the sandbox clients print. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
The external-driver lane renders server.hostGatewayIP, but only the in-process driver reads it from the gateway TOML. The standalone driver takes OPENSHELL_HOST_GATEWAY_IP, which the post-renderer never set, so sandbox policy DNS refused host.openshell.internal on that lane. Copy the chart's rendered value into the driver container when present. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Conformance has to run against any gateway, and the unpublished openshell/e2e-python:dev image only exists where a lane built or loaded it. Default to the pinned uv Python image, which any gateway can pull, and keep OPENSHELL_TEST_PYTHON_IMAGE as the override. Host lanes no longer build the workload image for conformance. The base test policy names no workload user, so drivers use a default non-root user on images without one, and host_gateway runs python3 from PATH. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
- The WebSocket recorder reads frames until the connection ends, and the test waits for that before checking that no binary frame arrived. - The HTTP recorder keeps the request head and body apart. The REST tests compare the body exactly and check the secret appears nowhere, and failure messages redact it. - testkit errors keep the gateway's gRPC status code, and the admission test checks for FailedPrecondition. - The policy create request destructures the SDK spec, so a new field fails to compile instead of being dropped. - host_gateway no longer retries its connection. - Docs state that conformance requires sandboxes to reach the test machine as host.openshell.internal. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…mage The tmachine installers load it and the Docker and Podman lanes build it, so those lanes don't pull an image inside a test's timeout or lose curl, git and openssl. The Kubernetes and VM lanes keep using the pinned public image through OPENSHELL_TEST_PYTHON_IMAGE, and the base test policy names the sandbox user again. Running conformance against any gateway without an override is left to a follow-up. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
The test passed whenever no binary frame reached upstream, whatever the reason. It now first sends the same frame from a sandbox without the provider and requires that it arrives, so the denial check can fail. Unit tests check that WebSocketRecorder sees a binary frame after a long text frame, and none among text frames. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Use tungstenite's derive_accept_key instead of hashing with sha1 and base64 directly, and split host-server into http.rs and websocket.rs. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
… is missing The post-renderer copied the host gateway IP whenever the chart rendered any hostAliases. It now matches the host.openshell.internal entry, and fails if the lane set server.hostGatewayIP but no such entry rendered. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…r create import_profile reads the profile's id from its YAML and returns it, and create_provider takes the provider name and type separately, so the provider name no longer has to match the profile id. install_provider does both. Document that policy::allow merges binaries under one name and that an override Python image must use a PYTHON_BINARIES path. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…gating-pilot/matthewgrossman Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
testkit creates things through Context methods, such as cx.sandbox(..) and cx.on_cleanup(..), so import_profile, create_provider and install_provider become cx methods instead of free functions. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Move the driver-agnostic sandbox lifecycle, labels, templates, settings, image, process, Landlock and upload tests from e2e/rust to the conformance suite, the Podman OCI identity test to the podman driver suite, and the doctor, proxy-preflight and WebSocket tunnel checks to crate tests. Move the live-Internet benchmark to mise run perf:live-internet. Delete the legacy files and their references. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…d test image helpers Add cx.cli_sandbox (with policy files), ExecOutput::success, cx.sandbox_details, cx.wait_absent and Engine::build_test_image, use them in the ported lifecycle tests, and replace the duplicate policy file, sandbox get and test image code in the host-gateway, oci-image and podman suites. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Replace hand-written exit-code checks in the smoke, lifecycle, persistent_workspace, file_transfer and policy_advisor suites. Polls and checks that expect a failure keep reading the exit code. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…e helpers Add SandboxBuilder::create_ready, ExecOutput::expect_stdout, Sandbox::poll_exec, Context::delete_and_expect_absent and testkit::with_deadline, and convert std::io::Error into testkit::Error so I/O results take .context directly. Use them across the conformance suite, which also gets shared template, global-setting and label-list helpers and a split on-failure restart poll. Make Engine::build_image private so callers go through build_test_image, which removes the image. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…proxy Replace the hand-written WebSocket bridge with openshell_sdk::edge_tunnel::start_tunnel_proxy, so the test covers the client half of the tunnel and its edge-token headers as well as the router. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Move expect_listed, expect_absent, wait_absent and the CLI's sandbox get JSON (now get_json) from Context onto Sandbox, and drop delete_and_expect_absent. Give ExecOutput failure() and CliOutput expect_stdout(), so both outputs offer the same checks. Put with_deadline's description first, like poll_until. Make CliSandboxBuilder::run consume the builder like create, and rename its policy setter to policy_yaml. Update the testkit README's list of what a test can do. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Replace CliSandboxBuilder's private PolicyFile with testkit::YamlFile, and drop YamlFile's map_err calls now that I/O errors convert into testkit::Error. CliOutput::success now includes stdout in its failure message, like ExecOutput::success, since many commands report errors there. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…ting changes Add SandboxBuilder::image and CliSandboxBuilder::create_ready, so both builders set an image and wait for Ready the same way. Convert tonic::Status into testkit::Error, keeping its code, so gRPC results take .context() directly, and drop Error::from_status. Create the host-gateway sandbox with cx.sandbox(..).image(..).policy(..) instead of a hand-written CLI create. In credential gating, use create_ready and poll_exec. Drop the remaining map_err calls on I/O results in host-server-support. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Add Cli::spawn and Cli::spawn_with_stdin (with Context wrappers) that start the CLI and keep it running, for commands that stream output or read input. Stdout and stderr are drained in the background so the pipe never blocks. Also add Cli::apply_env so another program, such as ssh with a ProxyCommand from sandbox ssh-config, reaches the same gateway. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Move the CLI exec, attached create, no-keep, attach, and SSH host identity tests from e2e/rust to the conformance suite, and the procfs-based connect transport recovery and signal forwarding tests to a new features/cli-attach suite. With the lifecycle port, no tests are left in sandbox_lifecycle.rs, so delete it with ssh_host_identity.rs, its tmachine follow-up entry, the now-unused PTY harness helper, and the stale Kubernetes nextest waiver. The gateway-restart check of SSH host identity moves to a later port. Part of #5281 Closes #5426 Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
Label |
Member
Author
|
/ok to test bbbe043 |
elezar
force-pushed
the
test/5425-port-sandbox-lifecycle/matthewgrossman
branch
from
November 10, 2026 10:44
1ce77a5 to
f8b1ced
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Port the e2e/rust tests that drive a long-running
openshellCLI process (exec streaming, attached create,--no-keep, attach/reconnect, and SSH host identity) to testkit, and delete the legacy files. Stacked on #5431 (PR1); review the last two commits.Related Issue
Closes #5426
Part of #5281. Stacked on #5431, which is stacked on #5318.
Changes
CliChild(Cli::spawn/spawn_with_stdin,cx.cli_spawn/cx.cli_spawn_with_stdin): a long-running CLI process with stdin, a line reader, its pid, signal/kill and wait. AlsoCli::gateway()andCli::apply_env(), which export the CLI environment for a plainsshclient.exec_cli.rs:cli_exec_large_output_is_complete,cli_exec_piped_stdin_crosses_grpc_message_limitcli.rs:cli_attached_create_streams_main_and_reports_exit,cli_no_keep_removes_sandbox_after_main_exitattach.rs:attach_reconnect_replays_history_for_the_same_main_process(it also stops and starts the sandbox while a connect session is open),attach_main_exit_255_is_not_retried_as_transport_failuressh.rs:ssh_host_identity_stable_across_stop_start_and_new_on_recreatetests/suites/features/cli-attachwith one Linux row (ubuntu-docker-rootful, deb) in.github/workflows/e2e-suites.yml:connect_recovers_its_ssh_transport,connect_forwards_pid_targeted_termination_and_reaps_ssh. It reads host procfs directly and fails with a clear message off Linux. The row isn't on k3s, because of Investigate delayed SSH input-owner release through kubectl port-forward #4445.e2e/rust/tests/sandbox_lifecycle.rs(PR1 and this PR together port all of its tests) ande2e/rust/tests/ssh_host_identity.rs. Also removed:sandbox_lifecyclefollow-up entry intests/tmachine/artifacts.nix;[profile.e2e-kubernetes]waiver forcanonical_main_connect_recovers_its_ssh_transportin.config/nextest.toml;openshell_tty_cmd,shell_escapeandnixdependency in e2e/rust.tests/testkit/README.md,tests/suites/conformance/README.md, andTESTING.md.Legacy test → new test
sandbox_lifecycle::sandbox_exec_large_output_is_completeexec_cli::cli_exec_large_output_is_completesandbox_lifecycle::piped_exec_stdin_crosses_grpc_message_limitexec_cli::cli_exec_piped_stdin_crosses_grpc_message_limitcanonical_main_exit_zero_completes_persistent_sandbox+canonical_main_nonzero_exit_preserves_statuscli::cli_attached_create_streams_main_and_reports_exitsandbox_create_with_no_keep_cleans_up_after_tty_command+…_preserves_failure_then_cleans_upcli::cli_no_keep_removes_sandbox_after_main_exitcanonical_main_disconnect_reconnect_replays_history_for_same_processattach::attach_reconnect_replays_history_for_the_same_main_processcanonical_main_exit_255_is_not_retried_as_transport_failureattach::attach_main_exit_255_is_not_retried_as_transport_failurecanonical_main_connect_recovers_its_ssh_transportfeatures/cli-attach::connect_recovers_its_ssh_transportcanonical_main_connect_forwards_pid_targeted_termination_and_reaps_sshfeatures/cli-attach::connect_forwards_pid_targeted_termination_and_reaps_sshssh_host_identity::ssh_host_identity_survives_restarts_and_changes_after_recreationssh::ssh_host_identity_stable_across_stop_start_and_new_on_recreate; the gateway-restart part moves to PR6Defaults applied
cli_pty. PlainCli::rundrives attachedsandbox create -- <cmd>, so the legacy TTY--no-keepcase now runs without a terminal. Onlyconnect_forwards_pid_targeted_termination_and_reaps_sshneeds one; it passes anopenptyfollower as theCliChild's stdin.features/cli-attachon one Linux row.sshhost identity test needs one, becausesandbox ssh-configwrites it into itsProxyCommand. Without one, the test fails with a clear message. It reads the fingerprint through the SDK and keeps the "auth.jsonunreadable" check.Testing
cargo fmtandcargo clippy --all-targetsare clean for testkit, conformance and features, on macOS and in Linux (Docker,rust:1.95-bookworm) for the Linux-only cli-attach codecargo test -p openshell-testkit --libpasses, including the newCliChildunit testscargo check --features e2e,e2e-host-gateway --testspasses in e2e/rustChecklist