[matthewgrossman]

Summary

Moves the driver-agnostic sandbox lifecycle and API tests from e2e/rust to testkit so they run on every conformance row, moves the rest to the podman driver suite or crate tests, and deletes the legacy files. PR1 of the #5297 plan.

Related Issue

Closes #5425
Part of #5281

Changes

• New conformance suites under tests/suites/conformance/src/: image, labels, templates, settings, process, filesystem_policy. New tests in lifecycle.rs and file_transfer.rs. The suite table in tests/suites/conformance/README.md is updated.
• tests/suites/drivers/podman/tests/oci_identity.rs: supervisor container identity and capabilities.
• Crate tests: crates/openshell-cli/tests/doctor_integration.rs (uses a fake docker on PATH), crates/openshell-gateway/tests/proxy_config_preflight.rs, and production_gateway_listener_serves_grpc_through_websocket_tunnel in crates/openshell-server/src/lib.rs.
• The live-Internet benchmark is now mise run perf:live-internet (tasks/scripts/live_internet_perf.py drives it, and live_internet_perf_round.py is the legacy benchmark body, moved unchanged apart from ruff fixes).
• testkit: CliOutput::message() moved from file_transfer.rs into tests/testkit/testkit/src/cli.rs.
• Shared test helpers, used by the tests in this PR:
  • cx.cli_sandbox(suffix) → CliSandboxBuilder (.image(), .policy_yaml(yaml), .args(), .command(); .create() detaches and adopts, .run() returns the attached CliOutput). It tracks the sandbox before the CLI runs and passes --no-tty.
  • SandboxBuilder::create_ready(): create, then wait for Ready.
  • On Sandbox: poll_exec(what, timeout, command, ready) reruns an exec every 250 ms until ready accepts its output; get_json() is sandbox get --output json; wait_absent(timeout) waits for the gateway to remove the sandbox by itself, then checks get and list.
  • On both ExecOutput and CliOutput: success(), failure() and expect_stdout(expected). ExecOutput.command names the exec in their messages, and a failed success() reports stdout and stderr.
  • testkit::YamlFile: YAML in a temporary .yaml file for CLI commands that read one. CliSandboxBuilder::policy_yaml uses it. It matches the YamlFile in test(conformance): port the credential gating e2e tests #5318.
  • testkit::with_deadline(what, timeout, future): a named deadline shorter than a step's own.
  • std::io::Error converts into testkit::Error, so I/O results take .context() directly. The .map_err(Error::from_error) calls on I/O results in testkit, tests/support and the features suites are gone.
  • Engine::build_test_image(cx, name, containerfile) in container-engine-support: builds localhost/openshell-test-<name>:<run id> and removes it on cleanup. Engine::build_image is now private, so callers can't skip the cleanup.
• Existing testkit API that changed, so the helpers are uniform: cx.expect_listed(&sandbox, phase) and cx.expect_absent(&sandbox) are now sandbox.expect_listed(phase) and sandbox.expect_absent(). The testkit README lists the current API.
• These helpers replace duplicate code: the PolicyFile copies and hand-written sandbox create/sandbox get calls in host-gateway and oci-image, the image build/cleanup in oci-image and podman oci_identity, and the create in podman userns. host-gateway drops its tempfile dependency and oci-image drops its serde_json dependency. The proposed CLI-delete cleanup helper was skipped because no test in this PR uses it.
• ExecOutput::success() also replaces the hand-written exit-code checks in the existing smoke, lifecycle, persistent_workspace, file_transfer and policy_advisor conformance tests. Polls and checks that expect a failure still read exit_code.
• The other helpers are used across the conformance suite, existing tests included. templates.rs, settings.rs and labels.rs also get small local helpers (create_template, set_global_setting/delete_global_setting, listed_names).
• Deleted 13 legacy files and their references: the [[test]] entries, PODMAN_CI_TESTS, e2e-vm.sh, TESTING.md, and --enable-websocket-tunnel in e2e/with-{docker,podman}-gateway.sh, which only edge_tunnel_e2e used. Also removed the now-dead SandboxGuard::create_with_gateway_default.
• Shared files:
  • sandbox_lifecycle.rs keeps its other tests for PR4 (test(e2e): port CLI-driven exec, attach and SSH host identity tests from e2e/rust #5426). Whichever of PR1/PR4 merges last deletes it.
  • The incoherent-config tests are removed from docker_corporate_proxy.rs and vm_corporate_proxy.rs. Whichever of PR1/PR6 merges last deletes docker_corporate_proxy.rs. vm_corporate_proxy.rs stays until PR9.

Outcome per legacy test

Legacy test	Outcome
sandbox_lifecycle::sandbox_stop_start_preserves_workspace	Merged: already covered by persistent_workspace_survives_stop_start. "Stop while connect is open" moves to PR4.
sandbox_lifecycle::sandbox_can_be_deleted_while_stopped	Merged: covered by lifecycle::stopped_sandbox_can_be_deleted
sandbox_lifecycle::sandbox_exec_outlives_startup_recovery_deadline	lifecycle::exec_outlives_startup_recovery_deadline
sandbox_lifecycle::detached_canonical_main_{exit_zero_reaches_completed,nonzero_exit_reaches_error}	lifecycle::main_exit_after_ready_sets_final_phase
sandbox_lifecycle::canonical_main_and_exec_receive_declared_environment + canonical_tty_main_uses_sandbox_environment	lifecycle::canonical_main_environment
sandbox_lifecycle::detached_main_exit_during_provisioning_is_classified_as_workload_result	lifecycle::main_exit_during_provisioning_is_workload_result
sandbox_lifecycle::on_failure_policy_replaces_runtime_and_preserves_workspace	lifecycle::on_failure_policy_restarts_main_and_keeps_workspace
ephemeral_cleanup::detached_ephemeral_{success,failure}_removes_sandbox_and_driver_resources	lifecycle::detached_no_keep_sandbox_is_removed_after_main_exits
landlock::hard_requirement_accepts_enriched_device_path	filesystem_policy::hard_requirement_accepts_added_device_paths
landlock::{best_effort,hard_requirement}_rejects_replaced_policy_{path,ancestor} (4)	filesystem_policy::replaced_policy_path_rejects_next_launch
core_dump_hardening::sandbox_processes_disable_core_dumps	process::core_dumps_are_disabled
default_image::sandbox_from_default_image	Merged into lifecycle::default_command_sandbox_is_listed_ready
default_image::sandbox_from_explicit_nvidia_ubuntu_image	image::explicit_image_is_used
sandbox_labels::sandbox_labels_are_stored_and_filterable	labels::labels_are_stored_and_filter_list
sandbox_templates::sandbox_create_from_template_uses_reusable_workload	templates::sandbox_from_template_uses_its_workload
sandbox_templates:: get-after-delete, duplicate name, missing template (3)	templates::template_crud_errors
settings_management::settings_global_override_round_trip	Split: settings::sandbox_setting_round_trip + settings::global_setting_overrides_sandbox_scope
upload_backpressure::large_upload_preserves_content_and_metadata_outside_workdir	Split: file_transfer::large_upload_backpressure + the outside-workdir metadata case in file_transfer
podman_oci_identity::podman_supervisor_runs_as_oci_identity_without_capabilities	drivers/podman oci_identity::supervisor_runs_as_oci_identity_without_capabilities
docker_preflight::doctor_check_fails_without_docker + doctor_check_output_shows_docker_label	doctor_check_fails_without_docker_on_path (+ doctor_check_fails_fast_when_docker_info_fails)
docker_preflight::doctor_check_passes_with_docker	doctor_check_passes_when_docker_info_succeeds
docker_preflight::doctor_check_error_includes_guidance	Dropped
docker_corporate_proxy::…rejects_incoherent_configuration + vm_corporate_proxy::…rejects_incoherent_configuration	Dropped → crate test proxy_config_preflight.rs (docker, podman, vm)
docker_corporate_proxy::proxy_settings_are_inserted_before_nested_driver_tables	Dropped
edge_tunnel_e2e::gateway_status_reports_healthy	Dropped
edge_tunnel_e2e::ws_tunnel_status_through_edge_proxy	Dropped → crate test production_gateway_listener_serves_grpc_through_websocket_tunnel
internet_network_perf::benchmark_complete_internet_path	Dropped
live_internet_traffic_perf::benchmark_live_internet_agent_traffic	Kept on the host: mise run perf:live-internet

Decisions and defaults applied

• The six drops (decision D in test(e2e): decide which e2e/rust tests to port and in what order #5297) were approved, and the work ships as one PR.
• large_upload_backpressure runs on every conformance row.
• The global settings test changes a gateway-wide setting. It depends on the runner running one test at a time (test(runner): apply per-family nextest config and a thread count to runner rows #5295) and restores the value on cleanup. sandbox_setting_round_trip now clears the global value first, as the legacy test did.
• The podman OCI identity test runs on all 8 userns rows. The assertions for the rootful rows haven't been verified. The first CI run will show whether they hold.
• The WebSocket tunnel test runs the SDK edge tunnel (openshell_sdk::edge_tunnel::start_tunnel_proxy, a new openshell-server dev-dependency) through the production listener, /_ws_tunnel, and the Health RPC. It fails when the tunnel is off.
• The proxy preflight test checks both config preflight (category=malformed, file unchanged) and real openshell-gateway --config startup, because only startup names the offending keys.
• Landlock tests now run on every row, through exec rather than as the main command with CLI --policy YAML. Their policy sets no process identity, so they run as the image's USER. The default test image has no sandbox user. Rows without Landlock fail hard_requirement, and they should.
• Test deadlines fit inside the runner's 2-minute kill. The on-failure restart allows 15 s for the first marker, 45 s for the restart and 15 s for the replacement marker. The --no-keep test releases both sandboxes before waiting, with a 60 s limit.

Testing

• mise run e2e:testkit (local Docker gateway): all 31 conformance tests pass. The lifecycle and settings suites were rerun after the timeout and settings fixes.
• After the ExecOutput::success() sweep: the full conformance suite on a Docker gateway, 31/31 (the #[ignore]d bug(sandbox): on Kubernetes, deleting a sandbox waits out the pods' termination grace period #5025 test is skipped, as on main)
• After the review-feedback helpers: the full conformance suite on a Docker gateway, 31/31 (bug(sandbox): on Kubernetes, deleting a sandbox waits out the pods' termination grace period #5025 test skipped), and the oci-image and host-gateway feature suites (7/7)
• After the helper migration: the 19 affected conformance tests (cli, lifecycle, process, templates, image, filesystem_policy), plus the oci-image and host-gateway feature suites on a Docker gateway (7/7)
• cargo test for the new crate tests on macOS: openshell-server WebSocket tunnel test, doctor_integration, proxy_config_preflight
• clippy -D warnings and fmt: openshell-server, -cli, -gateway, the conformance, podman and features suites, tests/support, testkit, e2e/rust
• mise run rust:lockfiles:check
• ruff check/format for the benchmark scripts. One direct-mode round of the benchmark ran.
• Podman oci_identity and userns (migrated to the helpers) on the userns rows: CI

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[matthewgrossman]

/ok to test c2eceee

[github-actions]

Label test:e2e applied for c2eceee. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[matthewgrossman]

/ok to test 45180d0

[matthewgrossman]

/ok to test e3b0f27

[matthewgrossman]

/ok to test 73e5bff

[matthewgrossman]

/ok to test caff38c

[matthewgrossman]

/ok to test 2cefa9a

[matthewgrossman]

/ok to test 1ce77a5

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-5431.docs.buildwithfern.com/openshell