ci: enable manual and GitHub Actions E2E testing with Kata Containers #4459

User Story

I run OpenShell’s Kubernetes E2E tests locally on a Kind cluster using Kata Containers. I want a documented manual testing workflow and automated GitHub Actions coverage so that changes affecting Kata workloads can be validated consistently.

Problem Statement

OpenShell needs a repeatable Kubernetes E2E workflow that supports a configurable Kata RuntimeClass both locally and in GitHub Actions.

This umbrella issue covers the manual testing changes, evaluation of runners suitable for nested virtualization, and integration of Kata E2E tests into CI.

Impact / Why This Matters

Manual testing enables contributors to validate Kata deployments and reproduce failures. Automated testing provides ongoing coverage for sandbox provisioning, process lifecycle, networking, file transfers, and policy enforcement.

Runner virtualization support, capacity, and resource limits also affect reliability. A local run successfully executed Kata workloads but encountered sandbox creation errors during concurrent testing.

Proposed Design

1. Enable and document manual Kata E2E testing

Update mise run e2e:kubernetes to accept KATA_CONTAINERS_RUNTIME_CLASS, pass its value to Helm as server.defaultRuntimeClassName, and label the selected RuntimeClass openshell.ai/sandbox-attachable=true for resource admission.

Preserve existing behavior when the variable is unset or empty. Keep the configuration independent of Docker or Podman while respecting the selected container engine, Kubernetes context, host gateway IP, and image settings.

Document the cluster prerequisites, how to run the tests manually, and how to inspect the resulting reports. Validate the workflow and record any failures or infrastructure limitations.

2. Evaluate GitHub Actions runners for Kata

Compare standard GitHub-hosted Linux runners, larger runners, and self-hosted runners.

Evaluate usable KVM access, Kata VM startup, CPU and memory capacity, test concurrency, resource limits, reliability, runtime, and cost. Verify that virtualization remains usable from inside the Kubernetes nodes.

GitHub documents that nested virtualization on hosted runners is technically possible but not officially supported. Runner selection therefore needs validation with actual Kata workloads. GitHub runner documentation

Document the recommended runner configuration, required setup, supported concurrency, and limitations.

3. Run Kata E2E tests in GitHub Actions

Add a Kata test lane that reuses the existing Kubernetes E2E workflow where practical.

The workflow should provision a Kata-enabled cluster, configure the RuntimeClass, verify that workload pods actually use Kata, and run CLI conformance and the applicable Kubernetes E2E suite.

Publish test reports and failure diagnostics, clean up test resources, and document how contributors trigger the workflow and reproduce failures locally.

Manual Testing Example

For an existing Kata-enabled Kind cluster running on Podman:
CONTAINER_ENGINE=podman
KATA_CONTAINERS_RUNTIME_CLASS=kata-qemu
IMAGE_TAG=dev
mise run e2e:kubernetes

The container engine, host gateway IP, Kubernetes context, RuntimeClass, and image tag should be adjusted to match the environment.

Initial Findings

A local run on Podman-backed Kind confirmed RuntimeClass labeling and workload execution with kata-qemu.

• All six CLI conformance scenarios passed.
• Kubernetes E2E results: 81 passed, 1 failed, 15 timed out, and 3 skipped.
• Kubernetes reported Creating watcher returned error too many open files during Kata sandbox creation.
• The assertion failure concerned an SSH session failing to reattach.

These results establish a working baseline but do not yet demonstrate reliable concurrent execution.

Acceptance Criteria

• Manual Kubernetes E2E testing supports a configurable Kata RuntimeClass.
• The selected RuntimeClass satisfies sandbox resource admission, and workload pods demonstrably use it.
• Existing behavior is preserved when no Kata RuntimeClass is specified.
• Manual testing prerequisites, commands, reports, and known limitations are documented.
• Runner evaluation identifies a suitable GitHub Actions configuration and documents its concurrency, cost, and limitations.
• A GitHub Actions Kata lane passes CLI conformance and the agreed E2E coverage repeatedly.
• Any test exclusions have documented reasons.
• CI publishes reports and useful diagnostics and cleans up test resources.
• Contributors can reproduce CI failures using the manual testing workflow.

Alternatives Considered

• Manual Kata testing only: useful for development and debugging, but insufficient for continuous regression coverage.
• Standard Kubernetes E2E coverage only: does not exercise workloads running inside Kata VMs.
• A separate Kata test suite: would duplicate existing coverage; extending the existing Kubernetes harness keeps local and CI testing aligned.
• Self-hosted runners: remain a candidate if hosted runners cannot provide reliable virtualization and sufficient capacity.

Agent Investigation

No response

Checklist

• I've reviewed existing issues and the published docs
• This is a design proposal, not a "please build this" request