ci(e2e): cut the VM install time of tmachine E2E suite rows #5438

User Story

As a contributor migrating E2E coverage to testkit suites (#5281), I want each tmachine E2E suite row to spend less time installing OpenShell into its VM, so that adding test packages and environments doesn't add a full VM install to every PR run.

Description

After #5429, a features or drivers row no longer compiles anything. The largest part of a row left is installing OpenShell into the VM, about 90s for every row on every commit.

Baseline row: features/host-gateway on fedora-podman-rootful (run 42707934133, job 129484694793). Ansible task times:

The tests themselves took 11s.

How it works today

tmachine builds each VM disk in layers. Each layer is a qcow2 overlay, a disk file that stores only its changes on top of the layer below:

1. Base image: Ubuntu or Fedora.
2. Setup layers: packages, nextest, test tools. These are cached in the GitHub cache between runs.
3. Install layer: OpenShell installed (tests/tmachine/src/qemu/install.rs).
4. Test session: a throwaway overlay on top of the install layer.

The install layer's cache key covers the installer's inputs, which include the RPMs, debs and images built from this commit. So the install layer never matches across commits. Yet every row still pays for it as a cached layer: it boots a VM, installs, shuts down, and then boots the VM a second time for the tests.

Rows in Branch E2E are one job per (package, environment). Several environments run more than one row, and each of those rows installs OpenShell again:

That's 13 rows and 13 installs for 5 environments.

Proposed changes, roughly in order of payoff

1. Don't reboot after the install. The install layer is per-commit, so caching it saves nothing. Install OpenShell in the test VM and run the tests in the same boot, or keep the install VM running for the tests. That saves the second boot (~18s) in every row. Keep the cached setup layers as they are.
2. Install once per environment. Run every row that shares an environment and installer in one VM: one install, then each family's bundle or program in turn. This has a trade-off. Fewer, longer jobs save runner time but can make the run longer overall: the k3s conformance rows (9–12 min) are already the slowest jobs in the run. One option is to merge only the features rows (host-gateway with oci-image) and leave conformance separate. Measure before deciding.
3. Faster image load. Copying and loading the images takes ~29s. Options to measure: load the runtime and test images in parallel; share the image archives into the guest instead of copying them; skip images the environment's tests don't use.
4. Split cacheable work out of the install layer. Anything the install does that doesn't depend on this commit's packages, such as creating users, enabling lingering, or configuring the container runtime, can move to a cached setup layer.

Context

• ci(e2e): cut tmachine E2E suite row setup time #5429 / PR ci(e2e): build the tmachine suite bundles once per run #5432: rows run prebuilt bundles, so the VM install is now the largest part of a row.
• ci(e2e): shrink the Nix download of tmachine E2E suite rows #5433: shrinks the Nix download of each row (~50s).
• tests/tmachine/src/qemu/install.rs: the install layer and its hash (install_hash)
• tests/tmachine/src/qemu/layer.rs: cached layers and overlays
• tests/tmachine/src/qemu/session.rs: the test session overlay and boot
• tests/tmachine/config.nix, tests/tmachine/ansible/: installers and their playbooks
• .github/workflows/e2e-suites.yml: rows matrix, tmachine disk cache key
• tests/runner/src/execute.rs: one bundle per session today

Definition of Done

• A row boots its VM once after the cached setup layers. There's no separate install boot and test boot.
• The decision on installing once per environment is recorded in this issue with measured run times, and implemented if it shortens the run or saves runner time without making the run longer.
• Provisioning-to-first-test time for a Podman row is recorded before and after in the PR.