Description
User Story
As a contributor migrating E2E coverage to testkit suites (#5281), I want each tmachine E2E suite row to download only what it uses, so that every row doesn't spend ~50s fetching compilers and VM images it never touches.
Description
Before it boots a VM, every tmachine E2E suite row runs nix build .#openshell-test-runner. That downloads the runner and everything it depends on, which Nix calls its closure. In the baseline row (run 42707934133, job 129484694793), that was 581 store paths, about 2.7 GiB compressed, and it took about 50s.
After #5429, rows run test bundles built once per run, so they compile nothing. The runner package still carries the compilers for both architectures, though, because the same package also builds bundles.
Download size by group in the baseline row (compressed, as fetched; some groups overlap):
| Group | Paths | Size | Used by a row that runs a prebuilt bundle? |
|---|---|---|---|
| VM base images (Ubuntu 24.04 and Fedora 44) | 2 | 1145 MiB | Only the row's own image |
| Rust toolchain (rustc, std for 4 targets, cargo, docs, rust-analyzer) | 12 | 491 MiB | No |
| aarch64 cross toolchains (gcc, binutils, z3) | 40 | 378 MiB | No (rows are amd64 only) |
| z3 libraries, every target | 29 | 168 MiB | No |
| Python and Ansible | 78 | 140 MiB | Yes (install layer) |
| QEMU GUI and audio dependencies (SDL2, ALSA, at-spi, flite, freepats, ffmpeg…) | 19 | ~100 MiB | No |
| x86_64 musl gcc and binutils | 6 | 97 MiB | No |
Removing what a row doesn't use, and keeping one base image, leaves an estimated ~1 GiB, down from 2.7 GiB.
Why it's this big
tests/tmachine/config.nix:testRunnerincludesrustToolchainand folds every toolchain's env intoruntimeEnv(aarch64 and x86_64, gnu and musl). So the cross gcc, binutils and z3 for all four targets are part of the package, whatever the host.rustToolchaincomes fromrust-toolchain.toml, which addsrust-srcandrust-analyzerfor the dev shell. The runner also ships those, plusrust-docs(75 MiB together).- Both base images are in the tmachine config, so every row fetches both. Each row only boots one: the install layer is a disk overlay on top of the row's own base image.
qemuPkgs.qemu.override { hostCpuOnly = true; }limits QEMU to the host's CPU type, but keeps the display and audio front ends and their libraries.
Proposed changes, roughly in order of payoff
- A compiler-free runner for rows. Split the package: one that only runs prebuilt bundles and programs (QEMU, Ansible, sshpass, nextest, tmachine config), and one that also builds them. Rows use the first. CI bundle builders and local
runwithout--bundleuse the second. - One base image per row. Fetch only the image of the row's environment, for example through a per-environment tmachine config or by realizing the image lazily.
- Host-arch toolchains in the builder. The builder package only needs the host's gnu toolchain (build scripts) and the host's musl toolchain (guest tests), not all four. Drop
rust-src,rust-analyzerandrust-docsfrom the runner's toolchain. - Headless QEMU (optional, smaller payoff). Build QEMU without display and audio front ends. That build isn't in cache.nixos.org, so it would need building and pushing to the OpenShell Cachix cache.
qemu_testis in the cache, but it includes every CPU target (343 MiB closure vs 423 MiB for the current build), so it doesn't help.
Context
- ci(e2e): cut tmachine E2E suite row setup time #5429: build the suite bundles once per run (PR ci(e2e): build the tmachine suite bundles once per run #5432). Rows then compile nothing, which makes change 1 possible.
tests/tmachine/config.nix:testRunner,runtimeEnv, theqemuoverride, base imagesflake.nix:toolchains,rustToolchain, theopenshell-test-runnerpackagerust-toolchain.toml: dev shell components.github/workflows/e2e-suites.yml: the row's "Run the suites" step (nix build .#openshell-test-runner).github/workflows/branch-e2e.yml:build-e2e-bundles(builder)tests/tmachine/src/qemu/{install.rs,layer.rs}: install layer hash and overlay on the base image
Definition of Done
- A tmachine E2E suite row running a prebuilt bundle or program downloads no Rust toolchain, cross gcc or z3.
- A row downloads only its own environment's base image.
- The row's Nix download is under ~1.2 GiB, and the before and after sizes and times are recorded in the PR.
- Building bundles locally and in CI still works on x86_64 and aarch64 hosts.
Activity