Repository navigation
ci: run the headless integration tests, and gate on them - #232
Merged
Merged
Conversation
`zig build test-integration` (146 integration tests on dvui's testing backend, plus the SDK, sizing and plugin-loader suites: 292 in all) was run by no CI job, so a layout or drag regression reached main unless someone remembered to run it; on 10-03 two of its tests had been failing unnoticed. It runs now in a Linux job of its own beside the test matrix, and `ci-ok` waits on it. Linux only: the code under test is the same everywhere, and on Windows it needs MSVC. The job notes why its log can say `failed command:` on a pass (a test that logs a warning; the demo-replay test always does). CONTRIBUTING.md and CLAUDE.md stop saying CI skips the suite. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2 of 4 tasks
foxnne
added a commit
that referenced
this pull request
Oct 8, 2026
Follows #232, now merged; rebased onto `main`. It touches only `ci.yml`. ## What changes **Every other CI run on `main` was building from nothing, and this fixes it.** I first guessed that cancelled runs were the cause. The logs show otherwise: | Run | What its Linux test cache did | |---|---| | 37320334563, `main`, 10-05 | Cold; ended at 3.67 GB, under the limit, so it was saved | | 37682171347, `main`, 10-07 | Restored 683 MB (compressed); ended at **6.94 GB** and was "exceeding limit of 5368709120 bytes; clearing cache", so it was saved empty | | 37785306976, `main`, 10-08 | Restored that empty cache (**188 B**) and built from scratch, about 10 min | | 37786098520 and 37786730490, 10-08 | 7.07 GB and 5.69 GB, both cleared and saved empty | The cause is how setup-zig caches. It saves each run under a new key and restores the newest, so a warm run saves what it restored plus everything it rebuilt. For Linux, one run is enough to go past the limit, and an over-limit cache is saved empty. Cancellation only took part because cancelled runs grow too. The near-duplicates also fill the repo's 10 GB of Actions cache, which stood at 9.6 GB. **Now:** - setup-zig's own caching is off (`use-cache: false`). It still points `ZIG_GLOBAL_CACHE_DIR` and `ZIG_LOCAL_CACHE_DIR` at `.zig-cache`. - Each job restores and saves `.zig-cache` itself with `actions/cache/restore` and `actions/cache/save`, under a **fixed key**: Zig version, job, OS, arch, and the hash of every `build.zig.zon`, taken before anything is unpacked. - A fixed key can't be overwritten, so **only a run that missed saves, and only if it succeeded.** A job's cache is one complete cold build (about 3.7 GB for Linux tests), and every later run with the same dependencies, Zig and OS starts from it. Nothing reaches a size limit, warm runs upload nothing, and a run cancelled or failed mid-build never leaves a partial cache. - **A bug fixed on the way:** `Windows build (fizzy backend, cross-compiled)` still computed its key inline, the bug the test job's comment already described. Restore asked for `d259c36b…` and save wrote `a0d09007…`, so it missed every time (run 37786098520: "Cache miss"). It now takes its key from a step, like the other jobs. **What a warm run keeps, and what it doesn't:** - It reuses the expensive, unchanging parts: package downloads (no network, so no `HttpConnectionClosing`), the C libraries (SDL3 ×5, freetype, tree-sitter), and the build tools. - The app, tests and wasm are rebuilt whenever their code changed, which is true of nearly every commit. That was already the case for warm runs. - A cache is rebuilt from cold only when a `build.zig.zon`, Zig or the runner OS changes. **Not touched:** `web.yml`'s `build` job also uses setup-zig's caching. Its entries stay around 264 MB, so it doesn't hit the limit. It could move to the same scheme later. ## SDK impact - [x] None ## Verified - [x] The workflow parses. Each of the three jobs runs: key step → setup-zig (`use-cache: false`) → restore → … → save, with save gated on `success() && cache-hit != 'true'`. No setup-zig cache inputs remain. - [ ] **Live:** this PR's first run will miss every new key (cold) and save one entry per job. Its next run should restore them ("Cache restored from key: zig-0.16.0-test-Linux-X64-…") and skip "Save Zig cache". I'll check both before asking for a merge. - [ ] After merge, `main`'s first run is cold once and saves `main`'s copies, which every PR then restores. The old `setup-zig-cache-v2-*` entries go unused and age out after 7 days, or sooner under the 10 GB limit. ## Follow-ups - Move `web.yml` to the same scheme if its cache ever grows. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up from #230.
What changes
zig build test-integrationnow runs in CI, andci-okwaits on it. The suite is 146 integration tests that drive real widgets, layouts and drags on dvui's testing backend, plus the SDK, sizing and plugin-loader suites: 292 tests in all. No CI job ran it before, so a layout or drag regression reachedmainunless someone remembered to run it locally. On 10-03, two of its tests had been failing unnoticed.Integration tests (Linux), beside the test matrix. It adds a compile to each run, but not to how long a run takes, and a failure shows under its own check name. It is Linux only because the code under test is the same on every OS, and on Windows the suite needs MSVC (build/app.zig). It runs whenever the builds run: on PRs, the merge queue, and pushes tomain.failed command: …fizzy-integration-testson a pass whenever a test logs a warning, and the demo-replay test always does. The exit code is the verdict.CONTRIBUTING.mdandCLAUDE.mdno longer say CI skips this suite.Cost: one more Linux job per run, and one more Actions cache entry (setup-zig keys caches by job). The repo's cache budget is already at about 9.6 of its 10 GB, so GitHub evicts the least recently used entries a little sooner. See the follow-ups for the larger cache problem I found while checking this.
SDK impact
Verified
main(4dc4d51a):Build Summary: 28/28 steps succeeded; 292/292 tests passedin 1:13, exit 0. The known replay warning printed itsfailed command:line.Integration tests (Linux)run is the first. It is the same headless code, but this is the first time the suite has run on a Linux runner.ci-okneeds[changes, test, integration, windows-fizzy-backend].Follow-ups
mainrestored a 188-byte cache and built from scratch, andgh cache listshows several 0 MB Linux caches. Saving only from runs that weren't cancelled would make most runs warm. Not changed here.🤖 Generated with Claude Code