Repository navigation
ci: compile once and hand binaries to the test jobs - #142
Merged
Merged
Conversation
The unit-test, spec, harness and both integration jobs each compiled the same gRPC, abseil and BoringSSL objects, about ten minutes per job, and the two integration jobs built an identical `dist`. A restored Zig cache never spared them that: Zig 0.17.0 and 0.17.1 treat a C or C++ source whose inode or mtime changed as modified even when its contents are identical (https://codeberg.org/ziglang/zig/issues/37200, fixed on master by 9343d0fe4b), so `test-server` took ~600s whether setup-zig restored 0 or 903 MB. A `build` job now compiles the test binaries, the release tree and the harnesses, and saves zig-out to the cache under the commit SHA. The other jobs restore it and run the binaries directly. `install-harness` is the new build step that installs the spec runner and the three concurrency harnesses without running them. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Member
Author
|
Zig 0.17 introduced a bug that caused metadata-based caching of C compilation to overrule conte-based digests caching - so each of our CI jobs is currently ~10 minutes as they recompile our gRPC components. This is a small improvement to save some runner time. 0.17.2 (or maybe 0.18) release will contain 9343d0fe4b which fixes the issue |
incidentist
approved these changes
Nov 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI now compiles once per run, in a new
buildjob, and the unit-test, spec, harness and integration jobs run the binaries it produced. Each of those jobs used to spend about ten minutes compiling the same gRPC, abseil and BoringSSL objects, and the two integration jobs built an identicaldist.No issue is linked: this is a CI change requested directly by a maintainer, so the BigQuery sections of the template do not apply.
Why the Zig cache could not do this
setup-zig already restores
.zig-cachein every job, but it has never saved any time. Over the last 40 runszig build test-servertook about 600s whether the restored cache was 0, 715 or 903 MB.The cause is a Zig bug, ziglang/zig#37200: a C or C++ source whose inode or mtime changed counts as modified even when its contents are identical. Dependency sources are unpacked fresh into
zig-pkg/on every runner, so every object misses. The fix (9343d0fe4b) is on Zig master only; 0.17.0 and 0.17.1 both have the bug. A tar archive cannot preserve inodes, so sharing the Zig cache between jobs through the S3 cache would miss the same way. I reproduced it locally:zig build install-testzig-pkg/re-extracted, or only touched.zig-cacheandzig-pkg/through a tar round trip, nanosecond mtimes keptsrc/touchedWhat changed
buildrunszig build install-test(debug) andzig build dist install-harness -Drelease=true, then saveszig-outto the cache underzig-out-linux-x64-<commit SHA>. Re-running a failed job finds those binaries and skips the compile.install-harnessis a newbuild.zigstep. It installsspec-runner,analyzer-threads,shared-catalog-threadsandgrpc-smoketozig-out/harness/without running them. They stay out ofbin/because that directory is the release tree the Docker image copies.zig-outand run the binaries directly. The integration jobs copyzig-out/bininto the Docker context and no longer need Zig or the shim.The push-only arm64 build is unchanged.
Where to look
./zig-out/tests/engine-testsrather thanzig build test). They find the shim through the rpath they were linked with, which is the same absolutedist/shimpath on every runner.Left out on purpose: the
buildjob still uses setup-zig's cache, which never hits and resets itself above the size limit.Checks
Run locally on macOS against the binaries the two build commands installed:
zig-out/tests/engine-tests(596 passed), the binary behindzig build testzig-out/tests/server-tests(499 passed), the binary behindzig build test-serverzig build spec -- --strict:spec-runner --strictexits 1 locally with regressions. I did not compare that againstmain.zig fmt --check build.zig entrypoints toolsanalyzer-threadsandgrpc-smokeexit 0.shared-catalog-threadsprintedRESULT: FAILon 2 of 5 direct runs and passed its onezig build shared-catalog-threadsrun, so watch that job here.🤖 Generated with Claude Code