Skip to content

ci: compile once and hand binaries to the test jobs - #142

Merged
ohaibbq merged 2 commits into
mainfrom
ci/build-once
Nov 13, 2026
Merged

ohaibbq merged 2 commits into
mainfrom
ci/build-once

Conversation

@ohaibbq

@ohaibbq ohaibbq commented Nov 13, 2026

Copy link
Copy Markdown
Member

CI now compiles once per run, in a new build job, and the unit-test, spec, harness and integration jobs run the binaries it produced. Each of those jobs used to spend about ten minutes compiling the same gRPC, abseil and BoringSSL objects, and the two integration jobs built an identical dist.

No issue is linked: this is a CI change requested directly by a maintainer, so the BigQuery sections of the template do not apply.

Why the Zig cache could not do this

setup-zig already restores .zig-cache in every job, but it has never saved any time. Over the last 40 runs zig build test-server took about 600s whether the restored cache was 0, 715 or 903 MB.

The cause is a Zig bug, ziglang/zig#37200: a C or C++ source whose inode or mtime changed counts as modified even when its contents are identical. Dependency sources are unpacked fresh into zig-pkg/ on every runner, so every object misses. The fix (9343d0fe4b) is on Zig master only; 0.17.0 and 0.17.1 both have the bug. A tar archive cannot preserve inodes, so sharing the Zig cache between jobs through the S3 cache would miss the same way. I reproduced it locally:

Scenario zig build install-test
Warm rebuild 1s
zig-pkg/ re-extracted, or only touched ~170s, every C library recompiled
.zig-cache and zig-pkg/ through a tar round trip, nanosecond mtimes kept ~170s
Only src/ touched 15s, gRPC still cached

What changed

  • build runs zig build install-test (debug) and zig build dist install-harness -Drelease=true, then saves zig-out to the cache under zig-out-linux-x64-<commit SHA>. Re-running a failed job finds those binaries and skips the compile.
  • install-harness is a new build.zig step. It installs spec-runner, analyzer-threads, shared-catalog-threads and grpc-smoke to zig-out/harness/ without running them. They stay out of bin/ because that directory is the release tree the Docker image copies.
  • The downstream jobs restore zig-out and run the binaries directly. The integration jobs copy zig-out/bin into the Docker context and no longer need Zig or the shim.

The push-only arm64 build is unchanged.

Where to look

Left out on purpose: the build job still uses setup-zig's cache, which never hits and resets itself above the size limit.

Checks

Run locally on macOS against the binaries the two build commands installed:

  • zig-out/tests/engine-tests (596 passed), the binary behind zig build test
  • zig-out/tests/server-tests (499 passed), the binary behind zig build test-server
  • zig build spec -- --strict: spec-runner --strict exits 1 locally with regressions. I did not compare that against main.
  • zig fmt --check build.zig entrypoints tools
  • ephor reports zero findings

analyzer-threads and grpc-smoke exit 0. shared-catalog-threads printed RESULT: FAIL on 2 of 5 direct runs and passed its one zig build shared-catalog-threads run, so watch that job here.

🤖 Generated with Claude Code

The unit-test, spec, harness and both integration jobs each compiled the
same gRPC, abseil and BoringSSL objects, about ten minutes per job, and the
two integration jobs built an identical `dist`. A restored Zig cache never
spared them that: Zig 0.17.0 and 0.17.1 treat a C or C++ source whose inode
or mtime changed as modified even when its contents are identical
(https://codeberg.org/ziglang/zig/issues/37200, fixed on master by
9343d0fe4b), so `test-server` took ~600s whether setup-zig restored 0 or
903 MB.

A `build` job now compiles the test binaries, the release tree and the
harnesses, and saves zig-out to the cache under the commit SHA. The other
jobs restore it and run the binaries directly. `install-harness` is the new
build step that installs the spec runner and the three concurrency
harnesses without running them.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@ohaibbq
ohaibbq requested a review from incidentist November 13, 2026 18:59
@ohaibbq

ohaibbq commented Nov 13, 2026

Copy link
Copy Markdown
Member Author

Zig 0.17 introduced a bug that caused metadata-based caching of C compilation to overrule conte-based digests caching - so each of our CI jobs is currently ~10 minutes as they recompile our gRPC components.

This is a small improvement to save some runner time. 0.17.2 (or maybe 0.18) release will contain 9343d0fe4b which fixes the issue

@ohaibbq
ohaibbq enabled auto-merge (squash) November 13, 2026 19:43
@ohaibbq
ohaibbq merged commit 62b4753 into main Nov 13, 2026
19 checks passed
@ohaibbq
ohaibbq deleted the ci/build-once branch November 13, 2026 20:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants