Repository navigation
Conversation
4a919f2 to
9598ac8
Compare
ashhart
changed the title
|
CI doesn't show a real gain from this yet, so I'm holding it as a draft. One full run on this head, against the last 29 green runs:
Linux is 1.4 min under its median and Windows 2.3 min over, both inside the normal spread. The test run itself is much shorter, but the runners rebuild from empty every time, so the slower LLVM compile takes most of that back. Saved CPU also doesn't turn into wall time one for one on four vCPUs. A warm build cache should change this, because the compile would only happen when the code changes and the shorter run would count on every PR. I'll measure again once #2140 is on main, and land this only if |
qwen4's take most of the test binaries' CPU time on CI, over six minutes on Linux and three on Windows, nearly all of it interpreting numeric kernels in Debug. Optimised, with safety checks still on, the same 113 tests run about nine times faster. The GGUF steps already ran each binary twice, Debug and ReleaseFast; their Debug run becomes ReleaseSafe, keeping the safety checks for most of the gain.
9598ac8 to
4f8bca5
Compare
|
Closing this, because #2274 has taken away what it was for. With a warm cache, ReleaseSafe makes these tests about 8 times cheaper: on an M3 Ultra, qwen4's and GGUF's test runs total 32 s against 264 s in Debug. But since #2274, a pull request reuses main's test passes and compiles, so these tests only run again when a change touches their inputs. Main still builds everything cold. There, this branch's earlier runs had the test step 1.4 minutes faster on Linux but 2.3 minutes slower on Windows, where the C++ compiles take longer optimized. That doesn't pay. |
What
The
qwen4_familyhost test binary is the slowest inzig build test: over 6 minutes on CI's Linux runners and over 3 on Windows, in a cold run from 4 Nov. Nearly all of that is its numeric tests running CPU kernels unoptimised.This compiles that binary with ReleaseSafe. Safety checks stay on. The change is two
.optimizevalues inzig/build/qwen4.zig, and it doesn't touch the workflow.The GGUF steps get the same change. They already run each binary twice, in Debug and ReleaseFast, and on CI the Debug runs took over a minute and 28 s against about 8 s and 1 s. Their Debug run becomes ReleaseSafe, so the safety checks still run.
Numbers
zig build test-qwen4-hoston an M3 Ultra:On the same machine, the GGUF steps' 25-test binary goes from 17 s to 2 s and the 15-test one from 8 s to 0.4 s, with compiles of 1-2 s becoming 8-9 s.
All tests pass both ways. The compile is slower, so CI's own times on this PR are the real result. For comparison, over recent runs the
zigjob's median is 17.2 min andzig-windows' is 23.4 min. Compile cost matters more onzig-windows, where LLVM is slowest, so that's the job to watch.What it gives up
Debug fills undefined memory with a marker value, and ReleaseSafe doesn't, so a read of undefined memory in these tests is less likely to fail. Bounds, overflow and
unreachablechecks all remain. The other test binaries still run in Debug. The rest of the slow ones aren't worth it yet: the server tests are a long tail with no single slow test, and many of them wait on sockets and timers.Exactness and tests
This changes how tests are built, not what ships. The numeric tests compare bits, and they pass in ReleaseSafe on this Mac; this PR's CI covers Linux and Windows.