[garvrao80]

What this changes

Linux no-GPU CI runs zig build -Dfixed-cpu test-host --test-timeout 20m to collect named host tests and compile checks. The build graph rejects empty named checks, stale exclusions, and aliases that bring excluded work back. The graph regression suite runs when any of its three inputs changes or Git cannot resolve a comparison base.

The previous Linux job reached its one-hour cap while running the 100,000-state startup benchmark. The two large-state timing commands stay available for explicit runs. The startup corpus, startup-scale test, smaller cache-index timing check, and Linux nightly shard steps remain in the per-PR graph.

• Issue: Refs CI never reaches 246 of the 393 named test steps, and nine of them fail on main #2031.
• Stacked on: main at e5b2d7dd507c83d140fdcc07f9dec56ea34ef39e, merged into the published branch.
• atc overlap --json: incomplete (pullFilesUnavailable, timeLimit). The GitHub inventory checked 185 open PRs and 26 overlaps with zero new conflicts. The reported lfm3 CUDA: serve the block FP8 export weight-only and the MLX 4-bit export as stored (stacked on #1508) #1634 conflict is identical on main and this branch.

Reviewed exclusions

• The optional xgrammar steps require a separate checkout and flag. The release-path check has its own CI job.
• test-startup-states-timing and test-imprint-ops-timing seed up to 100,000 states and run fresh processes for timing. The graph omits 78 benchmark invocations while still reaching the shared corpus check.
• test-metal-residency-owners-host reproduces the hosted crash on untouched Linux main. The recording runner leaves kv unset, so residency attempts to use an invalid buffer. Setting r.kv = null in an isolated test copy gives 63 passes and 2 skips.
• test-qwen35-draft-frame-host and test-qwen35-frame-invalidation-host still abort on main in the recording fixture at zig/src/cuda/affine4_kernels.zig:249. The fixture leaves seg_fn unset.
• test-gradient-serving still reports a reviewed-source mismatch in zig/kernels/cuda/flashnext_hc.cu on main.

The six checks restored after #2527 and #2550 remain in the graph. test-sessions-writes-host remains reachable through the Linux DeepSeek shard. Its real-file readback test still fails on untouched Linux main (2 passed, 1 failed), matching main's nightly shard.

Checks on this head

• Head commit: 7b7e1118081c0bde74ac860ad2e32e8a5b45220a.
• Lean checker, python3 tools/zig/lean_check.py: 386 findings on base and head, 0 new. zig fmt --check build.zig zig and git diff --check passed.
• Tests: all 9 graph regressions passed. The configured Linux graph reaches 471 named checks, 596 run steps and 624 compile steps, with zero guard failures. zig build -Dfixed-cpu test-startup-states-corpus-host test-cache-index-timing test-qwen35-live-host -j4 --test-timeout 10m --seed=0xece1bd70 --summary all passed 11/11 build steps and 26/26 tests.
• CI: No GPU and Zig format started on this head.
• Not run locally: the full aggregate after the earlier cold build filled the system drive, Windows, macOS/Xcode, or device and checkpoint tests.

Receipt

n/a for inference. This changes build graph collection, Linux CI, contributor guidance and the graph regression fixture. Local checks use Zig 0.17.0 on Linux x86_64 (WSL).

Checklist

• The title and description match the head commit, including the receipt and the stack.
• New tests fail before the change, pass after it, and skip cleanly without their dependency.
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.

[ashhart]

Thanks @garvrao80, this is the shape we agreed on #2031, and the guards against stale exclusions and aliases are a good addition. One change so it can land green: please put the checks still red on main on the exclusion list, each with the reason "red on main, #2031" and the PR that fixes it, so each comes off when its fix lands. #2098 has already fixed two of the eight. Of the rest, the two source-pin checks (test-qwen-serving-host and test-gradient-serving) are ours to review, and we'll do that next.

[ashhart]

@garvrao80, the two source pins are ours to settle. We'll re-baseline both to main's bytes in one reviewed commit that says no device re-qualification is claimed, and drop ck_experts.cu from the gradient manifest, since its own note says the learners don't read it. Until that lands, please put test-qwen-serving-host and test-gradient-serving on the exclusion list with the reason "source pins awaiting re-baseline (#2031)", beside the six fixture prerequisites, so the gate can land green.

[ashhart]

Thanks @garvrao80. CI changed under this PR today, so here's what it costs now and what's needed to land it. In short, it's worth having: it catches the 246 named steps CI never ran. With main's cache it costs a few minutes on most pull requests, and there are four asks.

What it costs

These are measured on hosted Linux runners, with this PR's test-host and exclusions on current main, one run each. Runners differ, so read them as rough sizes.

	test (today)	test-host
Cold, as every main commit runs	11.7 min	28.4 min (12,328 passed, 94 skipped, all green)
Restored cache, nothing changed	5.3 min	8.6 min
Restored, one family file changed	6.2 min	14.7 min
Restored, one zig/src/core file changed	5.8 min	over 21 min (cancelled, unfinished)

The restored rows used main's cache with test passes reused, as #2274 makes CI do. They also predate the tool that stops C and C++ from recompiling, so both columns should come down a little. A cold test-host run spends 12 minutes on one binary, deepseek_v41/device_host_test.zig, which is why it needs the 20-minute test timeout. Then qwen4_family (4 min) and rows_startup_scale_test (3 min).

The saved cache grows from 0.84 GB to 1.87 GB a save, against GitHub's 10 GB for the repository. That fits, but fewer main saves stay in the budget.

What's needed

1. Rebase on main. CI: main's build cache fits every runner, and release-paths and macOS use it too #2252 is on main, and CI: a pull request reuses main's test passes and C compiles, so it runs only what its diff changed #2274 is next in the landing queue. no-gpu.yml has changed a lot: the new line becomes zig build -Dfixed-cpu test-host --test-timeout 20m, without -Drerun-tests, which CI: a pull request reuses main's test passes and C compiles, so it runs only what its diff changed #2274 removes.
2. Run Host test graph regressions only when it can change. It takes about 2 minutes on every run, and only zig/build/host_tests.zig, tools/zig/host_tests_test.py and build.zig affect it. A git diff --quiet guard like CI: a pull request reuses main's test passes and C compiles, so it runs only what its diff changed #2274's Restored cache regressions step would do.
3. Ours: the tests' file reads. With CI: a pull request reuses main's test passes and C compiles, so it runs only what its diff changed #2274, a pull request reuses a test's pass unless the test's binary or a declared input changed. A test that reads a tracked file while it runs has to declare it in zig/build/test_reads.zig. The 246 new steps haven't been traced yet; I'll trace them and add what's needed. This shouldn't land before that.
4. Optional: the DeepSeek enumeration. Building device_host_test in ReleaseSafe, as Build: qwen4's and GGUF's host tests compile with ReleaseSafe #2240 does for the heaviest binaries, would likely bring its 12 minutes down a long way and let the timeout go back to 10 minutes.

Once 1 and 2 are in and the trace is done, it can go in the landing queue.

[ashhart]

@garvrao80, an update on ask 3 (the file-read trace). It's settled on our side. We traced a cold test-host on a hosted runner: the trace saw every one of the 618 runs the build summary lists, across 273 binaries. None of the 246 new steps reads tracked files without either declaring them or always running, so this PR needs nothing added to zig/build/test_reads.zig. The one gap the trace found was already on main (the portable-names test), and #2311 fixes it.

So what's left is yours: ask 1 (the rebase) and ask 2 (guarding the two-minute Host test graph regressions step). Ask 4 stays optional.

[ashhart]

/atc exchange 4659daa814b96b2b72c65474fef9b39704cbe31fab546e38f616b4fc0a2a626f

[ashhart]

Thanks @garvrao80. Asks 1 and 2 are done: the branch is current with main, and the graph regression step skips when its inputs haven't changed.

Two things landed on main today that affect this PR:

1. All nine temporary exclusions are fixed on main. Qwen3.8-27B on Metal: TF_PROMPT_ROWS below 128 gets the prompt floor's message again #2527 and Tests: thirteen host checks that no CI job ran build and pass again on main #2550 fixed the host checks that were red. Tests: thirteen host checks that no CI job ran build and pass again on main #2550 also fixes the test-qwen35-live-host crash in your last run, where a test fixture left tree_nodes unset. That was main's bug, not this PR's. Please remove the nine "red on main, CI never reaches 246 of the 393 named test steps, and nine of them fail on main #2031" entries and their matching fixture lines in tools/zig/host_tests_test.py, as your CONTRIBUTING note says, then merge main once more. That leaves the three permanent exclusions: the two xgrammar checks and check-release-paths.
2. CI: a nightly job for the 246 host checks no per-PR job runs #2554 added nightly jobs for the same 246 checks. They run in shards and print their wall times. Your test-host also collects their test-host-ci-linux-* aggregates because they're named test-*. That's harmless, since they only point at the same steps, so you don't need to change anything for it.

When that run is green, we'll compare its Linux job time with the nightly shard timings before landing it.

[ashhart]

Thanks @garvrao80, and thanks for the careful notes on each exclusion. The merge with main is in, and so are the guarded graph-regression step and the six restored checks.

We went through this head's run. None of the three red marks is caused by this PR:

1. Zig format never started: GitHub reports 0 jobs and a workflow-file problem. The workflow is identical to main's and passed on all your earlier heads, so we'll rerun it.
2. test-qwen-serving-host fails because main's pin at zig/dense_serving_test.zig:23 still expects the old kernels/cuda/qwen35_draft.cu. The file changed on main with the DSpine drafter after your previous run. That's ours to re-pin.
3. test-sessions-writes-host fails the same way in main's own nightly run, so that one is ours too.

On the four exclusions you kept at zig/build/host_tests.zig:13-16, your evidence holds up:

• We recomputed the test-gradient-serving pins on main and found flashnext_hc.cu and flashnext/cuda_batch.zig out of date.
• Your residency trace comes from a run that changes no code under test.
• For the draft-frame pair, the fixture lines in fix(cuda): allocate Qwen3.5 scratch at load #1879 (gemv_fn set and split = null) look like the same fix you describe.

Please keep all four for now; we'll take each one off as its fix lands. The two timing exclusions at :11-12 are fine with us.

Nothing more is needed from you on the code. We'll fix the two red checks on main first. Then please merge main once more (or tell us and we'll do it), and we'll rerun CI and compare the Linux job time with the nightly shards, as planned. If it's green, it goes into the landing queue.

[ashhart]

@garvrao80 both red checks were main's, and both are fixed there now: #2738 re-pinned the serving fixtures and #2731 fixed the sessions-writes load. I've merged main into your branch as we offered (77d428f5e, a plain merge on top of your 7b7e11180, nothing of yours changed), so CI reruns on it now. If it's green, it goes into our landing queue, and we'll compare the Linux job's time with the nightly shards as planned.

• Synthena (Ash's Agent)

[ashhart]

Thanks @garvrao80, this did exactly what it's for on its first train tonight: zig build test-host found two things on main that the current CI never builds. zig/qwen35_copies_host.zig stopped compiling after #2703 changed copyMatch's arguments (fixed in #2836), and dense_serving_test's pinned CUDA image check fails because a few sources changed without being re-pinned (tomorrow's release pins them all). So #2095 is held until both are on main, and then it goes back in the landing queue; nothing is needed from you in the meantime.

• Synthena (Ash's Agent)

[ashhart]

Thank you @garvrao80. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes.