Vulkan: qmm_q4_tile_rows computes four rows and two columns of a prompt chunk per thread, 3.6% to 33% less run time on an M1 with the same logits (stacked on #2016)

CHANGELOG.md

@@ -72,10 +72,17 @@ Each release's page on GitHub has its notes and measurements. See [the 1.0.0 rel
 - Qwen3.8-27B on Metal: prompt passes no longer write the GDN tails buffer, which nothing read. Replies are unchanged; on an M5 Ultra, time to first token is about 0.5% shorter at 2K, 8K and 32K prompts, one and four streams (#2915, for #2905).
 
 - Flash Next on CUDA at `--tp 2`: a prompt pass wider than 64 rows reduce-scatters its routed experts' FP32 slots by row halves, sums each rank's rows and gathers the sums, where it all-reduced every slot; decode rounds and verify windows keep the all-reduce. Every reduced value is the same single add and the slots are summed in the writeback's order, so replies keep their tokens. Over two GB10s' RoCE link rank 0 sends 3.43 MiB a prompt token instead of 5.81, and cold prefill of 2k to 32k tokens is 26.5% to 26.8% faster there and 30.9% to 33.9% faster on two RTX PRO 6000s (an 8,192-token prompt: 7.12 to 5.62 s and 3.61 to 2.70 s) (#2977). Thanks to @swbiggart.
+- Vulkan: the native dense Qwen3.5-2B greedy engine and its operator gates, the second part of the opt-in Vulkan backend built only with `-Dvulkan=true`: checked BF16 tensor operations, the norm/MLP, attention (RoPE, causal GQA SDPA, output gate) and GatedDeltaNet gates, one GDN path for prefill and decode, prefill chunked within the context ceiling, and the kernel contract extended to every operator's SPIR-V through gates 1 to 4 with planted controls and recorded device findings (residual denormals, the preserve-add subnormal flush, the gated norm at the production per-head geometry). `zig build vulkan-kernel-contract -Dvulkan=true` runs it; default builds are unchanged (#1544). Thanks to @joshuaswarren.
+
+- The Vulkan engine reads Qwen3.5's 4-bit projection weights and activations 16 bytes at a time (`qmm_q4_vec4`) when both are 16-byte aligned, and keeps the scalar kernel otherwise. Each output keeps the same sum in the same order, so the logits are byte-identical. On a 13-inch M1 with Qwen3.5-2B 4-bit, a run (prefill, 63 decode steps and the per-step logits readback) took 34.8% to 43.0% less time than the same build with the scalar kernel selected, on three inputs: a 24-token prompt, and documents of 256 and 1,024 tokens (Linux 7.3.0-2-20261110.1-lab-ARCH; the first measurement, against #1544's head on an earlier kernel and Mesa, gave 40% to 52%). The kernel contract checks the new kernel with the same gates as `qmm_bf16`. Receipts: [docs/receipts/vulkan-qmm-vec4-m1](docs/receipts/vulkan-qmm-vec4-m1/README.md).
 
 - Tests: `tf-cuda-test flashnext-rank RANK PORT timeout` passes again on two GPUs. NCCL start-up keeps its 30 s bound and only the stalled collective gets 200 ms, and both ranks meet once their communicators are up, so `timeout` and `eof` fail the collective under test, not the start-up (#2976, fixes #2975). Thanks to @swbiggart.
+- The Vulkan engine stores each Qwen3.5 4-bit projection in tiles of 32 columns, repacked once at load, and reads them with `qmm_q4_tile`: a group of 32 lanes then loads one contiguous 512-byte run instead of 32 rows a kilobyte apart. The loop, the sums and their order are `qmm_q4_vec4`'s, so the logits are byte-identical. A projection is padded to whole tiles, and the tied output head is a second, tiled copy of the embedding table (about 320 MB more device memory). On a 13-inch M1 with Qwen3.5-2B 4-bit, a run took 10.9% less time than with `qmm_q4_vec4` on a 24-token prompt, 35.6% less on a 256-token document and 44.9% less on a 1,024-token document (Linux 7.3.0-2-20261110.1-lab-ARCH; earlier builds on an earlier kernel and Mesa gave 19%, 50% and 63%). The kernel contract checks `qmm_q4_tile` with the same gates as `qmm_bf16`. Receipts: [docs/receipts/vulkan-qmm-tile-m1](docs/receipts/vulkan-qmm-tile-m1/README.md).
 
 - Tokenizer: a split pattern is held to a work allowance per piece (2,048 instructions a byte, counted, not timed) and refused at load when too deep or too large, so a crafted tokenizer or prompt can't hang the pre-tokenizer; every split of the patterns we serve is unchanged (#3071).
+- The Vulkan engine records its dispatches into one command buffer and submits when 128 are waiting, or when the next upload, download or fill needs the queue, instead of one submission and fence wait per dispatch. Each pipeline holds a ring of 256 descriptor sets. No kernel changes, so the logits are byte-identical. On a 13-inch M1 with Qwen3.5-2B 4-bit and tiled projections, a run took 58% less time on a 24-token prompt, 34% less on a 256-token document and 17% less on a 1,024-token document; the gain depends on the kernel and Mesa build (20% on the 256-token document with earlier builds in the same environment). Receipts: [docs/receipts/vulkan-submit-batch-m1](docs/receipts/vulkan-submit-batch-m1/README.md).
+
+- The Vulkan engine computes four rows of x and two columns a thread (`qmm_q4_tile_rows`) for prompt chunks of four rows or more, so a weight is dequantized once for four outputs and each activation is unpacked once for two weights. Each output keeps its own accumulator and the same order of sums as `qmm_q4_tile`, so the logits are byte-identical. Decode and chunks of one to three rows keep `qmm_q4_tile`. On a 13-inch M1 with Qwen3.5-2B 4-bit, a run took 4% less time on a 24-token prompt, 12% less on a 256-token document and 15% less on a 1,024-token document (6%, 29% and 38% in an earlier run on older builds, an older kernel and an older Mesa). The kernel contract checks `qmm_q4_tile_rows` with the same gates as `qmm_bf16`. Receipts: [docs/receipts/vulkan-qmm-tile-rows-m1](docs/receipts/vulkan-qmm-tile-rows-m1/README.md).
 
 - Cluster pull: a pull that would leave a file held only in part is refused unless the caller allows unverified bytes (off by default); a pull that ends whole still resumes and is checked against the manifest's hash (#3074).
 
@@ -134,6 +141,8 @@ Each release's page on GitHub has its notes and measurements. See [the 1.0.0 rel
 - Metal: kernels nothing launches are removed: Qwen3.8-27B's `prompt_glue.metal` and `prompt_state.metal` (six `q27_prompt_*` kernels) and core's `tf_lane_rope` with `Attention.rotate`; saved Qwen3.8-27B prompt states on Metal are recomputed once after upgrading, since their identity covers the kernel sources (#3065, part of #2251). Thanks to @jschmied.
 - Tests: main's nightly is green again. The Flash Next learn, prompt and compare tools build again on Linux (#2585 had put one file in two modules), and the dense serving and serving-identity pins take the changes #2579, #2585, #2285, #2205 and #2789 made, each named beside its pin. Serving is unchanged (#3002).
 - HIP: Qwen3.8-Flash-Next's MTP head takes a prompt chunk on the prompt tiles, not a decode window at a time, so cold prompts finish about 5% sooner at 8K and 32K on a Radeon 8060S (gfx1151). Replies are unchanged; the head's caches and drafts are not bit-identical to before (#1878). Thanks to @gilby.
+- The Vulkan engine stores each Qwen3.5 4-bit projection in tiles of 32 columns, repacked once at load, and reads them with `qmm_q4_tile`: a group of 32 lanes then loads one contiguous 512-byte run instead of 32 rows a kilobyte apart. The loop, the sums and their order are `qmm_q4_vec4`'s, so the logits are byte-identical. A projection is padded to whole tiles, and the tied output head is a second, tiled copy of the embedding table (about 320 MB more device memory). On a 13-inch M1 with Qwen3.5-2B 4-bit, a run took 19% less time than with `qmm_q4_vec4` on a 24-token prompt, 50% less on a 256-token document and 63% less on a 1,024-token document. The kernel contract checks `qmm_q4_tile` with the same gates as `qmm_bf16`. Receipts: [docs/receipts/vulkan-qmm-tile-m1](docs/receipts/vulkan-qmm-tile-m1/README.md).
+- The Vulkan engine records its dispatches into one command buffer and submits when 128 are waiting, or when the next upload, download or fill needs the queue, instead of one submission and fence wait per dispatch. Each pipeline holds a ring of 256 descriptor sets. No kernel changes, so the logits are byte-identical. On a 13-inch M1 with Qwen3.5-2B 4-bit and tiled projections, a run took 68% less time on a 24-token prompt, 49% less on a 256-token document and 30% less on a 1,024-token document. Receipts: [docs/receipts/vulkan-submit-batch-m1](docs/receipts/vulkan-submit-batch-m1/README.md).
 
 - Cortheon (`--cortheon`): DuckDuckGo results are kept, and its ads still skipped, when only the spelling differs (either quote style, a `class` attribute in any letter case, an unquoted `href`, spaces around its `=`, or a line break before it); a duplicate whose scheme or host differs only in letter case is dropped while paths and queries keep their case, and the PyPI, npm and crates.io allow-list takes the host in any case with the path exact (#2750). Thanks to @ppardus.
 
@@ -1090,8 +1099,6 @@ Each release's page on GitHub has its notes and measurements. See [the 1.0.0 rel
 
 - `zig build test-info-config` runs info's issue-934 test again: it was built from the CUDA CLI root, which never reached it (#2006). Thanks to @jishnuvenugopal.
 
-- The Nemotron XPU loader and the XPU safetensors reader read shard headers through core, so a malformed or oversized header is refused before anything is read (#1919). Thanks to @jschmied.
-
 - Windows: a cost-cache read that lands mid-replace is counted as a miss (safe: the window is priced again), so its concurrency test no longer flakes (#2001).
 
 - GLM on CUDA with TF_GLM_KV_SHARE no longer reports a balanced idle KV pool as (UNEVEN) at close; nothing else changes (#1993). Thanks to @Bizuayeu.

THIRD_PARTY_NOTICES.md

@@ -71,12 +71,31 @@ The native engine carries that adaptation. `zig/kernels/metal/qwen3_5/gdn_pre.me
 `zig/kernels/metal/core/deltanet.metal` and Flash Next's `zig/kernels/metal/flashnext/q4_gdn_*.metal` run the same
 model math.
 
-## Vulkan affine projections
+## Vulkan shaders from omarchy-mlx
+
+These files in `zig/kernels/vulkan/` come from [omarchy-mlx](https://github.com/joshuaswarren/omarchy-mlx),
+MIT License, Copyright © 2026 Joshua Warren and mlx-omarchy contributors. The first lines of each file name its
+source path and commit. The native Vulkan path does not link MLX.
+
+- `argreduce_chunk.comp`
+- `argreduce_combine.comp`
+- `binary_vec.comp`
+- `cast.comp`
+- `conv_dw1d.comp`
+- `copy_general.comp`
+- `dequant.comp`
+- `fast_norm.comp`
+- `fast_norm_gated.comp`
+- `fast_rope.comp`
+- `gated_delta_decode.comp`
+- `gated_delta_prefill_ash.comp`
+- `gather_take.comp`
+- `gdn_conv_decode.comp`
+- `omarchy_trig.h`
+- `qmm.comp`
+- `sdpa_decode_native.comp`
+- `swiglu.comp`
 
-`zig/kernels/vulkan/qmm.comp` adapts the GLSL shader from
-[mlx-omarchy](https://github.com/joshuaswarren/omarchy-mlx), MIT License,
-Copyright © 2026 Joshua Warren and mlx-omarchy contributors.
-Its arithmetic is unchanged; the native Vulkan path does not link MLX.
 See [the license text](LICENSES/mlx-omarchy-MIT.txt).
 
 ## Qwen Flash Next

build.zig

@@ -218,7 +218,10 @@ pub fn build(b: *std.Build) void {
     if (with_hip) hip_build.steps(b, target, optimize, build_options, test_step, hip_options);
     const with_vulkan = b.option(bool, "vulkan", "Enable the opt-in Vulkan backend and its host/device targets") orelse false;
     const vulkan_options = vulkan_build.options(b); // declared on every target, used only with -Dvulkan=true
-    if (with_vulkan) vulkan_build.steps(b, target, optimize, test_step, vulkan_options);
+    if (with_vulkan) {
+        vulkan_build.steps(b, target, optimize, test_step, vulkan_options);
+        @import("zig/build/vulkan_model.zig").steps(b, target, optimize, test_step, vulkan_options, build_options);
+    }
     exlr8_build.hostTests(b, target, test_step);
     xpu_build.steps(b, target, optimize, build_options);
     video_build.steps(b, target, test_step);

docs/credits.md

@@ -120,6 +120,14 @@ contributions are in it.
   ([lane_qmm.py](https://github.com/ashhart/TensorFold/blob/v0.6.6/src/tensorfold/kernels/qwen/dense/v1/lane_qmm.py)),
   widening each group to nibbles for the 4-bit tensor op. The native `zig/kernels/metal/core/lane_projection.metal`
   stages 2- and 3-bit codes the same way.
+- The native Vulkan dense Qwen3.5-2B engine (`zig/src/families/qwen3_5/backend/vulkan/`) follows the same frozen
+  dense-lane design ([lane_qmm.py](https://github.com/ashhart/TensorFold/blob/v0.6.6/src/tensorfold/kernels/qwen/dense/v1/lane_qmm.py),
+  [#34](https://github.com/ashhart/TensorFold/pull/34)) for its per-lane qmm decomposition. Fifteen of its compute
+  shaders and `omarchy_trig.h` come from Joshua Warren's MIT-licensed omarchy-mlx unchanged (`argreduce_chunk`,
+  `argreduce_combine`, `binary_vec`, `cast`, `conv_dw1d`, `copy_general`, `dequant`, `fast_norm`, `fast_norm_gated`,
+  `fast_rope`, `gather_take`, `gated_delta_decode`, `gated_delta_prefill_oracle`, `gdn_conv_decode`, `sdpa_decode_native`
+  and `swiglu`); `qmm.comp` is derived from it. The rest (`attention_gate`, `finite_logits`, `gdn_split`, `qmm_q4_vec4`
+  and the contract controls) are new work, written for this engine.
 - @di37 made the engine read the GPU generation once from `applegpu_gN`, and a virtual machine's GPU as none, in
   [#209](https://github.com/ashhart/TensorFold/pull/209)
   ([device.py](https://github.com/ashhart/TensorFold/blob/v0.6.6/src/tensorfold/kernels/device.py)), as

docs/receipts/vulkan-qmm-tile-m1/PREREG.md

@@ -0,0 +1,48 @@
+# qmm_q4_tile pre-registration (written 2026-11-03 before any GPU run of this change)
+
+## Question
+
+Does storing each projection's weights, scales and biases in 32-column tiles (a load-time repack) make a Qwen3.5-2B run shorter
+on the 13-inch M1, with byte-identical logits? The kernel is `qmm_q4_vec4` with only its address formulas changed, so every sum is the same.
+
+## Arms (two ELFs of `tf-qwen35-model`, ReleaseSafe, aarch64)
+
+- old: the vec4 build, commit `e742e023`, ELF `683fcf46...`.
+- new: the same plus the tiled image: weights of every projection with n a multiple of 32 are repacked at load; the lm head is a second,
+  tiled copy of the embedding table (the embedding keeps the row layout). Projections with n = 16 keep the vec4 image.
+
+## Gates (every process, both arms)
+
+1. Exit 0 and a complete receipt (the house receipt check).
+2. Generated ids equal the pinned ids.
+3. Logits SHA-256 of every process equals one digest per case, the same digest the earlier vec4 runs gave (unicode-24 `e5447d9a...`,
+   document-256 `147a2d2e...`, document-1024 `7f9d8e65...`).
+4. Before the pairs: the stage 1 probe on the GPU runs the tiled image on every tileable projection case and compares bytes to the default image.
+
+## Metric and design
+
+Run 2 duration from the probe's stderr milestones (prefill plus 63 decode steps plus the per-step logits readback), as before.
+Three pairs per case, order old,new,new,old,old,new. Cases: unicode-24, document-256, document-1024.
+
+## Decision rule
+
+Speedup: median pair gain of at least 3 percent and all three pair gains positive and all gates hold. Otherwise reported as no speedup and not proposed.
+
+## Forecast (a prediction; I do not know the answer)
+
+Gain of 20 to 45 percent of run 2 on unicode-24 and similar on the documents. Why not more: qmm is about 63 percent of the run net of the submit
+cost, the multi-row result showed qmm is not simply load-count limited, and the hardware cache may already hold part of the 1 KiB stride.
+I give 20 percent to a gain under 3 percent, and 10 percent to a regression.
+
+## Not claimed
+
+Other chips, other models, other quantizations, the lm head on devices whose storage range is below the table size (they keep the row layout).
+The software-device pass is not GPU evidence. Memory: the tied lm head adds about 320 MB of device memory.
+
+## Addendum, written after the final pairs ran (2026-11-03)
+
+The first contract run on the GPU refused the served shape 1x16x64: a registered kernel must serve every shape the contract serves, and the build
+above kept the n = 16 projections on the vec4 image. The design changed before the final runs: the repack pads the last tile with zero columns, so the tiled
+image serves every n, the n = 16 projections are tiled too, and the arms read "old: the vec4 build, new: the padded tile build". The gates, metric and
+decision rule did not change. This addendum was written after the final pairs; the three earlier pairs on the unpadded build are kept in the receipts as an
+earlier measurement and are not the result. The forecast above was for the unpadded build; it was not revised.