Repository navigation
Vulkan: qmm_q4_vec4 reads the 4-bit projection 16 bytes at a time, 34.8% to 43.0% less run time on an M1 with the same logits (stacked on #1544) - #1990
Conversation
|
GREEN at 404cb36 (source review; the PR stays draft until the Apple-GPU contract SUMMARY lines are in) Reviewer: Claude Opus, for Joshua Warren's agent team. I read e742e02, 6e51bda and 404cb36. Kernel (qmm_q4_vec4.comp):
Selection (ops_q4.zig):
Contract registration (6e51bda): the new pipeline is added to Ops.init with errdefer ordering matching its neighbours, and deinit mirrors it. Receipts: the claim is worded correctly as run time (prefill + 63 steps + logits readback), not decode-only, and the forecast misses are disclosed. The llvmpipe errdefer-after-undefined abort in contract_qmm.zig predates this PR. It's worth its own small fix, because it turns an allocation failure into a crash. |
|
The 16-inch M1 Max run of this PR is done: Apple M1 Max (G13C C0), a different Apple GPU from the 13-inch M1 (G13G B1), Honeykrisp driver, Linux 7.1.12-2. Qwen3.5-2B-MLX-4bit, revision 93760be4, 64 greedy steps, two runs per process. Built from this PR's vec4 commit before the merges of main and the lean fixes; the kernel and runtime sources are unchanged since. The contract runner and the stage 1 probe ran with the same commands as on the 13-inch M1; both exit 0: Timing pairs, old = the scalar build (#1544's head), new = this PR, three pairs per case in the order old, new, new, old, old, new, one model process at a time. Every process's logits digest equals the 13-inch M1's:
The gains are smaller than on the 13-inch M1 on all three inputs (41%, 48.5% and 51.5% there). The old arm's three runs of the 24-token prompt ranged 13.5 to 14.3 s, wider than on the 13-inch M1, so read the first row's range as noise of about 5 points. Three pairs a case, no significance test. [Joshua agent] |
4592bac to
071e6bd
Compare
|
vec4 (16-byte weight and activation loads) measured on the 13-inch M1 at unicode-24, with both arms built from their own trees in one session: old = the #1544 base tree
Median 40.5%, range 39.5 to 41.3%. All six exit 0, My pre-registered forecast for this input was 8 to 20 percent (55%) and it missed: all three pairs landed near 40%. So the figure I post is the measured one, not the forecast: about 40 percent at unicode-24 on the 13-inch M1, against the 24.4 / 32.8 / 39.7 percent the 16-inch M1 Max measured on its three inputs. Context for why this number is worth more than the published ones: the same day's replications moved two other published figures (the rows image read 5.9% in #2017's body and 12.3% in a fresh run; the hazard build read 7.1% and 6.84%). Every number in this comment and the two beside it now comes from six fresh processes with all digests equal, and from arms built in the same session rather than lifted from an older packet. [Joshua agent] |
|
vec4 on the second GPU family: 23.1% at unicode-24 on an Intel Arc Pro B60, Intel's open-source Mesa driver, six fresh processes old,new,new,old,old,new. Arms built in the same session as the M1 run: old = the #1544 base tree
Median 23.1%, range 22.1 to 24.9%. All six exit 0, My forecast for this run was 15 to 30 percent (45%) and the median landed inside it. Two families, one session's arms, digests equal on both:
For scale, the 16-inch M1 Max measured 24.4 / 32.8 / 39.7 percent on its three inputs. The long-input numbers here remain single-run for the reason recorded on #2017: the harness has no 256-token case to run. [Joshua agent] |
|
Replication of the B60 vec4 number, fresh output root, forecast written before the run (80% that the warm median lands in 20 to 26%):
Median 23.0%, range 22.2 to 24.8% — against the first run's 23.1% (22.1–24.9). All six exit 0, The B60 vec4 figure now rests on two independent runs: 23.1% and 23.0% at unicode-24. The M1 vec4 run (40.5%) is being replicated on that host next, so both families will have two runs each. [Joshua agent] |
|
The B60 summaries do not match the displayed timing pairs. Using
Could you reconcile the summaries with the source receipts and say whether the timing rows or the aggregation changed? These calculations use only the posted rows; I have not rerun the GPU measurements. |
|
Replication of the M1 vec4 number, fresh output root, forecast written before the run:
Median 40.45%, range 39.5 to 40.9% — against the first run's 40.5% (39.5–41.3). All six exit 0, Both families now rest on two runs each:
vec4 is the strongest measured lever in this stack: it halves the load instruction count on a step the microbenchmark shows is weights-bandwidth-bound, and it changes no output on either driver. The remaining gap in the receipts is the long-input cases, which cannot run until the harness carries a 256-token case (recorded on #2017). [Joshua agent] |
|
Third run of the vec4 term, and the third run of the end-to-end stack figure, both on the 13-inch M1 at unicode-24: vec4 over the #1544 base: 40.5%, 40.45%, 40.71% (range 40.26 to 41.09 in this run). All six processes exit 0, Stack head over vec4 alone: 78.6%, 78.67%, 78.5% (range 78.06 to 78.9). Same gates, same digest. Both three-run series are tight, which is the opposite of what the Intel B60 showed on the rows image (9.5%, 9.5%, 2.5%). That contrast is worth stating plainly: on this chip the measurements resolve to well under a point, so the figures above can be quoted as point estimates; on the B60 the same class of effect has a per-pair spread of several points, so only ranges are honest there. The final shape of the stack's measured contribution at unicode-24 on the M1:
[Joshua agent] |
|
vec4 at document-256 on the 13-inch M1 (256 prompt tokens, context 512, arms built this session): 48.52% over the scalar q4 projection.
All six exit 0, My forecast was 15 to 35 percent and it missed high: vec4 pays MORE at 256 tokens (48.5%) than at 24 tokens (40.5%, three runs). The width change scales up with prompt length rather than down, which is the opposite of what the decode-vs-prefill shift made me expect. Combined with the batching-at-256 result posted on #2016 (19.29%, in band), the long-input ladder on this device is: base to vec4 48.5%, vec4 to batched 19.3%, and the stack head figures posted earlier (82.88% over vec4 alone at this input). [Joshua agent] |
|
Thanks @joshuaswarren. With the full diff in hand we've now checked the rebase. Against the series we reviewed, every file is the same apart from main's own changes, so #1544, #1990, #2015 and #2016 can land in that order. One small thing is still open, and it doesn't block: @cwschroeder's question on the B60 medians. From the posted pairs they come to 24.17% and 22.48%, not 23.1% and 23.0%. If the medians use more readings than the tables show, please say which, or correct the two figures. |
|
Correction: my earlier reply recalculated the displayed rows, but those rows do not match the retained timing files. The originally reported medians came from different readings. The retained run-2 pairs are: first run 17326→13505 ms (22.054%), 17547→13177 ms (24.905%), and 17116→13168 ms (23.066%), median 23.066%. Replication: 17306→13457 ms (22.241%), 17106→13168 ms (23.021%), and 17806→13386 ms (24.823%), median 23.021%. Each retained timing record says |
|
A small licence check on the Vulkan kernels here and in #2016, @joshuaswarren: their headers read "Joshua Warren and mlx-omarchy contributors; MIT". Could you confirm on this PR that those kernels are your own work, or that any other contributors' lines are covered by that notice? It won't hold up landing. |
|
[Joshua's agent] The published #1990 and #2015 timings use Qwen3.5-2B. The 14-inch M2 Max currently has Qwen3.8-2B staged; I have not verified a Qwen3.5 checkpoint there. For the additional M2 comparison, do you want the same Qwen3.5-2B checkpoint as the published runs, or a Qwen3.8-2B run? I will keep the models separate in the results. |
|
[Joshua's agent] I checked the Vulkan source history and the license included in this stack. The #1990 vec4 shader was added in a6777fc, and #2016's submission change in 530b5b7. Both commits record Joshua Warren as author. The earlier Vulkan projection was carried into TensorFold in 2a41e71; the other shader imports include 2b35b5d and 3cde396, also authored as Joshua Warren. The upstream source is under the mlx-omarchy MIT license, and the stack includes its full notice in LICENSES/mlx-omarchy-MIT.txt. The raw pre-import histories for the corresponding shader paths include Joshua Warren and the agent labels QmmLayouts and DecodeKvDirect; mlx-omarchy's mailmap maps the QmmLayouts email to Joshua. For example, 57c2819 records QmmLayouts on qmm.comp. DecodeKvDirect appears in local branch history; its commit is not available on GitHub. I do not want to claim that every pre-import line is Joshua's solely from the TensorFold commit authors. |
|
[Joshua's agent] One additional source mapping: TensorFold's gated_delta_prefill_ash.comp records that it derives from mlx-omarchy's gated_delta_prefill.comp. The latter's pre-import history I checked records Joshua Warren as the author; for example, 7d3f69f and fa103c8. The TensorFold version was carried in 3cde396. |
|
[Joshua's agent] I checked the RoPE mapping further. A local branch commit, 495482535e12e3a5b1d2f82bc10b5840a8329c65, records DecodeKvDirect as the author of a BF16 pair-store implementation with matching fragments in TensorFold's fast_rope.comp. That branch commit is not available through the GitHub API; my earlier link to it was invalid. The upstream mainline replay is 8d9aaa2; the TensorFold shader was carried in 3cde396. DecodeKvDirect is an author label; I cannot infer a human identity from it. The stack includes the mlx-omarchy MIT notice. |
|
Thanks @joshuaswarren for tracing that, and for saying what you can't infer from an author label. It goes with the stack's shader-provenance question, which needs a decision on our side, and we'll answer both here. Until then, please don't rework |
|
[Joshua's agent] Here is the file-level evidence for the licence question, so you can decide the stack's shader-provenance question. The notice in this stack is upstream's own licence text, unchanged. Where the 21 shaders in this diff come from:
Upstream has other contributors, and the notice covers their lines only where it says it does. The upstream shaders directory has 18 commits whose raw author strings are not Joshua Warren. Ten labels account for 13 of them: What the copies contain: only two of those 18 upstream commits touch a file in this diff. Neither contributes a single line to the copy here.
The four files whose header carries the full MIT notice line ( What I am not claiming: The ask: the two questions on this thread are answered above in file-level terms. Those are which lines the notice covers, and where the third-party history sits. The one still open is your side's decision on the stack's shader-provenance question, which you said you would answer here. I am not reworking One open question of mine, from earlier in this thread: for the additional M2 comparison, do you want the same Qwen3.5-2B checkpoint as the published runs, or a Qwen3.8-2B run? The 14-inch M2 Max has Qwen3.8-2B staged, and I have not verified a Qwen3.5 checkpoint there. I will keep the models separate in the results either way. [Joshua's agent] |
|
Thanks @joshuaswarren. That's the file-level account we needed: which lines the notice covers, and that neither of the two third-party commits on those files has a line in this stack. It's on record for the decision. On the M2 comparison: please use the same Qwen3.5-2B checkpoint as the published runs, so the rows compare one to one. A Qwen3.8-2B run is welcome as well, kept separate as you say. We'll answer the provenance question here once #1544's held packet passes, since the stack lands behind it. |
|
Noted on the M2 comparison: we will use the same Qwen3.5-2B checkpoint as the published runs, and keep a Qwen3.8-2B run separate. That run needs the 14-inch M2 Max, which is unreachable right now (ssh to its Linux side returns "No route to host", the macOS side times out, checked 05:38Z). It is left until the machine is back, and no M2 number is claimed here in the meantime. |
|
Thanks @joshuaswarren for the file-level account; it's what we needed to decide. Our rule is that code from other projects stays out even with a notice, while a contributor's own work is welcome. Your account supports reading these shaders as your own work, so we'll take them that way, with three small things:
On the M2 comparison: the same Qwen3.5-2B checkpoint as before, as agreed. |
…elog line The receipts folder holds the identity probe, the pre-registration, a summary generated from the raw run files, and the SHA-256 of each raw file. On Qwen3.5-2B 4-bit, a run took 40% to 52% less time on three inputs, with the same logits digest in all six processes of each case. The contract runner's output on the GPUs follows in a later commit.
tf-vulkan-kernel-contract on the Apple M1 (G13G B1) under Honeykrisp: 2 kernels pass gates 1 to 4, 0 fail, 0 known findings, 8 of 8 planted controls caught, 0 limit failures. qmm_q4_vec4 and qmm_bf16 show the same worst shapes value (0.990 of the bound).
…m kernels Header becomes Copyright 2026 Joshua Warren; MIT on the omarchy-mlx-derived shader files; the licence file and the credits lines stay. Kernel comments now describe our own push-constant layout and macro names instead of the other backend's names. ASH_QWEN35, ash_gate and gated_delta_prefill_ash.comp become ORACLE_QWEN35, oracle_gate and gated_delta_prefill_oracle.comp. SPIR-V check: 25 server shader images compiled at the base and the modified tree with the shipped glslang flags. 24 of 25 are byte-identical. gdn_split_bf16 differs only by the embedded oracle_gate identifier; with debug names stripped both objects hash the same (01d232c2cdc2).
…an_check adds no problem
Why: the validated CPU store policy keeps every activation store inside layers 0-3 in float32 while weights, scales, biases, gains, A_log, dt_bias, the KV cache, the conv chunk state and the embedding stay bf16 (row 28 |V-R| 3.3 all-bf16 vs 0.05-0.6 with f32 in layers 0-3). What: one compile define per activation role, parameters stay bf16 - qmm.comp ACT_F32 (x/out f32, scales/biases uint16_t), fast_norm.comp ACT_IN_F32/ACT_OUT_F32/ACT_OUT_BF16 (gain bf16), binary_vec.comp ACT_F32 and ADD_WIDE (f32 stream + bf16 residual -> bf16), swiglu.comp ACT_F32, fast_rope.comp ACT_OUT_BF16 on the float fall-through (f32 rope with a bf16 cache store), sdpa_decode_native.comp F32_Q/F32_OUT inside the BF16_IO composition arm (cache stays bf16), attention_gate.comp ACT_F32 (element addressing), gdn_split.comp ACT_F32 (dt stays uint16_t, the Metal-matching intermediate roundings go transparent), gated_delta_prefill_ash.comp ACT_F32 (soa/beta/y f32, state f32 unchanged), fast_norm_gated.comp ACT_F32 (gain bf16), conv_dw1d.comp and copy_general.comp float/OUT_TYPE fall-throughs. Build tables gain the variants; the new gdn_cast_bf16_f32, gdn_cast_f32_bf16 and gdn_copy_u32 images serve the GDN conv-state copies. Proof: glslang recompile of every variant table entry; the 26 existing bf16 images are sha256-identical to base 4875da9; the 14 new variants compile.
Why: layers 0-3 keep every activation store in float32; each operator needs a pipeline that stores its f32 output (and the two mixed residual transitions) instead of rounding to bf16. What: ADD_WIDE_F32 (bf16 + f32 -> f32) in binary_vec.comp with its add_bf16_f32 variant; Ops gains qmm_f32 and copy_f32_bf16 with copyPipeline(kind); NormMlp gains norm_f32/norm_f32_in/norm_bf16_in, add_f32/add_wide_bf16/add_bf16_f32 and swiglu_f32 behind the pure normVariant/addVariant/swigluVariant kind maps; Attention picks rope_f32/rope_f32_kcache/sdpa_rows_f32q/gate_f32 by view kind with element addressing in RopePlan/GatePlan; Gdn picks conv/split/prefill/ gated-norm f32 images by splitVariant/gatedNormVariant with copyAny for the u32 state copies. Probe and contract shader literals name the new images. Proof: all five stage-gpu-build steps compile; the host module tests cover the new kind maps and element counts.
Why: the validated store policy keeps every activation store inside layers 0-3 in float32; layers 4-23 keep bf16, so bf16 outputs and their roundings are untouched there. What: fwd.actKind names the per-layer store kind; Core's layer bodies allocate every scratch store in the input view's kind (g stays f32 and beta bf16 in the GDN scan at both ends); runLayers re-kinds the four arena windows (h0/h1/r/x, now f32-backed) per layer: the input add and norm read the previous layer's kind and write the current kind, the post add writes the current kind over the previous sum, and the plain layer-0 norm takes the bf16 embedding into f32; finish stays bf16. The bundle serves the new images; extractQueries rides copyPipeline(kind). Proof: qwen35-model-host-test and vulkan-host-test pass; the arena test sizes the worst f32 layer at the cap inside the 8 MiB arena.
Why: the acceptance wants host tests that fail before and pass after for the store policy, and host-verifiable want formulas for the f32 shader pipelines. What: stage2_fixture_f32 pins add_f32/wide_bf16/wide_f32, swiglu_f32 and rms f32/f32_in formulas (no store rounding inside the wide layers, one RNE bf16 store in each transition) off the shared case tables; the engine test sweeps actKind over all 24 layers; the arena test names the wide and first narrow kinds; zig fmt on gdn.zig. Proof: vulkan-host-test and qwen35-model-host-test pass; lean_check is 434 problems at head vs 433 at base (the +1 is the new fixture file's own count, zero in edited files beyond fmt).
Why: lean_check counts consecutive comment lines as a problem; the five new two-line comments merged into single lines (431 problems now, 433 at base). Proof: lean_check 431 vs 433 base; vulkan-host-test and qwen35-model-host-test pass after the edit.
Why: the f32 work introduced 7 new lean problems (multi-line comments and three comments past 120 columns), which the CI gate compares against the merge base. What: fold or shorten the six comments in backend.zig, engine.zig, attention.zig, ops.zig, ops_q4.zig and stage2_fixture_f32.zig. No behavior change. Proof: python3 tools/zig/lean_check.py gives 424 problems on the head and 424 on the base 4875da9, the same set; zig fmt --check is clean; vulkan-host-test and qwen35-model-host-test pass.
The wide path was never run on a Vulkan device before the 13-inch run and failed at run1 with Invalid. Running it on lavapipe (scratch build) showed contract bugs and one shader bug: - convPrefill demanded f32 taps and an f32 conv state while the checkpoint taps and the state regions were bf16; the taps stay bf16 and the conv windows of the linear slots in layers 0-2 are now f32 (state.convKind) - the a, b and beta stores take the layer kind (forward.zig allocated beta bf16 while the split and scan shaders store f32 beta) - conv_dw1d_f32 is built with ACT_F32 so only the activations widen - norm_f32 stored bf16 into an f32 view (missing ACT_OUT_F32), which broke every post-attention norm in layers 0-3 Checked on lavapipe against the f64 reference on a 32-row prompt: per-layer relative error about 1e-3 in layers 0-3, final logits max abs error 0.12. The bf16 conv SPIR-V is byte identical to the base.
The choices field showed every backend name on every build; the served set was already computed for the honoured values. Listing what the binary does not serve invited --backend vulkan attempts on builds that refuse it.
The dense-lane decomposition follows the frozen lane_qmm design; the compute shaders are new work.
…e gated refusal Choices equal to the served set made an unserved pick fail the choice check first, whose message main's flag-deps pin does not expect; the pin expects the gated refusal for an unserved value. Take main's choices form so the choice passes and the native gate speaks: main's zig job passes this test at 6e9af93 and fde40f6, and the failure showed only on this stack's merge (run 42815854414, flag_deps_test.zig:45).
…y-mlx shaders The Vulkan build's own choices gate already carried vulkan, but the flag's static choice list did not, so --backend vulkan died as an invalid choice before the served check could refuse it on other builds. The cli test now covers both builds: it parses on the Vulkan build and gets the gated refusal elsewhere. The credits line said all of the engine's compute shaders were new work; fifteen of them and omarchy_trig.h come from omarchy-mlx unchanged, and the list now says so. Proved: zig build test-flag-deps-host failed first on the unfixed choice list, with the invalid-choice refusal instead of the gated one, then passed 22 of 23 with one skip without Vulkan and succeeded with -Dvulkan=true.
The first figures (40% to 52%, against ashhart#1544's head) came from builds before the float32 fix, on an older kernel and Mesa build. On the fixed head, on a 13-inch M1 with Linux 7.3.0-2-20261110.1-lab-ARCH, the three cases took 34.8%, 41.0% and 43.0% less time than the same head with the scalar kernel selected (three pairs for the first two, two pairs for the 1,024-token document, one logits digest per case). The gain depends on the environment; these runs do not separate the cause. The identity probe and the kernel contract output are from the earlier builds and were not re-run. Signed-off-by: joshuaswarren <816217+joshuaswarren@users.noreply.github.com>
…es for the oracle Each shader taken from omarchy-mlx now opens with its source path and commit, what changed, and the copyright line; THIRD_PARTY_NOTICES.md lists every such file. The 34 placeholder comment lines in 17 files become one-line comments of what the code does, or go. The push-constant block, the gate helper and the labels that named another project's layout now describe our own layout and say oracle. No code changes. Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
59145cf to
613a5bf
Compare
…le_f32 Adds the P-core pinned ashhart#1990 against ashhart#2015 timing on the 13-inch M1 (28.6%, 54.4%, 66.8%, equal logits digests), the output of the ten contract and stage runners at 2a5a06c including qmm_q4_tile_f32, and replaces the earlier-build kernel contract and identity probe files. Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
…le_f32 Adds the P-core pinned ashhart#1990 against ashhart#2015 timing on the 13-inch M1 (28.6%, 54.4%, 66.8%, equal logits digests), the output of the ten contract and stage runners at 2a5a06c including qmm_q4_tile_f32, and replaces the earlier-build kernel contract and identity probe files. Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
…le_f32 Adds the P-core pinned ashhart#1990 against ashhart#2015 timing on the 13-inch M1 (28.6%, 54.4%, 66.8%, equal logits digests), the output of the ten contract and stage runners at 2a5a06c including qmm_q4_tile_f32, and replaces the earlier-build kernel contract and identity probe files. Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
…le_f32 Adds the P-core pinned ashhart#1990 against ashhart#2015 timing on the 13-inch M1 (28.6%, 54.4%, 66.8%, equal logits digests), the output of the ten contract and stage runners at 2a5a06c including qmm_q4_tile_f32, and replaces the earlier-build kernel contract and identity probe files. Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
…le_f32 Adds the P-core pinned ashhart#1990 against ashhart#2015 timing on the 13-inch M1 (28.6%, 54.4%, 66.8%, equal logits digests), the output of the ten contract and stage runners at 2a5a06c including qmm_q4_tile_f32, and replaces the earlier-build kernel contract and identity probe files. Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
…le_f32 Adds the P-core pinned ashhart#1990 against ashhart#2015 timing on the 13-inch M1 (28.6%, 54.4%, 66.8%, equal logits digests), the output of the ten contract and stage runners at 2a5a06c including qmm_q4_tile_f32, and replaces the earlier-build kernel contract and identity probe files. Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
…le_f32 Adds the P-core pinned ashhart#1990 against ashhart#2015 timing on the 13-inch M1 (28.6%, 54.4%, 66.8%, equal logits digests), the output of the ten contract and stage runners at 2a5a06c including qmm_q4_tile_f32, and replaces the earlier-build kernel contract and identity probe files. Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
…le_f32 Adds the P-core pinned ashhart#1990 against ashhart#2015 timing on the 13-inch M1 (28.6%, 54.4%, 66.8%, equal logits digests), the output of the ten contract and stage runners at 2a5a06c including qmm_q4_tile_f32, and replaces the earlier-build kernel contract and identity probe files. Signed-off-by: Joshua Warren <816217+joshuaswarren@users.noreply.github.com>
Fix the stale shader filename and separate unchanged from adapted sources. Record source commit, full ELF hashes, and sizes from retained receipts. Verified the JSON summary parses; historical build commands are absent.
|
Updated on |
Stacked on #1544. Above #1544's branch (
a989a7b87, the 43 commits this branch also carries, plus one fix commit040a4fb0) this PR adds, in order: the vec4 kernel and its selection (e5fa7c5f), its kernel contract registration (9297100c), receipts and the changelog line (6cbbc58d,814522b6,27df9a06), the oracle rename and header fixes (64e13512,6116cb71), the float32 activation port for layers 0 to 3 (a35df20a,efa50978,08315a1a,a33db616,4983a411,411cbc1e,774c1789), the--backendflag listing only the choices this build serves (35175560), the credits page (1d1ae16d), and the merge of #1544's fix commit (59145cf8).mainis merged in at46571f11; the one conflict was aCHANGELOG.mdhunk and both entries are kept.On a 13-inch Apple M1 with Honeykrisp, a Qwen3.5-2B 4-bit run took 34.8% to 43.0% less time than the same build with the scalar kernel selected, on three inputs. The logits are byte-identical. These figures are from the float32-fixed head on Linux 7.3.0-2-20261110.1-lab-ARCH and Honeykrisp from the Omarchy Mesa build (library sha256
d75f4cbc). The old arm is this head built with the vec4 selection forced to the scalar kernel (qmmVariant = .scalarinops_q4.zig, not committed): the float32 fix sits above the vec4 commits, so an older #1544 build would not carry it. The first version of this description gave 40% to 52% against #1544's head, measured on earlier builds with Linux 7.1.12-2-20261013.18-sep-ARCH and Mesa git-9323171c12. The gain depends on the environment, and these runs do not separate the cause.e946185d64ae3188aaae7d1182c353787e89Run 2 is the prefill plus 63 decode steps plus the per-step logits readback. It is not decode-only, so do not read these as decode ms per token. All processes of a case wrote the same logits digest, in both arms. The 1,024-token case has two pairs, not three: the scalar arm takes about 250 s a process. I did not re-run the identity probe or the kernel contract on the fixed head; their output is from the earlier builds.
Second chip: 16-inch M1 Max
The same three-pair protocol on an Apple M1 Max (G13C), same old and new builds, order old, new, new, old, old, new, run-2 gain per pair.
On this chip unicode-24 is within the spread, so the 3% rule is not met there; the two documents meet it. Each case has one logits digest across all pairs and both arms, equal to the 13-inch M1 digest. The runs do not separate the cause of the difference between the chips, so I am not naming one.
The branch at
59145cf8The fix commits change comments, headers, names in tests and
THIRD_PARTY_NOTICES.mdonly. I compared the shader and source lines of46571f11(the measured head) and59145cf8with comments stripped; the only differences are test names and the renamed test helpergateRefOracle. The figures above therefore apply. The probe ELF itself differs (sha256 prefix930d857aagainst2a1d481c) because a safe-mode build records source lines.zig fmt --check, the Vulkan host tests build and run (34 s), andlean_checkadds no problem in the changed files.What changes
qmm_bf16reads each weight word and each activation value in its own load.qmm_q4_vec4reads 16 bytes of each per load. Each output keeps one fp32 accumulator and the same order of sums, so only the load width changes.ops_q4picks the new image when the x and w windows are 16-byte aligned, and keepsqmm_bf16otherwise. The image is registered in the kernel contract in its own commit.Setup
d75f4cbc), Linux 7.3.0-2-20261110.1-lab-ARCH.46571f111b2f0e92963561702b7eaab41868f823with localqmmVariant = .scalar; ELF SHA-2564d1c6ba6c702220c44303ba674f4c3247e671b3e585d3c83bcd6960d65c2cb31(6,087,144 bytes). Vec4 build: source commit46571f111b2f0e92963561702b7eaab41868f823; ELF SHA-2561e1b7d8cdddf57645c2090207eef24812e4cbc6e6b90a11dc9d43492ce90ac5a(6,087,520 bytes). Both are ReleaseSafe aarch64 builds oftf-qwen35-model; the retained run files do not include the exact shell build command.Identity
docs/receipts/vulkan-qmm-vec4-m1/identity-probe.txt). This probe ran on the earlier builds and was not re-run on the fixed head. The same probe passes 17 of 17 on llvmpipe. That run is not GPU evidence.qmm_bf16. The new image is byte-identical toqmm_bf16on every probe case, so I added no controls of its own.Pre-registration
docs/receipts/vulkan-qmm-vec4-m1/PREREG.mdwas written before any GPU run. The rule was: a median gain of at least 3% and every pair positive. All three cases meet it.I forecast a 10% to 35% gain on the 24-token prompt, and below 40% for the two documents. The 24-token prompt gave 34.8%, inside the forecast. The documents gave 41.0% and 43.0%, above it. On the earlier builds the three cases gave 40.1%, 48.5% and 51.5%.
Not covered
Checks
tf-vulkan-kernel-contracton the 13-inch M1 (Honeykrisp):qmm_q4_vec4passes all five gates (limits, nonfinite, repeat, shapes, sums) with the same worst shapes value asqmm_bf16(0.990 of the bound). The 8 planted controls are all caught. Full output:docs/receipts/vulkan-qmm-vec4-m1/kernel-contract-m1-13-inch.txt.On llvmpipe the runner aborted in
prepareafterqmm_bf16's nonfinite and repeat gates, in anerrdeferthat frees a value already set toundefined. That code is onmainand is reached when a device buffer allocation fails, and llvmpipe caps a buffer at 128 MiB. #1992 fixes it; this PR does not touch it.python3 tools/zig/lean_check.py: 466 problems before and after, none new.zig/build/vulkan.zig(676 to 683 lines) andzig/tests/vulkan/qwen35_model_probe.zig(646 to 647) were already over 600 lines.atc check 1420shows Native Vulkan backend for Linux GPUs (Apple Silicon first) #1420 complete and unclaimed with only Native Vulkan backend for Qwen3.5-2B (opt-in): engine and kernel-contract registration (stacked on #1526) #1544 attached.atc overlap --jsonreturnedcomplete: false(pull files, time limit and working tree unavailable), so conflicts are unknown.main: every Vulkan host suite passes (vulkan, stage 2 to 5, contract-pointwise, model), and the stage 1 probe and the contract runner build.raw-files-sha256.txtlists the hash of each one.