[scchumbletech]

What this changes

#2851 takes a non-reg prompt chunk's 16-row blocks in one dispatch on Ultras only (longPrompt ends with " Ultra"). On other tensor-unit Macs — an M5 Max here — the same chunks fall back to one 16-row window a dispatch, and each dispatch re-reads the projection's whole weight slab: a Qwen3.8-27B MLX 8-bit 8k prompt streams ~517 slab reads and sits at the memory bus. This opens the same row-block path on tensor units at large (k.device.tensorUnits()), Ultras keeping their long-key behavior exactly.

• Issue: Refs Metal M5: prompt lane projections at every width take a chunk's 16-row blocks in one dispatch #2851 (extends it to non-Ultra tensor-unit chips; the M5 Max receipt it lacked).
• Stacked on: main (93d77e5).
• atc overlap --json: not run (atc is not installed here); by hand, no open PR touches these gate lines.

Checks on this head

• Head commit: f2eb879
• Lean checker on the base and on this head: 342 and 342, 0 new.
• Tests: zig build test — 844/848 steps, 3215/3246 tests passed, 29 skipped, 2 failed; both failures are pre-existing on main on this machine (PDF message normalization in src/server/messages.zig:371, unrelated to quant). The updated dispatch test passes on tensor units and keeps the 128-dispatch expectation on non-tensor-unit devices.
• CI: pending at submission.
• Not run: CUDA/HIP/XPU — the change is inside a Metal-only family gate (k.device.tensorUnits()), unreachable from those backends.

Receipt

• Main against branch, Apple M5 Max 40-core GPU, 128 GB, macOS 26.x, Zig 0.17.0, Xcode Metal toolchain 27B47 (Metal 32023.922), checkpoint TensorFold/Qwen3.8-27B-MLX-8bit, serve string --drafter z-lab/Qwen3.8-27B-DFlash2 --parallel 4 --context 262144 --thinking --reasoning-effort xhigh --thinking-budget 131072, 8k salted unique-prefix prompt (fresh KV, no prefix retention), streaming TTFT-based rate, 3-run median, fresh server per run:

  	cold 8k prefill
  main 09a9eb3	336 tok/s
  this change	531 tok/s (+58%)

  Replies byte-identical main against branch on the measured prompts (5/5 greedy prompts, drafted and non-drafted paths). GPU pegged at 1620 MHz during passes.

• 4-bit unchanged: 842–858 tok/s stock-level across runs (the reg path is untouched).

• Decode: unchanged — prompt gating only; the decode/verify path keeps its 16-row keys.

• New switch: none. Non-reg prompt chunks on tensor-unit Macs take the row-block path automatically; every other caller keeps main's schedule.

• Paths reached: Metal qwen27 non-reg widths (8/6/5/3/2-bit, oQ mixes) on any tensor-unit Mac. Could not run: Ultra (no hardware) — behavior there is main's.

• Exactness: drafted = plain, byte-for-byte, on all measured prompts.

• Prompt speed: the numbers above. Decode speed: n/a (unchanged code path).

• Licence of new weights or drafters: n/a.

Checklist

• The title and description match the head commit, including the receipt and the stack.
• New tests fail before the change, pass after it, and skip cleanly without their dependency.
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.
• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo (byte-identical on the measured prompts; the dispatch change does not alter any arithmetic).
• No precision traded for speed. If the bits change, the description says which sums change and why. (No bits change; per-group reduction order is unchanged.)
• Decode and prompt processing are no slower than main at one stream and the widest admitted; with --drafter, no slower than without it, greedy and sampled

[ashhart]

Thanks @scchumbletech. The existing row-block dispatch is reused, with decode and the reg path preserved; all 35 host checks pass. Please shorten the two-line explanatory comment to one purpose line: it adds one lean finding on current main. The M5 Max receipt is useful. We will run the device check on our side, retaining the Ultra control separately from the newly enabled hardware.

• Synthena (Ash's Agent)

[scchumbletech]

Done — the gate comment is one purpose line, lean back to 342 (0e9d66a), follow-up commit (48926fa) removes an erroneous change.

[ashhart]

Thanks @scchumbletech. The gate comment is now one purpose line, the accidental iterator change is gone, and all 35 host checks pass with zero new lean findings. We will run the device check on our side on M5 Ultra as the unchanged-path control, keeping your M5 Max evidence separate for the newly enabled path.

• Synthena (Ash's Agent)