[gilby]

What this changes

On Metal, DeepSeek-V4.1's FP8 projections ran every decode row through the row kernel, so a launch of 6 to 16 rows (a drafted verify window, or several drafted streams) paid per-row scalar work on every projection. With this change launches of 6 to 16 rows run the FP8 projections on a simdgroup matrix tile (fp8_tile.metal), chosen by row count only; 1 to 5 rows keep the row kernel, where it is faster. A row's result is bit-identical to the row kernel's at every shape and row count, so rows at once equal one at a time and tokens are unchanged.

• Issue: Refs DeepSeek-V4.1 on Metal: decode and prompt speed on M3 Ultra and M5 Ultra, token-exact (gate and order for the drafts) #2984 (the speed work on this family; the stack itself is DeepSeek-V4.1 on Metal: a stack on main's deepseek_v41 family for the release checkpoint (one M3 Ultra) #1676)
• Stacked on: dsv41 metal: the lane target, the prompt cache and the native server over Metal #1720 (c97d661c4); this adds two commits: a synthetic timing step at the release's widths (time-dsv41-metal-synth, no checkpoint needed), and the tile with its contract records. Two commits since review: f2241b84a (the kernel contract's entry for the tile) and ff8365687 (the CHANGELOG line quotes the served result on both chips).
• atc overlap --json: not run yet.

Why it is exact: each 32-wide group root is computed as the prompt tile does it, four 8-wide steps in k order from +0; each simdgroup takes an aligned power-of-two run of groups, so its root is a subtree of the row kernel's pair tree, and the slice roots meet in that same tree. Zero padding adds x + 0 exactly, since no partial sum is ever -0. An output row depends only on its own input row. The form (8 or 16 columns, the number of simdgroups) cannot change bits, by the same argument.

What it does not do: plain rounds of 1 to 4 streams are unchanged. The per-row split measured for this change shows the experts' multi-row kernels, not FP8, dominate at 2 to 5 rows; that is a separate change.

Checks on this head

• Head commit: ff8365687
• Lean checker, python3 tools/zig/lean_check.py, on the base and on this head: 345 and 345, 0 new (tools/pr_check.py 0 errors, 0 warnings).
• Tests run, with the command and pass, fail and skip counts: the new commits touch documentation and the CHANGELOG only; at 2402e6eb4 zig build test-dsv41-metal-all passed on the M5 Max (layer 31/31, served 8/8) and on the M3 Ultra (9/9 steps, 39/39 tests).
• zig build time-dsv41-metal-synth (six synthetic layers at release widths, a 40-layer round weighted by layer kind, ms): 1 row 30.3 before and after, 4 rows 45.0 and 45.0, 6 rows 68.2 to 64.9, 8 rows 78.0 to 68.0, 16 rows 148.2 to 110.1, four slots of one row 45.6 and 45.6.
• CI: not run: the branch is still on dsv41 metal: the lane target, the prompt cache and the native server over Metal #1720's pre-landing head and conflicts with main (CHANGELOG.md), and per the order in DeepSeek-V4.1 on Metal: decode and prompt speed on M3 Ultra and M5 Ultra, token-exact (gate and order for the drafts) #2984 it is rebased onto the PR before it at landing.
• Not run, and why: the M3 Ultra dsv41_fp8_tile equality line and time-dsv41-metal-synth there (with them, whether 6 rows stays the crossover); run list in the comment.

Receipt

• Main against branch: two drafted streams faster on both chips, every other stream count level, every token unchanged. M5 Ultra: 2 streams 1.037 to 1.055, 1 stream 1.001 to 1.009, 3 and 4 streams 0.998 to 1.000. M3 Ultra (DeepSeek-V4.1 on Metal: decode and prompt speed on M3 Ultra and M5 Ultra, token-exact (gate and order for the drafts) #2984 rule): 2 streams 1.036 to 1.054 (4 cells faster), the rest 0.999 to 1.006 within noise, none slower. Receipts: https://gist.github.com/gilby/df7bed9a89d9c854cfe78479b4c8cb6f (M5 Ultra), https://gist.github.com/gilby/c020133ac9809d5f01d5b34dd60fc308 (M3 Ultra), measured at 2402e6eb4; the two new commits are documentation.
• Boxes it reaches: Apple Silicon Macs serving DeepSeek-V4.1 natively, in launches of 6 to 16 rows.
• New switch? Default, accepted values, profile row: none.
• Environment: measured at 2402e6eb4 on dsv41 metal: the lane target, the prompt cache and the native server over Metal #1720's c97d661c4; deepseek-ai/DeepSeek-V4.1-Flash-1017 release weights at revision 1bae0e1088993b9b82de08f59de5be5a34e42170; M5 Ultra 512 GB and M3 Ultra 512 GB; Zig 0.17.0.
• Paths and platforms the change reaches, which I ran and which I could not: Metal only. Ran the synthetic-weight device suites and the contract on an M5 Max. Not run: M3, M5 Ultra, the release weights.
• Exactness: token_sha256 equal to the stack tip in every cell on both chips, 0 failed, 0 --alone / --serial mismatches; the contract holds the tile bit-equal to the row kernel one row a launch.
• Main against this branch, same machine and checkpoint, posted before asking for a run on our machines: posted in the receipt comments on this PR (links above).
• Decode speed before and after: bench_openai 1.001 to 1.036 (M5 Ultra), and on the M3 Ultra fibonacci-raw sampled 1.031, the rest level.
• Prompt speed before and after: n/a for the claim: launches of a prompt chunk do not take this tile; the cold-prompt cells in the M5 Ultra receipt are single runs with base first and are not claimed.
• Licence of any new weights or drafter: n/a.

Kernel contract

• Kernels registered with the contract, and the kernel each one replaces: the FP8 tile in six registrations (f32, bf16, a forced 16-column form, one simdgroup, the release wkv fixture, and one declared as replacing the row kernel run one row a launch, bit-equal on all 33 shapes at gate 3). An equality probe of 4,500 launches covers every model FP8 shape in f32 and bf16, rows 1 to 16, every form, the 16/17 seam and a three-launch batch, with NaN and Inf planted in activations, codes and scales: every launch byte-equal to the row kernel one row at a time.
• A planted control for each new kernel, and each one fails as planted: four, each red through data: non-finite inputs flushed (gate 1), a write past the output (gate 2), roots added in a chain (gate 3), fp16 group roots (gate 4). At TF_CTL 0 the source is what ships.
• Each backend's SUMMARY line and exit status, M5 Max: base SUMMARY 89 kernels: 89 pass gates 1-4, 0 fail, 0 known findings; 186 controls: 186 red as planted, 0 missed; 0 limit problems; this head SUMMARY 95 kernels: 95 pass gates 1-4, 0 fail, 0 known findings; 190 controls: 190 red as planted, 0 missed; 0 limit problems. M5 Ultra at 2402e6eb4, exit 0: --only dsv41 SUMMARY 95 kernels: 95 pass gates 1-4, 0 fail, 0 known findings; 190 controls: 190 red as planted, 0 missed; 0 limit problems; full run (with the one-line fix from Kernel contract on tensor-unit Macs: the full run aborts with ControlEditMissing in the llama gemm_untransposed control since #2251 #3067) 756 pass, the same 57 known findings as base. M3 Ultra at 2402e6eb4, exit 0: --only dsv41 95 of 95 pass, 190 of 190 controls red; full run 545 pass, the same three known nonfinite findings as base. The new commits are documentation.
• Chips I could not run: M1, M2, M4. Still owed on the M3 Ultra: the dsv41_fp8_tile equality probe line from its contract run.

Checklist

• The title and description match the head commit, including the receipt and the stack.
• New tests fail before the change, pass after it, and skip cleanly without their dependency.
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.
• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo (synthetic weights, and on the release weights in the receipts).
• No precision traded for speed.
• Decode and prompt processing are no slower than main at one stream and the widest admitted; with --drafter, no slower than without it, greedy and sampled

[gilby]

First real-weights receipt, with the contract on the M5 Ultra. One M5 Ultra 512 GB, deepseek-ai/DeepSeek-V4.1-Flash-1017 release weights, ReleaseFast, --parallel 4 --context 1048576 --prompt-cache-gib 0, fresh server per arm, tools/bench_concurrent.py --levels 1,2,3,4 --tokens 128 --reps 3 --temperatures 0,1 --alone --serial then tools/bench_openai.py --tokens 256 --reps 3. Every request's token_sha256 equals the stack tip's in every cell, and concurrent equals alone equals serial. Thermal pressure was logged nominal at both ends of every cell, and control cells before each block read 46.6 to 46.9 tok/s. Arms in the order stack tip (c97d661c4), this, #2983, #2983, this, stack tip; the two stack-tip runs differ by 0.1 tok/s or less.

Contract on the M5 Ultra at 2402e6eb4 (tf-kernel-contract --only dsv41, with a fresh rotary receipt): SUMMARY 95 kernels: 95 pass gates 1-4, 0 fail, 0 known findings; 190 controls: 190 red as planted, 0 missed; 0 limit problems. test-dsv41-metal-all exits 0 there. The run without --only stops with ControlEditMissing before any SUMMARY, on this head and on the stack tip alike.

Greedy, aggregate tok/s, this over the stack tip:

streams	code	chat
1	61.6 / 61.2 (1.007)	56.6 / 56.5 (1.002)
2	85.9 / 81.4 (1.055)	80.2 / 76.9 (1.043)
3	96.1 / 96.2 (0.999)	93.0 / 93.1 (0.999)
4	113.8 / 114.0 (0.998)	109.0 / 109.2 (0.998)

bench_openai: code 71.8 against 71.2, chat 56.8 against 56.7. So the tile pays where launches reach 6 or more rows, which today is two drafted streams (4 to 5%); one stream's windows mostly stay under 6 rows, and rounds of three and four streams are plain, so those are level, as the description expected. Cold 2K, 8K, 32K and 64K cells and the M3 Ultra contract and receipt are running; I'll post them here. The issue line in the body now points at #2984.

[ashhart]

Thanks @gilby. The exactness argument holds up in the code. In fp8_tile.metal:86-151 each simdgroup folds an aligned power-of-two run of groups with a binary counter, so its root is a subtree of the row kernel's zero-padded tree, and the cross-simdgroup sum at :160-172 is that tree's top. The tile is chosen by row count alone, never by the data (fp8_tile.zig:74-76, ctx.zig:72), and each output row reads only its own input row, so with the bit-equality above a row's bits can't depend on what else is in the launch. The M5 Ultra receipt matches the description: two drafted streams run 4-5% faster, everything else is level, and every token_sha256 is equal.

Asks:

1. The M3 Ultra contract (--only dsv41) SUMMARY and the 4,500-launch equality probe at this head. Exactness rests on each chip's fp32 simdgroup multiply-accumulate, so the M3 has to pass before its tokens are trusted. Please also post the M3 receipt (stack tip, this, this, stack tip; every token_sha256 equal) and a time-dsv41-metal-synth run there. If the M3's crossover isn't at 6 rows, please pick a min_rows (fp8_tile.zig:13) that is no slower on either chip.
2. The CHANGELOG.md line quotes the synthetic M5 Max layer times (26% and 13% less GPU time). Please replace them with the served results from the release weights on both chips. On the M5 Ultra, for example, two drafted streams run 4-5% faster and the rest is level.
3. docs/kernel-contract.md doesn't mention the tile, while dsv41 metal: a multi-stream round's per-slot stages run as one dispatch each (stacked on #1720) #2954 and dsv41 metal: decode experts decode an expert's weights once for all of its rows (stacked on #1720, #2973) #2983 each add a short entry for their kernels. Please add one that says what the tile is held to: bit-equal to fp8_rows run one row a launch on all 33 shapes at gate 3, the equality probe across forms, row counts and NaN/Inf plants, and the four controls.
4. Order and rebase. This makes rows cheaper, so it should land before DeepSeek-V4.1 on Metal: opt-in draft cost table timed at start, default pricing unchanged (stacked on #1720) #2952, so that DeepSeek-V4.1 on Metal: opt-in draft cost table timed at start, default pricing unchanged (stacked on #1720) #2952's cost table is measured on these kernels. With dsv41 metal: Engram rows are warmed ahead and joined at their layer's command buffer (stacked on #1720) #2953 closed, the order is dsv41 metal: TF_DSV41_ROUND_PROF times each decode round and the drafter's passes (stacked on #1720) #2951, dsv41 metal: prompts in pieces between the other streams' rounds (stacked on #1720) #2950, dsv41 metal: a multi-stream round's per-slot stages run as one dispatch each (stacked on #1720) #2954, this, dsv41 metal: decode experts decode an expert's weights once for all of its rows (stacked on #1720, #2973) #2983, then DeepSeek-V4.1 on Metal: opt-in draft cost table timed at start, default pricing unchanged (stacked on #1720) #2952. When you rebase onto dsv41 metal: TF_DSV41_ROUND_PROF times each decode round and the drafter's passes (stacked on #1720) #2951, timing_real.zig needs a real merge: dsv41 metal: TF_DSV41_ROUND_PROF times each decode round and the drafter's passes (stacked on #1720) #2951 adds four slots of one row (Multi) to once and report, and this PR makes once take a Launch and moves report's timing into time. Please keep both. After dsv41 metal: a multi-stream round's per-slot stages run as one dispatch each (stacked on #1720) #2954, the four-slot shape in timing_synth.zig also needs the slots' buffers declared (ask 1 on dsv41 metal: a multi-stream round's per-slot stages run as one dispatch each (stacked on #1720) #2954).

• Synthena (Ash's Agent)

[gilby]

M3 Ultra, 512 GB: the series is being re-measured now under the noise rule in #2984 (one discarded first run, base on both sides, both-under-both test); it follows in this thread.

• Head: 2402e6eb4
• Base: c97d661c4
• Checkpoint: deepseek-ai/DeepSeek-V4.1-Flash-1017 release weights as published (pinned revision: not recorded in these runs)
• Hardware: M5 Ultra, 512 GB; macOS 27.2.1 (26B131); Metal compiler 383.1 (printed by the rotary sweep of this run); the device-test and kernel-contract builds set TOOLCHAINS=com.apple.dt.toolchain.Metal.32023.922.4; the Xcode and Zig versions are not recorded in these logs
• Tools: tools/bench_concurrent.py --tokens 128 --reps 3 --temperatures 0,1 --levels 1,2,3,4 --alone --serial; tools/bench_openai.py --tokens 256 --reps 3 --temperatures 0,1; tools/bench_cache.py --prompt-file <file> --expected-prompt-tokens N --require-cold --tokens 64 --repeats 1; zig build test-dsv41-metal-all; zig build tf-kernel-contract tf-rotary-sweep; tf-rotary-sweep; tf-kernel-contract --only dsv41 --rotary-accuracy <sweep json> and the full tf-kernel-contract --rotary-accuracy <sweep json>; zig build time-dsv41-metal-synth with TF_DSV41_SYNTH_ROWS=1,2,3,4,5,6,8,16 (and TF_DSV41_TIMING_DIFFERENT=1 TF_DSV41_SYNTH_EXPERTS=32 for the different-rows table)
• Server flags: --context 1048576 --parallel 4 --prompt-cache-gib 0 --vision --no-update-check, TF_EXPERIMENTAL=deepseek; one fresh server per cell
• Arm order (M5 Ultra): decode and bench_openai: base, the head of dsv41 metal: launches of 6 to 16 rows run their FP8 projections on a matrix tile (stacked on #1720) #2973, the head of dsv41 metal: decode experts decode an expert's weights once for all of its rows (stacked on #1720, #2973) #2983, the head of dsv41 metal: decode experts decode an expert's weights once for all of its rows (stacked on #1720, #2973) #2983, the head of dsv41 metal: launches of 6 to 16 rows run their FP8 projections on a matrix tile (stacked on #1720) #2973, base (run 1 and run 2 of each arm are its first and second run in this order); cold prompts: base at 2,048 / 8,192 / 32,768 / 65,536 tokens, then each head the same way, one server per cell

Summary: token hashes equal to base in every cell of every table. Decode, mean over mean against base, 1 to 4 streams, T 1 and 0, chat and code: 1 stream 1.001 to 1.009, 2 streams 1.037 to 1.055, 3 streams 0.998 to 1.000, 4 streams 0.998 to 1.000. bench_openai 1.001 to 1.036. Cold-prompt TTFT 0.907 to 0.996 of base at 2,048 to 65,536 tokens (single runs). The M5 Ultra --only dsv41 kernel contract is clean: 95 kernels: 95 pass gates 1-4, 0 fail, 0 known findings; 190 controls: 190 red as planted, 0 missed; 0 limit problems.

Cells under 0.98 of base: none.

Kernel contract

M5 Ultra, 512 GB (applegpu_g17, tensor units true), --only dsv41 with the rotary receipt (tf-kernel-contract --only dsv41 --rotary-accuracy <sweep json>, the sweep generated by tf-rotary-sweep in the same tree), exit status 0, SUMMARY line verbatim:

SUMMARY 95 kernels: 95 pass gates 1-4, 0 fail, 0 known findings; 190 controls: 190 red as planted, 0 missed; 0 limit problems; gates 109983 ms, limits 17 ms

M5 Ultra full run (tf-kernel-contract --rotary-accuracy <sweep json>, no --only): attempted, exit status 1, and no SUMMARY line was printed: it stopped with error: ControlEditMissing right after the core deltanet registrations (SHARED_CONV_LIMITS lines), before reaching any DeepSeek registration. The same abort happens on the earlier base a43ad7b98; the full run was not run on c97d661c4 on this chip. There is no M5 Ultra full-run SUMMARY yet; the logs are in the gist.

M3 Ultra, 512 GB (applegpu_g15, tensor units false; macOS 26.6.2 (25G83), Metal compiler 373.7), final contract run, both runs verbatim. --only dsv41:

SUMMARY 95 kernels: 95 pass gates 1-4, 0 fail, 0 known findings; 190 controls: 190 red as planted, 0 missed; 0 limit problems; gates 130681 ms, limits 20 ms

M3 Ultra full run (tf-kernel-contract --rotary-accuracy <sweep json>):

SUMMARY 545 kernels: 545 pass gates 1-4, 0 fail, 3 known findings; 579 controls: 579 red as planted, 0 missed; 0 limit problems; gates 2627255 ms, limits 184 ms

For comparison, the base tree on the M3 Ultra (the tree at c97d661c4, the base of this PR), full run:

SUMMARY 539 kernels: 539 pass gates 1-4, 0 fail, 3 known findings; 575 controls: 575 red as planted, 0 missed; 0 limit problems; gates 2634630 ms, limits 15026 ms

and its --only dsv41 run:

SUMMARY 89 kernels: 89 pass gates 1-4, 0 fail, 0 known findings; 186 controls: 186 red as planted, 0 missed; 0 limit problems; gates 119168 ms, limits 277 ms

The full run needs a rotary sweep receipt passed with --rotary-accuracy, generated by tf-rotary-sweep in the same tree; without it the full run stops at the start with UnrecordedRotaryCompiler (the first, receipt-less attempt of each M3 Ultra tree is in the gist as contract-full-without-rotary.log). The 3 known findings are the same three on base and on this head, all three of kind nonfinite: nonfinite metal affine_mm q4 g64, nonfinite metal affine_mm q4 g64 fp32 out, and nonfinite metal nemotron experts up oracle tf_xup_w3. The --only dsv41 runs report 0 known findings on both chips.

Device tests (zig build test-dsv41-metal-all)

M5 Ultra: build exit status 0, 1137 PASS lines, no line starting with FAIL; the log ends with failed command: ... test ... --listen=- (twice per log, once per test step) although the build exit status is 0. M3 Ultra, earlier run of the same step on this head: Build Summary: 9/9 steps succeeded; 39/39 tests passed, 1137 PASS lines, with the same failed command lines in the log. Selected M5 Ultra line:

PASS dsv41 metal FP8 kernel by rows: row kernel at 1-5 rows, the MMA tile at 6-16

The base tree was not put through the device tests on either chip.

Decode, bench_concurrent aggregate tok/s, M5 Ultra, 512 GB

Order as run: base 1, this 1, [the other head twice], this 2, base 2. Ratios: this 1 over base 1, this 2 over base 2, and mean over mean. Token hashes equal in every cell across the four arms: yes; failed / alone / serial mismatches: 0. All 16 cells are reported.

streams	T	load	base 1	this 1	this 2	base 2	this 1 / base 1	this 2 / base 2	this / base (mean over mean)
1	0	chat	56.5	56.6	56.5	56.5	1.002	1.000	1.001
2	0	chat	76.9	80.2	80.0	76.9	1.043	1.040	1.042
3	0	chat	93.1	93.0	92.9	93.0	0.999	0.999	0.999
4	0	chat	109.2	109.0	109.0	109.1	0.998	0.999	0.999
1	0	code	61.2	61.6	61.5	61.2	1.007	1.005	1.006
2	0	code	81.4	85.9	85.8	81.4	1.055	1.054	1.055
3	0	code	96.2	96.1	96.0	96.2	0.999	0.998	0.998
4	0	code	114.0	113.8	113.8	114.0	0.998	0.998	0.998
1	1	chat	62.6	62.8	62.7	62.6	1.003	1.002	1.002
2	1	chat	70.7	73.4	73.3	70.8	1.038	1.035	1.037
3	1	chat	75.0	75.0	75.0	75.0	1.000	1.000	1.000
4	1	chat	85.3	85.3	85.3	85.3	1.000	1.000	1.000
1	1	code	69.7	70.5	70.3	69.8	1.011	1.007	1.009
2	1	code	67.2	70.3	70.3	66.9	1.046	1.051	1.048
3	1	code	76.3	76.3	76.2	76.3	1.000	0.999	0.999
4	1	code	86.8	86.7	86.7	86.7	0.999	1.000	0.999

bench_openai decode tok/s (median of 3), M5 Ultra

Token hashes equal to base in all four cells across the four arms: yes. The run summary flags the two T 1 cells as unstable between the repeats inside one server, base included; their token hashes are still equal across all four arms.

prompt	T	base 1	this 1	this 2	base 2	this 1 / base 1	this 2 / base 2	this / base (mean over mean)	tokens per chunk, base / this
fibonacci-raw	0	71.2	71.8	71.8	71.2	1.008	1.008	1.008	3.16 / 3.16
fibonacci-raw	1	67.1	70.0	69.0	67.1	1.044	1.027	1.036	3.2 / 3.2
gpu-chat-no-think	0	56.7	56.8	56.8	56.7	1.002	1.001	1.001	2.349 / 2.349
gpu-chat-no-think	1	57.7	58.1	57.8	57.7	1.008	1.000	1.004	2.393 / 2.393

Cold prompts (bench_cache --require-cold, one run per cell, reply 64 tokens, T 0), M5 Ultra

TTFT ratio below 1.000 is faster than base. Each cell is a single run and base ran before the heads (base at all four sizes, then the head of #2973, then the head of #2983), so these cells do not separate the change from drift.

prompt tokens	base TTFT (s)	this TTFT (s)	this / base	cached tokens (base / this)	token_sha base	token_sha this	equal
2048	6.03	5.47	0.907	0 / 0	330cd07dd3cf	330cd07dd3cf	yes
8192	11.90	11.86	0.996	0 / 0	b2da1b2b327d	b2da1b2b327d	yes
32768	37.28	36.69	0.984	0 / 0	7cfb5e60de47	7cfb5e60de47	yes
65536	70.14	68.94	0.983	0 / 0	cce981123927	cce981123927	yes

Synthetic row timing, M5 Ultra (time-dsv41-metal-synth, release-shape layers, fastest of 7, microseconds, layer 0 only). Base has no such step, so there is no base column: the table shows this head next to the head of #2983 only because both were timed in the same run; the FP8 rows column is what #2973 changes.

Identical rows, 16 experts:

shape (streams x rows)	head of #2973 round	FP8 rows	experts	head of #2983 round	FP8 rows	experts	experts, #2983 / #2973
1x1	625.6	287.3	193.6	552.3	279.1	182.5	0.943
1x2	661.5	267.1	257.0	648.2	295.7	211.0	0.821
1x3	690.0	282.4	264.8	646.8	283.8	228.1	0.861
1x4	729.8	310.9	275.4	681.4	309.6	233.5	0.848
1x5	922.6	338.1	430.5	780.9	339.0	295.3	0.686
1x6	981.7	378.0	440.5	820.9	371.1	299.3	0.679
1x8	1014.6	382.8	466.2	905.3	372.7	353.3	0.758
1x16	1446.6	486.4	850.9	1205.1	487.6	618.3	0.727
4x1	713.9	304.7	346.0	734.5	312.1	302.2	0.873

Different rows, 32 experts:

shape (streams x rows)	head of #2973 round	FP8 rows	experts	head of #2983 round	FP8 rows	experts	experts, #2983 / #2973
1x1	590.4	260.6	193.0	586.1	269.3	194.2	1.006
1x2	695.3	260.2	320.1	642.5	260.6	280.3	0.876
1x3	821.0	284.6	404.5	754.4	285.6	351.2	0.868
1x4	866.4	308.5	436.8	815.9	311.3	401.4	0.919
1x5	1012.0	337.7	567.8	892.0	342.7	508.5	0.896
1x6	1096.9	377.9	644.9	1030.4	387.9	551.2	0.855
1x8	1210.4	379.7	674.4	1081.9	373.8	609.7	0.904
1x16	1653.4	484.1	1083.9	1442.4	488.5	848.3	0.783
4x1	780.5	309.0	417.5	782.7	309.2	368.7	0.883

Method as run: each cell is a fresh server started just before it; a thermal-pressure check (powermetrics thermal sampler) ran at the start and the end of every cell, and a cell that was not Nominal at either end was discarded and re-run after a pause. Three cold cells were discarded for thermal state and re-run after a pause (base at 32,768 tokens: ended Heavy; base at 65,536: ended Moderate; the head of #2983 at 65,536: started Moderate), and one further attempt of base at 65,536 tokens was started and interrupted when the script was restarted (no result). The cold-size script was then re-started with end-of-cell pressure informational for the long prefills, so two kept cells ended above Nominal: base at 65,536 tokens ended Moderate and the head of #2983 at 32,768 tokens ended Heavy. The raw files of the discarded attempts were not kept, only the thermal and discard logs. The decode, bench_openai and synthetic-timing cells were not discarded. One-stream --no-drafts control cells on the base tree ran between series (4 of them are relevant here) and read 46.6 / 46.6 / 46.6 / 46.8 tok/s, against a gate of at least 42 tok/s and Nominal thermal pressure. These M5 Ultra series were run before the noise rule in #2984 and have no discarded first run for that purpose; every arm is a fresh server and the order is base, this, this, base wherever a pair is shown. Kept cells that were not Nominal at an end, or where a macOS background process was busy: cold 65,536 tokens, base ended at Moderate thermal pressure; cold 8,192 tokens, head of #2973 a macOS background process (mediaanalysisd) was using 100% CPU at the end.

Not run: the full kernel contract on the M5 Ultra (it aborts, see above); the --only dsv41 contract on the base tree on the M5 Ultra; device tests on the base tree on either chip; synthetic timing on the base tree; the checkpoint's pinned revision is not recorded.

Raw JSON, server logs and driver scripts: https://gist.github.com/gilby/df7bed9a89d9c854cfe78479b4c8cb6f

[ashhart]

Thanks @gilby, this is a thorough write-up, and it answers part of ask 1.

• M5 Ultra receipt. It's in the form we asked for: base, this, this, base, with every token_sha256 equal and concurrent equal to alone and serial. Two drafted streams run 3.7-5.5% faster, everything else is level, and no cell is under 0.98 of base.
• M3 Ultra contract. --only dsv41 reads 95 kernels pass and 190 of 190 controls red. The full run reads 545 pass, with the same three nonfinite known findings as the base tree. That covers the contract part of ask 1.
• The M5 Ultra full run stops with ControlEditMissing after the LFM3 conv segments, before any DeepSeek kernel, on an earlier base of the stack too. So it isn't this PR's. We'll check whether main does the same on an M5.

Still open:

1. From ask 1:
   • The dsv41_fp8_tile equality line from the M3 run (zig/tests/contract/metal_dsv41_fp8_tile_eq.zig:119), so the launch count and "0 differ" are on record.
   • The M3 receipt you're re-measuring under the DeepSeek-V4.1 on Metal: decode and prompt speed on M3 Ultra and M5 Ultra, token-exact (gate and order for the drafts) #2984 rule, with the checkpoint revision recorded.
   • time-dsv41-metal-synth on the M3.
   • If the M3's crossover isn't at 6 rows, a min_rows (zig/src/families/deepseek_v41/metal/fp8_tile.zig:13) that is no slower on either chip.
2. The CHANGELOG.md line still quotes the synthetic M5 Max layer times. Please swap them for the served results on both chips. The cold-prompt cells are single runs with base first, so please leave prompt speed out of the line.
3. A docs/kernel-contract.md entry for the tile, as in ask 3.
4. The rebase in the order dsv41 metal: TF_DSV41_ROUND_PROF times each decode round and the drafter's passes (stacked on #1720) #2951, dsv41 metal: prompts in pieces between the other streams' rounds (stacked on #1720) #2950, dsv41 metal: a multi-stream round's per-slot stages run as one dispatch each (stacked on #1720) #2954, this, dsv41 metal: decode experts decode an expert's weights once for all of its rows (stacked on #1720, #2973) #2983, then DeepSeek-V4.1 on Metal: opt-in draft cost table timed at start, default pricing unchanged (stacked on #1720) #2952, with the timing_real.zig merge from ask 4.

No hurry on any of it. Post the M3 numbers here when they're in.

• Synthena (Ash's Agent)

[gilby]

M5 Ultra, 512 GB: the full kernel-contract SUMMARY that was missing above. The full run stops at registration on chips with tensor units since #2251 (#3067); with the one-line fix proposed there applied to this tree's copy, it completes on this head (2402e6eb4), exit status 0, macOS 27.2.1, Metal compiler 383.1:

SUMMARY 756 kernels: 756 pass gates 1-4, 0 fail, 57 known findings; 640 controls: 640 red as planted, 0 missed; 0 limit problems; gates 3792910 ms, limits 3718 ms

The base tree c97d661c4, same machine, same fix:

SUMMARY 750 kernels: 750 pass gates 1-4, 0 fail, 57 known findings; 636 controls: 636 red as planted, 0 missed; 0 limit problems; gates 3760102 ms, limits 21777 ms

And --only dsv41 on this head, from the same build:

SUMMARY 95 kernels: 95 pass gates 1-4, 0 fail, 0 known findings; 190 controls: 190 red as planted, 0 missed; 0 limit problems; gates 110294 ms, limits 227 ms

The 57 known findings are the same 57 lines on the base tree and on this head, and none is a DeepSeek-V4.1 record: 41 are the #1009 masked-value entries of other families' attention kernels that this chip registers (Gemma 4, Spark2.5, MiniMax M3, Llama, GLM, the step kernels), and the rest are those families' prompt-attention entries, the affine_mm q4 g64 entries and the nemotron experts up oracle entry; the --only dsv41 run has none. The Llama control that #3067 is about goes red as planted with the fix (CONTROL control: metal llama gemm_00, the weights are read untransposed, as [k, n]: red at shapes, as planted). For comparison, the M3 Ultra full run posted earlier has 3 known findings because that chip does not register the tensor-unit kernels.

Full logs for the four trees and the applied one-line diff: https://gist.github.com/gilby/fb5f4ae45e3089e7f742c17753623740

[gilby]

M3 Ultra, 512 GB: the receipt promised above, run on 2026-11-13 under the noise rule in #2984 (one discarded first run per series, base on both sides, both-under-both test).

• Head: 2402e6eb4
• Base: c97d661c4
• Checkpoint: deepseek-ai/DeepSeek-V4.1-Flash-1017 release weights as published, read from local disk (pinned revision: not recorded in these runs)
• Hardware: M3 Ultra, 512 GB; the macOS version, Metal compiler, Xcode and Zig versions are not recorded in these logs
• Tools, from the base tree in every arm: tools/bench_concurrent.py --tokens 128 --reps 3 --alone --serial at 1, 2, 3 and 4 streams, temperature 0 and 1; tools/bench_openai.py --tokens 256 --reps 3 at temperature 0 and 1
• Server flags: --context 1048576 --parallel 4 --prompt-cache-gib 0 --vision --no-update-check, TF_EXPERIMENTAL=deepseek; one fresh server per cell; every TENSORFOLD_PREFILL_* and TF_DSV41_* setting is unset unless the row names it
• Arm order, per series: one discarded run (base), base, this, this, base; a cell starts only at Nominal thermal pressure after a 60 s settle and its end level is recorded; a cell that ends at Heavy is discarded and retried once, and if it ends at Heavy again it is kept and flagged
• Server binaries (sha256 prefix): base 69a6bd19c1e9, this 3020fcac8ccf

Summary: this head gains at 2 streams and in one bench_openai cell and is level elsewhere. The four 2-stream bench_concurrent cells are all faster by the #2984 rule, this over base 1.036 to 1.054 (+3.6% to +5.4%); the one bench_openai cell is fibonacci-raw at temperature 1, 1.031 (+3.1%). The other 14 cells (1, 3 and 4 streams and the other bench_openai cells) are 0.999 to 1.006, within noise. The table also shows 4 streams, temperature 0, chat as faster at 1.011, but that is not a gain: its base 2 run ended at Heavy twice and read 77.1 tok/s against 78.6 for base 1, while both this runs (78.7, 78.7) equal base 1; read that cell as within noise. Token hashes equal in every cell, 0 failed or unequal comparisons. No cell is slower.

Cells under 0.98 of base: none.

bench_concurrent aggregate tok/s and bench_openai decode tok/s (median of 3 requests; the openai rows are 1 stream, load names the prompt)

Order as run: base 1, this 1, this 2, base 2. Ratios: this 1 over base 1, this 2 over base 2, and mean over mean. Verdict by the #2984 rule: slower only if both this runs are under both base runs and the mean gap is over 1%, faster the mirror image, otherwise within noise. Verdicts as the rule gives them: 6 faster (5 of them real gains per the paragraph above, one the Heavy-affected cell), 0 slower, 14 within noise, out of 20 cells.

streams	T	load	base 1	this 1	this 2	base 2	this 1 / base 1	this 2 / base 2	this / base (mean over mean)	verdict	end level (b1 / t1 / t2 / b2)
1	0	chat	43.5	43.5	43.6	43.6	1.000	1.000	1.000	within noise	Mod/Mod/Mod/Mod
2	0	chat	55.8	58.0	58.0	55.9	1.039	1.038	1.038	faster	Mod/Mod/Mod/Mod
1	0	code	46.4	46.6	46.5	46.4	1.004	1.002	1.003	within noise	Mod/Mod/Mod/Mod
2	0	code	59.3	62.6	62.4	59.3	1.056	1.052	1.054	faster	Mod/Mod/Mod/Mod
1	1	chat	47.8	47.8	47.8	47.8	1.000	1.000	1.000	within noise	Mod/Mod/Mod/Mod
2	1	chat	50.8	52.7	52.7	50.9	1.037	1.035	1.036	faster	Mod/Mod/Mod/Mod
1	1	code	51.8	52.1	52.1	51.8	1.006	1.006	1.006	within noise	Mod/Mod/Mod/Mod
2	1	code	48.9	51.3	51.3	48.9	1.049	1.049	1.049	faster	Mod/Mod/Mod/Mod
3	0	chat	68.7	68.9	68.8	68.4	1.003	1.006	1.004	within noise	Mod/Mod/Mod/Heavy*
4	0	chat	78.6	78.7	78.7	77.1	1.001	1.021	1.011	faster	Mod/Mod/Mod/Heavy*
3	0	code	72.2	72.2	72.2	72.4	1.000	0.997	0.999	within noise	Mod/Mod/Mod/Heavy*
4	0	code	84.1	83.9	84.1	84.1	0.998	1.000	0.999	within noise	Mod/Mod/Mod/Heavy*
3	1	chat	55.7	55.7	55.8	55.7	1.000	1.002	1.001	within noise	Mod/Mod/Mod/Heavy*
4	1	chat	61.7	61.7	61.5	61.6	1.000	0.998	0.999	within noise	Mod/Mod/Mod/Heavy*
3	1	code	57.5	57.5	57.5	57.4	1.000	1.002	1.001	within noise	Mod/Mod/Mod/Heavy*
4	1	code	63.8	63.9	63.9	63.8	1.002	1.002	1.002	within noise	Mod/Mod/Mod/Heavy*
1	0	fibonacci-raw	54.5	54.7	54.7	54.4	1.005	1.006	1.005	within noise	Mod/Mod/Mod/Mod
1	0	gpu-chat-no-think	43.8	43.8	43.7	43.8	1.001	0.999	1.000	within noise	Mod/Mod/Mod/Mod
1	1	fibonacci-raw	50.3	51.7	51.9	50.2	1.028	1.033	1.031	faster	Mod/Mod/Mod/Mod
1	1	gpu-chat-no-think	43.7	43.6	43.7	43.6	0.998	1.002	1.000	within noise	Mod/Mod/Mod/Mod

End level key: Nom = Nominal, Mod = Moderate, Heavy = Heavy, * = ended at Heavy twice, kept and flagged.

Exactness: token hashes are equal across all four runs in every one of the 20 cells (all reps of each cell compared), with 0 failed requests and 0 --alone / --serial mismatches. Checked against the raw JSON: hash equality over the four runs of each cell, and the failed, alone and serial counts over every run including the discarded ones.

Discarded, retried and flagged cells

• The first run of every series is the discarded run and is in no table (series d-fp8rows-c12, d-fp8rows-c34, d-fp8rows-oai).
• Flagged: base 2 of the 3-and-4-stream series (d-fp8rows-c34-base2) ended at Heavy on its first try, was discarded and re-run once after cooling, and ended at Heavy again; the second run is kept and flagged. It affects all 8 cells of the 3 and 4 stream rows (base 2 column). Its first try's raw files are in the gist as .try1 files. Apart from the 4-stream temperature 0 chat cell discussed above, its numbers agree with base 1 within 0.5%.
• Nothing else ended at Heavy; every other kept cell ended at Moderate (the 1-and-2-stream and bench_openai series all ended at Moderate in all four arms).

Method as run: each cell is a fresh server started after the previous one was stopped and a 60 s settle, and a cell starts only when the powermetrics thermal sampler reports Nominal (the driver polls every 30 s); the level is recorded again at the end of the cell. One run per series is discarded and not in any table, base is on both sides, and the series are per stream count and temperature (bench_concurrent: 1 and 2 streams in one series, 3 and 4 in another where noted) and per temperature (bench_openai). A 2-minute multi-stream cell that starts at Nominal usually ends at Moderate on this machine, so Moderate does not discard a cell; only an end at Heavy does: the cell is then discarded, cooled to Nominal and re-run once, and if it ends at Heavy again it is kept and flagged. Tools and server flags are the ones in the header. Every arm, the raw JSON, the server logs and the driver scripts are in the gist.

Not run: the checkpoint's pinned revision, the macOS version and the toolchain versions are not recorded; kernel-contract runs are not part of this series' data.

Raw JSON, server logs and driver scripts: https://gist.github.com/gilby/c020133ac9809d5f01d5b34dd60fc308

[gilby]

New head: ff8365687 (two commits on top of 2402e6eb4, not rebased).

• f2241b84a: ask 3 from 12 Nov 13:18, repeated at 23:50. docs/kernel-contract.md has an entry for the tile: the registration that replaces tf_dsv41_fp8_w* run one row a launch and must be bit-equal to it at every shape of its shapes gate (the seven release projections and every row count from 1 to 16), the f32, bf16, forced-form and release-fixture registrations against float64, the equality probe (every dense FP8 shape the decode forward runs, f32 and bf16, rows 1 to 16 from row 0 and 1 to 15 from row 1 at every form, the 16/17 seam and a three-launch batch, NaN and Inf planted), and the four controls.
• ff8365687: ask 2 from 12 Nov 13:18 and 23:50. The CHANGELOG line quotes the served result on both chips instead of the synthetic M5 Max layer times, and leaves prompt speed out: on the release weights two drafted streams decode 3.7 to 5.5% faster on the M5 Ultra and 3.6 to 5.4% faster on the M3 Ultra, other stream counts are level, and every token is unchanged.

Receipts: both new commits are documentation and the CHANGELOG, so the M5 Ultra and M3 Ultra receipts and both chips' contract lines above (taken at 2402e6eb4) describe this head's code. For the record, the checkpoint in both receipts is deepseek-ai/DeepSeek-V4.1-Flash-1017 at revision 1bae0e1088993b9b82de08f59de5be5a34e42170 (the receipts say "not recorded"; this is the revision those weights were downloaded at).

Your ask 1 from 12 Nov 13:18 and 23:50, item by item:

• The M3 Ultra contract (--only dsv41) and the M3 Ultra receipt under the DeepSeek-V4.1 on Metal: decode and prompt speed on M3 Ultra and M5 Ultra, token-exact (gate and order for the drafts) #2984 rule: posted on 12 and 13 Nov.
• The dsv41_fp8_tile equality line from the M3 Ultra: still owed. It prints in the same --only dsv41 run as the SUMMARY; I'll post the line with its launch count.
• time-dsv41-metal-synth on the M3 Ultra: still owed.
• min_rows: decided from that synthetic run together with the served receipts. The M3 Ultra receipt already shows no slower cell at 1, 3 and 4 streams (launches under 6 rows keep the row kernel there) and 3.6 to 5.4% faster at two streams, where launches reach 6 or more rows. If the M3 Ultra's FP8 rows column shows the tile ahead at 5 rows, or behind at 6, I'll propose the value no slower on either chip, with both chips' contract on each side of the crossover as DeepSeek-V4.1 on Metal: decode and prompt speed on M3 Ultra and M5 Ultra, token-exact (gate and order for the drafts) #2984 asks; otherwise 6 stays.

Ask 4 (the rebase in the #2984 order, with the timing_real.zig merge): I am not pushing a rebase of a branch under review to follow main; this head stays on c97d661c4 and is ready to be rebased at landing. I did the timing_real.zig merge in a local rebase of the stack, keeping both: once and time take a Launch as here, report takes #2951's Multi and builds its Launch from it, Timing.print takes the slot count so the header keeps #2951's "S slot(s) x N row(s)", and timing_synth.zig's one print call passes it. The four-slot shape there still needs segs.declare once #2954 is under it (see #2954).

At ff8365687: documentation only since 2402e6eb4, where test-dsv41-metal-all passed on the M5 Max and the M3 Ultra; lean_check.py 345 on base and head; pr_check.py --pr 2973 0 errors and 0 warnings.

[gilby]

M3 Ultra, 512 GB: the contract lines and the time-dsv41-metal-synth round table that the earlier comment said were still owed, at head ff8365687.

• Checkpoint: deepseek-ai/DeepSeek-V4.1-Flash-1017 revision 1bae0e1088993b9b82de08f59de5be5a34e42170, release weights as published, read from local disk
• Hardware: M3 Ultra, 512 GB
• Server flags: --context 1048576 --parallel 4 --prompt-cache-gib 0 --vision --no-update-check, TF_EXPERIMENTAL=deepseek, one fresh server per cell
• Thermal: a cell starts at Nominal pressure after a 60 s settle and the end level is recorded
• Head: ff8365687; tools built from the head (zig build tf-kernel-contract tf-rotary-sweep; zig build time-dsv41-metal-synth)
• Hardware as recorded by the rotary sweep: ROTARY SWEEP compiler 373.7/373.7, OS 26.6.2 (25G83), GPU generation 15; no partial receipt

Kernel contract, tf-rotary-sweep --out rotary.json then tf-kernel-contract --only dsv41 --rotary-accuracy rotary.json (both exit 0; the sweep ended COMPLETE 302008320 cases), the two lines verbatim:

SUMMARY 95 kernels: 95 pass gates 1-4, 0 fail, 0 known findings; 190 controls: 190 red as planted, 0 missed; 0 limit problems; gates 130743 ms, limits 253 ms
PASS dsv41_fp8_tile equality: 9 model shapes x f32/bf16, rows 1-16 from row 0 and 1-15 from row 1 at every form, the 16/17 seam and a 3-launch batch (4500 launches, NaN/Inf planted, 153374 NaN outputs in the f32 baselines) byte-equal to fp8_rows one row a launch: 0 differ

time-dsv41-metal-synth with TF_DSV41_SYNTH_ROWS=1,2,3,4,5,6,8,16. Each figure is the fastest of 7 timed runs of one synthetic round (release-shape layers, a 40-layer round weighted by layer kind, no head, Engram or host), one run of the tool per table: these are single samples, not a bracketed A/B, and the tool's per-kernel columns are measured by differencing, so small columns can read slightly negative per layer. The server was not running; the run started at Nominal pressure and ended at Nominal (identical rows) and Nominal (different rows). The 40-layer round, 16 experts, identical rows:

launch (slots x rows)	round, concurrent (us)	round, alone (us)	FP8 rows (us)	experts (us)	attention (us)	glue (us)
1 x 1	25,013.0	29,971.7	11,873.5	9,095.0	3,168.1	5,835.1
1 x 2	29,962.1	35,348.7	12,703.5	13,701.4	3,113.7	5,830.2
1 x 3	31,690.5	36,915.8	13,436.9	14,085.0	3,404.7	5,989.3
1 x 4	34,013.7	39,675.7	14,991.0	14,864.8	3,773.0	6,046.9
1 x 5	45,058.9	50,386.7	16,988.0	23,098.1	3,910.9	6,389.7
1 x 6	49,467.4	54,968.4	19,920.9	23,745.4	4,703.6	6,598.5
1 x 8	52,155.8	57,521.4	19,915.0	24,899.6	5,911.9	6,794.8
1 x 16	84,007.2	89,685.4	25,618.2	45,873.4	10,106.1	8,087.7
4 x 1	35,135.3	48,609.9	15,020.0	14,744.2	11,959.6	6,886.1

The same with 32 experts and different rows (TF_DSV41_TIMING_DIFFERENT=1 TF_DSV41_SYNTH_EXPERTS=32):

launch (slots x rows)	round, concurrent (us)	round, alone (us)	FP8 rows (us)	experts (us)	attention (us)	glue (us)
1 x 1	25,378.7	30,390.5	11,959.8	9,205.8	3,197.7	6,027.3
1 x 2	33,747.4	39,194.8	12,735.7	17,193.4	3,161.7	6,103.9
1 x 3	35,624.5	42,543.2	13,509.7	19,522.7	3,420.7	6,090.0
1 x 4	39,156.1	46,452.2	15,084.9	21,404.4	3,767.4	6,195.6
1 x 5	47,818.4	55,329.0	17,104.8	27,671.3	3,913.1	6,639.9
1 x 6	55,600.9	62,570.8	20,303.5	30,693.2	4,727.0	6,847.1
1 x 8	61,380.2	67,439.3	20,082.4	34,379.9	5,923.5	7,053.4
1 x 16	97,075.9	103,165.1	25,768.1	58,689.6	10,400.1	8,307.3
4 x 1	41,113.5	55,917.1	15,145.7	21,568.5	11,993.4	7,209.6

The min_rows question, from the FP8 rows column only. Rows 1 to 5 run the row kernel and rows 6 and up run the tile. In the identical-rows run the FP8 rows time goes 11,874, 12,704, 13,437, 14,991, 16,988 us for 1 to 5 rows (steps of 830, 733, 1,554 and 1,997 us), then 19,921 us at 6 rows (a step of 2,933 us), 19,915 us at 8 and 25,618 us at 16; the different-rows run shows the same shape (17,105 at 5, 20,304 at 6, 20,082 at 8, 25,768 at 16). The 6-row launch costs as much as the 8-row one (a difference of 6 us in the first table). This run times each row count with the threshold as shipped, so it has no row kernel at 6 rows and no tile at 5 rows to set against each other: the data does not decide whether a threshold of 6 is the best value on this chip. Settling it needs those two forms timed, which I have not done; until then 6 stays, and the served receipt (no slower cell at any stream count, 3.6 to 5.4% faster at two streams) is the only evidence for it on this chip.

Raw outputs and drivers: https://gist.github.com/gilby/f956f91d5bc63f30b6ccb4ba549e061a