Repository navigation
Conversation
…over Metal target.zig is the lanes' verify target: a slot's prompt, its windows of rows for several slots in one launch, keeps and rewinds, and kept prompt states (the prompt cache) whose resumed rows the reply reports as cached_tokens. boot.zig loads a release checkpoint folder and entry.zig serves it as tensorfold-dsv41 (dsv41-metal-serve). Tests serve the tiny model through LaneHost: concurrent equals solo equals serial, drafted equals draft:false, cache hits equal fresh runs, seeds and cancels.
…for the next A shared round with the plain switch at it asked no drafts and ran no head pass (core Rule.plain_streams, threaded from the family's resolved setting: 0 on CUDA, so its rounds are unchanged), and the family's speculation no longer launches a pass for a round that wide (armSpec/speculate plainRound).
…teps and the forward try's
…stays within 600 lines
…nything else by name
…treams streams decode
…er holds a rejected draft's taps
…nd the next ask catches the drafter up
…ers per row and dispatch class Six synthetic layers of the release's widths (one of each attention kind, few experts) are timed at 1-16 rows of one slot and at 4 slots x 1 row, each dispatch alone and the layer on the serving encoder, and split into FP8 rows, experts, attention and glue, with a 40-layer round weighted by kind. timing_real's report takes a launch (context and spans) so both harnesses share it.
…x tile, bit-equal to the row kernel Launches of 6 to 16 rows run the dense FP8 projections on fp8_tile.metal: 8 or 16 columns a threadgroup, K split over up to 16 simdgroups in aligned power-of-two runs of groups, each 32-wide group summed from zero in four 8-wide MMA steps in k order (the prompt tile's group root), the groups' roots and the slices' roots in the same zero-padded pair tree the row kernel uses. Every row's bits are the row kernel's one row a launch; 1-5 rows keep the row kernel (chosen by row count only), since below 6 rows it reads the weights faster. Contract: six registrations (f32, bf16, a forced 16-column form, one simdgroup, the row kernel one row a launch as replaced baseline, the release wkv fixture), four planted controls, gate-5 records with M1/M2 limits, and an equality probe over every model FP8 shape at 1-16 rows, every form, the 16/17 seam and a batch with NaN and Inf planted. Device tests: the layer's kernel by row count, and four slots of two rows each in one launch against each alone.
…f its rows, bit-equal to one row a launch The decode gate/up and down for an expert's slot of 2-16 rows ran Kimi's four-row passes: two rows paid for four rows of arithmetic, five rows for two passes of four, each rereading the weights. The new tf_dsv41_xp_upc_rows and tf_dsv41_xp_down_rows decode each 32-group of an expert's MXFP4 weights once per pass and apply it to every row in it (passes of 2, 4, 6 or 8 rows for gate/up, 2 or 4 for down, rows past the slot repeating its last row unwritten), reading the activations straight from device memory. Each row's chain is the one-row kernel's: 32 products in k order from zero, then its group's scale, then the same tree and epilogue, so a row's bits are the same alone or with any other rows and routing. Slots of one row keep tf_dsv41_xp_upc_one/down_one unchanged. Contract: registrations with every row on the same six experts (gate/up coded and down, gates 1-4 against the oracle), one planted control a kernel (rows reading their pass's first row, red at shapes), gate-5 records for both launches, and a probe holding a launch's act, out and combine output bit-equal to each row launched alone at 1-16 rows on the same, disjoint and random experts, with NaN and Inf planted, plus its own control. Device test: the same equality at 2-16 rows.
ae8373c to
b471c94
Compare
|
First real-weights receipt, with the contract on the M5 Ultra. One M5 Ultra 512 GB, Contract on the M5 Ultra at Greedy, aggregate tok/s:
Sampled cells have the same shape (two streams of sampled code 1.111). So on this chip the expert rows are worth 6 to 7% at one drafted stream, 15 to 17% at two (10 to 11% of it over #2973), and 2 to 3% at three and four, where rounds are plain and each expert has few rows. The M3 Ultra contract and receipt are running; I'll post them here. The issue line in the body now points at #2984. |
|
Thanks @gilby. This is a clean change. The code is ready. Owed before it can land:
|
|
M3 Ultra, 512 GB: the series is being re-measured now under the noise rule in #2984 (one discarded first run, base on both sides, both-under-both test); it follows in this thread.
Summary: token hashes equal to base in every cell of every table. Decode, mean over mean against base, 1 to 4 streams, T 1 and 0, chat and code: 1 stream 1.060 to 1.074, 2 streams 1.103 to 1.170, 3 streams 1.023 to 1.055, 4 streams 1.023 to 1.049. Cells under 0.98 of base: none. Kernel contract M5 Ultra, 512 GB (applegpu_g17, tensor units true), M5 Ultra full run ( M3 Ultra, 512 GB (applegpu_g15, tensor units false; macOS 26.6.2 (25G83), Metal compiler 373.7), final contract run, both runs verbatim. M3 Ultra full run ( For comparison, the base tree on the M3 Ultra (the tree at and its The full run needs a rotary sweep receipt passed with Device tests ( M5 Ultra: build exit status 0, 1138 PASS lines, no line starting with FAIL; the log ends with The base tree was not put through the device tests on either chip. Decode, Order as run: base 1, this 1, [the other head twice], this 2, base 2. Ratios: this 1 over base 1, this 2 over base 2, and mean over mean. Token hashes equal in every cell across the four arms: yes; failed / alone / serial mismatches: 0. All 16 cells are reported.
Token hashes equal to base in all four cells across the four arms: yes. The run summary flags the two T 1 cells as unstable between the repeats inside one server, base included; their token hashes are still equal across all four arms.
Cold prompts ( TTFT ratio below 1.000 is faster than base. Each cell is a single run and base ran before the heads (base at all four sizes, then the head of #2973, then the head of #2983), so these cells do not separate the change from drift.
Synthetic row timing, M5 Ultra ( Identical rows, 16 experts:
Different rows, 32 experts:
Method as run: each cell is a fresh server started just before it; a thermal-pressure check ( Not run: the full kernel contract on the M5 Ultra (it aborts, see above); the Raw JSON, server logs and driver scripts: https://gist.github.com/gilby/56e53417c07a002e3fb1c3aec95b661d |
|
Thanks @gilby. The M3 Ultra contract at The M5 Ultra full-run abort doesn't come from this PR. It's main's Llama The M5 Ultra receipt reads well. Every Still owed, as before:
|
|
M5 Ultra, 512 GB: the full kernel-contract SUMMARY that was missing above. The full run stops at registration on chips with tensor units since #2251 (#3067); with the one-line fix proposed there applied to this tree's copy, it completes on this head ( The base tree And The 57 known findings are the same 57 lines on the base tree and on this head, and none is a DeepSeek-V4.1 record: 41 are the #1009 masked-value entries of other families' attention kernels that this chip registers (Gemma 4, Spark2.5, MiniMax M3, Llama, GLM, the step kernels), and the rest are those families' prompt-attention entries, the Full logs for the four trees and the applied one-line diff: https://gist.github.com/gilby/fb5f4ae45e3089e7f742c17753623740 |
|
M3 Ultra, 512 GB: the receipt promised above, run on 2026-11-13 under the noise rule in #2984 (one discarded first run per series, base on both sides, both-under-both test).
Summary: every one of the 20 cells is faster by the #2984 rule and none is slower: this over base, mean over mean, is 1.024 to 1.166 (+2.4% to +16.6%). By group: 1 and 2 streams 1.053 to 1.166, 3 and 4 streams 1.024 to 1.048, Cells under 0.98 of base: none.
Order as run: base 1, this 1, this 2, base 2. Ratios: this 1 over base 1, this 2 over base 2, and mean over mean. Verdict by the #2984 rule: slower only if both this runs are under both base runs and the mean gap is over 1%, faster the mirror image, otherwise within noise. Verdicts: 20 faster, 0 slower, 0 within noise, out of 20 cells.
End level key: Nom = Nominal, Mod = Moderate, Heavy = Heavy, Exactness: token hashes are equal across all four runs in every one of the 20 cells (all reps of each cell compared), with 0 failed requests and 0 Cold first request ( First request only;
TTFT over the mean of the two base runs, mean of the two this runs: 2,048 tokens 0.9938, 8,192 tokens 0.9977, 32,768 tokens 0.9991; token hashes equal in all four runs of each prompt length. The cold decode rate in the same requests is higher on this head (for example 50.7 and 50.6 against 47.9 and 47.8 at 2,048 tokens), single runs. All four 32,768-token cells ended at Heavy twice and are flagged: each ended at Heavy on its first attempt, was discarded and re-run once after cooling, and ended at Heavy again; the second attempt is kept. On this machine a cold prefill of that size ends at Heavy whichever arm runs. Discarded, retried and flagged cells
Method as run: each cell is a fresh server started after the previous one was stopped and a 60 s settle, and a cell starts only when the powermetrics thermal sampler reports Nominal (the driver polls every 30 s); the level is recorded again at the end of the cell. One run per series is discarded and not in any table, base is on both sides, and the series are per stream count and temperature ( Not run: the checkpoint's pinned revision, the macOS version and the toolchain versions are not recorded; a base-versus-head comparison without #2973's change underneath; kernel-contract runs are not part of this series' data. Raw JSON, server logs and driver scripts: https://gist.github.com/gilby/9142a61266ba7f68358c34d288556e1f |
…, not synthetic percentages
|
New head:
Receipts: the new commit is the CHANGELOG, so the M5 Ultra and M3 Ultra receipts and contract lines above (taken at Your other items from 12 Nov 23:50:
At |
|
M3 Ultra, 512 GB: the
Each figure is the fastest of 7 timed runs of one synthetic round (release-shape layers, a 40-layer round weighted by layer kind, no head, Engram or host), one run of the tool per table: these are single samples, not a bracketed A/B, and the tool's per-kernel columns are measured by differencing, so small columns can read slightly negative per layer. The 40-layer round at this head, 16 experts, identical rows:
The same with 32 experts and different rows (
This head against the #2973 head, round totals and the experts column (recomputed from the two output files of each head). Identical rows, 16 experts:
Different rows, 32 experts:
At 16 rows the round goes 84,007 to 75,779 us with identical rows (-9.8%) and 97,076 to 88,364 us with different rows (-9.0%); the experts column is 45,873 to 37,376 us and 58,690 to 47,226 us. The FP8 rows column is the same at both heads within 161 us (identical rows) and 107 us (different rows), as it should be since only the experts differ. This run times the pass sizes as shipped and not other sizes, so it shows the gain but does not say whether another pass size (2, 4, 6 or 8 rows for gate/up, 2 or 4 for down) would be faster on this chip; I have not timed those, so I am not proposing a change of sizes from it. Raw outputs and drivers: https://gist.github.com/gilby/dec04808f0294eb3a588c9c1b80af0ad |
What this changes
On Metal, DeepSeek-V4.1's routed-expert kernels for more than one row ran every expert's rows in passes padded to four rows: a two-row launch did four rows of arithmetic, and a five-row slot ran two padded passes and decoded the expert's weights twice. With this change the multi-row gate/up and down kernels (
tf_dsv41_xp_upc_rows,tf_dsv41_xp_down_rows) decode an expert's weights once per pass and size the pass by the rows left (2, 4, 6 or 8 for gate/up; 2 or 4 for down). Each row's chain is the one-row kernel's, product by product, so a row has the same bits whatever other rows are in the launch and however they route. One-row slots keep the one-row kernels, untouched.c97d661c4) and, for the synthetic timing step only, dsv41 metal: launches of 6 to 16 rows run their FP8 projections on a matrix tile (stacked on #1720) #2973; this adds one commit that touches only the expert side and applies cleanly on dsv41 metal: the lane target, the prompt cache and the native server over Metal #1720 alone. One commit since review:10676dba8(the CHANGELOG line quotes the served result on both chips).atc overlap --json: not run yet.Why it is exact: each row's 32 fp4 by E4M3 (or bf16) products run as a sequential fma chain from zero in k order, then the same scaled fma, the same eight-way tree and the same epilogue as the one-row kernel. No value crosses between rows, and padding rows are computed but never written.
Checks on this head
10676dba8python3 tools/zig/lean_check.py, on the base and on this head: 345 and 345, 0 new (tools/pr_check.py0 errors, 0 warnings).b471c9479zig build test-dsv41-metal-allpassed on the M5 Max (40/40) and on the M3 Ultra (40/40).zig build time-dsv41-metal-synth(six synthetic layers at release widths, a 40-layer round, ms, base to this), rows on the same experts: 1 row 30.2 to 30.3, 2 rows 38.4 to 33.4, 3 rows 41.8 to 38.7, 4 rows 45.0 to 41.5, 5 rows 61.2 to 49.9, 8 rows 68.6 to 59.4, 16 rows 111.1 to 93.4, four slots of one row 45.6 to 41.9. Rows routed apart (about 18 distinct experts at 4 rows): 2 rows 39.5 to 36.6, 4 rows 50.3 to 46.6, 8 rows 75.5 to 65.2, 16 rows 116.0 to 96.3, four slots of one row 50.8 to 47.4.CHANGELOG.md), and per the order in DeepSeek-V4.1 on Metal: decode and prompt speed on M3 Ultra and M5 Ultra, token-exact (gate and order for the drafts) #2984 it is rebased onto the PR before it at landing.time-dsv41-metal-synthon the M3 Ultra, to confirm the pass sizes; run list in the comment.Receipt
b471c9479; the new commit is the CHANGELOG only.b471c9479on dsv41 metal: the lane target, the prompt cache and the native server over Metal #1720'sc97d661c4with dsv41 metal: launches of 6 to 16 rows run their FP8 projections on a matrix tile (stacked on #1720) #2973 under it;deepseek-ai/DeepSeek-V4.1-Flash-1017release weights at revision1bae0e1088993b9b82de08f59de5be5a34e42170; M5 Ultra 512 GB and M3 Ultra 512 GB; Zig 0.17.0.token_sha256equal to the stack tip in every cell on both chips, 0 failed, 0--alone/--serialmismatches; the contract's rows probe holds each row bit-equal to the row launched alone.bench_openai1.053 to 1.086 (M3 Ultra) and code 76.1 against 71.2 tok/s (M5 Ultra).Kernel contract
tf_dsv41_xp_upc_rows(replacestf_dsv41_xp_upc_multi) andtf_dsv41_xp_down_rows(replacestf_dsv41_xp_down_multi), each registered with every row on the same six experts against the float64 reference (the down is byte-equal to it). An equality probe checks a launch's activations, outputs and combine output bit-equal to each row launched alone, for same, disjoint and random routing at rows 1, 2, 3, 4, 5, 7, 8, 9 and 16, finite and with NaN and Inf planted: 54 cases, 0 differ.TF_CTL 6on each (rows read their pass's first row), both red at the shapes gate; the probe's own control is red too. AtTF_CTL 0the source is what ships.SUMMARYline and exit status, M5 Max: baseSUMMARY 95 kernels: 95 pass gates 1-4, 0 fail, 0 known findings; 190 controls: 190 red as planted, 0 missed; 0 limit problems; this headSUMMARY 97 kernels: 97 pass gates 1-4, 0 fail, 0 known findings; 192 controls: 192 red as planted, 0 missed; 0 limit problems. M5 Ultra atb471c9479, exit 0:--only dsv41SUMMARY 97 kernels: 97 pass gates 1-4, 0 fail, 0 known findings; 192 controls: 192 red as planted, 0 missed; 0 limit problems; full run (with the one-line fix from Kernel contract on tensor-unit Macs: the full run aborts with ControlEditMissing in the llama gemm_untransposed control since #2251 #3067) 758 pass, the same 57 known findings as base. M3 Ultra atb471c9479, exit 0:--only dsv4197 of 97 pass, 192 of 192 controls red; full run 547 pass, the same three knownnonfinitefindings as base. The new commit is the CHANGELOG.Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo (synthetic weights, and on the release weights in the receipts).--drafter, no slower than without it, greedy and sampled