Repository navigation
dsv41 metal: a multi-stream round's per-slot stages run as one dispatch each (stacked on #1720) - #2954
dsv41 metal: a multi-stream round's per-slot stages run as one dispatch each (stacked on #1720)#2954gilby wants to merge 22 commits into
Conversation
…over Metal target.zig is the lanes' verify target: a slot's prompt, its windows of rows for several slots in one launch, keeps and rewinds, and kept prompt states (the prompt cache) whose resumed rows the reply reports as cached_tokens. boot.zig loads a release checkpoint folder and entry.zig serves it as tensorfold-dsv41 (dsv41-metal-serve). Tests serve the tiny model through LaneHost: concurrent equals solo equals serial, drafted equals draft:false, cache hits equal fresh runs, seeds and cancels.
…for the next A shared round with the plain switch at it asked no drafts and ran no head pass (core Rule.plain_streams, threaded from the family's resolved setting: 0 on CUDA, so its rounds are unchanged), and the family's speculation no longer launches a pass for a round that wide (armSpec/speculate plainRound).
…teps and the forward try's
…stays within 600 lines
…nything else by name
…treams streams decode
…er holds a rejected draft's taps
…nd the next ask catches the drafter up
…, with its dispatches and barriers a layer time-dsv41-metal-slots records each layer's encode (dispatches, barriers) and the median wall time of a round over synthetic weights, at 1, 2 and 4 slots of one row and one slot of four rows, so the cost of the per-slot loops shows without the release checkpoint.
…ssor writes and decode attention A decode launch over several slots ran the window key (tf_dsv41_f_kv), the ratio-2 pool and the two scatters of its kept inputs, the FP4 round trip into the compressed rows, the index-key packing and the decode attention once a slot, because each slot's state lives in buffers of its own. Each of these now has a slot form (*_segs) that indexes a per-layer table of the slots' buffers (segs.zig, written as the layer is encoded, every buffer it reaches declared with useResource) by the row's slot (Meta.rowseg). The per-row body of every kernel moved into an inline function the per-slot kernel and its slot form both call, so a row runs the same instructions alone or beside other slots; launches of one slot keep the per-slot kernels. The pool's slot form copies the kv and score words it keeps in place of the two scatters. The attention picks its prefetch form when no slot has more than two rows. Contract: each slot kernel is registered as a replacement of its per-slot kernel, bit-equal to it launched a slot at a time at 1, 2, 3 and 4 slots of 1 to 17 rows (heads 4 and 64), NaN and infinities included, with a TF_CTL 6 control (every row reading or writing the first slot's state) that goes red at the shapes gate for all but the prefetch attention.
…idate blocks and top-512 An index-source layer of a launch over several slots ran the decode scorer, a barrier, the candidate blocks (at the candidate source), a barrier and the top-512 once a slot. With the layer's slot table the scorer (tf_dsv41_ix_scores_dec_segs) runs every row over its slot's keys, key count and first position, the candidates (tf_cand_segs) over its slot's key count, leaving a slot with no keys as the per-slot path does, and the unchanged top-512 over all rows; one barrier between stages instead of one a slot. The merged path keeps every row's scores at a fixed stride (the scratch's key width), so the top-512 reads each row's first `visible` scores exactly as before; launches of one slot keep the per-slot code. As for the attention side, each kernel's row body is an inline function both forms call. Contract: both registered as replacements, bit-equal to the per-slot launchers a slot at a time at 1 to 4 slots of 1 to 17 rows, ratios 1 and 2, with and without a candidate mask, NaN and infinities included, each with a TF_CTL 6 control (the first slot's keys or key count for every row) red at the shapes gate.
|
Thanks @gilby. This is careful work. Each per-slot body moved unchanged into a function both kernels call, every slot buffer is reached through its own table entry, ragged slots stop at their own key counts while the top-512 only reads each row's visible keys, and the contract plants a cross-slot control on the new kernels. No need to split it: the two halves share the slot table. Asks:
Two pointers, not blockers. These five all sit on #1720 and touch the same few fields, so they'll land in this order, each rebased on the one before: #2951 (the profiler, to read the others' receipts), #2950 (opt-in), #2954, #2953 (on #2951, so
|
|
Real-weights receipt. One M3 Ultra 512 GB,
Four streams of greedy code go from 83.8 to 83.9 tok/s to 86.1 and 86.3, and three from 72.2 to 73.3 and 73.5; the rest is level. The two 0.98 cells at one stream are against a base mean lifted by its first run (47.1 against 45.1 and 45.5 in the later two); one-stream rounds run the old kernels here, and this branch read 44.8 and 45.1 on code and 52.2 on sampled code, the same as the later base runs. So the gain is small, as the description expected: up to about 3% at four greedy streams, where routing is shared and these stages are a larger share of the round. An M5 Ultra run follows. I'd understand if this is not worth eight kernels on its own; it is the first half of making rows cheap, and the FP8 rows on the matrix tile are the larger half. |
…sources call segs.table declared every buffer a layer's slot kernels reach with its own useResource call, five to seven a slot a layer. segs.declare now hands every region buffer of the launch's slots (each layer's window, and the pools, index keys, carries and kept inputs of the KV sources) to the encoder in one useResources:count:usage: call, made by forward.encode and by forward.run for each command buffer before its first layer; a launch of one slot declares nothing, as before. ComputeEncoder gains useResources beside useResource. time-dsv41-metal-slots also reports the median host encode of a whole round (a command buffer encoded, never committed).
… and the per-slot bodies held to the base The slot table entry was declared in each of the five kernel sources against one Zig struct. It now lives in deepseek_v41/segs.h, which sources.zig puts before fused, compress, csa2_attn, index and cand (as kv_pages.h is shared); the contract compiles the header with a kernel that writes sizeof and every field offset and checks them against segs.zig's Seg, which the contract probes now use too. After the gates the contract also launches each per-slot kernel that now calls a moved body (tf_dsv41_f_kv, the pool, round trip and key packing, tf_dsv41_attn_dec in both prefetch forms, tf_dsv41_ix_scores_dec and tf_cand) and the base's body of it on the same inputs, random and with non-finite plants, at every probe shape, and requires the same bytes: 164 cases. The base's kernels are kept only in the test tree (tests/contract/dsv41_base), appended to today's source under tf_base_ names, so they share its helpers and nothing ships twice.
|
Thanks. Head is now
Your two pointers are done. The slots' buffers are declared once per encoder in one One thing I could not produce: the contract with no At this head on an M5 Max: |
|
Thanks. Head is now Pointers. The slots' buffers are now declared once per encoder in one Your item 2, the one-stream path: you are right that the served tests could not show it, and most per-slot registrations are float64-bounded. So the contract now keeps the stack tip's bodies of Item 1, the contract on the Ultras. M5 Ultra at M5 Ultra receipt at
Item 3: At |
|
Thanks @gilby, and for the M5 Ultra receipt. Both pointers are in at On item 3, agreed. With #1720's copy-window fix at Asks:
|
|
M3 Ultra, 512 GB: the series is being re-measured now under the noise rule in #2984 (one discarded first run, base on both sides, both-under-both test); it follows in this thread.
Summary: token hashes equal to base in every cell of every block. At one stream this head is level with base (mean over mean 0.999 to 1.000 with default drafts, 1.001 to 1.002 with Cells under 0.98 of base: none. Kernel contract M5 Ultra, 512 GB (applegpu_g17, tensor units true), A second build of the same head in a second tree gives (verbatim): M5 Ultra full run ( M3 Ultra, 512 GB (applegpu_g15, tensor units false; macOS 26.6.2 (25G83), Metal compiler 373.7), final contract run, both runs verbatim. M3 Ultra full run ( For comparison, the base tree on the M3 Ultra (the tree at and its The full run needs a rotary sweep receipt passed with Device tests ( Build exit status 0 for both builds of the head; 1135 PASS lines (second build: 1135); no line starting with FAIL. The log has no M3 Ultra device tests for this head: not run. Decode, Order as run: base 1, this 1, this 2, base 2 (the full block order is in the header). Token hashes equal in every cell across the four arms: yes; failed / alone / serial mismatches: 0. All 12 cells are reported.
Block A, Token hashes equal in every cell across the four arms: yes; failed / alone / serial mismatches: 0. All 12 cells are reported.
Block B (first build of the head), default drafts, 1 to 4 streams Order as run: base 1, this 1, this 2, then two runs of a different change (not reported here), then base 2. Token hashes equal in every cell: yes; failed / alone / serial mismatches: 0. All 16 cells are reported.
Token hashes equal to base in all four cells across the four arms: yes.
Method as run: each cell is a fresh server started just before it; a thermal-pressure check ( Not run: M3 Ultra device tests; cold-prompt cells for this head; Raw JSON, server logs and driver scripts: https://gist.github.com/gilby/c93f79e41189fbfcbc3674117fa12855 |
|
Thanks @gilby. The M5 Ultra receipt at The full-run abort on the M5 Ultra ( These M5 Ultra series came before the discarded first run in #2984's noise rule. Base ran first in each, so any first-run lift favours base. We'll take them as they are. Still open:
|
|
M5 Ultra, 512 GB: the full kernel-contract SUMMARY that was missing above. The full run stops at registration on chips with tensor units since #2251 (#3067); with the one-line fix proposed there applied to this tree's copy, it completes on this head ( The base tree And The 57 known findings are the same 57 lines on the base tree and on this head, and none is a DeepSeek-V4.1 record: 41 are the #1009 masked-value entries of other families' attention kernels that this chip registers (Gemma 4, Spark2.5, MiniMax M3, Llama, GLM, the step kernels), and the rest are those families' prompt-attention entries, the Full logs for the four trees and the applied one-line diff: https://gist.github.com/gilby/fb5f4ae45e3089e7f742c17753623740 |
|
M3 Ultra, 512 GB: the receipt promised above, run on 2026-11-13 under the noise rule in #2984 (one discarded first run per series, base on both sides, both-under-both test).
Summary: this head gains at 3 and 4 streams and is level elsewhere. At 3 and 4 streams (8 Cells under 0.98 of base: none.
Order as run: base 1, this 1, this 2, base 2. Ratios: this 1 over base 1, this 2 over base 2, and mean over mean. Verdict by the #2984 rule: slower only if both this runs are under both base runs and the mean gap is over 1%, faster the mirror image, otherwise within noise. Verdicts: 6 faster, 0 slower, 14 within noise, out of 20 cells.
End level key: Nom = Nominal, Mod = Moderate, Heavy = Heavy, Exactness: token hashes are equal across all four runs in every one of the 20 cells (all reps of each cell compared), with 0 failed requests and 0 Discarded, retried and flagged cells
Method as run: each cell is a fresh server started after the previous one was stopped and a 60 s settle, and a cell starts only when the powermetrics thermal sampler reports Nominal (the driver polls every 30 s); the level is recorded again at the end of the cell. One run per series is discarded and not in any table, base is on both sides, and the series are per stream count and temperature ( Not run: the checkpoint's pinned revision, the macOS version and the toolchain versions are not recorded; kernel-contract runs are not part of this series' data. Raw JSON, server logs and driver scripts: https://gist.github.com/gilby/1da4b87d59800ff945461fb7b81efc52 |
…e a multi-slot layer launch segs.table declares nothing since the useResources commit, so an encoder that runs a layer over several slots has to call segs.declare first. forward.encode and forward.run do; real_test's timeLaunch, which runs one layer over four slots, did not, so its slot kernels read buffers never declared to the encoder. It now declares them on its pass's encoder. The package exports segs so the tests can.
|
New head:
Receipts: the M5 Ultra and M3 Ultra receipts and both chips' contract lines above were taken at Your other points:
At |
What this changes
On Metal, a DeepSeek-V4.1 decode round with several streams ran one dispatch a stream for the window key, the KV source writes, decode attention, and the indexer's scorer and candidates, with the indexer taking a barrier a stream. With this change a round with two or more streams runs each of those stages as one dispatch over a per-layer slot table, and the indexer takes the same barriers as a one-stream round. Each kernel's per-row body is moved unchanged into an inline function that both the per-slot kernel and the new
*_segskernel call, so a row's arithmetic and order are the same and a row has the same bits alone or with others. One-stream rounds run exactly the old code.a43ad7b98); this adds three commits: a slot timing step on synthetic weights, the attention side, and the indexer side (which needs the attention commit's slot table). Happy to split it into two pull requests if you prefer. Two commits since review:cd32ace02(the release-layer test declares its slots' buffers before its four-slot launch) andc469d807b(one CHANGELOG line with the served result on both chips).atc overlap --json: run on this head against main; incomplete (its time and pull-request limits). Ten textual conflicts with other open pull requests, each in one file:CHANGELOG.md, the settings tables inplatform/environment.ziganddocs/environment.md, or the coverage text indocs/kernel-contract.md(idol: Idol's denoiser on CUDA (stacked on #1766) #1767, Vulkan: one submission per batch of dispatches, 30% to 56% less run time on an M1 with the same logits (stacked on #2015) #2016, Vulkan: qmm_q4_tile_rows computes four rows and two columns of a prompt chunk per thread, 3.6% to 33% less run time on an M1 with the same logits (stacked on #2016) #2017, Ling-3.1-flash on CUDA, on one or two GB10s #2340, server: --execution-witness serves a read-scoped snapshot of what runs, or why it is unknown #2901, server: "execution_witness": true binds a reply to the witness at admission and at its end, with its prompt, resolved request and tokens #2958, flashnext cuda: at --tp 2 a prompt pass reduce-scatters its routed experts' slots by rows and gathers their sums #2977, flashnext cuda: at --tp 2 on one host, collectives run on a blocking split of the nonblocking communicator #2978, flashnext cuda: the opt-in --tp 2 decode graph (TF_FLASHNEXT_DECODE_GRAPH=1) serves again instead of failing its first plain round #2980, flashnext cuda: the opt-in --tp 2 decode graph takes the dense split and keeps its graph across drafted verifies #2982). No shared source logic.What to expect: the dispatches inside these loops go from 114 a stream to 114 a round, and the indexer's barriers from 9 a stream plus 8 to 17. The attention-side loops had no barriers between streams, so only the indexer's really held the GPU up; my estimate for the release model is a few percent at four streams, and the receipt will say.
Checks on this head
c469d807bpython3 tools/zig/lean_check.py, on the base and on this head: 345 and 345, 0 new (tools/pr_check.py0 errors, 0 warnings).zig build test-dsv41-metal-all: 9/9 steps, 38/38 tests;zig build test-dsv41-metal-realbuilds (its tests skip withoutTF_DSV41_REAL);zig build test: 13000 of 13109 passed, 108 skipped, 1 failed (dashboard_test, which fails the same way on this base and was fixed on main by tests: the dashboard page's test runs on every OS, and expects only tiles the page has #2946).CHANGELOG.md), and per the order in DeepSeek-V4.1 on Metal: decode and prompt speed on M3 Ultra and M5 Ultra, token-exact (gate and order for the drafts) #2984 it is rebased onto the PR before it at landing.timeLaunchwith the release weights aftercd32ace02(it needs the checkpoint; it is a timing test, not served code).Receipt
--no-drafts. M3 Ultra (DeepSeek-V4.1 on Metal: decode and prompt speed on M3 Ultra and M5 Ultra, token-exact (gate and order for the drafts) #2984 rule): 3 and 4 streams 1.008 to 1.028, 6 of 8 cells faster, the rest within noise, none slower; 1 and 2 streams 0.999 to 1.009. Receipts: https://gist.github.com/gilby/c93f79e41189fbfcbc3674117fa12855 (M5 Ultra), https://gist.github.com/gilby/1da4b87d59800ff945461fb7b81efc52 (M3 Ultra), measured atc6d2413c7;cd32ace02touches a test and the package's export list only,c469d807bthe CHANGELOG only.c6d2413c7on dsv41 metal: the lane target, the prompt cache and the native server over Metal #1720'sa43ad7b98;deepseek-ai/DeepSeek-V4.1-Flash-1017release weights at revision1bae0e1088993b9b82de08f59de5be5a34e42170; M5 Ultra 512 GB and M3 Ultra 512 GB; Zig 0.17.0; tests on an M5 Max.token_sha256equal to the stack tip in every cell of every block on both chips, 0 failed, 0--alone/--serialmismatches; the contract holds every per-slot kernel byte-equal to the base's body.bench_openai(1 stream) 0.999 to 1.001 on both chips.Kernel contract
tf_dsv41_f_kv_segs(tf_dsv41_f_kv),tf_dsv41_c_pool2_segs(pool2 and its two scatters),tf_dsv41_c_fp4rt_segs,tf_dsv41_c_fp4pack_segs,tf_dsv41_attn_dec_segs_bf16and its prefetch form,tf_dsv41_ix_scores_dec_segs,tf_cand_segs. Each is checked bit-equal to its per-slot kernel launched a slot at a time: the attention side at 1, 2, 3 and 4 slots and rows 1, 2, 4, 16 and 17, heads 4 and 64; the indexer at both ratios, with and without the candidate mask; gate 1 the same with NaN and Inf planted. The per-slot kernels keep their float64 registrations and now run the shared body.TF_CTL 6(every row takes slot 0's state) on each new kernel except the prefetch attention; seven controls, all red at the shapes gate. AtTF_CTL 0the source is what ships.SUMMARYline and exit status, M5 Max: baseSUMMARY 89 kernels: 89 pass gates 1-4, 0 fail, 0 known findings; 186 controls: 186 red as planted, 0 missed; 0 limit problems; this headSUMMARY 97 kernels: 97 pass gates 1-4, 0 fail, 0 known findings; 193 controls: 193 red as planted, 0 missed; 0 limit problems. M5 Ultra atc6d2413c7, exit 0:--only dsv41SUMMARY 97 kernels: 97 pass gates 1-4, 0 fail, 0 known findings; 193 controls: 193 red as planted, 0 missed; 0 limit problems; full run (with the one-line fix from Kernel contract on tensor-unit Macs: the full run aborts with ControlEditMissing in the llama gemm_untransposed control since #2251 #3067) 758 kernels pass, 57 known findings, the same 57 as base. M3 Ultra atc6d2413c7, exit 0:--only dsv4197 of 97 pass, 193 of 193 controls red; full run 547 pass, the same three knownnonfinitefindings as base.cd32ace02andc469d807bchange no kernel or record.Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo (synthetic weights, and on the release weights in the receipts).--drafter, no slower than without it, greedy and sampled