Repository navigation
Conversation
…over Metal target.zig is the lanes' verify target: a slot's prompt, its windows of rows for several slots in one launch, keeps and rewinds, and kept prompt states (the prompt cache) whose resumed rows the reply reports as cached_tokens. boot.zig loads a release checkpoint folder and entry.zig serves it as tensorfold-dsv41 (dsv41-metal-serve). Tests serve the tiny model through LaneHost: concurrent equals solo equals serial, drafted equals draft:false, cache hits equal fresh runs, seeds and cancels.
…for the next A shared round with the plain switch at it asked no drafts and ran no head pass (core Rule.plain_streams, threaded from the family's resolved setting: 0 on CUDA, so its rounds are unchanged), and the family's speculation no longer launches a pass for a round that wide (armSpec/speculate plainRound).
…teps and the forward try's
…stays within 600 lines
…nything else by name
…treams streams decode
…er holds a rejected draft's taps
…nd the next ask catches the drafter up
…ers per row and dispatch class Six synthetic layers of the release's widths (one of each attention kind, few experts) are timed at 1-16 rows of one slot and at 4 slots x 1 row, each dispatch alone and the layer on the serving encoder, and split into FP8 rows, experts, attention and glue, with a 40-layer round weighted by kind. timing_real's report takes a launch (context and spans) so both harnesses share it.
…x tile, bit-equal to the row kernel Launches of 6 to 16 rows run the dense FP8 projections on fp8_tile.metal: 8 or 16 columns a threadgroup, K split over up to 16 simdgroups in aligned power-of-two runs of groups, each 32-wide group summed from zero in four 8-wide MMA steps in k order (the prompt tile's group root), the groups' roots and the slices' roots in the same zero-padded pair tree the row kernel uses. Every row's bits are the row kernel's one row a launch; 1-5 rows keep the row kernel (chosen by row count only), since below 6 rows it reads the weights faster. Contract: six registrations (f32, bf16, a forced 16-column form, one simdgroup, the row kernel one row a launch as replaced baseline, the release wkv fixture), four planted controls, gate-5 records with M1/M2 limits, and an equality probe over every model FP8 shape at 1-16 rows, every form, the 16/17 seam and a batch with NaN and Inf planted. Device tests: the layer's kernel by row count, and four slots of two rows each in one launch against each alone.
…f its rows, bit-equal to one row a launch The decode gate/up and down for an expert's slot of 2-16 rows ran Kimi's four-row passes: two rows paid for four rows of arithmetic, five rows for two passes of four, each rereading the weights. The new tf_dsv41_xp_upc_rows and tf_dsv41_xp_down_rows decode each 32-group of an expert's MXFP4 weights once per pass and apply it to every row in it (passes of 2, 4, 6 or 8 rows for gate/up, 2 or 4 for down, rows past the slot repeating its last row unwritten), reading the activations straight from device memory. Each row's chain is the one-row kernel's: 32 products in k order from zero, then its group's scale, then the same tree and epilogue, so a row's bits are the same alone or with any other rows and routing. Slots of one row keep tf_dsv41_xp_upc_one/down_one unchanged. Contract: registrations with every row on the same six experts (gate/up coded and down, gates 1-4 against the oracle), one planted control a kernel (rows reading their pass's first row, red at shapes), gate-5 records for both launches, and a probe holding a launch's act, out and combine output bit-equal to each row launched alone at 1-16 rows on the same, disjoint and random experts, with NaN and Inf planted, plus its own control. Device test: the same equality at 2-16 rows.
…tion blocks a seat TF_DSV41_POOL_TOKENS is checked against --parallel and --context and cuts the per-request window to the pool. A run is placed first fit; when the free blocks are split the live runs slide down in start order, so a request waits only while too few blocks are free in all. Blocks a request wrote are zeroed before the next run reads them. Host tests cover fit, wait, refuse, reuse, sliding, dirty blocks and the setting; test-dsv41-metal-host runs the package's host checks without a device run.
…the shared pool's runs With a pool set, each KV source has one latent buffer and one index-key buffer for every slot, and a slot binds them at its run's first row: the kernels see the same contiguous per-stream rows at a buffer offset, so no kernel changes. A prompt takes a run of its ids, its max new tokens and one window of drafts (admit), or LaneMemory while too few positions are free; the run goes back on release or a failed prompt. The window ring, carry, taps and prompt tail stay per slot. Unset, every slot allocates and binds as before.
…, a working-set check, the startup line The load reads the setting, refuses it by name when it is out of range, cuts --context to the pool when the pool is smaller, and refuses a pool whose buffers and slot rings pass the GPU's working set before any of them is allocated. The startup line names the pool's positions, bytes a position and MiB; unset it is the old line, which a host test checks. test-dsv41-metal-pool (a device step of its own) drives five requests over eight blocks so live runs slide down and a full pool waits, then four requests through LaneHost that cannot all fit, each reply against unpooled serial decode.
… not the first sixteen A slot past the sixteenth reserved the whole window because its admit had nowhere to go. The target now keeps one entry a slot; test-dsv41-metal-pool checks slot 19 of 20 takes a run sized by its max new tokens.
…ored after runs slid down Host tests apply each plan's moves and zeroes to a byte buffer with the device side's own copy and zero: a run overlapping its own destination slides down intact, a moved seat's writes are counted from its new place, a refused take changes no run or dirty block, and 4,000 random takes and gives keep every live seat's bytes while each new run reads zeros. test-dsv41-metal-pool gains the prompt cache: runs slide under live requests and a kept state restored into a new run, all replies equal serial.
…es, including its default and thinking reserve
Owner
|
Thanks @gilby. The default-off pool, compaction and cached-restore checks look sound. All 32 host checks pass, with no new lean findings. This remains stacked on #2983; the pool code applies over current main. We will run the device check on our side for pool pressure, moved-run restore and late slots before landing. No action needed from you.
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
DeepSeek-V4.1 on Metal: the new opt-in setting
TF_DSV41_POOL_TOKENSlets all slots share one pool of positions for the compressed latents and index keys, instead of each slot holding--contextpositions. With--parallel 4 --context 1048576 TF_DSV41_POOL_TOKENS=1048576the four slots share 1,048,576 positions and any one request may still use the whole window, so the server holds one 1M request or four 256K ones in 8.6 GB less GPU memory. Unset (the default) the server allocates and logs exactly as on the base.b471c9479when this was built; dsv41 metal: decode experts decode an expert's weights once for all of its rows (stacked on #1720, #2973) #2983 has since gained10676dba8, a CHANGELOG-only commit, which this branch does not yet contain). This pull request adds the 9 commitsb471c9479..0b285e817:068bfd7cbdsv41 metal: the shared pool's allocator, contiguous runs of 256-position blocks a seat42de9b15adsv41 metal: slots read their compressed latents and index keys from the shared pool's runsaff5426b9dsv41 metal:TF_DSV41_POOL_TOKENSat boot: the window cut to the pool, a working-set check, the startup line655a62ef5docs:TF_DSV41_POOL_TOKENSon Metal in the environment table and the changeloga1a56ee84dsv41 metal: each slot's admitted max new tokens kept for every slot, not the first sixteenbab88ced7dsv41 metal: the pool's moves checked on bytes, and a kept state restored after runs slid down03b4cc564docs:TF_DSV41_POOL_TOKENSreserves the reply limit the server enforces, including its default and thinking reserved86619e8achangelog:TF_DSV41_POOL_TOKENS's line quotes the served result on both chips0b285e817changelog: the line names this pull requestatc overlap --json: run on this head at 2026-11-13 23:56 UTC against mainf8dacd777. 65 entries, none with another contributor's work in conflict: the entries are this stack's own pull requests (dsv41 metal: the vision tower's kernels under the kernel contract #1721, dsv41 metal: image prompts #1722: untested, they change other files of the same family), glm cuda: a logit gather broadcasts one block a rank, so TP=3 prompt_logprobs past ~100 tokens no longer hangs #1571 (untested) and my own local worktrees of this stack.atc check 2984shows the issue claimed by this session's account with no other pull requests on it.How it works:
max_tokens, or the server's default of 4096 when the client omits it, plus--thinking-reservewhen thinking) and 16 rows of drafts, cut to the window. While too few positions are free it returnsLaneMemory, so LaneHost requeues it at the head and admits nothing new until a stream ends.--parallelx--context, and when the pool and slot rings pass the GPU working set. The window is cut to the pool when the pool is smaller. The startup line names the pool's positions, bytes a position and MiB.Checks on this head
0b285e817. Every receipt below was measured at03b4cc564; the two commits after it change the CHANGELOG line and no code, sogit diff 03b4cc564..0b285e817touchesCHANGELOG.mdonly.python3 tools/zig/lean_check.py, on the baseb471c9479and on this head: 345 problems on both, 0 new (tools/zig/lean_new.py). The 345 are inherited.zig build test-dsv41-metal-host(new step, no device): 32 pass, 0 fail, 0 skip. The same package tests on the base, run through the same step added to a copy of the base tree, are 13 pass: the allocator, setting, byte-level move and alignment tests are new and cannot build on the base becausepool.zigdoes not exist there.zig build test-dsv41-metal-pool(new step, device; not part oftest-dsv41-metal-all): 4 PASS lines on M5 Ultra, 512 GB and 4 on M3 Ultra, 512 GB at this head, and the build reports 5 tests passed, 0 failed, 0 skipped (M3 Ultra log). The 4 lines: 5 requests over 8 blocks with runs sliding down and a full pool waiting, 0 replies differ from unpooled serial; 4 requests of 3 blocks over 8 through LaneHost, 0 differ; slot 19 of 20 takes a run of 2 blocks sized by its max new tokens (2 expected, 6 the whole window); with the prompt cache on, 0 of 6 replies differ from serial, runs slid down and a kept state restored into a new run. The M5 Ultra log in the gist shows the same 4 PASS lines; its exit status was not recorded.zig build test-dsv41-metal-all(device): 1,138 PASS lines and no failure on this head and 1,138 on the base, on both chips.zig fmt --check build.zig zigandgit diff --check b471c9479..03b4cc564: pass.zig build test test-goldenon a development Mac: the same single failure on the base and on this head,dashboard_test"dashboard page polls the public stats snapshot without private details" (failed without output; outside the DeepSeek Metal package, so inherited here). Neither run printed a build summary, so no suite-wide pass count is quoted; CI on the pushed head decides.families/deepseek_v41/metalonly); other Apple chips (M1 to M4, M5 Max); a prompt-cache run at the pooled capacity beyond the device test and the 45K resume below.Receipt
b471c9479against this head03b4cc564, same machine, checkpoint and flags on each chip. The base is the head dsv41 metal: decode experts decode an expert's weights once for all of its rows (stacked on #1720, #2973) #2983 had when this was built.families/deepseek_v41/metal). CUDA's use ofTF_DSV41_POOL_TOKENS(its paged pool) is untouched.TF_DSV41_POOL_TOKENS: already declared for CUDA inplatform/environment.zig(scope dsv41, u64), read through that registry on Metal. Default unset or blank: each slot holds--contextpositions as before. Accepted: a whole number from 256 through--parallelx--context; any other text stops the start with the setting's name (refusal texts below). Documented indocs/environment.md(the row now covers Metal); the CHANGELOG Unreleased line is quoted at the end.03b4cc564and baseb471c9479; Zig 0.17.0 (.zig-version); the Metal sources are compiled at load by the system toolchain, as on the base; Mac Studio M5 Ultra, 512 GB (macOS 27.2.1 (26B131), Metal compiler 383.1) and Mac Studio M3 Ultra, 512 GB (macOS 26.6.2 (25G83), Metal compiler 373.7), the OS and compiler lines as recorded in the earlier receipts on this stack (these logs do not repeat them; Xcode is not recorded); checkpointdeepseek-ai/DeepSeek-V4.1-Flash-1017revision1bae0e1088993b9b82de08f59de5be5a34e42170, release weights as published, with its Engram tables; server--context 1048576 --parallel 4 --prompt-cache-gib 0(--prompt-cache-gib 8only in the resume test),--vision --no-update-check,TF_EXPERIMENTAL=deepseek, one fresh server per cell. Server binaries (sha256 prefix): M5 Ultra this head44c2dc372d97, base58850a7193f7(and730ee1cd7f6dfor655a62ef5); M3 Ultra this head33ad0738dce0, base4c65c90c6a57. Commands and drivers are in the gist.03b4cc564on both chips: startup, exactness, cold prompts, the resume test, both speed series and the device tests; on M5 Ultra also the capacity cells (900K, 4 x 250K, staggered, 1.04M); on M3 Ultra the scaled-down capacity cells. At the previous head655a62ef5, on M5 Ultra only: the 10-server cold-prompt alternation at 32K and 64K and the two-waves-of-four-200K and no-max_tokenscells. The three commits between them are per-slot max-new sizing (a1a56ee84, which changes nothing below the seventeenth slot, so four slots take the same path), byte-level host tests plus a device-test case (bab88ced7; its only non-test change makes two pool helperspub) and docs (03b4cc564). The655a62ef5cells are quoted as measured there and were not re-run.tools/bench_concurrent.py --alone --serialandtoken_sha256(all token hashes compared across base, this head unset and this head pooled; failed requests and--alone/--serialmismatches counted over every run):bench_concurrent --tokens 256 --reps 3 --temperatures 0,1 --levels 1,4 --alone --serialon M5 Ultra and the same with--tokens 128on M3 Ultra: base, unset and pooled, 8 cells (code and chat, T 0 and 1, 1 and 4 streams). The 60 hashes of each arm are equal across the three arms in all 8 cells on both chips; 0 failed, 0--aloneand 0--serialmismatches.token_sha256equal across base, unset and pooled on both chips (table below).--prompt-cache-gib 8(turn 3 resumed from 30,202 cached tokens on M5 Ultra and 30,219 on M3 Ultra) against the same final prompt sent cold to a fresh server. The reply hashes are equal (M5 Ultra1b66eb0a7bb16026, M3 Ultrada47341947d2eed7), unset and pooled give the same hashes in all three turns, and every reply holds the right needle.runs slid down: true, twice).tools/bench_openai.py --tokens 256 --reps 3 --temperatures 0,1andtools/bench_concurrent.py --tokens 128 --reps 3 --temperatures 0,1 --alone --serialat 1 to 4 streams (M5 Ultra) or 1 and 4 streams (M3 Ultra). Two series on each chip: A,TF_DSV41_POOL_TOKENSunset; B, pooled with 1,048,576. Per series: one discarded run, then base, this, this, base, each on a fresh server per piece (M5 Ultra) or a fresh server per arm (M3 Ultra); a cell starts at Nominal thermal pressure after a settle and its end level is recorded. Noise rule of DeepSeek-V4.1 on Metal: decode and prompt speed on M3 Ultra and M5 Ultra, token-exact (gate and order for the drafts) #2984: slower only when both this runs are under both base runs and the mean gap exceeds 1% (faster is the mirror image; for first-token times the directions swap). Columns are tok/s (bench_openaidecode median,bench_concurrentaggregate) or seconds for the cold rows; "this / base" is the mean over the mean.--alone/--serialcount is 0 in every cell.max_tokenscell and one M3 Ultra reference cell, each named where it appears.M5 Ultra, 512 GB, series A (unset against base):
bench_openaifibonacci-raw, T 0bench_openaigpu-chat-no-think, T 0bench_openaifibonacci-raw, T 1bench_openaigpu-chat-no-think, T 1M5 Ultra, 512 GB, series B (pooled against base):
bench_openaifibonacci-raw, T 0bench_openaigpu-chat-no-think, T 0bench_openaifibonacci-raw, T 1bench_openaigpu-chat-no-think, T 1M3 Ultra, 512 GB, series A (unset against base):
bench_openaifibonacci-raw, T 0bench_openaigpu-chat-no-think, T 0bench_openaifibonacci-raw, T 1bench_openaigpu-chat-no-think, T 1M3 Ultra, 512 GB, series B (pooled against base):
bench_openaifibonacci-raw, T 0bench_openaigpu-chat-no-think, T 0bench_openaifibonacci-raw, T 1bench_openaigpu-chat-no-think, T 1End level key: Nom = Nominal, Mod = Moderate, Heavy = Heavy. The cold rows are the measured request on a fresh server after one warm-up request (M5 Ultra: a different prompt of the same length; M3 Ultra: a 2,048-token prompt), both cold with
--prompt-cache-gib 0: cache statecoldand 0 cached tokens in every measured request. The token hashes in them are equal across all four runs.ttft_sfromtools/bench_cache.py --require-cold --tokens 64 --repeats 1, one run per arm (single runs: no verdict is drawn from them; the series above carry the verdicts):330cd07dd3cfb2da1b2b327d7cfb5e60de47330cd07dd3cfb2da1b2b327d0e3f2ebade35On M3 Ultra the 32,768-token cells ended Heavy on the first try, were discarded and re-run once; the second runs ended Moderate and are the ones shown (the discarded first tries are in the gist). Longer cold prompts, M5 Ultra at the earlier head
655a62ef5: 10 fresh servers in the order base, pooled, unset, pooled, base, unset, pooled, base, unset, pooled, the first token of the measured cold request, one hash across all ten:The mean gaps are +0.97% (unset) and +0.10% (pooled) at 32K and -1.12% and -1.29% at 64K, so the rule gives not slower for both. The first two 64K servers ended Heavy and are kept (one request each).
03b4cc564:03b4cc564)max_tokens100max_tokens100) and two 150K (max_tokens2000, a long numbered-list reply) at once, then a 500K sent when the 300K finishedmax_tokens(the server default of 4096 applies)At
655a62ef5:655a62ef5)max_tokens100); the second wave sent when the first finishedmax_tokensM3 Ultra, 512 GB, scaled down (its capacity cells were cut for time, and the four-prompt case was not run there):
03b4cc564, scaled down)max_tokens100) and two 60K (max_tokens1500) at once, then a 250K sent when the 150K finishedThe 900K and 1.04M requests fit a pool that is a quarter of what four whole windows take. The four 250K prompts all answer correctly because requests prefill one at a time as before, so the later ones wait in line.
ioreg"Alloc system memory"), this head: M5 Ultra base 331,405,393,920 B, unset 331,405,410,304 B, pooled 322,817,785,856 B (8,587,608,064 B less; (4 x 1,048,576 - 1,048,830) positions x 2,730 B agrees to within 0.5 MB). M3 Ultra, read when the server came up: base 330,621,255,680 B, unset 330,621,272,064 B, pooled 322,033,614,848 B (8,587,640,832 B less); read 20 s later: 330,621,272,064 B, 330,705,911,808 B and 322,537,652,224 B. Peak allocation during the capacity cells is in the tables above (pooled, 323.2 to 323.7 GB). M5 Ultraphys_footprintafter load: 31 GB base, 31 GB unset, 23 GB pooled.info: dsv41 metal: 40 layers, 41 shards (283238 MiB mapped, 7789 MiB copied), 4 slots of 1048576 positions, DSpark on. Pooled:info: dsv41 metal: 40 layers, 41 shards (283238 MiB mapped, 7789 MiB copied), 4 slots sharing a pool of 1048830 positions (TF_DSV41_POOL_TOKENS=1048576; a request up to 1048576; 2730 bytes a position, 2730 MiB), DSpark on.error: dsv41 metal: TF_DSV41_POOL_TOKENS=100 is less than one block of 256 positions; give at least 256, or unset it for a context window a seatanderror: dsv41 metal: TF_DSV41_POOL_TOKENS=4194305 is more than --parallel 4 x --context 1048576 = 4194304 positions; every seat would hold its own window: lower it or unset it, each followed bytensorfold: the DeepSeek-V4.1 Metal engine cannot load <checkpoint path> (BadPoolTokens). The working-set refusal has host-test coverage only.Known limits:
max_tokensplus 16 rows. A client that omitsmax_tokensgets the server's default of 4096 (plus the thinking reserve when thinking) and reserves that much; the 1.04M request and the two 100K requests in the capacity tables were sent withoutmax_tokens.runs slid down: trueis the direct evidence that runs move under live requests.CHANGELOG, Unreleased, as it stands in the diff: "DeepSeek-V4.1 on Metal:
TF_DSV41_POOL_TOKENSlets the slots share one pool of positions instead of--contexteach, so four slots can share 1,048,576 positions while any one request still uses up to--context; a request holds its prompt,max_tokensand one window of drafts, waits while too few positions are free, and its tokens equal an unpooled run; unset, memory and startup lines are as before."Kernel contract, only when a GPU kernel is new or changed
No GPU kernel is new or changed, so there is no contract record to post.
zig/kernelshas the same git tree on the base and this head (git rev-parse b471c9479:zig/kernels 03b4cc564:zig/kernelsprintsa9cd42ebfbf9a852ce4ef6d50ea1c35381877b10for both). The DeepSeek Metal libraries are compiled at load from that unchanged source text with unchanged options, so the compiled kernels are the same. Only the buffer offsets passed at bind time differ, which every encoder already passed.Checklist
The title and description match the head commit, including the receipt and the stack.
New tests fail before the change (they do not build on the base:
pool.zigdoes not exist there) and pass after it. The device step needs a Metal device and is its own named step, not part of the defaultzig build test.Comments and docstrings are one line. No measurements or history in the source.
No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.
Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
Drafted output equals the base's (token hashes equal in every cell with DSpark on), a resumed prompt equals a fresh one, and concurrent equals solo (
--alone --serialcounts 0).No precision traded for speed. No sums change: kernels, their inputs and their order are unchanged; only where a row lives in a buffer differs.
Decode and prompt processing are no slower than main at one stream and the widest admitted (0 slower cells of 70 in four series, plus cold prompts to 64K); drafts are on in every cell, so no drafter-off comparison applies.
Raw receipts (JSON, server logs, test logs, drivers) for both chips:
M5 Ultra part 1 (with the file list) https://gist.github.com/gilby/a991a53176c009468f9cda435b303d40 , M5 Ultra part 2 https://gist.github.com/gilby/798dc2033168266f5e1940de7782769b , M3 Ultra https://gist.github.com/gilby/065a78a4757ee72203f052c22a7c9d85 . Empty files named in the list are left out.