Repository navigation
Conversation
…over Metal target.zig is the lanes' verify target: a slot's prompt, its windows of rows for several slots in one launch, keeps and rewinds, and kept prompt states (the prompt cache) whose resumed rows the reply reports as cached_tokens. boot.zig loads a release checkpoint folder and entry.zig serves it as tensorfold-dsv41 (dsv41-metal-serve). Tests serve the tiny model through LaneHost: concurrent equals solo equals serial, drafted equals draft:false, cache hits equal fresh runs, seeds and cancels.
…for the next A shared round with the plain switch at it asked no drafts and ran no head pass (core Rule.plain_streams, threaded from the family's resolved setting: 0 on CUDA, so its rounds are unchanged), and the family's speculation no longer launches a pass for a round that wide (armSpec/speculate plainRound).
…teps and the forward try's
…stays within 600 lines
…nything else by name
…treams streams decode
…er holds a rejected draft's taps
…nd the next ask catches the drafter up
…in / piece / finish) The Metal target implements the lane core's chunked hooks over its whole-prompt path: `begin` clears the slot or restores a kept state, each `piece` runs encoder chunks from the slot's tail position, and `finish` runs the rows the pieces left, the decoder's cone once over the prompt's end, and the choice and keep exactly as `prefill` does. An `encode` that stops short of the prompt's end prepares the next call's first chunk (keyed by the slot's tail and rows), so a lone prompt in pieces of the chunk size runs the same chunks, inputs and host overlap as one call. A mid-prompt state is never kept: a released, cancelled or failed prompt is abandoned (nothing saved, the prepared chunk dropped) and the slot's next prompt starts cleared. The lanes backend fills prompts a step at a time (Backend.prefill_step over the pieces, draft/lane_steps.zig) only when Options.steps is set; serve_lanes turns it on for Metal with TF_DSV41_PREFILL_PIECES=1, TENSORFOLD_PREFILL_ORDER=srpt or a LaneHost prompt policy, share or budget. With none of them set prompts run whole at admission, as before; the CUDA backend never sets it. Tests: pieces of 1, 7, 64, 100 and 300 rows equal the whole prompt (every state byte, the first choice and 8 greedy tokens) with a decode stream running between pieces; two prompts' pieces taking turns at the same positions; refused and released mid-prompt prefills; kept prompts resumed in pieces; served replies in steps equal serial; a short request during a long prompt gets its first token first in steps (after it as whole prompts); a cancel mid-prompt keeps nothing; host tests over a stepped CPU twin for the step adapter, its failure and cancel paths and the switch.
3410755 to
3e49808
Compare
|
Thanks @gilby. This reads well: with nothing set the Metal server still runs whole prompts (the steps vtable is only chosen with
One pointer, not a blocker: when a step stops short of the prompt's end, These five all sit on #1720 and touch the same few fields, so they'll land in this order, each rebased on the one before: #2951 (the profiler, to read the others' receipts), #2950 (opt-in), #2954, #2953 (on #2951, so
|
|
Real-weights receipt. One M3 Ultra 512 GB,
Alone, the long prompt takes 190 to 194 s and the short reply 1.8 s. Lone cold 32,768-token prompt, What it shows: steps work and are exact, the long prompt pays 1 to 5 s, and a decoding stream's freeze drops from the whole prefill to one chunk. But the short reply still takes 144 to 156 s, because the host runs one decode round after each 4,096-row piece, so the reply gets one token per piece. That is one token every 6.5 s, not fair service. The next step is to give decoders a share between pieces (several rounds a piece, or smaller pieces while a decoder is live) through the lane host's existing share setting, which this branch does not wire for DeepSeek yet. I'll add that here and rerun before marking it ready. |
…pt's steps With prompts in steps the lane host ran one decode round after each 4,096-row piece, so a reply beside a long prompt drew one token a piece. The DeepSeek served lanes now use LaneHost's own share: `--decode-share` reaches LaneHost (its share and, while replies decode, its row cap on a step), else TENSORFOLD_PREFILL_SHARE, else replies get 0.25 of wall time beside a prompt in steps. A step taken while a reply decodes owes it a third of the step's time in rounds before the next step; with no reply decoding nothing is owed, so a lone prompt runs as before. `--decode-share` on Metal turns steps on. With steps off the share stays the host's default (whole prompts). The target counts the most launches between two of a prompt's pieces (Stats). Tests: on the device, a share of 0.75 gives a decoding reply at least 3 rounds between two pieces (37 here) where a share of 0 gives at most 1, replies equal serial; host tests of the same over the stepped twin and of which share applies (flag, env, default, whole prompts).
…refused by name, docs `serve_lanes.stepsFor` takes --decode-share: above 0 it turns prompt steps on (its share then reaches LaneHost), outside [0, 1) it is refused, and otherwise the environment decides as before. The Metal entry reports either refusal by name at open: the flag's range, or TF_DSV41_PREFILL_PIECES's expected forms when its text is malformed (blank keeps the default). Docs: TF_DSV41_PREFILL_PIECES's row in environment.md names what it does on each backend; api.md lists DeepSeek-V4.1 on Metal among the servers that apply --decode-share; profiles.md notes the Metal reader; the optimization ledger's DeepSeek cell for prompt chunk interleaving is open on Metal. Test: --decode-share 0.5 with no environment turns steps on at that share (and the row cap); 0 and unset keep whole prompts; 1, 1.5, -0.1 and NaN are refused; a blank switch keeps its default and malformed text fails.
…nk at its own size When a step stopped short of the prompt's end, `encode` prepared the next full chunk. Under a capped step (TENSORFOLD_PREFILL_POLICY=fcfs with _TOKENS, or the row cap beside replies with --decode-share) the next call asks for fewer rows, so that stage never matched and the call read its inputs again. The stage now takes the rows the next call will ask for when it repeats this one's size, which is the next full chunk for an uncapped step (the same as before) and for the rows a finish has left. `Prompt.stage_hits` counts calls that start on a staged chunk. Tests: on the device, pieces of 1, 7, 64, 100 and 300 rows each start every later call on a staged chunk (1 and 7 rows found one before this change) with state equal to the whole prompt; a served fcfs prompt in 128- and 100-row steps beside a decoding reply starts its steps on staged chunks (8 of 8 and 10 of 10; 1 before) and replies equal serial.
|
Thanks. Head is now
Your pointer was right, and it is fixed: a step that stops short now prepares the next call's first chunk at this step's own size (capped at the prompt's end), so a capped step starts on a prepared chunk. The uncapped path prepares the same full chunk as before. A counter of calls that start on a prepared chunk is checked at 1, 7, 64, 100 and 300 rows and in the fcfs test (8 of 8 and 10 of 10 steps; it was 1 of 8 and 1 of 10 before). Since your review the branch also gives decoding replies a share of wall time between steps, through the lane host's own share (0.25 by default when steps are on). The first receipt showed why it was needed: one round per step. Receipt at
Alone, the reply takes 1.8 s and the long prompt 165 to 194 s across the five servers. So the short reply goes from about 200 s to about 8 s, the long prompt's first token stays inside the spread between the two base runs, and the share-1 arm reproduces the earlier one-round-per-step behaviour. Lone cold 32,768-token request: stack tip 47.48 and 45.85 s, steps on 45.16 s, same The optimization ledger's cell for this family and item is updated, and the body's issue line now points at #2984. At this head on an M5 Max: |
|
Thanks @gilby. We read the three new commits on One small fix, then what's still owed:
|
|
M3 Ultra, 512 GB: the series is being re-measured now under the noise rule in #2984 (one discarded first run, base on both sides, both-under-both test); it follows in this thread.
Summary: with nothing set this head is level with base on this chip and changes no token. Default-path decode ( Cells under 0.98 of base: none. Default path, nothing set: Order as run: base 1, this 1, this 2, base 2. Ratios: this 1 over base 1, this 2 over base 2, and mean over mean. Token hashes equal in every cell across the four arms: yes; failed / alone / serial mismatches: 0. All 16 cells are reported.
Method as run: each cell is a fresh server started just before it; a thermal-pressure check ( Fill ( 'Reply first' sends the reply, then the long prompt 0.5 s later; 'long first' sends the long prompt, then the reply 10 s later.
Every fill cell started at Nominal thermal pressure and ended at Moderate thermal pressure after the 131,072-token prefill (the powermetrics thermal sampler's level, not memory pressure); the fill script sets end-of-cell pressure as informational for this long prefill, so none was discarded. Alone-run long TTFT of the seven fill cells ranges 134.17 to 137.23 s. In the cold 32,768-token series one kept cell had a macOS background process (mediaanalysisd) using 97% CPU at the end-of-cell check (this head, unset); thermal pressure was Nominal at both ends of all three cold cells. Cold 32,768-token prompt (
Cold TTFT over base (single runs): this unset 1.001, Not run: the pieces, decode-share and Raw JSON, server logs and driver scripts: https://gist.github.com/gilby/d41622e4da2c9b160eaa0ad8c2f36ee3 |
|
Thanks @gilby for the M5 Ultra receipt. It answers the policy cells we asked for. With These M5 Ultra series came before the discarded first run in #2984's noise rule. Base ran first in each, so any first-run lift favours base, not this branch. We'll take them as they are, with no need to rerun them. What's still open:
Then it lands second in the stack, right after #2951.
|
|
M3 Ultra, 512 GB: the receipt promised above, run on 2026-11-12 and 13 under the noise rule in #2984 (one discarded first run per series, base on both sides, both-under-both test).
Summary: with nothing set this head is level with base on this chip in the default path and changes no token: aggregate decode this over base, mean over mean, is 0.998 to 1.002 in all 16 Cells under 0.98 of base: none in the default-path series. In the fill table the nothing-set arm's alone-run long first token is 0.963 and 0.972 of the two base runs as a speed ratio (single runs, outside the noise rule; see the note under the fill table). Default path, nothing set: Order as run: base 1, this 1, this 2, base 2. Ratios: this 1 over base 1, this 2 over base 2, and mean over mean. Verdict by the #2984 rule: slower only if both this runs are under both base runs and the mean gap is over 1%, faster the mirror image, otherwise within noise. Token hashes equal in every cell across the four arms: yes; failed, alone or serial mismatches: 0. All 16 cells are reported: 0 slower, 0 faster.
Thermal pressure level at the end of each default-path cell (powermetrics thermal sampler; every cell started at Nominal, and the discarded first run of every series ended at Nominal):
The kept cells that ended above Nominal in this part are the four 4-stream T 1 cells and two of the four Fill ( 'Reply first' sends the reply, then the long prompt 0.5 s later; 'long first' sends the long prompt, then the reply 10 s later. 'Reply done' is the time from sending the reply to its last token.
Every fill cell started at Nominal and ended above it after the 131,072-token prefill, and none was discarded or re-run (a fill cannot be shortened): base run 1, this with nothing set and Cold 32,768-token prompt (
Cold first token over the mean of the two base runs (43.185 s), single runs: this with nothing set 0.999, All four cold cells are kept cells that ended above Nominal: each ended at Heavy on its first attempt, was discarded, was re-run once after cooling to Nominal, and ended at Heavy again, so the second attempt is kept and flagged. On this machine a cold-prefill cell of this size ends at Heavy whichever arm runs. The first attempts' raw files were not archived (their discard lines are in the window log). Method as run: each cell is a fresh server started after the previous one was stopped and a 60 s settle, and a cell starts only when the powermetrics thermal sampler reports Nominal (the driver polls every 30 s; the longest wait was 92 s); the level is recorded again at the end of the cell. One run per series is discarded and not in any table, base is on both sides, and the series are per stream count and temperature (bench_concurrent) and per temperature (bench_openai). Because a 2-minute 4-stream cell that starts at Nominal ends at Moderate on this machine and the longer cells at Heavy, a cell is not discarded for its end level unless it ended at Heavy on a short cell: then it is discarded, cooled to Nominal and re-run once, and if it ends at Heavy again it is kept and flagged. No default-path cell ended at Heavy; the fill cells are not re-run, and the four cold cells are the flagged ones above. Tools and server flags are the ones in the header; the tools come from the base tree and the head does not touch Not run: the pieces, Raw JSON, server logs and driver scripts: https://gist.github.com/gilby/d5d940706755000180b1855a8a82b5a3 |
…hare in steps, as unset does applyShare skipped the steps default whenever TENSORFOLD_PREFILL_SHARE was present, blank included, while the lane host reads a blank value as unset and so as its own default share of 1: with steps on and the variable blank, a prompt kept all the time and replies were back to one round a step. A blank value now counts as unset here too. The share test gains the blank case (it read 1 before).
|
New head:
Receipts: the M5 Ultra and M3 Ultra receipts above were taken at Your other points:
At |
|
M3 Ultra, 512 GB: the repeat of the fill cell with nothing set that the earlier comment said was still owed, at head
'Reply first' sends the 64-token reply, then the long prompt 0.5 s later; 'long first' sends the long prompt, then the reply 10 s later; 'reply done' is the time from sending the reply to its last token. Each arm is a fresh server and a single fill (a 131,072-token cold prompt is not repeatable inside one server).
The long answer's All five servers (the discarded run included) ended at Heavy thermal pressure; this is what a cell this long does on this machine, and the arms ran the same load in the same order, so it does not separate them. With nothing set this head equals base on every figure: this over base, mean over mean, is 1.000 for the alone-run first token, 0.999 and 0.997 for the long prompt's first token with the reply first and long first, and 0.999 and 0.997 for the reply's completion time. The alone-run first token is 170.0 to 170.3 s in all four runs (170.26, 170.04, 170.34, 170.04 s), so the single 174.07 s reading of the earlier receipt did not repeat; the two base runs are 0.22 s apart and the two this runs 0.30 s apart. Still owed: the lone cold 32,768-token prompt with Raw JSON, server logs and drivers: https://gist.github.com/gilby/c9070e544e81abb399bccd8a101cf1df |
What this changes
On Metal, a DeepSeek-V4.1 prompt ran whole in one lane-host call, so a long prompt held every other stream until it finished and
TENSORFOLD_PREFILL_ORDER=srptand the prompt policies had nothing to act on. With this change the Metal target implementsbegin/piece/finish, and the lanes offer a prompt step built on them, so a prompt can run a chunk at a time between the other streams' rounds. Nothing changes unless a setting asks for it: withTF_DSV41_PREFILL_PIECES, the SRPT order and the prompt policy, share and budget all unset, prompts run whole exactly as before.a43ad7b98); this adds one commit,dsv41 metal: prompts in pieces between the other streams' rounds (begin / piece / finish). Two commits since review:b8c0cb299(a blankTENSORFOLD_PREFILL_SHAREis unset inapplyShare) and49a92da01(the CHANGELOG line quotes the served result on both chips).atc overlap --json: run on this head against main; incomplete (its time and pull-request limits). Nine textual conflicts with other open pull requests, each in one file:CHANGELOG.md, or the settings tables inplatform/environment.ziganddocs/environment.md(qwen27: serve the 27B pair, refuse what a pair cannot keep, and die with the peer #1434, Vulkan: qmm_q4_tile_rows computes four rows and two columns of a prompt chunk per thread, 3.6% to 33% less run time on an M1 with the same logits (stacked on #2016) #2017, Ling-3.1-flash on CUDA, on one or two GB10s #2340, Flash Next TP=2 on CUDA: head and rollback graphs replay, and one-row split lanes skip the interleave copies #2837, server: --execution-witness serves a read-scoped snapshot of what runs, or why it is unknown #2901, server: "execution_witness": true binds a reply to the witness at admission and at its end, with its prompt, resolved request and tokens #2958, flashnext cuda: at --tp 2 on one host, collectives run on a blocking split of the nonblocking communicator #2978, flashnext cuda: the opt-in --tp 2 decode graph (TF_FLASHNEXT_DECODE_GRAPH=1) serves again instead of failing its first plain round #2980, flashnext cuda: the opt-in --tp 2 decode graph takes the dense split and keeps its graph across drafted verifies #2982). No shared source logic.How it stays exact:
finishruns the last rows. The chunk boundaries, inputs and the host work prepared ahead are the ones the whole-prompt loop uses, so the state is equal by construction.finish. A mid-prompt state is never kept; a kept entry is a whole prompt's state, and a cache hit atbeginrestores it before the pieces, asprefilldoes.Checks on this head
49a92da01python3 tools/zig/lean_check.py, on the base and on this head: 345 and 345, 0 new (tools/pr_check.py0 errors, 0 warnings).zig build test-dsv41-metal-all: 9/9 steps, 47/47 tests.zig build test: the share test's new blank case failed beforeb8c0cb299(share 1 where 0.75 is expected) and passes after; the only other failure in the full suite isdashboard_test, which fails the same way on this base and was fixed on main by tests: the dashboard page's test runs on every OS, and expects only tiles the page has #2946."draft": false; a short request during a 1,100-token prompt gets its first token first in steps and after it with whole prompts; a cancel mid-prompt keeps no prefix.CHANGELOG.md), and per the order in DeepSeek-V4.1 on Metal: decode and prompt speed on M3 Ultra and M5 Ultra, token-exact (gate and order for the drafts) #2984 it is rebased onto the PR before it at landing.TF_DSV41_PREFILL_PIECES=1with base on both sides, and a repeat of the M3 Ultra fill arm with nothing set (both requested on this PR, run list in the comment); CUDA beyond compiling (the DeepSeek CUDA engine sharesdraft/lanes.zig).Receipt
bench_concurrent1 to 4 streams 0.997 to 1.002 on the M5 Ultra and 0.998 to 1.002 on the M3 Ultra (all 16 cells within noise by the DeepSeek-V4.1 on Metal: decode and prompt speed on M3 Ultra and M5 Ultra, token-exact (gate and order for the drafts) #2984 rule),bench_openai0.997 to 1.001. Opt-in: a 64-token reply beside a 131,072-token cold prompt finishes in 6.3 to 13.0 s withTF_DSV41_PREFILL_PIECES=1against 130 to 142 s on base (M5 Ultra) and 7.6 to 13.8 s against 209 to 256 s (M3 Ultra). Receipts: https://gist.github.com/gilby/d41622e4da2c9b160eaa0ad8c2f36ee3 (M5 Ultra) and https://gist.github.com/gilby/d5d940706755000180b1855a8a82b5a3 (M3 Ultra), measured atfdcc22211;b8c0cb299changes only the blank-variable case, which no measured arm set.TF_DSV41_PREFILL_PIECES=1now turns steps on for Metal (on CUDA it still selects the round planner);TENSORFOLD_PREFILL_ORDER=srptand the prompt policy, share and budget settings turn them on too. Default off.fdcc22211on dsv41 metal: the lane target, the prompt cache and the native server over Metal #1720'sa43ad7b98;deepseek-ai/DeepSeek-V4.1-Flash-1017release weights at revision1bae0e1088993b9b82de08f59de5be5a34e42170; M5 Ultra 512 GB and M3 Ultra 512 GB; Zig 0.17.0; tests on an M5 Max.token_sha256equal in every cell of both chips' default-path series, and in every fill arm (fcfsandfcfswithTENSORFOLD_PREFILL_TOKENS=1000included) the long answer's hash equals the unset run's;bench_fillreports every reply equal to its run alone.bench_openai0.997 to 0.999 (M5 Ultra) and 0.998 to 1.001 (M3 Ultra), nothing set.TF_DSV41_PREFILL_PIECES=1: M3 Ultra 0.998 of base; M5 Ultra 1.021 on a single run, to be repeated with base on both sides.Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo (synthetic weights, and on the release weights in the receipts).--drafter, no slower than without it, greedy and sampled