Repository navigation
Conversation
…over Metal target.zig is the lanes' verify target: a slot's prompt, its windows of rows for several slots in one launch, keeps and rewinds, and kept prompt states (the prompt cache) whose resumed rows the reply reports as cached_tokens. boot.zig loads a release checkpoint folder and entry.zig serves it as tensorfold-dsv41 (dsv41-metal-serve). Tests serve the tiny model through LaneHost: concurrent equals solo equals serial, drafted equals draft:false, cache hits equal fresh runs, seeds and cancels.
…for the next A shared round with the plain switch at it asked no drafts and ran no head pass (core Rule.plain_streams, threaded from the family's resolved setting: 0 on CUDA, so its rounds are unchanged), and the family's speculation no longer launches a pass for a round that wide (armSpec/speculate plainRound).
…teps and the forward try's
…stays within 600 lines
…nything else by name
…treams streams decode
…er holds a rejected draft's taps
…nd the next ask catches the drafter up
The depth rule priced Metal rounds with the CUDA-shaped defaults. Metal now times dry windows of 1 to 16 rows, the second launch from row 17 and a DSpark pass with its ingest at load, on a time budget, through the lane core's calibration path. TF_DSV41_CALIB keeps its meaning; measure is the Metal default, and TF_DSV41_CALIB_BUDGET_MS bounds the repeats. The prompt cache is not fed and every slot is left as new.
…t table The first measurement took 12.7 s and priced the pass at 62 ms. It now times rows 1, 2, 4, 8 and 16, the 4 + 4 window, rows 17 and 32 and the pass once each after an untimed window and pass, repeats the passes only while the budget lasts, and interpolates the rows between. The startup line reports the ingest share of the pass and the timed runs.
…when no draft pays Drafting stays on at every stream count. Before a round's pass, the lanes price the next round from each stream's running confidences (calibrated, the last passes blended): the asks and drafts with the most expected tokens a ms, with no ask and no pass among the choices. A round where no draft pays runs no pass and no drafter ingest, as --no-drafts would; a stream asks again when its prediction pays, when it is new, or at a probe bounded to 1% of the plain rounds' modelled time. After the pass the depths come from the same price, k = 0 included. TF_DSV41_DRAFT_PRICED selects it; the Metal lanes turn it on, and with it the count switch defaults off (TF_DSV41_DRAFT_PLAIN_STREAMS still sets it). Off, every choice is the earlier rule's, checked against a hash of its choices taken before the setting existed. Pricing only: tokens never change.
… table by device and weights The Metal measurement now checks every TF_DSV41_CALIB* setting before it times anything, so a malformed value stops the start naming the setting, its text and the accepted forms; only a measurement that fails falls back to the defaults. The settings have rows in platform/environment.zig and docs/environment.md. A stored table carries the device and checkpoint identity, so another chip or other weights never read it. dropWindow's comment says it is for calibration, with wipe after.
…very window a lone stream can make The measurement prefilled the calibration text into each slot twice and ran thirty one-row windows to grow a continuation, which cost about nine seconds on the release model outside the timed runs. It now prefills one short prompt in each slot it needs once, takes the window tokens from the text itself, and clears only the slots it used. Rows 1 to 6 are timed rather than interpolated, since a lone stream's drafts make exactly those windows.
…efuse a malformed value at a Metal start Both settings have rows in platform/environment.zig and docs/environment.md, so a Metal start stops naming the setting, its text and the accepted forms; on CUDA the depth reader refuses them when the engine opens, as before. The ledger's shared-drafts cell for DeepSeek-V4.1 points at ashhart#2986.
The start-up measurement already timed the drafter's ingest inside its one-ask pass; the table now keeps it (stored with the table, read back to the microsecond), and the priced rule adds it for each further ask and once more for a stream that catches up after plain rounds. A table without it prices every pass flat, as before.
|
Closing this PR as a failed attempt. The priced draft rounds missed their bar on an M5 Ultra twice, with the version pushed here and with a reworked version that stayed local. The stream-count switch ( The bar set in advance: the default within 2% of What was tried. The pushed version (head The reworked version (local only, Both priced from the cost table timed at start, which is itself not adopted as the default: on an M5 Ultra it made three chat cells 2 to 4% slower and it is an opt-in in #2952. Result of the reworked version, M5 Ultra, 512 GB, 1 and 2 streams,
3 and 4 streams,
At 1 stream the priced rounds are 14 to 61% ahead of The pushed version, from an earlier series on the same machine and method (head
Against So the bar is not met at 3 and 4 streams in either version, and both versions lose to the current rule at 1 and 2 streams. The cells were not changed after seeing the result. Method as run: each cell is a fresh server, arms in the order A, priced, earlier rule, earlier rule, priced, A, a thermal-pressure check ( Ledger row ( Raw JSON, server logs and driver scripts: https://gist.github.com/gilby/9dc31f647b945d8522b255f1c88348aa |
What this changes
DeepSeek-V4.1 on Metal decided whether to draft by counting streams: rounds of three or more decoding streams ran plain. Below that, the drafter's pass for the next round was launched before that round's depths were known, so the depth rule could only choose how many drafts to verify, never to skip the pass, and a stream whose drafts were not landing still paid a pass every round. With this change the lanes price each round's asks before the pass: from each stream's own running draft confidences, calibrated, it picks the asks and depths with the most expected tokens per ms, including no ask for a stream and no pass at all for the round. A round with no ask does the work of a
--no-draftsround. Drafting is on at every stream count; the count switch defaults to off andTF_DSV41_DRAFT_PLAIN_STREAMSstill works when set.a43ad7b98) and DeepSeek-V4.1 on Metal: opt-in draft cost table timed at start, default pricing unchanged (stacked on #1720) #2952 (the measured cost table, nowafbc1ab33); this adds the priced rule and its follow-ups, with DeepSeek-V4.1 on Metal: opt-in draft cost table timed at start, default pricing unchanged (stacked on #1720) #2952's two newer commits cherry-picked on top so nothing published was rewritten. Draft until those land and the real-model receipt below is in.atc overlap --json: run on this head; the result is incomplete (its time and pull-request limits) and reports nine textual conflicts with other open pull requests: seven inCHANGELOG.md(server: --execution-witness serves a read-scoped snapshot of what runs, or why it is unknown #2901, server: "execution_witness": true binds a reply to the witness at admission and at its end, with its prompt, resolved request and tokens #2958, Vulkan: qmm_q4_tile_rows computes four rows and two columns of a prompt chunk per thread, 3.6% to 33% less run time on an M1 with the same logits (stacked on #2016) #2017, Flash Next TP=2 on CUDA: head and rollback graphs replay, and one-row split lanes skip the interleave copies #2837, Nemotron 3.5 Bolt on CUDA: the shared expert runs on the side stream in decode windows #2847, CUDA EXL3 grouped experts: launch and size storage at the caller's tile and pair count #2843, docs: dots.mocr's images on native CUDA #2773) and two in the settings tables ofplatform/environment.zig, its test anddocs/environment.md(Ling-3.1-flash on CUDA, on one or two GB10s #2340, qwen27: serve the 27B pair, refuse what a pair cannot keep, and die with the peer #1434), where each adds its own rows. No shared source logic.How it decides: a stream's prediction is the mean of the confidences its own passes reported, by draft position, calibrated with the shared per-position calibration. The planner scans the largest chain gains over all streams against the measured cost of the rows plus the pass. A fresh stream asks. A stream that went plain comes back through a probe: when the modelled time of rounds with a plain stream reaches a hundred passes' worth, every stale stream asks together in one pass, so probing costs at most 1% of plain-round time. After the pass the same scan prices the depths with the pass paid, and zero is allowed.
Scope: nothing under
zig/src/corechanges. The setting (TF_DSV41_DRAFT_PRICED) defaults on only in the Metal lanes; CUDA keeps it off unless set. With it off the choices are the old ones: a test hashes 1,800 depth choices (joint modes 0 to 2, five confidences, 1 to 4 streams, two cost tables) against a hash taken on the base commit.Checks on this head
c5d5687f6python3 tools/zig/lean_check.py, on the base and on this head: 345 and 345, 0 new.tools/pr_check.py: 0 errors.test-dsv41-draft160/160 (17 new);test-dsv41-serve-host14 passed, 5 skipped;test-shared-lane-regressions-host360 passed, 1 skipped;test-dsv41-metal-all43/43, with a new device test of four streams with the setting on (replies equal serial; 63 rounds priced, 6 with a pass, 57 without, 2 probes).zig build test: 12,649 passed, 107 skipped, 1 failed (dashboard_test, which fails the same way on main).Receipt
deepseek-ai/DeepSeek-V4.1-Flash-1017, fresh servers,--prompt-cache-gib 0:tools/bench_concurrent.py --levels 1,2,3,4 --alone --serial --temperatures 0,1.0 --tokens 256 --reps 3andtools/bench_openai.py, arms--no-drafts, this default, this withTF_DSV41_DRAFT_PRICED=0(the old behaviour). The bar: the default within 2% of--no-draftsin every cell, no slower than the old behaviour anywhere, and ahead of plain on code at 3 and 4 streams.--no-drafts: at confidence 0.5 the old rule gave 0.88 and 0.90 at one and two streams and this gives 1.00; at 0.9 with three and four streams the count switch gave 1.00 and this gives 1.18 and 1.08. These are the model's numbers, not measurements.TF_DSV41_DRAFT_PRICED(0, 1, off, on, false, true), default on in the Metal lanes and off elsewhere, with its rule row inplatform/environment.zigand itsdocs/environment.mdrow; a malformed value stops a Metal start by name. Nodocs/profiles.mdrow: that file lists the CUDA profile registry's settings and none of the depth-policy switches. The pass is priced per ask from the table's timed ingest, and a stream catching up after plain rounds is charged a second one.TF_DSV41_DRAFT_PLAIN_STREAMSnow defaults to 0 on Metal when it is on.a43ad7b98and DeepSeek-V4.1 on Metal: opt-in draft cost table timed at start, default pricing unchanged (stacked on #1720) #2952, Zig 0.17.0, Xcode 27.2 Metal toolchain, M5 Max for the tests above.bench_concurrent --alone --serialon the release weights to follow.Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo (synthetic weights; release weights to follow).