Repository navigation
Conversation
…over Metal target.zig is the lanes' verify target: a slot's prompt, its windows of rows for several slots in one launch, keeps and rewinds, and kept prompt states (the prompt cache) whose resumed rows the reply reports as cached_tokens. boot.zig loads a release checkpoint folder and entry.zig serves it as tensorfold-dsv41 (dsv41-metal-serve). Tests serve the tiny model through LaneHost: concurrent equals solo equals serial, drafted equals draft:false, cache hits equal fresh runs, seeds and cancels.
…for the next A shared round with the plain switch at it asked no drafts and ran no head pass (core Rule.plain_streams, threaded from the family's resolved setting: 0 on CUDA, so its rounds are unchanged), and the family's speculation no longer launches a pass for a round that wide (armSpec/speculate plainRound).
…teps and the forward try's
…stays within 600 lines
…nything else by name
…treams streams decode
…er holds a rejected draft's taps
…nd the next ask catches the drafter up
The depth rule priced Metal rounds with the CUDA-shaped defaults. Metal now times dry windows of 1 to 16 rows, the second launch from row 17 and a DSpark pass with its ingest at load, on a time budget, through the lane core's calibration path. TF_DSV41_CALIB keeps its meaning; measure is the Metal default, and TF_DSV41_CALIB_BUDGET_MS bounds the repeats. The prompt cache is not fed and every slot is left as new.
0b51e33 to
c8c30ea
Compare
|
Thanks @gilby. Pricing from the machine's own table makes sense, and the tests cover the parts that matter for exactness: every slot's state hashes the same before and after the measurement, the stats and the prompt cache are untouched, and replies after a measured start equal the serial decode. Since the verify window keys each row's draw by its position, depth can only change speed. Asks:
If #2950 or #2951 lands first: These five all sit on #1720 and touch the same few fields, so they'll land in this order, each rebased on the one before: #2951 (the profiler, to read the others' receipts), #2950 (opt-in), #2954, #2953 (on #2951, so
|
|
Real-weights receipt, and this is not ready. One M3 Ultra 512 GB, The measured table on this chip: Two problems:
I'll fix both (a pass timed the way a round runs it, and a calibration that fits in about 2 s) and rerun before asking for anything. The table itself is useful: about 9 ms a row from 1 to 16 rows on an M3 Ultra, and only a small step at row 17 (169.5 to 190.4 ms, where the second launch starts). |
…t table The first measurement took 12.7 s and priced the pass at 62 ms. It now times rows 1, 2, 4, 8 and 16, the 4 + 4 window, rows 17 and 32 and the pass once each after an untimed window and pass, repeats the passes only while the budget lasts, and interpolates the rows between. The startup line reports the ingest share of the pass and the timed runs.
… table by device and weights The Metal measurement now checks every TF_DSV41_CALIB* setting before it times anything, so a malformed value stops the start naming the setting, its text and the accepted forms; only a measurement that fails falls back to the defaults. The settings have rows in platform/environment.zig and docs/environment.md. A stored table carries the device and checkpoint identity, so another chip or other weights never read it. dropWindow's comment says it is for calibration, with wipe after.
|
Thanks. Head is now
On Rerun on one M3 Ultra 512 GB, release weights, at
The two start lines: What that fixes and what it does not. The two-stream loss is gone: two streams are 2.5 to 4.4% ahead of the stack tip, and one-stream code 2 to 3% ahead. Two things are still wrong: one cell is behind by more than noise (one stream, sampled chat, 46.1 and 46.5 against 47.5 and 47.5), and the calibration still reports 10.1 s over 11 timed runs, with load at 15.2 s against 5.1 s, though the timed windows add up to about 1.1 s. So most of that time is outside the timed runs and I have not found it yet. Not ready; I'll post again when both are closed, with three starts on each chip as you asked. At this head on an M5 Max: |
…very window a lone stream can make The measurement prefilled the calibration text into each slot twice and ran thirty one-row windows to grow a continuation, which cost about nine seconds on the release model outside the timed runs. It now prefills one short prompt in each slot it needs once, takes the window tokens from the text itself, and clears only the slots it used. Rows 1 to 6 are timed rather than interpolated, since a lone stream's drafts make exactly those windows.
…t be one The pass's entry was a single sample taken right after the largest windows, so whatever the first ingest after them costs became the table's pass. The pass is now run once untimed and five times back to back, as a round runs it, and the fastest is kept; the first ingest after the windows is reported apart. A table whose pass is dearer than twice its own 1-row window is refused when measured and when read back from a stored file.
…budget The slot term is the 4 + 4 window less the 8-row one, each a single sample once the budget was spent. Both now run twice more side by side after the passes, marked so the budget never drops them; only the other row counts are optional.
…ults On the M5 Ultra the measured table made code faster and sampled chat slower in three cells, with the rule near-indifferent between the depths. The table is still timed at start and reported, with its checks and settings, but the default table prices the rounds unless TF_DSV41_CALIB names a source, so served depths are unchanged. The lanes test and the Metal device test run from the Metal test steps.
Unset or blank keeps the default table with nothing timed and no extra startup line, so load time and behaviour are the base's. measure, zig, cached and real time or read the table, check it, report it and price from it. The table_prices switch is gone.
|
This push narrows the PR: with Commits since the last head Why the default changed. On an M5 Ultra, pricing from the measured table at
Two more checks on why. An unpushed variant that changed the acceptance estimate the depth rule uses did not recover the cells (chat 1 stream sampled 0.980, chat 2 streams greedy 0.981, Result: pricing from the measured table is not adopted for the default, and stays as an opt-in. Receipt at the final head,
Summary: token hashes equal to base in every cell of every arm (0 differences; Cells under 0.98 of base: none (unset). Opt-in
Opt-in
Startup lines, base and this head with the setting unset (identical in the 1 and 2 stream, the 3 and 4 stream and the Opt-in Refusal of a malformed value. Method as run: each cell is a fresh server started just before it, and the first server of each piece is run and discarded. Base is measured on both sides of this head (base, this, this, base). A thermal-pressure check ( Not run: M3 Ultra (pending, above); CUDA; cold-prompt cells for this head (nothing in the prompt path is changed, which is a reading of the diff and not a measurement); the kernel contract and device tests on this head (no GPU kernel is new or changed). Raw JSON, server logs and driver scripts: https://gist.github.com/gilby/d7cf0e381966791ac5ec9dc686e95de2 |
|
Thanks @gilby. Narrowing this to an opt-in is the right call, and the evidence for it is clear. Three chat cells were slower with the measured table in both series (for example, 1 stream sampled at 0.972 and Small things on
One pointer, not a blocker: In the order, this stays after #2983, so that its table is measured on the final kernels.
|
|
M3 Ultra, 512 GB: the receipt promised above, run on 2026-11-13 under the noise rule in #2984 (one discarded first run per series, base on both sides, both-under-both test).
Summary: two series, (a) Cells under 0.98 of base: none in series (a). In series (b) one, in the rerun only: 1 stream, temperature 1, chat at 0.978 (its first run was 0.981). (a) Order as run: base 1, this 1, this 2, base 2. Ratios: this 1 over base 1, this 2 over base 2, and mean over mean. Verdict by the #2984 rule: slower only if both this runs are under both base runs and the mean gap is over 1%, faster the mirror image, otherwise within noise.
End level key: Nom = Nominal, Mod = Moderate, Heavy = Heavy. Exactness: token hashes equal across all four runs in every one of the 12 cells, 0 failed requests, 0 Startup line. With the setting unset, all 25 servers of series (a) print exactly one cost line, the default one, and nothing else about calibration; no log of the series mentions calibration. Verbatim from one of them: (b) Same order, rule and columns. The first run of each series (all 12 cells), then the rerun of the two series that had slower cells.
Verdicts, first run: 5 faster, 3 slower, 4 within noise. Exactness: token hashes equal across all four runs in every one of the 12 cells, 0 failed, 0 mismatches. Rerun. The 1-stream series and the
All 6 rerun cells, in full:
The rerun has 3 faster cells, 3 slower (1 stream T0 code, 1 stream T1 chat, openai T1 gpu-chat-no-think) and 0 within noise; token hashes equal in all 6, 0 mismatches. The faster cells are the same in the first run and the rerun (1 stream T0 chat 1.017, 1.016; 1 stream T1 code 1.043, 1.044; What the measured table printed. The first Over the 14 Discarded, retried and flagged cells
Method as run: each cell is a fresh server started after the previous one was stopped and a 60 s settle, and a cell starts only when the powermetrics thermal sampler reports Nominal; the level is recorded again at the end of the cell. One run per series is discarded and not in any table, base is on both sides, and the series are per stream count and temperature ( Not run: Raw JSON, server logs and driver scripts: https://gist.github.com/gilby/8f3815c1164339f45cf8f9d6b2fa73f4 |
|
New head: Your "small things" of 12 Nov 23:50, point by point:
The Receipts: the three new commits are documentation, the CHANGELOG and the ledger, so the M5 Ultra and M3 Ultra receipts above (taken at At |
What this changes
On Metal, DeepSeek-V4.1's draft depth rule prices its rounds from the CUDA-shaped built-in table. This adds a way to time a table on the loaded model and price from it, as an opt-in: with
TF_DSV41_CALIBunset or blank on Metal nothing is timed and the built-in table prices the rounds exactly as on base (same startup lines, same load time, same depths, same tokens); namingmeasuretimes a table at start (about two seconds), checks it, reports it and prices the rounds from it,zigstores that table and reads it back on the device and checkpoint that stored it,cachedandrealread a stored one. A malformedTF_DSV41_CALIB*value stops a Metal start naming the setting, and a table whose pass costs more than twice its own 1-row window is refused. Pricing never changes a token.The PR started as "price the rounds from a table measured at load" by default. On an M5 Ultra that made the code cells 1 to 6% faster and three chat cells 2 to 4% slower (repeated on a rerun), so the default stays the built-in table and the measured table is kept as an opt-in. The measurements are in the receipt comment on this head.
a43ad7b98:c8c30ea8f(price drafts from a cost table timed on the loaded model),1a135017d(time a sparse set of windows and a warm pass),562b276a3(refuse a malformed timing setting, key a stored table by device and weights),afbc1ab33(time over a two-row prompt, every window a lone stream can make),c0e24f554(time the pass back to back, refuse an implausible pass),962135265(the slot term's windows keep their repeats),6bf0ec9c0and55ae488ba(time nothing unlessTF_DSV41_CALIBnames a source),284548471(the refusal text names the Metal default as it is). Three commits since review:1a3baab44(theTF_DSV41_CALIBrow describes each mode once),0d892bbcb(the CHANGELOG line says what a user gets),a97716e2d(the optimization ledger row).atc overlap --json: run on this head on 2026-11-12, no conflicts; base3cacb821c(the merge basetools/pr_check.pyreports).Checks on this head
a97716e2d(the receipts were measured at55ae488baon the M5 Ultra and284548471on the M3 Ultra; the commits since are documentation, the CHANGELOG and the ledger)python3 tools/zig/lean_check.py, on the base and on this head: 345 and 345, new findings 0 (tools/pr_check.py0 errors, 0 warnings)284548471,zig build test-dsv41-draft -Dcpu=apple_m1148 of 148, andzig build test-dsv41-metal-all,test-environment-hostandtest-dsv41-env-host45 of 45 over 28 build steps. Ata97716e2d(documentation only since):test-env-defaults-host102/104 (2 skipped),test-environment-report-host36/36,test-environment-host,test-dsv41-env-host14/14,test-profile21/21. Without the change: on this PR's base,cost_table_test.zig,calib_pass_test.zigandcalib_test.zigdo not build, since each uses what the PR adds (Plan.initWide,calib_measure.launch_bound, the stored table'sid,Costs.plausible, the pass'singesttiming,metal.calib); the malformed-setting case incalib_test.ziggot the default table instead of a refusal before562b276a3.CHANGELOG.md), and per the order in DeepSeek-V4.1 on Metal: decode and prompt speed on M3 Ultra and M5 Ultra, token-exact (gate and order for the drafts) #2984 it is rebased onto dsv41 metal: decode experts decode an expert's weights once for all of its rows (stacked on #1720, #2973) #2983 at landing.TF_DSV41_CALIB=measurestarts on the M5 Ultra (one is posted; they are best taken on the final kernels, at landing); CUDA (the shared filesdraft/costs.ziganddraft/calib_env.zigalso serve the CUDA engine; only the host tests that include them ran); cold-prompt cells (nothing in the prompt path changed, which is a reading of the diff and not a measurement)Receipt
TF_DSV41_CALIBunset: token hashes equal to base in every cell of every arm, and every cell level with base: mean over mean 0.998 to 1.002 in the 16bench_concurrentcells (1 to 4 streams, greedy and sampled, chat and code) and the 4bench_openaicells, 0.998 to 1.002 in the 8--no-draftscells, per pair 0.997 to 1.004. Cells under 0.98 of base: none. Startup lines, load time (3.5 to 3.6 s on both sides) and draft counters equal to base. WithTF_DSV41_CALIB=measure(one run, opt-in): code +1 to +6%, chat 2.7 and 3.0% slower in two cells, load +2.1 s, tokens equal. All cells and the method are in the receipt comment. M3 Ultra, 512 GB, at284548471, under the DeepSeek-V4.1 on Metal: decode and prompt speed on M3 Ultra and M5 Ultra, token-exact (gate and order for the drafts) #2984 rule: withTF_DSV41_CALIBunset all 12 cells within noise (0.998 to 1.001), token hashes equal, the server prints only the default cost line; withmeasure(opt-in) 5 cells faster, 3 slower (1.2 to 2.2%, repeated on the one rerun), 4 within noise. Receipts: https://gist.github.com/gilby/d7cf0e381966791ac5ec9dc686e95de2 (M5 Ultra), https://gist.github.com/gilby/8f3815c1164339f45cf8f9d6b2fa73f4 (M3 Ultra).deepseek_v41) on Apple GPUs; with the setting unset none of its behavior changes. The CUDA engine sharesdraft/costs.ziganddraft/calib_env.zig: a stored table whose pass is dearer than twice its 1-row window is refused on read-back there too, and a stored table carries a device and weights identity that is 0 (the old file name and match) when a caller names none; CUDA was not run.TF_DSV41_CALIB(existed for CUDA, now registered for Metal inplatform/environment.zig): unset or blank isdefaulton Metal (the built-in table, nothing timed) andcachedon CUDA; acceptedcached,default,real,measureorzig, in any case; anything else stops a Metal start naming the setting, the text and the accepted forms. Ten moreTF_DSV41_CALIB_*settings (_DIR,_FILE,_SHAPE,_STAT,_REPS,_WARM,_RECHECK,_CYCLE,_DEEP_REPS,_BUDGET_MS) are registered the same way.docs/environment.mdhas a row for each; there is nodocs/profiles.mdrow, since none of them selects a policy.a43ad7b98, head55ae488bafor the numbers and284548471pushed; Zig from.zig-version(0.17.0; the version used on the machine is not recorded in the logs); Metal toolchainTOOLCHAINS=com.apple.dt.toolchain.Metal.32023.922.4; M5 Ultra, 512 GB, macOS 27.2.1;deepseek-ai/DeepSeek-V4.1-Flash-1017release weights at revision1bae0e1088993b9b82de08f59de5be5a34e42170; M3 Ultra, 512 GB, for the second receipt--no-drafts, 1 to 4 streams,TF_DSV41_CALIBunset,measure, and a malformed value); ran the M3 Ultra server (unset andmeasure); could not run CUDA or another Metal chiptools/bench_concurrent.py --alone --serialortoken_shaagainst"draft": false:token_sha256equal in every cell across base, this head, the opt-in arm and the discarded first runs (0 differences in 13bench_concurrentarms and 5bench_openaiarms);--aloneand--serialunequal 0 and failed 0 in every celltools/bench_openai.py:TF_DSV41_CALIBunset, median of 3, base against this head: fibonacci-raw 71.0 / 71.1 (T 0) and 65.4 / 65.5 (T 1), gpu-chat-no-think 56.7 / 56.7 (T 0) and 56.8 / 56.8 (T 1); table in the commentChecklist
cost_table_test.zig,calib_pass_test.zigandcalib_test.zig; see Tests run)."draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.--drafter, no slower than without it, greedy and sampled (withTF_DSV41_CALIBunset, the default;measureis opt-in and not no-slower everywhere, as above)