Skip to content

DeepSeek-V4.1 on Metal: opt-in draft cost table timed at start, default pricing unchanged (stacked on #1720) - #2952

Draft
gilby wants to merge 26 commits into
ashhart:mainfrom
gilby:dsv41-metal-opt/01-cost-table
Draft

gilby wants to merge 26 commits into
ashhart:mainfrom
gilby:dsv41-metal-opt/01-cost-table

Conversation

@gilby

@gilby gilby commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

What this changes

On Metal, DeepSeek-V4.1's draft depth rule prices its rounds from the CUDA-shaped built-in table. This adds a way to time a table on the loaded model and price from it, as an opt-in: with TF_DSV41_CALIB unset or blank on Metal nothing is timed and the built-in table prices the rounds exactly as on base (same startup lines, same load time, same depths, same tokens); naming measure times a table at start (about two seconds), checks it, reports it and prices the rounds from it, zig stores that table and reads it back on the device and checkpoint that stored it, cached and real read a stored one. A malformed TF_DSV41_CALIB* value stops a Metal start naming the setting, and a table whose pass costs more than twice its own 1-row window is refused. Pricing never changes a token.

The PR started as "price the rounds from a table measured at load" by default. On an M5 Ultra that made the code cells 1 to 6% faster and three chat cells 2 to 4% slower (repeated on a rerun), so the default stays the built-in table and the measured table is kept as an opt-in. The measurements are in the receipt comment on this head.

  • Issue: Refs DeepSeek-V4.1 on Metal: decode and prompt speed on M3 Ultra and M5 Ultra, token-exact (gate and order for the drafts) #2984
  • Stacked on: dsv41 metal: the lane target, the prompt cache and the native server over Metal #1720. This PR adds 9 commits on a43ad7b98: c8c30ea8f (price drafts from a cost table timed on the loaded model), 1a135017d (time a sparse set of windows and a warm pass), 562b276a3 (refuse a malformed timing setting, key a stored table by device and weights), afbc1ab33 (time over a two-row prompt, every window a lone stream can make), c0e24f554 (time the pass back to back, refuse an implausible pass), 962135265 (the slot term's windows keep their repeats), 6bf0ec9c0 and 55ae488ba (time nothing unless TF_DSV41_CALIB names a source), 284548471 (the refusal text names the Metal default as it is). Three commits since review: 1a3baab44 (the TF_DSV41_CALIB row describes each mode once), 0d892bbcb (the CHANGELOG line says what a user gets), a97716e2d (the optimization ledger row).
  • atc overlap --json: run on this head on 2026-11-12, no conflicts; base 3cacb821c (the merge base tools/pr_check.py reports).

Checks on this head

  • Head commit: a97716e2d (the receipts were measured at 55ae488ba on the M5 Ultra and 284548471 on the M3 Ultra; the commits since are documentation, the CHANGELOG and the ledger)
  • Lean checker, python3 tools/zig/lean_check.py, on the base and on this head: 345 and 345, new findings 0 (tools/pr_check.py 0 errors, 0 warnings)
  • Tests run, with the command and pass, fail and skip counts: at 284548471, zig build test-dsv41-draft -Dcpu=apple_m1 148 of 148, and zig build test-dsv41-metal-all, test-environment-host and test-dsv41-env-host 45 of 45 over 28 build steps. At a97716e2d (documentation only since): test-env-defaults-host 102/104 (2 skipped), test-environment-report-host 36/36, test-environment-host, test-dsv41-env-host 14/14, test-profile 21/21. Without the change: on this PR's base, cost_table_test.zig, calib_pass_test.zig and calib_test.zig do not build, since each uses what the PR adds (Plan.initWide, calib_measure.launch_bound, the stored table's id, Costs.plausible, the pass's ingest timing, metal.calib); the malformed-setting case in calib_test.zig got the default table instead of a refusal before 562b276a3.
  • CI: not run: the branch is still on dsv41 metal: the lane target, the prompt cache and the native server over Metal #1720's pre-landing head and conflicts with main (CHANGELOG.md), and per the order in DeepSeek-V4.1 on Metal: decode and prompt speed on M3 Ultra and M5 Ultra, token-exact (gate and order for the drafts) #2984 it is rebased onto dsv41 metal: decode experts decode an expert's weights once for all of its rows (stacked on #1720, #2973) #2983 at landing.
  • Not run, and why: three fresh TF_DSV41_CALIB=measure starts on the M5 Ultra (one is posted; they are best taken on the final kernels, at landing); CUDA (the shared files draft/costs.zig and draft/calib_env.zig also serve the CUDA engine; only the host tests that include them ran); cold-prompt cells (nothing in the prompt path changed, which is a reading of the diff and not a measurement)

Receipt

  • Main against branch: M5 Ultra, 512 GB, with TF_DSV41_CALIB unset: token hashes equal to base in every cell of every arm, and every cell level with base: mean over mean 0.998 to 1.002 in the 16 bench_concurrent cells (1 to 4 streams, greedy and sampled, chat and code) and the 4 bench_openai cells, 0.998 to 1.002 in the 8 --no-drafts cells, per pair 0.997 to 1.004. Cells under 0.98 of base: none. Startup lines, load time (3.5 to 3.6 s on both sides) and draft counters equal to base. With TF_DSV41_CALIB=measure (one run, opt-in): code +1 to +6%, chat 2.7 and 3.0% slower in two cells, load +2.1 s, tokens equal. All cells and the method are in the receipt comment. M3 Ultra, 512 GB, at 284548471, under the DeepSeek-V4.1 on Metal: decode and prompt speed on M3 Ultra and M5 Ultra, token-exact (gate and order for the drafts) #2984 rule: with TF_DSV41_CALIB unset all 12 cells within noise (0.998 to 1.001), token hashes equal, the server prints only the default cost line; with measure (opt-in) 5 cells faster, 3 slower (1.2 to 2.2%, repeated on the one rerun), 4 within noise. Receipts: https://gist.github.com/gilby/d7cf0e381966791ac5ec9dc686e95de2 (M5 Ultra), https://gist.github.com/gilby/8f3815c1164339f45cf8f9d6b2fa73f4 (M3 Ultra).
  • Boxes it reaches: Metal DeepSeek-V4.1 (deepseek_v41) on Apple GPUs; with the setting unset none of its behavior changes. The CUDA engine shares draft/costs.zig and draft/calib_env.zig: a stored table whose pass is dearer than twice its 1-row window is refused on read-back there too, and a stored table carries a device and weights identity that is 0 (the old file name and match) when a caller names none; CUDA was not run.
  • New switch? TF_DSV41_CALIB (existed for CUDA, now registered for Metal in platform/environment.zig): unset or blank is default on Metal (the built-in table, nothing timed) and cached on CUDA; accepted cached, default, real, measure or zig, in any case; anything else stops a Metal start naming the setting, the text and the accepted forms. Ten more TF_DSV41_CALIB_* settings (_DIR, _FILE, _SHAPE, _STAT, _REPS, _WARM, _RECHECK, _CYCLE, _DEEP_REPS, _BUDGET_MS) are registered the same way. docs/environment.md has a row for each; there is no docs/profiles.md row, since none of them selects a policy.
  • Environment, with the TensorFold commit, Zig and backend toolchain, machine and GPU, checkpoint and revision: base a43ad7b98, head 55ae488ba for the numbers and 284548471 pushed; Zig from .zig-version (0.17.0; the version used on the machine is not recorded in the logs); Metal toolchain TOOLCHAINS=com.apple.dt.toolchain.Metal.32023.922.4; M5 Ultra, 512 GB, macOS 27.2.1; deepseek-ai/DeepSeek-V4.1-Flash-1017 release weights at revision 1bae0e1088993b9b82de08f59de5be5a34e42170; M3 Ultra, 512 GB, for the second receipt
  • Paths and platforms the change reaches, which I ran and which I could not: ran the Metal M5 Ultra server (default drafts and --no-drafts, 1 to 4 streams, TF_DSV41_CALIB unset, measure, and a malformed value); ran the M3 Ultra server (unset and measure); could not run CUDA or another Metal chip
  • Exactness, from tools/bench_concurrent.py --alone --serial or token_sha against "draft": false: token_sha256 equal in every cell across base, this head, the opt-in arm and the discarded first runs (0 differences in 13 bench_concurrent arms and 5 bench_openai arms); --alone and --serial unequal 0 and failed 0 in every cell
  • Main against this branch, same machine and checkpoint, posted before asking for a run on our machines (or why you can't): posted in the receipt comment on this head
  • Decode speed before and after, from tools/bench_openai.py: TF_DSV41_CALIB unset, median of 3, base against this head: fibonacci-raw 71.0 / 71.1 (T 0) and 65.4 / 65.5 (T 1), gpu-chat-no-think 56.7 / 56.7 (T 0) and 56.8 / 56.8 (T 1); table in the comment
  • Prompt speed before and after: not run for this head; nothing in the prompt path is changed, and the opt-in timing runs once at load over slots it clears afterwards
  • Licence of any new weights or drafter, or "n/a": n/a, no new weights or drafter

Checklist

  • The title and description match the head commit, including the receipt and the stack.
  • New tests fail before the change, pass after it, and skip cleanly without their dependency (cost_table_test.zig, calib_pass_test.zig and calib_test.zig; see Tests run).
  • Comments and docstrings are one line. No measurements or history in the source.
  • No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.
  • Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
  • Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.
  • No precision traded for speed. If the bits change, the description says which sums change and why.
  • Decode and prompt processing are no slower than main at one stream and the widest admitted; with --drafter, no slower than without it, greedy and sampled (with TF_DSV41_CALIB unset, the default; measure is opt-in and not no-slower everywhere, as above)

gilby added 15 commits November 11, 2026 22:56
…over Metal

target.zig is the lanes' verify target: a slot's prompt, its windows of rows for several slots in one launch,
keeps and rewinds, and kept prompt states (the prompt cache) whose resumed rows the reply reports as cached_tokens.
boot.zig loads a release checkpoint folder and entry.zig serves it as tensorfold-dsv41 (dsv41-metal-serve).
Tests serve the tiny model through LaneHost: concurrent equals solo equals serial, drafted equals draft:false,
cache hits equal fresh runs, seeds and cancels.
…for the next

A shared round with the plain switch at it asked no drafts and ran no head
pass (core Rule.plain_streams, threaded from the family's resolved setting:
0 on CUDA, so its rounds are unchanged), and the family's speculation no
longer launches a pass for a round that wide (armSpec/speculate plainRound).
The depth rule priced Metal rounds with the CUDA-shaped defaults. Metal now
times dry windows of 1 to 16 rows, the second launch from row 17 and a DSpark
pass with its ingest at load, on a time budget, through the lane core's
calibration path. TF_DSV41_CALIB keeps its meaning; measure is the Metal
default, and TF_DSV41_CALIB_BUDGET_MS bounds the repeats. The prompt cache is
not fed and every slot is left as new.
@ashhart

ashhart commented Nov 11, 2026

Copy link
Copy Markdown
Owner

Thanks @gilby. Pricing from the machine's own table makes sense, and the tests cover the parts that matter for exactness: every slot's state hashes the same before and after the measurement, the stats and the prompt cache are untouched, and replies after a measured start equal the serial decode. Since the verify window keys each row's draw by its position, depth can only change speed. Asks:

  1. metal/calib.zig:103 turns every error from the measurement into "the default table stands in", including a malformed setting. TF_DSV41_CALIB_BUDGET_MS=abc or TF_DSV41_CALIB_REPS=0 start the server with a warning naming InvalidCharacter or CalibKnob, where CUDA's measure mode refuses them. Please read the timing method before the fallback so a bad value stops the start, and name the setting in the message. TF_DSV41_CALIB_BUDGET_MS also needs its row in platform/environment.zig and docs/environment.md.
  2. This changes the default at one and two streams, where drafts win, so the receipt needs to show no cell slower than the stack tip. Please also post the startup table line ("dsv41 calibration (zig): ...") from three fresh starts on each chip, so we can see how much a table moves between starts, and the load time it adds. For the A/B arms, TF_DSV41_CALIB=zig keeps one stored table across restarts, if you want to take the table out of the comparison.
  3. Target.dropWindow (metal/target.zig:428) says the slot is as it was before the window, but it doesn't rewind the ratio-2 carry the way keep does (:376). For calibration that's fine because wipe follows. Please say so in its comment (calibration only, wipe after), since it's public.
  4. A note for the body: the calibration log line is shared, so CUDA's boot line now prints rows up to 32 (with 17) where it printed 48 and 64. Nothing else on CUDA changes.

If #2950 or #2951 lands first: wipe should also clear #2950's filling and hit_on and the prompt's staged chunk. With TF_DSV41_ROUND_PROF=1, the calibration windows would land in the round profile, so it should be paused around the measurement.

These five all sit on #1720 and touch the same few fields, so they'll land in this order, each rebased on the one before: #2951 (the profiler, to read the others' receipts), #2950 (opt-in), #2954, #2953 (on #2951, so forward.zig keeps one run), then #2952 (so its table is measured on the final kernels). Against main they only conflict in CHANGELOG.md.

  • Synthena (Ash's Agent)

@gilby

gilby commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Real-weights receipt, and this is not ready. One M3 Ultra 512 GB, deepseek-ai/DeepSeek-V4.1-Flash-1017 release weights, --parallel 4 --context 1048576 --prompt-cache-gib 0, ReleaseFast, fresh server per arm, base is #1720 at a43ad7b98. Arms base, this, this, base; tools/bench_concurrent.py --levels 1,2,3,4 --tokens 128 --reps 3 --temperatures 0,1 --alone --serial. Every token_sha256 equals base.

The measured table on this chip: dsv41 calibration (zig): draft 62.37 ms, slot 7.26 ms, verify 1/2/4/6/8/16/17/24/32 rows 30.6, 41.1, 56.6, 74.9, 90.7, 169.5, 190.4, 254.7, 323.2 ms, measured in 12.7 s. The default table it replaces says 42 and 117 ms at 1 and 16 rows and 5.6 ms a pass.

Two problems:

  1. Calibration took 12.7 s, and load went from 5.1 to 6.4 s to 18.0 s. The budget did not hold on the real model.
  2. Two streams, greedy, got slower in both runs: code 54.4 and 54.5 against 57.4 to 59.4 tok/s (0.937), chat 53.0 and 52.9 against 54.8 to 55.9 (0.960). Every other cell is within 2% (0.98 to 1.005). The pass is priced at 62 ms, eleven times the default; I think the timed pass includes a prompt-sized ingest that a decode round never pays, so depth is under-chosen at two streams.

I'll fix both (a pass timed the way a round runs it, and a calibration that fits in about 2 s) and rerun before asking for anything. The table itself is useful: about 9 ms a row from 1 to 16 rows on an M3 Ultra, and only a small step at row 17 (169.5 to 190.4 ms, where the second launch starts).

…t table

The first measurement took 12.7 s and priced the pass at 62 ms. It now times
rows 1, 2, 4, 8 and 16, the 4 + 4 window, rows 17 and 32 and the pass once
each after an untimed window and pass, repeats the passes only while the
budget lasts, and interpolates the rows between. The startup line reports the
ingest share of the pass and the timed runs.
… table by device and weights

The Metal measurement now checks every TF_DSV41_CALIB* setting before it
times anything, so a malformed value stops the start naming the setting, its
text and the accepted forms; only a measurement that fails falls back to the
defaults. The settings have rows in platform/environment.zig and
docs/environment.md. A stored table carries the device and checkpoint
identity, so another chip or other weights never read it. dropWindow's
comment says it is for calibration, with wipe after.
@gilby

gilby commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Thanks. Head is now 562b276a3 (two commits on top of the one you read; not rebased). Your points, then the rerun.

  1. A malformed calibration setting now stops the start: metal/calib.zig checks every TF_DSV41_CALIB* setting through platform/environment.zig and reads the timing method before the measurement, and the refusal names the setting, its text and the accepted forms. Only a failure inside the measurement itself falls back to the default table with a warning. Rule rows (Metal scope) and docs/environment.md rows are in for TF_DSV41_CALIB, _DIR, _FILE, _SHAPE, _STAT, _REPS, _WARM, _RECHECK, _CYCLE, _DEEP_REPS and _BUDGET_MS, each saying what it does on Metal and on CUDA. A device test starts with TF_DSV41_CALIB_BUDGET_MS=abc, TF_DSV41_CALIB_REPS=0 and a bad _STAT and gets the refusal with the slot state untouched; without the check it got the default table.
  2. Target.dropWindow's comment now says calibration only, wipe follows, and the compressor carry is not rewound.
  3. On the CUDA boot line: at this head it is unchanged. CUDA's line comes from calib_gpu.boot, which has its own copy of the line code and which this PR does not touch; the row list changed only in calib_lanes.boot, whose only caller is Metal. What does change on CUDA: it also reads and checks TF_DSV41_CALIB_BUDGET_MS (unused there), and a stored table's file gains an "id": 0 field under the same file name. I'll put that in the body.
  4. Noted for the order you gave: wipe will clear dsv41 metal: prompts in pieces between the other streams' rounds (stacked on #1720) #2950's filling, hit_on and the staged chunk when it sits on dsv41 metal: prompts in pieces between the other streams' rounds (stacked on #1720) #2950, and the calibration's enter and leave are now one place each, so pausing dsv41 metal: TF_DSV41_ROUND_PROF times each decode round and the drafter's passes (stacked on #1720) #2951's profile around it is one call.

On TF_DSV41_CALIB=zig: it stores one table and reads it back on later starts, so it does take the table out of an A/B. Before this push it was not safe on Metal: the key had no chip or weights in it. The table now carries an id hashed from the chip name, the drafts flag, config.json and model.safetensors.index.json, and is read back only when it matches; CUDA's file names are unchanged. It still has no executable identity, which main's core/lanes/cost_cache.zig has. Moving this onto that cache is the right next step and I'd do it as its own change.

Rerun on one M3 Ultra 512 GB, release weights, at 1a135017d (the commit before the review fixes, which change no timing code), arms stack tip, this, this, stack tip, tools/bench_concurrent.py --levels 1,2,3,4 --tokens 128 --reps 3 --temperatures 0,1 --alone --serial. Every token_sha256 equals the stack tip's.

N T0 chat T0 code T1 chat T1 code
1 1.000 1.020 0.975 1.031
2 1.025 1.030 1.030 1.044
3 0.997 1.001 1.001 1.000
4 0.995 1.002 1.004 0.998

The two start lines: draft 6.42 ms (ingest 0.73), slot 7.96 ms, verify 1/2/4/6/8/16/17/24/32 rows 28.4, 37.5, 50.9, 66.6, 82.3, 152.0, 171.5, 226.7, 289.9 ms and draft 6.43 ms (ingest 0.68), slot 8.24 ms, verify ... 28.4, 37.5, 51.0, 66.7, 82.3, 152.3, 170.9, 227.8, 292.9 ms. So the table repeats within about 1% between starts, and the warm pass is 6.4 ms, not the 62 ms the first version timed cold.

What that fixes and what it does not. The two-stream loss is gone: two streams are 2.5 to 4.4% ahead of the stack tip, and one-stream code 2 to 3% ahead. Two things are still wrong: one cell is behind by more than noise (one stream, sampled chat, 46.1 and 46.5 against 47.5 and 47.5), and the calibration still reports 10.1 s over 11 timed runs, with load at 15.2 s against 5.1 s, though the timed windows add up to about 1.1 s. So most of that time is outside the timed runs and I have not found it yet. Not ready; I'll post again when both are closed, with three starts on each chip as you asked.

At this head on an M5 Max: test-dsv41-draft 144/144, test-dsv41-metal-all 41/41, test-environment-host and test-dsv41-env-host 38/38, lean_check.py 345 on base and head, pr_check.py --pr 2952 0 errors and 0 warnings.

…very window a lone stream can make

The measurement prefilled the calibration text into each slot twice and ran
thirty one-row windows to grow a continuation, which cost about nine seconds
on the release model outside the timed runs. It now prefills one short prompt
in each slot it needs once, takes the window tokens from the text itself,
and clears only the slots it used. Rows 1 to 6 are timed rather than
interpolated, since a lone stream's drafts make exactly those windows.
…t be one

The pass's entry was a single sample taken right after the largest windows,
so whatever the first ingest after them costs became the table's pass. The
pass is now run once untimed and five times back to back, as a round runs it,
and the fastest is kept; the first ingest after the windows is reported apart.
A table whose pass is dearer than twice its own 1-row window is refused when
measured and when read back from a stored file.
…budget

The slot term is the 4 + 4 window less the 8-row one, each a single sample
once the budget was spent. Both now run twice more side by side after the
passes, marked so the budget never drops them; only the other row counts are
optional.
…ults

On the M5 Ultra the measured table made code faster and sampled chat slower
in three cells, with the rule near-indifferent between the depths. The table
is still timed at start and reported, with its checks and settings, but the
default table prices the rounds unless TF_DSV41_CALIB names a source, so
served depths are unchanged. The lanes test and the Metal device test run
from the Metal test steps.
Unset or blank keeps the default table with nothing timed and no extra
startup line, so load time and behaviour are the base's. measure, zig,
cached and real time or read the table, check it, report it and price from
it. The table_prices switch is gone.
@gilby gilby changed the title dsv41 metal: the draft depth rule prices rounds from a table measured at load (stacked on #1720) DeepSeek-V4.1 on Metal: opt-in draft cost table timed at start, default pricing unchanged (stacked on #1720) Nov 12, 2026
@gilby

gilby commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

This push narrows the PR: with TF_DSV41_CALIB unset on Metal nothing is timed and the built-in table prices the rounds, as on base; measure (or zig, cached, real) times or reads a table, checks it, reports it and prices from it.

Commits since the last head afbc1ab33: c0e24f554 times the DSpark pass back to back (one untimed run, five timed, the fastest kept) and refuses a table whose pass costs more than twice its own 1-row window, when measured and when read back from a stored file; 962135265 keeps the slot term's windows and their repeats whatever the time budget; 6bf0ec9c0 and 55ae488ba make the table opt-in, so an unset or blank setting times nothing and adds no startup line; 284548471 changes one string, the parenthesis of the refusal text, so it names the Metal default as it is (default).

Why the default changed. On an M5 Ultra, pricing from the measured table at 962135265 made code cells faster and three chat cells slower. The first series (base, this head, this head, base; 1 to 4 streams) had chat at 1 stream sampled at 0.979 of base, chat at 2 streams greedy at 0.978 and bench_openai chat sampled at 0.968 (one run per arm there); 3 and 4 streams were level (0.999 to 1.001). The rerun, in the same order as every series here (first server discarded, then base, this, this, base), gives the same three cells again, each with both this-head runs under both base runs: chat 1 stream sampled 0.972, chat 2 streams greedy 0.981, bench_openai chat sampled 0.965. The code cells gained 5.2 and 5.9% (1 and 2 streams, sampled), 2.2 and 3.9% (greedy), and bench_openai fibonacci-raw 2.8% sampled and 1.3% greedy; chat 2 streams sampled gained 2.1% and bench_openai chat greedy 3.7%.

streams T load base 1 this 1 this 2 base 2 this 1 / base 1 this 2 / base 2 this / base (mean over mean)
1 1 chat 62.7 61.0 61.0 62.8 0.973 0.971 0.972
2 1 chat 70.6 72.6 71.7 70.7 1.028 1.014 1.021
1 1 code 68.8 72.0 72.8 68.9 1.047 1.057 1.052
2 1 code 67.1 71.0 71.0 67.0 1.058 1.060 1.059
1 0 chat 56.5 56.7 56.5 56.5 1.004 1.000 1.002
2 0 chat 76.9 76.0 75.0 77.0 0.988 0.974 0.981
1 0 code 60.5 62.3 61.2 60.4 1.030 1.013 1.022
2 0 code 81.5 84.7 84.6 81.5 1.039 1.038 1.039
prompt T base 1 this 1 this 2 base 2 this 1 / base 1 this 2 / base 2 this / base (mean over mean)
fibonacci-raw 1 65.5 68.2 66.5 65.4 1.040 1.016 1.028
fibonacci-raw 0 71.2 72.2 72.0 71.2 1.014 1.012 1.013
gpu-chat-no-think 1 56.9 54.8 54.9 56.8 0.964 0.966 0.965
gpu-chat-no-think 0 56.8 58.6 59.1 56.8 1.032 1.041 1.037

Two more checks on why. An unpushed variant that changed the acceptance estimate the depth rule uses did not recover the cells (chat 1 stream sampled 0.980, chat 2 streams greedy 0.981, bench_openai chat sampled 0.960; code +1.4 to +6.1%), so it was not kept; its tables are in the gist as run6-*. A fixed-depth sweep (TF_DSV41_DEPTH=static, TF_DSV41_DRAFT_DEPTH=k for k = 1 to 5, one run per depth, the rule on at the start and end of each series) on base and on 962135265 gives the same tok/s on both trees at every fixed depth (largest difference 0.3 tok/s), so the trees differ only in the rule's table. With the built-in table the rule is ahead of the best fixed depth by 1.4 to 7.9% in the eight bench_concurrent cells and between +8.3% and -0.7% in the four bench_openai cells (fixed depth 3 is 0.7% ahead in chat greedy). In chat at one stream, depths 2 and 3 are within 0.2 tok/s of each other (58.5 and 58.3 sampled, 53.5 and 53.7 greedy); the rule beats both because it chooses per round. With the measured table the rule keeps less of that lead in chat (1 stream sampled: 62.8 built-in table, 60.6 measured, best fixed depth 58.5 to 58.6; 2 streams greedy: 77.0, 75.2, best fixed depth 75.4 to 75.6) and more in code (2 streams greedy: 81.5, 84.6, best fixed depth 75.3 to 75.5). The measured table lowers the mean number of drafts a round in every request group of run5-counters.txt (chat sampled 2.40 to 1.85, chat greedy 2.03 to 1.62, code greedy 2.36 to 1.90). I did not find which part of the table causes the chat loss, and the cells were not tuned after seeing them. The tables, including the kept rate by position and confidence derived from the accept logs, are in run7-tables.txt (the raw accept logs, about 23 MB, are not in the gist).

Result: pricing from the measured table is not adopted for the default, and stays as an opt-in.

Receipt at the final head, TF_DSV41_CALIB unset against base. M3 Ultra, 512 GB: pending; the series is being re-measured under the noise rule in #2984 and follows in this thread.

  • Head: 284548471; measured at 55ae488ba, which differs from it by one string (the parenthesis of the refusal text; the run below printed the earlier wording, see the refusal section)
  • Base: a43ad7b98
  • Checkpoint: deepseek-ai/DeepSeek-V4.1-Flash-1017 release weights as published (pinned revision: not recorded in these runs)
  • Hardware: M5 Ultra, 512 GB; macOS 27.2.1; Metal toolchain TOOLCHAINS=com.apple.dt.toolchain.Metal.32023.922.4; the Zig and Xcode versions are not recorded in these logs
  • Tools: tools/bench_concurrent.py --tokens 128 --reps 3 --temperatures 1.0,0 --alone --serial at --levels 1,2, --levels 3,4 and (--no-drafts pair) --levels 1,4; tools/bench_openai.py --tokens 256 --reps 3 --temperatures 1.0,0
  • Server flags: --context 1048576 --parallel 4 --prompt-cache-gib 0 --vision --no-update-check, TF_EXPERIMENTAL=deepseek; one fresh server per cell; --no-drafts only in the pair named so; TF_DSV41_CALIB=measure only in the opt-in cell
  • Arm order: each piece (1 and 2 streams, then 3 and 4 streams, then bench_openai): base (first server, discarded), base 1, this 1, this 2, base 2; then the --no-drafts pair (this, base); then the opt-in cell; then the malformed-value start
  • Builds: base binary sha256 d4689bd4b9ea2581…, this head 7bb76d017a3962185… (full hashes in run8-binaries.sha256); the gist's file-name labels are explained in its README

Summary: token hashes equal to base in every cell of every arm (0 differences; --alone and --serial unequal 0, failed 0). With the setting unset this head is level with base: mean over mean 0.998 to 1.002 in all 20 cells (per pair 0.997 to 1.004) and 0.998 to 1.002 in the 8 --no-drafts cells. Startup lines are identical to base's in all three pieces (below), load time 3.5 to 3.6 s on both sides, and the draft counters from the server logs are identical (1 and 2 streams: 2991 rounds, 3177 accepted of 5248 drafted, except base 1 at 5247 drafted; 3 and 4 streams: 12277 rounds, 993 of 1666; bench_openai: 1527 rounds, 2554 of 4333).

Cells under 0.98 of base: none (unset). Opt-in measure, one run: two cells, chat at 1 stream sampled (0.970 and 0.968 of the two base runs) and chat at 2 streams greedy (0.973 and 0.971).

bench_concurrent aggregate tok/s, TF_DSV41_CALIB unset (all token hashes equal, all 16 cells reported)

streams T load base 1 this 1 this 2 base 2 this 1 / base 1 this 2 / base 2 this / base (mean over mean)
1 1 chat 62.7 62.7 62.6 62.8 1.000 0.997 0.998
2 1 chat 70.6 70.7 70.5 70.7 1.001 0.997 0.999
3 1 chat 74.9 74.9 74.9 74.9 1.000 1.000 1.000
4 1 chat 85.2 85.3 85.2 85.2 1.001 1.000 1.001
1 1 code 68.8 68.8 68.8 68.7 1.000 1.001 1.001
2 1 code 67.1 67.0 66.9 66.9 0.999 1.000 0.999
3 1 code 75.5 75.5 75.5 75.4 1.000 1.001 1.001
4 1 code 84.9 84.9 84.9 84.8 1.000 1.001 1.001
1 0 chat 56.4 56.4 56.5 56.5 1.000 1.000 1.000
2 0 chat 76.9 76.9 77.0 77.0 1.000 1.000 1.000
3 0 chat 93.0 93.0 93.0 93.0 1.000 1.000 1.000
4 0 chat 109.0 109.2 109.1 109.0 1.002 1.001 1.001
1 0 code 60.4 60.4 60.4 60.4 1.000 1.000 1.000
2 0 code 81.4 81.4 81.5 81.5 1.000 1.000 1.000
3 0 code 96.0 96.1 96.1 96.1 1.001 1.000 1.001
4 0 code 113.9 114.0 113.9 113.9 1.001 1.000 1.000

bench_openai decode tok/s (median of 3), unset (hashes equal in all four cells)

prompt T base 1 this 1 this 2 base 2 this 1 / base 1 this 2 / base 2 this / base (mean over mean)
fibonacci-raw 1 65.4 65.6 65.3 65.3 1.004 1.001 1.002
fibonacci-raw 0 71.0 71.1 71.1 71.1 1.001 1.000 1.000
gpu-chat-no-think 1 56.8 56.8 56.8 56.7 1.000 1.002 1.001
gpu-chat-no-think 0 56.7 56.7 56.7 56.7 1.001 1.000 1.000

--no-drafts, unset (one run each, this head then base; hashes equal)

streams T load base this this / base
1 1 chat 46.1 46.2 1.002
4 1 chat 85.3 85.4 1.001
1 1 code 46.2 46.3 1.002
4 1 code 86.7 86.8 1.001
1 0 chat 46.5 46.6 1.002
4 0 chat 109.2 109.3 1.001
1 0 code 46.7 46.6 0.998
4 0 code 114.0 114.1 1.001

Opt-in TF_DSV41_CALIB=measure, one run between the base runs of the 1 and 2 stream piece (so it has no repeat; hashes equal to base in all 8 cells; 3141 rounds, 3032 accepted of 4447 drafted against base's 2991 and 3177 of 5248)

streams T load base 1 measure (one run) base 2 measure / base 1 measure / base 2
1 1 chat 62.7 60.8 62.8 0.970 0.968
2 1 chat 70.6 71.8 70.7 1.017 1.016
1 1 code 68.8 72.8 68.7 1.058 1.060
2 1 code 67.1 70.9 66.9 1.057 1.060
1 0 chat 56.4 56.5 56.5 1.002 1.000
2 0 chat 76.9 74.8 77.0 0.973 0.971
1 0 code 60.4 61.2 60.4 1.013 1.013
2 0 code 81.4 82.3 81.5 1.011 1.010

Startup lines, base and this head with the setting unset (identical in the 1 and 2 stream, the 3 and 4 stream and the bench_openai pieces; <url> stands for the local test address):

info: dsv41 draft costs: default, verify 1 / 16 rows 42.000 / 117.000 ms, pass 5.600 ms
info: dsv41 depth: cost, cap 5, skip on, joint 1, drafts priced from the cost tables
[tensorfold] serving DeepSeek-V4.1-Flash-1017 at <url> (sampling: temperature 1.0, top_p 1.0; drafts: DSpark + copies; context: 1048576; loaded in 3.6s)

Opt-in measure (startup lines as printed; load 5.7 s against 3.5 to 3.6 s, so the timing costs 2.1 s at start):

info(dsv41): dsv41 calib method (zig): table raw, min of 3 after 1 warm-up(s), rows 1-4 cycled, rows 1-0 rechecked, rows 9+ 5 runs; 2 slots, 45 timed runs, a 220-token text; rows 1-4 kept 20.740 / 27.779 / 33.757 / 37.633 ms (rank 0)
info(dsv41): dsv41 calibration (zig): draft 4.66 ms (ingest 0.39, first 0.43), slot 0.00 ms, verify 1/2/4/6/8/16/17/24/32 rows 20.7, 27.8, 37.6, 49.5, 59.0, 110.0, 128.8, 165.9, 208.3 ms, measured in 2.09 s over 21 timed runs (tokenizer and text 0.03 s, 5 prompt rows in 0.84 s, 19 windows and passes in 1.25 s) (not stored)
info: dsv41 draft costs: measured, verify 1 / 16 rows 20.740 / 109.972 ms, pass 4.655 ms
info: dsv41 depth: cost, cap 5, skip on, joint 1, drafts priced from the cost tables
[tensorfold] serving DeepSeek-V4.1-Flash-1017 at <url> (sampling: temperature 1.0, top_p 1.0; drafts: DSpark + copies; context: 1048576; loaded in 5.7s)

Refusal of a malformed value. TF_DSV41_CALIB=bogus stopped the start with exit status 1. This run printed the wording of 55ae488ba, tensorfold: TF_DSV41_CALIB="bogus": expected cached, default, real, measure or zig, in any case (unset or blank: measure on Metal), which was wrong for the new default; 284548471 corrects the parenthesis in zig/src/platform/environment.zig to (unset or blank: default on Metal) and changes nothing else. The corrected text was not run through the server.

Method as run: each cell is a fresh server started just before it, and the first server of each piece is run and discarded. Base is measured on both sides of this head (base, this, this, base). A thermal-pressure check (powermetrics) ran at the start and the end of every cell, and a cell that was not Nominal at either end would have been discarded and re-run after a pause; none was (38 of 38 samples Nominal, mediaanalysisd at 0.0% CPU at every sample). A control (base, --no-drafts, 1 stream, greedy) ran before the series at 46.8 tok/s against a gate of at least 42.

Not run: M3 Ultra (pending, above); CUDA; cold-prompt cells for this head (nothing in the prompt path is changed, which is a reading of the diff and not a measurement); the kernel contract and device tests on this head (no GPU kernel is new or changed).

Raw JSON, server logs and driver scripts: https://gist.github.com/gilby/d7cf0e381966791ac5ec9dc686e95de2

@ashhart

ashhart commented Nov 12, 2026

Copy link
Copy Markdown
Owner

Thanks @gilby. Narrowing this to an opt-in is the right call, and the evidence for it is clear. Three chat cells were slower with the measured table in both series (for example, 1 stream sampled at 0.972 and bench_openai chat sampled at 0.965). The fixed-depth sweep shows the two trees differ only in the rule's table. With TF_DSV41_CALIB unset, the M5 Ultra receipt is level with base: 0.998 to 1.002 in all 20 cells and in the 8 --no-drafts cells. The startup lines, load time and token hashes are all equal, and the draft counters match base's (one base run drafted one token fewer). We're happy to land it as an opt-in that changes no token.

Small things on 284548471:

  1. The TF_DSV41_CALIB row in docs/environment.md describes the Metal modes twice: once after "On Metal", then again from "default prices rounds from the built-in table". Please keep one description.
  2. The CHANGELOG line mostly describes the timing method. Per rule 4 on DeepSeek-V4.1 on Metal: decode and prompt speed on M3 Ultra and M5 Ultra, token-exact (gate and order for the drafts) #2984, please say what a user gets instead. It's opt-in, and on the M5 Ultra it made code 1 to 6% faster and three chat cells 2 to 4% slower, so the default stays the built-in table. Please add the M3 Ultra numbers when you have them.
  3. The ledger row you quote in the body isn't in the diff yet (docs/optimization-ledger.md).
  4. Three checklist boxes are still open: tests that fail before the change, one-line comments, and no slower than main. Please name which of cost_table_test.zig, calib_pass_test.zig and calib_test.zig go red without the change, and tick the first two once they hold. The third can wait for the M3 Ultra series.
  5. When this rebases onto the PRs before it, the two notes from our first review still apply. wipe should clear dsv41 metal: prompts in pieces between the other streams' rounds (stacked on #1720) #2950's filling, hit_on and the staged chunk, and dsv41 metal: TF_DSV41_ROUND_PROF times each decode round and the drafter's passes (stacked on #1720) #2951's round profile should pause around the measurement.
  6. The M3 Ultra series under the noise rule at the final head. Please also post the measure startup table line from three fresh starts on each chip, so we can see how far a table moves between starts.

One pointer, not a blocker: Boot.identity (metal/boot.zig) reads and hashes config.json and the weight index on every Metal open. That includes runs with TF_DSV41_CALIB unset, where nothing uses the result. Load time measured level, so it's cheap. Computing it only when the mode reads or stores a table would keep the default path exactly as before.

In the order, this stays after #2983, so that its table is measured on the final kernels.

  • Synthena (Ash's Agent)

@gilby

gilby commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

M3 Ultra, 512 GB: the receipt promised above, run on 2026-11-13 under the noise rule in #2984 (one discarded first run per series, base on both sides, both-under-both test).

  • Head: 284548471
  • Base: a43ad7b98
  • Checkpoint: deepseek-ai/DeepSeek-V4.1-Flash-1017 release weights as published, read from local disk (pinned revision: not recorded in these runs)
  • Hardware: M3 Ultra, 512 GB; the macOS version, Metal compiler, Xcode and Zig versions are not recorded in these logs
  • Tools, from the base tree in every arm: tools/bench_concurrent.py --tokens 128 --reps 3 --alone --serial at 1 stream (temperatures 1 and 0) and at 4 streams (temperature 0 and temperature 1 as separate series); tools/bench_openai.py --tokens 256 --reps 3 at temperature 0 and 1
  • Server flags: --context 1048576 --parallel 4 --prompt-cache-gib 0 --vision --no-update-check, TF_EXPERIMENTAL=deepseek; one fresh server per cell; every TENSORFOLD_PREFILL_* and TF_DSV41_* setting is unset except TF_DSV41_CALIB where the series names it
  • Arm order, per series: one discarded run (base), base, this, this, base; a cell starts only at Nominal thermal pressure after a 60 s settle and its end level is recorded; a cell that ends at Heavy is discarded and retried once (none did)
  • Server binary (sha256 prefix): base 0a5638a8e95e, this ac63111bad6f

Summary: two series, (a) TF_DSV41_CALIB unset against base, (b) TF_DSV41_CALIB=measure against base. (a) With the setting unset this head times nothing at start and equals base: all 12 cells are within noise, this over base 0.998 to 1.001 (mean over mean), and the server prints only the default cost line. (b) With measure, the 4-stream cells are within noise (0.998 to 1.001), 5 cells are faster, and three cells are slower by the #2984 rule and stayed slower on the one allowed rerun: 1 stream, temperature 0, code (0.984 first run, 0.983 rerun); 1 stream, temperature 1, chat (0.981, 0.978); bench_openai temperature 1, gpu-chat-no-think (0.988, 0.986). That is why the measured table stays opt-in and off by default. Token hashes are equal across all four runs in every cell of both series and of the rerun, with 0 failed requests and 0 --alone / --serial mismatches. Nothing ended at Heavy and nothing was discarded.

Cells under 0.98 of base: none in series (a). In series (b) one, in the rerun only: 1 stream, temperature 1, chat at 0.978 (its first run was 0.981).

(a) TF_DSV41_CALIB unset against base

Order as run: base 1, this 1, this 2, base 2. Ratios: this 1 over base 1, this 2 over base 2, and mean over mean. Verdict by the #2984 rule: slower only if both this runs are under both base runs and the mean gap is over 1%, faster the mirror image, otherwise within noise. bench_concurrent aggregate tok/s and bench_openai decode tok/s (median of 3 requests; the bench_openai rows are 1 stream and load names the prompt). All 12 cells reported: 0 slower, 0 faster.

streams T load base 1 this 1 this 2 base 2 this 1 / base 1 this 2 / base 2 this / base (mean over mean) verdict end level (b1 / t1 / t2 / b2)
1 0 chat 43.7 43.7 43.8 43.7 1.000 1.002 1.001 within noise Nom/Nom/Nom/Nom
1 0 code 47.1 47.1 47.1 47.1 1.000 1.000 1.000 within noise Nom/Nom/Nom/Nom
1 1 chat 47.8 47.7 47.8 47.7 0.998 1.002 1.000 within noise Nom/Nom/Nom/Nom
1 1 code 52.1 52.0 51.9 52.0 0.998 0.998 0.998 within noise Nom/Nom/Nom/Nom
4 0 chat 78.7 78.6 78.6 78.6 0.999 1.000 0.999 within noise Nom/Nom/Nom/Nom
4 0 code 84.0 84.1 83.9 83.9 1.001 1.000 1.001 within noise Nom/Nom/Nom/Nom
4 1 chat 61.7 61.4 61.7 61.5 0.995 1.003 0.999 within noise Mod/Mod/Mod/Mod
4 1 code 62.3 62.3 62.2 62.5 1.000 0.995 0.998 within noise Mod/Mod/Mod/Mod
1 0 fibonacci-raw 54.5 54.5 54.5 54.4 1.000 1.001 1.001 within noise Nom/Nom/Nom/Nom
1 0 gpu-chat-no-think 43.8 43.8 43.8 43.8 1.000 1.001 1.000 within noise Nom/Nom/Nom/Nom
1 1 fibonacci-raw 49.9 49.9 49.9 49.9 1.000 1.000 1.000 within noise Mod/Nom/Mod/Nom
1 1 gpu-chat-no-think 43.7 43.7 43.6 43.7 1.002 0.998 1.000 within noise Mod/Nom/Mod/Nom

End level key: Nom = Nominal, Mod = Moderate, Heavy = Heavy.

Exactness: token hashes equal across all four runs in every one of the 12 cells, 0 failed requests, 0 --alone / --serial mismatches, checked against the raw JSON of every run.

Startup line. With the setting unset, all 25 servers of series (a) print exactly one cost line, the default one, and nothing else about calibration; no log of the series mentions calibration. Verbatim from one of them:

info: dsv41 draft costs: default, verify 1 / 16 rows 42.000 / 117.000 ms, pass 5.600 ms

(b) TF_DSV41_CALIB=measure against base (information for the opt-in)

Same order, rule and columns. The first run of each series (all 12 cells), then the rerun of the two series that had slower cells.

streams T load base 1 this 1 this 2 base 2 this 1 / base 1 this 2 / base 2 this / base (mean over mean) verdict end level (b1 / t1 / t2 / b2)
1 0 chat 43.7 44.8 44.1 43.7 1.025 1.009 1.017 faster Mod/Mod/Mod/Nom
1 0 code 47.1 46.3 46.4 47.1 0.983 0.985 0.984 slower Mod/Mod/Mod/Nom
1 1 chat 47.8 46.9 46.9 47.8 0.981 0.981 0.981 slower Mod/Mod/Mod/Nom
1 1 code 52.1 54.5 54.2 52.1 1.046 1.040 1.043 faster Mod/Mod/Mod/Nom
4 0 chat 78.2 78.5 78.4 78.6 1.004 0.997 1.001 within noise Nom/Nom/Mod/Mod
4 0 code 84.0 83.9 83.9 83.7 0.999 1.002 1.001 within noise Nom/Nom/Mod/Mod
4 1 chat 61.6 61.6 61.7 61.6 1.000 1.002 1.001 within noise Mod/Mod/Mod/Mod
4 1 code 62.3 62.2 62.3 62.4 0.998 0.998 0.998 within noise Mod/Mod/Mod/Mod
1 0 fibonacci-raw 54.5 55.1 55.2 54.4 1.012 1.015 1.013 faster Nom/Nom/Nom/Nom
1 0 gpu-chat-no-think 43.8 44.9 44.7 43.8 1.026 1.023 1.024 faster Nom/Nom/Nom/Nom
1 1 fibonacci-raw 49.9 51.5 51.3 50.0 1.032 1.025 1.029 faster Mod/Mod/Nom/Mod
1 1 gpu-chat-no-think 43.8 43.4 43.1 43.7 0.991 0.985 0.988 slower Mod/Mod/Nom/Mod

Verdicts, first run: 5 faster, 3 slower, 4 within noise. Exactness: token hashes equal across all four runs in every one of the 12 cells, 0 failed, 0 mismatches.

Rerun. The 1-stream series and the bench_openai temperature 1 series each had a slower cell, so each got the one allowed rerun (four fresh servers each, base, this, this, base, same rule; the rerun has no discarded first run). The three slower cells repeated, each with both this runs under both base runs:

cell first run: base 1, base 2 first run: this 1, this 2 first run this / base rerun: base 1, base 2 rerun: this 1, this 2 rerun this / base
1 stream, T 0, code 47.1, 47.1 46.3, 46.4 0.984 (0.983, 0.985) 47.1, 47.1 46.3, 46.3 0.983 (0.983, 0.983)
1 stream, T 1, chat 47.8, 47.8 46.9, 46.9 0.981 (0.981, 0.981) 47.8, 47.8 46.6, 46.9 0.978 (0.975, 0.981)
bench_openai, T 1, gpu-chat-no-think 43.8, 43.7 43.4, 43.1 0.988 (0.991, 0.985) 43.7, 43.8 43.0, 43.3 0.986 (0.983, 0.989)

All 6 rerun cells, in full:

streams T load base 1 this 1 this 2 base 2 this 1 / base 1 this 2 / base 2 this / base (mean over mean) verdict end level (b1 / t1 / t2 / b2)
1 0 chat 43.7 44.4 44.4 43.7 1.016 1.016 1.016 faster Mod/Nom/Mod/Mod
1 0 code 47.1 46.3 46.3 47.1 0.983 0.983 0.983 slower Mod/Nom/Mod/Mod
1 1 chat 47.8 46.6 46.9 47.8 0.975 0.981 0.978 slower Mod/Nom/Mod/Mod
1 1 code 52.2 54.5 54.4 52.1 1.044 1.044 1.044 faster Mod/Nom/Mod/Mod
1 1 fibonacci-raw 49.8 51.0 51.1 49.9 1.024 1.023 1.024 faster Mod/Nom/Mod/Mod
1 1 gpu-chat-no-think 43.7 43.0 43.3 43.8 0.983 0.989 0.986 slower Mod/Nom/Mod/Mod

The rerun has 3 faster cells, 3 slower (1 stream T0 code, 1 stream T1 chat, openai T1 gpu-chat-no-think) and 0 within noise; token hashes equal in all 6, 0 mismatches. The faster cells are the same in the first run and the rerun (1 stream T0 chat 1.017, 1.016; 1 stream T1 code 1.043, 1.044; bench_openai T1 fibonacci-raw 1.029, 1.024); bench_openai temperature 0 is faster in both prompts (1.013 and 1.024) and was not rerun. Code and one chat cell gain, chat cells at one stream and the sampled bench_openai chat cell lose: pricing the rounds from the measured table is not a uniform win, which is why it is not the default.

What the measured table printed. The first measure server of the 1-stream series prints these three lines at start (verbatim):

info(dsv41): dsv41 calib method (zig): table raw, min of 3 after 1 warm-up(s), rows 1-4 cycled, rows 1-0 rechecked, rows 9+ 5 runs; 2 slots, 45 timed runs, a 220-token text; rows 1-4 kept 27.616 / 37.973 / 42.816 / 48.630 ms (rank 0)
info(dsv41): dsv41 calibration (zig): draft 6.27 ms (ingest 0.57, first 92.43), slot 0.00 ms, verify 1/2/4/6/8/16/17/24/32 rows 27.6, 38.0, 48.6, 67.0, 80.2, 148.1, 175.6, 228.6, 289.3 ms, measured in 4.10 s over 19 timed runs (tokenizer and text 0.04 s, 5 prompt rows in 2.40 s, 17 windows and passes in 1.71 s) (not stored)
info: dsv41 draft costs: measured, verify 1 / 16 rows 27.616 / 148.064 ms, pass 6.270 ms

Over the 14 measure servers (every this server of series (b), rerun included) the printed values range: verify 1 row 27.528 to 28.253 ms, verify 16 rows 146.287 to 148.198 ms, DSpark pass 6.247 to 6.317 ms, calibration time 4.09 to 4.16 s; the built-in default table prints 42.000 / 117.000 ms and 5.600 ms. No server refused its measured table.

Discarded, retried and flagged cells

  • The first run of every series, in both (a) and (b), is the discarded run and is in no table; none ended at Heavy (end levels in endstates.txt).
  • The rerun of two (b) series is the one rerun described above, allowed because of the slower cells; it is reported in full and replaces nothing: the first run is kept in the tables too.
  • No cell ended at Heavy, so none was retried for thermal reasons and none is flagged. Kept cells that ended above Nominal are the ones marked Mod in the tables.

Method as run: each cell is a fresh server started after the previous one was stopped and a 60 s settle, and a cell starts only when the powermetrics thermal sampler reports Nominal; the level is recorded again at the end of the cell. One run per series is discarded and not in any table, base is on both sides, and the series are per stream count and temperature (bench_concurrent) and per temperature (bench_openai). Moderate does not discard a cell; Heavy would. Tools and server flags are those in the header. Every arm, the raw JSON, the server logs and the driver scripts are in the gist.

Not run: measure against base at 2 and 3 streams; the other opt-in settings (zig, cached, real); the checkpoint's pinned revision, the macOS version and the toolchain versions are not recorded; kernel-contract runs are not part of this series' data.

Raw JSON, server logs and driver scripts: https://gist.github.com/gilby/8f3815c1164339f45cf8f9d6b2fa73f4

@gilby

gilby commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

New head: a97716e2d (three commits on top of 284548471, not rebased).

Your "small things" of 12 Nov 23:50, point by point:

  1. The TF_DSV41_CALIB row describing the Metal modes twice: 1a3baab44 keeps one description. Unset or blank is cached on CUDA and default on Metal; default prices from the built-in table with nothing timed, measure times, reports and prices from a table, zig reads this engine's stored table and times and stores one when there is none, cached and real read a stored one; on Metal a stored table also matches the device and checkpoint that stored it.
  2. The CHANGELOG line: 0d892bbcb says what a user gets instead of the timing method, and cites DeepSeek-V4.1 on Metal: decode and prompt speed on M3 Ultra and M5 Ultra, token-exact (gate and order for the drafts) #2984 rather than DeepSeek-V4.1 on Metal: a stack on main's deepseek_v41 family for the release checkpoint (one M3 Ultra) #1676: TF_DSV41_CALIB=measure is opt-in; on the release weights it made code 1 to 6% faster and three chat cells 2 to 4% slower on the M5 Ultra, and five cells 1 to 4% faster and three 1 to 2% slower on the M3 Ultra, so unset or blank keeps the built-in table with nothing timed, and served depths, tokens and load time are as before.
  3. The ledger row: a97716e2d adds it under DSpark drafters in docs/optimization-ledger.md: deepseek_v41 (M), timed cost table, opt-in, M5 Ultra code +1..+6 % and three chat cells −2..−4 %, M3 Ultra five cells +1..+4 % and three −1..−2 %.
  4. The three checklist boxes. All three new test files go red without the change: on this PR's base, cost_table_test.zig, calib_pass_test.zig and calib_test.zig fail to build, because each uses what the PR adds (Plan.initWide, calib_measure.launch_bound, the stored table's id, Costs.plausible, the pass's ingest timing, metal.calib), and with the PR they pass (test-dsv41-draft 148/148; test-dsv41-metal-all with the environment steps 45/45 at 284548471). Beyond building, the malformed-setting test in calib_test.zig got the default table instead of a refusal before 562b276a3, and the warm-pass test in calib_pass_test.zig covers the cold first ingest behind the first M3 Ultra table's 62 ms pass. Comments are one line (the lean count is 345 on base and head) and carry no measurements; the numbers in calib_pass_test.zig are the fixture's synthetic times. The no-slower box: with TF_DSV41_CALIB unset, both chips' receipts are level with base in every cell. The body ticks all three and names the tests.
  5. wipe clearing dsv41 metal: prompts in pieces between the other streams' rounds (stacked on #1720) #2950's filling, hit_on and the staged chunk, and pausing dsv41 metal: TF_DSV41_ROUND_PROF times each decode round and the drafter's passes (stacked on #1720) #2951's round profile around the measurement: neither exists on this branch's base, so both are changes for the rebase onto dsv41 metal: prompts in pieces between the other streams' rounds (stacked on #1720) #2950 and dsv41 metal: TF_DSV41_ROUND_PROF times each decode round and the drafter's passes (stacked on #1720) #2951 at landing. They are small: wipe resets those three fields with the rest of the slot, and the calibration's enter and leave are one place each, so the profile is paused and resumed there. They go in when this PR sits on those two.
  6. The M3 Ultra series under the noise rule: posted on 13 Nov. The measure startup lines: the M3 Ultra receipt gives the range over 14 fresh starts (verify 1 row 27.53 to 28.25 ms, 16 rows 146.29 to 148.20 ms, pass 6.25 to 6.32 ms, 4.09 to 4.16 s to measure). The M5 Ultra has one start posted; three fresh starts there are still owed. Since this PR comes after the kernel PRs so its table is measured on the final kernels, those starts are best taken on the rebased stack at landing; I'll post them then.

The Boot.identity pointer is not changed in this push; it reads config.json and the weight index once at open and measured level.

Receipts: the three new commits are documentation, the CHANGELOG and the ledger, so the M5 Ultra and M3 Ultra receipts above (taken at 55ae488ba and 284548471) describe this head's code. The checkpoint in both is deepseek-ai/DeepSeek-V4.1-Flash-1017 at revision 1bae0e1088993b9b82de08f59de5be5a34e42170.

At a97716e2d on an M5 Max: test-env-defaults-host 102/104 (2 skipped), test-environment-report-host 36/36, test-environment-host, test-dsv41-env-host 14/14 and test-profile 21/21 pass; lean_check.py 345 on base and head; pr_check.py --pr 2952 0 errors and 0 warnings.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants