[jschmied]

What this changes

A new page, docs/optimization-ledger.md: which speed and memory optimization applies to which model family, and where each one stands, so a change proven on one family gets ported to the others instead of re-invented.

• One table of 27 optimizations (rows) by 21 native families (columns). Each cell is yes (applies, not done), no (with a short reason), done, failed (tried, no gain or rejected) or open, with the backend where it matters and a link to the PR or issue. Four cells are ? (add-plus-norm on deepseek_v41, lfm3 and kimi_k3; shared draft rounds on deepseek_v41), for whoever knows. To stay within 120 columns, the table is split into five column groups with the same rows.
• After the table, a numbered list: how each optimization works and what it saves, with per-family gain tables where PRs measured them (fused projections, copy drafts, shared draft rounds, solo chunks, read-ahead, direct load, L2 prefetch), and the prompt tail's per-family estimates from qwen3_5 cuda: a prompt chunk's last layer runs q, attention, o and the MLP over the kept row's tile #2229's survey, marked as estimates.
• A rough-gain table: each optimization's estimated effect on prefill, decode and memory (low under 3 %, medium 3–15 %, big over 15 %, or what lets a case fit), for planning a port. Measured numbers stay in the list.
• The intro points at the shared helpers a port builds on. Two are not on main yet and say so: core/prompt_tail.zig (qwen3_5 cuda: a prompt chunk's last layer runs q, attention, o and the MLP over the kept row's tile #2229) and core/state_walk.zig (core: one walk for kept prompt states, GLM on it (CUDA and Metal) #2215).

The page asks that a PR changing a cell updates it in the same PR.

• Issue: n/a, documentation only.
• Stacked on: main 01e13a04a, one commit.
• atc overlap --json: no conflicts.

When a cited PR lands

These cells say open and flip to done when their PR lands:

• qwen3_5 cuda: a prompt chunk's last layer runs q, attention, o and the MLP over the kept row's tile #2229: row 1 (q3.5).
• CUDA: #1928's load read-ahead on Nemotron, Bolt, MiniMax M3, Kolibri-1, Spark 2.5 and DeepSeek-V4.1, opt-in (#2003) #2033: row 21 (nemo, and the kol1, spk25, mm3 and dsv41 cells).
• core: one walk for kept prompt states, GLM on it (CUDA and Metal) #2215: row 5. qwen CUDA: an integrated GPU keeps prompt states in device memory, without pinning a copy each save (#2043) #2113: row 4. qwen3_5 cuda: the 27B as a pipeline stage, logits byte-equal to the whole model #2115: row 26. GLM-5.3-Flash on CUDA keeps conversation states on disk, every rank's part (--learn-conversations) #1411: row 8 (glm, CUDA). KV pool: host tier for the core pool (5/8) #1501: row 7. metal nemotron: TF_KV_SHARE keeps prompt states as core pool pages (stacked on #1837) #1850: row 3 (nemo, Metal).
• glm cuda: TF_GLM_L2PF reads a decode window's next weights into L2 on a side stream, on by default on a GB10 #2162: row 24 (glm). In the later parts: minimax_m3 (CUDA): draft several streams' DSpark blocks in one pass #1426 (row 19) and DeepSeek-V4.1: read the pack's weight spans past the page cache (core/direct_io) #1924 (row 22) in the second, lfm2: DSpark drafting with owned context and confidence scheduling #1394 (row 17) and lfm2_moe CUDA: attention pages from the shared KV pool, and the prompt cache (stacked on #1492) #1503 (rows 2 and 3) in the third, qwen27: serve the 27B pair, refuse what a pair cannot keep, and die with the peer #1434 (row 25) in the fourth, kimi: K3 and K3.1 through the native server on Metal #2221 (row 17) in the fifth.

Checks on this head

• Head commit: 93c8a2be8.
• Lean checker: 425 problems on this head, 0 new. git diff --check clean. No line over 120 columns.
• Every link (117) was checked against the repository's PR and issue lists; every path in the intro exists on main except the two marked as not yet there.
• Not run, and why: no build or tests; the page cannot reach inference.

Receipt

n/a: documentation only. Corrections welcome, cell by cell.

[ashhart]

Thanks @jschmied, a page that says which family has which optimization, with every cell linked, is something we've needed. The links and switch names check out against main. A few corrections:

1. The prompt_tail.zig bullet names your branch; please drop the parenthesis, so it ends "it is on [qwen3_5 cuda: a prompt chunk's last layer runs q, attention, o and the MLP over the kept row's tile #2229]".
2. Row 8, glm: main's Metal server already keeps GLM-5.3-Flash's conversation states on disk (native/metal.zig lists it with Qwen3.8-27B, Nemotron and Flash Next), so that cell is done M¹ open C[#1411]. Nemotron's cell should gain M¹ for the same reason.
3. Rows 18, 23 and 27 cite Flash Next on CUDA: copy drafts where a reply repeats earlier text #468, Flash Next on CUDA: drop the checkpoint from the page cache after load (fixes #1868) #1880, CUDA: Qwen4 and tensor-parallel loaders drop the checkpoint's pages after every rank is ready #1887 and fix(profile): report Nemotron CUDA paging overrides #2061 as done, but none of them has a CHANGELOG line, and none is open. Could you check each one landed? If not, done¹ or the PR that did it.
4. GLM CUDA: opt-in context copies priced by the lane host #2146 has landed, and CUDA: #1928's load read-ahead on Nemotron, Bolt, MiniMax M3, Kolibri-1, Spark 2.5 and DeepSeek-V4.1, opt-in (#2003) #2033 and qwen3_5 cuda: a prompt chunk's last layer runs q, attention, o and the MLP over the kept row's tile #2229 are queued, so rows 1, 18 and 21 will flip soon. A note in the PR helps whoever lands them update the page.

[jschmied]

Thanks for checking the cells. All four are in 93c8a2be8:

1. The prompt_tail.zig bullet now ends "it is on [qwen3_5 cuda: a prompt chunk's last layer runs q, attention, o and the MLP over the kept row's tile #2229]".
2. Row 8: glm is done M¹ open C[#1411], and nemotron gained M¹.
3. All four landed, so the cells stay done with their PRs: Flash Next on CUDA: copy drafts where a reply repeats earlier text #468 (10-07), Flash Next on CUDA: drop the checkpoint from the page cache after load (fixes #1868) #1880 (11-01), CUDA: Qwen4 and tensor-parallel loaders drop the checkpoint's pages after every rank is ready #1887 (11-04) and fix(profile): report Nemotron CUDA paging overrides #2061 (11-04). None of them added a CHANGELOG line.
4. GLM CUDA: opt-in context copies priced by the lane host #2146 has landed, so row 18's glm cell is done [#2146]. The PR description now lists every open PR the page cites, with its rows, so whoever lands one knows which cells to flip.

To keep the page current, I'd propose one pre-flight line in docs/pr/SKILL.md: "If this PR adds, finishes or drops an optimization for a family, update that family's cell in docs/optimization-ledger.md." Should I add it in this PR, or in its own? Until then I'll flip the cells myself as their PRs land.

[ashhart]

Thanks @jschmied, all four are in, and listing every open PR the page cites with its rows is a good way to keep it current. One small thing: "[#2211] moves GLM onto the shared store" reads as done; please mark it "(open)" like the others. On the docs/pr/SKILL.md line: yes please, but in its own small PR, since it changes contributor instructions rather than the ledger. With that, this is ready to land.

[jschmied]

Done in 5e8895e26: #2211 now reads "(open)". The docs/pr/SKILL.md line is its own PR, #2243. It links the ledger page, so it should land after this one.

[ashhart]

Thanks @jschmied. Queued for after the 1.2.3 cut, with #2243 after it.

[jschmied]

One small correction before it lands, in 2bd212bab: row 3's Flash Next cell now reads yes (host store), and the prompt-store paragraph says why. Flash Next already resumes a prefix from the host store's whole-state copies, so the shared store would save memory (about 0.9 GiB of attention rows per kept 32k state, 27 KiB a token per cuda_snap.zig's test), not add resuming. Docs only; nothing else changed.

[Maxrio9]

One Kimi K3/K3.1 mapping correction before this lands: in the current head, the Kimi cells appear reversed. Row 17 (DSpark) says done [#2221], while row 18 (copy drafts) says yes. Could row 17 stay yes and row 18 become done [#2221]?

#2221 landed the native Metal path with copy_drafts = true, while the backend rejects nonzero draft-head depth. That leaves the release's DSpark drafter as a future port. Ledger rows 17–18 on this PR head · copy capability

[ashhart]

Thanks @jschmied. The Flash Next correction in row 3 reads right. Three lines no longer match main, so please fix them before it lands:

1. Part 2 says qwen CUDA: a faster opt-in paged path (TF_QWEN_KV_PAGES=256), --parallel a cap with pages set #2011 "makes them the default". qwen CUDA: a faster opt-in paged path (TF_QWEN_KV_PAGES=256), --parallel a cap with pages set #2011 keeps pages opt-in and makes --parallel a cap with pages set.
2. Part 3 lists TF_GLM_KV_SHARE, which now stops the server at start (TF_KV_SHARE instead), and calls glm cuda: TF_KV_SHARE keeps GLM's prompts in the shared prompt store, and GLM's own store goes #2211 open; it has landed.
3. qwen3_5 cuda: a prompt chunk's last layer runs q, attention, o and the MLP over the kept row's tile #2229 has landed: row 1's q3.5 cell and the intro's prompt_tail.zig line.

The 08:16 push also changed rows 4, 12, 19, 22 and 24 and part 2's row 2. Please list those in a comment so whoever lands it can check them.

[jschmied]

Thanks @ashhart and @Maxrio9. All in, on 27cb4f927, on top of 0d40ea8bf.

1. Part 2: qwen CUDA: a faster opt-in paged path (TF_QWEN_KV_PAGES=256), --parallel a cap with pages set #2011 keeps pages opt-in and makes --parallel a cap with pages set. It no longer says "the default".
2. Part 3: TF_GLM_KV_SHARE is gone and TF_KV_SHARE named, and glm cuda: TF_KV_SHARE keeps GLM's prompts in the shared prompt store, and GLM's own store goes #2211 reads as landed.
3. qwen3_5 cuda: a prompt chunk's last layer runs q, attention, o and the MLP over the kept row's tile #2229: row 1's q3.5 cell, the two lines in part 2's table and the intro's prompt_tail.zig line all say landed.
4. Kimi: my mistake in 0d40ea8bf, thanks @Maxrio9. Row 17 is back to yes and row 18 is done [#2221]: copy_drafts = true, and the backend rejects a nonzero draft-head depth.

Cells changed since the last review, for whoever lands this:

Push	Row	Cell	Now	Why
08:16	4 iGPU states	q3.5	done [#2113]	merged
08:16	4 iGPU states	fnext, gem4, nemo	done¹	already on main, no single PR
08:16	12 Add + norm	fnext	no (bound <1 %)	measured bound
08:16	19 Shared drafts	fnext	no (null [#1762])	measured null
08:16	22 Direct load	q3.5, gem4	open [#2282]	PR open
08:16	22 Direct load	nemo	done [#458]	already on main
08:16	24 L2 prefetch	fnext, gem4	no (null)	measured null
08:16	2 Paged KV, part 2	q27	open M[#2283]	PR open
08:16	item 24's table	flashnext, gemma4	new row: +0.45 % and +0.15 %, inside the spread	measured on a GB10, no PR
0d40ea8bf	1 Prompt tail	q3.5	done [#2229]	merged
0d40ea8bf	21 Read-ahead	nemo, kol1, spk25, mm3, dsv41	done [#2033]	merged

0d40ea8bf also has text with no cell change: item 10 now cites #2238 behind -Dnative-fp-decode, and item 20 gains #2246's row and #2288's note. Row 10's cells stay yes, as the decode is off by default.

The null rows for fnext and gem4 are our GB10 measurements, on #1762 where a PR exists.

[jschmied]

Cells corrected in d6686f5af. Checked by reading main at ed8ac7304 (5fa2dc461 adds only a build fix and CHANGELOG lines) and the PRs below; nothing was run.

Row	Family	Was	Now	Proof
3 Prompt store	nemo	open M[#1850]	yes C open M[#1850]	no share_attach in nemotron; nemotron_h not in TF_KV_SHARE's families (core/profile.zig)
4 iGPU states	dory	yes	done¹	dory/cuda_snap.zig: DeviceBuffer, DtoD copies
4	kol1	yes	done¹	kolibri1/snap.zig, same
4	spk25	yes	done¹	spark25/snap.zig, same
4	mm3	yes	done¹	minimax_m3/cuda_engine.zig keepAs, same
12 Add + norm	dory	yes	done [#1256]	tf_slerp_rows: Dory's residual and its L2 norm (no RMS norm here) in one launch; landed as 65f3b8107
12	q27	yes	done M¹	qwen27/norm_xs.metal: H + R, then the norm
12	lfm2m	yes	done M[#822]	lfm2_norm, lfm2_combine_norm
12	lfm3	?	done X[#1696]	lfm3_post_add_norm, every layer in xpu_model.zig
12	kimi	?	done M¹	k3_res_norm (kimi_k3/layer.zig residual)
12	dsv41	?	done M[#1714] ? C	metal/fold.zig prenormPost; CUDA emits programs, still unresolved
14 CUDA graphs	glm	yes [#1490]	open [#2279]	open PR for that issue; item 14 updated
18 Copy drafts	qwen4	yes	done¹	qwen4/lanes_adapter.zig copy_drafts = true, window width
18	step5	yes	done M¹	native/draft_sources_metal_test.zig asserts copies
18	gdsa	yes	done M¹	same test
18	llama	yes	done M¹	llama/backend.zig facts: width 16, default copy_drafts; no test names it
18	dsv41	yes	done¹	deepseek_v41/serve_lanes.zig: draftSources(.., max_copy > 0), width 16
18	lfm3	yes	done X[#1696]	xpu_engine.zig copy_only, native/xpu_lfm3.zig
19 Shared drafts	dsv41	?	done¹ (opt-in)	dspark_slots.zig, TF_DSV41_SLOT_DRAFTS=1; draft/lanes.zig .draft_streams
19	lfm3	yes	no (no head)	matches its row 15; XPU is copy_only
21 Read-ahead	q27, llama, step5, gdsa, kimi	yes	no (Metal)	checkpoint_prefetch.zig start: "none off Linux"; item 21 says so
21	lfm2m	yes	no (no CUDA)	same, wording as its row 1; turns yes with #1492
23 Page drop	dsv41	yes	done [#1884]	cuda_model.zig, cuda_weights.zig cache.files (4ffa9ae83)
23	q27, llama, step5, lfm2m	yes	done M[#1885]	checkpoint_metal.zig file_cache.release; q27 through closeFiles
23	gdsa	yes	done M[#1888]	glm_dsa/weights.zig closeCopied
27 Profiles	dory	yes	open [#2306]	registers TENSORFOLD_DORY_KV_PAGES; not in profile.zig on main

Left alone:

• Row 10 stays yes for q3.5, fnext, glm, nemo, dory. cuda: native FP4/FP8 pair decode behind -Dnative-fp-decode (prompt16 FP4 −4–5 %, bit-equal) #2238 is a build option that -Dgpu product builds refuse, unlike the runtime opt-ins the page counts as done, and it covers the prompt tiles only.
• Row 19, kimi, stays yes: the backend drafts nothing yet, but the release's DSpark drafter is row 17's yes, and shared rounds apply once it lands.
• Row 6, dsv41, stays yes: deepseek: reserve full requests and agree on pool growth #1682 reserves requests in the family's own admission, not in kv_ledger.

[ashhart]

/atc exchange 27b3e59fd493e5f63f4c42b9f86d17ca9a8a32ea8e59dcf91c65bf197597b967

[ashhart]

Thank you @jschmied. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes.

[ashhart]

Thanks @jschmied, this landed at 16:46, with the three corrected lines and its CHANGELOG line. One check for a small follow-up: row 21 marks read-ahead "done [#2033]" in several families, but we can't find #2033 on main, and main's CHANGELOG credits the six opt-in loaders to #2003. The same goes for "done M[#1714]" in row 12. Could you point to where those landed, or send a one-line PR switching the cites?

[jschmied]

@ashhart, both are on main, so I'd keep the ledger as it is:

• CUDA: #1928's load read-ahead on Nemotron, Bolt, MiniMax M3, Kolibri-1, Spark 2.5 and DeepSeek-V4.1, opt-in (#2003) #2033: merged on 5 Nov, 08:50 UTC. Its commits are 7916d80ce, 5e3cee69f and d0579ef6a, the last as the merge commit. Those commits and main's CHANGELOG line cite "(CUDA: the decode-window L2 prefetch from #1580 on Qwen 3.5/3.6 and Nemotron #2003)", which looks like a typo: CUDA: the decode-window L2 prefetch from #1580 on Qwen 3.5/3.6 and Nemotron #2003 is the closed L2 prefetch issue, and the read-ahead PR is CUDA: #1928's load read-ahead on Nemotron, Bolt, MiniMax M3, Kolibri-1, Spark 2.5 and DeepSeek-V4.1, opt-in (#2003) #2033. So it's the CHANGELOG line that needs the fix, not row 21. I can send that as a one-line PR if you'd like.
• dsv41 metal: hyper-connection mix, pre and post, and the fused and folded decode glue #1714: closed as included, as your 3 Nov note says. It landed with dsv41: the served engine's lane half as a backend-neutral module #1719's merge (cfcbecb0b), and the hyper-connection commit on main is 3ef0357d5. The cite still leads to the right work, but done M[#1719] would point at the merge. Tell me which you prefer and I'll send it with the CHANGELOG fix.

[ashhart]

Thanks @jschmied! This is in TensorFold 1.2.4, out now: https://github.com/ashhart/TensorFold/releases/tag/v1.2.4