[Bizuayeu]

What this changes

Under slots_only (GLM with TF_KV_SHARE named) and no --prompt-cache-gib, kv_pool.plan now sets aside the slots that fit past the engine's own window and each other stream's fixed part and page, TF_KV_SHARE at most, before windows and held count the rest: where an asked cache goes. The rest of the default cache still becomes pages after the windows. So a room that holds one more whole window no longer keeps fewer states. On three GB10s at TP=3 (--context 1048576 --parallel 8, TF_KV_SHARE=32, no --prompt-cache-gib):

Checkpoint	Build	Left after the 16 GiB reserve	Pool pages (whole windows)	Kept states' slots	16 chats sent again, newest first: resumed from their own state
Jammed XSFP4	main	41.8 GiB	6,152 (3)	8 of 32	8
Jammed XSFP4	this head	41.9 GiB	5,535 (2)	32	16
XSFP4	main	37.3 GiB	4,796 (2)	32	16
XSFP4	this head	37.3 GiB	4,790 (2)	32	16

XSFP4's two launches differ by six pages at the same printed 37.3 GiB: at any room that prints so, both plans give the same pages, and 4,790 and 4,796 lie about 37 MiB of room apart. Main warns at the Jammed XSFP4 start (TF_KV_SHARE=32 keeps 8 states in the prompt cache's slots, under 4 a GLM stream at --parallel 8: live streams evict each other's before a resume; 32 take TF_KV_SHARE=32 and --prompt-cache-gib 6); this head keeps all 32 there without it. Keeping them costs the room's third whole window, as --prompt-cache-gib 6 does on main. Plans that already kept all n, plans without TF_KV_SHARE and plans beside an asked cache are unchanged. GLM's advice under 4 states a stream names TF_KV_SHARE alone where its slots now fit.

On four GB10s at TP=4 (XSFP4, --context 1048576 --parallel 8 --mtp-confidence 0.5, TF_KV_SHARE=32, no --prompt-cache-gib, a 14 GiB reserve, one launch a build, main first), both builds print 56.1 GiB left. Main keeps 25 of 32 states' slots beside 8,196 pool pages (4 whole windows) and warns (TF_KV_SHARE=32 keeps 25 states in the prompt cache's slots, under 4 a GLM stream at --parallel 8: …; 32 take TF_KV_SHARE=32 and --prompt-cache-gib 4); this head keeps all 32 beside 8,058 pages (3 windows), without the warning. Our own grid client's replies (a 4,096-token prompt, "draft": false, greedy, 256 tokens, six reps at 1, 4 and 8 streams) are the same on both, and the round times sit within 0.3% (26.371 → 26.375, 48.071 → 47.935 and 71.721 → 71.689 ms).

• Issue: Fixes GLM CUDA: a named TF_KV_SHARE is planned after every window the room holds, so at TP=3 on a lighter checkpoint it keeps 8 of 32 states #2987 (option 1 and the five points in issuecomment-6824706877).
• Stacked on: main (c34ff562ca), three commits: the change (24566df958), docs: the CUDA memory plan, what each setting decides and the file that reads it #3028's paragraph in docs/api.md brought to the new order (682f389f9c), and the CHANGELOG line (eb08041b0c). Rebased from main 6c5cef8941, where the change was af61220202, the same patch, and the receipt below was taken; main changed none of the change's six files in between, and "this head" in the receipt means that build. Main has since taken glm cuda: under --mtp-confidence a greedy stream drafts its rates' depth, not fifteen a round #2864, which changes how deep a temperature-0 stream drafts under --mtp-confidence (this receipt's TP=3 runs at 0.5), so today's main can draft differently at T=0. Today's main ab78aef904 has since taken glm cuda: TF_GLM_CONTEXT_CEILING serves windows past 1,048,576 tokens, read at GLM's open #2765 and glm: the DFlash2 drafter's startup line and docs give its licence as Apache-2.0 (#1583) #3084, which also change families/glm/cuda_native.zig and the recipe, in other lines; merged with it, git merge-tree is clean.
• atc overlap --json against main ab78aef904: its listing shows thirty-four open pull requests sharing a file with this head, most of them only by CHANGELOG.md, docs/api.md or both, and marks qwen3-asr: host slice, the shared tower and Qwen3 schedules over a backend binding (#2105) #2239, qwen27: serve the 27B pair, refuse what a pair cannot keep, and die with the peer #1434, GLM on CUDA: the latent split's per-rank page book and owner (TF_GLM_KV_SPLIT, #1431 part 1; stacked on #1411) #2415 and hip flashnext: a shared round's streams draft in one head pass a step #2696 as conflicting. git merge-tree of main, and of main merged with this head, against each of them: this head adds a conflict to none. Twenty-six already conflict with main, each in the same files with as many conflict hunks with this head, GLM on CUDA: the latent split's per-rank page book and owner (TF_GLM_KV_SPLIT, #1431 part 1; stacked on #1411) #2415 among them (its CHANGELOG line and environment rows; it shares core/kv_pool.zig and families/glm/cuda_native.zig) and glm cuda: EXL3 mcg experts at any whole-bit width, mixed by projection #2094 (six EXL3 files; it shares families/glm/cuda_native.zig). The other eight merge cleanly with both, GLM-5.3-Flash on CUDA keeps conversation states on disk, every rank's part (--learn-conversations) #1411, KV pool: host tier for the core pool (5/8) #1501 and glm cuda: a logit gather broadcasts one block a rank, so TP=3 prompt_logprobs past ~100 tokens no longer hangs #1571, which share core/kv_pool.zig or families/glm/cuda_native.zig, among them. Of those the listing leaves out, glm cuda: the L2 prefetch's ffn-head site also reads a MoE layer's shared gate-up, after its router #3156 and docs: move README sections into docs/, short human README #3170 merge cleanly with both, lanes: shared rounds price plain overhead apart from drafted rows (refs #2605) #2687 already conflicts with main in two files this head doesn't touch, with as many hunks with this head, and glm cuda: TF_GLM_EXPERT_FORM names the expert form, so launches compare at one form #3151, mine and open, shares docs/recipes/glm-5.3-flash-cuda.md; this head adds a conflict to none of them. The listing stops at its limits (complete: false).

The five points, in order:

1. The slots go where an asked cache goes (kv_pool.zig plan: first), counted as holds counts them (held(l, b, fit, 0)), TF_KV_SHARE at most and bounded by the cache. windows and held then count on what is left.
2. Only slots_only plans without --prompt-cache-gib that kept fewer than TF_KV_SHARE change. Host tests hold the others: plans that keep all n today, no TF_KV_SHARE (unset, empty and whitespace-only, read through sharing → shareSlots → budgetOf → plan in native/cuda_streams.zig), beside --prompt-cache-gib, and a kept_ppm family's layout (Gemma 4's).
3. A host test through kv_pool.plan with today's TP=3 layout, kv pool: a window past the engine's own counts only where the rest past every fixed part holds it #2568's TP=3 one and kv pool: a window past the engine's own counts only where the rest past every fixed part holds it #2568's TP=2 one, red on main, and a sweep of rooms from 8 to 64 GiB in 64 MiB steps where the kept states never fall (GLM CUDA: under TF_KV_SHARE, a band of rooms leaves the startup cache at 0, and since #2358 no state is kept #2547's band and this edge stay gone).
4. The startup line, advice (families/glm/cuda_share.zig) and the recipe (docs/recipes/glm-5.3-flash-cuda.md) read the new order.
5. The receipt below has each launch's pool pages and whole windows beside its kept states. The CHANGELOG line goes in its own commit once this has a number.

When fewer than n slots fit, the startup line says why, as before: a cache too small names the --prompt-cache-gib that holds all n, a room too small says how many it holds at any --prompt-cache-gib.

Tests:

• Changed, red on main: "a named TF_KV_SHARE's slots come where an asked cache goes, before the windows past the engine's own (GLM CUDA: a named TF_KV_SHARE is planned after every window the room holds, so at TP=3 on a lighter checkpoint it keeps 8 of 32 states #2987)" (main keeps 8 of 32 at the Jammed XSFP4 room, 0 and 4 at kv pool: a window past the engine's own counts only where the rest past every fixed part holds it #2568's TP=2 rooms, 0, 1 and 7 at kv pool: a window past the engine's own counts only where the rest past every fixed part holds it #2568's TP=3 ones) and "with --parallel named, TF_KV_SHARE's kept states never fall as the room grows: no band, no window's edge" (main drops states at each window's edge, six times at TP=3).
• Unchanged, green on main and here: "plans stand where today's keeps all n, without TF_KV_SHARE, beside --prompt-cache-gib and at kept_ppm", and the band tests' other rows.
• Five assertions moved out of the band tests into the changed test, as they held the old order: "outside that band"'s TP=3 rows at 40.375 and 41.125 GiB and TP=2 rows at 16.4 and 17.2 GiB, and edge's snaps == 0.
• advice: wide and strip now advise TF_KV_SHARE=16 alone; Jammed XSFP4's TF_KV_SHARE=8 (a 1.5 GiB cache) advises 32 alone; beside a named 2 GiB cache, 12 fit and it names 6.

Checks on this head

• Head commit: eb08041b0c, the CHANGELOG line on top of the docs/api.md paragraph (682f389f9c) and the change (24566df958); neither of the two on top changes code. The checks below ran on this head; the receipt ran the change before the rebase, on main 6c5cef8941.
• Lean checker, python3 tools/zig/lean_check.py, on the base (main c34ff562ca) and on this head: 341 and 341, the same findings. On today's main ab78aef904 and this head merged with it: 341 and 341, the same findings.
• Tests run, with the command and pass, fail and skip counts: zig build test-kv-pool, 61 of 61. The whole gate on this tree in WSL (x86_64, Zig 0.17.0): test -Dfixed-cpu, test-kernel-contract, test-golden, test-cuda-host, compile-cuda-tests (x86_64 and aarch64), check-native-cuda-host, test-kv-pool, test-glm-pages-host, test-glm-snapshots-host, test-glm-image-cache-host, test-glm-startup-host: every step passed (test 9,261 of 9,330 with 69 skipped, test-kernel-contract 236 of 236, test-cuda-host 6,426 of 6,456 with 30 skipped, test-kv-pool 61 of 61). pytest from the root with the CI's options plus -rs (-q -rs -p no:cacheprovider): 450 passed, 86 skipped. Main's CI pytest at c34ff562ca has 452 passed and 84 skipped, and prints no skip reasons. On this head merged with today's main ab78aef904, check-native-cuda-host, the five GLM host steps and compile-cuda-tests (x86_64) pass too (test-kv-pool 61 of 61, test-glm-pages-host 13 of 13, test-glm-snapshots-host 21 of 21, test-glm-image-cache-host 3 of 3, test-glm-startup-host 7 of 7). Two of the skips here are tools/zig/test_nfkd_tables.py's, which want Unicode 16.0.0 (this Python 3.12 has 15.0.0). On the tree before the rebase, with this head's tests on main, exactly the two changed tests fail; with advice's body as main has it, the advice test fails ("expected null, found 3"); the empty-share test fails when its pages are off by one.
• CI: zig, zig-windows, zig-macos-dev-tools, pytest, release-paths, both benchmark-clients jobs and fmt passed; the two nightly host jobs report skipped.
• Not run, and why: Metal, HIP, Vulkan and XPU builds and tests, with no such machine here. slots_only is set only in CUDA's budgetOf, so their plans take the old path.

Receipt

• Main against branch: the TP=3 table and the TP=4 launches above, and the cells below.
• Boxes it reaches: three GB10s at TP=3 and four at TP=4, each on a ring of QSFP cables.
• New switch? None.
• Environment: TensorFold af61220202 (the change before the rebase, the same patch as 24566df958; the receipt's binary is built from its tree, 2e922da1ed) and main 6c5cef8941, each built from an empty Zig global cache with Zig 0.17.0 and zig build native -Dnvcc=<nvcc 13.3.73> -Dsm=121 in a CUDA 13.3 build container; GLM-5.3-Flash XSFP4 at 88736f91 and Jammed XSFP4 at 0a22bb46, each with its own MTP head drafting; --context 1048576 --parallel 8, TF_KV_SHARE=32, no --prompt-cache-gib, a 16 GiB reserve, the GPU clock capped at 2,200 MHz. At TP=4, main 6c5cef8941 and the head before the rebase, def2158752, each built from an empty Zig global cache with Zig 0.17.0 in a CUDA 13.3.1 build container; XSFP4 at 88736f91, --mtp-confidence 0.5, a 14 GiB reserve, TF_GLM_PREFILL_REDUCE=gather, NCCL_CUMEM_ENABLE=1 and TF_GLM_PROMPT_TAIL=1.
• Paths and platforms the change reaches, which I ran and which I could not: the new step in kv_pool.plan runs only under slots_only, which only budgetOf (native/cuda_kv_pool.zig) sets, from TF_KV_SHARE's slots. Of the families with TF_KV_SHARE's hooks, only GLM's layout gives each kept state a slot, so only GLM's CUDA streams reach it; every other plan takes the old path. Ran: GLM at TP=3 on GB10, and at TP=4 the plan, the startup line and the grid. Not run: GLM at TP=2 (the host test takes kv pool: a window past the engine's own counts only where the rest past every fixed part holds it #2568's TP=2 layouts through kv_pool.plan), and at TP=4 the 16 chats sent again and the drafted benchmarks.
• Exactness: tools/bench_concurrent.py --alone --serial at 1, 4 and 8 streams, T=0 and T=1, on all four launches: every concurrent reply equals the same request alone (3, 12 and 24 checks a cell) and the serial "draft": false replies (8 a cell), and none failed. The 16 chats' replies sent again equal their first replies on every launch, and each chat's replies on this head equal main's. Decode-check tokens and bench_openai and bench_concurrent token hashes equal main's on each checkpoint. At TP=4 the grid's replies equal main's in every rep and stream of its three cells.
• Main against this branch, same machine and checkpoint: this post.
• Decode speed before and after, in tok/s, main → this head, one launch each with main first: tools/bench_openai.py (the median of three 256-token runs) and tools/bench_concurrent.py (a cell's aggregate, the median of three reps; request i sends tools/bench_prompts.json's item i, 256 tokens):

Checkpoint	T	bench_openai fibonacci-raw	bench_openai gpu-chat-no-think	bench_concurrent 1 stream	4 streams	8 streams
Jammed XSFP4	0	56.38 → 56.47	39.55 → 39.57	54.5 → 54.4	94.0 → 94.0	95.5 → 96.0
Jammed XSFP4	1	55.98 → 56.67	47.34 → 47.54	71.4 → 71.0	85.1 → 85.4	91.7 → 94.1
XSFP4	0	45.25 → 45.09	38.95 → 38.77	54.1 → 54.3	89.1 → 88.8	95.1 → 94.7
XSFP4	1	51.30 → 50.40	47.10 → 47.23	59.2 → 60.1	85.2 → 85.1	92.9 → 93.3

At T=0 the widest gap between the builds in any cell is 0.52%, and on XSFP4's fibonacci-raw all three of this head's runs sit below main's (45.08 to 45.13 against 45.17 to 45.36); one launch a build can't tell gaps this size from launch to launch. At T=1 one launch's three bench_openai runs spread wider than the builds differ (XSFP4 fibonacci-raw: 47.31 to 52.72 on main, 47.77 to 54.03 here).

• Prompt speed before and after: n/a. The change sizes the pool and the slots at load; prompt passes run the same kernels over the same pages.
• Licence of any new weights or drafter: n/a.

Checklist

• The title and description match the head commit, including the receipt and the stack.
• New tests fail before the change, pass after it, and skip cleanly without their dependency. (The two changed tests fail on main; the native empty-share test skips on Windows, where it can't set the environment.)
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.
• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off. (The plan only sizes the pool and the slots.)
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (bench_concurrent --alone --serial and the 16 chats above.)
• No precision traded for speed. If the bits change, the description says which sums change and why. (No bits change: the same tokens as main.)
• Decode and prompt processing are no slower than main at one stream and the widest admitted; with --drafter, no slower than without it, greedy and sampled. (The table above, at 1 and 8 streams, greedy and sampled, one launch a build: gaps under 1% can't be told from launch to launch there, and the larger T=1 gaps favour this head or sit inside one launch's run spread. The change doesn't reach drafting: every launch drafts with its checkpoint's MTP head.)

[Bizuayeu]

Added a TP=4 receipt to the description (four GB10s, main and this head, one launch each).

[ashhart]

Thanks @Bizuayeu, the reasoning here is careful and so are the tests. Taking the named TF_KV_SHARE slots first, where an asked cache would go, means the kept states no longer fall as the room grows. The new band rows pin that, the removed rows were exactly the old behaviour, and the guard tests show unset, empty and whitespace values plan as before.

It's queued to land. It's safe because the new step runs only with TF_KV_SHARE named and no --prompt-cache-gib, so every other plan is main's, and kept-state counts never change a reply's tokens.

The CHANGELOG commit conflicts with newer lines; we'll settle that when it lands, so there's nothing to rebase.

• Synthena (Ash's Agent)

[ashhart]

Our device run on two GB10s at TP=2 (GLM-5.3-Flash XSFP4, fresh launches base, new, new, base; TF_KV_SHARE unset, so this checks that the default path is unchanged): GREEN. Tokens are identical (72 of 72 decode cells and 12 of 12 prompt cells), decode is level (-2.2% to +1.8%), and chat resume qualified on both arms. The prompt gains in the summary come from main having moved since this PR's merge base, not from this change. Thank you @Bizuayeu. It's going into the landing queue.

• Synthena (Ash's Agent)