[Bizuayeu]

What this changes

docs/api.md said the CUDA server "sizes its window" from the free memory less the reserve, on unified memory and on discrete cards. On native CUDA the window is --context, the model's own window at --context 0, or, when it is unset, the family's default capped at the model's own window; on the paged families the streams and pages fill the budget up to the streams' windows plus the cache's pages, so once those exceed it, the free memory after load is the reserve. The two paragraphs now say so. After them, under "Context and errors":

• the plan's order: the engine's own window, each other stream's fixed part, then the prompt cache, a named --prompt-cache-gib before every window but the engine's own and an unnamed one after the whole windows (GLM CUDA: a named TF_KV_SHARE is planned after every window the room holds, so at TP=3 on a lighter checkpoint it keeps 8 of 32 states #2987 for what that does to TF_KV_SHARE's states), then the kept states, and the rest of the budget as pages;
• a table of the settings that decide each part, with its kind, its scope (the host, the launch or a family), its default, what it decides and the file that reads it;
• a pointer to CUDA on unified memory: what the reserve, the KV pool and the prompt cache each decide, and at which scope #3000 for measured numbers, in place of the example the first version had (it ran on glm cuda: TF_GLM_CONTEXT_CEILING serves windows past 1,048,576 tokens, read at GLM's open #2765's head, and the reference describes main).

docs/recipes/cuda.md's paragraph on the Python 0.6.6 engine links to the table. No new names and no default changes.

• Issue: Refs CUDA on unified memory: what the reserve, the KV pool and the prompt cache each decide, and at which scope #3000, as asked there (CUDA on unified memory: what the reserve, the KV pool and the prompt cache each decide, and at which scope #3000 (comment)).
• Stacked on: main (3885dceb2bb6). It merges with main 6c5cef8941 cleanly (git merge-tree).
• atc overlap --json (2026-11-12 14:05 UTC, base main; complete: false, with the other pull requests' limit, unavailable pull files and the time limit): atc tested 5 pull requests that share a file, GLM-5.3-Flash on CUDA keeps conversation states on disk, every rank's part (--learn-conversations) #1411, Nemotron 3.5 Transcribe: audio input, the 65,536-token window, an hour of audio, MTP drafts #1632, CUDA: #1764's BF16 prompt tile for Qwen ModelOpt, Nemotron and Dory, plus two exact Qwen prompt kernels (#1762) #2008, ricursive: Ripple-65B-Preview on native Metal (NVFP4 and bf16 releases) #2231 and ricursive: the publisher's DFlash2 drafter for Ripple-65B (stacked on #2231) #2260, all clean there. By hand: 33 of the 288 open pull requests change docs/api.md or docs/recipes/cuda.md (the 25 whose file lists stop at 100 were read to the end). Test merges with this branch and with main 6c5cef8941 (git merge-tree): 8 merge with this branch cleanly, and the 25 that conflict with it conflict with main in the same files, so the branch adds none. In docs/api.md, ocr: PDF documents and pages on /v1/ocr, one page at a time (#2440 step 2) #2571 to ocr: POST /ocr in PaddleX's contract (#2440 step 6) #2575 conflict in the one block they already have with main.
• The CHANGELOG line is left for landing.

Checks on this head

• Head commit: c20358d550 (documentation only), the review's three edits on a748ac0581.
• Lean checker, python3 tools/zig/lean_check.py, on the base and on this head: 345 and 345, 0 new. tools/pr_check.py: 0 errors, 0 warnings.
• Tests run, with the command and pass, fail and skip counts: python -m pytest -q -rs -p no:cacheprovider from the root, on a748ac0581: 450 passed, 86 skipped and 519 subtests passed. Main's CI run of the same job on 3885dceb2bb6 has 452 passed and 84 skipped: the two more skips here are tools/zig/test_nfkd_tables.py's, since this interpreter has Unicode 15.0.0 and the tables pin 16.0.0. c20358d550 changes only docs/api.md, which no test reads, so it was not run again.
• CI: green on c20358d550 (runs 43330299393 and 43330299202): benchmark-clients (ubuntu-latest, windows-latest), pytest, release-paths (ubuntu-24.04), zig, zig-macos-dev-tools, zig-windows and fmt pass; host-nightly-linux and host-nightly-macos were skipped by the workflow.
• Not run, and why: inference, since the change is documentation only.

Receipt

n/a: documentation only, so it cannot reach inference.

Checklist

• The title and description match the head commit, including the receipt and the stack.
• New tests fail before the change, pass after it, and skip cleanly without their dependency. (No new tests: documentation only.)
• Comments and docstrings are one line. No measurements or history in the source. (No source changes.)
• No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.

[ashhart]

Thanks @Bizuayeu. I checked each setting and file in the new table against main, and nearly all of it holds:

• the reserve's floors and the 2 GiB and 0.5 GiB lower limits (zig/src/native/cuda_memory.zig:141-154);
• the limit counting the weights (Pool.room, :132);
• --parallel auto meaning up to 8 (zig/src/server/cli.zig:508);
• the plan's order and the named cache coming before the windows (zig/src/core/kv_pool.zig:81-104);
• TF_KV_SHARE (zig/src/core/kv_prompts.zig:119-121);
• GLM's host pages defaulting to one window and 16 pages (zig/src/families/glm/cuda_kv_host.zig:21-25).

The old "sizes its window" wording was wrong, and this fixes it.

Three small edits before it lands:

• docs/api.md:738-740: yes, the example should go. It was measured on glm cuda: TF_GLM_CONTEXT_CEILING serves windows past 1,048,576 tokens, read at GLM's open #2765's head, which isn't on main, and the reference describes main. A link to CUDA on unified memory: what the reserve, the KV pool and the prompt cache each decide, and at which scope #3000 for the numbers is enough.
• docs/api.md:707-708 and :731: an unset --context takes the family's default, capped at the model's own window, and --context 0 takes the model's window (zig/src/native/cuda_window.zig:16-22). Please say so, and add that file to the row.
• docs/api.md:733: the unnamed cache's size, half of what the windows leave with a 16 GiB floor, is set in zig/src/native/cuda_kv_pool.zig:164-178 (budgetOf) and zig/src/native/cuda_memory.zig:99-101 and :185-191, not in kv_pool.zig. Please name those files.

CI is green, and it's docs only, so it needs no device run. We'll add the CHANGELOG line when we land it.

• Synthena (Ash's Agent)

[Bizuayeu]

Three small edits before it lands

#3028 (comment)

Done in c20358d550, checked against main's zig/src/native/cuda_window.zig, cuda_kv_pool.zig and cuda_memory.zig:

• docs/api.md:738-740: the example is gone, and one line points to CUDA on unified memory: what the reserve, the KV pool and the prompt cache each decide, and at which scope #3000 for the numbers.
• :707-708 and :731: an unset --context takes the family's default capped at the model's own window, and --context 0 takes the model's window; the row adds zig/src/native/cuda_window.zig.
• :733: the row names zig/src/native/cuda_kv_pool.zig (budgetOf) and cuda_memory.zig for the unnamed cache's size, in place of kv_pool.zig.

The description matches the new head.

[ashhart]

Thanks @Bizuayeu, all three edits are in. It's queued to land.

• Synthena (Ash's Agent)

[ashhart]

Thank you @Bizuayeu. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes.

[ashhart]

Thanks @Bizuayeu! This is in TensorFold 1.2.8, out now: https://github.com/ashhart/TensorFold/releases/tag/v1.2.8

• Synthena (Ash's Agent)