[BlivionIaG]

Current head: 310133f35a. Main 90b4fae62d, bafe3e07cb, 948e748584, d001c28f29 and release 09a9eb3317 and the #1444 TP=2 paging fix were added after the checks and device measurements below; those device measurements were not repeated on this head. CI is running on this head.

What this changes

Paged EXL3 K/V from the core pool.

• Issue: Refs [Draft] EXL3 Flash Next on CUDA: TP=2, BF16 paging, cache reuse, graphs #1113 (part of the agreed chain; does not complete it).
• Stacked on: EXL3 Flash Next: deep prefill (3/8) #1443 5415afbd81; over it this PR adds 2 commits (cuda: use the core K/V page pool for EXL3 caches and the TP=2 refusal for non-EXL3 packs).
• Based on main 6c5cef8941, merged through 484437b6ac.
• Overlap: atc overlap --json against four known peers (Flash Next TP=2 on CUDA: head and rollback graphs replay, and one-row split lanes skip the interleave copies #2837, flashnext cuda: the opt-in --tp 2 decode graph (TF_FLASHNEXT_DECODE_GRAPH=1) serves again instead of failing its first plain round #2980, flashnext cuda: the opt-in --tp 2 decode graph takes the dense split and keeps its graph across drafted verifies #2982, flashnext: the MTP head's streams by row through the lease's pages, no per-lane block outside the pool (#2444 P3, F2 part c) #2992) only, run on the integrated stack before the latest merges of main; not a global clearance. Conflicting files: Flash Next TP=2 on CUDA: head and rollback graphs replay, and one-row split lanes skip the interleave copies #2837 2, flashnext cuda: the opt-in --tp 2 decode graph (TF_FLASHNEXT_DECODE_GRAPH=1) serves again instead of failing its first plain round #2980 6, flashnext cuda: the opt-in --tp 2 decode graph takes the dense split and keeps its graph across drafted verifies #2982 8, flashnext: the MTP head's streams by row through the lease's pages, no per-lane block outside the pool (#2444 P3, F2 part c) #2992 6. Owner composition agreements exist for flashnext cuda: the opt-in --tp 2 decode graph (TF_FLASHNEXT_DECODE_GRAPH=1) serves again instead of failing its first plain round #2980, flashnext cuda: the opt-in --tp 2 decode graph takes the dense split and keeps its graph across drafted verifies #2982, flashnext: the MTP head's streams by row through the lease's pages, no per-lane block outside the pool (#2444 P3, F2 part c) #2992; my notice on Flash Next TP=2 on CUDA: head and rollback graphs replay, and one-row split lanes skip the interleave copies #2837 is unanswered.

Checks before the release merge

• Head d2aaa3ed00. The device results below ran on 08abdaf8a5c0; this head differs from it by the merges of main at 0591c4e85d, c34ff562ca, a3aecd8d0f and 484437b6ac and the pin fixture gradient-serving.tsv. The device cells were not repeated on this head.
• python3 tools/pr_check.py --base <main> --pr 1444: exit 0, 0 errors, 0 warnings; lean 339 vs 341 on main, 0 new. zig fmt --check zig: clean.
• Host tests: test-cuda-host, test-exl3-cuda-host, test-flashnext-exl3-host, test-exl3-prompt-host, compile-flashnext-host, test-flashnext-memory-host, test-qwen-serving-host, test-lane-api-host and test-lanes-see pass on Darwin arm64 on the EXL3 Flash Next: draft lane guard for offloaded experts (8/8) #1446 head, which contains this head; the Linux host suite is CI's.
• Linux x86_64 build with CUDA (-Dtarget=x86_64-linux-gnu): exit 0 on the EXL3 Flash Next: draft lane guard for offloaded experts (8/8) #1446 head, which contains this head (build only, no GPU executed on this head).
• CI: see the checks on this PR.

Device results on 08abdaf8a5c0

Two 16 GB sm_120 GPUs, Linux x86_64, checkpoint above. One run per cell unless stated.

Cell	Result
Plain page-boundary resume	passes unpaged and paged
TP=1 speed with paging, unpaged/paged/paged/unpaged	level at one stream and 4 to 10% higher at four streams on code (no third baseline arm)
1444-perf-tp2-abba, TP=2 speed with paging, unpaged/paged/paged/unpaged	passes; token hashes equal in all four arms; paged is 0.99 to 1.01 of unpaged at one stream and 1.03 to 1.07 at four streams (greedy, code and chat); arms of the same kind differ by up to 5%, so no better than "no slower"
Drafted resume on fresh servers, TP=1	paged-all and unset pass and equal each other
Drafted resume on fresh servers, TP=2	paged-all and unset each pass; tp2-compare is red only because the two runs differ in which lane took the 4164-token cache hit (the gate pins the lane)
Ancestry	the frozen core is an ancestor of this head

Not run or open

• Before the TP=2 paging fix, a 10000-token prompt at --parallel 2 was refused; the targeted admission smoke passed with the fix. The full device suite has not been repeated on the current head.
• Main against this head (main refuses this checkpoint).

Receipt

• Boxes it reaches: paged K/V for EXL3 Flash Next, TP and resume paths.
• New switch: none.
• Environment: Zig 0.17.0. Host checks on Darwin arm64. Device runs on Linux x86_64 with two 16 GB sm_120 GPUs. Checkpoint turboderp/Qwen3.8-Flash-Next-exl3 at 69e33439ae950f17bcbe95c98f117d80f759ab6d.
• Kernel contract: n/a: no GPU kernel files change in this PR.
• Model fidelity, measured on an older head and not repeated: native against an independent reference, 70/72 top tokens; the two differences are teacher-forced ids 16 (step 11) and 17, native margins 0.25 and 0.5 against reference 0.125 and 0.25. No tie waiver and no perfect-equivalence claim.
• Licence of new weights or drafter: n/a (Qwen Community License 1.0 on the checkpoint; no weights distributed, no new drafter).

Review requests implemented in source

• Main's refusal of TF_FLASHNEXT_KV_PAGES at --tp 2 is kept for every pack except EXL3 (refusePages); EXL3 at TP=2 is accepted and the explain text says so. A host test covers non-EXL3 refused with the setting named, EXL3 accepted, TP=1 accepted, unset accepted.

Checklist

• The title and description match the head commit, including the receipt and the stack.
• New tests fail before the change, pass after it, and skip cleanly without their dependency. Host tests pass in CI and the device cells above pass; I did not run the new tests against the parent to show they fail before the change.
• Comments and docstrings are one line. No measurements or history in the source. Lean check (one-line comments within 120 columns): 0 new problems. I scanned every added comment line in the stack for measured results and history wording; the hits are worked examples and limits, not recorded results.
• No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines. Scanned every added line in the stack diff and every commit message for host and user names, addresses, local paths, AI trailers and tool names: none. No binary files are added. Receipts are not in the repository.
• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off. No direct demonstration. The concurrent and drafted device cells run through the lane rounds.
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. Resume and drafted resume pass at TP=1; the TP=2 drafted comparison is red on lane pinning (above). Concurrent equals solo was not run for this PR on its own.
• No precision traded for speed. If the bits change, the description says which sums change and why. No arithmetic change; paged and unpaged runs give the same tokens at TP=1.
• Decode and prompt processing are no slower than main at one stream and the widest admitted; with --drafter, no slower than without it. Paged against unpaged: level at one stream; at four streams 4 to 10% higher on code at TP=1 and 3 to 7% higher at TP=2 (about the size of the noise: no slower). Main refuses this checkpoint (n/a).

TP=2 admission fix: commit 69fe300ff9 negotiates the maximum rank state-run size before each rank plans its page pool. On the earlier device-tested #1444 source plus this change, --tp 2 --parallel 2 with 256-row paging served both 3,000- and 10,000-token prompts; the 10,000-token request previously returned HTTP 500. The current release-merged head has not been run on device.

[ashhart]

Thanks for the careful paging work. As I said on #1113, a second BF16 paging system beside main's core KV pool won't
land. This PR adds its own pager and shared arena next to TF_FLASHNEXT_KV_PAGES:
exl3/kv/cuda_exl3_kvpage*.zig, exl3/offload/exl3_shared_kv.zig, cuda_exl3_shared_kv.zig and the six kvp_*
kernels. cuda_prompt.zig:177 also reads TF_FLASHNEXT_KVPAGE on every Flash Next engine.

The way in is to extend core/kv_pool.zig and the paging that cuda_kv_lanes.zig already does. A host-RAM tier for
that pool would be welcome as its own issue, since other families could use it too.

Two other points:

• The PR has no measurements yet.
• tools/exl3_serve_tp2.sh:12 turns paging on, so every later receipt depends on it. Please measure later slices
  without it too.

[BlivionIaG]

1444-perf-tp2-abba passes (1300 s): TP=2 speed with paging, unpaged / paged / paged / unpaged order, one server per arm, same binary as the other cells (08abdaf8a5c0; the current head differs by merges of main and the pin fixture, not repeated on the device). Two 16 GB sm_120 GPUs.

Aggregate tok/s from bench_concurrent (greedy):

prompts	streams	U1 (unpaged)	P2 (paged)	P3 (paged)	U4 (unpaged)	paged / unpaged (means)
code	1	25.2	25.4	25.5	25.3	1.008
code	4	60.7	64.8	64.1	59.5	1.072
chat	1	24.8	24.7	24.6	24.8	0.994
chat	4	49.6	50.0	52.3	50.0	1.027

The bench_openai token hashes of all four arms are equal. Paged is level at one stream (0.99 to 1.01 of unpaged) and 1.03 to 1.07 at four streams. Arms of the same kind differ by up to 5% (paged chat at four streams: 50.0 and 52.3), so the four-stream difference is about the size of the noise: read it as no slower, not faster. One run per arm, no third baseline.

[BlivionIaG]

A TP=2 finding on paged K/V, found while trying to run #1522's TP=2 restore cells (the first server start there failed). Binary 08abdaf8a5c0 (this PR's tested head), two sm_120 GPUs, --tp 2 --context 12288, TF_FLASHNEXT_EXPERT_LINES=16 TF_FLASHNEXT_KV_PAGES=256 TF_FLASHNEXT_KV_PAGES_ALL=1, one raw token-id completion with max_tokens 8:

--parallel	prompt tokens	result
5	3000	served
5	10000	served
2	3000	served
2	10000	HTTP 500 at once: the engine has no memory for this request's caches with nothing else running: lower --context or --parallel

On the #1522-tested binary with the tier off (the paged path is this PR's), the same --parallel 2 server without TF_FLASHNEXT_KV_PAGES and _ALL serves the 10000-token prompt, so the refusal comes with paging, not with the model or the context. At TP=1 with the same paging settings and --parallel 2, 12000-token prompts are served (the restore cells). The startup line for the refused TP=2 server says 2 streams at once, the engine's own at a 12288-token window (0.70 GiB, in the model's), 1 more taking 96 pages of 256 rows (0.70 GiB).

So at TP=2, paged, --parallel 2, a prompt that fits the window (10000 of 12288 tokens) is refused for lack of page memory when nothing else is running. I have not found the cause and have not changed any code. The TP=2 speed cell above uses --parallel 5 and short prompts, so it does not hit this. Whether this is an intended limit of the page budget or a bug in the TP=2 page accounting is open; I'm not claiming either.

[BlivionIaG]

Follow-up to the TP=2 paged-admission refusal: commit 69fe300ff9 makes the ranks agree on the larger state-run size before either rank plans its page pool. Previously the rank without the MTP head could plan fewer pages, then adopt the other rank’s larger run at attachment.

On the previously tested #1444 source (08abdaf8a5c0) plus this change, two sm_120 GPUs, --tp 2 --parallel 2 --context 12288, TF_FLASHNEXT_KV_PAGES=256 and TF_FLASHNEXT_KV_PAGES_ALL=1: both 3,000- and 10,000-token requests completed with eight generated tokens. Before the change, the 10,000-token request returned HTTP 500 in this setup. This was a targeted admission smoke, not the full qualification suite; the release-merged current head was not run on device.