[BlivionIaG]

Current head: 5415afbd81. Main 90b4fae62d, bafe3e07cb, 948e748584, d001c28f29 and release 09a9eb3317 were merged after the checks and device measurements below; those device measurements were not repeated on this head. CI completed without reported failures on this head.

What this changes

Segmented deep EXL3 prefill with TP prefill partials in float64, and prompt-state restore at any row (the snapshot grid is gone).

• Issue: Refs [Draft] EXL3 Flash Next on CUDA: TP=2, BF16 paging, cache reuse, graphs #1113 (part of the agreed chain; does not complete it).
• Stacked on: EXL3 Flash Next: expert-parallel TP=2 (same tokens as TP=1) and MTP drafts (2/8) #1442 9d2bd21f28; over it this PR adds 12 non-merge commits (deep prefill, TP prefill dispatch, prefill settings registration and tests, FP64 TP prefill partials with a typed all-gather and one rounding at the finish, snapshot-grid removal, rotate-contract fault handling).
• Based on main 6c5cef8941, merged through 484437b6ac.
• Overlap: atc overlap --json against four known peers (Flash Next TP=2 on CUDA: head and rollback graphs replay, and one-row split lanes skip the interleave copies #2837, flashnext cuda: the opt-in --tp 2 decode graph (TF_FLASHNEXT_DECODE_GRAPH=1) serves again instead of failing its first plain round #2980, flashnext cuda: the opt-in --tp 2 decode graph takes the dense split and keeps its graph across drafted verifies #2982, flashnext: the MTP head's streams by row through the lease's pages, no per-lane block outside the pool (#2444 P3, F2 part c) #2992) only, run on the integrated stack before the latest merges of main; not a global clearance. Conflicting files: Flash Next TP=2 on CUDA: head and rollback graphs replay, and one-row split lanes skip the interleave copies #2837 2, flashnext cuda: the opt-in --tp 2 decode graph (TF_FLASHNEXT_DECODE_GRAPH=1) serves again instead of failing its first plain round #2980 6, flashnext cuda: the opt-in --tp 2 decode graph takes the dense split and keeps its graph across drafted verifies #2982 8, flashnext: the MTP head's streams by row through the lease's pages, no per-lane block outside the pool (#2444 P3, F2 part c) #2992 6. Owner composition agreements exist for flashnext cuda: the opt-in --tp 2 decode graph (TF_FLASHNEXT_DECODE_GRAPH=1) serves again instead of failing its first plain round #2980, flashnext cuda: the opt-in --tp 2 decode graph takes the dense split and keeps its graph across drafted verifies #2982, flashnext: the MTP head's streams by row through the lease's pages, no per-lane block outside the pool (#2444 P3, F2 part c) #2992; my notice on Flash Next TP=2 on CUDA: head and rollback graphs replay, and one-row split lanes skip the interleave copies #2837 is unanswered.

Checks before the release merge

• Head f654c5b60d. The device results below ran on 40e3a6ceb16e; this head differs from it by the merges of main at 0591c4e85d, c34ff562ca, a3aecd8d0f and 484437b6ac (cuda_batch_moe.zig: this branch's writeback and main's sumSlots both kept) and the pin fixture gradient-serving.tsv. The device cells were not repeated on this head.
• python3 tools/pr_check.py --base <main> --pr 1443: exit 0, 0 errors, 0 warnings; lean 339 vs 341 on main, 0 new. zig fmt --check zig: clean.
• Host tests: test-cuda-host, test-exl3-cuda-host, test-flashnext-exl3-host, test-exl3-prompt-host, compile-flashnext-host, test-flashnext-memory-host, test-qwen-serving-host, test-lane-api-host and test-lanes-see pass on Darwin arm64 on the EXL3 Flash Next: draft lane guard for offloaded experts (8/8) #1446 head, which contains this head; the Linux host suite is CI's.
• Linux x86_64 build with CUDA (-Dtarget=x86_64-linux-gnu): exit 0 on the EXL3 Flash Next: draft lane guard for offloaded experts (8/8) #1446 head, which contains this head (build only, no GPU executed on this head).
• CI: see the checks on this PR.

Device results on 40e3a6ceb16e

Two 16 GB sm_120 GPUs, Linux x86_64, checkpoint above. One run per cell unless stated.

Cell	Result
Deep prefill against the batched pass	fast-equal, fast-resume (45 kept states bit-equal), fast-mixed and the TP prefill cells pass at TP=1 and TP=2; TP=2 deep equals TP=1 deep, bit for bit
REDUCED_BRANCH, TWO_CHUNK	change neither tokens nor speed
Prompt time, deep against batched	3001 tokens 5.9 s against 37.7 s at TP=1 and 3.4 s against 20.9 s at TP=2; 12001 tokens 17.6 s at TP=1 and 11.8 s at TP=2
First token, cold 6144-token prompt, deep prefill off against on	76 to 81 s against 7.8 s repeated (14.5 s for the first request after start); token hashes identical in every request; decode unchanged at 1 and 4 streams
kernel-contract exl3 flashnext (filtered)	96/96 kernels pass gates 1-4, 43/43 controls red, 137 limit problems (the same set as #1441)

Not run or open

• Main against this head (main refuses this checkpoint).

Receipt

• Boxes it reaches: deep prefill and prompt-state restore, TP prefill.
• New switch: none.
• Environment: Zig 0.17.0. Host checks on Darwin arm64. Device runs on Linux x86_64 with two 16 GB sm_120 GPUs. Checkpoint turboderp/Qwen3.8-Flash-Next-exl3 at 69e33439ae950f17bcbe95c98f117d80f759ab6d.
• Kernel contract: This PR adds or changes kernels: flashnext_exl3_prefill.cu (+425), exl3_candidate.cu (+114), contract_tppf_controls.cuh (+275, new), contract_controls.cu (+3). The filtered contract above covers them.
• Model fidelity, measured on an older head and not repeated: native against an independent reference, 70/72 top tokens; the two differences are teacher-forced ids 16 (step 11) and 17, native margins 0.25 and 0.5 against reference 0.125 and 0.25. No tie waiver and no perfect-equivalence claim.
• Licence of new weights or drafter: n/a (Qwen Community License 1.0 on the checkpoint; no weights distributed, no new drafter).

Profile setting check

The final head has one TF_FLASHNEXT_PROFILE registration in zig/src/platform/environment.zig, the same entry as main. The EXL3 prefill and main paths read that shared setting; this slice adds no second registration, environment-table doc line or valid-case test.

Checklist

• The title and description match the head commit, including the receipt and the stack.
• New tests fail before the change, pass after it, and skip cleanly without their dependency. Host checks passed in CI; device results and failures are listed above. I did not run the new tests against the parent to show they fail before the change.
• Comments and docstrings are one line. No measurements or history in the source. Lean check (one-line comments within 120 columns): 0 new problems. I scanned every added comment line in the stack for measured results and history wording; the hits are worked examples and limits, not recorded results.
• No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines. Scanned every added line in the stack diff and every commit message for host and user names, addresses, local paths, AI trailers and tool names: none. No binary files are added. Receipts are not in the repository.
• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off. No direct demonstration. The concurrent and drafted device cells run through the lane rounds.
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. Resumed equals fresh: 45 kept states bit-equal; token hashes identical across the first-token runs. Concurrent equals solo and drafted equals "draft": false were not run for this PR on its own.
• No precision traded for speed. If the bits change, the description says which sums change and why. TP prefill partials are summed in float64 with one rounding to float32 at the finish, so the TP=2 sum equals the TP=1 sum; shown bit for bit on device (TP=2 deep equals TP=1 deep).
• Decode and prompt processing are no slower than main at one stream and the widest admitted; with --drafter, no slower than without it. Prompt processing is faster (cells above); decode is unchanged at 1 and 4 streams. Main refuses this checkpoint (n/a).

[ashhart]

The prefill speedup is impressive: the 16k-token prompt goes from 56 s to 10.5 s. The blocker is arithmetic.

The deep path is on by default for segments of 512 rows or more (exl3/prefill/cuda_exl3_prefill_state.zig:22,28;
cuda_exl3_prefill.zig:211-212). It runs dense projections of 64 rows or more on #806's candidate tiles
(cuda_exl3_prefill_tiles.zig:10-14) and experts on the new 64-member tiles, while decode and shorter segments use
Linear.matmul and runSlots. The docs prompt's hash moving from 1882ea29ebf6 to e6834625432c shows the
arithmetic changed.

Keeping cache states on the step grid (snap_grid) is a careful way to make resumes match fresh runs. Even so, a
prompt's tokens now depend on the step size and on whether the deep buffers could be reserved. When they can't, at
startup (cuda_exl3_prefill.zig:79) or at request time (:267), the batched pass takes over. Prompt rows also no
longer match serial decode, which is our oracle.

To land:

• The prompt kernels need to produce the decode kernels' bytes for every row, whatever the chunk or segment length.
  Main's NVFP4 Flash Next prompt tiles do this; see the coverage section of docs/kernel-contract.md.
• Register tf_exl3pf_gu and tf_exl3pf_down with replaces against the decode kernels, so the contract checks the
  bytes.
• Add a test that shows the same tokens with deep prefill on and off, and at two step sizes.

Once prompt rows match decode, the grid is no longer needed for exactness.

The ring, overlap and prefetch work can stay as it is.

[BlivionIaG]

First-token time and decode with deep prefill off and on, on 40e3a6ceb16e (1443-full-ttft-decode, passes, TP=1 on one sm_120 card, a 6144-token prompt with --prompt-cache-gib 0, so every request is cold; arms off, on, on again in that order, each a fresh server). The current head 86297964f5 differs from it by merges of main and the pin fixture, not repeated on the device.

Time to first token, 64-token greedy reply, cold, seconds:

request	off	on	on, second run
first after server start	80.54	14.50	14.52
repeated	76.36	7.83	7.89
the cell's concurrent phase	76.29	8.10	8.09

All nine requests have the same token hash (8e23f929d309), so the replies are bit-equal with it on and off. The first request after start includes the one-off cost of the first use of the path; later ones are the steady figure, about 9.7 times faster than off.

Decode is unchanged. Aggregate tok/s from bench_concurrent, greedy, off / on / on again:

prompts	1 stream	4 streams
code	13.3 / 13.2 / 13.3	40.4 / 40.6 / 40.6
chat	13.3 / 13.3 / 13.4	30.0 / 29.9 / 30.1

Slowest first-token time among the four streams: code 2.41 / 2.16 / 2.19 s, chat 3.58 / 3.57 / 3.64 s (not a claim; they are within the spread between arms elsewhere). The cell's title says 8192; the prompt in these requests is 6144 tokens and the table is for that.

For reference, #1441's cold 6144-token first-token time was 73.45 s, measured on #1441's own binary and server, not in this cell.

[ashhart]

Thanks. This is in good shape. Prompt rows now match decode, the snapshot grid is gone after your restores with it off, the tensor parallel cell turned out to be a harness mismatch and now passes, the filtered contract shows every kernel green with the planted controls red, and the first token and decode tables show the speedup without changing a single token. The float64 partials and single rounding match the decode path too.

The head you are about to push picked up a few merges of main and a release after those device runs and checks, so we will run the device check on our side on that head instead of asking you to repeat it. We will confirm CI there as well.

Two smaller things. First, you said the reduced branch and two chunk settings do nothing and offered to remove them. Since they do not change tokens we are happy to keep them, but if you would still rather drop them that is fine by us too. Second, please check that only main's copy of the profile setting (TF_FLASHNEXT_PROFILE) is left on the final head. Your earlier note said this slice's copy was gone, but the patch series we read still shows it. If both are there, please drop the extra row, doc line and test case.

• Synthena (Ash's Agent)