[BadAd84]

What this changes

TF_FLASHNEXT_MTP_WEIGHTS=checkpoint|mxfp8 says how the Flash Next MTP drafter's linears are kept in memory. checkpoint, the default, leaves the rows as stored and runs no new code. mxfp8 re-codes the drafter's BF16 linears at load into the MXFP8 projection the mixers already use, so a draft pass walks about half their bytes. A draft is only a proposal the verify forward checks, so replies are the same and only the rounds a reply takes move.

• Issue: Refs Flash Next on CUDA: an opt-in MXFP8 coding for the drafter's BF16 linears (TF_FLASHNEXT_MTP_WEIGHTS) #3162 (the proposal; no maintainer decision on its gate yet, so this is a draft until one lands).
• Stacked on: main.
• atc overlap --base origin/main --json against main at a3aecd8: complete is false (pullFilesUnavailable and timeLimit, so some pull requests are unknown). It lists textual conflicts with qwen27: serve the 27B pair, refuse what a pair cannot keep, and die with the peer #1434, DeepSeek-V4.1: read the pack's weight spans past the page cache (core/direct_io) #1924, Metal: serve the dense Qwen3.8-27B straight from a GGUF file (Q4_K_M, Q8_0) (#1920) #2078, deepseek_v41: reply logprobs normalize over all TP head columns (#1324) #2125 and CUDA admission: an integrated GPU under WSL2 reads the CUDA pool (RTX Spark N1X), plus an N1X receipt #2107. Each shares only CHANGELOG.md and/or zig/src/platform/environment.zig with this one (their changelog and registry additions land in the same places), and none touches the drafter's loader.

Checks on this head

• Head commit: e16ca9d (base a3aecd8, the tip of main when I pushed). Two commits: the change and its receipt.
• Lean checker, python3 tools/zig/lean_check.py, base and head: 341 and 341, 0 new. python3 tools/pr_check.py --base origin/main --pr 9999 reports 0 errors and 0 warnings.
• Tests run: zig fmt --check on the changed files rc 0; zig build compile-config-server -Dtarget=aarch64-linux-gnu rc 0; zig build test-cuda-host --summary all on Windows with Zig 0.17.0: 383 of 383 build steps succeeded and 5,911 of 6,054 tests passed with 143 skipped, none failed; zig build test-environment-host rc 0.
• CI: not yet run.
• Not run, and why: HIP and Metal (the change is in the CUDA family and platform/environment.zig's registry only); the complete zig build test (macOS Metal tests need a Mac).

Receipt

• Main against branch: docs/receipts/flashnext-mtp-weights/, cells below. Same machine, checkpoint and settings for all four arms, fresh server per arm, alternating main, branch with the setting unset, branch with mxfp8, main again.
• Boxes it reaches: Flash Next CUDA on GB10 with the NVFP4 checkpoint's BF16 mtp.* rows. A checkpoint whose drafter is not BF16 loads as stored whatever the setting says.
• New switch? TF_FLASHNEXT_MTP_WEIGHTS, default checkpoint, accepted checkpoint or mxfp8 (a registry row in platform/environment.zig; any other word refuses to start by name).
• Environment: TensorFold 6c5cef8 plus the same change (the branch was rebased onto a3aecd8 afterwards; only CHANGELOG.md conflicted), Zig 0.17.0, CUDA 13.3, one DGX Spark (GB10, sm_121, 128 GB unified memory), nvidia/Qwen3.8-Flash-Next-FP4-Local at revision b9ce18a6887e18c089fdea8509a33e63429a3300, --parallel 4, 131,072-token context, bf16 KV, MEM_CHECK=0.
• Paths and platforms: the drafter's load path on CUDA; the verify path is untouched. I ran GB10 only.
• Exactness, on main, the branch with the setting unset and the branch with mxfp8, each booted fresh with the prompt cache on and again with --prompt-cache-gib 0, --parallel 4, bf16 KV:
  • drafted equals "draft": false: served_exactness.py, 8 prompts, token ids, 256 tokens, equal 8 of 8 in all three arms with the cache on and again with it off. The plain request drafted 0 tokens and the drafted ones accepted 13 to 237 a reply, so the two paths really differ.
  • resumed equals fresh: 7 of 7 chat turns resumed from the cache equal the same turns on a cold server, in all three arms; tools/bench_cache.py repeated and concurrent requests equal the first, 4 of 4.
  • concurrent equals solo: tools/bench_concurrent.py --alone --serial at 1, 2 and 4 streams, code and chat, greedy and temperature 1.0: 42 of 42 equal in each, 0 failed requests; the six golden prompts at 1, 2, 4 and 6 streams: 13 of 13 equal.
  • against main: every reply of the branch with the setting unset and with mxfp8 has the same token ids or hashes as main's (16 of 16 concurrent, 6 of 6 golden, 8 of 8 drafted and plain, 7 of 7 turns).
  • zig/tests/qwen35/serve_check.py: 15 ok and 0 failures in each arm.
• Main against this branch: below, and raw files in the receipt.
• Decode speed before and after, from tools/bench_openai.py and tools/bench_concurrent.py (greedy, 256 tokens, 3 reps), aggregate tok/s:

prompt	streams	main	main again	branch, unset	branch, mxfp8	mxfp8 against the mean of main
code	1	73.2	72.4	72.4	74.1	+1.8%
code	2	138.0	137.4	137.6	142.0	+3.1%
code	4	242.2	240.1	240.3	247.7	+2.7%
chat	1	46.8	46.6	46.6	46.6	-0.2%
chat	2	86.8	86.3	86.5	88.6	+2.4%
chat	4	155.5	154.3	155.2	158.5	+2.3%

With the setting unset the branch is level with main within the run-to-run spread (-0.5% to +0.2% against the mean of the two main arms). One chat cell at one stream is flat. Sampled decode (temperature 1.0) and --drafter off were not run; this receipt is greedy only.

• Prompt speed before and after: cold first request, --prompt-cache-gib 0, 8k 3.44 to 3.46 s and 32k 14.14 to 14.19 s in all arms; the change cannot reach prompt processing (the drafter runs only in decode).
• Licence of any new weights or drafter: n/a. The drafter is the checkpoint's own mtp.* (NVIDIA Open Model Agreement).

Checklist

• The title and description match the head commit, including the receipt and the stack.
• New tests fail before the change, pass after it, and skip cleanly without their dependency.
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.

For a change that reaches inference:

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (Run on main, the branch unset and the branch with mxfp8; see Exactness above.)
• No precision traded for speed in the output: the verify forward is unchanged, so replies are identical. The drafter's own logits change under mxfp8, which only changes which tokens it proposes.
• Decode and prompt processing are no slower than main at one stream and the widest admitted; with --drafter, no slower than without it, greedy and sampled. (Greedy decode at 1, 2 and 4 streams is measured above, and 4 is the widest this boot admits at a 131,072-token window; prompt speed is level. Not run: sampled decode speed, and --drafter against none.)

[BadAd84]

Sampled decode, and the drafter on against off, measured after the PR was opened. Same GB10, the PR's base 6c5cef8, bf16 KV, --parallel 4, a fresh server per arm, upstream's tools/ from that base.

This family has no --drafter flag to drop, so "without the drafter" is the request field "draft": false.

Drafter on against "draft": false: faster in all 48 cells, in every arm. The ratio runs from 1.43x (sampled chat, 4 streams) to 2.79x (greedy code, 2 streams), and every "draft": false reply drafted 0 tokens.

Greedy, drafter on: with the setting unset the PR is level with main, and with mxfp8 it is +1.1% to +4.4% faster in all 6 cells.

Sampled (temperature 1.0), drafter on, against the mean of 4 boots of main (3 boots each for the PR unset and with mxfp8):

prompt	streams	PR unset	PR mxfp8
code	1	-0.3%	+1.9%
code	2	-0.4%	+0.4%
code	4	-0.4%	+0.9%
chat	1	-0.5%	-2.0%
chat	2	-0.3%	-1.2%
chat	4	-0.1%	+0.2%

Main's own boots spread by 1.6% to 2.0% in these cells, so the unset arm is inside main's range everywhere. mxfp8 is slower than every main boot on sampled chat at 1 and 2 streams. In those cells the re-coded drafter's drafts were accepted slightly less often (2.31 tokens a chunk against 2.39 at one stream). Over 9 seeds at one stream (bench_openai) mxfp8 is faster, +2.2% on code and +1.9% on chat, so the loss depends on the seed.

So the last checklist box stays unticked: mxfp8 isn't shown to be no slower than main for sampled chat at 1 and 2 streams. The raw files and scripts are in my tree, and I can add them to the receipt directory if you want them.

[ashhart]

Thanks @BadAd84. The host tests pass, and keeping checkpoint as the default looks good.

We will run the device check on our side with TF_FLASHNEXT_MTP_WEIGHTS unset, checkpoint and mxfp8. Each mode must preserve the plain target's complete generated tokens for greedy and seeded requests, including resumed turns and one and four streams. We will compare acceptance, resident memory, prompt speed and decode speed against main before merging.

• Synthena (Ash's Agent)