[ElderOrb]

Serves NVIDIA's ModelOpt NVFP4 and FP8 checkpoints on the HIP engine without widening a weight.

What the checkpoint holds, and what the engine keeps

For a projection of N outputs from K inputs ModelOpt writes weight U8 e2m1 codes [N, K/2] (input 2i in the low nibble), weight_scale E4M3 [N, K/16] and weight_scale_2, one global scale. An FP8 layer writes E4M3 codes [N, K] and one f32 scale a projection.

The host keeps exactly those tensors, and the device keeps that layout too - as quant.mlx does - so an upload is a copy. The one transform is at read: an E4M3 group scale widens to BF16, exactly (three mantissa bits fit seven). An F32 block table is refused (ModelOpt writes E4M3, and the kernel reads a 16-bit table), and a stack of experts is refused because its experts carry a global scale each.

The preflight counts what the device holds rather than the stored files: table.deviceBytes keeps U8 and E4M3 code tensors as stored and counts only an E4M3 group table at two bytes a value.

Kernels

quant/nvfp4_tiles.hpp is one product kernel, tf_nvfp4_gemm: a thread an output element, a group's 16 inputs folded in f32 with the group's own E4M3 scale, the one global applied to the sum once, the nibble through an e2m1 bit table.

quant/fp8_tiles.hpp is its sibling for FP8, tf_fp8_gemm: each E4M3 code turns into the exact f32 it names, all of K sums in f32, and the projection's one scale multiplies the result once, as CUDA's qmmf.cu reduce does. That keeps those layers row-invariant and off rocBLAS.

Both serve bf16 activations only, since the engine refuses an fp16 build for a ModelOpt checkpoint; one kernel serves a prompt's rows and a round's, so a row's bits never depend on its round.

Why dedicated kernels rather than the shared GEMM and stream tiles: both step along K in 32-code chunks (kGemmK = 32, nch = a.k >> 5, a chunk's scale index q >> gshift), and NVFP4's group is 16, so a chunk holds two scales: a.group / kGemmK is 0 and the shift would have to be negative. tiles/stream.hpp and tiles/dot2_tiles.hip keep only their MLX code.

The engine

A ModelOpt checkpoint's own bf16 layers (an lm_head, an unquantized linear, the MTP head) read as .dense on the row-invariant dense kernels rather than through rocBLAS, so rocBLAS is not opened at all. That route made plain() false, which had gated a bf16 checkpoint's refusals, so a ModelOpt checkpoint now carries its own: an fp16 build and more than one rank are refused by name, and graphs stay off. A ModelOpt checkpoint with routed experts is refused by name too, before its shards are read.

The format tag

core.quant.Format no longer names NVFP4 or FP8: it and Decoder are the three formats both engines read. A projection still has to carry them, so quant.Projection gains optional nvfp4 and fp8 fields beside a nullable handle, and format()/decoder() return optionals. The family's host projection is its own union, with rows, bytes, read, halves, sliceRows and sliceCols over it; its device projection is a union of quant.Device, quant.nvfp4.Device and its FP8 tensors.

Kernel contract

hip_records_nvfp4.zig holds both entry points' resources from hip-kernel-contract-limits, in gfx1030, gfx1100, gfx1151, gfx1201 order:

• tf_nvfp4_gemm: 29 VGPRs / 49 SGPRs on gfx1030, gfx1100 and gfx1151, 31 / 48 on gfx1201.
• tf_fp8_gemm: 15 / 36 on the first three, 17 / 38 on gfx1201.

No LDS either, bound 256, wave32. The same run compiles every group for the four chips, which is the gfx1100 and gfx1201 build. tests/hip/contract_nvfp4.zig carries a test type for both and the runner builds it; on gfx1151 (Radeon 8060S):

PASS nonfinite hip ModelOpt NVFP4 product: 256 non-finite and 1024 finite outputs match the float64 classes
PASS repeat hip ModelOpt NVFP4 product: five runs byte-identical (1024 bytes), inputs and guards untouched, no overlap
PASS shapes hip ModelOpt NVFP4 product: 4 shapes within the float64 bound (worst 0.001 of the bound), 0 edges refused by the kernel
PASS sums hip ModelOpt NVFP4 product: exact sums at k 4096, where fp16 sums would miss by 847512x and bf16 by 4765421x
PASS nonfinite hip ModelOpt FP8 product: 256 non-finite and 1024 finite outputs match the float64 classes
PASS repeat hip ModelOpt FP8 product: five runs byte-identical (1024 bytes), inputs and guards untouched, no overlap
PASS shapes hip ModelOpt FP8 product: 4 shapes within the float64 bound (worst 0.004 of the bound), 0 edges refused by the kernel
PASS sums hip ModelOpt FP8 product: exact sums at k 4096, where fp16 sums would miss by 442122x and bf16 by 4889576x
PASS limits live ModelOpt NVFP4 product: 256 threads launched, the runtime allows 256; 29 VGPRs, 0 + 0 LDS bytes (recorded 29, 0)
PASS limits live ModelOpt FP8 product: 256 threads launched, the runtime allows 256; 15 VGPRs, 0 + 0 LDS bytes (recorded 15, 0)
SUMMARY gfx1151 161 kernels: 161 pass gates 1-4, 0 fail, 2 known findings; 41 controls: 41 red as planted, 0 missed; 174 limit problems

The 174 limit problems are the repository's older records against this machine's hipcc (ROCm from the SDK, not the ROCm 10.1 the table was filled from); neither ModelOpt launch is among them.

Testing

• zig build test on Windows: passes, 0 failures (7,900+ host tests).
• zig build hip-kernel-test -Dhip=true -Dhipcc=... -Dhip-arch=gfx1151: 20/20 steps, 165/171 tests (6 skipped). Both products are compared with quant.nvfp4.reference and quant.fp8.e4m3 in f64 against the bound each kernel's chain derives, and print the worst error (0.00-0.004 of it), and a row's bits alone equal that row in a round of four, bit for bit.
• Host tests cover the reader (unfit, the exact widen, halves and slices), the preflight count, the FP8 and F32 refusals, the ModelOpt MoE refusal, and that the same BF16 tensor reads .bf16 on an unquantized checkpoint and .dense on a ModelOpt one.

Not in this revision

Served numbers - decode and prompt tok/s, drafted against "draft": false, concurrent against solo, resumed against fresh - are not here: this machine has no ModelOpt checkpoint to serve. The kernels' receipts above are the model-free ones; I will add the served numbers when a checkpoint is on it.

[ashhart]

Thanks, @ElderOrb. Serving ModelOpt NVFP4 and FP8 Qwen3.5 checkpoints on HIP through the BF16 rocBLAS path is in scope. A few things first:

1. The served weights aren't the checkpoint's. weights/modelopt.zig:73 and :90 round each dequantized weight to BF16. Code × E4M3 block scale fits BF16 exactly, and so does a bare E4M3 value. The FP32 global and per-tensor scales don't, and the product is rounded twice (f64, then f32, then BF16). We take no lossy weight formats (docs/pr/SKILL.md:74). Please widen only the exact part and apply those scales to the f32 sum, as CUDA does (qmmf.cu:89), or refuse any tensor BF16 would round.
2. The memory preflight: engine.zig:71 counts stored bytes, but the device holds 2 bytes a weight, about 3.5× NVFP4's stored size. Please count the widened size, so WeightsDoNotFit still refuses a checkpoint that doesn't fit.
3. Admission: please go through core/modelopt.zig, so unsupported recipes, E4M3 NaN scales and a zero, Inf or NaN global are refused by name. Please also refuse qwen3_5_moe ModelOpt experts, which still load through experts.routed unwidened, or say how they load.
4. Tests: the one test's values are all powers of two, so nothing rounds. Please add tests against core/quant/nvfp4.zig's f64 reference, with FP8, values that would round, and a zero global, and show them failing before the change.
5. hip/memory.zig:84 puts every HIP upload, for every family, in 64 MiB windows. Please split that into its own PR, or explain why it belongs here.
6. A CHANGELOG line ending "(hip: ModelOpt NVFP4 and FP8 checkpoints keep their codes and decode in flight #2490). Thanks to @ElderOrb.", and a rebase: hip/engine/mtp.zig, hip/engine/native.zig and weights/projection.zig conflict with main.

We have no AMD GPU, so please post your own main-against-branch numbers, naming the gfx target, along with drafted replies equal to "draft": false.

[ashhart]

@ElderOrb thanks, keeping the codes as stored is a much better design: the NVFP4 weights are now the checkpoint's, the E4M3 → BF16 table is exact, the NaN and zero-global refusals are in, and admission goes through core/modelopt.zig. On the format tag: please use the HIP-local type you offered, so core keeps no Format tag. What's left:

1. FP8 still rounds: weights/modelopt.zig:60 rounds e4m3 × scale to BF16, twice. Widen the bare E4M3 (exact) and apply the scale to the f32 output, or refuse FP8 layers. Please also refuse per-row and zero FP8 scales (:51-55).
2. F32 block scales: read keeps them F32 (core/quant/nvfp4.zig:70,98), but the stream tile reads every table entry as BF16 (nvfp4_decoder.hpp:180-183). Please refuse F32 block scales, since ModelOpt writes E4M3.
3. Row invariance: ops/project.zig:431 sends decode rounds over 16 rows to the GEMM, so a row's bits depend on the round's size. Please choose by o.prefill alone, as the MLX path does, with a test of 17 rows against one.
4. rocBLAS: these checkpoints now serve chat, but their BF16 linears (widened FP8 layers, the MTP head, an unquantized lm_head) run on rocBLAS, whose products vary with rows (engine.zig:195). Please post concurrent = solo and drafted = "draft": false on your gfx target, or route those layers through the row-invariant kernels.
5. Kernel contract: please register tf_nvfp4_gemm and the 8 stream entry points with hip_records, and post hip-kernel-contract on your chip and -limits for every served target. That also re-checks the MLX stream tile after the stream.hpp edit.
6. Other targets: nvfp4_wmma_gemm (nvfp4_tiles.hpp:32) compiles on every target. Please guard it as TF_MATRIX does (or drop the unused wrappers), and confirm gfx1100 and gfx1201 build.
7. Preflight: please count FP8 layers at 2 bytes a weight and the scale table at 2 bytes a group (engine.zig:69-75).
8. Tests: please add a non-power-of-two global, an FP8 case and the qwen3_5_moe refusal, show them failing before the fix, and say whether the kernel tests ran on the card.
9. Housekeeping: one-line comments (core/quant/nvfp4.zig:1-5, launches/nvfp4.zig:1-4, the .hpp headers); the stale doc at config.zig:43; route by recipes or drop the field; drop the docs' RDNA2 claim (engine.zig refuses fp16); drop the windows commit (landed as hip: upload large tensors in windows so the blit can pin them #2584); end the CHANGELOG line "(hip: ModelOpt NVFP4 and FP8 checkpoints keep their codes and decode in flight #2490). Thanks to @ElderOrb."

We have no AMD GPU, so please post main-against-branch numbers, naming the gfx target.

[ElderOrb]

Rebased and reworked since your note; the body above describes the current state. Summary of what changed:

• The shared tiles are gone from this path. Both step along K in 32-code chunks (kGemmK = 32, nch = a.k >> 5, a chunk's scale index q >> gshift), and NVFP4's group is 16, so a chunk holds two scales: a.group / kGemmK is 0 and the shift would have to be negative. quant/nvfp4_tiles.hpp is now one product kernel, tf_nvfp4_gemm plus tf_nvfp4_gemm_f16; tiles/stream.hpp and tiles/dot2_tiles.hip keep only their MLX code, so the MLX stream tile needs no re-check from this change.
• Row invariance by construction: one kernel for every row count, so ops/project.zig no longer picks by m and a row's bits cannot depend on its round.
• Preflight: table.deviceBytes counts what the device holds (U8 codes as stored, an E4M3 tensor at 2 bytes a value) instead of the stored files, with a host test.
• F32 block scales are refused (unfit), and the reader no longer keeps one; the kernel reads a 16-bit table only.
• recipes field dropped; it was admitted and then unused. The stale note at weights/config.zig is corrected and the module and header comments are one line each.
• Evidence on the card: zig build hip-kernel-test -Dhip-arch=gfx1151 on a 8060S is 19/19, with two NVFP4 kernel tests: the product at one row, three and five against quant.nvfp4.reference (2e-4), and a row's bits alone against a round of four (bit for bit). zig build test on Windows passes.

Still open, and in the body: the Format tag in core (needs the family-local projection type on both host and device sides), hip_records registration plus hip-kernel-contract/-limits, and FP8 exactness. On FP8 I would rather not pick for you: the scaled f32 product CUDA uses (qmmf.cu) is the exact one, and refusing FP8 layers refuses mixed checkpoints outright. Which do you want?

[ashhart]

Thanks @ElderOrb, this is a much better shape. The codes and E4M3 tables stay as stored, one product kernel serves
every row count, F32 block tables are refused, the preflight counts what the device holds, the recipes field and the
windows commit are gone, and the kernel tests ran on your 8060S. Thanks too for the clear account of why the shared
tiles can't take a group of 16.

On your FP8 question: the exact route, please. Widen the bare E4M3 values (exact in BF16), sum in f32, and apply the
per-tensor scale once to the f32 result, as CUDA's reduce does (zig/kernels/cuda/qmmf.cu:92). A sibling of your
NVFP4 kernel that decodes E4M3 in flight would do that, and it would also keep those layers row-invariant and off
rocBLAS. If that makes this PR too big, refuse FP8 layers by name here and bring the FP8 product as its own PR.

What's left, smallest first:

1. Docs and comments.
   • docs/quantization.md:140 still says FP16 where a card has no BF16 dot2. But engine.zig:208-212 refuses fp16 for
     this path, so tf_nvfp4_gemm_f16 can't run either.
   • docs/quantization.md:142 and the CHANGELOG line say "a quarter of the bytes". Codes plus a BF16 scale per 16
     come to 5 bits a weight on the device, about 31 %.
   • Some comments still describe the old version: hip/ops/project.zig:410 ("the dot2 tiles"),
     hip/kernel_tests/nvfp4.zig:1 and weights/checkpoint.zig:60 ("dequantizes"). There's also a dead branch at
     weights/modelopt.zig:59.
2. One rounding for the global. quant/nvfp4_tiles.hpp:51 multiplies each group scale by the global, which is a
   rounding per group. Each E4M3 scale is exact in f32, so fold each group with its own scale and apply the global
   once to the sum.
3. The kernel test's bound. 2e-4 (kernel_tests/nvfp4.zig:104) is far looser than an f32 sum needs at K ≤ 128.
   Please use the derived bound the dense test uses (kernel_tests/dense.zig:35-41) and log the worst error.
4. Tests still owed from last round. A non-power-of-two global (every global in the tests is 0.25, 0.5, 1 or 2),
   an FP8 case, and qwen3_5_moe ModelOpt experts refused by name with a host test. Please show each one failing
   before its fix.
5. The BF16 layers on rocBLAS. A text-only ModelOpt checkpoint isn't chatBf16 (engine.zig:80-84), so its BF16
   lm_head and widened layers take rocBLAS (project.zig:52-61), whose products vary with the round's rows. Please
   route them through the row-invariant dense kernels, as the chat path does, or show drafted equal to "draft": false
   and concurrent equal to solo on the card. The MTP head on rocBLAS is fine, since its drafts are verified.
6. The Format tag out of core (core/quant/types.zig:10), as you offered.
7. Kernel contract. Please register tf_nvfp4_gemm in hip_records, post hip-kernel-contract on gfx1151 and
   -limits for the served chips, and confirm that gfx1100 and gfx1201 build.
8. Served numbers on your card, naming the gfx target: decode and prompt tok/s, drafted equal to "draft": false,
   concurrent equal to solo, and resumed equal to fresh.

We have no AMD GPU, so your numbers are the receipt for this one. Main is closed for a release right now, but none of
this needs to wait for that.

[ashhart]

Thanks @ElderOrb. Here's what this head (786f8b61) has from our last note:

• the grouped and fused products decline a projection that has no handle (zig/src/hip/ops/project.zig:95, :104, :112, :153, :191, :195);
• FP8 NaN codes are refused at read (zig/src/families/qwen3_5/weights/modelopt.zig:52);
• the preflight counts lm_head.* and mtp.* (weights/table.zig:184-187, :205-208);
• table.zig:25 fits in 120 columns.

It looks like you're in the middle of getting the 27B to serve. Before the next push, here's what's left, blocking items first:

1. Debug logging. The PROBE lines are still in: hip/ops/project.zig:37, :44, :53, :62, :419, :424, :443, :469 and :473; backend/hip/engine/engine.zig:340-347; backend/hip/engine/mtp.zig:266-344; and the stack dump at backend/hip/engine/native.zig:226. Please remove them.
2. projectRouted returns 0 when a projection has no handle (project.zig:242). Its result is the output's device address, and hip/moe.zig:104 hands it to moeAct (:107), so 0 would be read as a buffer. Please keep error.BadShape there, as .dense and .bf16 do.
3. The plan leaves out rocBLAS's workspace. sizedBytes (engine.zig:307) now counts the 32 MiB workspace only when e.blas is set. But size() opens rocBLAS (:277) only after prepare has picked the stream count through served and planBytes (native.zig:225, engine.zig:336). So every unquantized checkpoint's plan now misses those 32 MiB, which changes main's path. Please count from what size() will open, not from e.blas.
4. rocBLAS for the vision tower. engine.zig:275 opens rocBLAS and allocates its workspace for every ModelOpt checkpoint, even one served without vision. If rocBLAS is missing while vision is on, it only warns (:283), and the first image fails later. Please open rocBLAS only when the vision tower is loaded, and in that case return the error, as the unquantized path does. The body also still says rocBLAS isn't opened at all.
5. CI. Lean finds a new two-line comment at weights/projection.zig:62-63 (zig job), so Linux zig build test still hasn't run on this head.
6. Still open from our last list:
   • The ModelOpt refusals still run after loadRankWith has read every shard (engine.zig:208, refusals at :217-228), and their messages still say "a bf16 checkpoint".
   • The .dense test still sets t.modelopt by hand (weights/checkpoint_hf_test.zig:368-371). Please open a small ModelOpt checkpoint through Checkpoint.open and mtp() instead.
   • Two tests: one that plants an FP8 NaN code for modelopt.zig:52 to refuse, and a host test that projectGroup declines an NVFP4 and an FP8 projection.
   • Leftovers:
     • the rocBLAS plumbing to the MTP head and its comment (mtp.zig:43-44, hip_lanes.zig:128);
     • kE2m1, kE2m1F16, route, out16 and take_item in quant/nvfp4.hpp, and its stale comment at :40;
     • the F16 and F32 table branches and out16 in quant/nvfp4_tiles.hpp:19-20 and :55;
     • Config/detect, matches and sliceStack in core/quant/nvfp4.zig;
     • the stale comments at weights/host.zig:94 (per_row) and hip/kernel_tests/nvfp4.zig:69 ("the tiles");
     • "other backends refuse the format" in docs/quantization.md:156-157 and the CHANGELOG line. The CUDA engine serves these checkpoints;
     • backend/hip/engine/native.zig:21 still lists the ModelOpt formats for qwen3_5_moe too, which is refused.
   • History: 41 commits, 19 of them probe:. Please squash so that each commit builds.
7. Served numbers on your card, as in our last note, using nvidia/Qwen3.8-27B-NVFP4 at 482ca0f3832238542f8f5295dde86b5f22711d80 and naming the gfx target and ROCm version:
   • greedy decode and prompt tok/s;
   • drafted equal to "draft": false;
   • two concurrent requests equal to their solo runs;
   • resumed equal to fresh;
   • one MLX checkpoint's greedy reply equal on main and this head.

We have no AMD GPU, so your numbers are the receipt for this PR.

• Synthena (Ash's Agent)

[ashhart]

Thanks @ElderOrb. This head (8faa0a8a) takes out the debug logging from our last note: the PROBE lines in hip/ops/project.zig, engine.zig and mtp.zig, and the stack dump in native.zig. The new 64 x 64 kernels sum each output in a fixed order whatever the row count, so a row's bits are still its own. For NVFP4 the order is the one you had before: a group's 16 inputs in order, fmaf with the group's scale, the global once (zig/kernels/hip/quant/nvfp4_tiles.hpp:74-106).

New on this head:

1. FP8 reads past its codes. fp8_tiles.hpp:55 loads w[c * k + kc + j] with no check that kc + j < k. When K isn't a multiple of 64, a column reads the next column's codes, and the last column reads up to 63 bytes past the tensor. The {48, 96} shape in hip/kernel_tests/fp8.zig:100 already does this. The sums stay right, since those inputs are zero, but the read is out of bounds. Please clamp it as nvfp4_tiles.hpp:57 does, or refuse such a K in launches/fp8.zig:48.
2. A decode row does 64 rows of work. Every block computes 64 rows. Rows past m repeat row m - 1 (fp8_tiles.hpp:59, nvfp4_tiles.hpp:68) and are dropped at the store. So a one-row decode round does 64 times the multiplies of the per-output kernel it replaces. Please post decode tok/s at one stream for this head against the previous kernel. If you add a path for few rows, it must sum each output in the tile's order, so a row's bits don't depend on its round.
3. The records are stale. hip_records_nvfp4.zig still records 0 LDS bytes and 29 and 15 VGPRs, but the kernels now stage 11 KiB and 12 KiB in LDS. Please re-derive the four chips' rows with hip-kernel-contract-limits, post hip-kernel-contract on gfx1151 at this head, confirm gfx1100 and gfx1201 build, and say whether hip-kernel-test ran on the card at this head.
4. Graphs are back on. engine.zig:227 now turns graphs off only for plain() or chat, so a ModelOpt checkpoint keeps them. That came in a probe: commit (4b2148e9). Please keep them off, as agreed, or show graphed replies equal to eager on the card.

Still open from our 12 Nov note:

5. Lean. weights/projection.zig:62-63 is still a two-line comment, so CI's zig job stops at the lean check and Linux zig build test hasn't run on this head.
6. projectRouted returns 0 for a projection with no handle (hip/ops/project.zig:238). Please keep error.BadShape there.
7. rocBLAS. The plan counts its workspace only when e.blas is set (engine.zig:307), which misses it for unquantized checkpoints. And rocBLAS still opens for every ModelOpt checkpoint, with only a warning when it's missing (:275-287). Please count from what size() will open, and open it for ModelOpt only with the vision tower, returning the error.
8. The refusals still run after loadRankWith (engine.zig:208), and still say "a bf16 checkpoint" (:220, :224).
9. Tests. The .dense test still sets t.modelopt by hand (weights/checkpoint_hf_test.zig:368-370). Nothing plants an FP8 NaN code for modelopt.zig:52 to refuse, and no test shows projectGroup declining an NVFP4 and an FP8 projection.
10. Leftovers. The rocBLAS plumbing to the head (mtp.zig:43-44, hip_lanes.zig:128). kE2m1, kE2m1F16, route, out16 and take_item in quant/nvfp4.hpp. The F16 and F32 branches of nvfp4_scale (nvfp4_tiles.hpp:20-26). Config/detect, matches and sliceStack in core/quant/nvfp4.zig. Stale comments at weights/host.zig:94 and hip/kernel_tests/nvfp4.zig:69. "Other backends refuse" in docs/quantization.md:155-156 and the CHANGELOG line. ModelOpt formats listed for qwen3_5_moe (native.zig:21). With the probes gone, the catch |err| { return err; } blocks in mtp.zig:266-334 and engine.zig:340-345 can be plain try.
11. Rebase and history. The branch sits on the 1.2.7 tag, so its CHANGELOG line is under ## 1.2.7 (CHANGELOG.md:111). Please rebase on main, move the line under Unreleased, and squash the 47 commits so each one builds.
12. The body still describes a thread an output element, rocBLAS never opened, and graphs off.
13. Served numbers on your card with nvidia/Qwen3.8-27B-NVFP4 at 482ca0f3, naming the gfx target and ROCm version: greedy decode and prompt tok/s, drafted equal to "draft": false, two concurrent requests equal to solo, resumed equal to fresh, and one MLX checkpoint's greedy reply equal on main and this head.

We have no AMD GPU, so your numbers are the receipt for this PR.

• Synthena (Ash's Agent)