[Shoestring-dev]

What this changes

A new HIP kernel group, exl3/exl3.hip, adds the HIP side of the single EXL3 projection that deepseek_v41's CUDA path calls as tensorfold_exl3_linear_v4.rot_in and .linear (exl3_block/linear.cu). It serves the LM head (5120 to 129,280, fp32 logits), Engram's wkv (6144 to 25,600) and the indexer's wk (512 to 128):

• tf_exl3_rot_in: x times suh, rotated by 128 and rounded to fp16 once. These are CUDA's rot_in operations in its order: the fp32 product, fwht128's butterflies, then the scale.
• tf_exl3_linear_k{4,6,8,10,12,16}_w{4,8}: the trellis linear at 2, 3, 4, 5, 6 and 8 bits (mul1, the codebook CUDA launches) on 4 and 8 waves a block.
  • A lane decodes its 16 x 16 tile values with one fp16 fma each. A row's sums are v_dot2_f32_f16 chains in a fixed order, so a row's bits do not depend on the rows that share its launch (1 to 16 rows a pass, any number of passes).
  • The waves' sums, the output rotation and svh follow CUDA's finish.
  • The wave count is CUDA's dense plan without a K split. CUDA's emitter refuses a split on this path, so the kernel has none, and it has no bias, because no caller passes one.
• Launches in launches/exl3.zig, with the shapes they refuse (fits, widthIndex) and the wave count (waves) host-tested there.

Nothing launches these outside their tests yet. Every HIP build compiles and embeds one more code object: 379 KB and about 3.9 s of hipcc on gfx1151 (the draft group is 39 KB and 0.7 s).

Only the K sums run in another order than CUDA's mma.m16n8k16. The rotations and the epilogue are CUDA's operations in CUDA's order, so the outputs are held to float64 (below), not to CUDA's bits. The end-to-end token gate against CUDA's serial path stays where #2065 put it: in the TP=2 pull request.

• Issue: Refs DeepSeek-V4.1 on HIP: a stack on main's deepseek_v41 family for two Strix Halo boxes (gfx1151, EXL3) #2065. The first deepseek_v41 HIP kernel pull request, after the TCP Kind (fabric/tp: a TCP Kind for hosts without RDMA #2718, merged) and its HIP device side (fabric/tp: HIP device side of the TCP Kind #2961). Next come the prompt path (unpack and the prompt tiles, registered bit-equal to this linear, so that prompt rows equal decode rows), then the grouped dense projections and the experts.
• Stacked on: main at 93d77e5. One commit, 6750d98. It merges clean on main 222f038, 59 commits later.
• Overlap: checked against the file lists of the 356 open pull requests (gh pr list), and with git merge-tree on each one that shares a code file.
  • hip: ModelOpt NVFP4 and FP8 checkpoints keep their codes and decode in flight #2490, Kernel contract: the HIP relu² stream tile at 5, 6 and 8 bits (part 1 of 3 for #3061) #3163, Nemotron on HIP: read each matrix at its own oQ width (part 2 of 3) #3242 and Nemotron on HIP: read the MTP head file at its own width (part 3 of 3) #3251 add lines to the same HIP registries (launches.zig, kernel_tests.zig, contract_main.zig, hip_records*.zig).
  • MoE route tap: optional TF_ROUTE_TAP trace of global top-k #1886, hip flashnext: a shared round's streams draft in one head pass a step #2696, cuda: GLM, MiniMax and Nemotron placed from one launch, by flags or aliases (T1b of #2560) #2919, ranks: every rank connection proves a shared key before its first protocol byte #3041, hip ranks: every byte after the key proof crosses the link in sealed records #3075, flashnext ranks: every byte after the key proof crosses the pair's link in sealed records #3076, tests: the HIP followers' step-parser tests run in CI (test-hip-followers-host) #3091, tp ranks: every byte after the key proof crosses the bootstrap, fate and plan links in sealed records #3101, cuda ranks: every record after the key proof crosses the liveness lines sealed #3102, cuda ranks: a keyed group's tensors cross NCCL under TLS, or the rank does not start #3103 and logit_bias: token biases and -100 bans on each row's draw for Qwen CUDA, Qwen3.5-2B Metal and Nemotron Parse HIP #3146 add build lines to zig/build/hip.zig.
  • None conflicts with this branch in a file it changes. The others share only docs/kernel-contract.md, where this adds a paragraph.

Checks on this head

• Head commit: 6750d98 on main 93d77e5. The suite and the limits ran on this head. The GPU runs used binaries built from a tree identical to this head except docs/kernel-contract.md.
• Lean checker: python3 tools/pr_check.py --base 93d77e5da reports 0 errors, lean 341 on the base and 341 on this head, 0 new. Its one advisory warning is the Unreleased line, which you write. No comment is past 120 columns, trailing comments included. hip_records.zig gains the file on an existing line and stays at 600 lines.
• Tests run:
  • zig build test test-golden -Dhip=true with ROCm 10.1's hipcc on Linux x86_64 (CPU only): 754 of 754 steps, 10,043 of 10,122 tests pass, 79 skip, exit 0. On main 93d77e5: 754 of 754 steps, 10,041 of 10,120 tests pass, 79 skip. Both runs print the same 25 failed command lines for passing steps that wrote to stderr.
  • zig build hip-host-test -Dhip=true on macOS (M4 Pro): 79 of 79 pass. One new host test covers the launches' refusals, the wave counts and the LDS size; a planted fault in it fails it by name.
  • zig build hip-kernel-test on gfx1151: 220 passed, 6 skipped, 0 failed, against 216 passed, 6 skipped, 0 failed on main 93d77e5 (another host of the same model and runtime). The three new tests:
    • The linear decodes every width's weights as core/exl3_format.zig's unpack does, bit for bit. The input is the identity, so each output row is a weight row through the output rotation, and the host repeats the kernel's fp32 butterflies. This runs at 4 waves on the checkpoint's tile layout and at 8 waves on the strips layout.
      • The trellises are built from one-signed tiles (a short search over each tile's stream): all positive weights against an identity padded with +0, and all negative ones against -0. Every zero product then enters dot2 as +0. RDNA 3.5's dot2 rounds a sum with a -0 lane apart from the same sum with +0 (caps.zig's dot2_signed_zero), so random weights against a zero-padded identity cannot be bit-exact there.
    • A row's outputs are the same bits alone, in windows of 2 to 24 rows from any first row, and in a 64-row launch (four passes of 16).
    • rot_in's fp16 rows equal the host's fp32 steps rounded once, from fp32, fp16 and bf16 rows: 391,680 roundings, so a 1-ulp fp32 error shows. Replayed on the host with the test's own inputs, a 1-ulp error in the scale (a division by sqrt(128) for the product) moves 22 of the fp16 outputs, one rounding of x * suh * scale in place of two moves 29, and fp16 products move 168,635. Without the kernel's opaque() guard, ROCm 10.1's hipcc fuses that product and its fp16 conversion into v_fma_mixlo_f16/v_fma_mixhi_f16 (one rounding); with it, v_mul_f32 then v_cvt_f16_f32, as on CUDA.
  • Each new kernel test fails with one planted fault, applied alone and rebuilt, and passes without it:
    • Without a plant: the three EXL3 tests pass.
    • The decoder reading each pair's second window one bit off: only the decode test fails (2-bit, + weights: row 1, column 0 is 0x40270d0b where 0x411da43f).
    • The multi-row path chaining a row's dot2 terms in another order: only the window test fails (rows 0 to 1 differ from the 64-row run at byte 0).
    • rot_in dividing by sqrt(128) instead of multiplying by its fp32 scale (a 1-ulp fp32 error): only the rot_in test fails (an fp16 output is 14274 where 14273, one ulp).
    • rot_in without its opaque() guard (one rounding): only the rot_in test fails (15453 where 15454, one ulp).
    • rot_in rounding its products to fp16 before the butterflies: only the rot_in test fails (42438 where 42418).
  • zig fmt --check build.zig zig and git diff --check: clean.
• CI: the eight checks pass on this head (fmt, pytest, zig, zig-windows, zig-macos-dev-tools, release-paths, and benchmark-clients on Ubuntu and Windows); the two nightly host jobs skip on pull requests.
• Not run, and why:
  • gfx1030, gfx1100 and gfx1201, because I have only gfx1151. Their records come from ROCm 10.1's compile, as in feat(hip): DFlash2 drafter kernels for qwen3_5 #2545.
  • Metal, CUDA and XPU: untouched.

Receipt

n/a for inference: nothing calls these kernels yet. For the same reason there are no token hashes; the pull request that wires deepseek_v41's HIP lane carries them, and the checkpoint needs TP=2 across two boxes.

• Environment:
  • Machine: AMD Ryzen AI Max+ 395, Radeon 8060S (gfx1151), 128 GB, Linux 7.2.7, ROCm 10.0.0's runtime (HIP 7.15).
  • Toolchain: Zig 0.17.0. The kernels, the contract and the records were built with ROCm 10.1's hipcc (HIP 7.16.26385), which main's HIP records hold.
• Paths and platforms the change reaches: every HIP build compiles and loads the new code object. The kernels launch only from kernel_tests/exl3.zig and the contract.
• Exactness, decode speed, prompt speed: n/a, no served path changes.
• Licence of any new weights or drafter: n/a.

Kernel contract

• Kernels registered (zig/tests/hip/contract_exl3.zig). None replaces a kernel, so each is held to float64:
  • tf_exl3_rot_in, bf16 rows to fp16: within float64's bound at depth 8 (the product and seven butterflies). Gate 4 follows CUDA's rot_group, inputs to 64: fp32 is not exact on 128-term rotations, so it proves fp16's miss alone.
  • tf_exl3_linear at each of the six widths, fp32 out (the head's), and at 4 bits with bf16 out. Each registration's shapes reach both of its kernels:
    • the indexer's wk in windows of 16 and 64 rows and at one row (4 waves);
    • Engram's wkv at 4 rows (8 waves);
    • the plan's 8-wave edge at 4 and 16 rows (8192 x 1024), and the shape below it (8064 x 1024);
    • gate 4's shape, Engram's K at one row of one column block (128 x 6144, 4 waves);
    • the edges at 1, 15, 16, 17 and 64 rows, and at K and N of 128, 256 and the largest served;
    • at 8 bits, also the head at one row (129,280 x 5120, 8 waves). N sets only the grid, so the head is registered at one width, which keeps its cost to one registration.
  • The linear's gate 4 is exact. All-ones trellis states give one weight at every width (-0.6513671875), so 6144 terms of 4096 times it are integers under 2^24 in any order, and fp16 overflows. A comptime check in the test pins both facts.
    • Gate 4 runs on 4 waves. The 8-wave kernel differs only in how many waves' partial sums the reduction adds, and gate 3 holds it within its float64 bound at every 8-wave shape.
    • The reference's output rotation is the unscaled butterflies in float64, then the scale, so it is exact on those integers: the 127 outputs of a block that are 0 come out 0. fmt.rotate scales each term first and leaves up to 5e-9 there, which the exact-sum bound would count against the kernel's exact zeros.
    • The bf16-output registration states that its output's rounding hides a bf16 sum's miss on these inputs, and leaves the proof to the fp32 ones.
• Planted controls, each the real kernel launched wrongly from the host, so production code does not change:
  • tf_exl3_rot_in reads the next block's suh: red at gate 3 (at 64 x 512 x 128, output 0 is 2,447 times its bound).
  • tf_exl3_linear reads its input rows one row late: red at gate 3 (at 16 x 128 x 512, output 0 is 235 times its bound).
• hip-kernel-contract on gfx1151 at 6750d98: SUMMARY gfx1151 169 kernels: 169 pass gates 1-4, 0 fail, 2 known findings; 44 controls: 44 red as planted, 0 missed; 1 limit problems; gates 444983 ms, limits 2447 ms. The exit status is 1, from the one limit problem.
  • main 93d77e5 on another host of the same model and runtime: SUMMARY gfx1151 161 kernels: 161 pass gates 1-4, 0 fail, 2 known findings; 42 controls: 42 red as planted, 0 missed; 1 limit problems; gates 429664 ms, limits 2425 ms. The same limit problem: tf_moe_combine_gate, launched at 256 threads where the runtime allows 1,024.
  • The EXL3 lines:
    • rot_in: 256 non-finite and 3840 finite outputs match float64's classes. Its 7 shapes are within float64's bound, worst 0.993 of it. Gate 4: fp16 sums would miss 250x and bf16 912x.
    • Each fp32-output linear: 256 non-finite and 768 finite outputs match. Its 15 shapes (16 at 8 bits, with the head) are at worst 0.001 of the bound. Gate 4 reads "exact sums at k 6144, where fp16 sums would miss by inf and bf16 by 2096x".
    • The bf16-output linear: its 15 shapes are at worst 0.789 of the bound.
    • Five runs of every registration are byte-identical, and every guard is intact.
  • Time: the EXL3 entries add about 7 to 15 s: the run's gates take 445.0 s, and main's take 429.7 s and 438.4 s on two other hosts of the same model and runtime to the gates.
    • Their float64 references unpack W by slices of K tiles and sum by ranges of columns on reference_threads, as affine.zig does, in each output's K order. They are bit-identical to the one-thread reference on the shapes I compared.
    • On the host alone (M4 Pro), the gate work of the seven linear registrations takes 7.8 s, against 261.8 s with one thread and the earlier shapes, which registered the head at every width.
• hip-kernel-contract-limits with ROCm 10.1's hipcc (no GPU), at 6750d98: all 52 new lines (13 launches on 4 chips) read record current. The run reports FAIL kernel-contract-limits: 3 problems, the same 3 as main 93d77e5: tf_moe_combine_gate's records are stale on gfx1030, gfx1100 and gfx1151.

  • The linears use 205 to 227 VGPRs, 43 to 46 SGPRs, no scratch, 2 to 16 KiB of static LDS, and at most 32 KiB of dynamic LDS (the waves' sums for 8 rows). rot_in uses 16 VGPRs and no LDS.

  • The records are in hip_records_exl3.zig, and hip_records_families.zig's count test lists them. The RECORD lines, in the order gfx1030, gfx1100, gfx1151, gfx1201:

    RECORD exl3 tf_exl3_rot_in: .usage = .{ .{ .vgprs = 16, .sgprs = 12, .lds = 0, .private = 0, .bound = 128, .wave = 32 }, .{ .vgprs = 16, .sgprs = 18, .lds = 0, .private = 0, .bound = 128, .wave = 32 }, .{ .vgprs = 16, .sgprs = 10, .lds = 0, .private = 0, .bound = 128, .wave = 32 }, .{ .vgprs = 16, .sgprs = 9, .lds = 0, .private = 0, .bound = 128, .wave = 32 }, }
    RECORD exl3 tf_exl3_linear_k4_w4: .usage = .{ .{ .vgprs = 206, .sgprs = 45, .lds = 2048, .private = 0, .bound = 128, .wave = 32 }, .{ .vgprs = 205, .sgprs = 45, .lds = 2048, .private = 0, .bound = 128, .wave = 32 }, .{ .vgprs = 205, .sgprs = 45, .lds = 2048, .private = 0, .bound = 128, .wave = 32 }, .{ .vgprs = 205, .sgprs = 46, .lds = 2048, .private = 0, .bound = 128, .wave = 32 }, }
    RECORD exl3 tf_exl3_linear_k4_w8: .usage = .{ .{ .vgprs = 214, .sgprs = 43, .lds = 4096, .private = 0, .bound = 256, .wave = 32 }, .{ .vgprs = 213, .sgprs = 43, .lds = 4096, .private = 0, .bound = 256, .wave = 32 }, .{ .vgprs = 213, .sgprs = 43, .lds = 4096, .private = 0, .bound = 256, .wave = 32 }, .{ .vgprs = 213, .sgprs = 44, .lds = 4096, .private = 0, .bound = 256, .wave = 32 }, }
    RECORD exl3 tf_exl3_linear_k6_w4: .usage = .{ .{ .vgprs = 209, .sgprs = 45, .lds = 3072, .private = 0, .bound = 128, .wave = 32 }, .{ .vgprs = 208, .sgprs = 45, .lds = 3072, .private = 0, .bound = 128, .wave = 32 }, .{ .vgprs = 208, .sgprs = 45, .lds = 3072, .private = 0, .bound = 128, .wave = 32 }, .{ .vgprs = 208, .sgprs = 46, .lds = 3072, .private = 0, .bound = 128, .wave = 32 }, }
    RECORD exl3 tf_exl3_linear_k6_w8: .usage = .{ .{ .vgprs = 217, .sgprs = 43, .lds = 6144, .private = 0, .bound = 256, .wave = 32 }, .{ .vgprs = 216, .sgprs = 43, .lds = 6144, .private = 0, .bound = 256, .wave = 32 }, .{ .vgprs = 216, .sgprs = 43, .lds = 6144, .private = 0, .bound = 256, .wave = 32 }, .{ .vgprs = 216, .sgprs = 44, .lds = 6144, .private = 0, .bound = 256, .wave = 32 }, }
    RECORD exl3 tf_exl3_linear_k8_w4: .usage = .{ .{ .vgprs = 209, .sgprs = 45, .lds = 4096, .private = 0, .bound = 128, .wave = 32 }, .{ .vgprs = 208, .sgprs = 45, .lds = 4096, .private = 0, .bound = 128, .wave = 32 }, .{ .vgprs = 208, .sgprs = 45, .lds = 4096, .private = 0, .bound = 128, .wave = 32 }, .{ .vgprs = 208, .sgprs = 46, .lds = 4096, .private = 0, .bound = 128, .wave = 32 }, }
    RECORD exl3 tf_exl3_linear_k8_w8: .usage = .{ .{ .vgprs = 217, .sgprs = 43, .lds = 8192, .private = 0, .bound = 256, .wave = 32 }, .{ .vgprs = 216, .sgprs = 43, .lds = 8192, .private = 0, .bound = 256, .wave = 32 }, .{ .vgprs = 216, .sgprs = 43, .lds = 8192, .private = 0, .bound = 256, .wave = 32 }, .{ .vgprs = 216, .sgprs = 44, .lds = 8192, .private = 0, .bound = 256, .wave = 32 }, }
    RECORD exl3 tf_exl3_linear_k10_w4: .usage = .{ .{ .vgprs = 213, .sgprs = 45, .lds = 5120, .private = 0, .bound = 128, .wave = 32 }, .{ .vgprs = 212, .sgprs = 45, .lds = 5120, .private = 0, .bound = 128, .wave = 32 }, .{ .vgprs = 212, .sgprs = 45, .lds = 5120, .private = 0, .bound = 128, .wave = 32 }, .{ .vgprs = 212, .sgprs = 46, .lds = 5120, .private = 0, .bound = 128, .wave = 32 }, }
    RECORD exl3 tf_exl3_linear_k10_w8: .usage = .{ .{ .vgprs = 221, .sgprs = 43, .lds = 10240, .private = 0, .bound = 256, .wave = 32 }, .{ .vgprs = 220, .sgprs = 43, .lds = 10240, .private = 0, .bound = 256, .wave = 32 }, .{ .vgprs = 220, .sgprs = 43, .lds = 10240, .private = 0, .bound = 256, .wave = 32 }, .{ .vgprs = 220, .sgprs = 44, .lds = 10240, .private = 0, .bound = 256, .wave = 32 }, }
    RECORD exl3 tf_exl3_linear_k12_w4: .usage = .{ .{ .vgprs = 214, .sgprs = 45, .lds = 6144, .private = 0, .bound = 128, .wave = 32 }, .{ .vgprs = 213, .sgprs = 45, .lds = 6144, .private = 0, .bound = 128, .wave = 32 }, .{ .vgprs = 213, .sgprs = 45, .lds = 6144, .private = 0, .bound = 128, .wave = 32 }, .{ .vgprs = 213, .sgprs = 46, .lds = 6144, .private = 0, .bound = 128, .wave = 32 }, }
    RECORD exl3 tf_exl3_linear_k12_w8: .usage = .{ .{ .vgprs = 222, .sgprs = 43, .lds = 12288, .private = 0, .bound = 256, .wave = 32 }, .{ .vgprs = 221, .sgprs = 43, .lds = 12288, .private = 0, .bound = 256, .wave = 32 }, .{ .vgprs = 221, .sgprs = 43, .lds = 12288, .private = 0, .bound = 256, .wave = 32 }, .{ .vgprs = 221, .sgprs = 44, .lds = 12288, .private = 0, .bound = 256, .wave = 32 }, }
    RECORD exl3 tf_exl3_linear_k16_w4: .usage = .{ .{ .vgprs = 219, .sgprs = 45, .lds = 8192, .private = 0, .bound = 128, .wave = 32 }, .{ .vgprs = 218, .sgprs = 45, .lds = 8192, .private = 0, .bound = 128, .wave = 32 }, .{ .vgprs = 218, .sgprs = 45, .lds = 8192, .private = 0, .bound = 128, .wave = 32 }, .{ .vgprs = 218, .sgprs = 46, .lds = 8192, .private = 0, .bound = 128, .wave = 32 }, }
    RECORD exl3 tf_exl3_linear_k16_w8: .usage = .{ .{ .vgprs = 227, .sgprs = 43, .lds = 16384, .private = 0, .bound = 256, .wave = 32 }, .{ .vgprs = 226, .sgprs = 43, .lds = 16384, .private = 0, .bound = 256, .wave = 32 }, .{ .vgprs = 226, .sgprs = 43, .lds = 16384, .private = 0, .bound = 256, .wave = 32 }, .{ .vgprs = 226, .sgprs = 44, .lds = 16384, .private = 0, .bound = 256, .wave = 32 }, }
    

• Chips I could not run: gfx1030, gfx1100 and gfx1201, whose records come from the compile.

Known limits

• Decode rows only. The prompt path follows, registered bit-equal to this linear.
• The linear takes 205 to 226 VGPRs on gfx1151, so a SIMD holds six or seven of its waves (1,536 VGPRs in steps of 24). This is the decode layout our fork serves, a tile decoded once for up to 16 rows. Tuning it is later work; the kernel test of a row's bits pins its sums.

Checklist

• The title and description match the head commit, including the receipt and the stack.
• New tests fail before the change, pass after it, and skip cleanly without their dependency. (Each new kernel test fails with its planted fault. The contract's two controls go red as planted.)
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.
• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off. (Nothing launches these kernels outside the tests yet.)
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (No served path changes.)
• No precision traded for speed. If the bits change, the description says which sums change and why. (New kernels, held to float64. The K sums' order is not CUDA's, as said above.)
• Decode and prompt processing are no slower than main. (No served path changes; every HIP build loads one more code object.)

[ashhart]

Thanks @Shoestring-dev. The launch geometry, trellis placement, stable row sums and FP32 barrier before the input's FP16 conversion hold up on review; the portable launch and record checks pass. Your gfx1151 runs cover the new operators and both controls. The full contract's existing tf_moe_combine_gate limit failure remains explicit, with no new failing entry. Ready for this nonserved kernel scope once CI is green. The later serving integration keeps the separate TP=2 token gate; no additional AMD run requested here.

• Synthena (Ash's Agent)