Repository navigation
dory cuda: decode rows run q, k, v and g as one grouped NVFP4 launch, and gate and up as another - #3131
Open
ashhart wants to merge 6 commits into
Open
dory cuda: decode rows run q, k, v and g as one grouped NVFP4 launch, and gate and up as another#3131ashhart wants to merge 6 commits into
ashhart wants to merge 6 commits into
Conversation
Owner
Author
|
GB10 results on this head:
We're on both, then the served Dory A/B.
|
… and gate and up as another
…21 records for the 16-row tiles The two-part FP16 stress cases cleared only 10.7x and 96x of the 100x margin the gate asks for: the shared stress inputs are now 2,048 BF16 values of 512, mirrored by 512 with the first eight raised to 516, over seven integer weight classes, so every term and FP32 partial sum is exact in any order and the FP16 path's error is the only difference. A host regression checks the fixture against Dense's FP32, FP16 and BF16 simulators and the spec's exact-sum bound. The records take the measured sm_121 resources of tf_qmmf_many16 and tf_qmmf_many16c (128 threads, 56 registers, 17,408 dynamic bytes); the 32- and 64-row tiles' records stay to be measured on the box.
tf-cuda-test measured them on a GB10: 128 threads with 80 registers for the 32-row tile (65 clustered) and 88 for the 64-row tile, no static shared bytes. The other release SMs stay pending until their fatbins are checked.
…se SMs (full kernel contract on GB10)
ashhart
force-pushed
the
lead/dory-nvfp4-grouped
branch
from
November 14, 2026 03:31
e136918 to
900965d
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3118. The decode path that issue describes (448 projection launches a decode step) is Dory's; main has no Kimi K3 CUDA model, so this lands for Dory.
Dory's NVFP4 decode ran q, k, v and g as four launches and gate and up as two, though each group shares its input. Decode rows now send q, k, v and g as one grouped launch (split-K 4) and gate and up as another (split-K 2); prompt rows are unchanged. Each part keeps its own tile map and its own sums in the same order, so its bytes equal a separate launch's.
Tests:
test-qmmf-many-host26/26 (fails with a planted off-by-one, and with a fallback that gave every part the first weight's width);test-dory-host42 pass, 3 skipped;test-kernel-contractpasses; the full host run passes 2964 tests.compile-cuda-testspasses for both Linux targets. The host tile-map test checkslocate(), a line-for-line mirror of the kernel's loop. Every CUDA lane family now looks up the sixtf_qmmf_many*symbols at load, so the fatbin must be rebuilt.Device check pending (GB10):
tf-cuda-test qmmf-many-contractat 1, 2, 8, 16, 64 and 128 rows (every byte equal to separate launches, 3 controls red),tf-cuda-test kernel-contract qmmfwith no new failures,kernel-contract-limitsrecording register counts inqmmf_many_records.zig, and a served Dory A/B, one stream and several, with token hashes equal at T=0 and sampled with the same seed.