Repository navigation
GLM-5.3-Flash EXL3 on CUDA: a whole prompt pass routes its experts in one call - #2852
Merged
Merged
Conversation
Owner
Author
|
Device check on a GB10 (sm_121):
|
This was referenced Nov 12, 2026
Closed
Closed
Plan all nine slots with E+1 expert ids and skip the shared id before prompt payload reads. Write routed FP32 rows in place, without compact-pick or output copies, in calls up to 4096 rows. Decode keeps its 1024-row compact path. Preserve its prompt variant as a device-test reference and remove staged-output storage from normal prompt scratch. Add real-driver host recording and large byte/sentinel tests, retaining the independent FP64 checks. Mark the two changed kernel resource records pending until this source is measured. Refs #2819
Remove superseded resource tuples, name per-rank scratch widths and make the skip control retain a guard after a member load. Keep the compact prompt byte reference in the CUDA test tree; production retains the compact slot fallback. Refs #2819
ashhart
force-pushed
the
lead/glm-exl3-prompt-one-call
branch
from
November 12, 2026 23:47
8dceae8 to
ad8788c
Compare
BadAd84
added a commit
to BadAd84/TensorFold
that referenced
this pull request
Nov 13, 2026
…anch, tests pin the item plan Main's whole-prompt-pass change (ashhart#2852) moved the prompt path out of the loop in cuda_exl3.zig's run(), so the item plan's `if (!prompt)` decode block no longer merged as written. The decode loop now sits in run() after the prompt branch, decodeItems and decodeBytes are as before, scratchBytes sizes a decode chunk by decodeBytes, and compact() serves only the prompt layers that are not eligible for the whole-pass kernel. Two host tests expected the old split-K decode launches and failed here (and on the original head): exl3 prompt: decode retains the existing split-K launches (zig/exl3_prompt_host.zig, step test-exl3-prompt-host) and GLM EXL3 prompt: decode retains compact 1024-row calls (zig/glm_exl3_prompt_host.zig, step test-glm-exl3-prompt-host). They now pin the new launches: for 4 rows the plan, rotate, grouped gate and up, gate-up epilogue, grouped down and combine in that order; for 1,030 and 2,048 rows no group launch, no prompt kernels, no compact copies, and one plan and one combine per 1,024-row chunk. With the decode branch made dead, both fail.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GLM-5.3-Flash EXL3 on CUDA ran its fused prompt path in 1,024-row expert calls, a cap inherited from the decode path's shared-memory limit, so each call streamed and decoded every routed expert's trellis again and each sub-chunk added two 2D copies.
What changes:
tf_exl3_prompt_gu,tf_exl3_prompt_down) take a skip id and return before any load for it; an unused argument is dropped.experts + 1and writes straight into the caller's output slots.Exactness: the EXL3 prompt GEMM accumulates the whole K in one pass with no split-K, each member is its own column with its own epilogue, and member order is a deterministic rank, so a row's bits don't depend on how many rows share a call.
Checks: host tests that a 2,048-row fused pass makes one expert call and no 2D copies, plus a planted missing-skip control; shared prompt scratch is 52 MiB at 2,048 rows and 104 MiB at 4,096.
tf-cuda-test glm-exl3is extended to 2,054 and 4,102 rows in both rotation modes, comparing routed output word for word against the old 1,024-call loop, with a sentinel in every ninth-slot word. The EXL3 prompt and CUDA host suites,compile-cuda-tests -Dtarget=aarch64-linux-gnu, fmt and lean pass. Both changed prompt kernels have pending launch records, so the old ones can't be reused silently.Still to run before landing:
tf-cuda-test glm-exl3on a GB10 with fresh records (queued), then a TP=2 A/B on two GB10s with a GLM-5.3-Flash EXL3 4-bit checkpoint: cold prompts at 2K, 8K and 32K at the default and 4,096-row chunks, decode at 1 and 4 streams, replies equal. #2094 edits the same kernels and can rebase onto this.For #2819.