[BadAd84]

What's slow today

A decode window's routed experts are grouped by tf_exl3_group (a 1024-thread block that scans every expert against every pair) and launched over min(pairs, E) x (n/128) x (mats*sk*ceil(rows/16)) blocks. At the widths a shared verify window reaches, most blocks have nothing to do: at 64 rows and the wide 768-column share that is 55,296 blocks, about 800 of them with work. The group kernel costs 0.054 ms a layer at 24 rows and 0.134 at 64, against 0.005-0.010 for a ranked plan.

Measured on one GB10 with both engines' kernels verbatim over one synthetic EXL3 weight set, per layer at the wide share, every row routed alike (a batching workload):

rows	this tree	a plan-based launch	this change
8	0.206 ms	0.169	0.193
32	0.437	0.247	0.300
64	0.932	0.382	0.506

44 distinct experts a window: 1.033 -> 0.958 ms at 24 rows and 1.221 -> 1.058 at 48 (a plan-based launch: 0.915 / 0.990).

What the change is

The prompt path already groups pairs into items of at most 16 through grouped.Router, and the grouped kernels already read item triples in their standard mode (exl3_grouped.cuh: standard, skip, direct pair indices). Decode only has to use them, so this is plumbing, not a kernel change:

• exl3_experts.runWith(..., ey, it: ?Items) runs the same single implementation for both paths; items = it != null is the only switch, and run is a one-line delegation to it with null. The plain branch inside is unchanged line for line.
• the family hands decode exactly what its block already holds - the Router the prompt path uses and the same plan buffer - so decodeItems runs for every non-prompt window.
• the decode arena is sized for the wider 9-slot layout (decodeBytes, the larger of the two), so no pool changes at GLM's shapes.
• a supplied plan is refused with error.TileNotBuilt for a non-mcg codebook, a non-zero wts/out, a zero ey, or a null plan array: the item kernels read one codebook a launch and write the caller's slot rows.

Nothing under zig/kernels/ changes.

Proof it's the same bits

tf-cuda-test glm-exl3-items compares the item decode's packed ey slot rows against the plain launch's own packed rows, over distinct windows (1, 2, 3, 8, 17 rows) and lockstep windows (16, 32, 48, 64 rows) at two widths, and again at GLM-5.3-Flash's served size (dims 4096 at widths 768 and 1024), with a plain-twice control and a NaN check: 0 of 66 checks differ on our tree, where a mutant that reads the plan's members one slot off fails exactly the item comparisons and none of the other cases.

What it's worth

Our fork's serve, TP=3 on three GB10s, production's probe, ABBA, every reply digest equal, the pool at boot equal:

streams	1	2	4	6	8
short decode	+2.4%	+1.7%	+3.9%	+5.2%	+6.5%
forks	+1.5%	+1.9%	+3.6%	+1.8%	+1.8%

One-stream prose +1.2%, cold and next-turn prompt cells inside their floors.

Same cells on main's own base (673a34b3b), both arms built from this repo's scripts/engine/build-image.sh on the same 3x GB10s, drafts on, fresh boot per stream count:

streams	1	2	4	6	8
main 673a34b3b	51.79	58.51	72.08	95.53	114.23
this PR	52.41	58.61	72.15	96.22	116.68
delta	+1.2%	+0.2%	+0.1%	+0.7%	+2.1%

Reply digests are equal at all levels, so the tokens match main's at all stream counts.

Files

• zig/src/cuda/exl3_experts.zig (+76/-29): Items, runWith, the layout split, the standard-mode launches, down_combine on ey.
• zig/src/families/glm/cuda_exl3.zig (+32/-2): decodeItems called for every non-prompt window, decodeBytes.
• zig/tests/cuda/glm_exl3_items_tests.zig (new, 221) and main.zig (+3): the check.
• zig/exl3_prompt_host.zig (+12/-11) and zig/glm_exl3_prompt_host.zig (+4/-3): the two host tests that pinned the old split-K decode now pin the item plan.
• CHANGELOG.md (+2): the Unreleased bullet.

Checks

fmt, check-native-cuda-host, compile-cuda-tests, test-cuda-host, every sm_121a fatbin through nvcc, and lean reports 0 new against main. glm-exl3-items runs on any GB10.

Overlap check, 2026-11-13 08:46 UTC: atc overlap --json on 702bb5d64, base 0fed2e4a5.

• CUDA (GB10): a second carveout source, spark-attic's atticd, opt-in #2762 shares only CHANGELOG.md with this change, the ordinary case.
• qwen3-asr: host slice, the shared tower and Qwen3 schedules over a backend binding (#2105) #2239 shares CHANGELOG.md and zig/tests/cuda/main.zig, and atc reports one conflicting file.
• MiMo-V2.6-Flash on CUDA, on two GB10s with its DFlash drafter #728 (MiMo) shares zig/src/families/glm/cuda_exl3.zig and zig/exl3_prompt_host.zig. It changes one line in each, the k.prompt.run call in the first and the k.prompt kernel table in the second. Neither is a line this change touches, so whichever lands second rebases a one-line difference.

The run was incomplete (rate and time limits).

[ashhart]

Thanks @BadAd84, this is a neat change: decode now reuses the item plan and grouped kernels the prompt path already
has, with no kernel touched.

I read it on current main. It applies cleanly, and the lean check is unchanged.

• run keeps main's launches exactly, so the other EXL3 callers are untouched.
• On the item path, the shared slot's sentinel is skipped by the router and by every kernel, so its ey row is never
  written.
• Each routed pair is rotated, summed and scaled exactly as before, and lands where the old copy put it.
• The arena and the plan buffer already cover the 16-pair plan.

Your 0-of-3,984 test is the right check.

Two small things:

1. Could the GPU test add one case at GLM's served size (dims 4,096, width 768 or 1,024)?
2. Please post the serve run on main's base when it finishes.

We don't have a GLM EXL3 checkpoint on our GB10s yet, so your serve run against main's base is the device receipt for now: equal token hashes and decode speed at 1 to 8 streams with
drafts on.

[ashhart]

Thanks @BadAd84 for adding the served-size case so quickly. Dims 4,096 at widths 768 and 1,024 is the shape a rank actually runs, so that ask is done.

Three small things before it lands:

1. The new 3-line comment at the top of the shape loop (lines 174-176) adds two lean problems. One line keeps the check at its current count.
2. lock = rows >= 16 turns the 17-row case into a lockstep window. Could you keep it distinct? It was the only distinct window spanning two tiles.
3. Please post the serve run on main's base when it finishes, and paste the RESULT glm-exl3-items line from the GB10. The test counts the NaN checks too, so it should print 66 comparisons, not 44 or 3,984.

The engine code is unchanged from the version we read, and we're happy with it.

[BadAd84]

Thanks, both are fixed.

1. The comment is one line now. I had already pushed that as 98317c364 just before your review landed, so the three-line block you saw is gone in the current head.
2. You are right about the 17-row window, and that was my bug: lock = rows >= 16 made it lockstep. Both shapes now carry an explicit case list instead of deriving the flag from the row count, so 17 rows is distinct again, and the served-size shape keeps 8 rows distinct and 64 lockstep. That is 4cda259bc.
3. The gates on that head: lean_new 424 against 424, 0 new; fmt clean; compile-cuda-tests exits 0.
4. Your arithmetic on the count is right: the RESULT line includes the NaN tally, so this set prints 66, which is 18 base cases and 4 served cases at three checks each. I will paste the actual line as soon as the GB10 run finishes, with the serve run against main's base in the same message.

[ashhart]

Thanks @BadAd84, that's all three test points handled: the one-line comment brings lean back to its count, 17 rows is
a distinct window again, and the case lists make the 66 easy to read off.

Two text fixes when you next push: the PR body's "0 of 3,984" should read 0 of 66, and commit 2's "36 to 44" is
now 66 (squashing commits 2-4 would tidy that). You can also drop the "reviewer" notes from the two test comments.

Then it's the serve run on main's base, plus the RESULT glm-exl3-items line from a GB10, and we can land it.

[BadAd84]

Squashed commits 2-4 into one, with the count corrected to 66. The only content change is two comment lines that drop the reviewer notes.

[ashhart]

Thanks @BadAd84, the squash is clean: the code is exactly what we reviewed, and only the two comment lines changed. Commit 2's message now gives the 66 and how it adds up.

One text fix left: the body's "Proof it's the same bits" section still says 0 of 3,984; it should say 0 of 66.

After that, we only need the serve run on main's base and the RESULT glm-exl3-items line from a GB10, and this can land.

[BadAd84]

RESULT glm-exl3-items: 0 of 66 comparisons differ
PASS glm-exl3-items: 66 comparisons equal the plain decode byte for byte

ai-engine:items2 built from 06fdcc8bd (main f23603801 plus this PR), single GB10, no checkpoint needed. The serve run against main's base is still filling its cells; its table follows.

[ashhart]

Thanks @BadAd84. The body now says 0 of 66, and the RESULT glm-exl3-items line is exactly what we wanted. The code is what we reviewed. The last thing is the serve run against main's base: equal token hashes, and decode at 1 to 8 streams with drafts on. Once that table is posted, this lands.

[ashhart]

@BadAd84, a side note that doesn't affect this PR. We read the notes on your ovn-solo and ovn-fair branches and measured both on today's main. GLM-5.3-Flash NVFP4, two GB10s at TP=2, main 86abaf097.

The prompt stall reproduces. --parallel 2, tools/bench_fill.py with a 49,152-token prompt beside a 1,024-token reply:

	Alone	Reply started first	Prompt started first
Reply done	31.9 s	68.3 s	65.7 s
Prompt's first token	32.8 s	39.7 s	39.2 s
Gap between reply pieces, 95th percentile		1.43 s	1.44 s
Largest gap		2.97 s	2.87 s

The 95th-percentile gap is about one 2,048-row prompt step on this pair, so a smaller step while another stream decodes looks like the right fix. Would you like to open ovn-solo as a PR on today's main? We'd want the prompt's tokens to match with and without another stream decoding, and main-against-branch numbers for the gaps and for prompt speed.

The fairness gap doesn't show here. --parallel 8, tools/bench_concurrent.py --mixed --temperatures 1, 512 tokens, three repeats:

• Streams ran at 9.2 to 11.1 tok/s, with fastest over slowest at 1.2.
• Effective streams, (Σx)² / Σx², came to 7.96 of 8, against the 5.6 in your note.
• All 24 replies equal the same request served alone.

That's a different setup from your EXL3 runs on one GB10, so it doesn't rule your number out. If you still see 5.6 on today's main, could you post the command and checkpoint? We'd like to look at it.

[ashhart]

Thanks @BadAd84 for the rebase; the code is unchanged from what we reviewed, and the item check still holds. The one thing left is the serve run against main's own base that the body mentions: token hashes equal to main, and decode at 1, 2, 4, 6 and 8 streams with drafts on, beside the same cells on main. Once that table is in the description, this is ready to land after the 1.2.3 cut.

[ashhart]

Hi @BadAd84, checking in on the last step here. It's the serve run against main's own base: token hashes equal to main, and decode at 1, 2, 4, 6 and 8 streams with drafts on, beside the same cells on main. Once that table is in the description, this is ready to land. Thanks again for the item-plan change; it's a neat one.

• Synthena (Ash's Agent)

[BadAd84]

Rebase only: onto main 673a34b3b, plus one test call updated for the glm.experts.Kernels.load signature change. No other content change.

[BadAd84]

Rebased onto today's main (673a34b3b): this branch is 2 commits (item plan, then the GPU check at served size), with one test call updated for the fourth argument of glm.experts.Kernels.load (block32, the same value main's own glm_exl3_tests.zig passes).

So, the serve run against main's own base. Both arms built on the same 3x GB10s using scripts/engine/build-image.sh, GLM-5.3-Flash EXL3 4bpw at rev 078455ffe, drafts enabled, fresh boot per stream count:

Streams	main 673a34b3b	this PR
1	51.79	52.41
2	58.51	58.61
4	72.08	72.15
6	95.53	96.22
8	114.23	116.68

Reply digests are equal at every level (3ec88b6418f7 / c36689783326), so the tokens match main at 1/2/4/6/8 streams and decode is equal to +0.1% to +2.1%.

[ashhart]

Thanks @BadAd84, that's the run we asked for: reply digests equal to main's at 1, 2, 4, 6 and 8 streams, and decode level to +2.1%. Could you copy that table into the PR description, so it lands with the change? We're running the GLM EXL3 kernel check on our GB10 on this head too. Main is frozen for the 1.2.7 release this afternoon, so it lands just after, once CI is green.

• Synthena (Ash's Agent)

[ashhart]

@BadAd84, we rebased this onto today's main to put it back in the queue, and found one thing only you can settle. Your own test exl3 prompt: decode retains the existing split-K launches (zig/exl3_prompt_host.zig:121, step test-exl3-prompt-host) fails on this PR, on its original head too. It expects the tf_exl3_group launch that this PR replaces with the router plan. The step runs only in our nightly, which is why CI stayed green.

1. Please update that test to pin the new decode launches (the item plan), so it fails if decode goes back to the old split-K path.
2. Heads-up for after GLM-5.3-Flash EXL3 on CUDA: a whole prompt pass routes its experts in one call #2852 lands: git merges your if (!prompt) decode block into GLM-5.3-Flash EXL3 on CUDA: a whole prompt pass routes its experts in one call #2852's new compact() without a conflict, but there prompt, ek and plan_buf are out of scope, so it won't compile. The decode loop belongs in run() after GLM-5.3-Flash EXL3 on CUDA: a whole prompt pass routes its experts in one call #2852's prompt branch, with GLM-5.3-Flash EXL3 on CUDA: a whole prompt pass routes its experts in one call #2852's prompt sizing and your decodeBytes in scratchBytes. If you'd rather we do that rebase once GLM-5.3-Flash EXL3 on CUDA: a whole prompt pass routes its experts in one call #2852 is in, say so.

• Synthena (Ash's Agent)

[BadAd84]

Both done, and main is merged in.

The test. exl3 prompt: decode retains the existing split-K launches now pins the item plan:

• 4 rows: plan, rotate, grouped gate and up, gate-up epilogue, grouped down, combine, in that order
• 1,030 rows: no group launch, no prompt kernels, no compact copies, one combine per 1,024-row chunk

GLM EXL3 prompt: decode retains compact 1024-row calls (zig/glm_exl3_prompt_host.zig, step test-glm-exl3-prompt-host) failed the same way, so its updated the same way. With the decode branch made dead, both fail.

After #2852. I merged main (0fed2e4a5) as a merge commit, no force-push. The decode loop is in run() after the prompt branch, decodeBytes is in scratchBytes, and compact() is just for prompt layers that arent eligible for the whole-pass kernel.

Gates (run on 55f498297; head 702bb5d64 differs only in the CHANGELOG line, and tools/pr_check.py was rerun on it): test-exl3-prompt-host 7 of 7, test-glm-exl3-prompt-host 5 of 5, check-native-cuda-host, compile-cuda-tests, test-cuda-host and zig fmt --check all pass. Lean is 341 vs 341, 0 new. tools/pr_check.py reports 0 errors and 0 warnings, with the CHANGELOG bullet as its own commit.

tf-cuda-test glm-exl3-items on a GB10, built from 55f498297, which differs from this head only in the CHANGELOG line: RESULT glm-exl3-items: 0 of 66 comparisons differ.

The serve table against mains base is still in the description at 673a34b3b. I have not rerun it on todays main.

[ashhart]

Thanks @BadAd84. The updated item-plan tests, their dead-branch controls, and the 0-of-66 GB10 result answer the two final asks. Thank you for keeping the older serve table’s source explicit too.

We still need to finish the current-main merge and verify the merged head’s two prompt-host suites, then complete our GLM EXL3 check on GB10. We’ll preserve the decode loop in run() and the separate prompt sizing. Nothing further is needed from you at this point.

• Synthena (Ash's Agent)

[ashhart]

As promised, we finished the merge with current main and pushed it to your branch (now a4a5d63): your three commits are unchanged apart from the base, and our one fix is on top. The decode loop stays in run(), and prompt sizing stays separate. The prompt-host suites pass on the merged head, with both dead-branch controls still failing as they should. What remains is our GLM EXL3 check on a GB10 pair.

• Synthena (Ash's Agent)