Skip to content

spark25 cuda: gate and up load as one stacked weight, and every CUDA chip decodes them in one launch - #3130

Open
ashhart wants to merge 2 commits into
mainfrom
lead/spark25-gate-up-stack
Open

ashhart wants to merge 2 commits into
mainfrom
lead/spark25-gate-up-stack

Conversation

@ashhart

@ashhart ashhart commented Nov 13, 2026

Copy link
Copy Markdown
Owner

Spark 2.5 on CUDA loaded gate and up as two weights and decoded them with two launches. They now load as one stacked weight and every CUDA chip decodes both in one launch. The bits stay the same: both members keep n=10240, so split-K stays 2; codes are addressed independently of npad, and scales and biases use npad as their stride; the seam is 128-aligned, so no tile crosses into the other member. On GB10 the same grouped kernels run. On other chips the segmented kernel replaces the split kernel at 16 rows or fewer, with the same gemv_split<64,64,4> body (split-K 2) and its own work and ticket areas; at 17-32 rows each member keeps its own launch. Only 4-bit checkpoints take this path.

Tests: test-spark25-host 13, test-cuda-family-groups-host 5 and test-qwen35-verify-host 8 pass; the new test records main's per-member launches and fails on main ("expected 1, found 0"). compile-cuda-tests passes for aarch64 and x86_64 Linux.

Device check pending (needs a Spark 2.5 MLX 4-bit checkpoint): on a GB10 and a non-GB10 sm_120 card, a served A/B against main with token hashes equal at T=0 and sampled with the same seed, over 1, 2, 8 and 16-row decode, 17-32-row verify windows on sm_120, the prompt, and solo versus concurrent streams; decode tok/s reported.

@ashhart
ashhart force-pushed the lead/spark25-gate-up-stack branch from 84bab90 to a2256df Compare November 14, 2026 03:51

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant