Repository navigation
Conversation
ashhart#1449's two kernels give tf_glm_scores' and tf_glm_select_rows' bytes, so a prompt chunk takes them unless TF_GLM_INDEX_CUDA=0; scores still bisects. Shapes the kernels do not take keep the old launches either way: a recorded-launch test pins prompt chunks to the kernels and a decode row, an unaligned key cache and a small pool cap to tf_glm_scores.
0b269f0 to
ff492c2
Compare
|
Thanks @Bizuayeu, the measurements here are thorough. The change itself is small and careful:
Because this changes the default prompt path on GLM, we'll run it on our own GB10s at TP=2 before it lands. That run is main against this branch with the variable unset: three alternated launches each with The cell we'll look at most closely is TP=2 at 32,768 tokens, where your own table shows it 0.5% to 1.7% later. If it holds, we may ask for the default to apply only above a prompt size, which Optional:
|
…ine for the default With the kernels on by default, the warning a failed load prints reaches people who never set TF_GLM_INDEX_CUDA, so it starts glm:, as GLM's other start-up lines do, and says the fallback is what TF_GLM_INDEX_CUDA=0 asks for. The changelog line records the new default: unset takes the kernels, 0 keeps tf_glm_scores.
Both are in If the 32,768-token delay holds in that run, a floor on the prompt size in |
On three GB10s at TP=3 and two at TP=2 (
Bizuayeu/GLM-5.3-Flash-XSFP4), measured before the rebase onto main, main against this commit with the switch unset: every token hash equal, the cold first token 0.6% to 13.7% sooner at TP=3 from 131,072 to 2,000,000 tokens and 6.7% sooner at TP=2 at 1,000,000 tokens, 0.6% to 1.7% later at TP=2 at 32,768 tokens with the prompt cache off (below), decode moves at TP=2 of up to 2.8% either way on requests that launch the same kernels with the switch at0or unset (the Receipt has the four rounds), and the same peak memory on every rank. Both arms ran with the GPU clock capped at 2,000 MHz on every rank, so the absolute times are not 2,200 MHz ones (at 2,200 MHz a 2,000,000-token prefill slowed one rank thermally).Each figure is the mean of an arm's two cold samples (greedy and seeded), one launch an arm. The TP=2 cells at 4,096, 32,768 and 131,072 tokens ran again in four more launches (main, this, this, main). This commit's first token against main's, in each adjacent pair: −0.3% and +5.3% at 4,096 (the +5.3% is one 3.37 s greedy sample; the other seven took 3.025 to 3.044 s), +0.7% and +0.6% at 32,768, −1.4% and −2.3% at 131,072; all four launches against all four, +2.5%, +0.6% and −1.8%. The decode table is in the receipt.
At 32,768 tokens on TP=2 this commit came out later in all three pairs with the prompt cache off: +1.7%, +0.7% and +0.6%, that is 0.41, 0.15 and 0.14 s. For agreed receipt 2 that is one comparison past its spread: in the alternated four (main, this, this, main) the eight samples overlap (main 23.33 to 23.73 s, this commit 23.56 to 23.71 s; +0.6% four against four), but the first adjacent pair's +0.7% is past that pair's own spread (each launch's two samples about 0.01 s apart). At serving defaults the four speed-group rounds moved −0.9%, −0.6%, +1.2% and −0.8% (
bench_cache.py's cold first row; the last two are this commit's binary with the switch at0against unset).To see whether the kernels account for the delay, four more launches on this commit's binary (
0, unset, unset,0) ran withTF_GLM_PROF=1at the 2,200 MHz cap. It times each prompt chunk's sections on the GPU, and it also turns the paired and solo prompt steps off and synchronizes every chunk, so these are not receipts, and they do not time the paired steps the receipts ran on. At 32,768 tokens the indexer's scoring and per-block selection took 0.138 to 0.146 s a request on each rank at0and 0.117 to 0.123 s unset (scoring 28 to 31 ms less, the kernels' selection 6 to 7 ms more thantf_glm_select_rows), over 15 of the 16 request and rank sums (one lost a stage line to an interleaved log line; the lost span was not an indexer one). The first token there was 23.415 and 23.291 s at0against 23.437 and 23.048 s unset (greedy; seeded 23.378 and 23.311 against 23.448 and 23.051), with up to 0.40 s between two launches at the same setting. Their token hashes equal the re-measure's.The paired steps were then timed as the receipts ran them (the paired prompt steps on, as by default, no profiling, the prompt cache off): twelve launches at TP=2 at the 2,200 MHz cap, so their seconds are not the table's, which ran at 2,000 MHz. Main's binary, this commit's binary with the switch at
0, and this commit unset, four launches each, in the order main,0, unset, unset,0, main, main,0, unset, unset,0, main (the last was cut by the end of the machines' window and ran again whole), each launch sending a 32,768-token prompt four times cold (greedy and seeded, twice each). Each launch's median first token: main 22.316, 22.867, 22.693 and 22.499 s;022.375, 22.806, 22.783 and 22.497 s; unset 22.481, 22.425, 22.440 and 22.485 s. No kind's four launches were all later than another kind's four, main against this commit included. The fifth to eighth launches (0, main, main,0) were each later than all eight others, and this commit unset had none of them. The 0.14 to 0.41 s above is not explained. Every token hash was equal across the twelve launches and equal to the re-measure's.What this changes
A prompt chunk's DSA indexer now runs on #1449's two kernels (
tf_glm_index_scoresandtf_glm_index_pools) unlessTF_GLM_INDEX_CUDA=0; until now it took them only withTF_GLM_INDEX_CUDA=1. They givetf_glm_scores' andtf_glm_select_rows' bytes, so no token changes.Issue: Refs glm cuda: a prompt chunk's DSA indexer in two CUDA kernels, byte-equal to tf_glm_scores and tf_glm_select_rows (TF_GLM_INDEX_CUDA), 1M-token first token 1,223 -> 1,033 s #1449 (the kernels, opt-in) and Proposal: GLM-5.3-Flash on CUDA past its 1,048,576-token window, with D-LSA (Flat) as an opt-in indexer for long prompts #2663, where the four receipts below were agreed (Proposal: GLM-5.3-Flash on CUDA past its 1,048,576-token window, with D-LSA (Flat) as an opt-in indexer for long prompts #2663 (comment)).
Base: main
ab78aef90, which has glm cuda: TF_GLM_CONTEXT_CEILING serves windows past 1,048,576 tokens, read at GLM's open #2765; three commits on it.2f2f4114echanges onlyzig/src/families/glm/cuda_index.zig, andff492c280folds that file's new doc comment onto one line for the lean checker and changes nothing else. They are17cb85a80and0b269f036, which stacked on glm cuda: TF_GLM_CONTEXT_CEILING serves windows past 1,048,576 tokens, read at GLM's open #2765's headf6d2a6d5e, cherry-picked onto main with the same patches (git patch-id); the receipts are17cb85a80's binary, from before the rebase.92f2645ecchanges only the text of the warning a load logs when the kernels are not available (zig/src/families/glm/cuda_forward.zig) and adds the CHANGELOG line, so the receipts were not taken again. The 2,000,000-token cells need glm cuda: TF_GLM_CONTEXT_CEILING serves windows past 1,048,576 tokens, read at GLM's open #2765's window past 1,048,576 tokens.atc overlap --json(atc 0.9.4) did not complete: it stopped at its time and pull-request limits and could not read some pull requests' files, so the conflicts here aregit merge-tree's. Against main (d6e982068), onlyCHANGELOG.mdconflicts: this head's line is at the head of Unreleased, where main has added lines sinceab78aef90. It is not rebased; as on kv pool: a named TF_KV_SHARE's slots come where an asked cache goes #3125, the CHANGELOG conflict is left for landing. Against the heads of the 355 other open pull requests, with this head and with mainab78aef90: 101 conflict with both, on the same files; 119 more conflict with this head, each inCHANGELOG.mdalone; none inglm/cuda_forward.zigorglm/cuda_index.zig.TF_GLM_INDEX_CUDA: unset, blank or1take both kernels;scorestakes the scores kernel withtf_glm_select_rows' pools, to bisect;0keepstf_glm_scoresandtf_glm_select_rows; any other value (on,off) still stops the load (BadIndexMode).Launches the kernels do not take (
Kernels.takes: fewer than 64 rows, fewer than 512 pools, or a key cache layout off 16 bytes) keeptf_glm_scoreswhatever the switch says, so a decode row never reaches them. A load whose kernels are not available logsglm: the indexer kernels are not used (…); prompt chunks use tf_glm_scores, as with TF_GLM_INDEX_CUDA=0and serves as today.A host test records each launched function through a driver without a device and pins that: in both modes, a 512-row prompt chunk at 65,536 tokens takes the kernels, while a decode row (the wide select), a key cache 8 bytes off alignment and a 256-pool cap take
tf_glm_scores.Checks on this head
92f2645ecpython3 tools/zig/lean_check.py, on mainab78aef90and on this head, as CI runs it: 341 findings against 341, 0 new (tools/pr_check.py: 0 errors, 0 warnings).cuda_index.zigis 181 lines.zig build test-glm-index-host --summary all: 11/11 tests, 4/4 steps, on this head, as on0b269f036and17cb85a80before the rebase17cb85a80: with the default back to off,GLM index: TF_GLM_INDEX_CUDA is on unless 0, scores bisects, and a typo is refusedfails (expected .both, found .off, 10/11). On17cb85a80, withKernels.takesletting a 1-row launch through, the recorded-launch test fails on the decode row (tf_glm_index_scoresandtf_glm_index_poolslaunched wheretf_glm_scoresand the wide select were expected), and so does thetakestest (9/11). The recorded-launch test pins main's own rule, so it also passes on main.zig build compile-config-server: 7/7, on this head and on17cb85a80. It compiles the new warning: with the warning's arguments emptied it fails on this head (too few argumentsandunused capture, 5/7).zig fmt --check build.zig zigandzig ast-checkoncuda_index.zigandcuda_forward.zig: clean on this head.-q -rs), on this head: 450 passed, 86 skipped, 519 subtests passed, with the same skips as onff492c280, where CI had 452 passed and 84 skipped. The skips here are the optional web packages (50), NumPy (30), the fake-serve fixtures (3), a CLI test that wants a native build (1), and the two NFKD table tests, which skip because this Python 3.12 has Unicode 15.0.0 and the tables pin 16.0.0.CHANGELOG.md(above). Onff492c280, the head before it, seven checks passed (pytest: 452 passed, 84 skipped, as on mainab78aef90); host-nightly-linux and host-nightly-macos skipped (no GPU); zig-macos-dev-tools waited for a macOS runner and was cancelled with the run before it started, as on mainab78aef90.tf-cuda-test glm-index-check) and the kernel contract, since this changes neither the kernels nor their records; TP=4; ROCm and Metal, which have no GLM indexer kernels.Receipt
Main against branch: main is glm cuda: TF_GLM_CONTEXT_CEILING serves windows past 1,048,576 tokens, read at GLM's open #2765's head
f6d2a6d5e, which this stacked on before the rebase, with the switch unset; the branch is17cb85a80, the same patch as2f2f4114e. Each was built fromgit archiveof its tree with an empty Zig global cache (binaries442a6e956b0b1565…and91be846b4c351d62…by sha256), with the same launch settings and binaries on every rank.Boxes it reaches: GLM-5.3-Flash on CUDA, every prompt chunk of 64 rows or more over at least 512 pools; decode rows are not reached.
New switch? No.
TF_GLM_INDEX_CUDA's default moves from off to both kernels; the accepted values are above. It has no profile row:cuda_index.zigreads it at load, as since glm cuda: a prompt chunk's DSA indexer in two CUDA kernels, byte-equal to tf_glm_scores and tf_glm_select_rows (TF_GLM_INDEX_CUDA), 1M-token first token 1,223 -> 1,033 s #1449, and it is not in the shared switch registry.Environment:
f6d2a6d5eand17cb85a80, Zig 0.17.0, nvcc 13.3.73 (-Dsm=121), three GB10s (sm_121) over RoCE for TP=3 and two for TP=2,Bizuayeu/GLM-5.3-Flash-XSFP4at88736f916f8939607fde8a4b881dcae749f7551d, MTP drafts, the GPU clock capped at 2,000 MHz on every rank for both arms.Paths and platforms the change reaches, which I ran and which I could not: CUDA GLM prompt chunks, run on sm_121 at TP=3 and TP=2. Not run: the other SMs the kernels' records cover since glm cuda: a prompt chunk's DSA indexer in two CUDA kernels, byte-equal to tf_glm_scores and tf_glm_select_rows (TF_GLM_INDEX_CUDA), 1M-token first token 1,223 -> 1,033 s #1449, TP=4.
Exactness:
Lcells): each arm on a fresh server,/v1/completionswith the prompt's token IDs, 64 tokens, greedy and seeded (temperature 1.0, seed 1234), prompt cache off, cold: everytoken_sha256equal, main against this commit, at TP=3 at 131,072, 524,288, 1,000,000 and 2,000,000 tokens, and at TP=2 at 4,096, 32,768, 131,072 and 1,000,000, and again at 4,096, 32,768 and 131,072 in the re-measure's four launches.bench_openai.py's reps'token_shafor two prompts at temperatures 0 and 1,bench_cache.py's first, repeated and concurrent rows at 4,096 and 32,768 tokens, and at TP=3bench_concurrent.py --alone --serialat 1, 4 and 8 streams, equal in all four launches. The TP=2 group ran three more times: the same four launches again, then twice four on this commit's binary alone with the switch at0and unset (0, unset, unset,0, and unset,0,0, unset); its hashes are equal across all sixteen launches.Main against this branch, same machine and checkpoint, posted before asking for a run on our machines: the tables here.
Decode speed before and after, from
tools/bench_openai.py(each launch's median over its reps; main's two launches against this commit's two; the change is the means'):At TP=3,
tools/bench_concurrent.py --alone --serial's aggregate tok/s at 1, 4 and 8 streams moved +0.3%, +0.2% and +0.3% (chat) and +0.7%, +0.5% and +0.3% (code). Each group's four launches ran the same expert form and window prices: GLM races its two expert forms at a build's first load and keeps the result, and the two builds picked differently at TP=3 (main classic, this factored), so before each group main's kept record was replaced by this commit's (factored at both TPs; a one-row window 29.66 ms at TP=3 and 39.49 ms at TP=2). The TP=2 pair's two builds both raced to factored.At TP=2 the decode rows ran in four rounds of four launches: main against this commit, the same again, then this commit's binary alone with the switch at
0against unset (0, unset, unset,0), and the mirror (unset,0,0, unset). The change, this commit's or unset's two launches against the other two:0→ unsetThese requests launch the same kernels with the switch at
0or unset.bench_openai.py's prompts are 14 and 32 tokens and each decodes 256, so no row gets to the 2,051 positions (dense_limit, the key width) before which a DSA layer attends densely and runs no indexer, with the switch at0or unset; past them a decode window is at most 32 rows, underKernels.takes' 64. After load, the one difference is that the kernels' module is loaded. The same two-launch rule, applied between the two sessions at the same setting, flags 5 of the 40 rows, among them −2.8% on fibonacci-raw at 0 (0against0) and −2.8% on gpu-chat-no-think at 1 (unset against unset).Prompt speed before and after: the first-token table above (the reply's own first-token time, two cold samples an arm), and
tools/bench_cache.py --require-cold's cold first row in the speed groups: TP=3 at 4,096 tokens 2.59 and 2.59 s against 2.59 and 2.59 s, at 32,768 18.54 and 18.53 s against 18.48 and 18.49 s (−0.3%); TP=2 at 4,096 3.13 and 3.38 s against 3.14 and 3.14 s, at 32,768 23.72 and 23.72 s against 23.51 and 23.48 s (−0.9%).Memory: every rank's
TF_MEMORYload_end peak and host peak equal, main against this commit, in every pair and group: TP=3 long prompts 64.61, 64.05 and 58.91 GiB, TP=3 serving defaults 64.01, 63.45 and 58.31 GiB, TP=2 91.26 and 90.83 GiB, host peaks 0.16 to 0.18 GiB. The pool after the reserve: 4,097 pages on both at TP=3 (36.7 GiB left), 2,482 and 2,484 at TP=2 (16.6 GiB), and 4,146 to 4,159 and 2,053 to 2,070 across the TP=3 and TP=2 speed groups' launches, each start sizing it from the memory free at that moment.Reserve: the TP=3 speed group ran both arms at
TENSORFOLD_MEMORY_RESERVE_GIB=20. At our image's 16 for that layout the pool took the rest of the budget and left 16.4 to 16.7 GiB free on rank 0 after load, under our harness's 19 GiB floor for a TP=3 run; at 20 it left 20.5 to 20.6 GiB. The reserve moves only the pool's size, and main and this commit ran at the same value.The fallback: the recorded-launch host test above.
Licence of any new weights or drafter: n/a.
Checklist
92f2645econ mainab78aef90: the first two commits are the same patches as0b269f036on glm cuda: TF_GLM_CONTEXT_CEILING serves windows past 1,048,576 tokens, read at GLM's open #2765'sf6d2a6d5e, and the receipts are17cb85a80's binary, from before the rebase; the second commit changes a comment only, and the third only the fallback warning's text and the CHANGELOG)Bizuayeu/GLM-5.3-Flash-XSFP4; every commit's author and committer are the noreply address)"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (in every TP=3 speed launch,bench_concurrent.py --alone --serial: every concurrent stream equals its alone run, 78, and every alone run equals its"draft": falserun, 6, none unequal or failed; in every speed launch at both TPs,bench_cache.py's repeated and concurrent rows equal its cold first row; the TP=2 group ran nobench_concurrent.py)0or unset; fibonacci-raw at 0 was lower in all four rounds, and moves of that size show between launches at the same setting; prompt: sooner at TP=3 from 131,072 tokens and at TP=2 at 1,000,000; at 32,768 on TP=2 later in the three receipt pairs (0.6% to 1.7%), and twelve launches on the paired path did not separate main,0and unset)