Repository navigation
Conversation
…re at one form GLM races its classic and factored NVFP4 expert forms when a build first loads and keeps the winner under a key that carries the build's hash. The race's window sums can land inside its 1% margin, so the same code built twice can serve different forms, and a comparison of two builds then carries the form along with the change it measures. TF_GLM_EXPERT_FORM=classic or factored names the form: rank 0 sets it on every rank before the warm-up, skips the race, and times the windows and the draft step at that form. A load that times nothing (no drafts, or too little scratch context) still sends the named form to every rank and logs `glm: expert form (untimed)`; unnamed, such a load stays on classic as before. The named form ends the cache identity, so its record and the race's never stand in for each other; unset or blank keeps the race and its key as they were. The startup line keeps its words and puts (TF_GLM_EXPERT_FORM) right after the form's word, which ends MTP's line. Any other text stops the load by name, as GLM's other rows do.
GLM's load-time race summed the 2-, 4- and 8-row windows and one draft step of its classic and factored NVFP4 expert forms, then ran every decode window on the winner. A sum hides the widths it loses: at TP=4 the factored form wins at 1 and 8 rows but loses at 4 and 6, so four streams without drafts decode about 2% slower than on classic. Both forms give the same bits (the NVFP4 fact contract holds the factored tiles and waves to the decode form's sums), so each window can take either. The race now times both forms at all ten widths, keeps classic at a width unless factored is more than 1% faster there, and prices each width at its kept form's time instead of timing the table again. A window runs on the form of the narrowest timed width that holds it: rank 0 sends the forms as a bit a row count, and the record keeps a bit a width. Records carry the build's hash in their key, so no record of a single-form build is read for its forms. The race sweeps every width five times on both forms, untimed, before it times a window, then each width's timed windows take the two forms in turn, classic first on even rounds and factored first on odd ones. Timed one form after the other, the order picked the form: on four GB10s at TP=4 a log of every run showed classic's narrow widths up to 60% over their later time in a load's first 4 s, and a 4 to 6% drift through the race on one form, so two launches of one build kept different forms at 3, 4, 6 and 12 rows. Five sweeps of both forms take about 5.4 s there. TF_GLM_EXPERT_FORM now names one form for every width: the warm-up sweeps that form ten times and the load times the windows and the draft step at it, and a load that times nothing sends it to every rank for every row count. The startup line names one form when every width shares it, else the more common form and the widths where the other runs (experts factored, classic at 4 and 6 rows).
8 tasks done
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
GLM's CUDA load now races its classic and factored NVFP4 expert forms, which give the same bits, at each of its ten timed window widths and keeps the faster form at each, where it kept one form for every width. This is not a complete fix: where factored's lead at a width sits near the race's 1% margin, two launches of one build can keep different forms there. On four GB10s at TP=4 this receipt's two racing launches did so at 12 and 32 rows (
factored, classic at 3, 4, 6 and 24 rowsandclassic, factored at 1, 2, 8 and 16 rows; across the two records, factored's time there is 1.1 and 1.2% under classic's), and two launches of a diagnostic build of the same race did at 8, 12, 16 and 32 rows. At 8 rows the two forms' decode rounds differ by about 2% in serving (the first table's 8-stream cell below), so a launch that keeps classic there gives that back.TF_GLM_EXPERT_FORM(#3151) names one form for every width when a comparison needs one.Main's race sums the 2-, 4- and 8-row windows and one draft step on each form and runs every decode window on the winner. A sum hides the widths it loses: on TP=4, #3151's receipt timed each form on its own, and against classic, factored's rounds are 4.82% shorter at one stream, 1.67% longer at four and 2.18% shorter at eight. This head times both forms at 1, 2, 3, 4, 6, 8, 12, 16, 24 and 32 rows, keeps classic at a width unless factored is more than 1% faster there, and prices each width at its kept form's time. A window runs on the form of the narrowest timed width that holds it: rank 0 sends the forms to every rank as a bit a row count, and the price record keeps a bit a width. The record's key carries the build's hash, so no single-form build's record is read for its forms.
The race sweeps every width five times on both forms, untimed, before it times a window; then each width's timed windows take the two forms in turn, classic first on even rounds and factored first on odd ones. Timed one form after the other, the order picked the form: on TP=4, a log of every run showed classic's narrow widths up to 60% over their later time in a load's first 4 s, and a 4 to 6% drift through the race on one form. From an empty price cache, this head's two racing loads loaded in 57.9 and 58.1 s, and #3151's head's first launch in #3151's receipt, on the same ring and racing as main does, in 56.7 s; loads that read a record loaded in 43.0 to 43.8 s here.
TF_GLM_EXPERT_FORMnow names one form for every width: the warm-up sweeps that form ten times, the load times the windows and the draft step at it, and a load that times nothing sends it to every rank for every row count. The startup line names one form when every width shares it, else the more common form and the widths where the other runs, as inglm: windows (timed): … experts factored, classic at 4 and 6 rows.On four GB10s at TP=4 (GLM-5.3-Flash XSFP4,
--context 1048576 --parallel 8), round time in ms from our own grid client ("draft": false, greedy, 256 tokens, each launch's median of six reps), lower is faster, two launches each:Here #3151's head raced to classic for every width (
glm: windows (timed): … experts classic), and this head's launches read the second race's record, factored at 1, 2, 8 and 16 rows. Taking a cell as different only where both launch pairs lean the same way by more than the larger of the two builds' launch-to-launch gaps, this head's rounds are 4.87% shorter at one stream and 2.27% shorter at eight, where the forms differ, and at four streams, where both keep classic, the two can't be separated (-0.33%). In #3151's receipt main and #3151's head raced to factored, and against that draw the forms differ at four streams instead, where that receipt timed factored's rounds 1.67% longer than classic's. So the grid ran again: #3151's head withTF_GLM_EXPERT_FORM=factored, which runs factored at every width, and two more launches of this head, which read the second race's record:factorednamedBy the same rule, this head's rounds are 2.70% shorter at four streams, where it keeps classic against factored: #3151's head's two launches there sit 1.5% apart, and every rep of this head's two is under every rep of theirs. At one and eight streams, where both run factored, the two can't be separated (-0.58% and +0.01%). Every launch's replies are the same.
TF_GLM_EXPERT_FORM(c628011007) and its CHANGELOG line (f935ab25cf), on mainab78aef904. This PR adds the change (e259495f26) and its CHANGELOG line (bc6349146d). Until glm cuda: TF_GLM_EXPERT_FORM names the expert form, so launches compare at one form #3151 lands, its two commits show here too; this PR's own diff is againstf935ab25cf. Both were rebased from main6c5cef8941, where glm cuda: TF_GLM_EXPERT_FORM names the expert form, so launches compare at one form #3151's head wasb92072b06d, this change4f9832ca00, and the receipt was taken; in the figures above and the receipt below, "this head" means that build. Beside context (glm cuda: TF_GLM_CONTEXT_CEILING serves windows past 1,048,576 tokens, read at GLM's open #2765'sTF_GLM_CONTEXT_CEILINGlines now sit above the row this change rewords),git range-diffshows no difference between the two changes. Main has since taken glm cuda: under --mtp-confidence a greedy stream drafts its rates' depth, not fifteen a round #2864, which changes how deep a temperature-0 stream drafts under--mtp-confidence(this receipt runs at 0.5), so on today's main a T=0 stream can draft to other depths and its windows take other widths. Merged with today's main4cfd955737,git merge-treeis clean, the CHANGELOG lines included.atc overlap --jsonagainst main4cfd955737, after the CHANGELOG line: its listing shows twenty-six open pull requests sharing a file with this head, sixteen of them byCHANGELOG.mdalone, and marks qwen3-asr: host slice, the shared tower and Qwen3 schedules over a backend binding (#2105) #2239, qwen27: serve the 27B pair, refuse what a pair cannot keep, and die with the peer #1434 and GLM on CUDA: the latent split's per-rank page book and owner (TF_GLM_KV_SPLIT, #1431 part 1; stacked on #1411) #2415 as conflicting.git merge-treeof main, and of main merged with this head, against each of them: eighteen already conflict with main, and seventeen of those conflict in the same files with as many conflict hunks with this head. The eighteenth, GLM on CUDA: the latent split's per-rank page book and owner (TF_GLM_KV_SPLIT, #1431 part 1; stacked on #1411) #2415 (TF_GLM_KV_SPLIT), conflicts with main in five files, with as many hunks with this head, and gains two:families/glm/cuda_follow.zig, where the order's field comments take this head's note on whatacarries for the forms and GLM on CUDA: the latent split's per-rank page book and owner (TF_GLM_KV_SPLIT, #1431 part 1; stacked on #1411) #2415's on whatdepthandflagscarry for its split, andfamilies/glm/cuda_lane_costs.zig, which glm cuda: TF_GLM_EXPERT_FORM names the expert form, so launches compare at one form #3151 brings, where the price cache's key takes GLM on CUDA: the latent split's per-rank page book and owner (TF_GLM_KV_SPLIT, #1431 part 1; stacked on #1411) #2415'ssplitfield and glm cuda: TF_GLM_EXPERT_FORM names the expert form, so launches compare at one form #3151's named tail, and the key test GLM on CUDA: the latent split's per-rank page book and owner (TF_GLM_KV_SPLIT, #1431 part 1; stacked on #1411) #2415's twelfth knob and glm cuda: TF_GLM_EXPERT_FORM names the expert form, so launches compare at one form #3151's name argument. Each resolves by keeping both sides, so whichever lands second takes the other's lines. The other eight merge cleanly with both. Of those the listing leaves out, glm cuda: TF_GLM_EXPERT_FORM names the expert form, so launches compare at one form #3151, which this PR stacks on, glm cuda: the L2 prefetch's ffn-head site also reads a MoE layer's shared gate-up, after its router #3156, docs: move README sections into docs/, short human README #3170 and lanes: shared rounds price plain overhead apart from drafted rows (refs #2605) #2687 merge cleanly with both, and kv pool: a named TF_KV_SHARE's slots come where an asked cache goes #3125, mine and open, which sharesdocs/recipes/glm-5.3-flash-cuda.md, already conflicts with main inCHANGELOG.md, the same with this head. The listing stops at its limits (complete: false).Tests:
glm_lane_costs_host.zig): "each width keeps the form its windows ran faster on, and classic inside the margin", "a dip is timed again at each width's own form, not the last raced one", "a slow start outlasting the old warm-up decides no width's form and sets no price", "a drift through the race falls on both forms alike", "a window run slow right after the other form decides no width's form" and "a window takes the form of the narrowest timed width that holds it"; and incuda_lane_costs.zig, "a raced line names the more common form and the widths where the other runs".test-glm-lane-costs-host(10 tests) unless named. One form for every width, the more common one, as main keeps: "each width keeps…" (expected 998, found 1023) and "a dip is timed again…" fail. The kept forms not set before the prices are smoothed and the rows sent: the same two. No sweeps: "a slow start…" (actual 36,expected 21). Classic first in every round: "a window run slow right after the other form…" (expected 1023, found 0). Each form's table timed in one block, as main times its forms: "a drift through the race…" (expected 0, found 1023). Every width priced at classic's time: the MTP price test and the three others that read prices. Only the 2-, 4- and 8-row widths raced, as main races: six, the DFlash2 and MTP race tests among them. A window on the widest timed width under it instead of the narrowest that holds it: "a window takes the form…" (expected 8, found 24). The startup line's majority counted at half the widths: only "a raced line names the more common form…" fails, intest-cuda-host(6,365 of 6,396). A copy that ignores the name in four places (as glm cuda: TF_GLM_EXPERT_FORM names the expert form, so launches compare at one form #3151's red copy did, at this head's code) fails exactly glm cuda: TF_GLM_EXPERT_FORM names the expert form, so launches compare at one form #3151's four naming tests intest-cuda-host(6,362 of 6,396) and the named-form test intest-glm-lane-costs-host(9 of 10,expected 1, found 241).Checks on this head
bc6349146d, the CHANGELOG line on top ofe259495f26. It changes no code. The gate below ran one259495f26's tree, and the lean checker and the overlap on this head; the receipt ran the change before the rebase,4f9832ca00.python3 tools/zig/lean_check.py, on the base (f935ab25cf) and on this head: 341 and 341, no new findings. On main4cfd955737and this head merged with it: 341 and 341, the same findings.e259495f26's tree in WSL (x86_64, Zig 0.17.0),zig buildwithtest -Dfixed-cpu,test-kernel-contract,test-golden,test-cuda-host,compile-cuda-tests(x86_64 and aarch64),check-native-cuda-host,test-glm-lane-costs-host,test-glm-window-prices-host,test-glm-fact-contract-host,test-cuda-startup-costs-host,test-glm-startup-hostandtest-glm-link-transport-host: every step passed (test9,322 of 9,391 with 69 skipped,test-kernel-contract238 of 238,test-cuda-host6,454 of 6,484 with 30 skipped). pytest from the root with the CI's options plus-rs(-q -rs -p no:cacheprovider): 450 passed, 86 skipped, the same skips and reasons as glm cuda: TF_GLM_EXPERT_FORM names the expert form, so launches compare at one form #3151's gate. Main's CI pytest at8642b1ee81, the newest main pytest that has finished, has 452 passed and 84 skipped, and prints no skip reasons. Two of the skips here aretools/zig/test_nfkd_tables.py's, which want Unicode 16.0.0 (this Python 3.12 has 15.0.0).zig,zig-windows,zig-macos-dev-tools,pytest,release-paths, bothbenchmark-clientsjobs andfmtpassed; the two nightly host jobs report skipped.priceinfamilies/glm/cuda_lane_costs.zig), and only GLM's CUDA decode windows read them (cuda_moe.zig).Receipt
factored) and the drafted figures below. Unset, glm cuda: TF_GLM_EXPERT_FORM names the expert form, so launches compare at one form #3151's head races and keeps one form as main does (its own receipt: within 0.1% of main in every cell).TF_GLM_EXPERT_FORM(glm cuda: TF_GLM_EXPERT_FORM names the expert form, so launches compare at one form #3151) now names one form for every width; its row inplatform/environment.ziganddocs/environment.mdsays so.4f9832ca00(the change before the rebase; the rebasede259495f26differs only in context) and glm cuda: TF_GLM_EXPERT_FORM names the expert form, so launches compare at one form #3151's head before its rebase,b92072b06d, each built from an empty Zig global cache with Zig 0.17.0 andzig build native -Dnvcc=<its nvcc> -Dsm=121in a CUDA 13.3.1 build container; GLM-5.3-Flash XSFP4 at88736f91with its MTP head drafting;--context 1048576 --parallel 8 --mtp-confidence 0.5,TF_KV_SHARE=32, a 14 GiB reserve,TF_GLM_PREFILL_REDUCE=gatherandNCCL_CUMEM_ENABLE=1(the grid addsTF_GLM_PROMPT_TAIL=1), the GPU clock capped at 2,200 MHz. This head raced twice from an empty price cache, the second time after the first's records were moved aside, and its later launches read the second's records; then the grid ran glm cuda: TF_GLM_EXPERT_FORM names the expert form, so launches compare at one form #3151's head, this head, this head and glm cuda: TF_GLM_EXPERT_FORM names the expert form, so launches compare at one form #3151's head, the drafted launches the same, the two loads without draftsfactoredand unset, and last the second grid: glm cuda: TF_GLM_EXPERT_FORM names the expert form, so launches compare at one form #3151's head namingfactored(timing its windows at it), this head, this head and glm cuda: TF_GLM_EXPERT_FORM names the expert form, so launches compare at one form #3151's head namingfactored(reading that record).priceraces the forms by width, and its decode windows, which take each window's form by its rows. Ran on TP=4 GB10s: racing loads and loads that read their record with"draft": falserequests, loads that read the record with drafted requests, and loads without drafts,factorednamed and unset. Without drafts the server runs one-row windows in turn (each load's 4- and 8-stream rounds are 4.03 and 8.08 to 8.09 times its 1-stream round), so those two loads check the named form at one row, where the named load's rounds are 4.8 to 4.9% shorter than the unnamed one's; that a load timing nothing sends it for every row count is the host test's. Not run: this head naming a form on a load that times (drafts on), whose ten sweeps and timing at the named form only the host tests run (the second grid's named loads are glm cuda: TF_GLM_EXPERT_FORM names the expert form, so launches compare at one form #3151's head's); the first racing launch's forms under drafts (every later launch of this head read the second race's record); DFlash2 drafters (the race times every width there too and keeps no prices), TP=2 and TP=3.tools/bench_concurrent.py --alone --serialfinds every concurrent reply equal to the same request alone (78 checks a launch) and to the serial"draft": falsereplies (48 a launch), none failed, and everybench_openaiandbench_concurrentreply hash equals glm cuda: TF_GLM_EXPERT_FORM names the expert form, so launches compare at one form #3151's head's first launch's (12 and 78).tools/bench_openai.py(the median of three 256-token runs) andtools/bench_concurrent.py(a cell's aggregate, the median of three reps; request i sendstools/bench_prompts.json's item i, 256 tokens):bench_openaifibonacci-rawbench_openaigpu-chat-no-thinkbench_concurrent1 streamAt T=0 the four launches draft the same rounds, accepted and drafted tokens. By the same pair rule, this head's
bench_openaigpu-chat-no-think median is 0.87% higher; fibonacci-raw (+0.64%) and the threebench_concurrentcells can't be separated. At eight streams one launch of this head sat at 117.0, every rep of it under every rep of #3151's head's two, and the other at 118.9, over them. At T=1 the drafting differs from launch to launch, so those figures are for the record.cuda_moe.zigsets it only when the pass is not a prompt's), so prompt passes run the same kernels on either build.Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (bench_concurrent --alone --serialon the four drafted launches above; the forms reach only decode rows, which give the same bits on either.)--drafter, no slower than without it, greedy and sampled. (Without drafts this head's rounds are 4.87% shorter than glm cuda: TF_GLM_EXPERT_FORM names the expert form, so launches compare at one form #3151's head's at one stream and 2.27% shorter at eight, and the two can't be separated at four; against glm cuda: TF_GLM_EXPERT_FORM names the expert form, so launches compare at one form #3151's head namingfactored, they are 2.70% shorter at four and can't be separated at one and eight. With drafts at T=0 one figure is 0.87% higher and the other four can't be separated; at eight streams one launch of this head sat 1.3% under glm cuda: TF_GLM_EXPERT_FORM names the expert form, so launches compare at one form #3151's head's two, every rep of it under every rep of theirs, and the other above them. The form reaches only decode rows. Where factored's lead at a width sits near the race's margin, two launches can keep different forms there, as the first paragraph says.)