Repository navigation
Conversation
…re at one form GLM races its classic and factored NVFP4 expert forms when a build first loads and keeps the winner under a key that carries the build's hash. The race's window sums can land inside its 1% margin, so the same code built twice can serve different forms, and a comparison of two builds then carries the form along with the change it measures. TF_GLM_EXPERT_FORM=classic or factored names the form: rank 0 sets it on every rank before the warm-up, skips the race, and times the windows and the draft step at that form. A load that times nothing (no drafts, or too little scratch context) still sends the named form to every rank and logs `glm: expert form (untimed)`; unnamed, such a load stays on classic as before. The named form ends the cache identity, so its record and the race's never stand in for each other; unset or blank keeps the race and its key as they were. The startup line keeps its words and puts (TF_GLM_EXPERT_FORM) right after the form's word, which ends MTP's line. Any other text stops the load by name, as GLM's other rows do.
b92072b to
f935ab2
Compare
|
Thanks @Bizuayeu, this is a useful diagnostic switch, and it's kept tidy. A named It's queued to land. It's safe because with the variable unset the cache keys, the timing calls and the startup lines are all exactly main's, and the form only picks which expert kernels run. Optional: a test showing that a record kept at one form isn't reused at the other would check the behaviour directly, not just the startup line's text.
|
…sed at the other The reuse check in price() moves into reuses(), which price() calls in its place. The new test keeps a classic record under a classic load's key in a temporary cache, then shows that a load naming factored finds nothing under its own key, and that reuses() refuses a record kept at the other named form while the race and the same form reuse it.
Added in |
What this changes
GLM's CUDA load races its classic and factored NVFP4 expert forms, which give the same bits, and decodes every window on the winner. The race's sums can land inside its 1% margin, so two launches or two builds of the same code can keep different forms, and a comparison of two builds then carries the form along with the change it measures.
TF_GLM_EXPERT_FORM=classicorfactorednames the form instead. Rank 0 sets it on every rank before the warm-up, skips the race, and times MTP's windows and draft step at that form. A load that times nothing (no drafts, or too little scratch context) still sends the named form to every rank and logsglm: expert form (untimed). The named form ends the price cache's identity, so its record and the race's never stand in for each other. Unset or blank keeps the race, its key and its startup line as they are. On four GB10s at TP=4 (GLM-5.3-Flash XSFP4,--context 1048576 --parallel 8), round time in ms from our own grid client ("draft": false, greedy, 256 tokens, each launch's median of six reps), lower is faster:classicfactoredMain and this head unset both raced to factored (
glm: windows (timed): … experts factored), and their later launches read that record ((kept)); unset, this head is within 0.1% of main in every cell. A named launch times its own record under the named key,classicright after the unset ones (glm: windows (timed): 26.62 ms at 1 row, 34.72 at 4, 99.16 at 32, a draft 1.37 ms; experts classic (TF_GLM_EXPERT_FORM)), and its second launch keeps it. The two forms trade places by width: againstclassic,factored's rounds are 4.82% shorter at one stream, 1.67% longer at four and 2.18% shorter at eight (the mean of two launches each), with the same replies. Without drafts (--no-drafts) the named load logsglm: expert form (untimed): factored (TF_GLM_EXPERT_FORM), the unnamed one logs nothing, and both answer as main does.ab78aef904), three commits: the change (c628011007), its CHANGELOG line (f935ab25cf) and, after review, the test the owner suggested (63dcb400ff). Rebased from main6c5cef8941, where the change was1f30d6c365and the receipt below was taken. Main had since taken glm cuda: TF_GLM_CONTEXT_CEILING serves windows past 1,048,576 tokens, read at GLM's open #2765, whoseTF_GLM_CONTEXT_CEILINGrow, registry entry and env-test lists sit where this change addsTF_GLM_EXPERT_FORM's; the rebase keeps both, and the env test's count of.glmrows adds both lists. Beside context, that is allgit range-diffshows between the two changes; the CHANGELOG commit is the same. Main has also taken glm cuda: under --mtp-confidence a greedy stream drafts its rates' depth, not fifteen a round #2864, which changes how deep a temperature-0 stream drafts under--mtp-confidence(this receipt runs at 0.5), so the receipt's main is6c5cef8941, "this head" in the receipt means the build before the rebase, and today's main can draft differently at T=0. Today's maind367f0fb7fhas since changed three of this PR's files:CHANGELOG.md,TF_GLM_PROF's row indocs/environment.md(docs: TF_GLM_PROF=1 runs GLM's prompt steps one chunk at a time #2990) and the GLM CUDA recipe's line on Jammed XSFP4 NZ5 (README: GLM-5.3-Flash Jammed XSFP4 NZ5 in Pick a model #2981); merged with it,git merge-treeis clean.atc overlap --jsonagainst maind367f0fb7flists sixteen open pull requests sharing a file with this head before it stops at its limits (complete: false), and marks qwen27: serve the 27B pair, refuse what a pair cannot keep, and die with the peer #1434 and qwen35 cuda: fuse the dense MLP's NVFP4 prompt gate and up projections into one launch #2410 as conflicting. With the open ones that earlier listings showed and those I added by hand, thirty-one share a file with this head, seventeen of them byCHANGELOG.mdalone.git merge-treeof main, and of main merged with this head, against each of them: nineteen already conflict with main, and all nineteen conflict in the same files with as many conflict hunks with this head. Two of them gain one more file,families/glm/cuda_lane_costs.zig. In GLM on CUDA: the latent split's per-rank page book and owner (TF_GLM_KV_SPLIT, #1431 part 1; stacked on #1411) #2415 (TF_GLM_KV_SPLIT), the price cache's key takes GLM on CUDA: the latent split's per-rank page book and owner (TF_GLM_KV_SPLIT, #1431 part 1; stacked on #1411) #2415'ssplitfield and this head's named tail, and the key test GLM on CUDA: the latent split's per-rank page book and owner (TF_GLM_KV_SPLIT, #1431 part 1; stacked on #1411) #2415's twelfth knob and this head's name argument; each resolves by keeping both, so whichever of the two lands second takes the other's lines. glm cuda: each decode window width keeps its faster expert form #3249, mine, is stacked on this PR's previous head, and its form check meets this head'sreuses()and its new tests this head's; I'll rebase it onto main once this lands. The other twelve merge cleanly with both.Tests:
glm_lane_costs_host.zig), "the expert form takes classic or factored, leaves blank to the race, and refuses the rest by name" (cuda_env_test.zig), "a load that times nothing sends a named form to every rank once, and nothing unnamed" and "a named form's startup line ends at (TF_GLM_EXPERT_FORM), and a raced one as it did" (cuda_lane_costs.zig).untimedsends nothing, the env row readsclassicasfactored) fails exactly four tests intest-cuda-host, 6,355 of 6,389: the set-once test (expected 1, found 3), the env test, the untimed test (expected false, found null) and the consumed key's named tail. The startup-line test reads only the line formats, which that copy keeps, so it passes there.cuda_lane_costs.zig) keeps a classic record under a classic load's key in a temporary cache, then shows that a load naming factored finds nothing under its own key, and that the reuse check refuses a record kept at the other form while the race and the same form reuse it. The check moved intoreuses(), whichprice()calls in its place.test-cuda-host, 6,447 of 6,478, where a classic record must be refused at factored. With the key's named tail always empty, the new test fails where the factored key must find nothing, and the consumed key's test where a named form must lengthen the key, 6,445 of 6,478; the third failure in that copy is cortheon fuzz's linear-time test, a timing test that passes in the gate below.Checks on this head
63dcb400ff, the test the owner suggested, on the CHANGELOG linef935ab25cfand the changec628011007. Its only code change movesprice()'s reuse check, as it was, intoreuses(). The checks below ran on this head; the receipt ran the change before the rebase, on main6c5cef8941.python3 tools/zig/lean_check.py, on the base (mainab78aef904) and on this head: 341 and 341, the same findings. On today's maind367f0fb7fand this head merged with it: 341 and 341, the same findings.zig buildwithtest -Dfixed-cpu,test-kernel-contract,test-golden,test-cuda-host,compile-cuda-tests(x86_64 and aarch64),check-native-cuda-host,test-glm-lane-costs-host,test-glm-window-prices-host,test-glm-fact-contract-host,test-cuda-startup-costs-host,test-glm-startup-hostandtest-glm-link-transport-host: every step passed (test9,315 of 9,385 with 70 skipped,test-kernel-contract238 of 238,test-cuda-host6,448 of 6,478 with 30 skipped). The new test runs and passes in bothtestandtest-cuda-host;test's one skip more than on the head before is Nemotron's ring-read test, which skips when its first read finishes before the second is asked. pytest from the root with the CI's options plus-rs(-q -rs -p no:cacheprovider): 450 passed, 86 skipped, the same skips as on the head before. Main's CI pytest at90b4fae62d, the newest main run that has finished, has 461 passed and 84 skipped, with the nine tests main has added since this base, and prints no skip reasons. Two of the skips here aretools/zig/test_nfkd_tables.py's, which want Unicode 16.0.0 (this Python 3.12 has 15.0.0).TF_GLM_EXPERT_FORM(priceinfamilies/glm/cuda_lane_costs.zig).Receipt
TF_GLM_EXPERT_FORM. Unset by default, which races the forms as main does;classicorfactorednames one; blank is unset, and any other text stops the load by name. Its row is inplatform/environment.ziganddocs/environment.md, and the GLM CUDA recipe says when to name it.1f30d6c365(the change before the rebase; the rebasedc628011007differs only beside glm cuda: TF_GLM_CONTEXT_CEILING serves windows past 1,048,576 tokens, read at GLM's open #2765's env lines, and this head's63dcb400ffonly moves the reuse check intoreuses()and adds a test) and main6c5cef8941, each built from an empty Zig global cache with Zig 0.17.0 andzig build native -Dnvcc=<its nvcc> -Dsm=121in a CUDA 13.3.1 build container; GLM-5.3-Flash XSFP4 at88736f91with its MTP head drafting;--context 1048576 --parallel 8 --mtp-confidence 0.5,TF_KV_SHARE=32, a 14 GiB reserve,TF_GLM_PREFILL_REDUCE=gatherandNCCL_CUMEM_ENABLE=1(the grid addsTF_GLM_PROMPT_TAIL=1), the GPU clock capped at 2,200 MHz. A first launch of this head, not counted, put its race's record in the price cache, as a build's first load does; then the grid ran main, unset,classic,factored,factored,classic, unset and main, the drafted launches main, unset, unset and main, and the two loads without draftsfactoredand unset.pricereads the name; unnamed, it makes main's calls. Ran on TP=4 GB10s: unnamed with drafted and"draft": falserequests, each named form with"draft": falserequests, and both without drafts. Not run: drafted requests at a named form. A named load prices its windows at that form (factorednamed: 34.84 and 97.97 ms at 4 and 32 rows, against the race's 37.18 and 102.10), so its drafted speed can differ from the raced load's at the same form. Also not run: TP=2 and TP=3, where the name is read and sent the same way and unnamed is main's path.tools/bench_concurrent.py --alone --serialfinds every concurrent reply equal to the same request alone (78 checks a launch) and to the serial"draft": falsereplies (48 a launch), none failed, and everybench_openaiandbench_concurrentreply hash equals main's first launch's (12 and 78).tools/bench_openai.py(the median of three 256-token runs) andtools/bench_concurrent.py(a cell's aggregate, the median of three reps; request i sendstools/bench_prompts.json's item i, 256 tokens):bench_openaifibonacci-rawbench_openaigpu-chat-no-thinkbench_concurrent1 streamAt T=0 the four launches draft the same rounds, accepted and drafted tokens. This head's
bench_openaimedians sit 0.4 to 0.8% below main's first launch, every run below every run of it, and above main's second, which ran both cases 3 to 4% below its first with no heat wait in any launch; the pairs lean opposite ways, so two launches a build can't separate them. Itsbench_concurrentcells sit with main's two, the higher 0.1% above them at most. At T=1 the drafting differs from launch to launch, so those figures are for the record.cuda_moe.zigsets it only when the pass is not a prompt's), so prompt passes run the same kernels under either name.Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (bench_concurrent --alone --serialon the four drafted launches above; the name doesn't reach the prompt cache.)--drafter, no slower than without it, greedy and sampled. (Unset, this head races to and keeps the same form as main: its round times are within 0.1% of main's, and its T=0 drafted figures can't be separated from main's two launches, as above. A named form is the caller's choice; the first table shows each form's round time by width, and a named form's drafted speed wasn't measured. The form reaches only decode rows.)