Repository navigation
docs: TF_GLM_PROF=1 runs GLM's prompt steps one chunk at a time - #2990
Conversation
|
Thanks @Bizuayeu, good catch. Checked against main at CI is green on
|
b53d77a to
e11fd72
Compare
With the profiler on, prompt_pair.pairedAt and prompt_solo.at return false (the e.prompt_profs[3] != null term in cuda_prompt_pair.zig and cuda_prompt_solo.zig), so every prompt step is a single chunk. The environment table did not say so, and a profile could be read as the served shape.
e11fd72 to
722eec3
Compare
|
Thank you @Bizuayeu. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes. |
What this changes
docs/environment.mddidn't say that the GLM profiler changes the prompt steps it times. WithTF_GLM_PROF=1,prompt_pair.pairedAtandprompt_solo.atboth return false (thee.prompt_profs[3] != nullterm atcuda_prompt_pair.zig:63andcuda_prompt_solo.zig:35), so every prompt step is one chunk. The table now says so, so a profile isn't read as the shape a server runs.a9f3f959d75f).atc overlap --json(2026-11-12 05:55 UTC, base main000b534a2845;complete: false, with the other pull requests' limit, unavailable pull files and the time limit): 35 open pull requests share a file, and none changes theTF_GLM_PROFrow. atc tested 9 of them: 5 conflict and 4 merge cleanly; the other 26 it could not test. The five conflicts are inCHANGELOG.mdonly and inherited: GLM-5.3-Flash on CUDA keeps conversation states on disk, every rank's part (--learn-conversations) #1411, dsv41 metal: the lane target, the prompt cache and the native server over Metal #1720, dsv41 metal: the vision tower's kernels under the kernel contract #1721, dsv41 metal: image prompts #1722 and server: images take the visual-token budget in prompt order (#2510) #2717 each conflict with main there in one block, and with this branch in the same one block. The 11 that also changedocs/environment.mdmerge with this branch cleanly there (git merge-tree); two of them, glm cuda: TF_GLM_GRAPHS=1 replays verify windows as CUDA graphs (#1490) #2279 and GLM on CUDA: the latent split's per-rank page book and owner (TF_GLM_KV_SPLIT, #1431 part 1; stacked on #1411) #2415, already conflict with main inCHANGELOG.mdand in Zig files this branch does not touch.Checks on this head
e11fd721b5f0, which only puts this pull request's number in the CHANGELOG line, on1d5b187ce056(docs only). The two areb53d77aebe71andd3bce23b9821rebuilt on maina9f3f959d75fafter the 1.2.8 release: the CHANGELOG line now sits under the new Unreleased, anddocs/environment.mdhas the same patch as before. On this head,tools/pr_check.py --base origin/main: 0 errors, 0 warnings (lean 341 vs 341, 0 new). The lean and test lines below ran ond3bce23b9821.python3 tools/zig/lean_check.py, on the base and ond3bce23b9821: 345 and 345, 0 new.tools/pr_check.py: 0 errors, 0 warnings.python -m pytest -q -rs -p no:cacheproviderfrom the root, on the base and ond3bce23b9821: 448 passed, 2 failed, 86 skipped and 519 subtests passed on each, with the same skip reasons. The two failures aretools/zig/zig_cache_restat_test.py'sRestatTests, which timed out at 300 s on both trees while theirzig buildwaited behind another build on the machine; run alone, they pass on both (2 passed each). Main's CI run of the same job on000b534a2845has 452 passed and 84 skipped: the two more skips here aretools/zig/test_nfkd_tables.py's, since this interpreter has Unicode 15.0.0.e11fd721b5f0(runs 43531503392 and 43531503418): benchmark-clients (ubuntu-latest, windows-latest), pytest, release-paths (ubuntu-24.04), zig, zig-macos-dev-tools, zig-windows and fmt pass; host-nightly-linux and host-nightly-macos were skipped by the workflow.Receipt
n/a: documentation only, so it cannot reach inference. What the row describes, measured on four GB10s at TP=4 (TensorFold 1.2.7,
93a2e93ece;Bizuayeu/GLM-5.3-Flash-XSFP4at88736f9; the median time to the first token of three cold prompts of 31,299 to 31,305 tokens; two starts each, one with the profiler): the default (pair steps) 16.856 s,TF_GLM_CHUNK_SOLO=409620.002 s andTF_GLM_PROF=120.115 s. The reply to a 32,611-token prompt had onetoken_sha256over all five starts.Checklist