Skip to content

docs: TF_GLM_PROF=1 runs GLM's prompt steps one chunk at a time - #2990

Merged
ashhart merged 2 commits into
ashhart:mainfrom
Bizuayeu:docs/glm-prof-one-chunk
Nov 14, 2026
Merged

ashhart merged 2 commits into
ashhart:mainfrom
Bizuayeu:docs/glm-prof-one-chunk

Conversation

@Bizuayeu

@Bizuayeu Bizuayeu commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

What this changes

docs/environment.md didn't say that the GLM profiler changes the prompt steps it times. With TF_GLM_PROF=1, prompt_pair.pairedAt and prompt_solo.at both return false (the e.prompt_profs[3] != null term at cuda_prompt_pair.zig:63 and cuda_prompt_solo.zig:35), so every prompt step is one chunk. The table now says so, so a profile isn't read as the shape a server runs.

Checks on this head

  • Head commit: e11fd721b5f0, which only puts this pull request's number in the CHANGELOG line, on 1d5b187ce056 (docs only). The two are b53d77aebe71 and d3bce23b9821 rebuilt on main a9f3f959d75f after the 1.2.8 release: the CHANGELOG line now sits under the new Unreleased, and docs/environment.md has the same patch as before. On this head, tools/pr_check.py --base origin/main: 0 errors, 0 warnings (lean 341 vs 341, 0 new). The lean and test lines below ran on d3bce23b9821.
  • Lean checker, python3 tools/zig/lean_check.py, on the base and on d3bce23b9821: 345 and 345, 0 new. tools/pr_check.py: 0 errors, 0 warnings.
  • Tests run, with the command and pass, fail and skip counts: python -m pytest -q -rs -p no:cacheprovider from the root, on the base and on d3bce23b9821: 448 passed, 2 failed, 86 skipped and 519 subtests passed on each, with the same skip reasons. The two failures are tools/zig/zig_cache_restat_test.py's RestatTests, which timed out at 300 s on both trees while their zig build waited behind another build on the machine; run alone, they pass on both (2 passed each). Main's CI run of the same job on 000b534a2845 has 452 passed and 84 skipped: the two more skips here are tools/zig/test_nfkd_tables.py's, since this interpreter has Unicode 15.0.0.
  • CI: green on e11fd721b5f0 (runs 43531503392 and 43531503418): benchmark-clients (ubuntu-latest, windows-latest), pytest, release-paths (ubuntu-24.04), zig, zig-macos-dev-tools, zig-windows and fmt pass; host-nightly-linux and host-nightly-macos were skipped by the workflow.
  • Not run, and why: inference, since the change is documentation only.

Receipt

n/a: documentation only, so it cannot reach inference. What the row describes, measured on four GB10s at TP=4 (TensorFold 1.2.7, 93a2e93ece; Bizuayeu/GLM-5.3-Flash-XSFP4 at 88736f9; the median time to the first token of three cold prompts of 31,299 to 31,305 tokens; two starts each, one with the profiler): the default (pair steps) 16.856 s, TF_GLM_CHUNK_SOLO=4096 20.002 s and TF_GLM_PROF=1 20.115 s. The reply to a 32,611-token prompt had one token_sha256 over all five starts.

Checklist

  • The title and description match the head commit, including the receipt and the stack.
  • New tests fail before the change, pass after it, and skip cleanly without their dependency. (No new tests: one table row and one changelog line.)
  • Comments and docstrings are one line. No measurements or history in the source. (No source changes.)
  • No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.

Bizuayeu added a commit to Bizuayeu/TensorFold that referenced this pull request Nov 12, 2026
@ashhart

ashhart commented Nov 12, 2026

Copy link
Copy Markdown
Owner

Thanks @Bizuayeu, good catch.

Checked against main at 3885dceb2b. With TF_GLM_PROF=1 the engine creates its prompt profilers, so the e.prompt_profs[3] != null term turns off both wide steps (zig/src/families/glm/cuda_prompt_pair.zig:63 and zig/src/families/glm/cuda_prompt_solo.zig:35). Leader.chunk then sends one chunk at a time, up to the next cut (zig/src/families/glm/cuda_follow.zig:246). The new row says exactly that. Your time-to-first-token numbers at about 31K tokens (16.856 s by default, 20.115 s profiled) show how far off a profile would be if someone read it as the served shape.

CI is green on b53d77aebe, it merges cleanly with main, and the changelog line is under Unreleased. Nothing to ask; it goes into the landing queue.

  • Synthena (Ash's Agent)

With the profiler on, prompt_pair.pairedAt and prompt_solo.at return false
(the e.prompt_profs[3] != null term in cuda_prompt_pair.zig and
cuda_prompt_solo.zig), so every prompt step is a single chunk. The
environment table did not say so, and a profile could be read as the
served shape.
@ashhart
ashhart force-pushed the docs/glm-prof-one-chunk branch from e11fd72 to 722eec3 Compare November 14, 2026 01:14
@ashhart
ashhart merged commit 8ff0477 into ashhart:main Nov 14, 2026
12 of 20 checks passed
@ashhart

ashhart commented Nov 14, 2026

Copy link
Copy Markdown
Owner

Thank you @Bizuayeu. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants