[russellmm]

What this changes

With --vision, GLM-5.3-Flash on CUDA keeps its whole vision tower on rank 0: 1.05 GiB of bf16 weights that are idle
except while an image is encoded. Rank 0's room bounds every rank's KV pages and prompt cache.

With TF_GLM_VISION_HOST=1:

• The tower's bf16 weights load into pinned host memory, mapped into the device's address space.
• An encode copies each unit of the pass into one device slot just before the work that reads it, on the same stream:
  the patch embedding, each block, the post-norm and downsample, the merger's projection, and its MLP.
• The device keeps only that slot, sized to the largest unit: 240 MiB for this tower instead of 1.05 GiB.
• The weights and the arithmetic are unchanged, so replies are identical.
• Quantized towers (MXFP8/NVFP4) are refused in this mode.
• Unset, nothing changes.

Fixes #1759. Follows #1736 (prompt states in host memory, merged); the two touch different files.

Modules:

• cuda_vision_weights.zig (new): where the tower's tensors live, plus the name lookups moved out of
  cuda_vision.zig, which stays under 600 lines.
• cuda_vision_stage.zig (new): the units, the slot and the copies.
• cuda_vision.zig: load places the weights, and the encode brings each unit in turn.

No kernel, scheduler or arithmetic change.

Why

This is the second step toward serving GLM-5.3-Flash on two 96 GB cards with system memory relieving the KV window.
The checkpoint's weights take 86.2 GiB of each card, which leaves rank 0 about 4-5 GiB for KV pages and the prompt
cache. The first step, #1736 (merged), moved kept prompt states to host memory. This one moves the idle vision tower.

Why staged copies and not zero-copy. Letting the kernels read the weights from host memory directly saved a little
more device memory, but cost about 5x per image:

zero-copy, 300 W	640x400	1920x1080	2560x1440
tower on the device	0.18 s	1.32 s	2.41 s
tower read directly from host memory	0.96 s	6.90 s	10.86 s

The GEMMs read each weight many times. Staging copies each byte across PCIe once a pass (~1.05 GiB, about 19 ms at
~57 GB/s).

Receipt

• Environment.

  • TensorFold main 94b90c9 with and without this commit, built with Zig 0.17.0 and nvcc 13.4.92 (-Dsm=120).
  • Two RTX PRO 6000 Blackwell Workstation cards (96 GB each, PCIe Gen5 x16), TP=2 on one host, NCCL 2.32.3,
    cuBLASLt 13.4, driver 615.71.09, Linux 7.3.
  • Host: Threadripper 9970X, 128 GB DDR5.
  • Checkpoint Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold @ 76c0b517.
  • Serve flags on both ranks: --vision --context 327680 --kv-cache bf16 --parallel 2,
    TENSORFOLD_MEMORY_RESERVE_GIB=2.6.
  • --chat-template: the checkpoint's own template with its media branch taken from zai-org/GLM-5.3-Flash. The
    checkpoint's template is text-only: it replaces images with a refusal, and the server warns so.
  • libjpeg-turbo 3.1.4.1 and libwebp 1.6.0. GPU cap 400 W for every timing below.

• Exactness.

  • tools/bench_concurrent.py --levels 1,2 --tokens 256 --temperatures 0 --alone --serial, code and chat. Baseline
    and candidate: alone 3/3 and 6/6 equal, serial 1/1 equal in every cell; 0 unequal, 0 failed.
  • Images: greedy 120-token replies to five test images (640x400 as PNG, JPEG and WebP; 1920x1080 and 2560x1440 PNG)
    are identical between baseline and candidate.

• Memory (rank 0's startup line):

  	baseline	candidate
  model	86.19 GiB	85.37 GiB
  left after the 2.6 GiB reserve, 320K window	4.8 GiB: 642 KV pages and a 0.5 GiB prompt cache	5.7 GiB: 642 KV pages and a 1.3 GiB prompt cache
  --context 393216, same flags	refused: "the CUDA memory budget leaves 4.81 GiB"	serves: 770 KV pages and a 0.5 GiB prompt cache

• Image speed. First token of a 1-token reply; each request is salted so nothing is reused; median of 3, two
  alternating server starts each:

  image (prompt tokens)	baseline	candidate
  640x400 (377)	0.146 / 0.146 s	0.158 / 0.158 s
  1920x1080 (2,724)	1.073 / 1.076 s	1.083 / 1.083 s
  2560x1440 (4,113)	1.991 / 1.992 s	2.005 / 2.006 s

• Decode speed (tools/bench_openai.py --tokens 256 --reps 3 --temperatures 0), median tok/s; identical
  token_sha in every cell.

  prompt	baseline	candidate
  fibonacci-raw	258.1	257.5
  gpu-chat-no-think	144.8	144.5

  bench_concurrent.py aggregate at 2 streams: code 378.7 vs 378.1, chat 241.6 vs 241.3.

• Prompt speed (tools/bench_cache.py --require-cold --tokens 64), fixed token arrays from the repository's docs,
  fresh server each run, alternating. Cold ttft_s, with the same token_sha and cached_tokens 0:

  prompt	baseline (2 runs)	candidate (2 runs)
  32,768 tokens	7.93 / 7.88 s	7.99 / 7.89 s
  65,536 tokens	16.24 / 16.31 s	16.42 / 16.33 s

• Tests run.

  • zig build test --test-timeout 10m: 5,753 passed (main's 5,752 plus the new unit test in
    cuda_vision_stage.zig), 27 skipped, 0 failed.
  • test-kernel-contract: 141/141. test-golden: pass. test-server-security test-pull-security: 26/26.
    fuzz-cortheon: 17/17.
  • Plain zig build: pass. build_options_test.py: pass. git diff --check: clean.
  • lean_new.py against main: 424 on the base, 424 on the head, 0 new.

• Not run, and why.

  • Metal: the tower there is a different path.
  • GB10 and other unified-memory hosts: there the slot saves nothing, because host and device memory are one.
  • TP=3, quantized towers (refused in this mode), and sm_80-sm_89 cards: not available.
  • Video input: needs libtfvideo, not built here. It uses the same encode path, so it is expected to behave the
    same, but it was not measured.

Checklist

• Every token still goes through the lane rounds. Only where the tower's weights live and when they are copied
  changes.
• Drafted output equals "draft": false, and concurrent equals solo (resumed prompts are untouched by this change).
• No precision traded for speed. The weights are the same bytes, so the bits don't change.
• No new or changed GPU kernel.
• Prompt processing is no slower than main (cold 32K/64K above; image encodes 10-15 ms slower).
• CUDA TP=2 on sm_120 only; Metal and the other CUDA classes are not run (above).
• The new test needs no device and fails before the change, which it doesn't compile without.
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths. No AI attribution lines.

[ashhart]

Thank you @russellmm. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes.

[russellmm]

I have the final piece that should be done soon which is to allow most of the KV Cache to sit pinned in system memory. This will allow the full 1M context of the GLM5.3-Flash model on a Dual RTX-6000. Without the final piece, we can get up to 512K KV with a 1.5GB VRAM reserve.
P.S. I am @russell_m on X.

[ashhart]

Thanks @russellmm! TF_GLM_VISION_HOST=1, which keeps GLM-5.3-Flash's vision tower in pinned host memory, is in 1.2.1: https://github.com/ashhart/TensorFold/releases/tag/v1.2.1