[jschmied]

Head: f366dd58319b00642c433e691dbe3cd0bc7df9e4 on main 223da7659b. Ledger row 20 (solo chunks), q3.5 cell, #1554 option C.

When the variable is set to 8192 and no other stream is decoding, a Qwen prompt pass takes up to 8,192 rows. Once another stream decodes, it goes back to the 4,096-row grid chunk.

The rule is the shared one, core/lanes/solo_chunk.zig (Limits, rows, read). It is the same rule Gemma 4 (#2391), Flash Next (#1733) and GLM (#1823) use.

When the variable is unset, nothing changes: the replies are the same as main's and so is the memory.

What changes

Core: nothing new.

• solo_chunk.rows and solo_chunk.read read the variable.
• platform/environment.zig gains a TF_QWEN35_CHUNK_SOLO row (.size, scope .cuda), with test rows in environment_test.zig and a line in docs/environment.md.

qwen3_5 CUDA only:

• Three row counts, where there used to be one (max_rows = 4096):

  • grid_rows = 4096. It keeps every cut-related meaning: cold cuts, kept states and resumes (snap_grid, onGrid), the pages' window, readout pieces, stage shapes and the disk probe.
  • max_rows = 8192 is the capacity for host tables, the round staging layout and the pocket row scales.
  • Engine.prompt_rows is read at load from the variable. It sizes the device memory that grows with the rows: the scratch rows, each stream's tap ring (tapRows() = 2048 + prompt_rows), the int8 prompt scratch, the EXL3 grouped scratch and a stage boundary. admit refuses a forward past it.

• cuda_prefill.soloEnd: an uncapped text pass merges consecutive cold chunks while all of these hold:

  • no other stream decodes (cuda_solo.othersDecode: a stream with a cache and no fill);
  • each merged chunk has at least solo_min_rows = 2048 rows;
  • the pass stays within the set rows;
  • it does not run past a mark the pass keeps.

  Images, --decode-share caps and short chunks (a tail, a request start's chunk, a capped step's rest under 2048) take the same cuts as main.

• Refusals by name, at load, before anything allocates:

  • error.BadChunkSolo, for any value other than unset, empty, 4096 or 8192;
  • error.QwenSoloStage, for 8192 on a pipeline stage.

  Both have messages in cuda_native_explain.zig.

Why there is a floor of 2048 rows: in an early version, every cold chunk merged. That moved a 27B reply. The prompt was 12,300 tokens, so main ends it with a 12-row tail, and that tail runs the projections' small-row tiles (qmmf.fused_rows 256; affine large tiles from 2048 rows). The merged pass ran those rows on the large tiles. On the head, a chunk under 2048 rows runs alone, as on main. The tails table below covers 12 to 4,000 rows on both bases.

Metal and HIP: none (this is a CUDA prompt-pass knob).

Evidence (GB10 sm_121, clock cap 2300, page cache dropped before every server start)

Binaries: main 223da7659b md5 78031f04, head md5 7b55bf9c. The arm order is main, off, on, on, off, main: off is the head with the variable unset, on is the head with TF_QWEN35_CHUNK_SOLO=8192. Each arm runs on a fresh server with --parallel 8 --context 40960, drafting on (DFlash2 for the 27B, MTP for the 35B-A3B). Paired medians are on vs main, by prompt; the "slower" counts are taken over those pairs.

Cold first token, alone (4 greedy prompts and 1 seeded prompt per size):

Model	6.2K	8.2K	12.2K	16.3K	32.7K	slower
Qwen3.6-35B-A3B 4-bit (MoE), on	−8.7 %	−5.8 %	−3.5 %	−5.8 %	−5.4 %	0 of 20
Qwen3.6-35B-A3B, off	+0.2 %	0.0 %	0.0 %	−0.2 %	−0.1 %	6 of 20
Qwen3.8-27B NVFP4 (dense), on	−1.9 %	−1.9 %	−1.2 %	−1.5 %	−1.6 %	0 of 20
Qwen3.8-27B, off	0.0 %	−0.2 %	−0.2 %	−0.2 %	−0.2 %	2 of 20

The 27B gains little: its prompt GEMMs are bandwidth-bound here (this box is bandwidth-bound). A discrete card (sm_120, sm_80) could show more.

Beside decoders: the same prompts while 1 or 3 streams decode.

• The prompt's first token is within ±0.75 % of main in every cell, both models, off and on.
• The decoders' longest token gap during the fill is equal to main's within 0.01 s. That gap is one grid chunk, so the busy chunk stays 4096.

Concurrency: 4 and 8 prompts sent at once.

• Wall time: −0.7..−1.9 % (35B) and −0.7..−0.9 % (27B).
• The median first token at c=8 drops by 18–19 % on both models, because the first passes run solo until the first stream decodes.

Multi-turn chats (/v1/chat/completions, A-B-B-A, 3 chats × 4 turns, greedy and seeded). With --prompt-cache-gib 8 the turns resume from kept states at 8,192 and 20,480 tokens.

• Resumed equals fresh in 72 of 72 turns on both models.
• Warm turns, on vs main: +0.04 % median (5 of 9 and 6 of 9 slower). These are level: a warm turn's rest is one chunk on both.
• Cold first turns: −4.0 % (35B) and −1.2 % (27B).

Exactness (on == off == main):

Model	Cells	Equal	Includes
35B	282 (solo, busy, concurrency)	282	312 decoder texts, compared on their common prefix (they stop by time)
27B	282 (solo, busy, concurrency)	282	312 decoder texts, the same way
35B	144 (chats)	144
27B	144 (chats)	144
Both	64 tails (32 per model): 4096 or 8192 + 12, 255, 256, 1000, 2047, 2048, 2100, 4000 rows, exact lengths, greedy and seeded	64	runs/tails27, runs/tails35

Memory (MemAvailable after load, --parallel 8):

• 35B: 94.1 → 93.4 GB on, unchanged off.
• 27B: 74.3 → 72.1 GB on, unchanged off. Most of this is the 8 tap rings at 10,240 rows.

Refusals:

• Loading with 6144, off, on, 2, " 0", 16384 or 8k exits 1 with the TF_QWEN35_CHUNK_SOLO message (runs/refuse.log).
• Unit tests by name: cuda_solo.zig covers the refused texts, the message and the stage refusal. cuda_prefill.zig and prefill_widths_qwen_host.zig cover merges, tails, marks, request starts, capped rests, and a decoding neighbour keeping the grid.

Checks on this head:

• x86 (q35solo-x86b): zig build -Dfixed-cpu test test-golden test-kernel-contract test-server-security test-pull-security fuzz-cortheon check-release-paths, tools/pr_check.py (lean 341 vs 341, 0 new), zig fmt --check, git diff --check: all green.
• GB10: zig build test-cuda-host 393/393 steps.

Overlap

These open PRs touch the same files:

• qwen3_5 cuda: a prompt chunk nothing reads stops its last layer at its keys #3060 and fix(qwen3.5/cuda): yield background prefill between chunks #2873 touch cuda_prefill.zig.
• qwen3_5 cuda: warm turns prefill the next turn's prompt and keep only a 4096-row-multiple end #2824 touches cuda_native.zig.
• qwen3_5 cuda: the GDN kernels read the state, its rollback copy and the log through a stream's state table (#2444 P3) #2620 touches cuda_cache.zig.
• qwen35: dense BF16 loading and native prompt logprobs #2020 touches cuda_prefill.zig, cuda_native.zig and qwen35_formats_host.zig.

Every conflict is textual. No other open PR does solo chunks for Qwen.

Proposed CHANGELOG line (not committed)

• Qwen 3.5/3.6/3.8 on CUDA: TF_QWEN35_CHUNK_SOLO=8192 lets a prompt pass take up to 8,192 rows while no other stream decodes, and the usual 4,096 once one does. On a GB10, a cold prompt's first token comes 3.5–8.7 % sooner on Qwen3.6-35B-A3B and 1.2–1.9 % sooner on Qwen3.8-27B NVFP4, at 6K–33K tokens, with the same replies. Each stream's tap ring and the prompt scratch grow with it (about 0.7 GB and 2.2 GB at --parallel 8). Unset, nothing changes. It uses the shared solo-chunk rule (gemma4 cuda: TF_GEMMA4_CHUNK_SOLO, on a solo-chunk rule shared in lanes (#1554 option C) #2391).

[ashhart]

Thanks @jschmied. The retained cache grid and short-tail guard look sound; 67 host checks and the native CUDA compilation pass. One small fix remains: the engine module reaches 601 lines on current main, adding a lean finding. Please bring it within 600 lines. After that, we will run the device check on our side for the wider passes and concurrent decoding.

• Synthena (Ash's Agent)