[SergiioB]

What this changes

Long-context decode on the Qwen3.8-27B XPU path is slow because every tree row of a round scores the whole prefix in scalar fp32: 15.1 tok/s at 131K prompt tokens on #2595, against 127 at a short prompt. With XPU_ATTN_PREF=256 a round reads the committed prefix once, on bf16 DPAS, for 8 (row, head) queries at a time: 50.7 tok/s at 131K, 90.1 at 32K, 114.7 at 8K. It is off by default; unset, decode runs today's kernels and token hashes equal main's.

The switch changes tokens: it sums the same products in another order and keeps attention weights in bf16, as attn_prompt does. Drafted output equals "draft": false under it.

Correction, same day: this pull request does not meet the rule you set in #2543 on 10 Nov ("a speedup has to keep main's tokens, even as an opt-in"). I opened it from the 8 Nov answer there and missed the later one; the first version of this description called it "the opt-in agreed in #2543", which was wrong. I have moved it to draft. It stays up only as evidence in case the numbers below are a reason to look at that rule again for a switch that is off by default, as XPU_KV_INT8 is. If the rule stands, close it and I will keep the kernel out of tree. Two exact variants were tried and neither helps: one exp a key in attn_decode keeps every token hash and is 1 % slower, and carrying P in two bf16 parts does not bring main's tokens back.

• Issue: Refs XPU Qwen3.8-27B: decode attention re-reads the KV cache once per tree row (11 tok/s at 131K); proposed fix +48 % #2543.
• Stacked on: XPU 27B decode attention: tree rows of a key slice run back to back (bit-identical, +23 % at 32K, +33 % at 131K) #2595 (head 078cac06b). This adds four commits: c7f5f11fd (kernels and launch), c7c1b2241 (docs), 9e78a0fe5 and aa19975fd (receipts).
• atc overlap --json: not run, atc is not installed on this machine. By hand: xpu: a sampled Qwen3.8-27B row draws from a top-k kept on the card #2895 also edits zig/src/xpu/dev.zig, forward.zig and qwen27.zig, and XPU: Qwen3.8-27B attention merge with 16 subgroups per head, bit-identical #2799 edits attn_merge in decode.cl. This change adds lines beside theirs (new kernel fields, one branch around the attn_decode launch, an #include at the end of decode.cl) and leaves attn_merge alone.

How it works

• pref_base(pos, CH) = ((pos - 16) / CH) * CH: the row at pos takes its keys below that from the prefix chunks and the rest (at most CH + 16) from slices. The cut depends on the row's position alone, and a tree row is at most 15 below its round's p0, so the chunks hold committed keys only and a drafted row cuts where the same row decoded serially does.
• attn_pref: one subgroup per (tile of 8 queries, K/V head, chunk of CH keys). Q·K on bf16_bf16_matrix_mad_k16, online softmax 16 keys a step, P in bf16, P·V on the same DPAS. fp32 partials a chunk.
• attn_tail: attn_decode's body from pref_base on. The body moved into attn_slices, which attn_decode calls with a constant 0, so its signature and its contract registration are unchanged.
• attn_merge2: merges a row's chunks, then its slices, in key order.
• XPU_KV_INT8=1, and tree rounds of several streams, keep the default path under the switch.

Checks on this head

• Head commit: aa19975fd.
• Lean checker: 116 problems on 078cac06b and 116 on this head; 0 new. python3 tools/pr_check.py --base 078cac06b: 0 errors, 1 advisory warning (no --pr number yet).
• xpu-kernel-contract-limits (no GPU): PASS: every registered launch fits bmg-g31, every record current, every control red.
• CI: none yet.
• tf-xpu-contract-run --device 0 on aa19975fd: SUMMARY bmg-g31 0xe223 10 kernels: 10 pass gates 1-4, 0 fail, 0 known findings; 7 controls: 7 red as planted, 0 missed; 0 limit problems, exit 0. attn_decode + attn_merge stay bit-equal to the upstream pair after the move into attn_slices.
• Not run: zig build test, test-golden, xpu-qwen27-tests.

Receipt

docs/receipts/xpu-qwen27-attn-prefix/ holds every timed request (requests.json) and the graded battery (battery.json).

• Main against branch: the default column is this branch with the switch unset, same session, same binary. Its token_sha equals XPU 27B decode attention: tree rows of a key slice run back to back (bit-identical, +23 % at 32K, +33 % at 131K) #2595's main column at every prompt length (26 requests), so it stands for main in the table.
• Boxes it reaches: Intel XPU, Qwen3.8-27B decode and verify rounds of one stream, bf16 K/V, only when XPU_ATTN_PREF is set.
• New switch: XPU_ATTN_PREF=CH, unset by default. Accepted: a multiple of 32 from 64 to 4096; anything else stops the load. No profile row.
• Environment: two Intel Arc Pro B70 (BMG-G31, 32 GB), issue running server #1 at a 230 W cap, Cannot get more than 8% GPU utilization on M4 with 48 gb (even using --resident-budget 30 GiB for example) #2 at 180 W. Ubuntu 26.04, kernel 7.0, xe, oneAPI 2026.1, Zig 0.17.0, ocloc 26.05.37020, plus the local build change XPU 27B decode attention: tree rows of a key slice run back to back (bit-identical, +23 % at 32K, +33 % at 131K) #2595 describes (not part of this PR). Checkpoint TensorFold/Qwen3.8-27B-MLX-4bit @ 22d8d538, drafter z-lab/Qwen3.8-27B-DFlash2. --parallel 1 --context 139264 --prompt-cache-gib 0 --no-thinking, greedy, 512 tokens, one stream, one discarded warmup a cell. The speed table was measured on a1d896a88 with these kernels inside decode.cl; this head moves them to decode_prefix.cl and returns the same token_sha on both paths at 8K and 32K (16 requests on B70 Cannot get more than 8% GPU utilization on M4 with 48 gb (even using --resident-budget 30 GiB for example) #2).
• Exactness under the switch: drafted equals "draft": false equals a second drafted run at 68 / 7,976 / 32,552 tokens (two repeats) and 130,856 (four), code and prose, and on the 275 battery items run with "draft": false (275 of 275; the other 54 were not run that way). The second card returns the first card's drafted text on 329 of 329 items.
• Decode, tokens/s after the first token, drafted, code / prose, B70 issue running server #1:

Prompt tokens (code)	default	XPU_ATTN_PREF=256	ratio
68	not run in this session	127.4 / 100.7	see below
7,976	90.0 / 71.6	114.7 / 91.5	1.27
32,552	45.9 / 33.9	90.1 / 66.5	1.96
130,856	15.1 / 13.6	50.7 / 45.6	3.35

"draft": false: 30.4 / 23.7 / 13.0 default and 30.4 / 27.3 / 18.8 with the switch at 8K / 32K / 131K. The two repeats at 8K and 32K agree within 0.05 tok/s; four at 131K give 50.70 to 50.79 (code) and 45.64 to 45.75 (prose). Correction (this row first read 1.00, "the same kernels"): that was wrong. A 512-token answer passes position 272, where the switch starts, and on B70 #2 in one session the switch is slower at a 68-token prompt: 123.3 / 98.6 default against 122.3 / 96.7 drafted (−0.8 % / −1.9 %), and 31.3 against 30.4 with "draft": false (−2.9 %). I am measuring a fix now: rows below position 1,040 stay on the default slices, so short contexts run today's kernels and return today's tokens. The cut still depends on the row's position alone. I will push it with its numbers and say here if it does not remove the loss. On B70 #2 at 180 W, this head: 85.0 to 109.0 at 8K and 42.3 to 84.5 at 32K (code), while the other card was serving.

• How the tokens change: against the default path the answer differs at 4 of 6 depth cells and on 67 of 329 battery items. XPU_ATTN_PREF=512 against 256 differs again at 3 of 4 cells, so the text follows the order of the sums, not the weights' precision: a variant that carries P in two bf16 parts (16 significant bits) did not bring the default text back and was 2–12 % slower.
• Quality, 329 items graded by rule checks and sandboxed execution, default / switch: HumanEval 78 / 78 and 76 / 76 of 82, knowledge 40 / 42 of 55, JSON schema 55 / 55, instruction following 33 / 33 of 55. Two items gained, none lost.
• Prompt speed: n/a, prompt rows use attn_prompt. Cold TTFT at 130,856 tokens is 134.0 s on both.
• Licence of new weights or drafter: n/a.

Not done, and what I propose for the gate

On 8 Nov #2543 asked an opt-in for an FP64 attention reference, teacher-forced error and margins, its own contract records and drafted equal to "draft": false. Only the last is here, and the 10 Nov rule comes before all of it.

1. Contract: register attn_pref + attn_tail + attn_merge2 as one operator against the float64 attention of xpu_attn_ref.zig, with store = bf16 for P, and a planted control (a merge that drops the last prefix chunk). Gate 4 needs your call: P·V sums bf16 weights on DPAS, as attn_prompt does.
2. Teacher-forced histories at 8K / 32K / 131K: worst per-column error against FP64 for the default path and the switch, and every top-token difference with its margin.
3. tools/bench_openai.py, prompt-cache resumes under the switch, sampled decoding.

If you'd rather see 1 and 2 before any review, say so and I'll keep this as is until they are in.

Revert: unset XPU_ATTN_PREF (the default), or git revert the four commits; no format or public interface changes.

Checklist

• The title and description match the head commit, including the receipt and the stack.
• New tests fail before the change, pass after it, and skip cleanly without their dependency. No new test yet; see the gate items.
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.
• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. Drafted equals "draft": false at every cell and on 275 battery items; resumed and concurrent were not run.
• No precision traded for speed. Under the switch P is bf16 where the default path keeps fp32, and the sums run in DPAS order; the default path's bits do not change.
• Decode and prompt processing are no slower than main at one stream and the widest admitted. One stream, greedy only.

[SergiioB]

Note

🤖 Claude Fable 5 responding on behalf of SergiioB

Closing this: it changes tokens under its switch, and #2543 (10 Nov) rules that out even as an opt-in. It was opened from the 8 Nov answer without reading the later one; that was our mistake, not a challenge to the rule.

What stays useful from it, all with main's tokens:

• Under the switch unset, token_sha equals XPU 27B decode attention: tree rows of a key slice run back to back (bit-identical, +23 % at 32K, +33 % at 131K) #2595's main column at every prompt length (26 requests), and tf-xpu-contract-run --device 0 on a head that includes XPU 27B decode attention: tree rows of a key slice run back to back (bit-identical, +23 % at 32K, +33 % at 131K) #2595's 078cac06b reads 10 kernels: 10 pass gates 1-4, 0 fail; 7 controls: 7 red as planted; 0 limit problems. That is the count asked for in XPU 27B decode attention: tree rows of a key slice run back to back (bit-identical, +23 % at 32K, +33 % at 131K) #2595; I'll repeat it there on XPU 27B decode attention: tree rows of a key slice run back to back (bit-identical, +23 % at 32K, +33 % at 131K) #2595's own head.
• Two exact attempts on attn_decode that are not worth repeating: one exp a key instead of two (one of them is always exp(0)) keeps every token hash and is 1 % slower at 8K and 32K; the same with two branches is far slower.

The kernel stays out of tree. Receipts remain on the branch SergiioB:xpu-qwen27-attn-prefix-dpas if anyone wants the numbers.