[gilby]

What this changes

On HIP, Flash Next's draft chains stop under a 0.3 confidence floor by default, the HIP-wide default every HIP family shares. Flash Next on one RDNA 3.5 GPU (the Strix Halo and other gfx115x iGPUs) serving alone now defaults to 0.8 through a family default (mtp_confidence_rdna3_5 in families/flashnext/hip/native.zig, resolved by hip_families.confidenceFloor with the same RDNA 3.5 one-rank test Policy.forCaps uses for prefill_step). Other cards, groups of GPUs and every other HIP family keep 0.3, and --mtp-confidence still overrides it. The floor only decides how many drafts a chain proposes, so replies are unchanged. The start-up line prints the floor in use. A sweep of floors from 0.4 to 0.9 on this chip (in the comments) put 0.8 at the top of every cell and found it the only floor with no slower cell.

• Issue: n/a. It is a default change for one family on one backend; HIP: sampled chat at 4 streams on Qwen3.6-35B-A3B is 6% faster with --mtp-drafts 0, because the depth rule underprices sampled drafted rounds #2605 discusses a related depth-rule gap, which this does not address.
• Stacked on: main.
• atc overlap --json against main (000b534a2): incomplete (pullFilesLimit, timeLimit). Of the PRs it could compare, nine conflict in one file each and one is clean; the file this PR shares with almost every open PR is CHANGELOG.md.

Checks on this head

• Head commit: 52800fb98, a merge of main 000b534a2 into the branch (conflict in CHANGELOG.md only). The change is the 0.8 default, recipe and changelog (e5e757b52) and its scope to one RDNA 3.5 GPU (391df1d05). The receipt below ran on 52800fb98.
• Lean checker, python3 tools/zig/lean_check.py: 345 problems on the base and 345 on this head, 0 new findings. python3 tools/pr_check.py --base origin/main --pr 2697: 0 errors, 0 warnings.
• Tests run on this head: zig build test -Dhip=true --summary all: 889 of 891 steps succeeded, 13,643 of 13,757 tests passed, 113 skipped, 1 failed; the failure is dashboard_test "dashboard page polls the public stats snapshot without private details", which fails the same way on main 000b534a2. zig build hip-kernel-test on gfx1151: 20 of 20 steps, 159 of 165 tests passed, 6 skipped, the same as main. With rdna3_5Alone forced to true, the new test fails at gfx1151 on two ranks ("expected 0.3, found 0.8"). The host test builds each card's Caps.of row: Flash Next resolves 0.8 on gfx1150–gfx1153 at one rank and 0.3 on gfx1151 at two ranks and on gfx1030, gfx1100 and gfx1201; qwen3_5 dense and MoE keep 0.3 on gfx1151; --mtp-confidence wins over the default on every card.
• CI: not yet reported for this head; I will post each failure and whether this change caused it.
• Not run: other HIP chips (no card), other Flash Next checkpoints.

Receipt

• Main against branch on this head: one Strix Halo iGPU, four fresh servers with the page cache dropped before each, in the order main (000b534a2), this, this, main. Both arms of each side are shown.
  • One stream (bench_openai --tokens 256 --reps 3): code 46.25 / 46.25 tok/s on main vs 49.06 / 49.13 here (+6.2%), chat 40.09 / 40.08 vs 42.37 / 42.42 (+5.8%).
  • Distinct streams (bench_concurrent --levels 1,2,4 --reps 3 --prompts tools/bench_prompts.json --alone --serial --temperatures 1.0,0), aggregate tok/s, steady state in brackets. Greedy: 1 stream 51.5 / 51.5 vs 51.6 / 51.7; 2 streams 67.5 / 67.4 (69.4 / 68.7) vs 66.8 / 66.9 (70.4 / 70.5); 4 streams 67.1 / 67.2 (73.9 / 73.2) vs 69.0 / 69.1 (76.5 / 76.3). Sampled at 1.0: 1 stream 45.2 / 45.2 vs 45.3 / 45.4; 2 streams 59.5 / 59.5 (62.8 / 62.7) vs 61.6 / 61.3 (64.3 / 64.4); 4 streams 64.4 / 64.6 (69.5 / 69.4) vs 67.4 / 67.5 (72.5 / 72.3).
  • One cell is slower in aggregate: two streams greedy, −0.9%, while its steady state is 2.0% faster. Not yet traced.
  • Cold, four streams on a fresh server: greedy 52.8 / 52.9 vs 55.4 / 55.2, sampled 39.4 / 39.5 vs 42.4 / 41.0.
• Boxes it reaches: one RDNA 3.5 GPU (gfx1150–gfx1153) serving Flash Next alone with MTP drafts and no --mtp-confidence. Discrete RDNA 2, 3 and 4 cards and groups of GPUs keep 0.3, unchanged from main.
• New switch? None; the existing --mtp-confidence keeps its range and wins over the default.
• Environment: TensorFold main 000b534a2 plus this change, Zig 0.17.0, zig build native-hip -Dhip=true -Dhip-arch=gfx1151 -Doptimize=ReleaseFast with the system hipcc (HIP 7.1, clang 21), one Strix Halo iGPU (gfx1151), Linux 7.0, unigilby/Qwen3.8-Flash-Next-oQ4e-HIP (revision 5a0fb414), --context 131072 --parallel 4 --prompt-cache-gib 0.
• Paths and platforms the change reaches: the HIP Flash Next draft chain's stop (run at 1, 2 and 4 streams); other cards', groups' and other HIP families' defaults (host test).
• Exactness, greedy and sampled: every request's token_sha256 equals main's (42 of 42 warm and 8 of 8 cold, in each arm); bench_concurrent --alone --serial equal in every cell of every arm at both temperatures (alone 3/3, 6/6 and 12/12 at 1, 2 and 4 streams; "draft": false 4/4), 0 unequal, 0 failed.
• Main against this branch, same machine and checkpoint: above.
• Decode speed: above.
• Prompt speed: the floor only decides how many drafts a chain proposes and doesn't reach prompt processing; cold 8K first token (bench_cache --require-cold) 12.04 / 12.04 s on main vs 12.00 / 12.00 s here, reply hash equal.
• Licence of new weights: n/a.

Checklist

• The title and description match the head commit, including the receipt and the stack.
• New tests fail before the change, pass after it, and skip cleanly without their dependency.
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.
• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. Drafted and concurrent are checked above; resumed prompts not re-run.
• No precision traded for speed. No sums change.
• Decode and prompt processing are no slower than main at one stream and the widest admitted. True at one and four streams; the two-stream greedy cell is 0.9% slower in aggregate and 2.0% faster in steady state (above).

[gilby]

We're running a sweep to see what the optimal default should be, since it's 0.4% slower on one of the official benchmarks.

@AbrahamCete if you have a chance to run the 3 PRs to confirm there is no degradation with the stuff you're working on, we'd appreciate it.

[gilby]

0.8 benchmarked the best and matches main's single stream performance. 6% improvement for code vs. 3% for 0.7 compared to main. Receipts are being prepared.

[AbrahamCete]

@gilby, I ran the three together on gfx1151 (Strix Halo), against what I'm working on: no degradation.

I don't have the Flash Next checkpoint here (its 115 GB is more than this box holds), so this covers what the PRs share with my work, not Flash Next serving. A is main 26e1b07; B is the same main with #2695, #2696 and #2697 merged in that order. They merge with main with only CHANGELOG.md conflicting, since main now builds it from lines/<pr>.txt. I took main's side there.

• Flash Next's fused prompt prep through linear(): on B, the GPU test from test(hip): Flash Next's fused prompt prep through linear() on the GPU #2706, my follow-up to perf(hip): prompt DeltaNet at half a value head a block, and its prep in one launch #2582 (flashnext-hip-gpu-test), passes 3/3, as on main. It runs fused against unfused, bit for bit, at one and two segments, so the engine, forward, head and lanes changes leave the fused branch's wiring alone.
• Qwen3.6-35B-A3B-MLX-4bit-MTP served with its default MTP drafts and no draft flags (--context 4096 --parallel 4, a fresh server per run in the order A, B, B, A, bench_concurrent --levels 1,4 --tokens 256 --reps 3 --temperatures 0 --alone --serial):
  • Both print mtp drafts 3 under 0.3 at start-up, so hip flashnext: draft chains stop under a 0.8 confidence floor by default #2697 leaves qwen3_5 at 0.3.
  • token_sha256 is equal in every cell of every run (code 9f2923ac14d0, chat a8c28b8297b4), and drafted equals serial.
  • Aggregate tok/s, the median of 3 reps, as the range over the two runs of each:

cell	main	the three PRs	means
code, 1 stream	122.7–126.9	122.6–123.1	−1.6%
code, 4 streams	243.2–244.6	244.2–244.5	+0.2%
chat, 1 stream	92.8–97.9	98.2–100.0	+3.9%
chat, 4 streams	201.9–202.1	202.3–203.5	+0.4%

At 4 streams it's flat. At one stream, main's own two runs differ by 3-5% between fresh servers, so I read both one-stream moves as noise.

[gilby]

I changed the default from 0.7 to 0.8 after sweeping the floor on this chip. Same machine and checkpoint, the opened head built once with --mtp-confidence set per server, fresh servers alternating with main (main, 0.5, 0.6, 0.7, main, 0.7, 0.6, 0.5, main; then main, 0.8, 0.9, 0.7, main, 0.7, 0.9, 0.8, main; then main, 0.4, 0.4, main). Aggregate tok/s, two arms each, every --alone --serial cell equal:

floor	1 stream, bench_prompts.json	4 streams	bench_openai code	bench_openai chat
0.3 (main)	51.1–51.6	65.4–66.9	45.8–46.3	39.7–40.1
0.4	51.4 / 51.8	65.7 / 66.1	46.2 / 46.5	39.9 / 40.2
0.5	51.4 / 51.4	66.8 / 67.7	46.6 / 46.7	40.9 / 41.0
0.6	51.3 / 51.3	67.4 / 67.5	46.4 / 46.4	41.6 / 41.6
0.7	51.3 / 51.3	67.1 / 68.5	47.6 / 47.7	42.1 / 42.1
0.8	51.6 / 51.6	68.1 / 68.6	49.0 / 49.0	42.3 / 42.3
0.9	50.5 / 50.6	67.6 / 67.7	48.1 / 48.2	41.6 / 41.6

0.8 is the top of every column and the only floor whose one-stream bench_prompts.json cell equals main's; at 0.9 the chains stop too often. The new head (7d95c746a) has 0.8 as the default, and the description now carries its own main-against-branch receipt: code +5.9%, chat +5.5%, 4 streams +2.6%, one stream on bench_prompts.json equal. Since it no longer matches CUDA's 0.70, the "as on CUDA" wording is gone. I've marked this ready for review.

[ashhart]

Thanks @gilby, and thanks @AbrahamCete for checking the three together. The sweep makes a clear case: on the Strix
Halo, 0.8 tops every column and is the one floor with no cell slower than main. Replies stay equal greedy and seeded,
since bench_concurrent runs both temperatures by default. The change itself is small and right. The flag still
wins, --policy can't set the floor, and the start-up line prints the floor in use.

Two things before it lands:

1. Which cards get 0.8. The new default reaches every HIP card serving Flash Next, including discrete RDNA 3 and
   RDNA 4 and two-card groups, but the sweep ran on gfx1151 at one rank. A card where verify rows are cheaper may want
   longer chains. Could you scope it the way Policy.forCaps scopes prefill_step (zig/src/hip/policy.zig:57-58:
   RDNA 3.5 at one rank) and leave 0.3 elsewhere? A main-against-branch run on one discrete card would work too. Either
   is fine with us.
2. The recipe (docs/recipes/qwen3.8-flash-next.md:481) now says only Flash Next's CUDA and HIP engines have this
   rule. Qwen 3.5, 3.6 and 3.8 on HIP have it too, at 0.3 (hip_lanes.zig:455), as your CHANGELOG line says. "The MLX
   path refuses it" would be accurate.

The CHANGELOG conflict is ours, and we'll add the PR number and your credit to the line when it lands.

[gilby]

Thanks @ashhart. Both are done, rebased onto main 33ada77c3:

1. Scoped to RDNA 3.5 at one rank. Flash Next's 0.8 now applies only when caps.generation == .rdna3_5 and the run has one rank, the same test Policy.forCaps uses for prefill_step. Discrete RDNA 2, 3 and 4 cards, groups of GPUs, and Qwen 3.5, 3.6 and 3.8 keep 0.3, and --mtp-confidence still wins everywhere. The family declares mtp_confidence_rdna3_5, hip_families.confidenceFloor takes the RDNA 3.5 one-rank answer from native/hip.zig, and the test now builds that answer from each card's real Caps.of row: 0.8 on gfx1150–gfx1153 at one rank, 0.3 on gfx1151 at two ranks, 0.3 on gfx1030, gfx1100 and gfx1201, 0.3 for both qwen3_5 families on gfx1151, and the flag over the default in both directions.
2. The recipe now reads: "The native HIP engine defaults to 0.80 for Flash Next on one RDNA 3.5 GPU (the Strix Halo and other gfx115x iGPUs) and to 0.3 on other cards, across several GPUs, and for Qwen 3.5, 3.6 and 3.8. The MLX path refuses it." The Strix Halo section says the 0.80 is for that iGPU serving alone, and the CHANGELOG line names the same scope.

On a Strix Halo serving alone the floor is the same 0.8 as before, so the sweep and the receipt in the description still describe this head; nothing changes there. Main's new commits since the receipt touch no file this PR changes.

One correction to "equal greedy and seeded": my sweep and receipt passed --temperatures 0, so they checked greedy replies only. I'm running main against this head on gfx1151 now with bench_concurrent --levels 1,2,4 --prompts tools/bench_prompts.json --alone --serial --temperatures 1.0,0, comparing every request's token_sha256 with main's, and will post the sampled cells here.

Checks on 391df1d05: zig build test -Dhip=true --summary all: 700/700 steps succeeded, 0 failed (9,120 of 9,190 tests passed, 70 skipped, on the same code before the test moved to its own file to keep native/hip.zig under 600 lines). With rdna3_5Alone forced to true, the new test fails at gfx1151 on two ranks ("expected 0.3, found 0.8"). zig fmt --check clean on the three touched Zig files.

[ashhart]

Thanks @gilby. Both points are done: the 0.8 floor now uses the same RDNA 3.5 one-rank test as prefill_step, the test covers every card in the table plus both qwen3_5 families and the flag, and the recipe reads correctly. Thanks also for correcting the greedy-versus-seeded point. What's left: the sampled run you're doing (bench_concurrent --levels 1,2,4 --prompts tools/bench_prompts.json --alone --serial --temperatures 1.0,0 on gfx1151, every request's token_sha256 equal to main's and drafted equal to "draft": false), and the CI results for 391df1d05.

• Synthena (Ash's Agent)

[gilby]

Thanks @ashhart. The sampled run is done, on a new head.

Head. 52800fb98 merges main 000b534a2 into the branch as a plain merge commit, with no force push. The only conflict was CHANGELOG.md, and resolving it lets CI start. The change itself is as you reviewed it at 391df1d05.

Receipt, sampled and greedy. One Strix Halo iGPU (gfx1151), four fresh servers with the page cache dropped before each, in the order main, this, this, main. Main is 000b534a2. bench_concurrent --levels 1,2,4 --reps 3 --alone --serial --temperatures 1.0,0 --prompts tools/bench_prompts.json, aggregate tok/s with steady state in brackets, both arms of each side shown:

temperature	streams	main	this head	aggregate	steady
0	1	51.5 / 51.5	51.6 / 51.7	+0.3%
0	2	67.5 / 67.4 (69.4 / 68.7)	66.8 / 66.9 (70.4 / 70.5)	−0.9%	+2.0%
0	4	67.1 / 67.2 (73.9 / 73.2)	69.0 / 69.1 (76.5 / 76.3)	+2.8%	+3.9%
1.0	1	45.2 / 45.2	45.3 / 45.4	+0.3%
1.0	2	59.5 / 59.5 (62.8 / 62.7)	61.6 / 61.3 (64.3 / 64.4)	+3.3%	+2.5%
1.0	4	64.4 / 64.6 (69.5 / 69.4)	67.4 / 67.5 (72.5 / 72.3)	+4.6%	+4.2%

• Tokens. Every request's token_sha256 equals main's in all three other arms: 42 of 42 in the warm run and 8 of 8 in the cold four-stream run, at both temperatures. In every arm and every cell the concurrent replies equal the alone ones (3/3, 6/6 and 12/12 at 1, 2 and 4 streams) and the alone replies equal "draft": false (4/4), with 0 unequal and 0 failed.
• One cell is slower in aggregate: two streams, greedy, −0.9% (66.8 and 66.9 against 67.4 and 67.5), while the same cell's steady state is 2.0% faster. The arms repeat within 0.1 tok/s, so I read it as real and not noise. I have not traced where the aggregate loses what the steady state gains. On the earlier main this cell was faster here (67.1–67.5 against 66.4–66.5), so main has gained in this cell since. Your call whether that holds the change; I can look into it if you want it resolved first.
• Cold, four streams on a fresh server: greedy 52.8 / 52.9 on main against 55.4 / 55.2 here, sampled 39.4 / 39.5 against 42.4 / 41.0.
• One stream, bench_openai --tokens 256 --reps 3: code 46.25 / 46.25 against 49.06 / 49.13 (+6.2%), chat 40.09 / 40.08 against 42.37 / 42.42 (+5.8%).
• Cold 8K first token, bench_cache --require-cold: 12.04 / 12.04 s against 12.00 / 12.00 s, reply hash equal in all four arms.
• The start-up line reads mtp drafts 3 under 0.3 on main and mtp drafts 3 under 0.8 here.

Checks on 52800fb98. python3 tools/pr_check.py --base origin/main --pr 2697: 0 errors, 0 warnings; lean 345 on the base and 345 here. zig build test -Dhip=true --summary all: 889 of 891 steps succeeded, 13,643 of 13,757 tests passed, 113 skipped, 1 failed. The failure is dashboard_test "dashboard page polls the public stats snapshot without private details", which fails the same way on main 000b534a2 without this change. zig build hip-kernel-test on the GPU: 20 of 20 steps, 159 of 165 tests passed and 6 skipped, the same as main; this PR adds no GPU test.

CI. I'll post the results for 52800fb98 here once the run finishes, with each failure and whether this change caused it.

Environment: Zig 0.17.0, zig build native-hip -Dhip=true -Dhip-arch=gfx1151 -Doptimize=ReleaseFast, system hipcc (HIP 7.1, clang 21), Linux 7.0, unigilby/Qwen3.8-Flash-Next-oQ4e-HIP at revision 5a0fb414, --context 131072 --parallel 4 --prompt-cache-gib 0. The description is updated to this head.

[ashhart]

Thanks @gilby, this is the run we asked for, and a thorough one. Every request's token_sha256 equals main's in all three other arms, warm and cold, at both temperatures, and concurrent equals alone everywhere. So the 0.8 floor changes speed only, never tokens.

On the numbers: one stream is +5.8–6.2%, and four streams +2.8% greedy and +4.6% sampled. Cold four-stream and the 8K first token are level or better. The one aggregate dip, two streams greedy at −0.9%, comes with a +2.0% steady state. So it sits in the first rounds, not in the steady decode, and against the gains everywhere else we're happy to take it as it stands.

Nothing more is needed from you beyond the CI results you mentioned. Once 52800fb98 is green, it goes into our landing queue, and we'll write the CHANGELOG line.

• Synthena (Ash's Agent)

[gilby]

CI on 52800fb98: all checks that ran passed (zig, zig-windows, zig-macos-dev-tools, pytest, fmt, release-paths, and benchmark-clients on Ubuntu and Windows); host-nightly-linux and host-nightly-macos were skipped, as they are for pull requests. No failures to report. Main has moved since the merge, so GitHub shows a conflict again; it is in CHANGELOG.md only.

[gilby]

The raw files behind the receipt above (the tools' JSON for all four arms: bench_concurrent warm and cold, bench_openai, bench_cache) are at https://gist.github.com/gilby/cfdab1d77815e737ada9259c9aa9ce91.

[ashhart]

Thank you @gilby. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes.