Repository navigation
Conversation
…o seven tokens a round
… copies grow to the window
# Conflicts: # CHANGELOG.md
…lane core's setup
… three-draft cap has a test
--mtp-drafts N now takes up to seven, the verify window's eight rows less the pending one. A stream that is alone in a round chains up to N drafts; rounds shared by several streams keep at most three a stream. Without the flag the head drafts three as before.
…ther family keeps the refusal past three The --mtp-drafts limit is a family's mtp_max_drafts, three when it declares none, so Flash Next is refused past seven and the other HIP families past three as before.
|
CI is green on |
|
Thanks @gilby. Letting a lone stream fill the rows #2730 opens up, while shared rounds keep three drafts a stream, is a tidy follow-up. The receipt is complete: base against this head on the same box, and with the flag unset, every cell matches the base, including tokens a round and the server's drafted and accepted totals. Every We checked the default path. With This lands after #2730, which is queued. Once that's in, please rebase so the diff holds only this change's commits. Asks
Optional, and fine as a follow-up: the lone ladder now jumps from 2 straight to The one-line fold in
|
…ilities, granted and allocated rows and a stream's first drafts beside a live one
… the recipe says when a lower N pays
|
Thanks for the review. Three new commits on top of 1. The shared cap in every path. I applied the cap in those paths rather than refusing the combination in
2. A stream that joins mid-reply. One Strix Halo iGPU (gfx1151), a fresh server with the page cache dropped, this head with
Both replies keep the same 3. 4. Changelog and recipe. No token changes and level speed on this head. Four fresh servers on one Strix Halo iGPU (gfx1151), all with
The default path (flag unset) on this head. The review commits change code the default path runs, so I ran four more fresh servers the same way with no
Tests: Conflict with main. Main has since changed four lines of On the optional item (pricing depths 4 to 7 from the stream's own rates): I have not built it. Flash Next on HIP times no window costs, so |
… stays within 600 lines
…e seven drafts ran slower
|
I merged main What conflicted and how I resolved it (
#2728 is in main; #2730, the base of this stack, is not. Under
Checks on
Merged head (
The driver waited until no other process used more than 20% of a core, checked that again every 7 s during each benchmark window (model load excluded), and recorded
Raw JSON, server start-up lines, the load records and the driver: https://gist.github.com/gilby/f1284f25978e7190af42f4014be918c3 |
|
I merged main Main's new On head
Since |
|
CI on |
|
Thanks @gilby. The cap now covers every shared path, including a stream's first drafts beside a live one. The staggered-join runs, TP refusal, test moves and disclosed slower seven-draft cases answer the four substantive asks. Deeper cost pricing can stay a follow-up. The one remaining item is a resolved net diff after #2730 lands, containing only this change and preserving the resolution that moved the test import and kept modules within 600 lines. The separate commit patches omit those resolutions, so we haven't verified the final merged tree yet.
|
What this changes
With the eight-row verify window (#2730), a Flash Next stream on HIP that has a round to itself has room for seven drafts, but the MTP head stops at three.
--mtp-drafts Nnow takes up to seven on Flash Next. A stream alone in a round chains up to N head drafts, and rounds shared by several streams keep at most three a stream. Without the flag the head drafts three as before, and replies, rounds and draft counts are the same as the base's.The lane core's depth rule gets one setting,
shared_mtp_cap. Its default is 0 for every other family, which means off and leaves the rule as it is. When it is set, a lone stream's top rung is the family's most drafts, and a shared round chains at most the setting a stream in every path: the cost-allocated budgets, the confidence-floor and node-probability branches, granted and allocated shared rows, and the first drafts of a stream that opens while another is live. Flash Next's HIP facts set it to 3.The
--mtp-draftsrefusal is per family. Flash Next is refused past 7 at--tp 1and past 3 at--tpabove 1, with a message that names both flags. Every other HIP family is refused past 3 with main's message.Issue: related to HIP Flash Next: about 850 small kernels per verify round cost a fifth of decode #1603 (the decode gap on gfx1151). This is not one of that issue's kernel fusions. It comes from the same trace work, which showed that a lone stream's round has verify rows the head does not fill.
Stacked on: hip flashnext: an eight-row verify window, so a lone stream's copies land up to seven tokens a round #2730 (the eight-row window). Part 1 of 1. This change's commits are
135d9c128and8b1c81720(before the merge of main), the changelog number commita43e12597, the three review commits4edc31fd0,de94f186cand7365c00f7, a merge of mainab78aef90(c2456fb8d),ed04933ce, which moves the MTP draft cap's test to its own file, the recipe commitfa6f49385, a merge of mainaaac299d1(f8b2f8f22) and0dde4b661, which moves the--mtp-draftsrefusal's test to its own file. Everything before them is hip flashnext: an eight-row verify window, so a lone stream's copies land up to seven tokens a round #2730.atc overlap --json: run on59aa7cf50(this head less the changelog number and the review commits) against base6c5cef894. The result is incomplete (ATC reports its pull-request, file and time limits). Source files shared with open PRs:core/lanes/config.zig,depth.zig,engine.zig; ATC reports a conflict there);core/lanes/depth.zig,engine.zig);native/hip.zig);CHANGELOG.md.hip flashnext: an eight-row verify window, so a lone stream's copies land up to seven tokens a round #2730 is the base of this stack. The review commits also touch
core/lanes/shared_drafts.zig,core/lanes/open.zigandnative/hip_families.zig.Checks on this head
0dde4b661. The device receipt is oned04933ce. Since then there are three commits: one docs-only commit, a merge of main that changes nothing undercore/lanes,families/flashnext/hiporhip, and a test move. This isa43e12597, the first posted head, plus:4edc31fd0: the shared cap in every lane-core path;de94f186c:--mtp-draftsabove 3 refused with--tpabove 1;7365c00f7: the changelog line and the recipe;c2456fb8d: a merge of mainab78aef90.CHANGELOG.mdandcore/lanes/depth.zigconflicted. The changelog keeps both sides' lines, this PR's first. Indepth.zig, main's one-line plain-streams check and this PR's cap are both kept unchanged; the deep-chain test's import moves fromdepth.zig's test block tolanes.zig's, sodepth.zigstays at 600 lines;ed04933ce: the MTP draft cap's test moves fromflashnext/hip/lanes.zigtoflashnext/hip/mtp_drafts_test.zig, because main's plan-path rounds broughtlanes.zigto 604 lines in the merge;fa6f49385: the recipe names sampled chat among the replies where seven drafts ran slower;f8b2f8f22: a merge of mainaaac299d1. OnlyCHANGELOG.mdconflicted, and it keeps both sides' lines, this PR's first;0dde4b661: the--mtp-draftsrefusal's test moves, unchanged, fromnative/hip.zigtonative/mtp_drafts_hip_test.zig, andcheckGroupbecomespubso the test can reach it. Main's new.ocr = offer.ocr,line had broughthip.zigto 601 lines.aaac299d1and 341 on this head, 0 new.python3 tools/pr_check.py --base origin/main --pr 3053: 0 errors, 0 warnings.zig fmtandgit diff --checkare clean. Every touched file is at most 600 lines.zig build test -Dhip=true --summary all: 898 of 901 steps succeeded, 14,282 of 14,407 tests passed, 122 skipped, 3 failed. Mainaaac299d1on the same Mac: 898 of 901 steps, 14,264 of 14,389 passed, 122 skipped, and the same 3 failures:dashboard_test"dashboard page polls the public stats snapshot without private details" and the twohip.tablesgather tests ("the parked pool's parts copy the same bytes as one thread for every id count" and "a cold call wakes the pool and the same ids gathered again stay on the caller").zig build hip-kernel-teston the receipt head8b1c81720: 20 of 20 steps, 159 of 165 passed and 6 skipped, the same as the base. This PR adds no kernel and no GPU test.--tp 1, and accepts 3 and refuses 4 at--tp 2with the message naming both flags. Aqwen3_5family accepts 3 and refuses 4 with main's text at both.mtp_max_drafts, and three above one rank.0dde4b661when the run finishes.--tpabove 1 (no second box with a link), which is why more than three drafts are refused there;Receipt
One Strix Halo iGPU (gfx1151). The table is from five fresh servers with the page cache dropped before each, in the order base, this with
--mtp-drafts 7, this with--mtp-drafts 7, base, this with the flag unset. The base is #2730's head2e51ea822, and "this" is the receipt head8b1c81720. Tools:bench_concurrent --levels 1,2,4 --reps 3 --alone --serial --temperatures 1.0,0 --prompts tools/bench_prompts.jsonandbench_openai --tokens 256 --reps 3. Cells give aggregate tok/s with tokens a round in brackets, both arms shown.--mtp-drafts 7bench_openaicodebench_openaichat--mtp-drafts 7: the edit workload at one stream, 0.8% at both temperatures (73.5 against 74.1 and 71.6 against 72.2). The arms repeat exactly, so I read it as real. In that workload the copy drafts already fill most of the window, and the deeper head chain adds more draft time than the extra 0.3 tokens a round return. The recipe now says so, so users know when to pick a lower N. The code prompt is level although its tokens a round rise from 2.61 to 2.94. The flag is opt-in and the default is unchanged, so no served cell moves unless the flag is passed.token_sha256equals the base's in all four other arms: 42 of 42 warm, 8 of 8 on a cold four-stream start and 4 of 4 on the edit workload, greedy and sampled. In every cell of every arm, concurrent equals alone and alone equals"draft": false, with 0 unequal and 0 failed.bench_cache --require-cold): 12.02 / 11.99 s on the base, against 12.04 / 12.06 s with seven drafts and 12.10 s unset. Repeat: 9.55 / 9.53 against 9.53 / 9.55 and 9.56 s. The reply hash is equal in all five arms.mtp drafts 3 under 0.3on the base and unset, andmtp drafts 7 under 0.3with the flag.Merged head (
ed04933ce) against main (ab78aef90). One Strix Halo iGPU (gfx1151), both built from fresh trees at the same main commit. Six fresh servers, each with the page cache dropped before it, in this order: main, this, this, main (all with the flag unset), then this twice with--mtp-drafts 7. Each server ran two tools:bench_concurrent --levels 1,2,4 --reps 3 --alone --serial --temperatures 1.0,0 --prompts tools/bench_prompts.json;bench_openai --tokens 256 --reps 3, covering code and chat at temperatures 1.0 and 0.The driver waited until no other process used more than 20% of a core, checked that again every 7 s during each benchmark window (model load excluded), and recorded
uptimeand the busiest processes before and after each arm. No arm was disturbed.bench_concurrent, aggregate tok/s, median of three reps (the reps in brackets), tokens a round after the slash:--mtp-drafts 7, this, two armsbench_openai, decode tok/s (median of three), unset arms in the order main, this, this, main, then the two--mtp-drafts 7arms:--mtp-drafts 7Level. No cell is more than 1% slower than main in both pairs. The two cells past 1% in one pair, two streams sampled (−1.4% and then +1.3%) and two streams greedy (−1.3% and then 0.0%), sit inside main's own spread between its two arms (−1.3% in both).
bench_openaiagrees within 0.4% in every row.Lone stream: this branch has hip flashnext: an eight-row verify window, so a lone stream's copies land up to seven tokens a round #2730's eight-row window, so a lone stream's copy drafts grow past three and tokens a round rise: 3.16 to 3.28 greedy and 2.59 to 2.64 sampled. Greedy is 0.6% to 0.8% faster than main and sampled is 0.2% slower in both pairs. That is inside the 1% gate, and the sampled cell's own reps do not vary.
Tokens. With the flag unset, every request's
token_sha256equals main's in all four arms: 126 of 126 fields (14 distinct requests, with their alone and"draft": falseruns). They also equal the earlier unset arms of7365c00f7.bench_openai's greedy replies have the same hash in every arm, with seven drafts too. The two--mtp-drafts 7arms equal each other and the earlier seven-draft arms, 126 of 126. Concurrent equals alone and alone equals"draft": falsein every cell of all six arms, with 0 unequal and 0 failed. The start-up line readsmtp drafts 3 under 0.3unset andmtp drafts 7 under 0.3with the flag.Seven drafts,
bench_openaisampled chat is slower than three: medians 36.44 and 37.31 against 37.55 to 37.78 unset (0.6% to 3.5% slower), on the same seeded replies (every rep'stoken_shais equal in all six arms); the second rep is about 4% slower in both arms. Chat greedy is +2.9%, code greedy +1.0%, code sampled −0.7% to −1.1%. The flag is opt-in and the default path is unchanged; the recipe now names this case.Raw files for the merged head: https://gist.github.com/gilby/f1284f25978e7190af42f4014be918c3
This head (
7365c00f7). One Strix Halo iGPU (gfx1151). Four fresh servers, all with--mtp-drafts 7and the page cache dropped before each, in the order: this head, the receipt head8b1c81720as a same-box control, this head, the control. Each ranbench_concurrent --levels 1,2,4 --reps 3 --alone --serial --temperatures 1.0,0 --prompts tools/bench_prompts.json, and the first one also ran three staggered-start runs. The driver waited until no other process used more than 20% of a core, logged any that did during the run, and reran that arm;uptimeand the busiest processes were recorded before and after each arm. A first single arm run earlier in the day was disturbed by other work on the box and is not used.Aggregate tok/s, median of three reps (the reps in brackets); tokens a round are the same in all four arms.
token_sha256for every request, its alone run and its"draft": falserun (126 of 126 fields in each arm), and they equal the earlier receipt's--mtp-drafts 7arms. Concurrent equals alone and alone equals"draft": falsein every cell of every arm, with 0 unequal and 0 failed. The start-up line readsmtp drafts 7 under 0.3in all four.--stagger-ms, request Aquicksort-raw, request Brust-reverse-words, 256 tokens each, sampled with fixed seeds and greedy, three reps each,--mtp-drafts 7, first arm above):"draft": falseBoth replies'
token_sha256are the same at every join point, at both temperatures and in every rep.This head with the flag unset, the default path, which the review commits reach (the depth budgets, the allocated shared rows and a stream's first drafts). Four more fresh servers, interleaved the same way (this, control, this, control), no
--mtp-drafts, the samebench_concurrentcommand, and the three staggered starts on the first; the same quiet-box rule, with the disturbance check over the benchmark window. Aggregate tok/s, median of three reps (the reps in brackets).token_sha256for every request, its alone run and its"draft": falserun (126 of 126 fields in each arm), and they equal the base's and the earlier unset arm's in the receipt above. Concurrent equals alone and alone equals"draft": falsein every cell of every arm, with 0 unequal and 0 failed. The start-up line readsmtp drafts 3 under 0.3in all four."draft": false, with A at 9 / 59 / 119 tokens sampled and 10 / 67 / 135 greedy.With #2697's 0.8 confidence floor the deeper chain pays more. This is an earlier run on a second box of the same kind, on this change's first commit merged with #2728 and #2697: 14 fresh servers, base and this interleaved, the same tools. The base has three arms; the others have two arms or one.
--mtp-drafts 7--mtp-drafts 6--mtp-drafts 5--mtp-drafts 4With seven drafts on that stack: code +6.8%, chat +1.6%, one stream sampled +6.3% and greedy +15.6%. Two and four streams stay inside the base's own spread. All 68 request hashes equal the base's in every arm.
--mtp-drafts 4is 1.2% slower than that base on the code prompt (49.6 against 50.1–50.2) and 0.7% slower on chat. 5, 6 and 7 are faster in every one-stream cell there. The edit workload was not run on that stack.New switch? None.
--mtp-draftskeeps its meaning and gains the range 4 to 7 on Flash Next HIP at--tp 1. The default is unchanged (3), and unset is identical to the base, with a test.Boxes it reaches: every HIP device serving Flash Next, and only when
--mtp-draftsabove 3 is passed at--tp 1.Shared code:
core/lanes/depth.zig,config.zig,shared_drafts.zigandopen.zigare the shared lane core. Every new bound is behindshared_mtp_cap, which only Flash Next's HIP facts set. Host tests pin that a zero cap changes nothing in each path.core/lanes/engine.zigsat at the 600-line cap, so the first commit folds one unrelated three-line loop there into one line to make room for the new argument; say if you prefer room made another way.Environment: Zig 0.17.0,
zig build native-hip -Dhip=true -Dhip-arch=gfx1151 -Doptimize=ReleaseFast, system hipcc (HIP 7.1, clang 21), Linux 7.0,unigilby/Qwen3.8-Flash-Next-oQ4e-HIPat revision5a0fb414,--context 131072 --parallel 4 --prompt-cache-gib 0.Licence of new weights: n/a.