Repository navigation
hip flashnext: verify windows run their DeltaNet from the round's plan, with no state snapshot a row - #2728
Conversation
|
Thanks @gilby. The replay matching the per-row snapshot for every kept row is the right check. A few asks before this can land:
|
…bytes once The slots follow the rest of the open path's errdefers, and streamBytes no longer adds the plan: the slots are made for the streams asked for before room is measured (ashhart#2728). Thanks to @gilby.
|
Thanks @ashhart. All six points, on head
Checks on With #2730. A commit that runs the window test at every row count from one to eight sits on the two branches merged together, and its GPU run passes (213 of 219, 6 skipped). The receipt cells on both together are running now; I'll post them here and link them from #2730. Environment: Zig 0.17.0, |
|
#2728 and #2730 together (the item 4 ask on #2730). The two branches merged,
Receipt: one Strix Halo iGPU (gfx1151), four fresh servers with the page cache dropped before each, in the order main (
Same environment as the receipt above. |
|
CI is green on |
|
Thanks @gilby. We read
CI is green on A small follow-up, not for this PR. The note at The test that runs the window at one to eight rows belongs in #2730. That PR lands after this one, so it will have both changes under it.
|
A stream's words, a slot's descriptor and the rows its keep replays, one Plan a cache slot. No kernel launches yet.
…laying the kept rows A window's conv split, gates and delta rule read its rows from the plan, and the keep rebuilds the conv tail from the kept inputs and re-runs the scan over the kept rows, so the state is serial decode's while the round writes no snapshot a row.
The engine makes a plan a cache slot where the shapes are the plan kernels' and sends its words inside a round's graph, a window's linear layer goes through the plan, and a keep replays the kept rows where the plan took it and copies the snapshots where it did not. The plan's bytes count in a stream's admission.
… k, v, state and conv tail At Flash Next's shape: the plan's conv split against today's launch and the kept conv tail for one to four rows, the plan's delta rule y against serial and the state a keep replays for each kept row (a one-row window is its own keep; a longer one leaves the state as the round found it), and two streams each replayed from its own rows. Host: the words' layout, the kept sections' offsets, and the shape guard, which is the only choice now.
…paths The kernel tests now reach the family's model-free side through kernel_reach, so a conv split test can build a real plan and compare today's launch with the plan's q, k, v, gate and beta bit for bit.
…bytes once The slots follow the rest of the open path's errdefers, and streamBytes no longer adds the plan: the slots are made for the streams asked for before room is measured (ashhart#2728). Thanks to @gilby.
… and keep the kernel-test wiring out of hip.zig
50a52d7 to
f70f7e6
Compare
|
Thank you @gilby. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes. |
Figures and scope corrected to the PRs (#2948, #2863, #2934, #2933, #2921, #2906, #2878, #2892, #2851, #2728, #3072, #2915, #2864, #2854 and others), promises of later work removed (#2903, #2875, #1630, #1720), #3074 and #3047/#3046 said as their PRs scope them, the duplicate #2861, #2585, #2321 and #2719 entries and two copies of released 1.2.5 lines removed, #1953's line named for its PR, and a line for #3255. Released sections unchanged.
|
Thanks @gilby! This is in TensorFold 1.2.8, out now: https://github.com/ashhart/TensorFold/releases/tag/v1.2.8
|
What this changes
Flash Next's HIP verify windows ran their 36 DeltaNet layers one row at a time, writing every layer's conv tail and recurrent state after every row of the window, so a keep could copy the snapshot of its last kept row. This moves the window onto the plan path the 27B HIP engine already uses: a window's conv split, gates and delta rule read their rows from the round's plan and leave the state where the window began, and the keep replays the kept rows through the same scan (
planGdnReplay) and rebuilds the conv tail from the kept inputs (tf_plan_keep). The round no longer writes a snapshot a row, which is where the decode time goes.The plan kernels are the existing ones (
plan_ops.convSplit, the plan delta rule,planGdnReplay,tf_plan_keep); no kernel source changes. A one-row window is its own keep: the plan kernel writes its state back in place and the replay stops at one row, as those kernels already do.gdn_replay.zig(new) holds the plan buffers a cache slot, the words a round sends, the window's linear layer on the plan, and the keep. The engine,mixers.zig,forward.zigandmodel.zigtake one-line hooks: the window branch of the linear layer, the words sent inside the round's graph (so a replayed graph sends that round's own), and the keep.The plan path is the default wherever its shape guard accepts (key dim 128, value dim a multiple of 8, value heads a multiple of key heads, conv width 1 to 8; Flash Next's shapes pass). Shapes it refuses keep the per-row path. There is no switch: it is a pure speedup with the same tokens.
The plan's buffers are allocated once at open for the streams asked for and counted there, before room is measured; a stream's admission does not add them again. The per-row snapshot buffers are still allocated, so device memory grows by the plan's few MiB a stream rather than shrinking; dropping the snapshots under the plan path is a separate follow-up.
Prompt chunks are untouched (the prefill branch of the linear layer is unchanged).
Issue: n/a.
Stacked on: main.
atc overlap --jsonagainst main (000b534a2): incomplete (pullFilesLimit,timeLimit); the file shared with most open PRs isCHANGELOG.md. hip flashnext: an eight-row verify window, so a lone stream's copies land up to seven tokens a round #2730 widens the verify window this plan is sized from; the two are tested together (see the comments).Checks on this head
caee124a5: the reviewed commits, a merge of main000b534a2(conflict inCHANGELOG.mdonly), the conv split test with norms and gates on (0c5103dca), the plan's bytes counted once and its slots freed when opening fails (ede2870e2,caee124a5), and the window test through a real plan, fill to keep (b3212e660). The receipt ran on this head.python3 tools/zig/lean_check.py: 345 problems on the base and 345 on this head, 0 new findings.python3 tools/pr_check.py --base origin/main --pr 2728: 0 errors, 0 warnings.zig build test -Dhip=true --summary all: 889 of 891 steps succeeded, 13,645 of 13,759 tests passed, 113 skipped, 1 failed; the failure isdashboard_test"dashboard page polls the public stats snapshot without private details", which fails the same way on main000b534a2.zig build hip-kernel-teston gfx1151: 20 of 20 steps, 213 of 219 tests passed, 6 skipped, 0 failed (main: 159 of 165).zig build hip-kernel-contract:SUMMARY gfx1151 159 kernels: 159 pass gates 1-4, 0 fail, 2 known findings; 41 controls: 41 red as planted, 0 missed; 173 limit problems, the same line as main000b534a2on this install (this PR registers no kernel).gdn_plan.zig: the plan's conv split gives the per-row launch's q, k and v bits at one to four rows; the plan's delta rule with the first n rows kept leaves the state the per-row snapshot of row n holds; two streams' plans keep their own rows.gdn_window.zig, through a realgdn_replay.Plan:fillwrites every slot's words and the linear layer's descriptor where the kernels read them; the conv split on the plan with norms and gates on writes the per-row launch's q, k, v, gate and beta; a window through its plan,fill→send→ conv split → delta rule →keep(n)over two layers (one linear, one zeroed sparse layer), rows 1–4 and every n, equals the per-row window in y, the state and the conv tail. Planted control: withPlan.keepkeeping one row fewer, the last test goes red (slices differ).Receipt
000b534a2) against branch, arms in the order main, this, this, main, a fresh server an arm with the page cache dropped (tools/bench_concurrent.py --levels 1,2,4 --reps 3 --prompts tools/bench_prompts.json --alone --serial --temperatures 1.0,0, 16 distinct prompts, 256 tokens;tools/bench_openai.py --tokens 256 --reps 3;tools/bench_cache.py --require-cold). Both arms of each side are shown:bench_openaicode 46.31 / 46.27 vs 47.32 / 47.33 tok/s (+2.2%), chat 40.14 / 40.12 vs 40.98 / 41.02 (+2.2%).000b534a2plus this change, Zig 0.17.0,zig build native-hip -Dhip=true -Dhip-arch=gfx1151 -Doptimize=ReleaseFastwith the system hipcc (HIP 7.1, clang 21), one Strix Halo iGPU (gfx1151), Linux 7.0,unigilby/Qwen3.8-Flash-Next-oQ4e-HIP(revision5a0fb414),--context 131072 --parallel 4 --prompt-cache-gib 0, MTP 3 drafts.token_sha256equals main's in each arm (42 of 42 warm, 8 of 8 cold);--alone --serialequal in every cell of every arm (alone 3 / 6 / 12 equal at 1 / 2 / 4 streams, serial 4 / 4: drafted equals"draft": false, concurrent equals solo), 0 unequal, 0 failed; the cold prompt'stoken_sha25625c0728f9043in every arm. The kernel tests above hold the replayed state bit-equal to the per-row snapshots.Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (--alone --serialabove. Resume was not run separately: the prompt cache keeps prefill states, and prompt chunks do not take this path.)