Repository navigation
Conversation
|
Coordination before publishing the EXL3 stack update (#1441–#1446, #1522): its new, unpublished capture-admission correction touches This is separate from your paged head streams and shared verify scores; I am not implementing or replacing those. The current textual conflict is in |
|
@BlivionIaG thanks for flagging it first. This stack's edits in those two files are one line each:
So both are pure admission and key changes, and they compose with your guard. Your host-planning check is one more early |
|
Thanks @jschmied, this is the cell we asked for on #2444: main and F2 + (c) admit the same streams at every limit from 76 to 79 GB, the kit rows equal main's in all three cells, and at 256 pages the pooled A-B-B-A means sit within 0.25% of main, with no cell keeping one sign across rounds. Thanks as well for the correction on the 3 pages. The pieces pack to 48 pages, the same as main's run, so nobody who turns this on loses a stream, and a lane can now open on any free pages instead of a run. That's enough for us. As we said on #2444, 8-head groups only come in if a measured cell shows the whole-layer pad (18.53 MiB a lane) costing a stream. I read the head through, and it holds up:
@BlivionIaG's guard and these two lines compose as you described. What's left before this and #2898 go in together:
On overlap:
|
adae273 to
25108af
Compare
|
@ashhart thanks for the read-through. All four are done; heads are #2898 d432157 and this PR 25108af, both on main 3885dce.
Host |
kv_pieces packs a family's state pieces first-fit decreasing on whole pages (each piece inside one page, 256-byte aligned) and checks a map for overlap. Lease.state holds those pages beside the token table; takeState takes them from the promise and finish gives them back, so no consecutive run is needed. Layout.pieces counts them wherever a run counted (need, plan, holds), and Layout.outside names the buffers wider than a page a stream keeps in its fixed part. The load report shows the pieces, their pages and pad, and that fixed part.
…aft as pieces Under TF_FLASHNEXT_KV_PAGES, TF_FLASHNEXT_STATE_PAGES=1 maps each lane's DeltaNet states, conv tails, n-gram tail and its MTP head's spares, saved rows, small buffers and pending-row log onto pieces of whole pages from its lease (no claimRun). The head's streams and window scores, wider than a page, are a block of the lane's own outside the pool, charged in its fixed part. The saved n-gram tail and rows get their own pointers, which the rollback key hashes. Unset, the run and every buffer stay as before.
… 0 or 1 Read through environment.value, so the CUDA backend refuses any other text by name at startup, before a model loads; words, 2 and padded text are refused.
…allest page that holds it TF_FLASHNEXT_STATE_PAGES's PieceOverPage said pages need 128 rows or more, but the floor is lower and depends on the head: the head's attention layer widens a page, so 104 rows hold a 3 MiB DeltaNet state with the head and 112 without. The load now logs the page's bytes, the largest piece and the smallest TF_FLASHNEXT_KV_PAGES that holds it (smallestRows, from the same piece sizes the packer places), and the refusal points at that line, as Qwen's does.
…ith the other Flash Next settings As a .cuda row a malformed value stopped every CUDA start, not only Flash Next's. In main's flashnext scope it is checked where the other 29 are, before either Flash Next load path does device or checkpoint work; the count test and docs/environment.md move to 30 settings, 21 switches.
…read through pages by row
…e lease's pages, nothing outside the pool
…er, then each row to its piece
…he rows its verify wrote, no per-row copies
25108af to
f0fe4b7
Compare
|
Thanks @jschmied. f0fe4b7 is 25108af restacked on main 6c5cef8. The only difference is #2939's TooMany text in Every ask is met, so we'll land #2898 and this PR together, #2898 first. #2939 carries the same core commit, so whichever of them lands second drops it. These are the changelog lines we'll use:
Thanks also to @BlivionIaG for flagging the
|
What this changes
Stacked on #2898 (F2, head 59ba8b4). The two land together, as ashhart asked in #2444 (comments 6796185011 and
6824699204). This PR's first five commits are #2898's; the six after them are (c).
On #2898 with
TF_FLASHNEXT_STATE_PAGES=1, two of the MTP head's buffers are each one matrix wider than a page:streams(31.5 MB at 1536-row chunks) and the window'slogits(15.9 MB). They stayed outside the pool as a lane's ownblock (option (a)), and at a 77 GB limit that block cost a concurrent stream (fit table on #2444, comment 6796148352).
This PR removes the block:
streamslives in the lease as 48 pieces of 32 rows (20,480 B a row), and the head reads it by row through the pagetable. Block 0 holds rows 0..31, so every window (at most 32 rows) reads and writes there exactly as before. A prompt
pass writes a chunk's rows block by block. A head prefill piece whose rows cross a block is first gathered into the
pass's
s.hrows. The piece doesn't touch those rows until after it has read its input.logits: a paged draft keeps no scores of its own. Its window's scores stay where the verify pass already writes them,the lanes' shared 64-row buffer (
verifyMany'sout, which main copies each lane's rows from). The draws read themthere with the same launches as main, before the next verify reuses the buffer. A lone window (a round with one
drafting stream) scores into the same buffer through
Ops.verifyIn. The plain single-streamverifyrefuses a pageddraft by name (
FlashNextPagedDraftVerify), which keeps the rank path and the tests honest.An idle pool lane now holds nothing outside the pool, and the layout's fixed part is main's again ("0 MiB fixed each" in
the load line, the same as main).
Saving 3 pages a stream isn't possible: main's 48 is the floor, and pieces reach it. With the head and pending rows
(the default), a lane's state plus head is 374,645,504 B, which is 47.54 pages of 7,880,704 B (256-row pages with the
head). Main's run is 48 pages. Pieces pack to 48 as well: 271 pieces, 342.22 MiB on 48 pages, 18.53 MiB of padding. That
holds even though the 15.9 MB of scores no longer live in the lane, because the 3 MiB DeltaNet states leave gaps the
rest can't fill, so first-fit-decreasing still needs 48 pages. With TF_FLASHNEXT_KEEP_LATE=0 it is 234 pieces on 41, the same as the run's 41. So F2 + (c) ties main on
bytes. What pieces add is the thing #2898 is for: a lane opens on any free pages, with no consecutive run.
No kernel changed. No file under
zig/kernels,zig/buildorbuild.zigdiffers from main. The 114 fatbinsembedded in the served binary are byte-identical to main 3885dce's, in the same order (md5 of each blob). The kernel
contract rows are unchanged, so no new contract-runner rows or limits apply.
Unset (
TF_FLASHNEXT_STATE_PAGESoff), every new branch is skipped (no blocks,logit_pagedfalse), and windows, drawsand prompt passes run main's code. The kit and A-B-B-A rows below show the bits and the speed both ways.
Core vs family:
Rows: a matrix as row blocks (sizes, a run that never crosses a block, a row's address);Placed: those blocks placed, walked from any rowcore/kv_pieces.zigcore/kv_pieces_test.zigHooks.stream_rows(aPlaced); per-block stream copies at the pass's write-backcuda_batch_state.zig,cuda_batch_head.zig,cuda_prompt.zig,cuda_prompt_windows.zigDraft.pieceSizesadds the blocks,initPiecestakes their addresses,streamAt,window_scores; the lane's wide block is gonecuda_mtp_draft.zig,cuda_kv_lanes.zigprompt_last,verifyIn; draws readwindow_scores; plainverifyon a paged draft refused by namecuda_mtp_ops.zig,cuda_mtp_draw.zig,cuda_lanes_verify.zigCode sharing, checked first: no Qwen, GLM or core path reads a buffer wider than a page through pages. Their pool lanes
hold only whole-layer state. So
Rows/Placedgo in core next to the packer, and Q3 (or a Qwen head, if one comes) canuse them as is. Metal has no Flash Next pool.
on flashnext: pool lanes keep their states and draft as pieces on whole pages, no run (#2444 P3, F2) #2898 (6825628750): it carries the reworded
kv_pieces_test.zigheader, Q3'stakeStatefix (highest free ids,free stack's order) and qwen3_5 cuda: a pool lane's state as pieces on any whole pages of its lease, no run (#2444 P3, Q3) #2939's
servedspare-lane line innative/cuda_kv_pool.zig(off for Flash Next, which setsneither
all_on_pagesnorengine_paged). So this PR's former last commit (thetakeStatefix) is gone, its contentalready in the first. flashnext: pool lanes keep their states and draft as pieces on whole pages, no run (#2444 P3, F2) #2898's other commits applied clean. Here
cuda_mtp_draw.zigconflicted in two commits (a672853 and293f9b4; main's c6a3377 / 80e0fd7 added the non-finite checks, dfd9ac5 moved
on()to the environmenttable). Resolved by keeping main's checks,
pendingrows andon(), and our row reads (window_scoresfor a pageddraft) inside them. Every other file's added and removed lines are the same as on adae273. No layout, packer or
ledger code changed in the rebase (
core/kv_pieces.zig,core/kv_pool.zig,cuda_kv_lanes.zig,cuda_mtp_draft.zigpiece sizes), so the fit table below, measured on adae273, still stands. The kit and A-B-B-A rows are new, on this
head.
Checks on this head
qwen3_5 cuda: a pool lane's state as pieces on any whole pages of its lease, no run (#2444 P3, Q3) #2939's updated core commit fac7b74 (same patch-id 02a90f2d91d9): its only change from 7e2957d is the TooMany text
in
native/cuda_kv_pool.zig, "state pages" for "state runs", and its test.kv_pool.zig,kv_pool_state_test.zigandcuda_kv_pool.zigare byte-equal to qwen3_5 cuda: a pool lane's state as pieces on any whole pages of its lease, no run (#2444 P3, Q3) #2939's head 250994a.zig/src/core/kv_pool.zigandkv_pool_state_test.zigare byte-equal to qwen3_5 cuda: a pool lane's state as pieces on any whole pages of its lease, no run (#2444 P3, Q3) #2939's head 250994a (checked withcmp),takeStatefix included.zig build -Dfixed-cpu test test-golden test-kernel-contract test-server-security test-pull-security fuzz-cortheon check-release-paths: 724/724 steps.zig build check-native-cuda-host compile-cuda-tests: 16/16.python3 tools/pr_check.py: 0 errors, lean 345 vs 345 (0 new); its one warning is the CHANGELOG line, which is in this body forashhart to place.
zig fmt --check,git diff --check: clean.zig build test-kv-pool test-cuda-host: 396/396 steps.row shas equal main's: 149 rows, 137 keys, 0 differ in each cell (main's rows are from the same day's run on 3885dce).
The 114 fatbins are byte-identical to main's. The A-B-B-A rows below are from e89fb3c1e, the rebased head before
ashhart's flashnext: pool lanes keep their states and draft as pieces on whole pages, no run (#2444 P3, F2) #2898/flashnext: the MTP head's streams by row through the lease's pages, no per-lane block outside the pool (#2444 P3, F2 part c) #2992 asks. 25108af differs from it only in the test header, the PieceOverPage refusal and its load
line, the switch's environment scope, and qwen3_5 cuda: a pool lane's state as pieces on any whole pages of its lease, no run (#2444 P3, Q3) #2939's spare-lane line (off for Flash Next). None of these is on the served
path of a page size that loads.
main's 10 new commits touch no Flash Next, KV pool or environment file. So the kit, fit and A-B-B-A rows carry over.
kv_pieces_test.zig: rows by block; a placed walk, with two planted controls that must fail and do (swappedblocks; a walk that lost its offset).
cuda_kv_pieces_test.zig: Flash Next 271 pieces on 48 pages with pending rows, equal to the run's 48; 234 on 41without, equal to 41; nothing outside, fixed bytes equal to main's. Two lanes with the head place every stream block
inside their own state pages with no overlap, and each of the 1536 rows' addresses is in its block. A lone verify on a
paged draft is refused by name.
cuda_kv_pieces_test.zig(flashnext: pool lanes keep their states and draft as pieces on whole pages, no run (#2444 P3, F2) #2898's last commit): the smallest page that holds the largest piece is 104 rows with thehead and 112 without; that page loads and one step less refuses. Planted red: smallestRows one step short (red by name, "expected 104, found 100", 1 test failed).
cuda_lanes_verify_test.zig: a paged draft's lone window goes toverifyInwith the lanes' buffer, a group stilltakes one shared pass, and with ranks the lone window keeps the plain verify (which refuses).
FlashNextPagedDraftVerifyfiringin 4 of 76 concurrent cases. That was the refusal catching a lone window I had missed, not a wrong bit, and
verifyInfixes it.
Served gates (GB10, Qwen3.8-Flash-Next-FP4, --context 16384 --parallel 4)
Conformance kit (
tools/conformance.py, flashnext-cuda profile with the cancel/evict blocks), page cache dropped beforeevery server, run after the rebase: main 3885dce (md5 82a75428) and the rebased head e89fb3c1e (a4381ec5), in the
same job. Cells: pages off;
TF_FLASHNEXT_KV_PAGES=256; and 256 +TF_FLASHNEXT_STATE_PAGES=1(the branch only). All 7 gates pass in everycell, main's included (35 of 35): drafted == plain (448 of 724 drafts accepted), repeat == first, concurrent == solo at
1/2/4/8 (drafted and plain, greedy and seeded), resumed == fresh, refusals, cancel (census back to idle), evict. Every kit
row's sha equals main's in the same paging: 149 rows, 137 keys, 0 differ, for off, 256, and 256 with pieces.
The final head 25108af (b9788991) reran the three branch cells: 21/21 gates, 0 of 137 keys differ from main's rows.
Load line, pieces cell: "state, in the pool as pieces: 271 pieces, 342.22 MiB on 48 pages a lane, 18.53 MiB padding
(5.1 %)" and "0 MiB fixed each". Main's is "state run, in the pool: 357.39 MiB on 48 pages", also with "0 MiB fixed
each".
Fit (ashhart's landing cell): admitted streams by memory limit
TENSORFOLD_CUDA_MEMORY_LIMIT_GB=L, --parallel auto (8 asked), --context 16384, 256-row pages, --prompt-cache-gib 0.
Main 33ada77 against this PR's pre-rebase head adae273 (md5 376329c7) with TF_FLASHNEXT_STATE_PAGES=1. It stands for
this head: the rebase changed no layout, packer or ledger code, and the load line above is the same (271 pieces on 48
pages, 0 MiB fixed each). A 16k-window stream needs 65 pages plus its state (48
both ways).
Main and pieces admit the same streams at every limit, and the pool has the same pages. #2898 alone lost a stream at 77
GB (16k streams 5 vs 4) and one at 76/76.5/77 GB with 4k prompts. That loss is gone.
Speed: A-B-B-A, page cache dropped before every server
Rerun after the rebase: main 3885dce (82a75428) against the rebased head e89fb3c1e (a4381ec5). c1 = 3 x 256 tokens
solo, c4 = 3 rounds of 4 x 256, TTFT = two 8.8k-token prompts. Two jobs: one ran A-B-B-A x2 at both pagings, the second
another A-B-B-A x2 at pages 256. Pooled: 4 rounds at 256, 2 off. Token shas are identical across every arm of each
paging, and solo == c4 in every arm.
Pooled at pages 256: c1 +0.01 %, c4 -0.05 %, TTFT -0.23 % / -0.47 %. Off: c1 +0.10 %, c4 +0.04 %, TTFT -0.23 % / -0.26 %.
No cell at 256 keeps one sign across the rounds. The c4 rounds go -0.55 / -0.13 / -0.15 / +0.61 %; the low arm is 80.6 in
one pieces arm and 80.8 in one main arm, the same dip that hit either arm before the rebase (adae273: pooled c1 -0.05 %,
c4 -0.07 %).
An earlier head that copied each score row to its own piece lost 0.69 % at c=4 in both rounds; this head does not copy.
Proposed CHANGELOG line (for ashhart to place): "Flash Next: with TF_FLASHNEXT_STATE_PAGES, the MTP head's streams live in
the stream's pages too and its window scores stay in the shared verify buffer, so a pool lane holds nothing outside the
pool (#2444)."