Repository navigation
Conversation
|
Thanks @wojo. This is the part that makes the split usable for serving, and it reads correctly. The home's copy alone holds the latent rows, MTP included, while every rank leases the full size. A restore reads latent rows only from that copy, either back at home or carried to the new owner through the room, with a third rank posting nothing. The disk image keeps its offsets and routes the latent runs to the home, and the words refuse another layout by name. A resume never lands in a twinned stream, and part 3's fetch can't fire during a twin. With the switch unset, the walk, plan, image and lanes changes read byte-identical. This part also removes the On your questions:
A few things:
This follows parts 2 and 3. Once the rows are in, we'll run main, unset and set on our two GB10s, with the prompt cache on.
|
|
Restacked on main now that #2205 has landed, with #2415 below it: parts 2, 3 and 4 are From the asks on #2935:
Device rows, 12 Nov (two GB10s, TP=2, bf16 KV, MTP drafts,
Host on the three heads: Not done: part 2 and part 3 were not run on GPUs on their own today (part 2 alone was last measured at |
|
Thanks @wojo for the restack and the rows on The code points from 11 Nov are still open on
One more from your table: set decodes at 33.00 tok/s against 33.97 unset, and 4 concurrent at 46.13 against 47.56. Both are about 3% lower, where part 2's head showed about 1% and none. If that holds when you rerun, please say so in the setting's row beside the cold-prompt cost. #2935 needs nothing more. The three parts land together once these are in, after #1411 and #2415. Then we'll run main, unset and set on our two GB10s with the prompt cache on.
|
|
Thanks for the review. Each point below is in 1. Twins borrow the prompt's pages. 2. Poisoned twins. 3. The give-back. 4. Three ranks. 5. Docs. 6. Cold first token, each part's own build. Two GB10s at TP=2, the full checkpoint,
"unset" is part 4's build with the setting off; main Twins add almost nothing over part 3 here: 0.2% at 17k and 28k, 1.6% at 93k. I checked whether they engage when serving by reading the RoCE port counters around two cold 28k prompts on each build:
The twins do engage: the fetch's broadcast (about 2 GiB a port, one way) is gone. But that broadcast was a small share, and both split builds move about a third more bytes than unset in both directions. The one difference I can find in the code is 7. Decode. The drop holds on the rerun. One stream, 1,024 tokens, three runs: 32.2 / 33.2 / 33.2 tokens a second set against 32.9 / 33.9 / 34.0 unset (about 2.5%); four streams, 46.7 aggregate against 47.6 (about 2%). The settings row now says so, with the prompt numbers above, in this part's new tip commit. Other device rows on this build: Host on |
|
A correction to point 6 of my last reply, and the fix, in The cold prompt gap was mostly pieces, not pairing. Under the split,
So pairing was about 3.6 of the 16 points, and the lost pieces (the split reduce plus the overlap) were the rest. The change. Device rows on
Replies hash the same on both. Decode stays about 2% slower, and I don't have a cheap fix for it. A section profile puts about 0.9 ms of the 1.4 ms a verify round adds in the 11 MLA layers: the owner's second attention pass for the other rank's heads, and two more sync points a layer. Two prototypes changed nothing measurable against a same-build control: sending a window's owner exchange as two one-shot RoCE all-gathers, and starting the owner's own pass before the queries arrive. |
e816b16 to
4b44f78
Compare
|
Restacked parts 2-4 onto main Conflicts, all next to main's new
Each commit's own diff matches its earlier version, with two exceptions:
Host, on each head:
Device rows on
Replies hash the same on both. |
…re each byte sits in its lease The rows store's view of a GLM kept state, without the store: the words (format, position, head rows, ring bases, KDA copy bits), the KDA prefix, each DSA layer's latent rows, scales and whole pools by position, its rings and pools in progress as the state's own, the head's caches at a whole verify window (rows past the head's own read as zeros), then the tail. Copies run a page of the lease at a time. Nothing is wired to --learn-conversations yet.
…state on disk Rank 0's engine thread leases, fills, reads and seals each rank's copy through the follow link (cuda_disk_ranks: blank, put, take, seal, prefixes). The store's image is rank 0's image followed by each other rank's prefix, the one part a rank holds alone; the rest are the same bytes on every rank, checked at start by the identity probe. A blank state is refused by restore until its seal. A blank whose disk step fails on the link gives rank 0's lease back, and its id goes on the dropped list, so the next step drops it on any rank the step reached; the id is not used again. An NCCL failure there already ends every rank (the liveness lines), but a CUDA error in the step's launch or upload returns to the disk read, which tries one shorter state and then prefills while the server goes on. cuda_disk_ranks_test.zig runs the steps on host memory with a fake link: that failed blank, a blank state refusing bytes past its words and front until sealed (DiskOrder), each BadStep a follower refuses before its states or the link, and each other rank's shared bytes taken past its prefix and compared to rank 0's (RanksDiffer, in ranks.same, which the probe calls). attach's refusals of the shared store and of more ranks than prefix_buf holds (KvShareWithLearnConversations, TooManyRanks) have their own test in cuda_glm_disk.zig. link and prefix_buf stay process-wide: the process serves one model. With TF_GLM_KV_LIVE_HALF=1 a keep holds one KDA half and the disk image both, so GLM refuses --learn-conversations with it by name at startup (cuda_refusal.zig); the test that pinned GLM refusing --learn-conversations outright now pins that it takes the flag alone and refuses it beside live-half keeps. Rank 0's link becomes the disk steps' link in `follow.Link.lead` (which also sets whether the other ranks begin asleep), at TP only, and `Link.deinit` clears it, so cuda_native.zig stays at its 600 lines. cuda_disk_ranks_test.zig: one rank leaves the steps local, two set the link, another link's deinit leaves it, its own clears it; the test fails with the clear removed.
…sk state into that storage A family whose kept states live in preallocated storage it reuses (GLM's snapshot pages on CUDA) and cannot trim never had a disk read chosen: the selection counted its spare storage as fixed and its kept states as not reclaimable, so no state ever fit. Such a family's room for a read is now its spare storage plus what evicting unborrowed states frees, and a read is ready once the family says its spare storage takes the state, as a keep's admission already asks. Families with trim, or without reused storage, are unchanged.
…t, as before The disk read's early admission for a family whose spare storage takes the state now applies only to a family without trim, as the selection's capacity already does. A family with trim, such as Flash Next's or Qwen3.8-27B's pools on Metal, makes room for a read by trimming its spare storage and evicting, exactly as it did before, and a new test pins that its read stays within the budget when its reuses answers yes.
…he state's rows An index ring based at `base` holds the rows from `base` up to the state's position, and the image keeps at most a ring's rows of each. The seal now checks the words' main base against the state's position and the head's base against its rows, so an image whose words name a ring that starts past them, or too far back to be held, is refused before the blank state is made whole.
…/O worker GLM's store uses The write-behind gave GLM's store its own I/O worker, while the grid test wrote on the test's thread (sync_writes). It now runs on each, drained by write.drain, so a worker path that loses the grid's chunk starts fails the test.
…rn-conversations alone keeps its states Since ashhart#1413 CUDA serves --spill-gib and --checkpoint-slots, and --learn-conversations admits GLM, so --learn-conversations --spill-gib (or --checkpoint-slots) opened GLM and ignored the flag: its disk store writes every rank's part between passes and stages nothing at eviction. GLM now refuses both by name. Its store's spill stays off, since an eviction would gather on rank 0 mid-pass while the other ranks wait in the gate's all-reduce. docs/api.md lists every CUDA family that keeps states on disk.
…ts passes for every rank's heads, outputs back cuda_owner_attn.run takes one DSA layer's segments (one stream's rows each) on rank r of W, each with the rank that holds its latent rows or none. In one NCCL group every other rank sends a segment's owner its absorbed queries; the owner attends its own heads and then each other rank's, and in a second group sends each rank its outputs, which land in that rank's own output rows. A rank without a segment's rows writes them and picks its token lists alone. Every rank sees the same segments and owners, so the sends between any two ranks meet the receives in order; a layer whose segments have no owner posts nothing and attends as before. Room holds each other rank's queries and the outputs made for it, a slot a rank at the widest share's heads: 256 MiB at TP=2 for 4,096 rows. body, lastRow and shared are the forward's entry points (one sequence, a prompt MTP chunk's last row, a shared window); without a room, or with in-process ranks, a stream on owner-only rows is refused by name rather than attended where its rows are not. Forward gains the room's field, unset: nothing routes through it yet. Host tests: 2 and 3 ranks on threads, groups as barriers that pair every send with its receive: each rank's outputs equal its unsplit ones, and a rank reads a segment's rows only where they are, over fixed windows and 200 random ones of up to 16 segments; the room's slots at TP=3's 22/21/21 heads; and three mutants caught.
…l they carry rows only their owner holds With each stream's latent rows on its owner alone (the next commits), a kept state copied from a stream's pages would hold every other rank's trash. As agreed in ashhart#1431, kept states are refused with the switch on until part 4 takes them: the server's options say whether it keeps prompt states (a prompt cache, which is on unless --prompt-cache-gib 0, or --learn-conversations' disk tier), and every rank refuses that before NCCL with KvSplitKeeps, naming both flags. snap_save refuses the same as a backstop. Off, nothing changes. The split's refusals are explained in cuda_start.explain, which cuda_native.explain falls back to, so cuda_native.zig stays within the lean rule's 600 lines. The rule is `cuda_start.keeping(split, keeps)`, beside `sharing`, which `openRank` calls. test-glm-startup-host: refused with the split and kept states, allowed with either alone; it fails with the rule emptied. test-native-options-host pins what counts as kept: no --prompt-cache-gib keeps (the cache is on by default), 0 does not, any size does, and --learn-conversations keeps; it fails with the default or the learn term changed.
…s for every rank's heads, prompt chunks whole A paged sequence records the rank its lease's class names (State.owner, set where its pages are mapped on every rank, cleared at close), and every DSA block on it goes through cuda_owner_attn: the forward's mixer (prompt chunks, decode and verify windows), the shared windows (several streams' verify windows or whole prompts) and the MTP head (a prompt chunk's last row, a chain's rows; rows committed alone only write). A sequence with no owner (a whole window, or the switch off) runs dsa_block as before. Every rank makes the exchange room at start, before its memory is read, for the prompt buffers' rows. The owners' exchanges run on the main stream, so with the switch on a prompt chunk's exchanges are whole, not in pieces on the side stream, and the engine makes no paired-chunk twin: the prompt history fetch of part 3 is where pieces come back. Engine.checkGathers and copy move to cuda_engine_pick to keep the engine's file in the lean rule. In this commit every rank still holds every latent row, so the owner's attention reads what each rank would have: replies are unchanged. The next commit keeps the rows on their owner alone. Off, every forward is as before (no room, no owner, the same pieces and pairs).
…t rows and a trash page, 6,912 bytes a token The latent region holds this rank's class alone, page id at id / W, then one trash page; the pooled keys stay every page's on every rank. A stream's latent table holds those places on its owner and the trash page on every other rank, so a rank without the rows writes them to its trash and never reads them (its attention is the owner's). The table is written in Pages.lat_ids, a table's worth a slot, and uploaded from there as the pooled ids are, so its source outlives the upload. TF_KV_POISON fills a page's latent rows only where they are held. The layout costs a token a rank its pooled keys and a W-th of the latent rows: 6,912 bytes at TP=2 (13,056 unsplit, 1.89 times the tokens) and 4,864 at TP=3, so the server's plan funds that many more pages. Pages.init gives a page of each class to the trash page, alike on every rank from the planned count (cuda_pages_owner.funded), so the arena stays inside what the plan counted; the start-up line gives both counts and the bytes a token. Tests: the cost, region and funding arithmetic; the split fixture's pools planned with the trash page's share; a stream's latent table its owner's places here and the trash for another rank's class, on rank 0 and on a follower mapping rank 0's ids; a table past 32 pages uploaded on a driver that reads each source only after the stack is clobbered; the NaN fill of only the held latent pages; the arena's bytes; and the pool step's planned count funding the same pages on the follower.
…as unsplit and set aside first, the pages the rest With a rank holding only its class's latent rows, a window costs about half the bytes, so the plan's whole windows before an unasked cache grew from two to four at a 1M window and 8 streams, and the cache got what they left: 0.40 GiB where the unsplit server keeps 1.90 GiB. As agreed in ashhart#1431, the cache gets its budget first and the pool the rest. The layout names the unsplit token (`whole_bytes`, GLM's page over its rows under owner classes). `budgetOf` plans an unasked cache over that layout, exactly the cache the unsplit server would keep, and marks it asked, so it comes before every window but the engine's own, as --prompt-cache-gib would. A named cache and every one-class layout plan as before. kv_pool.zig's two test blocks become one, which keeps it at 600 lines. Where that cache would leave a class short of a window (the 1104 shape on 18.2 GiB of room: classes of 1,941 pages, a 1M window takes 2,049), `kv_pool.fitClasses` lowers it only as far as every class holds one, so the server starts instead of refusing (ClassTooSmall). A named cache is never lowered: its refusal now names --prompt-cache-gib too. The plan's class check counts the trash page (a page a class gives way to it, `funded`): planned at a bare window, a class held 2,048 pages at start where the stream takes 2,049, and the engine refused it after the plan passed. GLM at TP=2, a 1M window, 8 streams and 30 GiB of room: unsplit 1.90 GiB and 4,098 pages (2.10M tokens); split before, 0.40 GiB and 8,196 pages; split now, 1.90 GiB and 7,740 pages (3.96M tokens). The test in test-cuda-cache-budget-host pins these, and fails (0.40 GiB) with the old plan; its second test pins the lowered cache (each class a window and the trash page, a page more of cache refused), and fails without `fitClasses`. Kept states stay refused with the split on until part 4 (KvSplitKeeps), so the server runs this split with --prompt-cache-gib 0 for now; this plan is what part 4's cache gets.
…on its owner alone, rank 0's table checked The check's split mode now runs on owner-only rows by construction: each prompt's paged stream, on owner k % W, is attended by its owner for both ranks' heads, prompts, windows and MTP drafts alike, and compared byte for byte with its twin on a whole window, which every rank attends as before. Rank 0 also reads back the paged stream's latent table after each open: its region's places when it owns the stream, the trash page alone when rank 1 does. It fails if no layer attended owner-computed, and reports rank 0's exchanged layers, rows and bytes. The new readback has its routing case among cuda_readback_sites.
…t 2 of ashhart#1431) The changelog says what the switch now saves, at what prompt cost, and that kept states are refused with it on. The contract notes that the owner-only region and the owner's passes for other ranks' heads are the same launches with other tables and pointers, so part 2 adds no case; glm-paged-check covers it. The setting's rows (`profile.zig`'s effect and evidence, `docs/environment.md`) describe this part: latent rows on the owner alone, about 1.89 times the tokens at TP=2, kept prompt states refused.
…teps with their indexer rows split The served ladder (g34) went wrong with TF_GLM_KV_SPLIT past 11,220 tokens while the check stopped at 5,000 and prefilled whole, without TF_GLM_INDEX_SPLIT's split chunks. The window grows to 32,768, the lengths add 11,220, 16,810 and 30,001, and those go chunk by chunk through Leader.chunk as the lanes send them, on both the paged stream and its twin.
…s against the exchange room before the first group The room check ran in the exchange itself (Gpu.move), on the rank that takes a segment, inside the first NCCL group: a segment past the room would have stopped that rank with its peers already waiting in the group. Now Room.holds checks every owned segment on every rank, from host values alone, before the group begins, so an overflow is the same refusal (KvSplitRoom) on every rank. A whole window takes no room and is never refused. The check in move stays as the backstop. Test (the GLM set in test-cuda-host): segments to the room's edge pass, one a row past it and one starting at the edge are refused, and a whole window past it is not.
…from the owner, each rank its own heads A prompt chunk of one sequence on owner-only rows (Buffers.prefill, State.owner set) no longer sends its queries to the owner: cuda_owner_fetch.body has the owner pack the layer's latent rows [0, pos) from its pages into the exchange room with tf_glm_kvh_rows, broadcasts them (ncclBroadcast, in place on the owner, none for a stream's first chunk), and every rank attends its own heads: the owner on its pages, the others on the room with no table, their own chunk rows written there and their pooled and index keys to their caches as before. Nothing comes back. It fetches while the chunk's rows to its end fit the room (262,144 rows at TP=2, where the two moves cost the same bytes on a link); past it, and for decode and verify windows, shared windows and the MTP head, part 2's owner exchange runs unchanged. Every rank decides from host values alone. The room is part 2's, so the fetch takes no memory of its own. owner_attn.Stats counts the fetching layers, rows and bytes.
…l its unsplit ones, at 2 and 3 ranks test-glm-owner-fetch-host (in the GLM host set) runs cuda_owner_fetch.run on host ranks: the owner's pages out of order and several layers apart, every other rank's latent pages poisoned and mapped to a trash page, two layers sharing one room, chunks across the dense limit and the pages' edges. Every rank's outputs equal its unsplit ones, dense and sparse; no rank reads a poisoned row; the owner never reads the room and the others never their pages. 200 random streams, the decision at the room's edge, and four mutants caught: no broadcast, the pack a row short, the owner attending the room, another rank attending its pages. A receive left out fails it, 8 of 10.
…fetches, and fails if no chunk fetched Its chunked prompts to 30,001 tokens fetch every chunk past a stream's first, so a split run with no fetching layer did not reach the fetch. The receipt prints rank 0's fetching layers, rows and bytes next to the owner exchange's.
… (part 3 of ashhart#1431) The changelog says what a prompt chunk now does with the switch on and where it still takes the owner's exchange. The contract notes that the fetch's pack is tf_glm_kvh_rows (its rows case) and its attention the same launches over rows in order, so part 3 adds no case; glm-paged-check covers it.
…ts history's broadcast moves no more bytes than its exchange `fetches` checked only that the chunk's rows fit the room. The fetch broadcasts every earlier row of the layer to each other rank; the exchange moves the chunk's rows of the other ranks' heads there and back. At TP=2 with the default 2,048-row chunks the two meet at 131,072 rows, the room, so a cold prompt's full chunks fetch as before. A short chunk on a long history did not: a 200-row follow-up at 60,000 rows broadcast 58.6 MiB a layer where the exchange moves 12.5 MiB, and a 1-row tail at 100,000 rows 97.7 MiB against 64 KiB. `fetches` now also takes what the exchange would move (`Room.away`: the heads on the ranks that are not the owner, and how many ranks the broadcast reaches) and fetches only while `host_pos x ranks reached <= 2 x rows x those heads`; the room bound stays. Every rank computes it from the same numbers, so they agree. test-glm-owner-fetch-host: a new test of the rule at TP=2 (full chunks to the room's end, the two cases above, the boundary at 64 rows a row, a first chunk) and TP=3 (each owner's heads elsewhere); it fails with the rule removed. The room's default size is stated as 131,072 rows (CHANGELOG, the test), 262,144 with TF_GLM_CHUNK_SOLO=4096.
…s latent rows at its region places, where held Part 4 of ashhart#1431 starts here. A paged sequence now carries `lpages`, its latent rows' places in this rank's region in row order: on the stream's owner, the latent table's entries as uploaded (`Pages.mapped`); elsewhere empty, and empty again on `close`. The kept-state plan reads a split sequence's latent parts through those places at the latent region's stride (part 2 read them through the page ids, which index the pooled region). On a rank that holds none of them the latent parts stay in the plan, in order and with their lengths, but are neither read nor counted, so every rank's plan has the same parts and a copy holds its own. Sizing counts them on every rank. Tests (test-glm-snapshots-host): on ten pages of class 1 of 2, the owner's plan reads every latent row once at its place id / 2 and never the trash page, every pooled key once at its id, and nothing else; another rank's plan has the same parts, the latent ones not held, and is shorter by exactly their bytes; both size the same. The existing two-region tests give `lpages` the identity places (every row on every rank). Each fails with the places read as page ids or with the latent parts counted where not held. The split pages test checks `lpages` on the owner, empty elsewhere and after close. The kept-state walk is core's (state_walk): its Part gains `alt` and `eachAlt` an `Alt` table, `each` unchanged (every caller's parts are not alt). GLM's plan sets `alt` on split latent parts and walks them on `lat_ids`; a rank without them leaves them out. state_walk's new test walks alt parts on their own pages, as a gap, and left out.
…es its size on every rank, restores at home A kept state's `home` is the owner of the stream it was kept from, the same on every rank (`Snap.home`, null unsplit); its copy alone holds the latent rows. Every rank leases the home's size (the store's charge, `snapshots.bytes`, which counts the latent rows), so rank 0's admission, which charges its own lease, can never pass a keep another rank's pool cannot hold. A restore onto a stream whose owner is the home moves each rank's own parts back, as unsplit. A restore anywhere else, or of a split state onto an unsplit sequence, is refused by name (`KvSplitAway`) here; the next commits place a resume at its home and carry the rows when it is not. Tests (test-glm-snapshots-host): with 512-wide latent rows, a state of class 1 kept on its owner's copy and on another rank's records home 1 on both, both lease the store's charge in pages though only the owner's copy holds the latent rows, and the owner's restore gives back every byte its plan read; away from home and onto an unsplit sequence: KvSplitAway. The test fails without the home, with each copy leasing only its own bytes, or without the refusal.
…state's home class when that class has room A stream that resumes a kept state takes its pages from the state's home class (`Pages.openNear`) when that class has them now, so its restore is at home and moves no latent row between ranks. Otherwise it takes the roomiest class as any stream does (ashhart#1313's rule, unchanged), and the restore carries the rows (next commit). The lane host reads the home from the stream's kept state (`Engine.homeOf`) where it opens the pass, and passes it to `seat`; admission is unchanged (a request fits when the roomiest class holds it). Off, every state's home is null and every stream takes the roomiest class, which with one class is today's. Tests (test-cuda-host, GLM): with class 0 the roomiest, a resume of a class-1 state opens on class 1; one that class 1 cannot hold opens on class 0; a home past the classes and no home take the roomiest. Fails with the home ignored.
…carries its latent rows from the home to the owner When a resumed stream's owner is not its kept state's home (the home class lacked the pages), every rank moves its own copy's parts back (`Pool.moveParts`: the home's copy holds the latent parts, which it now skips without reading; the others' copies never held them), then the home sends its copy's latent rows to the owner (`cuda_kv_carry.zig`). Both walk their plans' latent parts in order (the same parts and lengths on every rank) through the owner-attention room, the same pieces on both sides: the home copies a piece out of its snapshot pages (device or the pinned host tier) into the room and `ncclSend`s it; the owner `ncclRecv`s it into the room and copies it into its region at the stream's places. Rank pairs only: at TP=3 the third rank takes no part. A restore's size check is the copy's own (`kept`): the home's counts the latent parts, the others' do not. A split state on an unsplit sequence, or the reverse, stays refused by name (`KvSplitAway`). Tests (test-glm-snapshots-host, glm_kv_carry_host.zig, two ranks on a recording driver and NCCL on the host): - the carry alone, rooms of 1,000 bytes to 5 MiB: the owner's places get exactly the home's latent rows in plan order, in as many sends as receives of the same sizes, and nothing outside the stream's places (the trash page included) is written; the roles (home sends, owner receives, a third rank and a restore at home take none); - a restore off its home through `restoreState` on both ranks with a 3,000-byte room: each rank's non-latent parts are its own copy's, the owner's latent rows the home's; onto an unsplit sequence, KvSplitAway. Each fails with the home's lease offsets counting only latent parts, with `moveParts` not stepping over the latent parts the home's copy holds, or without the carry.
…ersations stays refused, by name With kept states' homes, resumes near them and carries in place, the prompt cache serves with the switch on: the start-up refusal of a prompt cache and `snap_save`'s backstop are gone. --learn-conversations alone is still refused (`KvSplitLearn`, replacing `KvSplitKeeps`), until its disk images record the split (next commit). `KvSplitAway` and `KvSplitCarry` are explained as bugs (the stream's owner and a state's home disagree in a way the carry does not cover). Off, nothing changes. The remaining rule is `cuda_start.learning(split, learn)`, in `keeping`'s place, with its test in test-glm-startup-host (refused with both, allowed with either alone); the options' `keeps` and its test go with the prompt cache's refusal.
…image holds its own rows, the words the split and home is gone, and with it the options' last kept-state flag). - Each rank's image is its own copy: the home's holds every part; another rank's lacks the latent runs (each DSA layer's latent rows and the head's), which its segments mark absent and its lease skips (`segments(.., lat)`); asked for one, a copy refuses by name (`KvSplitAbsent`). - The store's image keeps its format: rank 0's image, then each other rank's prefix. `route` sends the latent runs of rank 0's image to the home, at the home's offsets, and fans out the rest as before; the start-up probe compares the other ranks' shared bytes but those runs, and reads only the bytes it compares (`gaps`), so a rank off the probe state's home is never asked for one (asked, it stopped the server at start with KvSplitAbsent). - The words record `layout = split` and `owner = home` (the words ashhart#1411's `ddefe7cf7` set aside); a read refuses a split image on an unsplit server and the reverse, another world, or an owner past the world, by name (`DiskLayout`/`DiskWorld`). - A read's home is the class a stream would take now (rank 0's roomiest, in the blank step; the bytes are the same for any home). Each rank leases the home's size and seals its own copy's (`Plan.sized`, then `kept`), so the read is a kept state like any other: resumed at home, or carried. Tests: test-glm-snapshots-host (two ranks, a class-1 state kept on its owner's copy and another's): each rank's image out of its copy (the home's whole, the other's without the latent runs), read back into a blank at home 1, sealed by its words to its keep's size, then restored: every byte its plan holds equals the kept one; the words refused by an unsplit server; a home past the ranks refused. test-cuda-host (GLM): a copy off its home lacks exactly the latent runs, in image order, with lease bytes following each other, and refuses to copy one; route sends every latent byte to the home at its offsets, none shared, every other byte as before; `gaps` gives the bytes past a probe's loose spans, and `same` takes only those from each rank (four takes for two ranks and one loose span, none inside it). Fails with the seal counting the latent rows off the home, or `same` taking the whole range. `cuda_start.learning` and its test go with the refusal.
…of its pages on every other rank, given back at its first decode A fresh prompt pass (no kept state to resume) borrows, from each other owner class, as many pages as its own, when every one of those classes has them now (`cuda_pages_twin.zig`, rank 0's book). On each other rank the stream's latent table points into its twin, so every rank writes and attends its own heads for the pass, as unsplit (`State.twin`; the forward reads `State.attends()`, null while twinned), with no exchange. The begin step carries the twin ids after the stream's own (flag 2), and each follower maps its own class's, refusing another class (`ResidueMismatch`) or a wire that does not split into the classes. Before the stream's first decode round (`verify`, `verifyShared`, on every rank) the twin goes back: the latent table maps to the trash page again, the owner attends for every rank, and rank 0 frees the twin's pages. A stream that ends or is cancelled in its pass gives the twin back with its pages. Without the pages in every other class, or for a pass that resumes a kept state (its restore is at home or carried), there is no twin and the pass runs owner-computed, as before. A state kept during a twinned pass is kept as at home: the plan reads latent rows through `lpages`, which only the owner sets, so the owner's copy alone holds them. Admission is unchanged: a twin is borrowed only from pages free now, and a request never waits for one. Tests (test-cuda-host, GLM): a class-1 stream on rank 0 borrows 4 pages of class 0, its pooled-key table its own ids and its latent table the twin's places, attends null; after the first decode round the latent table is the trash page, the owner attends, class 0's pages are free, and the pool is idle at close. No twin when class 0 lacks the pages or the pass resumes a kept state; a fresh pass on the roomiest class is twinned, and closing it in its pass frees both. A follower maps a twinned begin to its class's twin, refusing a twin of another class or a short wire, and twins its own class's stream with no remap. With part 3 a prompt chunk takes the first of: its twin (every rank attends its own rows, `attends()` null), the history fetch (owner-only rows that fit the room), the owner attending every rank's heads (part 2). The split test checks a twinned stream does not fetch and fetches again once the twin is given back.
…tes, carries and twins (part 4 of ashhart#1431) `tf-cuda-test glm-kv-split-keep-check <ckpt> <rank> <world> <idfile>`, every rank with TF_GLM_KV_SPLIT=1, pages NaN-filled before use: a 9,000-token prompt and four windows of 1 to 4 rows against the same on a fresh whole window, byte for byte (first token, its logits, the windows' tokens and logits), over five passes on one paged stream: twinned on owner 1 keeping a state at 4,096; owner-computed on owner 0 keeping one; resumed at home 1; resumed off home 1 (carried to 0); resumed off home 0 (carried to 1). It fails if a twinned pass found no twin, a state is not homed on its owner, or the pool is not idle after. Built for aarch64 here (compile-cuda-tests). The disk tier's read under the split is the served check (--learn-conversations, restart, resume). The changelog's Unreleased line says what part 4 serves and that unset nothing changes. The setting's rows (`profile.zig`, `docs/environment.md`) say a kept state's rows stay on the rank it was kept on, in place of part 2's refusal, and name this part as evidence. The row in `docs/environment.md` also states the trade on this part's head: a cold 18.7k-token prompt's first token took 16.90 s against 14.92 s unset (about 13%) on two GB10s at TP=2.
…ges alone, NaN-filled under TF_KV_POISON A twin was as long as the stream's table, `need(prompt, max_tokens)`, though it goes back before the first decode round: on a busy pool it held the other classes' pages for a reply it never wrote, and a short request could wait out another stream's prompt pass. `openOwned` now borrows `need(prompt, 0)` pages. A twin's wire is still as long as the stream's ids: its pages, then the stream's own ids past the prompt, which a rank that does not hold them maps to the trash page, as untwinned. `mine` refuses a twin with no page of its rank's class, or whose ids past its pages are not the stream's. `map` NaN-fills a twin's latent places under TF_KV_POISON before the pass (`Pages.poisonLatent`), as `mapped` does a stream's own: a twinned pass that read a row it never wrote read the last stream's bytes. verifyShared's loop over its parts is `Engine.sharedParts`, which a host test reaches without a device. test-cuda-host (cuda_pages_split_test.zig): - the twin test: 3 pages for a 1,500-token prompt on a 4-page stream, the 4th table entry the trash page, the reply's page free for another stream; - a follower maps a twin shorter than the ids, and refuses a tail that is not the stream's ids or a twin with no page of its class; - TF_KV_POISON: rank 0 fills the stream's pooled keys then its twin's latent rows, a follower its twin page's; - `sharedParts` gives a twinned stream's twin back (no twin, the latent table on the trash page, the class's pages free) before the forward; - three ranks: for each owner, rank 0's wire holds the other two classes' twins, and each rank maps its own (the owner its places, the others their twin and the trash page), then gives it back. Each fails with its code removed: the give-back out of `sharedParts`, the fill out of `map`, the whole table borrowed, the tail check out of `mine`.
…s, and is back after its first verify window The twinned pass asks for a reply of 2,048 tokens, so its 9,000-token prompt's twin (18 pages a class) is shorter than its table (22). On rank 0 the check now fails if the twin took another count, or if after the pass's first verify window the slot is still twinned, its sequence still attends its own rows, or any other class has fewer free pages than before the stream opened. The pool runs under TF_KV_POISON as before, so a twinned row read before its write turns the reply to NaN.
…ne and the third rank posts nothing The recording wire now keeps which rank each send goes to and each receive comes from (its driver passed rank 0 whatever the peer). The two-rank test checks the home sends to the owner and no other, and that a third rank's side of `run` moves nothing. A new test restores a state kept on rank 1 onto a stream of rank 0 on three ranks through `restoreState`: ranks 1 and 0 carry as at two ranks, rank 2 restores its own parts with no send or receive, and no call names rank 2. It fails if a third rank takes a side, or if the peers are not checked. (test-glm-snapshots-host.)
…shhart#2937 in the evidence `docs/environment.md` says a prompt chunk fetches its history from the owner while that moves no more bytes than the owner's exchange, and that a fresh prompt pass borrows a twin of its prompt's pages until its first decode. `profile.zig` lists ashhart#2937 with the other parts.
…r again A split prompt chunk ran its exchanges whole and unpaired: piecesOf gave no pieces under the split, and the engine made no pair. That, not the owner work, was most of the split's cold prompt cost (an all-gather of every row a site, 39.8 GiB a port for a 28k-token prompt against 30.0 GiB pieced). Every split chunk now takes the engine's pieces. A step pairs two chunks only when both attend their own heads on every rank (a twin's pages, or a history fetch); a chunk the owner attends for every rank stays unpaired. The pair's twin buffers cost about 0.21 GiB a rank. Unset, nothing changes.
|
Restacked parts 2-4 onto main Only Host, on this head: No device rerun: no code of ours changed, and the rows in my last comment (on |
Current head (2026-11-12, evening):
4116e5daeon main223da7659Rebased onto main
223da7659, with the review's points as new commits on this part: a prompt twin holds the prompt's pages alone (not the whole table) and is NaN-filled underTF_KV_POISON;glm-kv-split-keep-checkchecks the twin's size and that it is back after the first verify window; the shared-window give-back and three-rank carries and twin wires have host tests;docs/environment.mdand the profile evidence cover every part. Where the text below differs, this section is the current one. Measured 12 Nov (one main earlier; since then a CHANGELOG-only rebase and the settings row):Two GB10s at TP=2, the full checkpoint,
--prompt-cache-gib 0, split rows withTF_KV_POISON=1, one build per head (12 Nov, built on main0fed2e4a5; the heads now pushed differ from those by a CHANGELOG-only rebase onto223da7659, plus on part 4 the settings row). Cold first token, seconds:"unset" is part 4's build with the setting off; main
0fed2e4a5takes 19.21 s on the 28k row. B's reply has the same token hash staggered as alone on every build, and every build's replies hash the same as main's.Twins add little over part 3's fetch on these rows (0.2% to 1.6%). The port counters show they engage (the fetch's one-way broadcast is gone), but a split prompt pass is never paired, twinned or not, and both split builds move about a third more bytes than unset; details in the 12 Nov reply. Decode with the setting on is about 2.5% slower on one stream (33.2 against 34.0 tokens a second) and 2% on four (46.7 against 47.6 aggregate).
Device checks on that build:
glm-paged-checkunset and split PASS,glm-kv-split-keep-checkPASS, served suite with and without the cache with main's token hashes, rank 1's log clean.Host on this head:
fmt, lean 341 vs 341,pr_check0 errors,test-cuda-host6,520 of 6,558 with 38 skipped (fresh cache), aarch64nativeandcompile-cuda-tests,test-glm-owner-fetch-host.Earlier head (2026-11-12):
3e12528aaon main6c5cef894Rebased onto main now that #2205 has landed; the last 8 commits are this part's, on part 3's
34ae6cb4f(cuda_native.zigis 587 lines). The setting's row indocs/environment.mdnow states the cold-prompt cost (asked on #2935): an 18.7K-token prompt's first token in 16.90 s against 14.92 s unset, about 13%. The last commit's message no longer says the keep check has not run on GPUs. Host on this head:fmt, lean 345 vs 345,pr_check0 errors,test-cuda-host6,477 of 6,515 with 38 skipped (fresh cache), aarch64nativeandcompile-cuda-tests.Device rows, 12 Nov (two GB10s, TP=2, bf16 KV, MTP drafts,
--context 131072 --parallel 4), run onc0cb964c9, the same 41 commits on main3885dceb2; main's3885dceb2..6c5cef894is dots.mocr on CUDA and touches none of these files:3885dceb2's from the same morning's run of GLM on CUDA: the latent split's per-rank page book and owner (TF_GLM_KV_SPLIT, #1431 part 1; stacked on #1411) #2415. Set isTF_GLM_KV_SPLIT=1 TF_KV_POISON=1. Turn 2 of the 30K conversation has to quote three marker sentences planted in it.--learn-conversations: written (26,624 tokens), resumed from RAM, read from disk after eviction (769 MiB in 498 ms) and after a restart (489 ms); the same hash and all three markers quoted in every row.glm-paged-checkunset and set PASS (set: 2 classes of 131 pages, idle after),glm-kv-split-keep-checkPASS,glm-index-checkPASS, kernel contractregions8 of 8 with 8 controls red as planted.--context 1048576 --parallel 8: 2,497 pages unset (one 1M stream), 4,740 set with no prompt cache (two), 4,100 set with the default cache (two).The text below is from the previous head.
Part 4 of #1431: with
TF_GLM_KV_SPLIT=1, kept states (the prompt cache and--learn-conversations), carries and prompt twins. Stacked on part 3 (#2937), which is on part 2 (#2935); please review the last 8 commits only (head473781255, restacked on part 3'sce004acb9on 11 Nov). Off by default: unset, every byte is as before.Draft, kept rebased as the parts below it move. Like parts 2 and 3, the branch sits on main with #2205 merged in until #2205 lands.
What changes
Kept states keep each rank's own rows.
lpages, its latent rows' places in this rank's region (empty where it holds none).home(the owner it was kept on). Every rank leases the home's size (the store's charge), so rank 0's admission stays safe.Resumes go home when they can. A resumed stream opens on its state's home class when that class has the pages (
Pages.openNear). Otherwise it opens on the roomiest, as any stream.Carries. Restored off its home, every rank moves its own parts back. Then the home sends the latent rows to the new owner through the owner-attention room, in the same pieces on both sides (
cuda_kv_carry.zig: NCCL send/recv on the main stream; a third rank takes no part).The prompt cache serves with the switch on; part 2's
KvSplitKeepsrefusal (cuda_start.keeping, the options'keepsand their tests) is gone. Until the learn commit,--learn-conversationsis refused bycuda_start.learning(split, learn), with its own test intest-glm-startup-host; the learn commit removes it.The setting's rows (
profile.zig,docs/environment.md) now say a kept state's rows stay on the rank it was kept on, in place of part 2's refusal; evidence GLM on CUDA: the latent split's per-rank page book and owner (TF_GLM_KV_SPLIT, #1431 part 1; stacked on #1411) #2415, GLM on CUDA: owner-only latent rows and owner-computed attention (TF_GLM_KV_SPLIT, #1431 part 2; stacked on #2415) #2935, GLM on CUDA: kept states, carries and prompt twins under the latent split (TF_GLM_KV_SPLIT, #1431 part 4; stacked on #2937) #2938.Disk images.
routesends rank 0's latent runs to the home, at the home's offsets.layout = splitandowner = home(the words GLM-5.3-Flash on CUDA keeps conversation states on disk, every rank's part (--learn-conversations) #1411 set aside). A split image on an unsplit server, the reverse, another world, or an owner past it are each refused by name.KvSplitAbsent); the first GPU run caught it.Prompt twins.
State.attends()is null while twinned).verify,verifyShared), the twin goes back on every rank, and rank 0 frees it. A stream that ends in its pass gives it back with its pages.With part 3, a prompt chunk takes the first of these that applies:
attends()is null while twinned, so the fetch does not fire then; the split pages test checks both sides.Core: the kept-state walk is
core/state_walk.zig. ItsPartgainsalt, andeachAlttakes anAlttable:eachis unchanged for every other caller, and a newstate_walktest covers the three cases.No new kernel: carries are copies and NCCL point-to-point, and twins are page tables.
Tests (host)
test-glm-snapshots-host:restoreStateon both ranks, with host NCCL;Each fails with its piece removed or broken (checked by mutation).
test-cuda-host(GLM):lpagesset and cleared, and resumes opening near home;gapsgives the bytes past its loose spans, andsametakes only those (four takes for two ranks and one loose span, none inside it);test-glm-pages-host,-startup-host,-disk-charge-host,test-cuda-disk-open-hostand the rest of the GLM host steps pass.On GPUs: two GB10s at TP=2 (2026-11-11)
These rows ran on this part's head before the 11 Nov afternoon restack (
fbf6ebdee); the restack changed the refusal's place, the setting's rows and part 2's room check, and was not rerun on GPUs. I'll rerun with the restack onto main.glm-kv-split-keep-check(new,TF_GLM_KV_SPLIT=1, 5 layers): PASS. Each pass's first token, four verify windows and their logits are byte-equal to a fresh whole window:The pool is idle after.
Served with the split,
TF_KV_POISON=1,--learn-conversations. A 30k conversation:Every read gave the same token hash as turn 2 computed fresh, and rank 1 logged no errors. On restart the server found 7 states on disk, with both ranks' parts.
Served with the split and the default prompt cache: a cold turn 2 and a resumed one (26,624 of 28,063 tokens cached, 1.44 s prefill), drafted and
"draft": false, with the same hashes as unset. Part 2 (GLM on CUDA: owner-only latent rows and owner-computed attention (TF_GLM_KV_SPLIT, #1431 part 2; stacked on #2415) #2935) has the full table.glm-paged-checkunset and set,glm-index-check, and the kernel contract'sregions: PASS.Host
On the head:
test-cuda-hostin full (5,221 of 5,258, 37 skipped),test-glm-snapshots-host, and the GLM host steps;fmtis clean, lean 386 vs 386 (0 new), the CUDA fragment check is clean, andtools/pr_check.pyreports 0 errors;aarch64-linux-gnunative build passes at the head, along with the CUDA test runner.Questions