Repository navigation
Conversation
448f3da to
26fe518
Compare
|
Coordination for #2980 and this PR before publishing the EXL3 stack update (#1441–#1446, #1522): my unpublished correction adds an EXL3 host-planning admission check to Both exclusions must survive: learned pockets remain eager as your PR requires, and an EXL3 pool requiring host planning cannot enter whole-pass capture. I am not changing your staging, abort/release handling, split or buffer keys. Flagging this before push to coordinate the shared predicate; no device-pass or publication-clearance claim for my correction yet. |
|
Thanks for raising this before you push. Agreed: both exclusions should hold. A folder with learned pockets stays eager, and an EXL3 pool that needs host planning stays out of whole-pass capture. Both of these PRs edit that line. #2980 adds if (e.collective != null and round_graph.enabled() and !has_media and e.pockets.mask() == 0 and <your host-planning check>) {Until this PR lands, Whoever lands second can keep the other's terms when resolving the conflict. If yours lands first, I'll carry your check through the rebases of #2980 and this PR. |
|
Thanks @swbiggart. We reviewed this PR's own commits (
Three things before it lands:
This carries #2980 and #2978, so it lands right after them.
|
26fe518 to
fbd535f
Compare
|
Thanks for the review.
|
|
Resumed prompt with the graph on: two RTX PRO 6000s at this head (
Turn 1 hashes to |
|
Thanks @swbiggart. The resumed turn hashes to Nothing else is owed on this PR's own commits. What's left is the landing order:
Against today's main only #2977 changes the prompt pass's collectives under this graph. So once this is restacked, we'll run
|
…ed and that a 4 s stall stays bounded The sum mode issues one raw all-reduce that must return ncclSuccess, as on a blocking communicator, and reports init time and device memory. The timeout mode's peer outlives rank 0's 3 s bound, so a collective that waits on the peer inside NCCL fails it. An optional MASTER address lets the two ranks run on two hosts.
…mmunicator Every collective on a nonblocking communicator ran its launch on a thread of NCCL's and returned in progress, so the engine polled it with a 50 us sleep, about 105 us a collective and 192 a decode pass. The parent stays nonblocking and bounds start-up under the control link and a deadline as before; a blocking communicator split from it, sharing its connections, carries every collective and returns once the kernel is enqueued. NCCL_RUNTIME_CONNECT=0, set when unset, makes NCCL connect inside the parent's init and the split's job, which the parent's poll bounds, so no collective waits on its peer for a connection. Abort ends the parent, which stops a split in flight, then the split.
…ECT and the measured decode The two-rank recipe says how start-up stays bounded and what NCCL_RUNTIME_CONNECT=1 would give up; the CUDA recipe's transport report names Flash Next with GLM; the changelog and the ledger's tensor-parallel row carry the measurements.
…communicator; one host keeps the blocking split
…icator across hosts, and the GB10 decode
…e capture TF_FLASHNEXT_DECODE_GRAPH=1 at --tp 2 failed every first plain round with UnownedCaptureSource: Cache.run set only `prepared`, so Pass.chunk uploaded the round's token ids through the copy-source owner inside the capture, which refuses any host source there. The round now queues its ids, positions, lane table and n-gram rows on the stream with stageWindow, as the one-GPU shared windows do, then captures or replays kernels and collectives alone; an unstaged pass is refused before any capture. A replay stages every round's inputs afresh, the lane table included.
A graph that captured NCCL collectives holds persistent plans, and ncclCommAbort waits without a bound until every such graph is destroyed (commDestroySync polls localPersistentRefs). With the round graph held by the engine, a peer lost during a replay, a capture or between rounds would hang the surviving rank in the abort instead of returning RankDisconnected or RankTimeout. The cache that captures a communicator's collectives now registers a release that the abort runs first: it ends an open capture, frees its graph and destroys the executable, so NCCL's abort can drain and return.
… eager A round of taught and untaught streams scales its pocket rows through a buffer rowScales allocates anew every round, which the round graph's key does not cover, so a replay would read the freed one. Like the window graphs, the round graph now leaves pockets out: with any pocket loaded both ranks run their rounds eagerly (the mask is the checkpoint's, so the ranks agree).
…and a lost peer Each of 52 two-rank rounds (one lane, two lanes at different positions, the other owner order, the 512-key seam) must equal an eager one-row forward of a twin stream byte for byte; with TF_FLASHNEXT_DECODE_GRAPH=1 every round must take the graph and the one-lane rounds must replay the first one's executable. `leave` decodes until the peer is stopped from outside: after four rounds it must hold the graph exactly when it is on, and the leader must then stop with RankDisconnected or RankTimeout rather than wait (the run's own timeout bounds a hang). The context is 1024 so the rounds cross 512 keys.
cuda_round_graph.zig stages its round before capture, cuda_collective.zig runs the round graph's release before an abort, and cuda_prompt.zig leaves rounds with pockets eager; all three reach only the opt-in TF_FLASHNEXT_DECODE_GRAPH path, so test-gradient-serving follows the new hashes.
It also puts the pull request number on the notes of the three rows re-pinned in gradient-serving.tsv.
…graph-split The abort releases the round graph before it aborts the parent and the split it made, cuda_collective.zig is re-pinned for both changes, and the round graph host test's fake communicator has no parent.
Every op the split adds to a round (the head and column gathers, copyCols and the FP32 matmul, the split shared expert) is ordered on the compute stream and reads scratch the round graph's key already names, so the graph captures it like the sum. Drop the load refusal and the round's dense_cut gate.
A clear keeps the counts; every 64th capture logs them, as the window graphs do. Host tests cover a split-shaped round's gathers inside the capture and a clear between rounds recapturing each one.
… drafted window's verify and keep leave it held A settle swaps a sequence's DeltaNet state buffers with its window's spares, and a captured round bakes those addresses in. The round graph's key now holds each lane's buffer digest (the one the window graphs key on) and capacity, so a settled lane is a new key, and verify, verifyMany and rollback no longer clear the graph. launch refuses a pass that carries a pocket before it captures anything.
…en graphed plain rounds
…se split, keyed by each lane's buffers
fbd535f to
6d6bc8e
Compare
|
Restacked on the new heads, as you asked: #2980 |
|
Thanks @swbiggart. The resumed turn's matching hashes already close that exactness ask. The CHANGELOG wording is fixed, and your restack names both updated parents, including the one-host agreement. The one thing we still need to finish reviewing the combined code is a single net diff of the resolved stack at
|
|
The resolved stack at
|
|
Thanks @swbiggart. The net diff and remerge-diff link answer the artifact request, including graph release before both communicator aborts and the combined serving pins. No further author change requested. The final resolved-code review and the two-GB10 graph-on/off check remain on our side before landing.
|
What this changes
With the opt-in
TF_FLASHNEXT_DECODE_GRAPH=1at--tp 2, start-up refused the default dense split (DecodeGraphWithDenseSplit), and every drafted verify or keep cleared the graph, so a drafted stream beside a plain one forced a recapture each verify. Now the graph captures the split's ops with the round, and a key holding each lane's buffers replaces those clears. Tokens are unchanged; unset, so is the served path.0ddafbc80) and flashnext cuda: at --tp 2 on one host, collectives run on a blocking split of the nonblocking communicator #2978 (headb4a758b2f) on main4cfd95573, through the merge42032c865; review42032c865..6d6bc8e65.42032c865..6d6bc8e65(11 files, +249/-50 before the number): split taken; counters and host tests; device test checks both ranks; the key, with a pocket guard; a count line; device test verifies between graphed rounds; re-pins; changelog and ledger.atc overlap --json: ATC 0.9.6 on2f4cdfe7e(base93a2e93ec): 8conflicts, 2 clean, 242unknown(complete: false). Re-run right before this pull request opened:atc overlap --jsonat 2026-11-12 14:10 UTC on headfbd535faf; everyconflictsentry is named in this section, or adds no conflict beyond those it already has with main (1), or conflicts only inCHANGELOG.md's## Unreleasedlines, as this section describes (5); each checked withgit merge-tree. I trial-merged the 123 open pull requests touching a stack file.CHANGELOG.md, none from these commits: dsv41 metal: the lane target, the prompt cache and the native server over Metal #1720, dsv41 metal: the vision tower's kernels under the kernel contract #1721, dsv41 metal: image prompts #1722, Vulkan: qmm_q4_vec4 reads the 4-bit projection 16 bytes at a time, 34.8% to 43.0% less run time on an M1 with the same logits (stacked on #1544) #1990, Vulkan: qmm_q4_tile reads the 4-bit projection from 32-column tiles, 29% to 67% less run time on an M1 with the same logits (stacked on #1990) #2015 and Vulkan: one submission per batch of dispatches, 30% to 56% less run time on an M1 with the same logits (stacked on #2015) #2016 already conflict with main; llama on Metal: Qwen2 checkpoints, Bimo-7B-CyberAgent's bf16 release with its q, k and v biases and YaRN at HF's rounding points #2890 and server: --execution-witness serves a read-scoped snapshot of what runs, or why it is unknown #2901 meet the parents'## Unreleasedheading.cuda_lanes_load.zig(EXL3 Flash Next on CUDA: one rank, offloaded experts, sm_120 gate (1/8) #1441-EXL3 Flash Next: draft lane guard for offloaded experts (8/8) #1446, EXL3 Flash Next: host-resident K/V on the tier (6/8) #1522) andcuda_mtp_ops.zig(EXL3 Flash Next: CUDA graphs and the device planner (7/8) #1445), all already conflicting with main. The parents add MoE route tap: optional TF_ROUTE_TAP trace of global top-k #1886 incuda_round_graph.zig.flashnext-model-rank, graph on.Checks on this head
6d6bc8e65on main4cfd95573;zig fmt --check build.zig zigandgit diff --checkpass on6d6bc8e65. Receipts on2f4cdfe7e,1bfb3e0f8(RTX) andbee6b7caf(GB10) stand for6d6bc8e65: rebasing ontoeab305db0changed their code only by main's flashnext: refuse a malformed TF_FLASHNEXT_* setting by name before device work #2585, which refuses malformedTF_FLASHNEXT_*settings at start-up; the later rebases onto4f8317dac,000b534a2and3885dceb2movedCHANGELOG.mdlines and took the parents' final heads, whose only code change is three test lines that set round-graph-capture'sreleasehook to null. Two hosts now keep the blocking-collectives parent's nonblocking communicator, so the GB10 solo cells were re-measured atbee6b7caf; earlier GB10 rows are labelled. The restack onto4cfd95573takes both parents' heads after flashnext-rank: the failure modes start once both communicators are up, and timeout bounds only the collective #2976 and flashnext cuda: at --tp 2 a prompt pass reduce-scatters its routed experts' slots by rows and gathers their sums #2977 landed, including flashnext cuda: at --tp 2 on one host, collectives run on a blocking split of the nonblocking communicator #2978's one-host agreement; the merge is resolved as before, and these eight commits are unchanged apart from pin rows andCHANGELOG.mdlines.python3 tools/zig/lean_check.py, on the base and on this head: 341 and 341 (base4cfd95573), 0 new.pr_check --base upstream/mainon this head: 0 errors, 3 advisory warnings (no--pr; the blocking-collectives parent's RFC 5737 192.0.2.x test addresses).zig build test-cuda-host: exit 0 on this head (Linux x86_64, CUDA 13.3.1 image); on2f4cdfe7e(--summary all, fresh cache), 6193/6232 passed, 39 skipped. New tests: the split round's capture; the key across settles and seams; mixed streams; the pocket refusal.2f4cdfe7e(plants.txt): withoutkey.buffers[i], 2 failed; without the pocket guard, 1;clearresetting the counters, 3; the split test's fake pass without its gathers, 1.zig build test test-golden -Dcpu=apple_m1ond532b9f05, this stack on maineab305db0before the parents' final heads: 7606/7680 passed, 73 skipped, 1 failed:dashboard_test, which maineab305db0also fails (same Mac: 13041/13152, 110 skipped).zig buildon2f4cdfe7e: pass.zig build test-gradient-servingon this head: pass.tf-cuda-test flashnext-model-rank MODEL RANK PORT ids.txt rounds MASTERon 2 x GB10 (fb8fdf3d8builds, blocking split across the hosts), one rank a host (MASTERis rank 0's), graph on; each rank asserts its counts and every graphed round equals an eager row byte for byte:f415e5f2c):FAIL round graphs captured and replayed { 0, 0 }, want { 6, 47 };93baf0599):FAIL plain rounds between verifies captured 4 more;071c02ed2):PASS 63 rounds equal eager rows byte for byte (round graph true, dense split 1), both ranks; graph off also passes.1bfb3e0f8, noMASTER: the same PASS.TF_FLASHNEXT_SHARD_HEAD=1,leave, or its fail-before: not run; graph off and the fail-before ran on GB10.bee6b7cafsolo cells ran that path graph on; RTX ran the rest on the same graph code./metrics.--tp 1, Metal, HIP: unreachable (needs a collective).Receipt
tp2_bench.py(median of 3 greedy 256-token requests) andgs_mixed.py/rc_mixed.pyovertools/bench_concurrent.py'sstreamclient;cell_sum.pyrecomputes (https://gist.github.com/swbiggart/659bf94526ad1920cacde6737ca27356).2 x GB10 (200G RoCE) at
bee6b7caf, one start each, graph off then on:Both ranks: 5 captures, 1,024 replays.
2 x GB10, earlier build (
fb8fdf3d8, blocking split across the hosts), six starts ABCCBA, means of two: new3385b84e2, old70bb77b92(split without the key), off: new with the graph off.After the mixed cell, both ranks: new 17 captures, 3,072 replays; old 257, 2,608.
2 x RTX PRO 6000 Blackwell Max-Q (PCIe Gen4, 300 W),
1bfb3e0f8, one start each:Both ranks: 21 captures, 4,096 replays after the mixed cells.
--tp 2with the graph on, plain one-row rounds. Unset, nothing changes (the removed clears acted on an empty cache).TF_FLASHNEXT_DECODE_GRAPHexists.MASTERis a test argument.6d6bc8e65; 2 x GB10 (sm_121), 2 x RTX PRO 6000 (sm_120), host tests on an Apple M5 Max; Zig 0.17.0, CUDA 13.3.1, NCCL 2.32.3; drivers 580.178.04 and 580.159.03 (GB10), 595.99.02 (RTX);nvidia/Qwen3.8-Flash-Next-NVFP4atfc694b54fb0174e0913e6adf86691ef85a4ead47.Draft.settleswaps thestatebuffers whose addresses a capture bakes in, so each lane's buffer digest is in the key;launchrefuses an unkeyed pocket.tools/bench_concurrent.py --alone --serialortoken_shaagainst"draft": false:token_sha4e759dbd859c(code) and764da77fbd3f(chat) in every cell, graph on and off. RTX, graph on,--levels 1,4 --alone --serial: alone 3/3 and 12/12, serial equal, 0 failed.tools/bench_openai.py: the tables, samestreamclient.Checklist
The title and description match the head commit, including the receipt and the stack.
New tests fail before the change, pass after it, and skip cleanly without their dependency.
Comments and docstrings are one line. No measurements or history in the source.
No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.
Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
Drafted output equals
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. Open: no resumed prompt was served with the graph on.No precision traded for speed. If the bits change, the description says which sums change and why.
Decode and prompt processing are no slower than main at one stream and the widest admitted; with
--drafter, no slower than without it, greedy and sampled. Open: on RTX, four streams at once are 0.3% lower with the graph on.