[swbiggart]

What this changes

At --tp 2, Flash Next on CUDA runs every collective on a nonblocking NCCL communicator: each returns in progress, runs on NCCL's thread, and is polled with 50 µs sleeps, four a layer. On one host (two RTX PRO 6000) that wait is the bottleneck.

Each rank now learns from the control link's two addresses whether both share a host (Link.oneHost). On one host, every collective runs on a blocking communicator split from the nonblocking one (ncclCommSplit, split_share) and returns once enqueued. The split has its own deadline, so start-up is bounded by twice the rank timeout. NCCL_RUNTIME_CONNECT=0, set there only when unset, makes NCCL connect inside the bounded init and split, not in a collective. On two hosts, where the split cost two GB10s 0.8-1.3% of plain decode, the collectives stay on main's nonblocking communicator, with no split or setenv. A stalled or departed peer still ends a collective within the bound. Same collectives, order and sums: replies are token-identical.

Two RTX PRO 6000 Blackwell Max-Q go from 28.9 to 73.3 tok/s plain and 112.6 to 210.8 drafted (code); two GB10s are level with main.

• Issue: Refs Roadmap: one N-rank layer for TP, PP and EP, shared by every family #2560 (part of T6, Flash Next CUDA). No new family, backend or arithmetic, so no gate; T6's move onto fabric/tp.Collective remains.
• Stacked on: main (4cfd95573). de20debb5: rank-test checks (enqueued return, bounded 4 s stall) on flashnext-rank: the failure modes start once both communicators are up, and timeout bounds only the collective #2976's start-up fix. 24a913da0: the split, host tests, pins; e5cafb123: docs. b40053cde: Link.oneHost, nonblocking across hosts, host tests, the enqueued check on one host only, pins; a82050e8b: its docs. b4a758b2f: both ranks exchange their one-host answer before NCCL starts, host test, pins.
• atc overlap --json (ATC 0.9.6, on 5da839cce, base 93a2e93ec, before the rebase): complete: false, 12 changed files, 250 overlaps: 5 conflicts, 5 clean, 240 unknown.
  • All five conflicts (GLM on CUDA: the latent split's per-rank page book and owner (TF_GLM_KV_SPLIT, #1431 part 1; stacked on #1411) #2415, qwen27: slot 0 keeps its rewind point through the warm pass #2723, llama on Metal: Qwen2 checkpoints, Bimo-7B-CyberAgent's bf16 release with its q, k and v biases and YaRN at HF's rounding points #2890, xpu: a sampled Qwen3.8-27B row draws from a top-k kept on the card #2895, core: Admit and Ledger move to engine_admission.zig, unchanged #2896) are CHANGELOG.md's re-opened ## Unreleased heading.
  • git merge-tree trial merges of open pull requests touching a changed file: only the EXL3 Flash Next stack (EXL3 Flash Next on CUDA: one rank, offloaded experts, sm_120 gate (1/8) #1441-EXL3 Flash Next: draft lane guard for offloaded experts (8/8) #1446, EXL3 Flash Next: host-resident K/V on the tier (6/8) #1522) adds conflicts (tests/cuda/main.zig; cuda_collective.zig for EXL3 Flash Next: CUDA graphs and the device planner (7/8) #1445, EXL3 Flash Next: draft lane guard for offloaded experts (8/8) #1446).
  • link-wake (flashnext cuda: at --tp 2 a control-link wait polls with a 50 us timeout for its first 50 ms, then blocks #2971) also edits cuda_rank_wire.zig, its pin and docs lines, so the heads conflict there.
  • Re-run right before this pull request opened: atc overlap --json at 2026-11-12 14:48 UTC on head c7907837d; every conflicts entry is named in this section, or adds no conflict beyond those it already has with main (5), or conflicts only in CHANGELOG.md's ## Unreleased lines, as this section describes (1); each checked with git merge-tree.

Checks on this head

• Head commit: b4a758b2f9ad2be537f4ed507f0c6708260d48a7. Receipts ran on the named builds; the rebases since, onto f0d8c8d50 (before f4a2caadc) and 4f8317dac, 000b534a2 and 3885dceb2, moved only CHANGELOG.md lines, and onto eab305db0 only the sameHost test's addresses (and their pin row) and the changelog's drafted figure (flashnext: refuse a malformed TF_FLASHNEXT_* setting by name before device work #2585 there changes only how TF_FLASHNEXT_* settings are read), so they stand. Onto ab78aef90 and 4cfd95573 (flashnext-rank: the failure modes start once both communicators are up, and timeout bounds only the collective #2976 and flashnext cuda: at --tp 2 a prompt pass reduce-scatters its routed experts' slots by rows and gathers their sums #2977 landed): the first commit keeps flashnext-rank: the failure modes start once both communicators are up, and timeout bounds only the collective #2976's rank-test fix, the source scan allows ncclReduceScatter (11 calls), and the pins come from the merged files; the flashnext-rank modes ran again on ea13840eb, this code on ab78aef90.
• This head: zig fmt --check build.zig zig, git diff --check pass; pr_check --base upstream/main 0 errors, 3 advisory warnings (no --pr; the sameHost test's RFC 5737 192.0.2.x addresses).
• Lean checker, python3 tools/zig/lean_check.py: base 4cfd95573 341, this head 341, the same findings, 0 new.
• Tests run, with the command and pass, fail and skip counts:

  • zig build test-cuda-host: this head exits 0 on Linux x86_64 (CUDA 13.3.1 image), as 0fc22f003, this code before flashnext-rank: the failure modes start once both communicators are up, and timeout bounds only the collective #2976 and flashnext cuda: at --tp 2 a prompt pass reduce-scatters its routed experts' slots by rows and gathers their sums #2977, did on macOS (run without --summary); uncached on 7e20ed85c: "379/379 steps succeeded; 6139/6178 tests passed (39 skipped)". Fake-NCCL tests: split and abort order, split failures and deadline, a scan allowing only the needed NCCL calls, no split across hosts, sameHost cases. Seven of eight planted mutations fail a test; the eighth (error path aborting the child first) is unobservable.

  • zig build test test-golden -Dcpu=apple_m1 (Apple M5 Max) on 20387dd00, this code on eab305db0 before the changelog-only rebases: "Build Summary: 837/839 steps succeeded (1 failed); 7595/7669 tests passed (73 skipped, 1 failed)". The failure, dashboard_test, is inherited: main eab305db0 fails it on the same Mac ("Build Summary: 837/839 steps succeeded (1 failed); 13041/13152 tests passed (110 skipped, 1 failed)").

  • zig build test-gradient-serving: pass on this head.

  • Fail before, pass after, tf-cuda-test flashnext-rank, one rank per GPU: main + the test is d41b7001e; this change is da506edfc, 2567f4211 (RTX) and f4a2caadc (both pairs):

    Mode	main + the test	this change
    sum (a collective returns ncclSuccess; from f4a2caadc on one host only)	FAIL: 7 (ncclInProgress), both	PASS, both; f4a2caadc GB10: "two hosts: collectives on the nonblocking communicator"
    timeout (4 s stall, 3 s bound), eof	PASS, both	PASS, both
    missing (one rank alone)	GB10 PASS; RTX not run alone	PASS, both (da506edfc; RTX also 2567f4211)
    timeout, negative control	NCCL_COMM_BLOCKING=1: hangs (killed at 120 s), both	NCCL_RUNTIME_CONNECT=1: hangs (killed at 120 s), both

    On ea13840eb, this code on ab78aef90, RTX pair: sum, timeout, eof, eof-scatter, timeout-scatter and exchange PASS on both ranks; sum reports "one host: a collective returns enqueued".

• CI: runs on this head.
• Not run, and why:
  • Alternated RTX starts: one start each, the gap is 1.9-2.5x.
  • GB10 four or more streams and cold prompts at the head: main's communicator path, equal to main by construction.
  • Served SIGSTOP or kill of a peer: covered by the rank test's timeout and eof modes.
  • A launch blocked during a long prefill while the peer is frozen: not covered, as on main.
  • Resumed prompts, --tp 1: no prompt state touched; one rank opens no NCCL.
  • GB10 sampled requests: no sampling state touched.
  • NCCL 2.30.x: source-checked only; runs used 2.32.3.

Receipt

• Main against branch: cells and numbers, or why they can't be run:

  2 x RTX PRO 6000 Blackwell Max-Q (PCIe Gen4, one host), main 93a2e93ec then 2567f4211, one start each. Decode tok/s at 256 tokens, greedy, median of 3; streams: tools/bench_concurrent.py --alone --serial aggregate tok/s; cold prompt tok/s, two requests:

  Cell	main	this change
  code, drafted	112.63	210.79
  code, plain	28.89	73.29
  chat, drafted	75.04	149.87
  chat, plain	28.89	73.18
  code, 1/4/8 streams	113.8/347.6/526.0	211.1/466.6/599.6
  same, temperature 1.0	111.1/283.4/435.0	213.3/384.7/485.4
  cold prompt, 32,768 tokens	2246.7, 2286.9	2295.3, 2293.5

  2 x GB10 (200G RoCE, one rank a host), starts alternated main, forced nonblocking, blocking, blocking, forced nonblocking, main, then f4a2caadc once. "Blocking" is 7e20ed85c (a split on every host), "forced nonblocking" the same with NCCL_COMM_BLOCKING=0. Decode tok/s, mean of each mode's starts:

  Cell	main	forced nonblocking	blocking	f4a2caadc
  code, drafted	51.14	51.12	51.12	50.97
  code, plain	23.05	23.05	22.87 (-0.8%)	23.11
  chat, drafted	37.83	37.73	37.84	37.68
  chat, plain	23.02	23.05	22.71 (-1.3%)	23.04

  The head is within 0.4% of main; main's two starts differ by up to 0.6%.

  Start-up: RTX init 0.50 → 0.61 s, 48.1 GiB left for streams (main 48.2), 8 streams each; GB10 rank-test init 0.30-0.34 s, main 0.30-0.31 s.

• Boxes it reaches: Flash Next on CUDA at --tp 2, both ranks on one host.

• New switch? Default, accepted values, profile row: none. On one host it sets NCCL_RUNTIME_CONNECT=0 when unset, as GLM's CUDA path does.

• Environment, with the TensorFold commit, Zig and backend toolchain, machine and GPU, checkpoint and revision: commits per table; Zig 0.17.0, CUDA 13.3.1, -Doptimize=fast, NCCL 2.32.3. RTX: sm_120, driver 595.99.02, P2P. GB10: sm_121, 121.7 GiB, Ubuntu 24.04.4, drivers 580.178.04 / 580.159.03, NET/IB. nvidia/Qwen3.8-Flash-Next-NVFP4 revision fc694b54fb0174e0913e6adf86691ef85a4ead47; --tp 2 --context 65536 --prompt-cache-gib 0, default MTP drafts.

• Paths and platforms the change reaches, which I ran and which I could not: sm_120 on one host and sm_121 on two hosts, Linux; not other SMs, NVLink or HIP.

• Exactness, from tools/bench_concurrent.py --alone --serial or token_sha against "draft": false: every decode request in every start, drafted and plain, has the reference hashes (chat 764da77fbd3f, code 4e759dbd859c). RTX cold prompts match across builds (cached_tokens 0); every RTX bench_concurrent row, greedy and sampled: alone and serial equal, 0 failed.

• Main against this branch, same machine and checkpoint, posted before asking for a run on our machines (or why you can't): above. Raw results: https://gist.github.com/swbiggart/12edb97d751417e5093bcaacbf32bc22

• Decode speed before and after, from tools/bench_openai.py: tp2_bench.py, above.

• Prompt speed before and after: tp2_bench.py's cold prefill, above; 64k exceeds the context.

• Licence of any new weights or drafter, or "n/a": n/a.

Checklist

• The title and description match the head commit, including the receipt and the stack.

• New tests fail before the change, pass after it, and skip cleanly without their dependency.

• Comments and docstrings are one line. No measurements or history in the source.

• No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.

• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. Resumed prompts not checked; drafted equals plain on both pairs, concurrent equals solo on RTX.

• No precision traded for speed. If the bits change, the description says which sums change and why. (Same collectives, same order.)

• Decode and prompt processing are no slower than main at one stream and the widest admitted; with --drafter, no slower than without it, greedy and sampled. RTX: faster at every level; cold prompts no slower. GB10: one stream level; more streams and prompts run main's path.

[ashhart]

Thanks @swbiggart. Plain decode on the one-host pair going from 28.9 to 73.3 tok/s with the same tokens is a big result. Keeping two hosts on main's nonblocking path, after measuring what the split cost there, was the right call. The fake-NCCL tests for split order, deadline and abort are thorough.

Thanks for the rebase. 1b9fc3c1da has the same five commits as before, so these points from review still stand:

1. Make sure both ranks agree on one_host. Link.oneHost (zig/src/families/flashnext/cuda_rank_wire.zig:196) says both ends read the same answer, but that only holds when neither end sits behind a proxy or NAT. Take rank 0 in a container with a published port and rank 1 on the host connecting to 127.0.0.1. Rank 1 sees a loopback peer and splits, but rank 0 sees the proxy's bridge address and doesn't. The split then waits for a peer that never joins, and start-up fails with RankTimeout where main would have served. Please have each rank send its answer to the other before NCCL init (one header each way, or alongside the unique id), and only split and set NCCL_RUNTIME_CONNECT (cuda_collective.zig:29-31) when both say one host. A host test with two different answers would cover it.
2. This branch has its own version of flashnext-rank: the failure modes start once both communicators are up, and timeout bounds only the collective #2976's rank-test fix (the 30 s init followed by timeout_ms = 200, and the exchange at sequence 1 in zig/tests/cuda/flashnext_rank_tests.zig:41-48), so the two conflict in that file. flashnext-rank: the failure modes start once both communicators are up, and timeout bounds only the collective #2976 is landing first. Please rebase onto it and keep one version.
3. flashnext cuda: at --tp 2 a prompt pass reduce-scatters its routed experts' slots by rows and gathers their sums #2977 adds ncclReduceScatter through c.api, and it's landing ahead of this. The source scan at cuda_collective.zig:333-353 allows a fixed list and counts exactly 10 calls, so after the rebase it needs ncclReduceScatter in allowed and the new count. The test's reasoning still holds, since it's a collective that NCCL connects at init.
4. Against today's main only CHANGELOG.md conflicts, and we sort the changelog out when landing. Once flashnext cuda: at --tp 2 a prompt pass reduce-scatters its routed experts' slots by rows and gathers their sums #2977 is in, though, docs/optimization-ledger.md, zig/tests/cuda/main.zig and the gradient-serving.tsv pins (cuda_collective.zig and cuda_launch.zig among them) conflict as well, so the rebase in 2 and 3 should be onto main after flashnext cuda: at --tp 2 a prompt pass reduce-scatters its routed experts' slots by rows and gathers their sums #2977.

Once 1 is in, a rerun of flashnext-rank sum/timeout/eof on the one-host pair at the new head is enough. The decode receipts stand, since the collectives don't change.

• Synthena (Ash's Agent)

[swbiggart]

Thanks for the review.

1. Pushed as c7907837d. Each rank now sends its one-host answer over the link before NCCL starts (agreeOneHost in cuda_collective.zig), and the split and NCCL_RUNTIME_CONNECT apply only when both say one host. A host test covers agreeing, disagreeing and malformed answers.
   2-4. Once flashnext cuda: at --tp 2 a prompt pass reduce-scatters its routed experts' slots by rows and gathers their sums #2977 is on main, I'll rebase onto it, keep flashnext-rank: the failure modes start once both communicators are up, and timeout bounds only the collective #2976's version of the rank-test fix, add ncclReduceScatter to the scan's allowed list and count, and re-pin. Then I'll rerun flashnext-rank sum/timeout/eof on the one-host pair at the new head.

[ashhart]

Thanks @swbiggart. c7907837d is what we asked for in point 1:

• agreeOneHost (zig/src/families/flashnext/cuda_collective.zig:52-64) sends one byte each way before the unique id. Rank 0 sends first and rank 1 receives first, so the two can't wait on each other.
• It returns one host only when both ends say so. Any byte other than 0 or 1 fails with BadRankFrame (:62). It runs under the same 30 s start-up bound as the id exchange.
• NCCL_RUNTIME_CONNECT is set only after the agreement (:29-31).
• The host test covers both ranks agreeing, each side disagreeing, and a malformed answer.

Two hosts keep main's nonblocking path, with one extra byte each way at start-up.

Your plan for 2 to 4 works. #2976 lands first and #2977 after it. Once #2977 is on main, please:

1. rebase onto it and keep flashnext-rank: the failure modes start once both communicators are up, and timeout bounds only the collective #2976's rank-test fix;
2. add ncclReduceScatter to the scan's allowed list and its call count;
3. re-pin from the merged files;
4. post the flashnext-rank sum, timeout and eof run on the one-host pair at the new head.

After that, we'll run start-up and a short serve on two GB10s, one rank a host, to check that the two-host path still matches main. #2982 carries this PR at 1b9fc3c1d, so it will need your new head as well.

• Synthena (Ash's Agent)

[swbiggart]

Rebased onto main 4cfd95573, which has #2976 and #2977, as b4a758b2f:

1. The first commit keeps flashnext-rank: the failure modes start once both communicators are up, and timeout bounds only the collective #2976's rank-test fix as landed: the 30 s start-up bound, the 200 ms bound for the stall alone, and meet(98) before the injected failure. It now adds only what this PR checks: the MASTER address, the init time and memory report, the raw ncclAllReduce that must return enqueued, and the 4 s stall that outlives rank 0's 3 s bound.
2. The source scan in cuda_collective.zig allows ncclReduceScatter, and its call count is 11. The collective test rig sets the new ncclReduceScatter field, and flashnext cuda: at --tp 2 a prompt pass reduce-scatters its routed experts' slots by rows and gathers their sums #2977's exchange rig sets parent = null, as the other rigs do.
3. Re-pinned from the merged files. The cuda_collective.zig and cuda_launch.zig notes keep flashnext cuda: at --tp 2 a prompt pass reduce-scatters its routed experts' slots by rows and gathers their sums #2977's text and add this PR's.
4. tf-cuda-test flashnext-rank at ea13840eb, this code on ab78aef90, on the one-host pair (2 x RTX PRO 6000, one GPU each, real NCCL), both ranks PASS in every mode:

Mode	Rank 0
sum	a collective returns enqueued on one host; chained sums exact; init 881 ms, 294 MiB
timeout	RankTimeout, aborted, reuse refused within the bound
eof	RankDisconnected, aborted, reuse refused within the bound
eof-scatter, timeout-scatter	the same through the reduce-scatter
exchange	#2977's exchange equals the all-reduce byte for byte, 65 to 8,192 rows

zig build test-cuda-host (Linux x86_64) and test-gradient-serving pass, and lean is 341, 0 new.

[ashhart]

Thanks @swbiggart. Both ranks now agree on one_host before NCCL starts. The rebase keeps #2976's failure test, adds ncclReduceScatter to the source guard and preserves the merged pins. Your new-head sum, timeout, EOF, scatter and exchange results close the requested one-host checks.

This now waits on one startup and short-serve run that we'll schedule on two GB10s, one rank per host, checking the nonblocking path and token equality against main. No action needed from you.

• Synthena (Ash's Agent)