[swbiggart]

What this changes

At --tp 2, a Flash Next rank ran its first collective (the benchmark gather in cuda_lanes_load.zig's open) as
soon as its own weights were in, under the collective's 30 s bound, so ranks whose loads finished more than 30 s apart
could not start. Now each rank first runs one more load-ready exchange (Session.room, with its 10-minute bound),
then the gather; a peer that exits, fails its load or closes its link ends the wait with RankDisconnected. No new
frame, setting or bound.

• Issue: Fixes Flash Next on CUDA: a --tp 2 start fails when one rank loads over 30 s later #2970. Refs Roadmap: one N-rank layer for TP, PP and EP, shared by every family #2560 (its "Start-up agreement" row); completes none of its items.
• Stacked on: main (4cfd95573); 36be03cac (change, tests, recipe line, serving pins), c40a84b19 (changelog) and fa5e3d56d (a failed load wait
  frees the prompt engine: its errdefer moves up to just after Engine.initRank; test 3 and the pin row follow).
• atc overlap --json (ATC 0.9.6, 2026-11-11 12:01 UTC, before the rebases, base 93a2e93ec, complete: false):
  "conflicts" with dsv41 metal: image prompts #1722, GLM on CUDA: the latent split's per-rank page book and owner (TF_GLM_KV_SPLIT, #1431 part 1; stacked on #1411) #2415, qwen27: slot 0 keeps its rewind point through the warm pass #2723, llama on Metal: Qwen2 checkpoints, Bimo-7B-CyberAgent's bf16 release with its q, k and v biases and YaRN at HF's rounding points #2890 and xpu: a sampled Qwen3.8-27B row draws from a top-k kept on the card #2895; every open pull request sharing a file was trial-merged
  with git merge-tree. New conflicts: zig/tests/cuda/main.zig with EXL3 Flash Next on CUDA: one rank, offloaded experts, sm_120 gate (1/8) #1441-EXL3 Flash Next: draft lane guard for offloaded experts (8/8) #1446 and EXL3 Flash Next: host-resident K/V on the tier (6/8) #1522 (keep their usage text,
  add late|gone); the re-opened ## Unreleased heading in CHANGELOG.md with ricursive: Ripple-65B-Preview on native CUDA (NVFP4) #2302, ricursive: Ripple-65B-Preview's DFlash2 drafter on native CUDA #2475, ricursive: load Ripple's EXL3 packs through main's shared EXL3 code #2514, ricursive: serve Ripple-65B on two GB10s with up to 1M tokens of context #2522, ricursive: speed up Ripple-65B on two GB10s #2523,
  qwen27: slot 0 keeps its rewind point through the warm pass #2723, server: the last --compact-at wins (auto then a fraction kept auto) #2883, server: serve --config FILE.toml reads flags from a file #2885, llama on Metal: Qwen2 checkpoints, Bimo-7B-CyberAgent's bf16 release with its q, k and v biases and YaRN at HF's rounding points #2890 and xpu: a sampled Qwen3.8-27B row draws from a top-k kept on the card #2895. Whichever of this and CUDA (GB10): Flash Next KV in the display carveout, opt-in #2763 lands second re-pins cuda_lanes_load.zig in
  gradient-serving.tsv. Re-run right before this pull request opened: atc overlap --json at 2026-11-12 14:08 UTC on head 25a26afab; every conflicts entry is named in this section, or adds no conflict beyond those it already has with main (1), or conflicts only in CHANGELOG.md's ## Unreleased lines, as this section describes (5); each checked with git merge-tree.

Checks on this head

• Head commit: fa5e3d56defa745f4757fb7dc0110ed91b426b4d
• Receipts were measured on 1da83e13d (tree equal to 17f7d67d2) before the rebases onto main eab305db0 (flashnext: refuse a malformed TF_FLASHNEXT_* setting by name before device work #2585
  there changes only how TF_FLASHNEXT_* settings are read, in cuda_lanes_load.zig too) and then 4f8317dac
000b534a2 and 3885dceb2 (only the CHANGELOG.md line and pin rows moved), then ab78aef90 and 4cfd95573 (flashnext-rank: the failure modes start once both communicators are up, and timeout bounds only the collective #2976 and flashnext cuda: at --tp 2 a prompt pass reduce-scatters its routed experts' slots by rows and gathers their sums #2977 landed; this branch's
  late and gone join flashnext cuda: at --tp 2 a prompt pass reduce-scatters its routed experts' slots by rows and gathers their sums #2977's mode list, and the two flashnext-rank usage lines become one); none changed this branch's other lines, so every
  receipt stands, and the device modes ran again on ab6d2ceaf, this code on ab78aef90.
• Lean checker, python3 tools/zig/lean_check.py: main 4cfd95573 341 (inherited), this head 341, the same
  findings, 0 new.
• Tests run, with the command and pass, fail and skip counts:
  • zig build test-cuda-host: this head passes on Linux x86_64 in the CUDA 13.3.1 image (exit 0; Zig prints no Build Summary without --summary).
    Uncached on 17f7d67d2: 379/379 steps, 6139/6178 passed, 39 skipped; flashnext module 270/270, with three new
    tests in cuda_rank_start_test.zig (socket pair, not on Windows):
    1. A peer's .ready 400 ms in, against a 50 ms collective bound, is awaited before the first collective.
    2. A peer closing its link during the wait gives RankDisconnected, no collective.
    3. cuda_lanes_load.zig calls r.afterLoad, not benchmark_group. directly, and registers
       errdefer own.engine.deinit(); before it.
  • Fail before: tests 1 and 2 do not compile without Session.afterLoad; with main's cuda_lanes_load.zig put back,
    test 3 fails (flashnext module: 269 pass, 1 fail); with 25a26afab's order (the errdefer after the wait), test 3
    fails.
  • zig build test test-golden -Dcpu=apple_m1 (macOS), on 75a849407, this head's code on main eab305db0
    (the patch-id without CHANGELOG.md is the same): "Build Summary: 837/839 steps succeeded (1 failed);
    7593/7667 tests passed (73 skipped, 1 failed)". The failure, dashboard_test "dashboard page polls the public
    stats snapshot without private details", is inherited: main eab305db0 fails it on the same Mac ("Build
    Summary: 837/839 steps succeeded (1 failed); 13041/13152 tests passed (110 skipped, 1 failed)").
  • zig fmt --check build.zig zig, git diff --check, zig build test-gradient-serving: pass on this head.
  • Device modes late, gone (2 x RTX PRO 6000, 1da83e13d): PASS (Receipt); again on ab6d2ceaf, with sum: PASS
    (rank 0 waited 35,061 ms for late; RankDisconnected after 2,956 ms for gone).
• CI: runs on this head.
• Not run, and why:
  • late/gone on main: main's tf-cuda-test lacks them.
  • eof, missing, timeout: unchanged code (late/gone branch off first).
  • A skew past 10 minutes, or a hung peer keeping its link open: now caught after 10 minutes, not 30 s.
  • bench_concurrent, prompt speed, sampled requests: the change runs once per start.
  • HIP, Metal, Windows: not reached.

Receipt

• Main against branch: unpaced served starts on 2 x GB10 (200G RoCE); main fb8fdf3d8 against this change on
  fb8fdf3d8 (cuda_rank_engine.zig byte-identical to this head's, cuda_lanes_load.zig apart from flashnext: refuse a malformed TF_FLASHNEXT_* setting by name before device work #2585's setting
  reads):

  Start	Rank 0 loaded	Rank 1 loaded	Outcome
  main	23:42:34	layer 36 of 48 at 23:43:02	rank 0 RankTimeout at 23:43:05, 31 s after its load; start failed
  branch	00:00:50	00:01:26, 36 s later	served at 00:01:32 ("loaded in 163.8s"); every reply has main's hash

  Not a same-conditions A/B (a build ran on rank 1's host during main's start).

  # main, rank 0
  23:43:05 tensorfold: the native CUDA engine cannot load <snapshot> with its own glue kernels (RankTimeout)
  # branch, rank 0
  00:01:32 [tensorfold] serving bench at http://127.0.0.1:8090/v1 (sampling: greedy; drafts: MTP + copies; context: 65536; loaded in 163.8s)
  

  Device modes at 1da83e13d, tf-cuda-test flashnext-rank RANK PORT MODE, two processes, one GPU each, real NCCL,
  production 30 s collective bound, 2 x RTX PRO 6000 Blackwell Max-Q (PCIe Gen4):

  Mode	Rank 0	Rank 1	Result
  late: rank 1 loads 35,000 ms late	waited 35,069 ms	waited 69 ms	PASS both
  gone: rank 1 leaves 1,000 ms in, as a failed load does (959 ms)	RankDisconnected after 2,961 ms (bound 10 s)	left	PASS both

• Boxes it reaches: Flash Next on CUDA at --tp 2, once per start; one-GPU Flash Next and other families unchanged.

• New switch? Default, accepted values, profile row: none.

• Environment, with the TensorFold commit, Zig and backend toolchain, machine and GPU, checkpoint and revision: Zig
  0.17.0; CUDA 13.3.1, -Doptimize=fast -Dcpu=native -Dsm=121; NCCL 2.32.3; 2 x GB10, 121.7 GiB each, Ubuntu
  24.04.4, drivers 580.178.04 and 580.159.03; nvidia/Qwen3.8-Flash-Next-NVFP4 revision
  fc694b54fb0174e0913e6adf86691ef85a4ead47, MTP drafts, --tp 2 --context 65536 --prompt-cache-gib 0, greedy. RTX:
  sm_120, driver 595.99.02.

• Paths and platforms the change reaches, which I ran and which I could not: the CUDA --tp 2 rank start; ran sm_121
  (served) and sm_120 (device modes); not run: a served start on sm_120 (covered by GB10 and late/gone).

• Exactness, from tools/bench_concurrent.py --alone --serial or token_sha against "draft": false: 3 greedy
  256-token requests per cell: chat 764da77fbd3f and code 4e759dbd859c in every branch cell, drafted equal to
  "draft": false and to a paced start of main.

• Main against this branch, same machine and checkpoint, posted before asking for a run on our machines (or why you
  can't): the tables above; rank logs, results, runner scripts and host-check records: https://gist.github.com/swbiggart/ddb5f44cffd779ee3a0842ca2d743d2d

• Decode speed before and after, from tools/bench_openai.py: no claim (once per start). Median tok/s, code drafted /
  plain, chat drafted / plain: main (paced) 51.79 / 23.26, 38.09 / 23.23; branch 51.80 / 23.08, 38.26 / 23.03.

• Prompt speed before and after: n/a: the change runs before the first prompt.

• Licence of any new weights or drafter, or "n/a": n/a.

Checklist

• The title and description match the head commit, including the receipt and the stack.

• New tests fail before the change, pass after it, and skip cleanly without their dependency.

• Comments and docstrings are one line. No measurements or history in the source.

• No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.

• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (Drafted equals "draft": false; resumed and concurrent not run: no per-request state changes.)

• No precision traded for speed. If the bits change, the description says which sums change and why.

• Decode and prompt processing are no slower than main at one stream and the widest admitted; with --drafter, no slower than without it, greedy and sampled (not compared: once per start; one start's decode is level with main's at one stream, greedy)

[swbiggart]

Thanks for the review on #2970.

Pushed as 7371ef346. errdefer own.engine.deinit(); now sits right after Engine.initRank, before the if (own.ranks) block, so RankDisconnected and the 10-minute bound free the engine before the session is poisoned. The source check in cuda_rank_start_test.zig now also expects the errdefer before r.afterLoad. With 25a26afab's order it fails, and on this head it passes. cuda_lanes_load.zig is re-pinned. zig build test-cuda-host and test-gradient-serving pass on macOS; lean is 345, 0 new.

[ashhart]

Thanks @swbiggart, 7371ef346 is what we asked for on #2970.

• errdefer own.engine.deinit(); now sits right after Engine.initRank (zig/src/families/flashnext/cuda_lanes_load.zig:66-67), before r.afterLoad (:72). So RankDisconnected and the 10-minute bound free the engine, and since errdefers run in reverse, that still happens before the session is poisoned.
• The source check in cuda_rank_start_test.zig:93-99 now expects the errdefer before the wait, and it fails with 25a26afab's order.
• Session.afterLoad (cuda_rank_engine.zig:156) runs the same room exchange both ranks already make at the end of open, in the same order on both. A slow peer is now waited for up to 10 minutes instead of 30 s, and one that fails or closes its link ends the wait at once.

No more asks on the change itself. One rebase is needed before it can land:

1. flashnext-rank: the failure modes start once both communicators are up, and timeout bounds only the collective #2976 lands first and flashnext cuda: at --tp 2 a prompt pass reduce-scatters its routed experts' slots by rows and gathers their sums #2977 after it, and flashnext cuda: at --tp 2 a prompt pass reduce-scatters its routed experts' slots by rows and gathers their sums #2977 conflicts with this in zig/tests/cuda/. Once flashnext cuda: at --tp 2 a prompt pass reduce-scatters its routed experts' slots by rows and gathers their sums #2977 is on main, please rebase onto it. In flashnext_rank_tests.zig, keep flashnext cuda: at --tp 2 a prompt pass reduce-scatters its routed experts' slots by rows and gathers their sums #2977's modes check and add late and gone to its list, so startWait is still reached. In zig/tests/cuda/main.zig, merge the two flashnext-rank usage lines into one that lists every mode.
2. Your pins for cuda_lanes_load.zig and cuda_rank_engine.zig in zig/tests/fixtures/gradient-serving.tsv can stay as they are, since flashnext cuda: at --tp 2 a prompt pass reduce-scatters its routed experts' slots by rows and gathers their sums #2977 doesn't change either file. Please run test-gradient-serving on the new head to confirm.
3. Post flashnext-rank late, gone and sum on your pair at the new head.

#2978 and #2980 touch the same usage text and pins, and #2978 the same recipe file. Whichever of those lands after this keeps both sides. You don't need to do anything about CHANGELOG.md. We sort that out when landing.

After that, we'll run late and gone and a served start with one rank loading well after the other on two GB10s, with token hashes against main.

• Synthena (Ash's Agent)

[swbiggart]

Rebased onto main 4cfd95573, which has #2976 and #2977, as fa5e3d56d, as you described:

• zig/tests/cuda/flashnext_rank_tests.zig keeps flashnext cuda: at --tp 2 a prompt pass reduce-scatters its routed experts' slots by rows and gathers their sums #2977's modes check, with late and gone added, so startWait is still reached.
• zig/tests/cuda/main.zig has one flashnext-rank usage line that lists every mode.
• The cuda_lanes_load.zig and cuda_rank_engine.zig pins stand; zig build test-gradient-serving passes.

tf-cuda-test flashnext-rank at ab6d2ceaf, this code on ab78aef90, on the one-host pair (2 x RTX PRO 6000, one GPU each, real NCCL):

Mode	Rank 0	Rank 1
late: rank 1 loads 35,000 ms late	PASS, waited 35,061 ms	PASS, waited 60 ms
gone: rank 1 leaves 1,000 ms in	PASS, RankDisconnected after 2,956 ms	PASS, left in 954 ms
sum	PASS	PASS

zig build test-cuda-host (Linux x86_64) passes, and lean is 341, 0 new.

[ashhart]

Thanks @swbiggart. The rebase keeps every rank mode in one usage line, the engine cleanup is registered before the load wait, and your new-head late, gone and sum results answer the requested pair checks. The serving pins and loader-order check pass too.

This now waits on one two-GB10 startup/serve run that we'll schedule, covering unequal load times, a peer leaving during load and token equality against main. No action needed from you; we'll handle the CHANGELOG placement.

• Synthena (Ash's Agent)