[swbiggart]

What this changes

tf-cuda-test flashnext-rank RANK PORT timeout fails on main on both two-GPU machines I have: it hands
Collective.init the 200 ms bound meant for the injected stall, and NCCL's initialization takes longer than that,
so both ranks stop with RankTimeout inside the initialization. Now both ranks initialize under the usual 30 s bound,
meet over the control link once both communicators are up, and only then does timeout set the collective's bound to
200 ms and rank 1 go silent (or, for eof, leave). The meet also closes a race on two hosts: a rank that finished
initializing first could leave before its peer had, failing eof inside the initialization instead of the
collective. The final handshake also uses meet. Test code only; nothing served changes.

• Issue: Fixes Flash Next on CUDA: flashnext-rank timeout stops both ranks inside NCCL initialization #2975. Refs Roadmap: one N-rank layer for TP, PP and EP, shared by every family #2560 (the multi-rank roadmap); completes none of its items.
• Stacked on: main (3885dceb2). One commit, 35a084d79 (zig/tests/cuda/flashnext_rank_tests.zig, +17 -8).
• atc overlap --json (ATC 0.9.6, 2026-11-11 11:25 UTC, base 93a2e93ec): complete: false; its four conflicts entries (GLM on CUDA: the latent split's per-rank page book and owner (TF_GLM_KV_SPLIT, #1431 part 1; stacked on #1411) #2415, dsv41 metal: the lane target, the prompt cache and the native server over Metal #1720, dsv41 metal: the vision tower's kernels under the kernel contract #1721, dsv41 metal: image prompts #1722) come from another of my open pull requests and already conflict with main in CHANGELOG.md. atc who: no pull requests or agents; no open pull request touches the file. Re-run right before this pull request opened: atc overlap --json at 2026-11-12 14:06 UTC on head 35a084d79; no conflicts entries.

Checks on this head

• Head commit: 35a084d794a2d3e22da544cdb2329e96395d8afb
• Lean checker, python3 tools/zig/lean_check.py, on the base and on this head: base 3885dceb2 345 (inherited),
  head 345, the same findings, 0 new (python3 tools/pr_check.py --base 3885dceb2: 0 errors, 1 advisory warning,
  no PR number).
• Tests run, with the command and pass, fail and skip counts:

  • zig fmt --check build.zig zig, git diff --check: pass on this head.

  • zig build test-cuda-host (it does not compile zig/tests/cuda) passes on this head on Linux x86_64 (CUDA 13.3.1
    image) and on e2f0b7f15, the same commit, on macOS (exit 0; Zig prints no Build Summary without --summary). zig build test test-golden -Dcpu=apple_m1, run on d04c5250c, this commit
    on main eab305db0 before the rebases onto 4f8317dac, 000b534a2 and 3885dceb2 (git range-diff: =): "Build Summary: 837/839 steps
    succeeded (1 failed); 7587/7661 tests passed (73 skipped, 1 failed)". The failure, dashboard_test "dashboard
    page polls the public stats snapshot without private details", is inherited: main eab305db0 fails the same test
    on the same Mac ("Build Summary: 837/839 steps succeeded (1 failed); 13041/13152 tests passed (110 skipped, 1
    failed)").

  • The changed file is in the CUDA device runner; it was built for sm_121 (GB10) and sm_120 (RTX) with
    zig build install native in a CUDA 13.3.1 image, -Doptimize=fast, and run as two processes, one rank a GPU.
    Fail before, pass after:

    Machine	Build	timeout	eof	sum
    2 x RTX PRO 6000 Blackwell Max-Q, one host	main 93a2e93ec	FAIL 1/1: both ranks RankTimeout in NCCL's initialization	PASS	PASS
    same	the fix, 10bef7792	PASS 1/1	PASS	PASS
    2 x GB10 (200G RoCE), one rank a host	main 7d97cc656	FAIL 2/2 (same signature)	PASS 1/1
    same	the fix, 10bef7792	PASS 2/2	PASS 2/2	PASS

    The fix's device runs were measured on 10bef7792, before the rebases onto f0d8c8d50, eab305db0,
    4f8317dac, 000b534a2 and 3885dceb2: 10bef7792, d04c5250c, e2f0b7f15 and 9a2858f70 are this commit (git range-diff: =), and the rank test, its
    collective and NCCL wrapper are identical from 7d97cc656 to 3885dceb2 (Flash Next changed only in flashnext: refuse a malformed TF_FLASHNEXT_* setting by name before device work #2585's
    setting reads, in eab305db0), so the measured code is unchanged.
    missing (one process alone) passes before and after.

• CI: not yet run.
• Not run, and why: HIP, Metal and Windows (the test is CUDA-only).

Receipt

n/a: test code only; it cannot reach inference. The device runs above are the check.

Checklist

• The title and description match the head commit, including the receipt and the stack.
• New tests fail before the change, pass after it, and skip cleanly without their dependency. (No new test: the existing timeout mode fails before and passes after; the runner needs two GPUs.)
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.

[ashhart]

Thanks @swbiggart, and for the clear report in #2975.

The fix is right. Collective.init now gets the usual 30 s bound, and only the collective under test sees 200 ms (zig/tests/cuda/flashnext_rank_tests.zig:42). The meet at sequence 98 (line 47) means neither rank fails before both communicators are up, so timeout and eof now test the collective instead of NCCL's start-up. Moving the final handshake into meet is a nice cleanup too.

35a084d794 is the same commit as 9a2858f70, which passed CI, moved onto today's main, and it merges cleanly. It's test code only, so nothing served changes. Landing it. #2977 is stacked on this commit, so this goes first.

• Synthena (Ash's Agent)

[ashhart]

Thank you @swbiggart. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes.

[ashhart]

Thanks @swbiggart! This is in TensorFold 1.2.8, out now: https://github.com/ashhart/TensorFold/releases/tag/v1.2.8

• Synthena (Ash's Agent)