Repository navigation
docs: verify NCCL selection for switchless GLM TP4 - #2993
vdalesandro wants to merge 3 commits into
Conversation
|
Thanks @vdalesandro, and welcome. A note on which NCCL library the switchless ring needs is useful, you were careful about what it doesn't claim, and you found a real gap: GLM ignores
CI on this head is waiting for our approval, and we'll start it.
|
|
Updated in 3d0e9f3.
The separate shared-loader fix is implemented and independently reviewed: I stopped before opening that second PR because ATC's merge comparison found conflicts in the shared environment registry/test table with #1441, #1442, #1443, #1444, #1445, #1446, #1522, #2008, #2415, #2551, #2648, #2762, #2919, #2935, #2937 and #2938. The complete discovery snapshot enumerated 293 open PRs and compared all 53 same-file heads. The fix itself is only three existing files, 32 added lines and two removed. Which landing order or coordination target would you prefer for the environment-table entries? I have not opened a competing PR or changed the running machines. |
|
Thanks @vdalesandro. Two small edits. We'll make them when we land unless you'd rather:
CI on this head is waiting for our approval again, and we'll start it. It's docs only, so no device run is needed. On the
|
|
Opened the loader fix against main as #3106 (three files, +32/−2; callers of |
|
Thanks @vdalesandro. The public version references, four-rank gather excerpts, simpler startup check and shorter recipe answer all five points. Opening #3106 also closes the loader follow-up. This goes to the landing queue. We'll make the two editorial changes you left with us: wrap the long lines and change “private” to “newer”. No device run is needed for this docs change; #3106 keeps its separate checks.
|
…libnccl.so.2' (review editorial changes)
Scope
Docs-only follow-up to #1971: the NCCL routing support needed for switchless RoCE links on separate IPv4 subnets. No engine/runtime change and no claim that ring or split prefill now works.
Public version references
The first public upstream release with these two settings is NVIDIA NCCL v2.30.7-1:
The tested private binary reports 2.30.7 and demonstrates the routing below; its upstream commit/build provenance is not established, so I am not calling that binary an unmodified NVIDIA release.
Review changes
Reduced the note to eight lines. Removed the engine-loader gap and redundant closing disclaimer. Selection is now checked with every rank’s startup version and NCCL INFO routing records, not process maps/file hashes. The separate loader correction will be linked here once published; until that fix lands, the recipe retains a process-local library-path override.
Archived run evidence (anonymized)
TensorFold native 1.2.6, GLM EXL3, TP=4, four switchless-ring GB10 hosts,
TF_GLM_PREFILL_REDUCE=gatherin both attempts. The library change plus the two routing settings fixed startup/peer-warm-up: the 2.28.9 attempt timed out inibv_modify_qpentering RTR before readiness; the 2.30.7 attempt initialized all four ranks and completed the existing five short functional checks. This does not qualifyring/splitprefill or establish a performance improvement. No new run was started for this revision.Hostnames, IP addresses, process/thread IDs and communicator ID below are replaced with placeholders; the startup/routing text is otherwise preserved. Historical failure excerpts are the retained tail, not a reconstructed version log.
Failure, NCCL 2.28.9 (one retained line per rank):
Success: each rank's startup version, NIC list and observed routing overrides (
NCCL_DEBUG=INFO):One completed capital-city request, fields extracted from its captured result (not a new or synthetic response):
{ "http_status": 200, "content": "Paris", "finish_reason": "stop", "done_seen": true, "usage": { "completion_tokens": 2, "completion_tokens_details": { "reasoning_tokens": 0 }, "prompt_tokens": 21, "prompt_tokens_details": { "cached_tokens": 0 }, "total_tokens": 23 } }Validation
Eight-line note, public links/source definitions checked; archived four-rank logs and completed-response fields rechecked against their original receipts; privacy screening and
git diff --checkpass. No new GPU run, model request, deployment or baseline repetition.