Skip to content

glm: resolve TF_NCCL_LIB in the shared NCCL library loader - #3106

Open
vdalesandro wants to merge 2 commits into
ashhart:mainfrom
vdalesandro:fix/glm-native-nccl-library
Open

vdalesandro wants to merge 2 commits into
ashhart:mainfrom
vdalesandro:fix/glm-native-nccl-library

Conversation

@vdalesandro

Copy link
Copy Markdown

Requested separately in #2993 (review by @ashhart).

Change

Native GLM called the shared cuda.nccl.Library.open(), which always opened libnccl.so.2, while fabric/tp already selected TF_NCCL_LIB. Resolve the override in the shared loader so GLM and its sibling callers use the same explicit selection.

Callers of nccl.Library.open() (zig/src/cuda/nccl.zig:80), all verified at these lines in this checkout:

  • GLM: zig/src/families/glm/cuda_native.zig:264
  • Flash Next's --tp 2 collective: zig/src/families/flashnext/cuda_collective.zig:25
  • MiniMax M3: zig/src/families/minimax_m3/cuda_native.zig:159
  • Shared TP group: zig/src/cuda/tp_group.zig:35

Reuse the native environment path validator: unset, empty and whitespace-only values keep libnccl.so.2; nonblank paths retain their exact bytes; control characters are refused before loading. openPath, required-symbol resolution, cleanup, minimum-version checks and communicator behavior are unchanged.

Three existing files change: the loader and its inline CPU regression; one environment rule; one entry in the existing environment valid-case table. No new dependencies or GPU work.

Verification

With the repository's exact Zig 0.17.0, CPU-only:

  • Focused regression: RED on the unchanged default, GREEN after the fix.
  • Invalid-path refusal: RED without validation, GREEN through the existing path validator.
  • Direct native loader boundary: 8/8 tests passed.
  • zig build test-runtime-host: 328/328 tests passed.
  • zig build test-environment-host: 23/23 tests passed.
  • zig build native: 8/8 build steps succeeded; the built executable reports tensorfold-native 1.2.7.
  • zig fmt --check and git diff --check: passed.
  • Same lean checker against the pinned base: 345 inherited findings, 345 on the candidate, zero new.

The successful native executable build contains the final production changes. A later rebuild after adding only the environment test's valid-case row exceeded the local 180-second time limit; final runtime/environment tests and formatting/lean replay passed after that row was added. Initial missing sparse-checkout inputs and the test-table coverage failure were corrected; their failing receipts are retained locally.

No GPU, actual NCCL runtime, checkpoint inference, performance, complete no-GPU CI or release qualification is claimed. Native host checks prove path selection and validation, not device execution.

Once this lands, the switchless-ring recipe in #2993 can point at TF_NCCL_LIB in place of LD_LIBRARY_PATH.

@ashhart

ashhart commented Nov 14, 2026

Copy link
Copy Markdown
Owner

Thanks @vdalesandro, this is a clean fix. Routing TF_NCCL_LIB through the shared loader means GLM, Flash Next's collective, MiniMax M3 and the shared TP group all honour the same override. Unset, empty and blank values keep libnccl.so.2, so nothing changes for anyone who doesn't set it, and the test covers each case.

One thing before it can land:

  1. The serving pin for nccl.zig needs updating. zig/src/cuda/nccl.zig is a pinned serving file, so zig build -Dfixed-cpu test-gradient-serving fails on this branch with "serving source changed: zig/src/cuda/nccl.zig". The pins run on every pull request now, so CI will show the same. The fix is one line: in zig/tests/fixtures/gradient-serving.tsv (the zig/src/cuda/nccl.zig row), set the hash to 0e36d9d84d3194d0cb63a73c3f66533e60b4da0ef08fa81bc2851e384b28b7b3 and add this PR's number to that row's commits column, as the other re-pinned rows do.

Optional: zig/src/fabric/tp/config.zig:119 still reads the variable raw, so a set-but-blank TF_NCCL_LIB= opens "" there rather than the default. Routing it through configuredPath() would make the two paths agree.

  • Synthena (Ash's Agent)

… re-pinned

Review follow-up for ashhart#3106.

- zig/tests/fixtures/gradient-serving.tsv: the zig/src/cuda/nccl.zig row carries
  the new hash and this PR's number, so test-gradient-serving passes again.
- Library.pathFrom(raw) holds the one rule (a path is kept as written; unset,
  empty and blank keep libnccl.so.2; a control byte is refused). Library.open
  applies it to the process environment as before, and fabric/tp/config.zig
  applies it to its own getter, so a set-but-blank TF_NCCL_LIB no longer opens
  a blank path there. A malformed value is error.BadSetting, like the other
  TP settings.

Checked on a CPU-only host with Zig 0.17.0: test-gradient-serving (fails before
the re-pin with "serving source changed: zig/src/cuda/nccl.zig", passes after),
test-tp (the new config test fails on the old line and passes on the new one),
test-runtime-host, test-environment-host, the nccl.zig tests, zig fmt --check
and the lean check (0 new). No device run.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants