Repository navigation
Nemotron 3.5 Transcribe on CUDA: the language model and the audio encoder - #1631
BobClawblaw wants to merge 8 commits into
Conversation
|
CI on Linux job · Windows job. The stacked #1632 also has the same seven crashes on Linux at This is from the CI logs; I have not reproduced it locally or established whether the fault is in the test setup or the cleanup path. |
|
Thanks for this, @BobClawblaw. It reads well: everything new sits behind the nemo_speechlm config, and the shared Nemotron loader changes keep the old tensor names for every other checkpoint. It rebases cleanly onto #1630, so we'll land it once #1628, #1629 and #1630 are in. Two small things before then, if you have time:
We also noticed that |
|
Thanks @cwschroeder and @ashhart. Pushed cb83265:
#1632 carries the same crash and will be rebased onto this. The drafted == |
|
Thanks @ashhart.
And yes, |
cb83265 to
bc2fbea
Compare
|
Correction to my comment above: cb83265 was replaced on this branch by bc2fbea (which carries the nemo_audio test step instead). The |
64098fe to
51f18b4
Compare
|
Rebased onto d377fcc (with #1630 under it). Main's growth pushed Checked: fmt, lean (0 new) and |
51f18b4 to
a69a13a
Compare
|
Thanks @BobClawblaw, both asks are answered, and thanks for the |
a69a13a to
282c782
Compare
|
Restacked onto #1630's 8fd6c58; the head is now 282c782. The one conflict was |
|
Thanks @BobClawblaw. The restack is clean, and the two sets' summaries on this head are what we wanted. One small |
282c782 to
091c0cc
Compare
|
Restacked on #1630's rebase (859efdd, main 933b969); now 091c0cc. The nemo_audio contract set sits beside main's Flash Next and EXL3 sets in |
|
Thanks @BobClawblaw. Both 30 Oct asks are answered and survive the rebase, the CI crash fix holds, and everything new sits behind |
091c0cc to
cb2c925
Compare
|
Done (cb2c925): |
|
Thanks @BobClawblaw: the push restacks only (the diff is byte-identical to what we reviewed), and nothing is owed from you. The stack lands in order (#1630, #1631, #1632) after our GB10 run, which needs Transcribe's BF16 release on a box first. |
cb2c925 to
c82ba78
Compare
|
Thanks @BobClawblaw. The fp32 vectors are still accepted only for a Transcribe checkpoint, with the five cases tested, and folding the |
c82ba78 to
0555383
Compare
|
Thanks @BobClawblaw. The rebase keeps both earlier changes intact: fp32 One question we should have asked earlier: the nemo_audio commit, and Since this push crossed code conflicts, the
|
|
Thanks. Straight answer: it follows NeMo's source. We cloned NVIDIA-NeMo/Speech at 7b05f096 and wrote a torch reference that mirrors it op for op, then ported to Zig/CUDA against that reference. The parts that follow NeMo:
The kernels in |
|
Thanks @BobClawblaw for the straight answer and the file-by-file list; that makes this easy to settle. Code that follows another project's source line by line is something we can't take, even with a notice, so the encoder in #1631 can't land as written. Please don't close anything, though: we'd like the model, and #1630 is unaffected and still lands on its own. The way forward is the rewrite you offered:
|
|
Thanks, we'll do the rewrite on those terms, and #1631 and #1632 are drafts until it's done. How it will be kept clean:
We'll post the stage-by-stage match and the requested contract and test summaries from the new head. |
|
That's exactly right, thank you. A writer who has never had the source open, NeMo run only as a black box, and the old commits gone is the clean way to do it. We'll read the new head when it's up.
|
0555383 to
1fb1354
Compare
|
The clean-room rewrite is up at |
|
Thanks @BobClawblaw for the rewrite and for the file-by-file sources. As far as the branch can show it, the encoder is clean. Neither diff has a torch reference or a NeMo script. The nemo_audio files come from one new commit, and none of the old encoder's commits or files are left in the history. The comments cite config.json, the papers and the dumps, not another project's code. On your MTP question: keep the splice in Line numbers below are on #1632's head (
After that we'll run two checks on the rebased head: Transcribe from the BF16 release with the six clips through
|
1fb1354 to
f5c1da6
Compare
|
Thanks. All seven items are addressed. The stack is rebased on main
The full serving run is in #1632's description: the ten clips as |
f5c1da6 to
d57fabc
Compare
|
A correction to the heads above: #1631 alone did not compile its tests. The new |
|
Thanks @BobClawblaw, all seven are in on
The stage tables with NeMo's own bf16-against-fp32 beside ours are what we asked for, and the speaker-cache sweep settles which key matters. A few small things (CI is already green on both heads):
Keeping our own
|
|
Thanks. All three are in: #1631's head is
The full kernel contract on the previous head's tree (the new commits touch no kernel): |
…nt blocks (relu^2 up, fp32 or bf16 down)
…and bf16 down, gates 1-5 with two controls
…, as pack does, with a test
…m) from its bf16 weights, audio rows spliced after the embeds The checkpoint's llm.* tensors load as published: NeMo's layer names, fp32 A_log/D/dt_bias, the stacked [E, in, out] experts on the bf16 grouped format with the shared expert as experts E and E + 1, TE's empty _extra_state skipped. The audio encoder and CTC aligner tensors are left for the audio path. A prompt pass's audio rows replace the <|audio|> positions' embeddings, as image rows do.
The Bolt load-recovery tests build an Engine by hand and null each optional kernel set; bf16x was left undefined, so deinit unloaded a garbage module (seven tests crashed on Linux and Windows CI).
…nscribe (salm) checkpoint, with a test
The perception encoder as a family of its own (zig/src/families/nemo_audio): WAV decoding, log-mel features and their per-feature normalisation, the ASR and streaming speaker transformers window by window (speaker cache and queue with AOSC compression, arXiv 2507.18446), the sinusoidal speaker fusion and the projection to the language model's width. Weights read as stored (bf16), perception.* only. nemotron/cuda_audio.zig holds the encoder, prepares clips (samples, rows, hash) and encodes each span at a prompt pass, up to an hour of audio. tf-cuda-test audio-check writes every stage under the reference dumps' names; audio-contract runs nemo_audio.cu's kernels through gates 1-4, each with a control built from nemo_audio.cuh.
…ore_threshold as the release does, with brief-speaker tests A speaker's frame score is log max(p, pred_score_threshold) plus the other speakers' log(1 - p); frames under 0.5 never score. A score above log 0.5 is positive, and a speaker with floor(share x min_pos_scores_rate) positive scores keeps only those (share = (188 - 8 x silence rows) / 8, rounded down: 22 shipped, so 11). The latest boost and the strong and weak boost counts follow, from the same share. When fewer than 188 frames can score, the rest of the cache is silence rows, after the scored rows. Fitted to black-box NeMo dumps at several values of both keys and of spkcache_sil_frames_per_spk; the tests use window 0 of a brief-speaker clip and of a clip with 37 s of silence, against the release's own caches.
6fdd49b to
d329cef
Compare
|
Rebased on main
The full kernel contract was not rerun after this rebase; the descriptions quote it from the stack on |
|
Full kernel contract after the rebase, on the GB10, on #1632's head
The same line as before the rebase. None of the limit problems is in this stack's launches, and none of its FAIL lines names a kernel from this stack apart from the planted controls. The descriptions now quote this run. |
|
Thanks @BobClawblaw. All three of this morning's asks are in, and they survive the rebase onto
Nothing is owed from you. We'll now run the two GB10 checks we promised, on #1632's head
If both are clean, this and #1632 land right after #1630.
|
What this changes
Nemotron 3.5 Transcribe (
nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B@c421d6acdb436edd7ae00565f6414631fb9bd56d,the bf16 release as published) on CUDA:
cuda_salm.zig), with exact bf16 caches andno conversion.
families/nemo_audioandkernels/cuda/nemo_audio.cuh/.cu(log-mel on the GPU, the 32-layer ASRencoder, the streaming speaker encoder with its cache, the fusion and projection).
tf-cuda-test audio-checkwrites every encoder stage, for comparison with a reference.Part of #1549; stacked on #1630.
How the encoder was written (clean room)
The previous encoder in this PR followed NeMo's source and was withdrawn. This one was rewritten from scratch, with
two separate roles:
torch reference. The only sources were the release's
config.json, the tensor names and shapes, the model cardand its architecture figure, the weights themselves, the Sortformer papers (arXiv 2409.06656, 2507.18446), the
SALM paper the card names (arXiv 2310.09424), textbook definitions (DFT, Hann window, LayerNorm, softmax attention,
RoPE (arXiv 2104.09864), GELU, sigmoid, ReLU, 1-D convolution) and TensorFold main.
7b05f096, installed unmodified and run as a black box in the served (online,windowed) mode, bf16 and fp32. Forward hooks dumped the stages you listed for ten clips: samples, log-mel before
and after normalisation, the stacked frames entering each encoder, per-layer states for one clip, both encoders'
outputs, the speaker-cache state before and after each window, the fused output, the projected rows, and the row
count and placement in the prompt. The writer saw only those numbers and a manifest of names and shapes.
Behaviour the config does not state was settled by matching the dumps. For example: pre-emphasis, the frame
offsets, the window and row geometry, the RoPE pairing, the speaker-cache sizes and its scoring per the Streaming
Sortformer paper. The served cache sizes differ from config.json's
sortformer_modules(spkcache_len264,fifo_len0,chunk_size_seconds30): the 188 cache rows, 40 queue rows and 40-second cores (4000 frames, with400 frames of context each side) come from the shapes of the dumps of NeMo's served mode, which uses those sizes for
this checkpoint. No torch reference is in the branch.
Two cache keys,
pred_score_thresholdandmin_pos_scores_rate, are not described in either paper. The writerfitted them to NeMo's caches. NeMo was rerun, unmodified, with only config values changed:
min_pos_scores_rateat0, 0.5, 0.52, 0.54, 0.55, 0.6, 0.7 and 1;
pred_score_thresholdat 0.1, 0.25 and 0.6;spkcache_sil_frames_per_spkat 1 and 2. The clips were four with a brief second speaker, a 48-clip sweep,
multispk_longand one with 37 s ofsilence (the commit
d329cef88).nemo_audio/wav.zignemo_audio/config.zigconfig.jsonkeys; window and cache sizes from the dumps' shapes; the card (one hour of audio)nemo_audio/weights.zigradio/weights.zig's planner patternnemo_audio/speaker.zigsortformer_modules; the dumps (sizes, which rows survive, and the two keys' variant runs)nemo_audio/encoder.zig,kernels.zig,nemo_audio.zigkernels/cuda/nemo_audio.cuh,nemo_audio.cumoss_image.cu's style andglue_math.cuhnemotron/cuda_salm.zigmodel.safetensorsheader) andconfig.json'sllm_config; the Nemotron family's existing loaders for its HF checkpoints (cuda_weights.zig,cuda_modelopt.zig) and main's Nemotron MTP headnemotron/cuda_audio.zigcuda_vision.zig's begin/end/prepare patterntests/cuda/audio_check.zigcontract_nemo_audio_controls.cuh,tests/cuda/contract_nemo_audio{,_ref,_feats,_rows,_attn}.zig,contract_family_sets.zig,contract_limits.zigdocs/kernel-contract.md,core/contract/gates.zig;contract_moss_image*.zigandcontract_attn.zig; the kernels' own definitions for the float64 referencescore/contract/nemo_audio_cuda_records.zig,registry_joins.zig,registry_sets_test.zigkernel-contract-limitsoutput;moss_image_cuda_records.zig's formdocs/kernel-contract.mdparagraphReceipt
d329cef88, on main6c5cef894with cuda experts: a bf16 format for a checkpoint's own bf16 experts #1630'sthree commits rebased onto it. Main's CUDA EXL3 grouped experts: decode launches and sizes storage at the 64-row prompt tile #2823 added
pairsandtiletoGrouped.decodeandpromptand movedtheir refusal test to
experts_host_test.zig; the bf16 tests now pass both, and the bf16 refusal test sits besidemain's. The serving checks and CI ran on the stack's top (Nemotron 3.5 Transcribe: audio input, the 65,536-token window, an hour of audio, MTP drafts #1632), which leaves these commits unchanged.
including a 42 s clip that takes two windows and one whose frame count is a multiple of 8). Each cell is the largest
difference over the stage's largest value, the lowest row cosine, and the rows below 0.999 cosine; NeMo's own bf16
against fp32 is beside ours. The projected rows (the full tables are at the end):
ls_1272_135031_0009ls_6930_75918_0000ls_1272_128104_0000twospk_1272_6930frames_mult8frames_4200(42 s)ls_1272_128104_0004mid34s_1272long130s_1272multispk_long(153 s)7 and 12, 17 and 30). On
long130s_1272our worst row is further off than NeMo's (0.297 against 0.886).mid34s_1272's rows first fall below 0.999 at layer 17(row 34); NeMo's bf16 and fp32 runs part there on the same row, and on row 132 from layer 13.
long130s_1272'sfirst move at layer 14 (window 1, row 334) and layer 13 (window 2, row 44); NeMo's runs part at layers 12 to 17.
Windows 0 and 3 have no row below 0.999 at any layer. At every layer of every window, no more of our rows move
than differ between NeMo's bf16 and fp32 runs.
long130s_1272's worst row(final row 994, window 1's row 544, six rows before the core ends) is one:
to NeMo fp32, against NeMo bf16's 0.99993).
is at 1,020, and the row has become a massive-activation row that float64 does not have. Dim 862 carries 21 of
the model's own 60 massive-activation rows at layer 31, which match across all three runs.
cosine 1.000000 (within one or two bf16 steps), runaway included (43, 55, 125.5, 342, 704).
to 1,272; cosine 0.21 to float64). NeMo's bf16 run starts to tip row 543 too (0.94 at layer 17, 0.88 at 18)
and recovers to 0.97. Which of the two rows grows dim 862 is decided by bf16 rounding.
input begins with the cache, so its rows compare by position only where the caches agree (hence the speaker
encoder's many moved rows in the full tables, ours and NeMo's alike). As sets of frames, our cache shares more
with NeMo's fp32 cache than NeMo's bf16 cache does, at every window of these clips (of 188 rows):
frames_4200w0long130s_1272w0 / w1 / w2multispk_longw0 / w1 / w2on every clip, and the replies have 1.55% WER against the LibriSpeech text (17 of 1,095 words, nearly all names
and spellings). Whether the replies equal NeMo's bf16 replies token for token was not checked: it needs NeMo's own
language model run, which this machine could not hold beside its other work.
pred_score_thresholdandmin_pos_scores_rateas NeMo applies them(
sil_thresholdchanged no cache on any clip, in either precision, and is not read):pred_score_threshold) + the other speakers' log(1 − p); frames withp < 0.5 never score. A score above log 0.5 is positive, and a speaker with at least floor(share ×
min_pos_scores_rate) positive scores keeps only those. The share is (188 − 8 × silence rows) / 8 rounded down(22 as shipped, so 11). Then the latest boost and the strong and weak boosts, with counts from the same share.
When fewer than 188 frames can score, silence rows fill the end of the cache.
pred_score_thresholdas a floor (no score under 0.25); that was wrong.(every variant, the sweep, the silence clip), 80 keep NeMo's 188 rows exactly, and in the other 116 every miss
is an exact fp32 tie at the cut. The bf16 runs agree on the brief speaker's rows at every value. Unexplained:
pred_score_threshold0.1 changes which tied frames NeMo keeps, although under this rule no score changes.brief_spk_1at rates 0, 0.5 and 1 and at threshold 0.6, and window 0 of the silence clip,against NeMo's caches row for row; two silence rows a speaker; the 0.54 / 0.55 boundary.
d57fabcd9). The brief clips' caches,as sets of frames shared with NeMo's fp32 cache, move only where the brief speaker is cached:
brief_spk_2w3brief_spk_4w1 / w2 / w3SUMMARY nemo audio: 9 kernels, 9 pass gates 1-4, 0 fail; 8 controls: 8 red as planted, 0 missedSUMMARY bf16 experts: 6 kernels, 6 pass gates 1-4, 0 fail, 0 known findings; 2 controls: 2 red as planted, 0 missed6f2f64e59(which carries this head unchanged):SUMMARY 658 kernels: 658 pass gates 1-4, 0 fail, 8 known findings; 424 controls: 424 red as planted, 0 missed; 488 limit problems, the same line as before the rebase. The limit problems are all in other families' launches (e.g.qmmf, and main'sdf_sumn). Every launch in this stackpasses its limits; the nemo_audio records were made with nvcc 13.2 (13.0 was not available).
test-kernel-contract,test-golden, the security tests and the fuzz test pass;zig build -Dfixed-cpu testpasses exceptdashboard_test's "polls the public stats snapshot", which fails on main6c5cef894alone too.test-cuda-hostpasses on the GB10 (this head). Lean reports nothing new.nemotron/config.zigreadsnemo_speechlm. No A/B on other Nemotron checkpoints was run onthis head.
Overlap
atc overlap --jsonafter this rebase (2026-11-12, the stack on main6c5cef894) found conflicts with four PRs. A mergecheck against each of them, and against the three it found last time, gives these files; all are shared registration
lists, the backend table or the contract doc, and whichever lands second rebases.
native/cuda_families.zig,tests/cuda/contract_limits.zig.kernels/cuda/contract_controls_set.cu,core/contract/registry_joins.zig,tests/cuda/contract_limits.zig.kernels/cuda/contract_controls_set.cu,core/contract/registry_joins.zig,tests/cuda/contract_family_sets.zig.core/contract/registry_joins.zig,tests/cuda/contract_limits.zig.native/cuda.zig.docs/kernel-contract.md.Its scan hit its PR, file and time limits (326 of 336 overlaps unchecked), so it is incomplete.
Stage by stage, all ten clips (largest difference over the stage's largest value / lowest row cosine / rows below 0.999)
ls_1272_135031_0009ls_6930_75918_0000ls_1272_128104_0000ls_1272_128104_0004twospk_1272_6930frames_mult8mid34s_1272frames_4200long130s_1272multispk_long