Repository navigation
Conversation
|
Ready for review. #2225 has landed, so I rebased onto main
A Metal binding of the same schedules passes on an M3 Pro (3 dev clips byte-identical). It follows as its own PR, stacked on this one. |
|
Thanks @jschmied. 38 of 40 clips byte-identical, the other two at a bf16 tie and a 0.125 near-tie, with stage-by-stage cosines beside them, is a strong first gate. Two things before it can go further. First, this adds the family and uses the shared-schedule layout from #2118, and both still need a decision on our side; we'll answer on #2105 and #2118, so please keep this as a draft until then. Second, a small one for whenever it moves: |
|
Thanks. Back to draft until #2105 and #2118 are answered. Shared kernels at nvcc 13.0. I compiled
So every kernel main already has compiles to the same registers, shared and constant bytes and the same instructions; the PR only adds the three functions, and their registry rows are in the PR. nvcc
Also on this push: |
|
Thanks — the host/CUDA split is easy to follow, and tapping every stage in the case generator is useful. I went through the host and CUDA paths and found two issues I think should be fixed before this is usable as a 0.6B/1.7B gate, plus a few smaller points. 1. A single chunk shorter than 100 frames uses the wrong convolution geometry
This is not only the shape of the conv dumps: the right-edge conv2/conv3 outputs read the extra padded rows, so the kept rows can differ. I reproduced it with a 50-frame (0.5 s) clip:
Suggested fix: when 2.
|
|
Thanks @shuuul, all fixed on
Host checks on this head, each step with its own exit code, all 0: The GPU on our box is down until a reboot. The CUDA re-run follows after it: the 0.6B and 1.7B 40-clip gates plus the 50- and 99-frame cases with stages. I'll post the results here. |
|
The CUDA gates on
All four differences are near ties against the package, with the top two logits at most 0.25 apart on either side:
The two Since the tools now exit 1 on a differing transcript, these clips fail by exit code, which is what a gate should do. If you'd rather have a margin rule for exact and near ties, say which, and I'll add it. |
|
Thanks @jschmied for taking every point from @shuuul's review, and for the GB10 rerun with each difference listed with its margins. Thanks too, @shuuul, for the careful read. Nothing is needed from you while this is a draft. The family and layout questions on #2105 and #2118 are still with the maintainer, and we'll answer your margin-rule question with that decision. As we said on #2105, the audio route (#1632) comes first.
|
|
Thanks @jschmied for the GB10 rerun with every margin listed, and @shuuul for the careful review. The family is a yes (#2105), and this layout is fine for a new family (#2118). On your margin question, here is the rule. It's the one we used on #2648.
Before it leaves draft:
The windowed reference is right, with the global comparison listed as you have it. Since this doesn't touch the server, it can land before #1632. The served path waits for #1632.
|
|
@ashhart thanks. Everything below is on 1. Rebase.
Tie rule.
So the 0.6B passes 44 of 45 clips and the 1.7B passes 45 of 45. Every other step matches. The failing step isn't a swap. In the package's bf16 run our 468 is third: 9197 25.75, 36 25.5, 468 25.375. The package's fp32 run of the same step puts 468 first, at 25.7277 against 9197's 25.7162. Under the rule as written it fails. I've left the rule alone and listed the step. Planted changes on dev clip 0000:
2. One Qwen3 forward on CUDA through #2285. Here's what I propose. I'll build it once #2285 lands, and nothing here depends on it yet.
3. Decode against prompt, two passes against one.
4. Distance to the package's fp32 run (teacher-forced on the package's greedy ids, final logits, worst column / mean |diff|).
Today's path isn't uniformly closer to fp32 than the package's bf16. On 0.6B 0000 its mean is 0.133 against 0.102, and that's the clip where the multi-row algorithms differ most. One algorithm a shape brings that mean to 0.090. I'll repeat this table through #2285's pass. 5. Provenance. The description now has the line, worded to match what I actually read: TensorFold's code, the checkpoint's config files, and the publisher's own reference package qwen-asr 0.0.6. From that package I read its transformers-backend modeling and processor, its inference wrapper, and a search of its vLLM-backend file for the attention windows. I also checked the log-mel front end against transformers' Verify on |
|
Thanks @jschmied. The rebase, the SASS table, the tie-rule run with every split listed, the fp32 distances and the provenance line answer what we asked. The #2285 plan (one Qwen3 layer path on CUDA, the schedule in Two decisions from our side:
One small thing for now: the exact-tie swap in Once it runs through #2285's pass, please post the tie-rule run, the fp32 distance table, and
|
|
@ashhart thanks. On
Verify on |
|
@jschmied yes, that shape works for us: one Qwen3 layer path on CUDA, the start, splice and head read only when set, and the embedding hashes on main against the new head. Three things from our side:
For the embedding hashes: we're rerunning #2285's receipts for the 0.6B, 4B and 8B on its rebased head now, and we'll post the per-vector hashes with them, so your comparison has a fixed baseline. |
|
Thanks @jschmied. The exact-tie order now lives in Nothing is needed until #2285 lands. When you rebase onto it, two small things from today's main:
|
…hart#2105) Qwen3-ASR's 0.6B and 1.7B admitted from config.json: the audio tower's geometry (chunks of 100 frames, windows of n_window_infer frames) and the Qwen3 language model's, its interleaved M-RoPE refused unless its sections span the head (audio positions are equal on all three axes, so it is plain RoPE). Every stored tensor checked by name, shape and dtype, none left over; the stored lm_head must be bit-equal to the tied embedding. WhisperFeatureExtractor's log-mel rows in float64, and the tower in float64 stage by stage, either exact or at torch's bf16 rounding points. The checkpoint ships no tokenizer.json; the shared tokenizer reads its slow export (ashhart#2225). tf-qwen3-asr-check compares weights, prompt ids, the reply's decode, mel rows and every tower stage with a case from tools/zig/qwen3_asr_case.py (the qwen-asr 0.0.6 package's transformers backend, hooked stage by stage).
…ing (ashhart#2105) tower.zig and lm.zig load weights by name through the binding's put and run the schedule through r.conv, r.linear, r.attention and the rest; they import no backend. schedule_test.zig runs both on a host binding: the tower matches the float64 reference at torch's points (global attention fails it), and the LM's ops stay inside their buffers.
…case's bf16 and fp32 logits (ashhart#2105) Each clip runs on the reference's greedy ids to the end. A step whose top token differs passes only if the two tokens swap places and both margins are at most 0.25 logit; every such step is listed, anything else fails with exit 1. With logits_fp32 and logits_bf16 in the case (qwen3_asr_case.py --logits), it prints our distance to the package's fp32 run beside the package's own bf16 distance. --rows K checks the decode steps against one prompt pass and a prompt split in two passes against one, on CUDA.
…y round, plants swapped q and k norms, and --row-gemms (ashhart#2105) topk may list an exact tie (margin 0) in either order; the greedy id is the reference's pick. --plant norms swaps q_norm and k_norm in every layer, a planted change that moves real tokens. --row-gemms runs every projection a row at a time, so --rows can tell cuBLASLt's per-row-count algorithms from the schedule's positions and caches.
…o forced.zig, with its test (ashhart#2105) forced.reference takes the reference's step from its topk pair and greedy id: an exact tie (margin 0) listed the other way round is the greedy pick's; anything else whose first token is not the greedy id is refused. Its test fails without the swap.
…ith a start position, an audio splice and a tied head over chosen rows (ashhart#2105) qwen3/cuda/lm.zig's forwardFrom takes an Extra: the input's first position, every layer's own k and v caches, rows spliced over the embedded ones, the final norm and the tied head over the pass's last rows, and stage taps. Each is read only when set: forward(.{}) issues the embeddings' launches with the same arguments. qwen3_asr/cuda/lm.zig (the schedule's CUDA binding) goes; the ASR checkpoint loads through weights.Model.fromCheckpoint under its thinker.model. scope, and the position and RoPE uploads are lm.tables, shared with the engine. tf-qwen3-asr-forced drops --row-gemms: every projection keeps one algorithm per (n, k).
… its caches before any launch, with its tests lm.admit names the refusals (InvalidBatch, ContextFull); its test and Splice.within's run in test-qwen3-cuda-host.
|
@ashhart #2285 is in, so here is the rebase and the run through its pass. Head
Verify on |
|
@ashhart @Arminova here are the embedding bytes I owed in the r5 reply. #2285's vectors don't move on this head. Main
|
|
@jschmied here are the token ids behind the per-vector hashes in #2285 (#2285 (comment)): https://gist.github.com/Arminova/416fab254972967b7e4307e5fdea25ed
|
|
Thanks @Arminova for the id lists. All 8 input sha256 matched in each file before any run. On head
Part B is #2995. When #2995 lands, I'll move |
|
Thanks @jschmied, and thanks @Arminova for the id lists. Byte-equal vectors on all three sizes, from a second GB10 build, is a strong cross-check. #2285 is on main as of last night, so this can rebase whenever suits you. #2995 (part B) is in its first review now, and we'll post on it shortly. Moving
|
|
@ashhart thanks. The rebase is already done: |
|
Thanks @jschmied, and sorry we missed this question in our last reply. Here is the call on 0.6B Why: one Qwen3 layer path on CUDA with #2285's fp32 norm weights and RoPE table is worth more than matching one bf16 step. On #2285's numerics your distance to the package's fp32 run went down (0.6B The rule for listing a step, fixed now so it covers both: the package's own bf16 and fp32 runs put the two tokens in opposite orders, and either our pick is the fp32 pick or our distance to the fp32 run on that clip is no worse than the package's bf16 distance. Step 10 meets it (bf16 has 9197 first, fp32 has 468, which is ours). Two things before it leaves draft:
On #2995: this doesn't need to wait for it. Both PRs change
|
…s audio tower (ashhart#2105) The three pinned serving sources gain only new code: a vision_kernel<64, 8> instance, tf_vit_conv_cols and tf_vit_conv_flatten, and their wrappers. The existing kernels' SASS is unchanged on sm_80/86/89/120/121.
|
Thanks @ashhart. Head 1. 0.6B
Step 3 alone is the clip with the case's prompt extended by the forced ids of steps 0-2. The step 0-2 and 0-3 slices give the same step-3 means by difference. By your rule, step 3 doesn't qualify. The whole clip is closer to fp32 than bf16 is, but step 3 alone is clearly worse: mean +41%, and our worst column there is 18284 itself (ours 5632 by 0.625; fp32 has 18284 by 0.281, bf16 has 5632 by 0.125). So I haven't listed it: the description has the rows and keeps it as the open fail, and the PR stays draft. Happy to look at it with you. 2.
The branch alone sits on #2995: agreed. Whichever lands second takes the other's changes in |
|
Thanks @jschmied for the What matches. Thank you for these numbers:
0.6B The look we promised. We said we'd look at it with you if step 3 came out worse, so here is what we'd like to see. Please post the layer-by-layer stage compare at step 3 against the package's fp32 run: the stages Re-pin. The three re-pinned rows in One small thing you can do now: add the CHANGELOG line from your description under Unreleased, ending with Before it lands we'll run our own copy check against qwen-asr 0.0.6, as we said on 10 Nov, and a short GB10 check that Qwen3-Embedding vectors from main and this head are byte-equal. If this lands before #2995, moving
|
|
Thanks @ashhart. Head Stage compare, 0.6B The last column is a logit lens: each layer's row through the final norm and the 18284 and 5632 head rows, in fp64. It
Head logits, 18284 / 5632:
Measured from fp32, ours is 0.18 behind on the lead at layer 18, 0.62 at layer 21 and 0.91 in the head logits. bf16 Note: this bf16 pass covers 25 rows, so its head logits are one bf16 step lower than the 30-row logits case (22.625 / CHANGELOG:
Full table (every stage) and the scripts: on request. |
|
Thanks @jschmied for the stage compare at step 3, which is what we asked for, and for the CHANGELOG line on Below, "lead" is 18284's logit minus 5632's, read through your lens. Where 18284 loses its lead.
No stage is out of line. Our mean is level with or below bf16's through the tower and layers 0 to 20. After that it is at most 1.29 times bf16's (layer 25: 0.2126 against 0.1653). The one large worst column is layer 21 (0.860 against 0.431), and it is level again at layer 22 (0.655 each). The final norm and head barely move the lead: ours reads -0.637 through the lens at layer 27 and -0.625 at the head. Our arithmetic or precision. This step is a near-tie. fp32 picks 18284 by 0.281. At logits near 22.5 one bf16 step is 0.125, so that is just over two steps, and the package's own bf16 run flips it too. A defect in our arithmetic, like a wrong scale or a misplaced row, shows as a stage far out of line with bf16, and on every step through it: your planted splice gave a worst column of 14.45 against about 1. Here no stage is. On the clip as a whole ours is closer to fp32 than bf16 is in both columns (0.667 / 0.0867 against 0.938 / 0.1226), and its mean is closer in 5 of the 6 dev cells. Your earlier runs point the same way: the move onto #2285's pass, with its fp32 norm weights in place of bf16-rounded norms, took this step from a 0.25 margin to 0.625, while it brought the 0.6B So we read it as precision: where bf16 rounding falls, ours and the package's, against an fp32 reference on a two-step tie. We can't point to a defect, and nothing here asks for a code change. What follows.
The full table isn't needed for this. The rows you posted are enough to sit beside the fail in the description.
|
|
Thanks @ashhart. Agreed: step 3 stays an open fail, the PR stays draft, and the new gate is fixed in writing before Clips (point 1): 20 from each of the gate's two public sets, 40 per checkpoint, taken by the gate's own rules one
How it runs (points 1-5):
The 40 clips, with SHA-256
LibriSpeech hashes are of the published FLAC members. The 20 FLEURS clips aren't on disk: I'll fetch the pinned The description now has the step-3 rows beside the open fail. |
|
@ashhart the 20 FLEURS clip hashes I promised, before anything runs. The tarball is
|
Refs #2105. Rebased on main
000b534a2(with #2285); head2fe5b72d9. Draft: one gate step is open (below).On top of
51dc03072,89d85b860re-pins three rows ingradient-serving.tsvand2fe5b72d9adds the CHANGELOGline. Every run and check below was made on
51dc03072; the code they cover is the same.The first slice of #2105: the family's host parts, a tower schedule shared by every backend, a Qwen3 decoder schedule
for the host reference and the Metal and XPU bindings, and the CUDA path, measured on a GB10. It doesn't touch the
server and doesn't depend on #1632.
It reads the checkpoint's Qwen2 slow-tokenizer export through #2225, now on main, because the checkpoint ships no
tokenizer.json.
Shape
families/qwen3_asr/(host, any OS):tower.zigandlm.zig: the schedules, generic over a bindingr. They never import a backend.schedule_test.zigruns both schedules on a host binding. The tower must match the float64 reference at torch'sbf16 points (global attention fails it), and every LM op must stay inside its buffers.
families/qwen3_asr/cuda/: a device store, the tower over vit.cu's LayerNorm, pointwise and cuBLASLt Linearkernels, and
model.zig, which runs the language model through qwen3 cuda: Qwen3 embedding checkpoints (BF16) on the native CUDA engine (#2261, 1 of 2) #2285's Qwen3 pass (next section).One Qwen3 layer path on CUDA: #2285's pass
qwen3/cuda/lm.ziggainsforwardFrom(…, Extra)(:343).Extra(:200) holds:start: the input's first position; the attention reads its positions froma.pos + 4 * (start + p)(:457);cache: every layer's own k and v caches in place of the scratch's one (:445);splice: rows copied over the embedded ones, cut to the pass (:348);head: the final norm over the pass's last rows and the tied head on the device, as one more pinned (vocab,hidden) plan that only ASR prepares (
:363,qwen3_asr/cuda/model.zig:91);tap: stage reads fortf-qwen3-asr's stage compare.forwardisforwardFrom(.{}): the same launches with the same arguments, andthe embedding engine's
prepareplan list is unchanged.lm.admit(:210) refuses a start or head on more than one input or without its caches, a head over no rows ormore rows than the pass, and positions past the caches, before any launch. Its test (
:218) names each refusal;with the cache check planted out it goes red (18/19), and green again (19/19) with it back.
weights.Model.fromCheckpoint(weights.zig:46) loads the ASR checkpoint's Qwen3 under itsthinker.model.scope. The position and RoPE uploads moved from the engine intolm.tables(lm.zig:127), whichboth call (
engine.zig:75).qwen3/pass_cuda.zigis the small module root the ASR tools import.qwen3_asr/cuda/lm.zig(our binding of the schedule on CUDA) andtf-qwen3-asr-forced --row-gemms.attention, isn't open yet. SoExtra.startisthe start for now; when B lands I'll move it onto B's offset.
qwen3_asr/lm.zig) stays in the family for now. It moves tofamilies/qwen3/schedule.zigwiththe Metal and XPU bindings (qwen3-asr: the Metal binding (#2105, stacked on #2239) #2254, qwen3-asr: the XPU binding on Gen12LP (#2105, stacked on #2239) #2255), its only other users.
Provenance
The family was written from TensorFold's code, the checkpoint's config files, and the publisher's reference package
qwen-asr 0.0.6. From that package I read the transformers-backend modeling and processor, the inference wrapper, and a
search of its vLLM-backend file for the attention windows. The log-mel front end was checked against transformers'
WhisperFeatureExtractorsource, which the package calls. No code was copied from any of them.Gate: the tie rule (as on #2648), teacher-forced against qwen-asr 0.0.6, on
51dc03072and a 99-frame cut, and 3 LibriSpeech dev clips. That's 45 per checkpoint.
tf-qwen3-asr-forcedruns each clip on the reference's greedy ids to the end. A step whose top token differspasses only if the two tokens swap places and both margins are at most 0.25 logit. Every such step is listed, and
anything else fails with exit 1.
1272-128104-0004step 11 (ours 3138 at 0.25, the reference's 705 at 0.000),cut99step 7 (13 at 0.000, 30 at 0.125),cmn-10369927130798325266step 19 (3837 at 0.000, 67338 at 0.25),cmn-15324743446256875074step 23 (17 at 0.25, 40820 at 0.25).Swaps:
1272-128104-0004step 11 (705 at 0.125, 11 at 0.000);cmn-10104380129516909856step 27 (3837 at0.25, 1773 at 0.000).
Listed exception (agreed on review, unchanged):
cmn-15324743446256875074step 10. Ours is 468 (margin 0.125).The bf16 reference picks 9197 (margin 0.25), with 468 third. The package's fp32 run picks 468, at 25.7277 against
9197's 25.7162.
Open fail,
cut50step 3: ours is 5632 with margin 0.625 over 18284. The reference case picks 18284 at0.125. The package's own teacher-forced logits on this clip disagree with that case: its bf16 run puts 5632 first
(22.75 against 22.625) and its fp32 run puts 18284 first (22.7922 against 22.5113). With the bf16-rounded norms
(r4) this step was a swap at 0.25; qwen3 cuda: Qwen3 embedding checkpoints (BF16) on the native CUDA engine (#2261, 1 of 2) #2285's fp32 norms move it to 0.625.
The rule for listing a step (review, 6826729645): the package's bf16 and fp32 runs order the two tokens oppositely,
and either our pick is the fp32 pick or our fp32 distance on that clip is no worse than the package's bf16. Step
10 meets it.
cut50step 3 meets the first part, and the whole clip is closer to fp32 than bf16 is, but step 3alone is not (0.667 / 0.1191 against 0.536 / 0.0843, table below; our worst column there is 18284 itself). So it
is not listed, and this stays draft.
Stage compare at step 3 against the package's fp32 run: the tower is level with bf16. Through LM layer 20 our
mean is at or below bf16's. From
lm_layer21both columns are above it, and 18284's lead (a logit lens throughthe final norm and head rows) falls further behind fp32 over layers 21-27. At the head the lead is -0.625 for
ours, -0.125 for bf16 and +0.281 for fp32: the gap builds up, with no single stage out of line.
The prompt is extended by the forced ids of steps 0-2. LM rows are at step 3's row; worst column / mean |diff|
to fp32; the lead is 18284 minus 5632 through the lens:
On review (6840111036) this reads as bf16 precision on a two-step tie, not a defect. The fail stands, and a new
gate on fresh clips is being agreed on the thread before any run.
rows one position late move no top token, but they raise the worst fp32 distance to 14.26 / 7.68.
Distance to the package's fp32 run, on
51dc03072At the final logits, teacher-forced on the package's greedy ids. Each cell is worst column / mean |diff|:
cut50, 9 stepscut50, step 3 aloneOn #2285's pass, our mean distance is below the package's bf16 run in 5 of 6 dev cells. The worst column is below it
in 5. On
cut50the whole clip is below it in both; step 3 alone is above it in both.cut50step 3 alone: the same clip and tool, with the case's prompt extended by the forced ids of steps 0-2. As acheck, the step 0-2 and step 0-3 slices give the same step-3 mean by difference (0.1191 ours, 0.0843 bf16).
In r4 the 0.6B means were 0.133 / 0.102 / 0.085.
Decode against prompt, two passes against one, on
51dc03072tf-qwen3-asr-forced --rows 8, no other flag): the prompt pass's last 9 rows equal the 9 decode steps bitfor bit. A prompt split in two passes inside the audio rows leaves 0 cache values different and the same last logits.
That holds on all six dev runs (0.6B and 1.7B, three clips each), because
gemm.zigkeeps one algorithm per (n, k)at every row count.
lm_test.zig): the same two equalities, and a planted cache-slot fault fails them.#2285's embedding bytes
The default call issues the same launches with the same arguments. The new fields are only read when set, and the
plan list is unchanged. Measured on the GB10 with Qwen3-Embedding 0.6B, 4B and 8B (Arminova's revisions,
serve --context 32768), main000b534a2against51dc03072, sha256 of every returned fp32 vector:as token ids), seeded inputs of 2048, 8192, 16384 and 30000 ids (alone, 7 inputs in one request, and 2048 with
30000);
4257e6f01hashes, 4 of 4 in each size.All 8 in one request is refused on both builds by the 32768-token admission.
Kernels
tf_vit_conv_cols: 3x3 stride-2 conv taps, channels last;tf_vit_conv_flatten: torch's [c, f] rows;vision_kernel<64, 8>instance for windowed attention at head dim 64.qwen3-asr-contract: all 3 kernels pass gates 1-4, and both planted controls (one per copy kernel) are caught.These kernels compile to the same SASS on this head (below).
vision_kernel<64, 8>takes 128 registerson sm_120/121 instead of 120 and has its own record; the two copies compile the same.
Checks
Rebased on main
000b534a2. One conflict:contract_controls_set.cu's include line; it keeps both main'scontract_dense_controls.cuhand ours.vit.cuandprefill_attention.cuat nvcc 13.0.88, main against this head, sm_80/86/89/120/121 (after main'stwo-query-heads change):
vit.cu: 55 of 55 records identical in res-usage and SASS;prefill_attention.cu: 70 of 70;zig build test-gradient-serving, each head merged locally with main6c5cef894: red on51dc03072(
prefill_attention.cu); on89d85b860the three re-pinned rows pass, and it stops only atzig/src/cuda/experts.zig, which main has since re-pinned (037e9d9018). With that row re-pinned locally too,it's green.
On
51dc03072: every check exits 0:git diff --check,zig fmt --check,cuda_common_check, lean (0 new),pr_check,zig build test,test-cuda-host,test-qwen3-asr-host,tf-qwen3-asrandtf-qwen3-asr-forced,test-qwen3-cuda-host,test-qwen3-checks-host,qwen3-cuda-gpu-build, the CI set (test-kernel-contract, test-golden, test-server-security, test-pull-security, fuzz-cortheon),compile-cuda-testsfor aarch64 and x86_64, andcheck-release-paths.x86 host set on
51dc03072(test,test-golden,test-kernel-contract,test-server-security,test-pull-security,fuzz-cortheon,check-release-paths,check-native-cuda-host,compile-cuda-tests,pr_check,zig fmt): all exit 0.Seeded sampling and concurrency don't apply yet: nothing here is served (no lane until Nemotron 3.5 Transcribe: audio input, the 65,536-token window, an hour of audio, MTP drafts #1632), and the tools are
greedy on one stream. Resume is the two-pass split above (caches and logits bit-equal to one pass).
CHANGELOG (Unreleased,
2fe5b72d9):qwen3-asr: Qwen3-ASR 0.6B and 1.7B on CUDA (tower and greedy decode), the language model through the Qwen3 CUDA pass (#2239).Next
The served path (a
qwen3_asrCUDA lane backend) waits for #1632. WER/CER, throughput and the 1.7B's eval follow.