Repository navigation
Conversation
Gates 1-4 run the relu^2 stream tile at every width Nemotron's oQ checkpoints store it (4, 5, 6 and 8 bits in groups of 64), each wider width with a planted control that reads its codes as 4-bit and goes red at gate 3. Gate 5 records the 24 launches at 5, 6 and 8 bits on the four RDNA chips. The 1x8 tile at those widths spills to scratch on gfx1100; the records name it as a known finding and a host test holds the list to the records.
The loader, the memory plan and admission read config.json's per-module widths: MLX affine 4, 5, 6 and 8 bits in groups of 64, one width per tensor. Routed stacks are checked against their stated width. Other widths, unquantized projections and an MTP head inside the model's shards are refused by name. The uniform 4-bit checkpoint reads as before. The MTP head file is still read at the model's own width, so admission refuses it by name beside a model not stored 4-bit in groups of 64.
|
CI on |
|
The gate item "prompt state independent of chunk size", measured on part 2's head
In every run the logits are finite and 0 rows differ, with a maximum absolute logit difference of 0. A last chunk under 16 rows.
So the shorter last chunk changes the output head's logits on this head exactly as it does on main, here by at most 0.015625 on this history, with the same greedy tokens; this PR does not change that behaviour. The chunk sizes whose last chunk is 16 rows or more (232 and 100 rows above) are bit-identical on every checkpoint. I did not run the trunk comparison for the 8-row case. The box's load average and top five processes before and after each run are in the raw files (no process other than the test and the tooling above 1% of a core; the power mode was not recorded for these runs). Raw per-run logs and comparison JSON: https://gist.github.com/gilby/b5177e19b978d05604050ec41df4744b. |
|
Thanks @gilby. The loading and byte planning tests pass. The temporary MTP refusal is explicit, so this part does not silently read a head at the body's width. We will run the device check on our side for stock 4 bit, uniform 8 bit and mixed oQ6 bodies. We will check actual loaded bytes, the named MTP refusal, concurrent and resumed replies, and prompt and decode speed. Stock tokens must match main; each newly admitted checkpoint must match its own plain target.
|
What this changes
Nemotron 3.5 Lightning on HIP read only
mlx-q4g64. With this, the loader, admission and the memory plan read each matrix at the widthconfig.jsonstates for it, so the oQ mixed-width MLX checkpoints the family's Metal backend already serves load on HIP: MLX affine at 4, 5, 6 or 8 bits in groups of 64, a width per matrix. No kernel source or launch changes: the shared HIP tiles already decode these widths (part 1 adds the relu² tile's contract records at them), and the single width lived in the Nemotron HIP admission, loader and plan.config.zigalready parses (asqwen3_5does on HIP); routed stacks are read at their own width and their geometry checked against it.lm_head's.Nemotron-H model mlx-q4g64+q5g64+q6g64+q8g64 18.14 GiB.mtp.layers.*inside the main shards.loadMtpbuilds the head's table at the model's width and admission plans the head atc.bits, so an oQ6 or oQ8 model beside the stock 4-bitmtp-4bit.safetensorswould pass admission and then fail at load. The refusal readsmtp-4bit.safetensors sits beside a model stored 6-bit in groups of 64: Nemotron on HIP reads the MTP head beside a model stored 4-bit in groups of 64 only; move mtp-4bit.safetensors out of the folder to serve it without drafts. Such a checkpoint is served on this part without drafts by moving the head file out of the folder;--mtp-drafts 0does not help, because admission reads the folder before the policy decides whether to load the head. An oQ4e checkpoint, whose base width is 4-bit in groups of 64, keeps its head. Part 3 removes the refusal, and its test becomes part 3's acceptance test.Part 2 of 3, for Nemotron 3.5 Lightning's oQ checkpoints on HIP. Part 3 (#3251) reads the MTP head file at its own width.
2a2da5b8b(loader, admission and plan) and5e4fc8035(changelog line), then two merges with no code of their own:deffa794c(part 1's head46e9e1f70) and771530870(mainaaac299d1), both conflicting inCHANGELOG.mdonly, so CI can start.atc overlap --json: run on771530870against baseaaac299d1. The result is incomplete (ATC reports its pull-request, file and time limits). For this tree it lists 13 open PRs, each sharing one file with this change; four of them are reported as conflicting in that file, which I read asCHANGELOG.md.Checks on this head
be9fe6237(771530870plus the changelog number). The receipt and the device checks below were measured on5e4fc8035, before the two merges, against main223da7659; I have not remeasured on the merged head. Since223da7659main has changed the routed prompt tile on gfx1151 (perf(hip): the routed prompt tile takes K-parallel passes for an item of few rows #2878), which this family's prompt pass uses on main and here alike; this PR's own diff against main is the same 13 files before and after the merges.python3 tools/zig/lean_check.py, on the base and on this head: 341 problems on mainaaac299d1and 341 here, 0 new (341 on223da7659, part 1 and5e4fc8035too).python3 tools/pr_check.py --base origin/main --pr 3242: 0 errors, 0 warnings.5e4fc8035):zig build test -Dhip=true --summary all: 896 of 899 steps succeeded, 13,756 of 13,875 tests passed, 116 skipped, 3 failed. Main at223da7659: 896 of 899 steps succeeded, 13,748 of 13,867 tests passed, 116 skipped, 3 failed. The three failures fail the same way on main:dashboard_test"dashboard page polls the public stats snapshot without private details", and the Flash Next HIP host testsgather: the parked pool's parts copy the same bytes as one thread for every id countandgather: a cold call wakes the pool and the same ids gathered again stay on the caller.zig fmt --check build.zig zigandgit diff --checkpass.config.json's width for it (red whenwidthignores its key); every refusal by name (red when every width is accepted); the plan's bytes equal the real oQ6 and oQ8e header bytes (red with a 4-bit-only count); the stock 4-bit test still pins the checkpoint at exactly 17,775,339,392 bytes, so the uniform path is unchanged. (The reverts after the first were checked on nemotron hip: read oQ mixed-width MLX checkpoints (4, 5, 6 and 8 bits, a width per matrix) #3068's head; the code under them is unchanged here.)zig build hip-kernel-contract, exit 1:SUMMARY gfx1151 162 kernels: 162 pass gates 1-4, 0 fail, 2 known findings; 44 controls: 44 red as planted, 0 missed; 173 limit problems(part 1's, as this part changes no kernel or record).zig build hip-kernel-test: 159 of 165 passed, 6 skipped, 0 failed, the same as main.Nemotron-H model mlx-q4g64 17.34 GiB, MTP head loaded) and drafts (91 of 108 drafts accepted on one 128-token reply); a local oQ6e build (6-bit base, 8-bit elsewhere) with the stock head beside it is refused by name with the message above before any weights load (device memory unchanged), with--mtp-drafts 0too; the same folder with the head file left out loads (Nemotron-H model mlx-q6g64+q8g64 24.43 GiB, MTP head off) and replies.771530870, this head less the changelog number) against mainaaac299d1, on the gfx1151 box, checks only:zig build test -Dhip=true --summary all747 of 747 steps, 10,036 of 10,115 tests passed, 79 skipped (main 747 of 747, 10,028 of 10,107, 79 skipped); lean 341 and 341;zig build hip-kernel-contract, exit 1,SUMMARY gfx1151 164 kernels: 164 pass gates 1-4, 0 fail, 2 known findings; 45 controls: 45 red as planted, 0 missed; 174 limit problems(main: 161 kernels, 42 controls, 174 limit problems, exit 1);hip-kernel-test216 passed, 6 skipped, 0 failed on both.5e4fc8035):tf-nemotron-hip-test teacherover a 1,000-id history in chunks of 256 and 300 gives the last chunk's rows bit-identical to the one-chunk run on the oQ4e, oQ6, oQ8e and stock 4-bit checkpoints (232 of 232 and 100 of 100 rows each). A last chunk under 16 rows (8 rows) differs in the output head's logits by at most 0.015625 with the same greedy tokens, on this head and on main alike. Details and raw files in the comment below.be9fe6237.Receipt
One Strix Halo iGPU (gfx1151), fresh servers with the page cache dropped before each,
--context 131072 --parallel 4 --prompt-cache-gib 0; every number below was measured on this head (5e4fc8035) against main (223da7659), binaries built from those two heads (sha256f8a6c92a10bb…for this head,ae9d7e30e28e…for main). The power mode readperformanceat the start and end of every arm, and no other process on the box used more than 20% of a core during any arm (load and top-process records are in the raw files). Sampled cells carry the tools' default seeds:tools/bench_concurrent.pysends--seed(default 1234), stream i at T=1.0 with seed 1234 + i;tools/bench_openai.pysends 1234 + rep for rep 0–2 (and 1234 for its warm-up). Raw tool JSON, server logs and driver scripts for every arm: https://gist.github.com/gilby/77553fdd074a0c4a59ec30b3212bb4e1.A. The 4-bit checkpoint, main against this head (
TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bitatd9d758fbwithmtp-4bit.safetensors; arms main, this, this, main). Aggregate tok/s, both arms of each side:bench_openaicode, T=0 / T=1.0bench_openaichat, T=0 / T=1.0--no-draftscode / chat, T=0bench_openaicode T=1.0 at −1.1% and 0.0%, 1 stream T=0 at −0.1% and −0.9%; no cell is 2% or more slower in both pairs, and the aggregate changes above are all under 1%. Cold first token is the first request of each arm (it reports zero cached tokens; the 12 cold rows of every arm are cold).bench_openaichat at T=0 reads either about 95 or about 99 tok/s from one repetition to the next on both builds (main's first arm 95.4 in one of three repetitions, this head's first arm 94.9 in one), so its medians move by that step.tools/bench_concurrent.pyandbench_cache.pytoken_sha256,bench_openai.pytoken_sha, 12 hex digits) of this head equal main's in every request: for each of this head's two arms against each of main's two arms, 66 of 66 equal, 0 unequal, 0 failed (12 cold replies, 42 requests at 1, 2 and 4 streams greedy and seeded T=1.0, 12bench_openairequests); main's two arms equal each other and this head's two arms equal each other, 66 of 66. With--no-drafts: 12 of 12 equal between main and this head, and each build's draftedbench_openaireplies equal its own--no-draftsreplies, 12 of 12. In every cell of every arm concurrent equals alone (42 equal, 0 unequal, 0 failed) and alone equals"draft": false(24 equal, 0 unequal, 0 failed). Server-log totals per arm: 19,172 reply tokens in every arm of both builds, 2.29–2.31 tokens a round on main and 2.30–2.31 here.Nemotron-H model 17.34 GiB, MTP head loadedon main andNemotron-H model mlx-q4g64 17.34 GiB, MTP head loadedhere.B. Mixed-width public checkpoints on this head (they do not load on main: an oQ4e fails at load with
backbone.layers.0.mixer.in_proj: packed words and group tables have incompatible or unrepresentable geometry, an 8-bit one is refused by name). The oQ4e with the stock head beside it (its base is 4/64); the 6- and 8-bit ones with the head file moved out, as this part requires:user3892r4mlx-oQ4e (3948ddc5), head besideTensorFoldoQ6 (8cf893c6), no headuser3892r4mlx-oQ8e (5b942665), no head--no-draftsarm (one for the oQ4e, one each for the oQ6 and the oQ8e); the oQ6 and oQ8e streams and cold columns come from the default-policy arm without a head, where the n-gram drafter still runs (889 of 1,126 and 778 of 1,098 drafted tokens accepted; 1.05 tokens a round), which is why its decode is not the plain figure. Main refuses or fails all three on the same box: the oQ4e at load withbackbone.layers.0.mixer.in_proj: packed words and group tables have incompatible or unrepresentable geometry, the oQ6 withNemotron on HIP reads MLX affine 4-bit groups of 64; this checkpoint is 6-bit in groups of 64, the oQ8e with the same message for 8-bit."draft": false(24 equal, 0 unequal, 0 failed) in every cell of the five default-policy arms and, for the oQ4e, both arms equal each other (66 of 66) and its draftedbench_openaireplies equal its--no-draftsreplies (12 of 12); the oQ6 and oQ8e default-policy replies equal their--no-draftsreplies (12 of 12 each). Every cold row is cold (4 of 4 at each size, in every arm).Fidelity against an independent reference
Reference:
mlx_lm0.32.2 onmlx0.32.4 (Metal), loading the same public checkpoint files (each file checked against the revision's LFS sha256 on both machines). The 16 prompts oftools/bench_prompts.json(main223da7659) were each followed by the reference's own greedy reply of up to 256 tokens. Engine:tf-nemotron-hip-testat this head (5e4fc8035) on gfx1151, teacher-forced on those histories with--no-mtp:teacherfor every row on the prompt path,decode --steps Kfor every reply row on the decode path. I compared each row's top token. Every row's top-32 ids and logits are identical to the same run at5474a338e, the earlier head of #3068. A margin is a side's top-1 logit minus its logit for the other side's token.user3892r4/...-mlx-oQ4e@3948ddc5TensorFold/...-oQ6@8cf893c6user3892r4/...-mlx-oQ8e@5b942665TensorFold/...-MLX-4bit@d9d758fb(main serves it)Most differences are larger than one bf16 step, and every one is listed in
differences.md. Twelve have a margin of 2 or more on one side (oQ4e 1, oQ6 6, oQ8e 4, 4-bit 1), and the largest is oQ6, heist-story, row 210: referencethe31.0 againsta26.5, engine 29.25 against 29.5. None of this is specific to a width: the uniform 4-bit checkpoint that main serves has the same count and the same spread of logits (p99 per-row error 2.4 to 2.75 at every width), while the planted control has 7.5% of rows different and 212 at a margin of 2 or more.The engine's prompt path and decode path also disagree with each other on 43 to 50 top tokens per ~4,000 reply rows at every width. Main
223da7659on the 4-bit checkpoint does the same, and the earlier head's prompt rows were bit-identical to main's. Whether Nemotron on HIP is meant to hold prompt rows equal to decoded rows is a separate question for main, which I will raise in its own issue. The tools, both sides' top-32 rows for every position, every difference with both margins, and the prompt-versus-decode measurements on main: https://gist.github.com/gilby/68a3888329b36947d80b7f277a202d24zig build native-hip -Dhip=true -Dhip-arch=gfx1151 -Doptimize=ReleaseFast, system hipcc (HIP 7.1, clang 21), one Strix Halo iGPU (gfx1151), Linux 7.0.tools/bench_concurrent.py --alone --serial --temperatures 1.0,0: this head equals main in 66 of 66 requests for each pair of arms (Receipt A), concurrent equals alone (42 equal) and alone equals"draft": false(24 equal) in every cell of every default-policy arm, 0 unequal and 0 failed anywhere.tools/bench_openai.py --tokens 256 --reps 3: within −0.6% to +0.8% on code and chat, T=0 and T=1.0 (Receipt A);--no-drafts87.9 / 87.9 against 88.1 / 88.0 tok/s code / chat.tools/bench_cache.py --require-coldat 2K, 7K and 28K tokens: first token +0.4%, +0.5% and +0.8% over main's mean of two arms (Receipt A), every row cold.Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.--drafter, no slower than without it, greedy and sampled