[ajensenwaud]

What this changes

This PR adds Meta's Muse Spark Open to the native CUDA engine. It is a text-only mixture-of-experts model with 816B parameters, 42B of them active per token. The PR serves it on two GB10s from a 1.8-bit EXL3 pack, with the prompt cache on.

• Drafter. The code can also run RadixArk's DFlash2 drafter, a small model that proposes tokens for Muse to check, which speeds up decoding. RadixArk has published no licence for it, so the recipe serves Muse without it.
• Chat format. Muse's chat format needs server: split recipient-addressed replies into reasoning, answer and tool calls #1336, which splits a reply into reasoning, answer and tool calls.
• Kernels. Muse reuses main's shared EXL3 code. Its own kernels cover attention, the norms, the router and the output head, and they keep activations in fp16, as exllamav3's reference does.
• Limits. Only two GB10s were tested, so any other GPU or number of GPUs is refused with an error that says why.
• Memory. Each GB10 holds about 108 GiB while serving, so the recipe lowers the server's memory reserve from 12 GiB to 3 GiB.

Results

• Accuracy. On five texts, Muse's output stays as close to exllamav3's as exllamav3 stays to itself, and the two pick the same top token 96% to 99% of the time.
• Exactness. Drafted replies equal undrafted ones, concurrent requests equal the same requests run alone, and 115 of 115 replies with the prompt cache on equal those with it off.
• Quality. GSM8K 195 of 200 and MMLU-Pro 167 of 200.
• Speed. With the drafter on, decode runs at about 30 tokens a second for one stream and 43 for four. llama.cpp on the same two Sparks manages 7.4 and 17.4. Plain-decode speed on this head is not yet measured.
• Kernel contract. All 45 of Muse's kernels pass. The run's few failures are all in main's own kernels.
• Host tests. Everything passes except one prompt-cache test, which fails the same way on main.

Details

• Hardware. Two ASUS Ascent GX10s (GB10, sm_121, 128 GB each), CUDA 13, Zig 0.17.0, linked over RoCE.

• Checkpoints. Pack ajensenwaud/Muse-Spark-Open-816B-A42B-EXL3-1.8bpw at 38ad2925, quantised from amd/Muse-Spark-Open-816B-A42B-Quark-MXFP4 at 4885eb2a. Drafter runs used RadixArk/Muse-Spark-Open-DFlash2 at 46c90f41.

• Run dates. On 2ff01910. Since then the branch has only moved onto server: split recipient-addressed replies into reasoning, answer and tool calls #1336 and main 2836e9b2; no kernel changed.

• Accuracy figures. KL is 0.67 to 1.24 times exllamav3's own noise, and the true next token's loss is within +0.61% and -0.08% of exllamav3's. This branch's logits are byte-identical to the build that measured them.

• Speed figures. Greedy decoding, 512-token replies, drafter on, in tokens a second:

  	Chat	Code	Agent	Report
  1 stream	29.4	32.1	30.1	28.7
  4 streams, total	42.2	43.3	45.5	41.6

• Prompt speed. A cold 15,375-token prompt reads at 196 tokens a second.

• Context. 471,040 tokens fit for one stream, or 110,592 each for four.

• Kernel contract. 45 of 45 Muse registrations pass gates 1-4, and 46 of 46 launch records pass gate 5. Main's own kernels account for the run's 2 failures and 291 limit problems.

• Host tests. 738 of 740 tests pass. The lean checker finds 426 problems, the same as main.

• Untested setups. One GPU, other GPUs, Metal and image inputs, all of which are refused.

Checklist

• Every token goes through the lane rounds.
• Drafted equals "draft": false, concurrent equals solo, and resumed equals fresh.
• No precision traded for speed.
• Every new GPU kernel is registered with the kernel contract.
• New tests fail before the change and pass after it.
• No personal data, hosts or local paths, and no AI attribution lines.

[ashhart]

Thank you for this. Bringing Muse Spark Open up on two GB10s with drafts and exact repeats is serious work. To fit how we land things, it needs splitting before we can run it:

1. int8 KV (TF_MUSE_KV8): please drop it. TensorFold's rule is exact serial decode only, so we don't take lossy formats, even opt-in ones (same call as qwen27 CUDA: an opt-in 4-bit attention cache (262K on a 24 GB card) — agreeing on its gate #799).
2. The lanes sampling change (one-pass Gumbel race with skip bounds) changes how every family samples when no top_k/top_p cut applies. Please send it as its own PR with the exactness argument and a test. We'd A/B it across families on Metal and CUDA.
3. Recipient-reply parsing in chat.zig/server.zig: a separate small PR too, please.
4. Kernel contract: the EXL3 kernels need registering with the contract (the checklist box), and where the shared EXL3 operators already cover a shape, please use those instead of a vendored copy.

After that, the family PR will just be the Muse Spark family, and we'll run it on two GB10s.

- Synthema, Ash's AI Assistant.

[ashhart]

Thanks @ajensenwaud. We rebased it onto current main (conflicts in chat.zig next to #1298's cut tool calls, build lists and the CUDA registry; all kept). Host suites and both Linux CUDA compiles pass, and lean adds no findings. Before a device run:

1. TF_MUSE_KV8 and the e4m3 attention mode are still here (forward.zig, engine.zig, ops.zig, kept.zig, attention.cu, the recipe). The engine takes no lossy KV, even opt-in, so please remove them. The served exactness results were taken with int8 KV on, so they need rerunning with fp16.
2. Please drop the recipient-replies and sampler-race commits now that they're #1336 and #1337, and stack on #1336.
3. Register the muse_* kernels with the CUDA kernel contract.
4. Where the shared EXL3 operators cover a shape, please use them instead of the vendored copies (muse_spark/exl3_linear.cu, exl3_experts*.cu, experts_grouped.cuh, decode.cuh).
5. memory_reserve_gib = 3 lowers the OS reserve from about 12.8 GB to 3 GB by default on a shared GB10. We'd rather this be the operator's explicit choice, with measured per-rank peak footprint showing the headroom, than a family default.
6. Please give the full revisions of the pack and the DFlash2 drafter, and add host tests for the lane backend's verify/keep/draft.

Then we'd run, on our GB10 pair: shared fatbins identical, the contract SUMMARY, drafted vs plain (greedy and seeded), concurrent vs solo, resumed vs fresh, graphed vs eager, and a GLM control.

[ajensenwaud]

Thanks. All four done:

1. int8 KV removed, fp16 only.
2. Sampler split out: lanes sampling: a keyed draw that cuts nothing by rank finds the sorted draw's token in one pass #1337.
3. Recipient replies split out: server: split recipient-addressed replies into reasoning, answer and tool calls #1336.
4. Vendored EXL3 copies gone. It runs on the shared operators. The two launch forms they don't cover are registered as byte-equal replacements, and every Muse kernel passes the contract on GB10.

Output is byte-identical to before, so the parity numbers stand. The branch is just the family now, rebased on main. Happy to run anything on our two GB10s.

[ashhart]

Re-review of e2231f5 against main 3f408b6. Thank you, every point from the earlier review is addressed: the fp8/int8 KV is gone, the vendored EXL3 copies are replaced by the shared operators (registered as replaces with byte-equality checks), the sampler and recipient-reply commits are dropped (the first two commits match #1336), the reserve default is gone, revisions are pinned, and rounds.zig has host tests. Host checks pass here, including test-muse and the Linux compiles. This lands after #1336 and a device check on our GB10 pair. Three small asks before then:

• A licence line for Muse Spark Open, DFlash2, and exllamav3 (MIT, used as a reference).
• Concurrent equal to solo, shown directly (the 12/12 sha match covers serial and four at a time together).
• The contract SUMMARY and GPU receipts on this head (the posted ones are from 2ff0191).
  On the pair we'll cmp every non-muse_* fatbin against main, run the contract and limits, tf-muse m1 teacher-forced with MUSE_GRAPHS=1 vs eager, prompt and drafted, then serve TP=2 for concurrency 1/4 at temperature 0 and 0.7, resumed vs fresh, and a GLM control.

[ajensenwaud]

Thanks Ash. All six from your review are addressed on the branch (e2231f5a):

1. 8-bit KV: removed; fp16 only. The served exactness counts in the description (drafted == plain 12/12, resumed == fresh 115/115) were rerun with fp16 KV.
2. Split commits: the recipient-reply and sampler commits are gone from this branch, and it is now stacked on server: split recipient-addressed replies into reasoning, answer and tool calls #1336: the first two commits are server: split recipient-addressed replies into reasoning, answer and tool calls #1336's as you reviewed them, and the last four are this PR's. lanes sampling: a keyed draw that cuts nothing by rank finds the sorted draw's token in one pass #1337 isn't needed.
3. Kernel contract: every muse_* kernel is registered and passes gates 1-4 on GB10, and all 46 Muse launch records pass gate 5 (1eb206df).
4. Shared EXL3 operators: the vendored copies are gone. Projections and routed experts run on cuda.exl3.Linear and exl3_experts. Two launch forms remain because no shared form covers them (q/k/v/gate in one launch with 32-row passes, and 32-member expert tiles); each is registered as a byte-equal replacement of main's kernel.
5. Memory reserve: memory_reserve_gib is removed from the family and from cuda_family_memory.zig, so the reserve is the operator's choice again. The recipe now passes TENSORFOLD_MEMORY_RESERVE_GIB=3 on both ranks. Measured on two GB10s (1 stream x 454,656 tokens, 1 GiB prompt cache), each rank holds 107.7 GiB idle and 108.8 GiB with a 24K-token prompt running. MemAvailable bottomed at 5.9 and 5.5 GiB on the two nodes, and rank 0's /health peak was 107.8 GiB. The server reports 2.7-3.9 GiB left for streams after a 3 GiB reserve, so the default 12.2 GiB would leave none.
6. Revisions and host tests:
   • Full revisions are in the description and in docs/recipes/muse-spark.md. The pack is 38ad2925945840c3a677bd3f213867d30368636c (weights as of efcdf2ec4222c4597e291f1b07edd19c3c02c652), quantised from AMD's MXFP4 export at 4885eb2a901c2638854ef35cf6833e6447622afb. The drafter source is RadixArk's DFlash2 at 46c90f412549c472941a68382d91a99e5e6372a0, quantised to EXL3 4.0 bpw with exllamav3 058b8630 plus the recipe's patches.
   • The lane backend's host side of a round is now in rounds.zig with host tests for verify (rows, positions, slots, held vs host drafts, greedy detection), keep (chains only) and draft (entry token, position, depth cap, held drafts). The refusals are covered too: trees, unknown streams, windows past the rows, drafts not held, no drafter, no first token. native.zig calls it, and behaviour is unchanged.

Host suites pass on the new head (test-muse 24/24, test-cuda-host, test-kernel-contract, test-native-options-host, test-kernel-symbols-host), and lean adds no findings. No kernel changed, so the GPU results stand. Ready for your GB10 run; happy to run anything on ours too.

[ajensenwaud]

The desktop failure on this run is the intermittent bot-vm guest test (guest.test.ts:55), not this PR (no app/ files touched; #1336 on the same base passed it). The diagnostics from 24199de caught the exception this time: {"ok":false,"output":"VM tool refused: [Errno 1] Operation not permitted"} on the macOS runner. Could you rerun the job? I don't have the rights to.

[ajensenwaud]

Please let me know what more is pending.

[ashhart]

Thanks @ajensenwaud. The Muse Spark family files are byte-for-byte what we re-reviewed on 26 Oct. Only the registration lists and #1336's two newer commits changed, so that review stands. Rebased on main b30d5b9d3 it applies cleanly. zig build test, the CUDA host suites, test-muse and both Linux compiles pass, and lean adds nothing. It lands after #1336.

Three small things, then our GB10 run:

1. A licence line in docs/recipes/muse-spark.md for Muse Spark Open, the DFlash2 drafter and exllamav3 (used as the reference).
2. From your exllamav3 teacher-forced run, the top-token misses with their contenders and margins.
3. Concurrent equal to solo on its own (bench_concurrent.py --alone --serial). We'll record it on our pair too.

On our pair we'll cmp every non-muse_* fatbin against main and run the contract on this head. Then tf-muse m1 with graphs and eager, prompt and drafted. Then we'll serve TP=2 at 1 and 4 streams at temperatures 0 and 0.7, check resumed against fresh, and run a GLM control.

[ajensenwaud]

Thanks @ashhart. All three are done. The branch is rebased on main 41abcf40. One conflict: main's loop-guard stop and the recipient hook edit the same line of chat.zig, and both are kept. That pushed chat.zig to 603 lines, so the sampling-field flag reader moves unchanged to fields.zig (aa9588a4; the same commit is on #1336). Muse Spark's own code is unchanged.

1. Licences. docs/recipes/muse-spark.md now has a licence paragraph under the checkpoint table (af83ecbf, docs only):
   • Muse Spark Open, AMD's MXFP4 export and our EXL3 pack are under Meta's Muse Model License and its Usage Policy (the pack ships it as LICENSE.md).
   • RadixArk's DFlash2 drafter states no licence (no licence field, file or text on its card), so we publish no drafter pack and build it locally.
   • exllamav3, the quantiser and the parity reference, is MIT.
2. Top-token misses from the exllamav3 teacher-forced run behind the PR's fidelity receipt (exllamav3 muse_spark, streamed, the 1.8 bpw pack, fp32 logits of its fp16 head, against tf-muse m1 at TP=2; the PR head's logits are byte-equal to that engine's). 80 misses in 3,348 positions:

Text	Rows	Misses	Top-1	exllamav3 vs itself (prefix vs full) top-1	ref top-1/top-2 exact ties among misses
English prose	157	4	0.9745	0.9872 (prefix_78), 0.9714 (prefix_140)	1
Chinese prose	138	1	0.9928	1.0000 (prefix_69), 0.9917 (prefix_121)	1
Code	214	2	0.9907	0.9907 (prefix_107), 0.9949 (prefix_197)	1
Agent turn	239	9	0.9623	0.9664 (prefix_119), 0.9820 (prefix_222)	1
Long mixed	2600	64	0.9754	0.9777 (prefix_1300), 0.9729 (prefix_2583)	15
All	3348	80	0.9761		19

• Our top token is exllamav3's Cannot get more than 8% GPU utilization on M4 with 48 gb (even using --resident-budget 30 GiB for example) #2 in 65 of the 80 (docs: add selected TensorFold mesh logo and hero #3 in 11); exllamav3's top is our Cannot get more than 8% GPU utilization on M4 with 48 gb (even using --resident-budget 30 GiB for example) #2 in 75.
• How far exllamav3 puts our pick below its own: exactly 0 in 18 (a tie in its fp16 head), within one fp16 step in 27, median 0.035 logits, p90 0.20, max 1.06 (agent turn, _return p 0.47 vs our _call p 0.35).
• exllamav3 against itself (a prefix run against the full run) lands in the same range: 0.966 to 1.000 top-1 on the same texts.
• The long mixed text is an internal session transcript with our hosts' configuration in it, so only its counts are here; the four short texts' misses are below in full.

Every miss on the four short texts, with both sides' top 5, margins and gaps

English prose (4 misses of 157)

pos	context (last 8 tokens)	true next	ref top	our top	ref top-5	our top-5	ref margin	our margin	ref gap	our gap
36	planks: timber from the hills,	95896 wool	28104 coal	95896 wool	28104 coal 18.3438 (0.0940) 95896 wool 18.2969 (0.0897) 34370 grain 18.2500 (0.0856) 15913 fish 18.0938 (0.0732) 28750 flour 17.7344 (0.0511)	95896 wool 18.3059 (0.0906) 28104 coal 18.2846 (0.0887) 34370 grain 18.2769 (0.0880) 15913 fish 18.0888 (0.0729) 28750 flour 17.7598 (0.0525)	0.0469	0.0214	0.0469	0.0214
86	importance began a slow decline that nobody at	290 the	290 the	1660 first	290 the 18.6250 (0.4994) 1660 first 18.6250 (0.4994) 13587 town 11.6875 (0.0005) 7204 City 9.7109 (0.0001) 4215 home 9.6094 (0.0001)	1660 first 18.6531 (0.5173) 290 the 18.5813 (0.4815) 13587 town 11.7754 (0.0005) 7204 City 9.7856 (0.0001) 6996 city 9.7147 (0.0001)	0.0000 (tie)	0.0718	0.0000	0.0718
109	mornings, the harbour master still logged every	38380 vessel	36744 arrival	32645 crate	36744 arrival 16.6094 (0.1943) 32645 crate 16.3750 (0.1537) 38380 vessel 16.3281 (0.1467) 9782 ton 16.1875 (0.1274) 58884 barrel 15.5781 (0.0693)	32645 crate 16.5392 (0.1813) 36744 arrival 16.5321 (0.1800) 38380 vessel 16.2187 (0.1316) 9782 ton 16.1980 (0.1289) 58884 barrel 15.7103 (0.0791)	0.2344	0.0071	0.2344	0.0071
130	jump from the end at high tide.	2473 What	35036 Yet	589 The	35036 Yet 18.5469 (0.1382) 589 The 18.5312 (0.1360) 1573 It 18.4844 (0.1298) 15307 Change 18.2812 (0.1059) 3696 But 17.9062 (0.0728)	589 The 18.5859 (0.1446) 35036 Yet 18.5616 (0.1412) 1573 It 18.4596 (0.1275) 15307 Change 18.2564 (0.1040) 3696 But 17.8925 (0.0723)	0.0156	0.0242	0.0156	0.0242

Chinese prose (1 misses of 138)

pos	context (last 8 tokens)	true next	ref top	our top	ref top-5	our top-5	ref margin	our margin	ref gap	our gap
69	面积广阔，气候温和，	18253 雨	4520 物	18253 雨	4520 物 17.1719 (0.1823) 18253 雨 17.1719 (0.1823) 21240 资源 17.0312 (0.1584) 42337 土地 16.9062 (0.1398) 131939 降水 16.5312 (0.0961)	18253 雨 17.2214 (0.1844) 4520 物 17.1974 (0.1800) 21240 资源 17.0605 (0.1570) 42337 土地 16.9481 (0.1403) 131939 降水 16.5729 (0.0964)	0.0000 (tie)	0.0240	0.0000	0.0240

Code (2 misses of 214)

pos	context (last 8 tokens)	true next	ref top	our top	ref top-5	our top-5	ref margin	our margin	ref gap	our gap
48	(compare=False, default=float("inf	7018 "))	7018 "))	68687 "))\n\n	7018 ")) 19.2969 (0.4099) 68687 "))\n\n 19.2969 (0.4099) 11609 "))\n 18.4688 (0.1791) 9811 "), 13.3438 (0.0011) 1584 ") 10.2969 (0.0001)	68687 "))\n\n 19.3269 (0.4245) 7018 ")) 19.2767 (0.4037) 11609 "))\n 18.4168 (0.1708) 9811 "), 13.1784 (0.0009) 1584 ") 10.2754 (0.0001)	0.0000 (tie)	0.0502	0.0000	0.0502
51	, default=float("inf"))\n\n\nclass	113133 Scheduler	59129 Priority	113133 Scheduler	59129 Priority 18.2500 (0.3074) 113133 Scheduler 18.2344 (0.3026) 21866 Job 17.6250 (0.1645) 101104 Heap 16.3594 (0.0464) 10347 Task 16.1406 (0.0373)	113133 Scheduler 18.2459 (0.3060) 59129 Priority 18.2410 (0.3045) 21866 Job 17.6322 (0.1657) 101104 Heap 16.3341 (0.0452) 10347 Task 16.1444 (0.0374)	0.0156	0.0049	0.0156	0.0049

Agent turn (9 misses of 239)

pos	context (last 8 tokens)	true next	ref top	our top	ref top-5	our top-5	ref margin	our margin	ref gap	our gap
16	working in a Linux terminal. Use the	40555 bash	2182 following	40555 bash	2182 following 17.0312 (0.1515) 40555 bash 17.0312 (0.1515) 10619 tools 16.6094 (0.0993) 15752 terminal 16.4844 (0.0877) 7935 tool 16.1562 (0.0631)	40555 bash 17.0152 (0.1526) 2182 following 17.0124 (0.1522) 10619 tools 16.5817 (0.0989) 15752 terminal 16.4703 (0.0885) 7935 tool 16.1170 (0.0622)	0.0000 (tie)	0.0028	0.0000	0.0028
21	. Use the bash tool to inspect and	3332 change	8415 edit	18176 modify	8415 edit 17.3906 (0.1070) 71828 manipulate 17.3594 (0.1037) 18176 modify 17.2031 (0.0887) 21362 interact 17.2031 (0.0887) 4969 understand 16.8750 (0.0639)	18176 modify 17.3314 (0.0978) 8415 edit 17.2930 (0.0941) 71828 manipulate 17.2343 (0.0887) 21362 interact 17.1469 (0.0813) 4969 understand 16.9794 (0.0688)	0.0312	0.0383	0.1875	0.0383
68	.<|eot|><|start|>assistant to=bash<|message|>	40 <	1488 '''	7564 {"	1488 ''' 7.3711 (0.2590) 7564 {" 7.3242 (0.2471) 785 {\n 6.8555 (0.1547) 26652 bash 6.8438 (0.1529) 2991 We 6.2539 (0.0847)	7564 {" 7.5782 (0.3144) 1488 ''' 7.5224 (0.2974) 785 {\n 6.7471 (0.1370) 26652 bash 6.6794 (0.1280) 2991 We 5.8940 (0.0583)	0.0469	0.0558	0.0469	0.0558
102	-3 && git show --stat HEAD	695 </	695 </	1932 &&	695 </ 18.1094 (0.2424) 198 \n 18.0938 (0.2386) 1932 && 18.0625 (0.2313) 39 ; 17.7812 (0.1746) 721 | 16.5000 (0.0485)	1932 && 18.0806 (0.2351) 198 \n 18.0800 (0.2349) 695 </ 18.0584 (0.2299) 39 ; 17.8407 (0.1849) 721 | 16.5639 (0.0516)	0.0156	0.0006	0.0469	0.0222
124	><|eom|><|start|>tool bash<|message|><tool	18107 _output	35485 _return	38048 _call	35485 _return 6.2266 (0.4700) 18107 _output 5.4883 (0.2246) 38048 _call 5.1641 (0.1624) 123688 _calls 3.8535 (0.0438) 47644 _use 3.6660 (0.0363)	38048 _call 6.2237 (0.3513) 35485 _return 5.9735 (0.2735) 18107 _output 5.6118 (0.1905) 123688 _calls 4.6672 (0.0741) 47644 _use 4.2569 (0.0491)	0.7383	0.2502	1.0625	0.2502
140	9e2 parser: read the manifest	71383 laz	2017 file	4518 version	2017 file 18.2969 (0.1268) 4518 version 18.2656 (0.1229) 198 \n 18.1719 (0.1119) 948 's 17.6562 (0.0668) 610 from 17.5156 (0.0580)	4518 version 18.3224 (0.1294) 2017 file 18.2474 (0.1200) 198 \n 18.1267 (0.1064) 948 's 17.7379 (0.0721) 610 from 17.5113 (0.0575)	0.0312	0.0751	0.0313	0.0751
151	7d0f113 ci: pin	17914 python	32892 dependency	17914 python	32892 dependency 16.3750 (0.1810) 17914 python 16.3594 (0.1782) 21250 dependencies 16.1719 (0.1477) 64929 pytest 16.1719 (0.1477) 18547 coverage 14.6562 (0.0324)	17914 python 16.3966 (0.1845) 64929 pytest 16.3137 (0.1698) 32892 dependency 16.2993 (0.1674) 21250 dependencies 16.0257 (0.1273) 18547 coverage 14.7425 (0.0353)	0.0156	0.0829	0.0156	0.0973
162	.12\n52bb1a0	37801 docs	16213 parser	62576 feat	16213 parser 17.4688 (0.2556) 62576 feat 17.4375 (0.2478) 37801 docs 17.2656 (0.2086) 9461 fix 15.6094 (0.0398) 180805 refactor 15.3828 (0.0318)	62576 feat 17.5806 (0.2972) 16213 parser 17.3612 (0.2386) 37801 docs 17.1679 (0.1967) 9461 fix 15.5790 (0.0401) 180805 refactor 15.3060 (0.0306)	0.0312	0.2194	0.0312	0.2194
171	: changelog\n src/parser.py |	981 14	34 6	32 4	34 6 17.5156 (0.0789) 32 4 17.4688 (0.0753) 36 8 17.3750 (0.0686) 738 12 17.3594 (0.0675) 30 2 17.2969 (0.0634)	32 4 17.5868 (0.0887) 34 6 17.5741 (0.0876) 30 2 17.4432 (0.0769) 33 5 17.3419 (0.0695) 36 8 17.3144 (0.0676)	0.0469	0.0127	0.0469	0.0127

3. Concurrent == solo, on a build of this branch before the rebase (7d1ac01f; the rebase changes none of this PR's Muse Spark or engine code), TP=2, 4 slots x 110,592, drafter on, bench_concurrent.py --alone --serial at its defaults (4 at once, 256 tokens, 3 reps, seed 1234):

   Prompt, temperature	concurrent == alone	alone == "draft": false	aggregate tok/s (per stream)
   code, 1.0	12/12	4/4	38.3 (10.9)
   chat, 1.0	12/12	4/4	48.2 (13.1)
   code, 0	12/12	1/1	92.0 (24.5)
   chat, 0	12/12	1/1	83.5 (20.9)

   0 unequal and 0 failed in every cell.

[ashhart]

Thanks @ajensenwaud. The licence paragraph, the per-miss tables and the concurrent-alone runs answer our three asks, and the family's code is unchanged since 26 Oct apart from the recipe. We merged aa9588a onto main a1e6d34 (clean). fmt and lean are clean, and the server changes are byte-for-byte #1336's, which pass test-server-cpu and test-golden.

Three things before our GB10 run:

1. The drafter's licence. RadixArk's DFlash2 states no licence, which by default means all rights reserved, so we can't point users at it or test with it. Could you ask RadixArk to state one? Until then, please make plain decode the served setup in the recipe and say the drafter is unlicensed. Our run will be plain decode.
2. The misses. 64 of the 80 are in the long internal text and aren't listed. Please list them without the context column, or swap in a public long text and list its misses. Please also add a line on position 124 of the agent turn, where our _call sits 1.06 logits above exllamav3's: does exllamav3's own prefix run move it too?
3. Admission. The records start at sm_121 (muse_cuda_records.zig:8) and only two GB10s were run. But open (native.zig:440) checks no GPU, config.zig:128 admits one rank, and the recipe offers one GPU (:59). Please refuse anything but sm_121, and one rank, by name for now, as deepseek_v41/kernels.zig:79 does. Also, weightShare (native.zig:488) still turns a bad TF_TP_* into one GPU's share; MiMo-V2.6-Flash on CUDA, on two GB10s with its DFlash drafter #728's 8be3e8b is the fix.

Then on our pair, with the 1.8 bpw pack and a 3 GiB reserve on both ranks, we'll run:

• cmp of every non-muse_* fatbin against main, and the contract;
• tf-muse m1 with graphs against eager, and prompt against decode rows;
• TP=2 at 1 and 4 streams at temperatures 0 and 0.7, checking concurrent against solo and resumed against fresh;
• server: split recipient-addressed replies into reasoning, answer and tool calls #1336's six served cases and a GLM control.

[ajensenwaud]

Thanks, Ash. All three are addressed on the branch (rebased onto current main; #1336's server commits are unchanged).

1. The drafter's licence. You're right: RadixArk's Muse-Spark-Open-DFlash2 states no licence on its card or in
its files. The recipe now serves plain decode. It names no drafter pack and gives no --drafter command, and it says
the drafter is unlicensed, so all rights are reserved by default, and stays out of the served setup until RadixArk
states a licence. I've asked them on the model's discussion page (RadixArk/Muse-Spark-Open-DFlash2, discussion #2). The family's DFlash2 code stays in, unused by the
recipe. One figure in the recipe was measured with the drafter loaded (about 108 GiB a rank); it now says so, and
says the drafter took 1.64 GiB of that a rank.

2. The misses. The long text's 64 are below, without the context column, in the same columns as the short texts'.
That text is an internal working note, so I checked every token the table lists (true next, both tops, both top-5s)
for host names, addresses, paths, user names and key-like strings. I checked each token's own text, and also where
the predicted token sits in the source. 11 token ids are shown as [redacted], id and text both, wherever they
appear: 34 cells over 5 of the 64 rows. They are the candidates at the start of a private network address, a
host-name prefix, and the completions of a local path. Their logits and probabilities are kept. Over the 64: our top is
exllamav3's #2 in 52 and its #3 in 8. 15 are exact fp16 ties between exllamav3's own top two. The median gap from
exllamav3's top to ours, read on exllamav3's logits, is 0.043 logits (max 0.453).

Position 124 of the agent turn: I reran exllamav3 over that text and kept the 222-token prefix run's logits this time (its full run reproduced the stored reference byte for byte). exllamav3's own prefix run does move 124, but not across: its lead of _return over _call goes from 1.06 to 0.82 logits (over _output, 0.74 to 0.45), and _return stays top. Ours goes past it: _call 6.224 over _return 5.974, 0.25 the other way. At that row our KL against exllamav3's full run is 0.135; its prefix run's is 0.015 (11th largest of its 222 rows; its largest, 0.317, is at row 121). So 124 is a real outlier for us, about 9x the reference's own movement there, though under the largest movement the reference shows elsewhere in the same text. In the prefix run 4 of 222 rows change top token against exllamav3's full run.

3. Admission. open now refuses anything but sm_121 (MuseSparkRequiresSm121). It also refuses one rank and
any other world size, by name (MuseSparkRequiresTwoRanks), as deepseek_v41/kernels.zig does. TF_TP_WORLD
unset counts as one rank. tf-muse follow makes the same check, and so do the target's and the drafter's
rankView. The recipe offers two GB10s only. weightShare takes #728's fix: a malformed TF_TP_* is BadSetting,
and admission reports it through the family's explanation instead of sizing one GPU. The change to
cuda_family_memory.zig is the same hunk as #728's, so the two merge cleanly there. One rank is refused at
weightShare too, so admission names it before it sizes memory. Host tests cover the sm/world rule, rankView(1),
the weight share unset, at 1, 2 and 4 ranks, and with six malformed settings. They fail on the previous code and pass
now.

The long text's 64 misses (no context column; 11 token ids redacted)

Same run and columns as the short texts' table: exllamav3's muse_spark forward (streamed full model, 1.8 bpw pack,
fp32 logits of an fp16 head) against this engine (TP=2, fp16 KV), on the 2,600-token mixed text (an internal working
note: prose, markdown tables, figures, code). Row pos holds the logits after ids[0..pos], predicting ids[pos+1]
("true next"). The context column is left out. Logits with softmax probability in brackets; margin = top-1 minus
top-2 logit on that side; "ref gap" = the reference's logit of its top minus its logit of our top; "our gap" likewise
for us; "(tie)" = an exact fp16 tie in the reference's head.

Summary: 64 misses of 2,600 rows (top-1 0.9754); exllamav3 against itself (prefix vs full forward) 0.9777 over the
first 1,300 rows and 0.9729 over the first 2,583. Over the 64: our top is the reference's #2 in 52, its #3 in 8,
lower in 4; the reference's top is our #2 in 61, #3 in 2, lower in 1. Ref gap: 0 exactly (an fp16 tie) in 14 (15 of
the 64 are exact ties between the reference's own top two), median 0.043, p90 0.172, max 0.453 logits. Our gap:
median 0.036, p90 0.183, max 0.431. The true next token is the reference's top in 6 and ours in 12 (neither in 46).

Redactions: the text is an internal note, so every token listed below (true next, both tops, both top-5s) was
checked for host names, addresses, paths, user names, e-mail addresses and key-like strings, by pattern on the token
and by where the predicted token sits in the source text. 11 token ids are shown as [redacted], id and text both, in
every cell they appear (34 cells over 5 rows: 80, 196, 198, 1132, 2056): the candidates at the start of a private
network address (row 80), a host-name prefix (row 196) and the completions of a local path (row 2056); rows 198 and
1132 list one of the same ids. Their logits and probabilities are kept.

Long mixed text (64 misses of 2600)

pos	true next	ref top	our top	ref top-5	our top-5	ref margin	our margin	ref gap	our gap
22	6940 Oct	13340 Feb	38 :	13340 Feb 17.9219 (0.0921) 38 : 17.8125 (0.0826) 5551 hours 17.5938 (0.0664) 26 . 17.5781 (0.0653) 25 - 17.2188 (0.0456)	38 : 17.9956 (0.1011) 13340 Feb 17.7619 (0.0800) 26 . 17.5439 (0.0644) 5551 hours 17.4911 (0.0610) 4118 Dec 17.2398 (0.0475)	0.1094	0.2336	0.1094	0.2336
34	2395 read	32721 pulled	19106 checked	32721 pulled 18.1719 (0.1087) 19106 checked 18.0469 (0.0960) 35808 verified 17.7031 (0.0680) 854 public 16.9219 (0.0312) 25016 confirmed 16.7812 (0.0271)	19106 checked 18.0334 (0.1047) 32721 pulled 17.9808 (0.0993) 35808 verified 17.6285 (0.0698) 854 public 16.9807 (0.0365) 25016 confirmed 16.6033 (0.0250)	0.1250	0.0526	0.1250	0.0526
72	154260 ---|---|	154260 ---|---|	17756 ---	154260 ---|---| 19.0625 (0.4764) 17756 --- 18.8594 (0.3888) 58231 ---| 16.4688 (0.0356) 186148 :--- 16.1250 (0.0253) 721 | 15.2109 (0.0101)	17756 --- 19.0712 (0.4563) 154260 ---|---| 18.9794 (0.4163) 58231 ---| 16.4425 (0.0329) 186148 :--- 16.0795 (0.0229) 721 | 14.9487 (0.0074)	0.2031	0.0917	0.2031	0.0917
80	[redacted]	[redacted]	[redacted]	[redacted] 18.9688 (0.4079) [redacted] 18.9375 (0.3954) [redacted] 17.7656 (0.1225) [redacted] 15.6875 (0.0153) [redacted] 15.2812 (0.0102)	[redacted] 18.9884 (0.4068) [redacted] 18.9493 (0.3912) [redacted] 17.8408 (0.1291) [redacted] 15.6021 (0.0138) [redacted] 15.1432 (0.0087)	0.0312	0.0391	0.0312	0.0391
139	1965 start	4215 home	19930 boot	4215 home 16.2031 (0.0761) 756 / 15.9219 (0.0574) 2822 rest 15.7891 (0.0503) 19930 boot 15.7500 (0.0484) 220 15.6484 (0.0437)	19930 boot 15.9922 (0.0612) 4215 home 15.9834 (0.0607) 220 15.9114 (0.0564) 756 / 15.9069 (0.0562) 2822 rest 15.8375 (0.0524)	0.2812	0.0088	0.4531	0.0088
149	24249 TP	7909 running	330 (	7909 running 17.7031 (0.0866) 330 ( 17.6875 (0.0853) 25 - 17.4844 (0.0696) 584 + 17.3281 (0.0595) 220 17.2969 (0.0577)	330 ( 17.7869 (0.0907) 7909 running 17.5900 (0.0745) 25 - 17.5632 (0.0726) 220 17.5470 (0.0714) 584 + 17.3358 (0.0578)	0.0156	0.1970	0.0156	0.1970
196	145373 uploads	5370 ~	[redacted]	5370 ~ 18.3750 (0.1110) [redacted] 18.3281 (0.1059) 4557 single 18.1875 (0.0920) 220 17.6562 (0.0541) 262 a 16.6250 (0.0193)	[redacted] 18.3347 (0.1033) 5370 ~ 18.3206 (0.1019) 4557 single 18.1539 (0.0862) 220 17.7276 (0.0563) 262 a 16.6158 (0.0185)	0.0469	0.0141	0.0469	0.0141
198	33 5	451 20	[redacted]	451 20 19.0625 (0.0705) [redacted] 19.0156 (0.0673) 1028 30 18.9219 (0.0612) 1391 40 18.5938 (0.0441) 33 5 18.4688 (0.0389)	[redacted] 19.0620 (0.0698) 451 20 19.0604 (0.0697) 1028 30 18.8971 (0.0592) 1391 40 18.5694 (0.0427) 33 5 18.5279 (0.0409)	0.0469	0.0016	0.0469	0.0016
256	24077 accessible	79374 usable	7441 yet	79374 usable 18.2188 (0.0772) 1792 used 18.0625 (0.0661) 393 for 18.0312 (0.0640) 7441 yet 17.9375 (0.0583) 45442 downloaded 17.6875 (0.0454)	7441 yet 18.3273 (0.0827) 79374 usable 18.2196 (0.0743) 393 for 17.8796 (0.0529) 191570 gated 17.7769 (0.0477) 436 on 17.7459 (0.0462)	0.1562	0.1077	0.2812	0.1077
366	20435 006	28 0	30 2	28 0 19.7969 (0.0363) 29 1 19.7969 (0.0363) 30 2 19.7969 (0.0363) 37 9 19.7969 (0.0363) 31 3 19.7812 (0.0357)	30 2 19.7981 (0.0362) 28 0 19.7943 (0.0361) 37 9 19.7925 (0.0360) 29 1 19.7924 (0.0360) 31 3 19.7868 (0.0358)	0.0000 (tie)	0.0038	0.0000	0.0038
417	2310 90	28 0	30 2	28 0 19.7344 (0.0361) 29 1 19.7344 (0.0361) 30 2 19.7344 (0.0361) 31 3 19.7344 (0.0361) 34 6 19.7344 (0.0361)	30 2 19.7095 (0.0340) 31 3 19.6895 (0.0333) 29 1 19.6860 (0.0332) 28 0 19.6846 (0.0332) 37 9 19.6841 (0.0332)	0.0000 (tie)	0.0200	0.0000	0.0249
418	82 f	78 b	80 d	78 b 19.8125 (0.1102) 82 f 19.8125 (0.1102) 80 d 19.7969 (0.1085) 79 c 19.7812 (0.1069) 81 e 19.7812 (0.1069)	80 d 19.8180 (0.1089) 78 b 19.8094 (0.1079) 82 f 19.7971 (0.1066) 81 e 19.7965 (0.1066) 79 c 19.7896 (0.1058)	0.0000 (tie)	0.0086	0.0156	0.0086
474	34 6	78 b	82 f	78 b 19.7500 (0.0358) 80 d 19.7500 (0.0358) 81 e 19.7500 (0.0358) 82 f 19.7500 (0.0358) 77 a 19.7188 (0.0347)	82 f 19.7611 (0.0360) 78 b 19.7561 (0.0358) 81 e 19.7550 (0.0358) 80 d 19.7516 (0.0357) 77 a 19.7328 (0.0350)	0.0000 (tie)	0.0049	0.0000	0.0049
476	35 7	28 0	31 3	28 0 19.7500 (0.0321) 29 1 19.7500 (0.0321) 30 2 19.7500 (0.0321) 31 3 19.7500 (0.0321) 33 5 19.7344 (0.0316)	31 3 19.7311 (0.0324) 29 1 19.7307 (0.0324) 28 0 19.7258 (0.0322) 30 2 19.7248 (0.0322) 36 8 19.7241 (0.0321)	0.0000 (tie)	0.0004	0.0000	0.0053
498	26 .	54068 -layer	26 .	54068 -layer 18.7656 (0.2348) 26 . 18.7344 (0.2276) 18866 layers 18.1406 (0.1257) 9278 × 17.2656 (0.0524) 100 x 17.0312 (0.0415)	26 . 18.8039 (0.2312) 54068 -layer 18.7846 (0.2268) 18866 layers 18.1649 (0.1220) 9278 × 17.3524 (0.0541) 100 x 16.9870 (0.0376)	0.0312	0.0193	0.0312	0.0193
608	97751 towers	3446 enc	97751 towers	3446 enc 18.5781 (0.2701) 97751 towers 18.5156 (0.2537) 1339 sub 17.1406 (0.0641) 22215 experts 17.1250 (0.0631) 19997 modules 16.7656 (0.0441)	97751 towers 18.5443 (0.2620) 3446 enc 18.5230 (0.2565) 22215 experts 17.1710 (0.0664) 1339 sub 17.1676 (0.0661) 19997 modules 16.8108 (0.0463)	0.0625	0.0213	0.0625	0.0213
645	330 (	38 :	330 (	38 : 19.0938 (0.2358) 330 ( 19.0625 (0.2285) 517 with 17.5781 (0.0518) 19 ' 17.5000 (0.0479) 583 are 17.4375 (0.0450)	330 ( 19.1316 (0.2396) 38 : 19.0982 (0.2317) 517 with 17.6484 (0.0544) 19 ' 17.5292 (0.0483) 583 are 17.4929 (0.0465)	0.0312	0.0334	0.0312	0.0334
666	34 6	35 7	36 8	35 7 18.8594 (0.2001) 36 8 18.8594 (0.2001) 34 6 18.7969 (0.1879) 37 9 18.7812 (0.1850) 33 5 18.1406 (0.0975)	36 8 18.8706 (0.1966) 35 7 18.8624 (0.1950) 34 6 18.8214 (0.1871) 37 9 18.7889 (0.1811) 33 5 18.1900 (0.0995)	0.0000 (tie)	0.0082	0.0000	0.0082
709	923 16	6879 81	2561 06	6879 81 18.8750 (0.0118) 2561 06 18.8438 (0.0114) 5181 86 18.8125 (0.0111) 981 14 18.7969 (0.0109) 3022 07 18.7969 (0.0109)	2561 06 19.1906 (0.0113) 6879 81 19.1813 (0.0112) 3022 07 19.1692 (0.0111) 981 14 19.1397 (0.0108) 5181 86 19.1395 (0.0107)	0.0312	0.0093	0.0312	0.0093
717	44588 Architecture	15018 Software	2473 What	15018 Software 18.3594 (0.0801) 2473 What 18.2344 (0.0707) 13395 GL 17.4062 (0.0309) 6287 Model 17.3438 (0.0290) 22263 Tool 17.2812 (0.0272)	2473 What 18.5399 (0.0837) 15018 Software 18.1092 (0.0544) 6287 Model 17.6319 (0.0338) 13395 GL 17.2938 (0.0241) 22263 Tool 17.1748 (0.0214)	0.1250	0.4307	0.1250	0.4307
788	3441 46	923 16	28 0	923 16 18.9375 (0.2100) 28 0 18.7031 (0.1661) 29 1 18.3281 (0.1142) 859 15 17.8906 (0.0737) 32 4 16.9375 (0.0284)	28 0 18.9859 (0.1913) 923 16 18.8289 (0.1635) 29 1 18.4931 (0.1169) 859 15 18.2548 (0.0921) 32 4 17.2333 (0.0331)	0.2344	0.1571	0.2344	0.1571
825	39 ;	24 ,	1809 only	24 , 19.0469 (0.3235) 1809 only 18.9844 (0.3039) 39 ; 18.9531 (0.2945) 330 ( 17.1719 (0.0496) 3699 |\n 15.6016 (0.0103)	1809 only 19.0083 (0.3193) 24 , 18.9821 (0.3110) 39 ; 18.9152 (0.2909) 330 ( 17.1570 (0.0501) 3699 |\n 15.6059 (0.0106)	0.0625	0.0262	0.0625	0.0262
855	24 ,	330 (	24 ,	330 ( 19.6875 (0.4697) 24 , 19.5781 (0.4210) 39 ; 17.6562 (0.0616) 39485 (= 16.0469 (0.0123) 319 = 15.1484 (0.0050)	24 , 19.6283 (0.4441) 330 ( 19.6244 (0.4424) 39 ; 17.7274 (0.0664) 39485 (= 15.9510 (0.0112) 319 = 15.0763 (0.0047)	0.1094	0.0039	0.1094	0.0038
864	85743 ,o	53793 projections	85743 ,o	53793 projections 17.0469 (0.1257) 85743 ,o 16.6094 (0.0811) 319 = 16.2969 (0.0594) 666 per 16.1719 (0.0524) 583 are 16.0156 (0.0448)	85743 ,o 17.0580 (0.1341) 53793 projections 16.8749 (0.1116) 319 = 16.2002 (0.0569) 666 per 16.0760 (0.0502) 583 are 15.9982 (0.0465)	0.4375	0.1831	0.4375	0.1831
868	12814 -width	1632 '	91986 -rank	1632 ' 17.7969 (0.1092) 91986 -rank 17.7188 (0.1010) 95280 -att 17.4844 (0.0799) 12814 -width 17.0625 (0.0524) 54068 -layer 17.0312 (0.0508)	91986 -rank 17.8554 (0.1112) 1632 ' 17.8109 (0.1063) 95280 -att 17.3803 (0.0691) 54068 -layer 16.9042 (0.0429) 12814 -width 16.8966 (0.0426)	0.0781	0.0445	0.0781	0.0445
957	100 x	7173 weight	15422 mean	7173 weight 18.2031 (0.1559) 15422 mean 18.1875 (0.1535) 1537 no 17.3281 (0.0650) 158274 RMS 17.1719 (0.0556) 126230 subtract 17.1094 (0.0522)	15422 mean 18.2420 (0.1673) 7173 weight 18.1143 (0.1472) 1537 no 17.2573 (0.0625) 126230 subtract 17.2172 (0.0600) 158274 RMS 17.1455 (0.0559)	0.0156	0.1277	0.0156	0.1277
1002	4316 normal	4686 weight	80838 RMS	4686 weight 19.3281 (0.2295) 80838 RMS 19.2344 (0.2089) 831 x 18.5938 (0.1101) 8053 norm 18.1875 (0.0733) 9237 multiply 17.3906 (0.0331)	80838 RMS 19.2915 (0.2250) 4686 weight 19.2808 (0.2226) 831 x 18.5611 (0.1084) 8053 norm 18.1391 (0.0711) 9237 multiply 17.3152 (0.0312)	0.0938	0.0107	0.0937	0.0107
1080	220	220	25648 residual	220 17.6562 (0.0546) 25648 residual 17.6094 (0.0521) 2277 post 17.5469 (0.0489) 3721 ** 17.4375 (0.0439) 2917 input 17.3125 (0.0387)	25648 residual 17.5494 (0.0570) 6267 device 17.5289 (0.0558) 220 17.4519 (0.0517) 4469 full 17.1749 (0.0392) 2277 post 17.1611 (0.0386)	0.0469	0.0205	0.0469	0.0975
1132	13653 expert	22215 experts	13653 expert	22215 experts 18.1406 (0.4744) 13653 expert 18.1250 (0.4670) [redacted] 14.5078 (0.0125) 27 / 13.8281 (0.0064) 61602 /shared 13.7344 (0.0058)	13653 expert 18.1329 (0.4847) 22215 experts 18.0790 (0.4592) [redacted] 14.4108 (0.0117) 27 / 13.7274 (0.0059) 61602 /shared 13.5904 (0.0052)	0.0156	0.0539	0.0156	0.0539
1154	6506 256	36 8	6506 256	36 8 18.9531 (0.1779) 6506 256 18.8750 (0.1645) 7325 128 18.1875 (0.0827) 32 4 18.1719 (0.0814) 923 16 18.1250 (0.0777)	6506 256 18.9124 (0.1774) 36 8 18.8892 (0.1733) 7325 128 18.1396 (0.0819) 32 4 18.0992 (0.0786) 923 16 18.0509 (0.0749)	0.0781	0.0232	0.0781	0.0232
1191	24 ,	24 ,	757 ),	24 , 19.5000 (0.3248) 757 ), 19.5000 (0.3248) 21 ) 17.5625 (0.0468) 220 17.4531 (0.0420) 39 ; 17.4062 (0.0400)	757 ), 19.5342 (0.3363) 24 , 19.4793 (0.3184) 21 ) 17.5866 (0.0480) 39 ; 17.3683 (0.0386) 220 17.3665 (0.0385)	0.0000 (tie)	0.0548	0.0000	0.0548
1280	15216 176	19798 007	29 1	19798 007 15.3281 (0.0761) 29 1 15.2891 (0.0732) 2253 05 14.6719 (0.0395) 27626 025 14.2969 (0.0271) 18921 020 14.2344 (0.0255)	29 1 14.9829 (0.0822) 19798 007 14.9237 (0.0774) 2253 05 14.3830 (0.0451) 27626 025 13.8383 (0.0261) 18921 020 13.8224 (0.0257)	0.0391	0.0592	0.0391	0.0592
1297	36 8	29 1	30 2	29 1 19.2812 (0.1033) 30 2 19.2812 (0.1033) 31 3 19.0625 (0.0830) 32 4 18.7969 (0.0636) 33 5 18.7812 (0.0626)	30 2 19.2960 (0.1029) 29 1 19.2877 (0.1021) 31 3 19.0825 (0.0832) 32 4 18.8162 (0.0637) 33 5 18.8116 (0.0634)	0.0000 (tie)	0.0083	0.0000	0.0083
1328	78 b	78 b	82 f	78 b 19.7031 (0.0371) 82 f 19.7031 (0.0371) 80 d 19.6875 (0.0365) 81 e 19.6719 (0.0360) 77 a 19.6562 (0.0354)	82 f 19.7065 (0.0370) 78 b 19.6998 (0.0367) 80 d 19.6806 (0.0360) 81 e 19.6710 (0.0357) 79 c 19.6631 (0.0354)	0.0000 (tie)	0.0067	0.0000	0.0067
1358	83 g	742 res	2499 input	742 res 18.8438 (0.1079) 2499 input 18.8438 (0.1079) 4171 post 18.7812 (0.1013) 7173 weight 18.2188 (0.0578) 93 q 18.0156 (0.0471)	2499 input 18.8909 (0.1150) 742 res 18.8489 (0.1103) 4171 post 18.7240 (0.0973) 7173 weight 18.2168 (0.0586) 93 q 17.9875 (0.0466)	0.0000 (tie)	0.0420	0.0000	0.0420
1362	31976 scal	25648 residual	191570 gated	25648 residual 18.7500 (0.0803) 191570 gated 18.7188 (0.0779) 2917 input 18.5469 (0.0656) 2277 post 18.4375 (0.0588) 916 no 18.3594 (0.0544)	191570 gated 18.7795 (0.0836) 25648 residual 18.7036 (0.0775) 2917 input 18.5751 (0.0681) 2277 post 18.3995 (0.0571) 916 no 18.2200 (0.0478)	0.0312	0.0759	0.0312	0.0759
1370	4469 full	436 on	24 ,	436 on 19.1094 (0.2218) 24 , 18.9688 (0.1927) 21 ) 18.3281 (0.1015) 4469 full 18.2500 (0.0939) 23 + 17.4062 (0.0404)	24 , 19.1240 (0.2132) 436 on 19.0531 (0.1986) 21 ) 18.3787 (0.1012) 4469 full 18.0371 (0.0719) 23 + 17.4401 (0.0396)	0.1406	0.0709	0.1406	0.0709
1373	14620 soft	25648 residual	2917 input	25648 residual 18.6406 (0.0856) 2917 input 18.5781 (0.0804) 171263 tanh 18.5312 (0.0767) 2431 same 18.2344 (0.0570) 2277 post 18.1562 (0.0527)	2917 input 18.7392 (0.0909) 25648 residual 18.5550 (0.0756) 171263 tanh 18.4339 (0.0669) 2431 same 18.3126 (0.0593) 2277 post 18.2246 (0.0543)	0.0625	0.1842	0.0625	0.1842
1392	53496 latent	290 the	49557 sliding	290 the 19.0625 (0.0913) 3318 No 18.8750 (0.0757) 49557 sliding 18.8438 (0.0733) 53496 latent 18.5625 (0.0554) 2917 input 18.2656 (0.0411)	49557 sliding 19.0897 (0.0966) 290 the 18.9839 (0.0869) 3318 No 18.7374 (0.0679) 53496 latent 18.5618 (0.0570) 10131 Mo 18.3349 (0.0454)	0.1875	0.1058	0.2188	0.1058
1421	12069 attention	341 and	49557 sliding	341 and 19.0312 (0.1018) 2917 input 18.9219 (0.0913) 49557 sliding 18.8906 (0.0885) 26355 gate 18.7656 (0.0781) 12069 attention 18.2344 (0.0459)	49557 sliding 19.0545 (0.1030) 341 and 19.0439 (0.1019) 2917 input 19.0067 (0.0982) 26355 gate 18.7493 (0.0759) 12069 attention 18.1772 (0.0428)	0.1094	0.0106	0.1406	0.0106
1437	36119 loader	16658 kernel	79744 kernels	16658 kernel 17.0625 (0.0982) 79744 kernels 17.0625 (0.0982) 8514 layer 16.5781 (0.0605) 4036 quant 16.3594 (0.0486) 2978 path 15.9531 (0.0324)	79744 kernels 17.2875 (0.1116) 16658 kernel 17.2402 (0.1065) 8514 layer 16.6306 (0.0579) 4036 quant 15.9795 (0.0302) 2978 path 15.9447 (0.0291)	0.0000 (tie)	0.0473	0.0000	0.0473
1454	325 e	7191 scale	29530 scales	7191 scale 18.8594 (0.3659) 29530 scales 18.8594 (0.3659) 325 e 18.0938 (0.1702) 4061 block 16.2031 (0.0257) 427 E 16.0000 (0.0210)	29530 scales 18.8506 (0.3653) 7191 scale 18.8191 (0.3540) 325 e 18.1126 (0.1746) 4061 block 16.3138 (0.0289) 427 E 16.0734 (0.0227)	0.0000 (tie)	0.0315	0.0000	0.0315
1463	30195 Mi	81 e	1271 32	81 e 17.2344 (0.0444) 1271 32 17.1719 (0.0417) 63410 every 16.9062 (0.0320) 7731 see 16.8906 (0.0315) 22145 scale 16.7969 (0.0286)	1271 32 17.2393 (0.0467) 81 e 17.2307 (0.0463) 63410 every 17.1295 (0.0418) 22145 scale 16.7447 (0.0285) 7731 see 16.7246 (0.0279)	0.0625	0.0085	0.0625	0.0085
1541	5023 52	28 0	32 4	28 0 19.2656 (0.1162) 32 4 19.2344 (0.1127) 33 5 19.2031 (0.1092) 30 2 19.1875 (0.1075) 29 1 19.1719 (0.1058)	32 4 19.2545 (0.1158) 28 0 19.2531 (0.1156) 33 5 19.2167 (0.1115) 30 2 19.1775 (0.1072) 29 1 19.1506 (0.1044)	0.0312	0.0013	0.0312	0.0013
1560	721 |	584 +	721 |	584 + 19.0156 (0.2645) 721 | 18.9688 (0.2524) 24 , 18.6875 (0.1905) 330 ( 17.5469 (0.0609) 756 / 17.1719 (0.0418)	721 | 19.0179 (0.2639) 584 + 18.9921 (0.2572) 24 , 18.6510 (0.1829) 330 ( 17.5954 (0.0636) 756 / 17.1204 (0.0396)	0.0469	0.0257	0.0469	0.0258
1581	3284 66	3195 65	3284 66	3195 65 17.9688 (0.4612) 3284 66 17.9688 (0.4612) 35 7 14.7422 (0.0183) 3804 67 13.8281 (0.0073) 34 6 13.8125 (0.0072)	3284 66 17.9281 (0.5151) 3195 65 17.6953 (0.4082) 35 7 14.5453 (0.0175) 40216 656 13.5821 (0.0067) 3804 67 13.5705 (0.0066)	0.0000 (tie)	0.2328	0.0000	0.2328
1606	2859 44	3763 42	4073 43	3763 42 18.0938 (0.3698) 4073 43 18.0938 (0.3698) 2859 44 16.8594 (0.1076) 2127 45 15.6094 (0.0308) 4040 41 15.1562 (0.0196)	4073 43 18.1860 (0.4055) 3763 42 17.9439 (0.3183) 2859 44 16.9721 (0.1204) 2127 45 15.7323 (0.0349) 4040 41 15.1889 (0.0203)	0.0000 (tie)	0.2422	0.0000	0.2421
1627	124235 replicated	19418 kept	583 are	19418 kept 18.3125 (0.1278) 583 are 18.2656 (0.1219) 173298 untouched 17.9844 (0.0920) 9353 stay 17.9375 (0.0878) 43890 excluded 17.8438 (0.0799)	583 are 18.2799 (0.1230) 19418 kept 18.2659 (0.1212) 9353 stay 18.0057 (0.0935) 43890 excluded 17.9094 (0.0849) 173298 untouched 17.8742 (0.0819)	0.0469	0.0140	0.0469	0.0140
1650	721 |	2386 size	721 |	2386 size 18.3125 (0.2008) 721 | 18.2031 (0.1800) 17460 GB 17.9062 (0.1337) 330 ( 17.5938 (0.0978) 8571 Gi 17.4062 (0.0811)	721 | 18.3228 (0.2016) 2386 size 18.2601 (0.1894) 17460 GB 17.7976 (0.1193) 330 ( 17.6605 (0.1040) 436 on 17.2257 (0.0673)	0.1094	0.0627	0.1094	0.0627
1662	50074 ceiling	330 (	3699 |\n	330 ( 18.2969 (0.1271) 3699 |\n 18.2656 (0.1232) 16025 tok 18.0625 (0.1005) 721 | 17.3906 (0.0513) 22274 tokens 17.1719 (0.0413)	3699 |\n 18.3570 (0.1284) 330 ( 18.2951 (0.1207) 16025 tok 18.0186 (0.0915) 721 | 17.4919 (0.0541) 260 t 17.1856 (0.0398)	0.0312	0.0619	0.0313	0.0619
1663	829 @	330 (	3699 |\n	330 ( 19.2344 (0.3066) 3699 |\n 19.1719 (0.2880) 22 * 18.5469 (0.1542) 721 | 17.6094 (0.0604) 829 @ 17.2812 (0.0435)	3699 |\n 19.2093 (0.2954) 330 ( 19.1883 (0.2892) 22 * 18.5056 (0.1461) 721 | 17.7754 (0.0704) 829 @ 17.5047 (0.0537)	0.0625	0.0209	0.0625	0.0209
1751	2530 75	33 5	28 0	33 5 18.5156 (0.3603) 28 0 18.3906 (0.3179) 34 6 16.7656 (0.0626) 2530 75 16.6719 (0.0570) 963 25 15.9844 (0.0287)	28 0 18.4395 (0.3439) 33 5 18.4124 (0.3347) 2530 75 16.7337 (0.0625) 34 6 16.6920 (0.0599) 963 25 15.9743 (0.0292)	0.1250	0.0271	0.1250	0.0271
1907	25326 cheap	220	5370 ~	220 19.0781 (0.2198) 5370 ~ 19.0625 (0.2163) 54758 negligible 18.7969 (0.1659) 702 not 17.9844 (0.0736) 30612 tiny 17.6719 (0.0539)	5370 ~ 19.1278 (0.2256) 220 19.0797 (0.2150) 54758 negligible 18.8088 (0.1640) 702 not 17.9327 (0.0683) 30612 tiny 17.6731 (0.0527)	0.0156	0.0481	0.0156	0.0481
1929	7325 128	7325 128	1915 64	7325 128 16.5156 (0.4464) 1915 64 16.5000 (0.4395) 3195 65 13.4609 (0.0210) 1271 32 13.3672 (0.0192) 13336 131 13.3125 (0.0181)	1915 64 16.5057 (0.4836) 7325 128 16.3324 (0.4067) 3195 65 13.4062 (0.0218) 1271 32 13.2310 (0.0183) 13336 131 13.1940 (0.0176)	0.0156	0.1732	0.0156	0.1732
1935	12733 184	30 2	32 4	30 2 18.7031 (0.1378) 32 4 18.6250 (0.1274) 36 8 18.5312 (0.1160) 923 16 18.5000 (0.1124) 29 1 18.3906 (0.1008)	32 4 18.6925 (0.1352) 30 2 18.6715 (0.1324) 36 8 18.5541 (0.1177) 923 16 18.5083 (0.1124) 29 1 18.3171 (0.0929)	0.0781	0.0210	0.0781	0.0210
2056	[redacted]	[redacted]	[redacted]	[redacted] 17.1094 (0.1352) [redacted] 17.0781 (0.1311) [redacted] 16.8906 (0.1087) [redacted] 16.5156 (0.0747) [redacted] 16.4375 (0.0691)	[redacted] 17.1944 (0.1485) [redacted] 17.0752 (0.1318) [redacted] 16.8573 (0.1060) [redacted] 16.4469 (0.0703) [redacted] 16.4076 (0.0676)	0.0312	0.1192	0.0313	0.1192
2097	22670 generic	86670 CUDA	7885 EX	86670 CUDA 18.2969 (0.0755) 7885 EX 18.1250 (0.0636) 38161 MX 17.4062 (0.0310) 1809 only 17.2812 (0.0274) 48570 Spark 17.1562 (0.0241)	7885 EX 18.3182 (0.0747) 86670 CUDA 18.2401 (0.0691) 38161 MX 17.6005 (0.0365) 1809 only 17.3528 (0.0285) 12184 relevant 17.1933 (0.0243)	0.1719	0.0781	0.1719	0.0781
2099	18326 pieces	24249 TP	7885 EX	24249 TP 18.4688 (0.0981) 2978 path 18.4531 (0.0966) 7885 EX 18.3594 (0.0879) 7270 engine 18.0000 (0.0614) 10131 Mo 17.4688 (0.0361)	7885 EX 18.6092 (0.1049) 24249 TP 18.4928 (0.0934) 2978 path 18.4271 (0.0874) 7270 engine 18.0514 (0.0600) 10131 Mo 17.6722 (0.0411)	0.0156	0.1164	0.1094	0.1164
2109	13038 Ro	86670 CUDA	1632 '	86670 CUDA 17.2344 (0.0391) 66449 CX 17.2188 (0.0385) 1632 ' 17.1562 (0.0361) 47227 RD 16.6250 (0.0212) 220 16.4531 (0.0179)	1632 ' 17.1985 (0.0371) 86670 CUDA 17.1843 (0.0366) 66449 CX 16.6883 (0.0223) 47227 RD 16.6279 (0.0210) 28625 IB 16.5791 (0.0200)	0.0156	0.0143	0.0781	0.0143
2118	6267 device	666 per	1632 '	666 per 17.2031 (0.0223) 1632 ' 17.2031 (0.0223) 7539 session 16.8906 (0.0163) 853 pre 16.8594 (0.0158) 86670 CUDA 16.6875 (0.0133)	1632 ' 17.2286 (0.0219) 666 per 17.2211 (0.0218) 7539 session 16.9690 (0.0169) 853 pre 16.8876 (0.0156) 86670 CUDA 16.6990 (0.0129)	0.0000 (tie)	0.0075	0.0000	0.0075
2143	2978 path	12462 cache	27452 chunk	12462 cache 16.1875 (0.0234) 27452 chunk 16.1719 (0.0230) 34371 -mode 15.7891 (0.0157) 25 - 15.7812 (0.0156) 13237 batch 15.7656 (0.0154)	27452 chunk 16.1916 (0.0241) 12462 cache 16.1703 (0.0236) 25 - 15.7978 (0.0163) 13237 batch 15.7387 (0.0153) 27 / 15.7326 (0.0152)	0.0156	0.0213	0.0156	0.0213
2253	1487 ml	3615 mo	113235 attention	3615 mo 17.6562 (0.1197) 113235 attention 17.6250 (0.1161) 89 m 17.0156 (0.0631) 9806 linear 16.8594 (0.0540) 1683 att 16.8438 (0.0531)	113235 attention 17.6088 (0.1154) 3615 mo 17.5631 (0.1103) 89 m 16.9804 (0.0616) 9806 linear 16.8682 (0.0550) 1654 util 16.8360 (0.0533)	0.0312	0.0456	0.0312	0.0456
2293	108918 Spark	108918 Spark	48570 Spark	108918 Spark 17.6719 (0.0844) 63 S 17.6094 (0.0793) 48570 Spark 17.5156 (0.0722) 103612 Attention 17.4062 (0.0647) 948 's 17.3906 (0.0637)	48570 Spark 17.6849 (0.0861) 108918 Spark 17.6830 (0.0859) 63 S 17.6644 (0.0843) 948 's 17.3839 (0.0637) 103612 Attention 17.3190 (0.0597)	0.0625	0.0019	0.1563	0.0019
2421	24 ,	335 .\n	420 ,\n	335 .\n 19.3281 (0.3919) 420 ,\n 19.2969 (0.3799) 198 \n 17.7344 (0.0796) 308 ;\n 17.3438 (0.0539) 24 , 16.9219 (0.0353)	420 ,\n 19.3628 (0.4112) 335 .\n 19.2069 (0.3519) 198 \n 17.7442 (0.0815) 308 ;\n 17.4788 (0.0625) 24 , 16.9683 (0.0375)	0.0312	0.1559	0.0312	0.1559

[ajensenwaud]

@ashhart could you rerun zig-windows when you get a moment? Its one failure is main's rows_close_test "a blocked spill-only flush cannot hold shutdown past its deadline", a timing check in code this PR doesn't touch; the Windows build itself compiled and every other test passed. Rerunning needs admin rights, so I can't.

[ajensenwaud]

Thanks @ashhart. Pushed, rebased onto current main (df5e4de6):

• tp.module fix in 2028559c: zig/build/tp.zig keeps one tp module per cuda module and returns it on later calls. Before it, zig build check-release-paths failed with file exists in modules 'tp' and 'tp0'; after it, it passes. The fix is byte-identical on MiMo-V2.6-Flash on CUDA, on two GB10s with its DFlash drafter #728.
• zig/src/server is server: split recipient-addressed replies into reasoning, answer and tool calls #1336's rebased version, so both PRs carry identical server files.
• Other conflicts were additive: the CUDA registry (minimax.native and muse_spark both kept), and the contract record and image lists (exl3_grouped_records, MiniMax-M3 and MOSS-Image kept, Muse appended).
• Main's contract_tests.zig is at 599 lines, so Muse's contract group joins contract_family_sets.zig (beside the GB10 affine-4 set), as fdd8b6dc and a4b3b060 did. This is folded into 358da6ad, and contract_tests.zig is main's.

Locally on x86_64 Linux: zig build test, test-kernel-contract, test-golden, test-server-security, test-pull-security, check-release-paths, compile-cuda-tests and tf-muse pass. fmt is clean, and lean finds 424 problems, as main. I couldn't run nvcc or a device here, so the cluster fix and the two-GB10 cells are still with you.

[ajensenwaud]

All good to go.

[ashhart]

Thanks @ajensenwaud, and sorry this has waited since 1 Nov. Everything we asked for is in. As agreed, we'll carry the cluster fix (traps below sm_90), resolve the new conflicts with main when we prepare it, and run the two-GB10 cells at TP=2 with pack revision 38ad2925: the fatbin cmp and contract, graphs against eager, 1 and 4 streams at temperature 0 and 0.7, resumed against fresh, #1336's cases, and a GLM control. We'll post the results here.

[ashhart]

/atc exchange b748653a7860856b1e83ee99c65ffc53f24a6ba6ab843548e7738a08d54cbaa4

[ashhart]

@ajensenwaud the same for this one: we're merging today's main into 2028559c2, with the cluster fix (traps below sm_90) carried as its own commit. The merge touches the TP build, the contract registry, native/cuda.zig and server/chat.zig, and it's checked against #728, since the two share those files. The two-GB10 TP=2 cells with pack revision 38ad2925 follow on the merged head, after the 1.2.6 release reads. Thanks for your patience.

[ashhart]

@ajensenwaud, brought up to today's main and pushed (2f238883e6): main merged (registrations and the TP build keep both sides), the cluster kernels trap below sm_90 as the other cluster kernels do, TF_TP_WORLD goes through the strict environment rules, the drafter knobs are tool-only names, kernels.zig is re-pinned, and the copies are in the shared-copies allowlist. test-muse 26/26, test-server-cpu and the Linux CUDA builds pass. #728 and this touch the same registration lines, so whichever lands second gets a small merge from us. What's left is ours: the two-GB10 TP=2 cells on pack 38ad2925.

• Synthena (Ash's Agent)

[ajensenwaud]

OK, thank you.

[ashhart]

Thanks @ajensenwaud. Everything we asked you to change is accounted for, including the plain-decode recipe, admission checks and cluster-kernel guards.

This still waits on us: the merge with current main, followed by the two-GB10 TP=2 checks on pack 38ad2925. We’ll check the shared kernels, graph/eager and resumed/fresh equality, and #1336’s served cases together. Nothing further is needed from you.

• Synthena (Ash's Agent)