[yguanrong]

What this changes

Qwen CUDA development builds on SM110 refuse unqualified serving opt-ins by name before paging, sharing or weight loading. Existing BF16 and DFlash2 paths keep their kernels, arithmetic and lane scheduling.

• Issue: Refs CUDA: qualify Jetson AGX Thor (sm_110) for native Qwen3.8-27B MLX 4-bit and DFlash2 #2319. This is Qwen development-target admission; product targeting and packaging remain separate.

• Stacked on: rebased onto a9f3f959d75fcc32e6d1ed5c049dd54e72db91f9; current main is d001c28f296a0b886876afa158dd8d9151da03c4. A read-only merge-tree against current main is clean and GitHub reports MERGEABLE. No parent PR.

• atc overlap --json: Final audit on 2026-11-13 against the pinned main: 324/324 current open PR heads and 27/27 other local worktrees, zero unknown, zero added conflict files and zero added conflict blocks in candidate-touched files versus main. One transient object-write error cleared on serial retry. Native ATC still fails with unexpected error after an extended deadline (earlier otherPullRequestsLimit/pullFilesLimit); a fresh rebase-head attempt on 2026-11-14 again returned unexpected error, so this supplemental audit is not labeled a complete native ATC result. Windows: source-build the 5090 preview as tensorfold.exe #2809 now has the same five conflict files and 12 blocks as main, including one block in cuda_native.zig; the additional startup block is removed. The guard follows the existing captured-kernel and slide-format refusals and precedes paging/sharing.

• Rebase scope: all 11 commits were replayed. git range-diff shows commits 4-10 exactly equal; commits 1-3 changed only CHANGELOG.md conflict context, and the final docs commit moved (#3057) to the current Unreleased. The final diff still touches the same 11 files. git diff --check and the conflict-marker scan pass.

Checks

• Head commit: 6182325c48830b5954f3c75c0daa4d3f29969cdd, verified on the remote branch. The rebased functional source commit is 609b309d7788a8b94fa6b9c676eb62ecb49d97b5; git range-diff shows it is exactly equivalent to the historically measured 4bde3c04de85434729263ec1d412fd8708beb9f8. The final commit changes only CHANGELOG.md and docs/recipes/qwen3.8-27b.md. Device receipts below remain tied to the historical original-base binary and are not relabeled as exact receipts for the rebased merged tree.
• Final python tools/pr_check.py --base upstream/main --pr 3057: 0 errors, 0 warnings; lean 341 vs 341, 0 new.
• Tests run, with the command and pass, fail and skip counts: Windows zig build test-qwen35-sm110-host test-qwen35-request-starts-host test-qwen35-draft-config-host -j2 --summary all: 11/11 steps, 59/59 tests, exit 0. Linux ARM64 zig build native test-qwen35-sm110-host test-qwen35-request-starts-host test-qwen35-draft-config-host -Dnvcc=/usr/local/cuda/bin/nvcc -Dsm=110 -j2 --summary all: 135/135 steps, 59/59 tests, exit 0. Negative control on the measured functional head 4bde3c04de85434729263ec1d412fd8708beb9f8 with the guards disabled: 18/20 tests, two expected startup failures, exit 1; restored source passes. git diff --check passes. Full-tree zig fmt --check build.zig zig finds cuda.zig and cuda_native_explain.zig; both also fail the same check on base. Focused new/modified guard files and cuda_native.zig pass formatting.
• CI on the rebased head: 7 applicable checks pass, 2 nightly jobs are skipped by policy, and Linux zig was cancelled at 1h0m17s by the repository's 60-minute job timeout: https://github.com/ashhart/TensorFold/actions/runs/43532254037 . Within that job, zig build test-host and zig build test-kernel-contract both succeeded; zig build test-golden reported 322/335 equal to server goldens, 0 unexpected differences before the runner cancelled it, and five later steps were skipped. The cancellation is not recorded as a pass. A failed-job rerun was requested with gh run rerun --failed, but GitHub refused it because this account has no repository admin rights. Historical context: all eight applicable checks passed on the prior docs-only tip 818102d657431a8761d40f4d431d81d4f7628b68 in https://github.com/ashhart/TensorFold/actions/runs/43350914812 .
• Not run, and why: the complete host suite is not claimed. Historical cuda.stager failure is every byte lands where the mapping would have put it, for any copies and read limit, at zig/src/families/drama3/cuda/stager.zig:294, expectEqualSlices(u8, want, got), with the build tree on ext4. This is tracked separately from the Qwen admission change. Metal, HIP, XPU and other CUDA targets were not device-tested because those devices are unavailable.

Historical receipt (pre-rebase)

• Exact final-head repeat: https://gist.github.com/yguanrong/15f1aab59a7798abfb458dd484b4aec9 . Final 4bde3c04de against the same main: complete base1/new1/new2/base2, 20/24 strict decode rows; not passed, analyzer exit 1; drafted/plain 12/12 pairs in both candidates; 896 decode and 32 cold requests, zero token differences. Four failed rows are new2 chat greedy width1 plain -0.1895%, chat sampled width1 plain -0.0074%, chat sampled width4 drafted -1.5644%, code sampled width1 plain -0.2537%. Deltas use min(base1,base2), no tolerance. The failed-cell set differs from the prior round; this does not by itself prove noise. The maintainer subsequently reviewed both rounds together with the token equality, startup scope and embedded fatbin identity evidence and determined the misses are run-to-run noise; the raw strict result remains recorded unchanged.
• Main against branch: the complete frozen base1/new1/new2/base2 receipt is public at https://gist.github.com/yguanrong/cd89ee85d655093ef37e22a36a13253b . It measured candidate 006566740dea1e73be59f8c38dbfc77547946763, before the final startup-order adjustment. Strict zero-tolerance decode result: 21/24 rows; not passed. Drafted/plain: 12/12 pairs in both new phases. These results are not relabeled as exact-head timings of the final commit or as noise.
• Boxes it reaches: Qwen native CUDA startup on SM110. Other GPUs retain existing admission behavior; no kernel, shared scheduler or per-request path is modified.
• New switch? Default, accepted values, profile row: n/a; no switch added. Unqualified opt-ins are refused by name, including --slide, DSpine drafter and TF_DSPINE_WINDOW; DFlash2 remains admitted.
• Historical device environment, with the TensorFold commit, Zig and backend toolchain, machine and GPU, checkpoint and revision: original base 6c5cef8941b27d86b06a986834e13deb9b515e1e, measured functional head 4bde3c04de85434729263ec1d412fd8708beb9f8, documentation-only tip 818102d657431a8761d40f4d431d81d4f7628b68; NVIDIA Thor SM110, Linux ARM64, Zig 0.17.0, NVCC 13 from runtime image ID sha256:62e412469ae102e21a939d3c65dbcd38c9999162681052ebb431a259f59e9b63. TensorFold native binary built from source and mounted read-only over that runtime image. TensorFold/Qwen3.8-27B-MLX-4bit and z-lab/Qwen3.8-27B-DFlash2 contents pinned by SHA256 manifest in the public JSON; no model files added. Final native SHA256 d17f9e74f52c2c6ded8c675d1e5fe066065e4456de09c27c43490aca8ce62b3a; measured predecessor native SHA256 8085ecd9aacd33297522ab3f1b624ae03cb6ac8271f2cc40a2ee21650dbc7673; main native SHA256 31c939181d03adf412fa2c2648f6cf66afea25a0515ee19ef82b6117cadc9333.
• Paths and platforms the change reaches, which I ran and which I could not: Windows x86_64 host tests, Linux ARM64/SM110 source build and checkpoint validation. All other chips and backends unrun; no claim of cross-platform device qualification. Context 8448 for performance, one GPU, BF16 KV/prompt, widths 1/2/4, 256 output tokens, cache 0, greedy and temperature 0.8 with fixed seeds.
• Exactness, from tools/bench_concurrent.py --alone --serial or token_sha against "draft": false: measured predecessor has zero mismatches across 896 decode requests and 32 cold requests. Its full conformance is 6 pass, 0 fail, 1 agreed skip (evict), exit 0; drafted/plain 10/10, repeat 10/10, concurrent/solo 44/44, refusal 4/4, cancel 4/4. Separate text/PNG checks, 7424-token long prompt, vision concurrency widths 2/4 and post-cancel next request match. Final-head checkpoint verification completed: startup refusals 14/14 expected, device smoke/state/vision exit 0, and conformance 6 pass/0 fail/1 agreed skip, exit 0. Drafted/plain 10/10, repeat 10/10, concurrent/solo 44/44, resumed/fresh 6/6, refusal 4/4 and cancellation 4/4; 7424-token long prompt, image inputs, vision widths 2/4 greedy/sampled and post-cancel next requests match. Sanitized per-case receipts are in the public evidence. An exact-head four-phase repeat is frozen before timing in comparison-method-4bde3c04de.md in the public receipt; it completed after all device/correctness and corrected-limit jobs, with distinct artifacts and unchanged thresholds. The preceding 21/24 verdict is retained.
• Main against this branch, same machine and checkpoint, posted before asking for a run on our machines: the complete four-phase receipt is public, and the maintainer reviewed it with the exact-head conformance, 14 startup refusals and embedded-fatbin identity evidence. The maintainer explicitly settled the performance gate and said no further timing is needed.
• Decode speed before and after: the frozen base1/new1/new2/base2 receipt retains all 24 rows and per-request data. The raw strict analyzer remains 20/24 with no token differences; the four misses were -0.1895%, -0.0074%, -1.5644% and -0.2537%. After reviewing the repeated cells, equal tokens and startup-only scope with byte-identical embedded fatbins, the maintainer determined these changes are noise and completed the gate. The raw result is not rewritten as a strict 24/24 pass.
• Cold first TTFT (base1/new1/new2/base2): 2k = 3.559389/3.545296/3.545845/3.551591 s; 8k = 14.483654/14.480751/14.491142/14.476264 s. Cached tokens were zero. The maintainer reviewed the 8K spread below 0.1% under the same settled gate and ended further timing. Full cold rows, frequencies and memory are in the public receipt.
• Prompt speed before and after (retained predecessor): cold first TTFT at 2k, base1/new1/new2/base2 = 3.584139/3.562550/3.535614/3.551932 s; at 8k = 14.487238/14.487723/14.485811/14.496990 s, cached tokens 0. The 8k new1 first request is slightly slower than the faster main; no universal prompt-speed pass is claimed. 32k/64k not run under the frozen 8448 context. Device peak allocation 23,855,552,288 bytes in all phases; cgroup peaks are reported separately and do not establish process physical footprint.
• Licence of any new weights or drafter, or "n/a": n/a, none added.

Supplemental full device-contract receipts

No GPU kernel is new or changed. The measured predecessor's 114 production fatbins and 114 normalized SM110 SASS listings match main. On the final served binaries, all 116 literal fatbin blobs found embedded in both binaries are byte-identical, with no embedding-membership differences. All 148 images in the post-contract asset directories are byte-identical and all 148 normalized SM110 SASS listings match (exit 0); that larger directory set includes contract assets and is not mislabeled as 148 production images. Exact lists, hashes and embedded membership are public in embedded-fatbins-sass-4bde3c04de.json. Base full kernel-contract completed with exit 1: SUMMARY 610 kernels: 610 pass gates 1-4, 0 fail, 8 known findings; 407 controls: 407 red as planted, 0 missed; 168 limit problems; gates 5387675 ms, limits 1125357 ms. This is not a full pass. The first standalone base limits invocation exited 127 because the verification script expected an installed runner instead of the actual cached runner; this is not a contract pass and the correction is retained separately. The predecessor full run was deliberately stopped as superseded (incomplete, not passed) so the final-head full run takes priority. Final head full kernel-contract also completed with exit 1: SUMMARY 610 kernels: 610 pass gates 1-4, 0 fail, 8 known findings; 407 controls: 407 red as planted, 0 missed; 168 limit problems; gates 5208888 ms, limits 1114689 ms. A line-for-line multiset comparison finds zero added/removed known findings and zero added/removed limit messages against the pinned base. All 168 problems are inherited missing compile-usage records; seven additional limits messages are N/A. Both full runs remain nonzero, not full passes. The sanitized complete summaries and finding comparison are public as device-contract-comparison-4bde3c04de.json in the receipt. Final-head standalone kernel-contract-limits completed with exit 0: every registered launch fits SM110 from compiled cubins, both controls red. This distinct SM110 cubin audit does not erase the full-run release-record findings. The corrected base standalone invocation also completed with exit 0: 942 passing launch records and two red controls on each side; all messages are identical. The first queued checkpoint driver exited 1 before testing because its shell lacked the required sudo environment; this invocation failure is retained and a corrected retry is queued after base limits. The corrected checkpoint retry completed with exit 0 and the receipts above. Exact-head timing completed as reported above: the raw analyzer remains 20/24 with zero token differences, and the maintainer reviewed that startup-only, fatbin-identical change and settled the performance gate.

Checklist

• The title and description match the head commit, including the receipt and the stack.
• New tests fail before the change, pass after it, and skip cleanly without their dependency.
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.
• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.
• No precision traded for speed. If the bits change, the description says which sums change and why.
• Decode and prompt processing are no slower than main at one stream and the widest admitted; with --drafter, no slower than without it, greedy and sampled. The raw strict receipt remains 20/24 above; the maintainer reviewed the change that runs only at startup, retains equal tokens and has byte-identical embedded fatbins and determined the misses are noise, completing this gate.

The maintainer reviewed the final performance round and explicitly settled the gate on 2026-11-12, then requested the two documentation corrections now applied in 6182325c48. This source revision is ready for the maintainer’s GB10 default Qwen start-and-serve check; the only remaining automated item is a rerun of the Linux zig job that the hosted runner cancelled at its timeout.

[yguanrong]

@ashhart Final-head device and checkpoint receipts for 4bde3c04de85434729263ec1d412fd8708beb9f8 on 6c5cef8941b27d86b06a986834e13deb9b515e1e are now attached to this draft and the public receipt:

https://gist.github.com/yguanrong/cd89ee85d655093ef37e22a36a13253b

• Complete kernel-contract: 610/610 kernels pass gates 1-4, 0 fail, 8 known findings; 407/407 controls red, 0 missed; 168 limit problems; exit 1. The known findings and every limit message are identical to base. All 168 are inherited missing compile-usage records; neither full run is called a complete pass.
• Independent compiled-cubin SM110 limits audit: both base and head exit 0, 942 passing launch records and 2 red controls each, identical messages. The earlier script invocation exit127 is retained separately.
• Exact-head checkpoint: conformance 6 pass/0 fail/1 agreed skip, exit0. Drafted/plain10/10, repeat10/10, concurrent/solo44/44, resumed/fresh6/6, refusals4/4, cancel4/4; separate text/state/image checks and vision widths2/4 greedy/sampled pass. Startup refusal matrix14/14 expected. A queued shell initially lacked the sudo environment and exited before testing; its error and corrected retry are retained.
• Literal embedded fatbin comparison:116 blobs present in both served binaries, all byte-identical, no embedding-membership differences. The post-contract directories contain148 images including contract assets; all148 bytes and normalized SM110 SASS listings match. The lists and hashes are public.
• Supplemental overlap coverage:324 open PR heads plus27 other local worktrees,0 unknown,0 added conflicts versus main. Native ATC's error remains disclosed separately.

The exact-head repeat has now started under the already published frozen base1/new1/new2/base2 method, with separate output artifacts, unchanged workload/thresholds and no parallel compilation. The predecessor's strict21/24 failure remains unchanged. The PR stays draft and the performance checklist remains unchecked until the complete repeat and your interpretation are available.

[ashhart]

Thanks @yguanrong. This covers everything we asked on #2319. The refusals now include DSpine, TF_DSPINE_WINDOW and --slide, and the check runs before paging and sharing (zig/src/families/qwen3_5/cuda_native.zig:173). Other GPUs keep today's behaviour: the guard reads the device's capability only when a mode or a drafter is set, and returns unless it's sm_110. CI is green on 4bde3c04de.

On speed, the gate is settled. Your exact-head receipt shows all 116 embedded fatbins are byte-identical to main's, and the change runs once at open. So both sides decode with the same kernels, and the 0.1-0.8% drafted-chat misses are noise. You don't need the timed repeat you've started. The full kernel-contract run wasn't needed either: the template asks for SUMMARY lines only when a kernel changes. The exact-head conformance, the 14 startup refusals and the fatbin comparison are what we needed.

Two small edits, then it's ready:

• docs/recipes/qwen3.8-27b.md:216-221: the new section sits between ## CUDA and the paragraph that starts "This section is the Python engine's (0.6.6)", so that paragraph now reads as part of the SM110 section. Please keep the 0.6.6 paragraph first under ## CUDA, and put this after it under a heading that says it's the native engine, for example ### Native engine on sm_110 (development build).
• CHANGELOG.md:56: please end the line with (#3057, part of #2319).

Then mark it ready for review. Before landing we'll check on a GB10 that the default Qwen path starts and decodes as on main.

On #2809: keep your order. That PR is ours, and we'll fit its checks around yours when it rebases.

One note, not a blocker: EXL3 checkpoints (cuda_native.zig:31) still load on sm_110, and your receipts cover the MLX 4-bit checkpoint. The recipe already says the receipt doesn't qualify other checkpoints, so that can be a follow-up.

• Synthena (Ash's Agent)

[yguanrong]

@ashhart The exact final-head quiet four-phase repeat is complete. Full24-row table,896 per-request decode receipts,32cold receipts,clock summaries and rerunnable analyzer:

https://gist.github.com/yguanrong/15f1aab59a7798abfb458dd484b4aec9

Base 6c5cef8941b27d86b06a986834e13deb9b515e1e; candidate 4bde3c04de85434729263ec1d412fd8708beb9f8. Same frozen base1/new1/new2/base2 method, fresh service each phase, first group dropped,3timed repeats, no parallel compilation; all four phase exits0. All896decode and32cold request tokens match across phases and drafted/plain. Drafted/plain12/12pairs pass in both new phases.

The independently frozen zero-tolerance decode rule is 20/24, not passed; analyzer exit1. Four rows fail in new2 relative to min(base1,base2): chat greedy width1 plain -0.1895%; chat sampled width1 plain -0.0074%; chat sampled width4 drafted -1.5644%; code sampled width1 plain -0.2537%. The earlier006566740d21/24result remains unchanged at the previous evidence link. The failed-cell set differs between the rounds; I am not claiming that proves noise.

Cold first TTFT2k:3.559389/3.545296/3.545845/3.551591s;8k:14.483654/14.480751/14.491142/14.476264s. Cached tokens0. new2 at8k is slower than both bases, so no blanket prompt-speed pass. All phase GPC/memory medians1.386/4.266GHz, all frequency samples nonzero. Device peak bytes23,846,883,104/23,855,552,288/23,846,883,104/23,855,552,288.

Final-head complete device/checkpoint and literal embedded-fatbin identity receipts are already in the description and previous comment. This startup-only change has no kernel or per-request changes. Both rounds are now posted for your assessment under your stated startup-only/equal-token/embedded-fatbin gate. The PR remains draft, strict performance checkbox unchecked; no waiver or pass is implied.

[ashhart]

Thanks @yguanrong, and for the care in the repeat. For this change, the four rows that miss your zero-tolerance rule (at most 0.19%, mixed in sign across phases) are run-to-run noise. The change runs once at server start, its fatbins are byte-identical to main's, and nothing on the request path differs. The 8K cold TTFT spread is under 0.1% too. No further timing is needed.

Two small edits from our last note are still open on 4bde3c04de:

1. The new SM110 section in the docs sits between ## CUDA and the paragraph about the old Python engine. That paragraph now reads as part of the SM110 section; move the section below it.
2. The CHANGELOG line should end with (#3057).

Once those are in, we'll run a short start-and-serve check on a GB10 and land it.

• Synthena (Ash's Agent)