[akol1]

What this changes

Adds an experimental dense BF16 Qwen3.5 format with native nonstreamed /v1/completions prompt_logprobs through the existing CUDA lanes. Stored FP32 GDN parameters stay FP32; scoring uses BF16 inputs, FP32 head sums/output and full-vocabulary normalization. Existing formats retain their generation/scoring admission. The review follow-up adds finite, opt-in observation of the production prompt body for diagnosis; no numerical tolerance or kernel arithmetic changes.

• Issue: Refs Proposal: dense BF16 Qwen3.5 CUDA scoring #1136 and its agreed gate. This is a functional text/scoring experiment. Whole-model equivalence, production and performance qualification remain unclaimed.
• Stacked on: main 9200a3064153b8041930f61f5ba92102d4bea692, with CUDA: #1764's BF16 prompt tile for Qwen ModelOpt, Nemotron and Dory, plus two exact Qwen prompt kernels (#1762) #2008 pinned at d19ff3f075eb144870af1af159de1eccafd0b69b as an actual ancestor. Its credited callback fix travels with the parent. The requested rebase resolves cuda_geometry.zig::parse by reading the shared layer pattern before the BF16 return; the FP32-scale guard remains. The refreshed parent contributes its bounded host-test registration placement; the BF16 guards and measured resource assertions remain. The parent is still open; its later f332655d changes only the GDN receipt assertion, already covered here by the stronger measured/missing-receipt controls. Base-history removal remains conditional on the parent landing; no merge is performed here.
• atc overlap --json: run after the final rebase, 2026-11-05 09:21 UTC, against main 9200a306. Coverage is incomplete (pullFilesLimit). The latest scan reports EXL3 Flash Next on CUDA: one rank, offloaded experts, sm_120 gate (1/8) #1441–EXL3 Flash Next: draft lane guard for offloaded experts (8/8) #1446 and EXL3 Flash Next: host-resident K/V on the tier (6/8) #1522; their conflict files reproduce on main or main plus the refreshed parent. The previously detected qwen CUDA: a faster opt-in paged path (TF_QWEN_KV_PAGES=256), --parallel a cap with pages set #2011/qwen3_5 cuda: the 27B as a pipeline stage, logits byte-equal to the whole model #2115 were also checked and add no conflict files beyond those baselines. These are read-only merge checks, not integration of unrelated branches. CUDA: #1764's BF16 prompt tile for Qwen ModelOpt, Nemotron and Dory, plus two exact Qwen prompt kernels (#1762) #2008 is an ancestor. The local warning is a preserved checkout of this same PR. This PR is mergeable against main.

Checks on this head

• Head commit: 47dd8cadc6f09ece60dd72613630c839b44bfe21. Measured native/capture source is 9416f0cf94bddbea49becb4c14047d0c16d08faf. After the later main/parent integration, the complete Qwen CUDA family and all CUDA kernel sources remain byte-identical to that producer. New upstream load/cache/stage changes are covered by the refreshed host checks; no device execution is relabeled. Transformers capture source is b54d8ff7691385fe420399e72272c932a881e29e; the source and binary identities in the assets are not relabeled.
• Lean checker on main and final head, with the same checker: 425 inherited / 425 candidate / zero new findings. zig fmt --check build.zig zig and git diff --check pass. The registry stays within 600 lines by registering the receipt test through the existing bounded test module.
• macOS: zig build test test-kernel-contract test-golden fuzz-cortheon dev-tools --test-timeout 10m -j8 --summary all, 917/917 steps, 7417/7487 executed tests passed, 70 skipped, other results cached, exit 0, on this refreshed integration head.
• Linux: zig build test test-kernel-contract test-golden fuzz-cortheon -j8 --summary all, 599/599 steps, 7396/7455 executed passed, 59 skipped, other results cached, exit 0, on b54d8ff7. Native/capture and plain builds at 9416f0cf pass; plain is 30/30 steps.
• python3 -m unittest discover -s tools/zig -p 'test_qwen35*.py' -v: 20/20 passed with the pinned reference dependencies; 17 passed / 3 skipped without the optional ML dependencies. The complete source fixes a superseded partial-export setup failure, retained in the packet.
• The trace control is red when the callback implementation is removed and 1/1 passes when restored. The scoped BF16 suite passes 19/19. Comparison controls enforce a nonzero CLI exit for planted drift, reject incomplete/nonfinite captures and keep an unscored-final-row difference separate from the primary served-row verdict.
• CI on this exact head: format and No GPU were cancelled by the maintainer while queued, at 09:34/09:35 UTC. GitHub check annotations identify the cancellation actor; no newer head or replacement runs exist. This is a maintainer-controlled cancellation, not a passing check or a demonstrated code failure. An approved CI path requires maintainer direction; no cancellation is overridden.
• Not run: a complete CUDA device-suite rerun after the rebase, other physical chips/backends, multiple ranks and optional grammar/tool paths. The review follow-up changes observation/parser integration, not GPU arithmetic. The earlier complete CUDA execution retains its actual producer below; compilation/resource inspection does not qualify another device.

Receipt

The refreshed host log and exact source/overlap checks are attached in the integration supplement. The original 217-file device/reference packet remains unchanged.

Layer diagnostic and reproduction, compact result, archive hashes, immutable evidence.

• Environment: one RTX PRO 6000 Blackwell Max-Q workstation GPU, sm_120/rank 1, 96 GB class; Linux x86_64, Ryzen 9 9950X3D, 128 GB RAM class; Zig 0.17.0, NVCC 13.0.88, ReleaseFast. Public Qwen/Qwen3.5-27B, revision fc05daec18b0a78c049392ed2e771dde82bdf654; all 11 actual shard hashes/sizes match pinned public LFS after capture. CPU reference: Python 3.12.3, Torch 2.15.0+cu132, Transformers 5.22.1, accelerate 1.15.0, safetensors 0.8.0, NumPy 2.5.4, eager attention, CUDA disabled. Both arms preserve all 96 stored FP32 parameters. Promoted FP32 uses the stored BF16 checkpoint values, not a separate FP32 training checkpoint.
• Paths/platforms: the diagnostic uses the production CUDA prompt forward. Its trace defaults off and has no serving configuration. Native capture has no CPU/RAM cgroup limit; the CPU reference has 16 CPUs/96 GiB. This synchronized capture is not a four-CPU/32-GiB latency benchmark. Original running deployments and active-service health were restored. Other physical chips/backends and multiple ranks are unqualified; offered GB10 cells remain with the maintainer.
• Exactness: traced versus fresh-cache untraced final BF16 logits match. Transformers BF16 final hidden rows equal its public library entry point; promoted-FP32 layers 0, 3 and 63 equal independently cloned FP32 layers. The earlier 32/32 API, drafted/plain, concurrent/solo, retained/fresh and cancellation checks are retained at their actual serving producer b168b36d, not claimed as new executions on this head.
• Decode speed before/after: no new performance run in this diagnostic follow-up. Historical measurements at b168b36d remain fully linked: native/SGLang C4 aggregate decode is about 9/109.5 tokens/s. Performance remains unqualified.
• Prompt speed before/after: no new timing claim from synchronized layer capture. Historical cold 64k TTFT is 1601.814 s native / 24.608 s SGLang. The original serving quota was four CPUs/32 GiB, distinct from this diagnostic. Complete warm/cold, sampled, concurrency, memory and cache constraints remain in the serving receipt.
• Licence of new weights/drafter: n/a; no new checkpoint or drafter is supplied. References use the pinned public checkpoint.

Requested layer comparison

The exact 32-token public fixture has 31 hidden rows used by served prompt scores. The last input row is retained in a separate all-32-row view. At each boundary the rule is max |native − FP32| <= max |Transformers BF16 − FP32|, with no added allowance.

The first excess is zero-based layer 0, the first linear-attention block. Embeddings match exactly. Layer 0's worst coordinate is input row 21, hidden column 3994: native 15.5625, FP32 15.63897705078125, BF16 reference 15.625.

Boundary	Worst native−FP32	Worst BF16−FP32	Within own maximum
Embedding	0	0	passed
Layer 0 residual	0.07647705078125	0.056326866149902344	failed
Layer 1 residual	0.09848594665527344	0.09848594665527344	passed
Layer 2 residual	0.19491863250732422	0.1948375701904297	failed
Layer 3 residual	0.199066162109375	0.188201904296875	failed
Layer 63 residual	4.7076416015625	7.054046630859375	passed
Final normalization	0.31170082092285156	0.4594573974609375	passed

The rule fails at 18/64 residual layers. Both row scopes find the same first layer. All 66 boundaries, maxima/medians/RMS, worst coordinates and 198 unchanged tensor arrays are external. Later boundaries returning inside their maximum do not waive the separately failed served-logprob rule. This locates observed drift in the first block; it does not establish an isolated accumulation bug or an unavoidable format floor.

The earlier direct HTTP comparison remains failed at all four prompt lengths, over all 4511 scored positions, including the 4097-ID seam, in both reference arms. At 32 IDs, worst/median native−FP32 logprob distance is 0.0415096283 / 0.004390716553, against own BF16−FP32 worst 0.02388763428. Same-weight FP64 checks still cover the head on native rows, not the whole body. No broader hidden-state bound replaces these failures.

The 217-file supplement includes full build/capture/audit logs, exact IDs, numerical failures and actual source/binary identities. Archive/member checksums and normalized header identities are recorded; all three public downloads were rehashed successfully. Source, outgoing history, JSON metadata and the complete packet were reviewed for private identifiers/credentials, with zero findings; approved public contributor aliases/noreply credit remain.

Kernel contract and historical producer

The BF16 copy/dense/FP32 scoring and FP32-scale RMS operators remain registered with their planted controls; no numerical tolerance or known exemption is added. The CPU five-SM audit at native producer 9416f0cf passes, with no stale records or applicable launch failures, and every planted launch control red. This inspects compiled resources; it is not five physical-device runs.

The complete historical sm_120 command at b168b36d0cc3503ae69e8bffcad494574019caf5, unchanged in the existing asset, is:

SUMMARY 559 kernels: 559 pass gates 1-4, 0 fail, 6 known findings; 367 controls: 367 red as planted, 0 missed; 0 limit problems; gates 4023784 ms, limits 760889 ms
exit: 0

The same six declared NaN findings remain; no new known entry is added. A complete device-suite rerun on the rebased head is not run, and this historical SUMMARY is not relabeled. Other CUDA chips, Metal/HIP and multiple ranks are unqualified. GB10 execution remains with the maintainer.

Checklist

• Title, description, receipt and stack match the published head and distinguish actual execution producers.
• Negative controls fail as planted, restored implementations pass, and optional ML dependencies skip cleanly.
• Source modules are bounded, comments/docstrings remain one line, and there are zero new lean findings.
• Source, outgoing commits and evidence contain no private identifiers or credentials; approved public attribution remains, with no AI trailers.
• Serving stays on native lanes; the finite diagnostic adds no serial serving path or draft-disable setting.
• No precision or tolerance is traded for speed; independent numerical failures are disclosed.
• New-head full token/lifecycle and performance qualification: historical results are retained; no new claim from the diagnostic.
• Final-head hosted pass: runs were maintainer-cancelled; awaiting maintainer direction.
• Maintainer landing decision, parent cleanup after landing and offered GB10 execution.

[akol1]

Qualification update on 91197649ca24d2234f9cde2ff8e6e68b7a09b730: the public API/FP64 reports and exact fixtures are now committed under docs/receipts/qwen35-bf16-scoring/. The complete strict CUDA contract is still running; this remains a draft.

Hosted format passed. The No GPU run was cancelled before completion, as was the previous head's run. There has been no subsequent push to supersede this head. The macOS zig build dev-tools step completed successfully, but that does not establish a complete hosted pass. The unprivileged Linux host suite passed locally, with the counts retained in the PR description.

Could a maintainer confirm whether the repeated cancellation is intentional and how to obtain the remaining hosted checks? I have not changed workflow policy or repeatedly retried cancelled runs. The outstanding checks remain explicitly unchecked.

[akol1]

The complete strict CUDA run is now retained in the receipt, including its full text output and binary identity:

SUMMARY 550 kernels: 550 pass gates 1-4, 0 fail, 6 known findings; 365 controls: 365 red as planted, 0 missed; 342 limit problems; gates 4013421 ms, limits 726300 ms
exit: 1

The numerical gates and every planted control passed their intended checks; six existing NaN findings reproduce. All four new function records pass live sm_120 checks, and all 20 new function/SM records are current in the five-SM asset audit.

The remaining failure is 188 registry problems plus 154 live record/device mismatches. The separate parent/candidate five-SM audits both exit 1 with 637 problems and identical diagnostic multisets; the comparison retains those diagnostics. The parent's full/live-only run was not run, so this is not presented as a matching parent live SUMMARY or a clean qualification.

Could the maintainer advise the appropriate catalog-resolution scope for these existing records? The PR stays draft. Native serving measurements have resumed, with performance, lifecycle, cold-prefill and existing-format receipts still pending.

[ashhart]

Thank you, @akol1. This is careful work: the dense BF16 path keeps every existing checkpoint on main's code, the
prompt scores have their own FP32 head with planted controls, and your callback-lifetime fix in commit 8 found a real
bug in the base we stacked under you. We'll take that fix into #1762 with credit. Three requests before this can land.
First, please drop the four base commits once #1762 is in, and keep only the summary receipts here, moving the
per-request dumps and the full contract log to a linked asset with their checksums. Second, the FP64 check covers the
score head on the engine's own rows; could you add a comparison of the served prompt logprobs against a transformers
forward of the pinned checkpoint, bounded by transformers' own BF16 distance from FP32? Third, a small wording change
in the recipe's line about the GPU lock, and a fuller message on the main commit. We'll run the GB10 cells on our side.

[ashhart]

Your callback fix (d8f5efbc93) is now on #2008 under your name (789846adb), so it lands with the base. You can drop it from this branch when you rebase.

[ashhart]

Thanks @akol1. Finding it at layer 0, with embeddings exact and the reference's own layers cloned and checked, is exactly what we needed. One more split would settle fix against floor: inside layer 0, could you compare the conv, the gates, the recurrence and the output norm with FP32 on the same 32-token fixture? Main has also moved under docs/recipes/cuda.md since your rebase, so it needs one more. We'll decide on the experimental landing once that's in.

[ashhart]

Our GB10 run on this head (47dd8cadc), against its merge base with main, Qwen3.8-27B MLX 4-bit, the path this PR must leave alone: no token differences in 20 prompt rows. Prefill moves -0.4% to 0.0%, decode after a prompt -0.4% to +0.3%, and the stream sweeps 0.0% to +0.3%. Level, so the shared loading changes don't touch the 4-bit path. The layer-0 split I asked for above is still the open question for the BF16 path itself.

[ashhart]

Our GB10 run with Nemotron 3.5 Lightning MLX 4-bit on this head, against its merge base, read REVIEW on one cell. Two-stream greedy code went from 318.1 to 248.9 tok/s (-21.8%; base runs 305.3 and 330.8, new 249.3 and 248.4). Four-stream greedy chat moved the other way, +13.8%, and everything else, prefill, decode after a prompt and the other sweeps, is within about 1%, with no token differences in 20 rows.

Nemotron's greedy multi-stream cells have two speed modes on GB10s that flip between runs, and a single pair can land on different modes. We saw the same thing on main this afternoon and settled it with repeats. So this is probably that, but we don't take it on two runs. A six-repeat rerun of the two- and four-stream sweeps is queued, and I'll post what it shows. Nothing needed from you for this one.

[ashhart]

The six-repeat rerun on Nemotron 3.5 Lightning MLX 4-bit is in, and it's level, so the earlier -21.8% was the multi-stream mode flip, not this PR.

• 2 streams: chat greedy +0.4%, chat sampled -0.4%, code greedy +1.9%, code sampled +2.1%.
• 4 streams: chat greedy -0.6%, chat sampled -0.2%, code greedy +0.5%, code sampled -0.3%.
• No token differences.

Since then we've found why those cells flip: identical greedy streams that start together run in lockstep, and that alone can move Nemotron about 50% between repeats. Our sweeps now stagger their starts by 150 ms. So both shared-path runs are level (the 27B and Nemotron), and what's left is the layer-0 split for the BF16 path itself.

[ashhart]

Thanks @akol1 for keeping the branch current with main. Two things need you before we can go further, and one
needs us.

1. The layer-0 split from our last round is still open. Inside layer 0, could you compare the conv, the gates,
   the recurrence and the output norm with FP32 on the same 32-token fixture? That tells us whether the drift is one
   accumulation we can fix or the format's floor, and we're holding the experimental-landing decision until it's in.
2. After the merges, this head no longer serves the checkpoint it's for. Main's qwen3_5 cuda: a plain BF16 checkpoint loads as a decision backbone, chat refused by name (#2324 step 1) #2383 made a plain BF16 Qwen3.5 on
   CUDA a decision backbone only. open now runs decide.check first
   (zig/src/families/qwen3_5/cuda_native.zig:141), which refuses any BF16 Qwen3.5 without joint_head_config.json
   before your admission at :151. Your test at zig/src/families/qwen3_5/cuda_bf16_format_test.zig:48-57 now
   expects that refusal, so Qwen/Qwen3.5-27B can't open for scoring here. Please don't work around it yet. Our rule for it: if the layer-0 split shows the drift is one accumulation we can
   fix, a scoring-only BF16 path beside qwen3_5 cuda: a plain BF16 checkpoint loads as a decision backbone, chat refused by name (#2324 step 1) #2383 is fine, serving prompt logprobs only, with generation refused by name and
   the FP32 comparison as its receipt. If it's the format's floor, the path stays out.

On our side, #2008 (the base) has to be rebased and land first. Main has also moved under four of your files in code
(zig/src/cuda/moe.zig, qwen3_5/cuda_native.zig, qwen3_5/cuda_prefill.zig, zig/build/cuda_family_host.zig).
Please hold the next rebase until #2008 is in, so you only do it once. Main is frozen for a release right now, so none
of this moves until that's out.

• Synthena (Ash's Agent)

[ashhart]

Thanks @akol1. We went through where this stands so it doesn't sit on our side. One question only you can answer, and one step that's ours.

From you: the layer-0 split. Your layer diagnostic puts the first excess in zero-based layer 0, the first linear-attention block. There, the worst native-to-FP32 distance is 0.0765 against the BF16 arm's own 0.0563. The next question is still inside layer 0. On the same 32-token fixture, how do the conv, the gates, the recurrence and the output norm each compare with FP32? Your trace already captures the production CUDA prompt forward at the residual boundaries. This needs capture points inside that one block, run on the same GPU and checkpoint as your other receipts. That's why we're asking you rather than rerunning it on different hardware.

What the split decides. Main's #2383 makes a plain BF16 Qwen3.5 on CUDA a decision backbone only. open runs decide.check first (zig/src/families/qwen3_5/cuda_native.zig:141 on this head), so Qwen/Qwen3.5-27B is refused before your BF16 admission at :151. Your own test at zig/src/families/qwen3_5/cuda_bf16_format_test.zig:48-57 expects that refusal. If the split shows one accumulation we can fix, a scoring-only BF16 path beside #2383 is fine: prompt_logprobs only, generation refused by name, with the FP32 comparison as its receipt. If it's the format's floor, the path stays out. Please don't work around the refusal before then.

Ours: the base. #2008 has to land before this can. On today's main this head conflicts in seven files: zig/src/cuda/moe.zig, qwen3_5/cuda_native.zig, qwen3_5/cuda_prefill.zig, qwen3_5/cuda_weights.zig, zig/build/cuda_family_host.zig, contract/registry_joins.zig and tests/cuda/contract_limits.zig. Please keep holding the rebase until #2008 is in, so it's done once, after the four base commits drop out.

• Synthena (Ash's Agent)