[jschmied]

What this changes

Ling-3.1-flash (#850) on CUDA at TP=1, built on #804's GLM kernels as agreed there and on #850. #804 is in main since 1.0.12, so this PR is stacked on nothing: the branch is rebased onto main 46a08c18 (TensorFold 1.0.12), and every commit on it is this PR's. The family is registered with the native CUDA server (model_type bailing_hybrid) and serves through the lane core at TP=1, with the prompt cache and MTP drafts. The whole model does not fit one Spark, so serving is checked on synthetic checkpoints and a 4-layer slice of the real one. docs/recipes/ling-3.1.md lists what exists, what is open, and how to test without the 572 GB checkpoint.

• glm: GLM-5.3-Flash (NVIDIA's NVFP4 checkpoint) as a CUDA family at TP over hosts #804's kernels, extended, with GLM's bytes unchanged:
  • group-limited top-k (8 groups, keep 4) and a per-layer shared SwiGLU limit (f9fc1a78); a division-free branch for groups that are whole lane columns, which GLM's single group never enters (e78d23bb);
  • GLM's swiglu keeps its limit as given, 0 included, as at 6380c8da; only Ling's swigluWith maps 0 to unclamped (32055e9e, 991383f3);
  • MLA with Ling's 64 roped key columns, 128-wide heads and a head-wise gate (7d184cb7);
  • KDA checked as is on Ling's layer 0 (ef8f12f3). The opt-in 16-row chunked prompt path that commit added is removed again in 03171bf5: nothing served called it, and GLM's cuda_kda.zig is 6380c8da's file again;
  • Ling's SwiGLU clamp order, min(silu(g), L) * clip(u, -L, L), as new kernel instances (bfb44ddc);
  • tf_pack_fp8_experts in modelopt_pack.cu, an added kernel for Ling's loader; the existing packers are unchanged (bbe360f9).
• zig/src/families/ling/, format-agnostic:
  • config and weight map (a6559571);
  • a loader that takes each tensor's format from the checkpoint, bf16, block FP8 or ModelOpt NVFP4, through qlinear and experts. EXL3 is refused and points to EXL3 checkpoints on the native engine (tracker) #832 (d46e3723);
  • ling-synth for small synthetic checkpoints in each format;
  • the TP=1 layer graph with prompt rows, decode windows and commit(keep) (6f583baf);
  • serving (a3f14d30): Engine with a sequence per slot, prompt chunks cut at any row, verify windows of up to 16 rows, shared windows of up to 8 streams, greedy and sampled draws; Lanes, GLM's cuda_lanes.zig without the ranks; native.zig with the server hooks and the prompt cache's snap_* hooks. A kept state is the sequence's arena copied whole, and a mark may fall on any row;
  • the MTP layer (model.layers.72) as the drafter: eh_proj(enorm(embed(t+1)) | hnorm(h_t)), MLA + MoE on its own cache, chains of up to 3 (577436b0). The server loads it unless --no-drafts;
  • routed experts staged in batches through pinned memory and packed on the GPU (bbe360f9);
  • a speed pass on the copies and small FP8 expert runs, same bytes (99652bab, b3faea69, 5b006370), and timing tests (7bd7a40a, 412f66fc, 0ffce4f7, bd7c23e3);
  • the recipe (faf7af1c).
• Review fixes (one commit each, each with a regression test that fails with the fix reverted):
  • config refuses every geometry the kernels don't take, at load, with a named error: KdaGeometry, MlaGeometry, Width, TiedHead (66579cba);
  • Store.alloc owns a buffer only once it is in both lists; fake_driver.zig fails every driver call in turn (674163fa);
  • rows past a sequence's capacity are refused before any layer runs (82b3817b);
  • routing groups of one and a single pick are refused at load, as HF's group_limited_topk can't run them (92afed61);
  • Engine.init frees sequence 0 and the slot list when bind(0) or the last sync fails (0ba2adfb);
  • a new stream's slot is chosen before its map entry exists (0ba510ad);
  • every tensor of the full checkpoint checked against the loader's names, dtypes and shapes, from shard headers alone, through one shape table, names.dims (5649c63b);
  • the oracles check their HF source against inclusionAI/Ling-3.1-flash@a6943f4b by sha256 (e4c9bcc5);
  • a segment the scratch cannot hold is refused before any layer runs, also on the MTP layer (5e931c3a);
  • staging and loaded weights are freed only after the stream drains; a failed drain leaks them on purpose (1395e267).

Receipt

• Environment: TensorFold 46a08c18 (1.0.12) plus this branch (e99e6505; its last commit only updates the recipe, so the checks below ran at 1395e267), one GB10 (sm121), CUDA 13.0, Zig 0.17.0. Checkpoint: inclusionAI/Ling-3.1-flash, block FP8, pinned at a6943f4b; its real layers read one at a time.

• Serving (ling-lanes, under upstream's round loop, prompts in 37-row chunks), bf16, FP8 and NVFP4, each with and without --mtp: 100 and 109 checks pass in each format. Tokens in every check, state bytes in the engine-level ones:

  • chunked == whole, at chunks of 512, 37 and 16 rows;
  • drafted == plain, greedy and sampled, windows of 3 to 16 rows cut anywhere; a 9-row window kept to 4 then a 5-row window equals 9 single rows;
  • resumed == fresh, at marks 37, 101, 150, 263 and 349 (not multiples of 16);
  • concurrent == solo, nine cases of three streams, and shared windows of 2 streams equal lone twins;
  • MTP-drafted == plain, greedy, sampled, resumed and concurrent; chained drafts equal ones computed by hand;
  • the reply equals the serial forward's argmax at all 160 of 160 rows;
  • a refused request (past the sequence's capacity, or past the scratch's) leaves the arenas byte for byte as they were, and the slot then serves as a fresh one does.

  Served logits against ling_forward.py's float64 on the same picks: 2.09e-2 to 2.24e-2 rel L2, as far as the HF model's own bf16 rounding points are (2.10e-2 to 2.25e-2). On the real FP8 release, --layers 0,1,2,3 (real embedding, layers 0-3, final norm and head) passes the same exactness checks; that run is not in the re-run at this head.

• Graph exactness, bit for bit, in bf16, FP8 and NVFP4 (ling-forward-synth): a prompt cut at 48 equals the uncut prompt; an 8-row verify window equals 8 decode rows; keeping 3 of 8 rows and then running 5 more leaves every KDA state, conv window and MLA cache row equal to the serial run; two sequences batched equal each run alone. On real layers, a prompt cut at 300 equals the uncut prompt and a 16-row window equals 16 decode rows.

• Against the reference:

  • real FP8 layers 0, 4, 7, 65 and 71 against the HF bf16 model's captured chain: rel L2 3.1e-3 to 3.8e-3, excluding the 0 to 21 of 512 rows whose top-8 picks flip on a near tie. Layer 4 at this head: 3.126e-3, its output bytes equal to a78a7a49's;
  • MLA layer 7 alone: 4.4e-3 from HF's bf16 output (HF's bf16 output is 3.6e-3 from float64);
  • KDA layer 0: 3.591e-3 from fla, within the float64 bound;
  • the MTP layer (ling-mtp-real) against HF's BailingMoeV3MTPLayer: 7.28e-3 on the 488 of 511 rows whose top-8 agrees;
  • routing on layer 4: the same experts in the same order on all 512 rows;
  • synthetic forwards: within the float64 bound, as far from float64 as the bf16 model's own rounding is.

• Full checkpoint (ling-ckpt-check, shard headers only): 54 shards, 214,025 tensors (3,100 of the MTP layer), 573,753,281,152 bytes, no name, dtype or shape mismatch, nothing missing or extra. The shard data is not hashed.

• Fault paths (ling-init-faults, on fake_driver.zig): every driver call and host allocation of the open and a kept state fails in turn with transfers pending, then every call from N on. Over bf16, FP8 and NVFP4, with and without MTP: 15,180 failure points (1,804 failing with a transfer pending), none leaves anything held, leaked, freed twice or freed before a drain.

• Loading one MoE layer's 512 FP8 experts (7.5 GiB): 6.1 s from local disk with the GPU packer, 14.3 s with the host packers before; host peak 1.3 GiB. Every packed byte equals the host packers' (ling-pack).

• Decode and prompt speed: not measured. The whole model does not run on one Spark. The speed pass is measured per op only: prompt KDA and MLP column copies -50 % (99652bab); FP8 expert time -7 to -27 % for prompt chunks up to 768 rows (5b006370).

• GLM unchanged:

  • glm-route-old (select, and swiglu at limits 10 and 0) and glm-mla-same (13 launches, 14.3 MB of outputs) run glm: GLM-5.3-Flash (NVIDIA's NVFP4 checkpoint) as a CUDA family at TP over hosts #804's 6380c8da fatbins next to the new ones, byte for byte; ling-route-old checks the division-free branch against the older select's bytes;
  • every kernel that existed before keeps its SASS and resource usage (cuobjdump, all fatbins);
  • ling-clamp-old checks the kept expert instances' bytes;
  • @Bizuayeu ran GLM's checks on the GLM-5.3 NVFP4 checkpoint on three GB10 hosts at c48be499, ac58df63 and e29e0c20 (now 03171bf5): one host, TP=2 and TP=3, and the server at TP=2 against 6380c8da's replies. All pass. GLM's and the shared code have not changed since 03171bf5.

• Tests run at 1395e267: glm-ref, glm-route-old, ling-route-old, glm-mla-same, glm-kda-ling, ling-slots, ling-ckpt-check (three synthetic checkpoints and the full release), ling-init-faults and ling-lanes (three formats, with and without --mtp; ids and picks byte-equal to a78a7a49), ling-weights (three formats, worst 0.433 of the bound), ling-pack layer 4, ling-layer-real layer 4; zig build test (1390/1398, 8 skipped), test-ling 30/31 (1 skipped: it needs TF_LING_INDEX), test-cuda-host 237/237, zig fmt --check, lean_check 517, as on main. Deliberate mutations, all caught: routing 6, MLA 6, config 8, loader 5, graph 6, clamp 5, serving 5, MTP 5, and each review fix reverted alone.

• Not run, and why:

  • a full 72-layer Ling forward, the real model served end to end, and MTP acceptance on real text (572 GB in FP8 needs five Sparks and TP);
  • TP=2 for Ling (one Spark);
  • Metal.

What is still open

• TP>1. The loader and engine are TP=1; cuts would follow glm/cuda_shard.zig and GLM-5.3-Flash on two CUDA ranks, in families/glm #855.
• EXL3, through EXL3 checkpoints on the native engine (tracker) #832. It is refused at load, naming EXL3 checkpoints on the native engine (tracker) #832.
• The recorder. lanes/recorder: --record writes each stream's committed rows and tapped features for drafter training; Nemotron CUDA gives its final norm's rows #848 is in main, but Ling gives no tapped rows yet, so --record is refused for bailing_hybrid. The rows are the final-norm rows Engine.verify already keeps.
• MTP acceptance on real text, once a whole model runs.
• bf16 routed experts run every expert on every row (no grouped bf16 kernel): correct, for tests, not for speed.
• Fused same-format projections (q|k|v, gate|up) without column copies.
• A real NVFP4 checkpoint. Only synthetic NVFP4 checkpoints have been loaded.

Checklist

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (On synthetic checkpoints in three formats and the real model's first 4 layers; not on the full model.)
• No precision traded for speed. If the bits change, the description says which sums change and why.
• Prompt processing is no slower than the last release. (Not measured for Ling; GLM's kernels are unchanged.)
• The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not.
• New tests fail before the change, pass after it, and skip cleanly without their dependency.
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths. No AI attribution lines.

Open question for @ashhart (from #850 and your reply of 2026-10-15): group-limited top-k is built into #804's tf_glm_topk, since Ling's MoE runs through #804's routing. It ranks sigmoid(logit)+bias as HF does for Ling; #798's moe_glue.cu ranks logit+bias, so the two can't simply replace each other. Where should the shared routing live?

[Bizuayeu]

@jschmied, we ran GLM's checks on this branch at c48be49 on our three GB10 hosts, with the GLM-5.3 NVFP4 checkpoint, and they all pass:

• One host: the build, zig fmt --check, test-cuda-host and zig build test, then glm-ref, the thin and synthetic lanes checks (cover and concurrent) and glm-shared-check.
• TP=3 and TP=2: the concurrent check (every stream equals its solo run) and the shared check (108 rows of 5 streams equal their lone twins' logits, tokens and drafts byte for byte), and at TP=2 the cover check.
• The server at TP=2 gives the same replies as glm: GLM-5.3-Flash (NVIDIA's NVFP4 checkpoint) as a CUDA family at TP over hosts #804's head (6380c8d) on the requests we compared, by hash, rounds and accepted drafts.

So the Ling commits leave GLM's checks and replies as they were. We have no Ling weights here, so this covers GLM only.

[jschmied]

Thank you, @Bizuayeu, that's the check we couldn't run.

We'll push three more pieces to this branch shortly: serving at TP=1 (engine, lane backend, kept prompt states), the MTP drafter (model.layers.72) and a GPU packer for the experts. None of them touches GLM's code: git diff c48be499 -- zig/src/families/glm zig/kernels zig/src/cuda only adds a packer kernel and its host call to modelopt_pack.cu / cuda/modelopt.zig, which GLM doesn't use. A speed pass on the Ling kernels may follow; if it touches any shared GLM kernel, we'll say which and how we checked GLM is unchanged.

A rerun once it's up would be welcome, if you have the time.

[jschmied]

Pushed ac58df63 (serving, MTP, expert packer, speed pass). The one shared GLM file touched is glm_glue_moe.cuh, a fast path inside the group-limited select that GLM (one group) never enters; glm-route-old, glm-mla-same and glm-ref are byte-equal / pass on this head.

[ashhart]

Thank you, @jschmied, for the GLM TP=2/3 checks and for identifying the upcoming serving, MTP and packer changes separately.

The routing comparison needs care: zig/kernels/cuda/moe_glue.cu:46 in #798 ranks logit+bias, while #874 ranks sigmoid(logit)+bias. Replacing one with the other changes picks and weights. We'll come back on where the shared routing should live.

At c48be49, the zero-limit GLM behavior and the 16-row helper's arbitrary-resume behavior remain open. Please send the pushed serving/MTP head and its delta from c48be49 so we can recheck those paths and the unchanged-shared-kernel claim.

Posted by Ash's AI Assistant Synthena

[jschmied]

Thank you. Both open items are closed at e29e0c20:

• Zero limit (6676c13c): at 6380c8da GLM's glue swiglu, like Metal's glm_rows_glu, clamps at the limit it's given, 0 included; 51c01573 had mapped 0 to unclamped for every caller. GLM's swiglu passes its limit as given again, and only Ling's swigluWith maps 0 to unclamped. glm-route-old now also compares GLM's swiglu at limit 0 with 6380c8da's kernels, byte for byte (it compared only 10 before; putting the old mapping back fails it).
• The 16-row chunk helper (e29e0c20): nothing served called it, so it's out: glm_kda_chunk.cu, its tests and the chunked path in glm-kda-ling. GLM and Ling run the row path, which resumes at any row, and zig/src/families/glm/cuda_kda.zig is 6380c8da's file again.

Delta from c48be499 in GLM's and shared code (git diff --stat c48be499 e29e0c20 -- zig/src/families/glm zig/kernels zig/src/cuda zig/build zig/src/native):

• glm_glue_moe.cuh: a division-free branch inside the group-limited select, taken only when groups are whole multiples of 32 experts (Ling's 64); GLM's single group never enters it.
• glm/cuda_glue.zig: the swiglu split above.
• modelopt_pack.cu / cuda/modelopt.zig: an added kernel, tf_pack_fp8_experts, for Ling's loader; the existing packers' SASS is unchanged.
• ling_glue.cu, probe.cu (a timing spin kernel for tests), build/cuda.zig (Ling's module now also imports lanes), native/cuda.zig (Ling in the registry).
• Removed: glm_kda_chunk.cu and its build and kernel-list entries.

Checked at e29e0c20: glm-ref, glm-route-old (select, and swiglu at 10 and 0), glm-mla-same (13 MLA launches byte-equal to 6380c8da), glm-kda-ling, zig build test, zig fmt --check, lean count unchanged. Everything else (zig/src/families/ling, its tests, oracle/, the recipe) is Ling-only: serving at TP=1, the MTP drafter, the expert packer and a speed pass.

[ashhart]

Thank you, @jschmied, for restoring GLM's zero limit behavior and removing the 16 row WY path.

We checked e29e0c2. glm/cuda_glue.zig:132 now preserves GLM's literal limit, and its KDA host file matches the earlier implementation. The KDA blocker remains at ling/config.zig:388: unsupported head widths and convolution tap counts still reach fixed kernels. Other open cases are Store.alloc ownership on allocation failure, capacity refusal after KDA state commits, singleton routing groups, engine initialization cleanup, and the undefined slot read at ling/lanes.zig:143. Full checkpoint checks and the exact HF source comparison also remain.

Please send a focused revision covering these cases, with geometry rejection and allocation failure regressions, so we can recheck it.

Posted by Ash's AI Assistant Synthena

[Bizuayeu]

@jschmied, we ran GLM's checks again at e29e0c2 on our three GB10 hosts, with the GLM-5.3 NVFP4 checkpoint, and they all pass:

• One host: the build, zig fmt --check, test-cuda-host and zig build test, then glm-ref, the thin and synthetic lanes checks (cover and concurrent) and glm-shared-check.
• TP=3 and TP=2: the concurrent check (every stream equals its solo run) and the shared check (108 rows of 5 streams equal their lone twins' logits, tokens and drafts byte for byte), and at TP=2 the cover check.
• The server at TP=2 gives the same replies as glm: GLM-5.3-Flash (NVIDIA's NVFP4 checkpoint) as a CUDA family at TP over hosts #804's head (6380c8d) on the requests we compared, by hash, rounds and accepted drafts.

We had also run ac58df6 the same way, with the same results. As before, this covers GLM only.

[jschmied]

@ashhart, thank you. @Bizuayeu, thanks for rerunning the GLM checks at e29e0c2.

A focused revision on top of e29e0c20, one commit per case (head 7c446062). Each fix has a regression test, and each test fails with its fix reverted.

1. Geometry rejected at config load (19cf6f4f)
config.parse now refuses with a named error, before any allocation or launch, every shape the kernels don't take:

• KdaGeometry: head_dim other than 128 and short_conv_kernel_size other than 4 (glm_kda.cu's DK and TAPS), and key/value heads other than the query heads.
• MlaGeometry: kv_lora_rank other than 512, qk_nope_head_dim/v_head_dim other than 128, qk_rope_head_dim other than 64, NeoX rope, and a q_lora_rank without an rmsnorm instance.
• Width: hidden_size other than 512/1536/4096, and MLP and expert widths outside whole 128-column blocks.
• TiedHead: tied embeddings.

Regression (test-ling): 26 values refused one at a time on the tiny config, each checked for its error and reason. The tiny config with the kernels' own values and the real config are accepted. A test also holds the fixed widths to glm.kda.dk, glm.latent.rope_width and fp8.block.

2. Store.alloc ownership on failure (d11ad3f9)
Before, a failed name copy or named.append freed a buffer already in buffers, so deinit freed it again, and the copied name leaked. alloc now reserves both lists and copies the name first, then allocates and appends without any step that can fail: on error the store holds nothing new.

Regression: fake_driver.zig is a host stand-in for the driver (device memory in host memory, kernels as no-ops, call N failing on request). store_test fails every host allocation in turn (checkAllAllocationFailures) and every device call in turn. No leak, no buffer left held, no double free. The old alloc fails both tests.

3. Capacity refusal before KDA commits (c578f33f)
The capacity check sat in the MLA block, after the KDA layers before it had run. A prompt chunk past the capacity had already committed their states and conv windows, and a window had already written its saves. Forward.rowsOf now refuses any segment without room before any layer runs. The MTP layer is a single MLA block that already refused before its cache write.

Regression (ling-lanes, engine checks, at 1012 of 1024): a 30-row prompt chunk, a 16-row window and, with --mtp, MTP rows past the end each return CacheFull. The sequence's arenas stay byte for byte as they were. The next window then equals a twin's in tokens, logits and states, and a new request on that slot equals one on a fresh slot. With the old check the arenas differ and the next commit fails with KdaLayersOutOfStep.

4. Singleton routing groups (d6f5ad88)
HF's group_limited_topk scores a group by its top two (topk(2) raises on a group of one) and doesn't renormalize a single pick. Groups of one and num_experts_per_tok 1 are now refused at load. n_group 1, topk_group == n_group and groups of two stay accepted.

Regression: the refusals and the four accepted edge cases in test-ling. glm-ref adds five glue.select shapes against the float64 rule: one group, 8 of 8 kept, groups of two (64 in 32 and 16 in 8), and the tiny config's 16 in 4. Picks are equal and weights are within 2^-16.

5. Engine init cleanup (d523c01d)
The one gap was sequence 0 and the slot list, which leaked when bind(0) or the closing synchronize failed. Both are freed now.

Regression: ling-init-faults DIR [--mtp] runs native.open (engine, sequence 0, lane backend) on the fake driver. It opens once cleanly, then fails each driver call and each host allocation of the open in turn, and requires each failed open to leave nothing held, leaked or freed twice. Over bf16, FP8 and NVFP4, with and without MTP, that is 8,591 failure points, all clean. Dropping this fix, or any of five existing errdefers (engine, graph, state, weights, mtp), fails it.

6. Undefined slot read in lanes.zig (67fb3c22)
getOrPut was followed by freeSlot(), which walked the new entry before its slot was set. In a release build a reused bucket keeps its old value, so a free slot could count as taken. Lanes.claim now picks the slot first and then puts the entry.

Regression: slotCheck claims slots over map buckets that still hold slot 0 from cleared entries. It runs as a host test and as ling-slots in the release build, where the old order fails with SlotTaken. In a debug build the hash map writes 0xAA into a new value, which hides the bug there.

7. Full checkpoint and HF source

• Full checkpoint (def8beba): the loader's shapes now come from one table, names.dims. ling-ckpt-check DIR [INDEX] reads only the shard headers and checks every tensor against the weight map at the dtype and shape the loader reads, including FP8 scale_inv blocks, NVFP4 codes and scales, bf16 exceptions, small tensors and convs. It also requires nothing missing, nothing extra, and the bytes to add up to the index's total_size. On the FP8 release (54 shards): 214,025 tensors, 3,100 of them in the MTP layer, 573,753,281,152 bytes, no mismatch.
• HF source (7c446062): oracle/ling_hf.py pins inclusionAI/Ling-3.1-flash at a6943f4bf04f8041ad983d2b58dd43b5ba86ae4a, with the sha256 of modeling_bailing_moe_v3.py, configuration_bailing_moe_v3.py, config.json and the index. config.json is as of that revision; the other three are unchanged since the upload. ling_mtp.py, ling_layer.py (now given the modeling folder the chain was captured with) and ling_weights.py stop on any other copy. test-ling holds fixtures/config.json to the pin. Our local copies match HF at that revision byte for byte.
• Still not possible on one 128 GB Spark: a 72-layer forward of the 572 GB FP8 checkpoint, so no end-to-end logits against HF, no serving of the real model and no real MTP acceptance. HF's own end-to-end run would need about 1.1 TB in bf16, so the HF reference stays the layer-by-layer captured chain. The real-weight checks remain single layers (0, 4, 7, 65, 71 and the MTP layer); the remaining layers could be checked one at a time from that chain at about 80 s per layer, which we haven't run. We also haven't hashed the shard data in this revision; the check reads headers only.

Re-run at 7c446062:

• Host and lint: zig build test, test-ling 30/31 (1 skipped), zig fmt --check, lean_check 527 (unchanged).
• GLM unchanged: glm-ref; glm-route-old, glm-mla-same (13 launches, 14.3 MB equal to 6380c8da) and ling-route-old.
• Ling: glm-kda-ling; ling-lanes bf16/FP8/NVFP4 with and without --mtp (95 and 104 checks); ling-weights for the three formats; ling-pack layer 4.
• Real layer: ling-layer-real L4 at rel L2 3.126e-3, as before. Its output bytes equal e29e0c20's.

[ashhart]

Thank you, @jschmied, for the focused fixes and regression cases at 7c44606.

The config refusals, Store ownership, normal sequence-capacity checks, slot initialization and engine cleanup now address the earlier cases in source. GLM's zero-limit behavior remains restored and WY remains removed.

Two follow-ups remain. ling/forward.zig:37 checks sequence capacity but leaves scratch capacity to the later MLA block; manually mismatched scratch and state can still commit KDA before refusing. Normal Engine allocations use matching capacities. In ling/weights_proj.zig:273, a failed packer launch can free pinned upload storage before the stream drains.

Please add the scratch preflight and drain-before-free error path, with a mismatched-capacity case and a test that keeps a transfer pending. We can fold these small fixes in at landing. We'll compare the newly pinned HF sources and run the host/fault checks, followed by full-model drafted/plain, resumed/fresh and concurrent/solo tests with arbitrary cuts and partial keeps.

Posted by Ash's AI Assistant Synthena

[jschmied]

@ashhart, thank you. Both follow-ups are in, one commit each, on top of 7c446062 (head a78a7a49).

1. Scratch preflight (84050923)
block_mla.room checks a segment against the sequence's capacity and against the scratch's attention partials (Buffers.nch, the only scratch buffer sized from a capacity; row counts were already checked). Forward.rowsOf calls it for every segment before any layer runs, with the same CacheFull. block_mla also calls it for every segment before its first cache write. That covers the MTP layer, which runs the block without rowsOf.

Regression (ling-lanes, engine checks): Buffers built for 512 positions against sequences of 1024, with twins at 500. A 30-row prompt chunk, a 16-row window and, with --mtp, 16 MTP rows each fit the state but not the scratch, and each returns CacheFull. The arenas (KDA states, conv windows, MLA and MTP caches) and the position stay byte for byte as they were. The next window then equals the twin's in tokens, logits and states, and a new request on that slot equals one on a fresh slot.
With the preflight reverted, the arenas differ and the next commit fails with KdaLayersOutOfStep. With only block_mla's ordering reverted, the MTP case fails on its own.

2. Drain before free (a78a7a49)
Reader now records whether an upload or packer launch is queued, and settled() drains it before anything is freed. That covers the expert stage (pinned host and device, weights_proj.zig:273) and the bf16/FP8/NVFP4 lane staging. I found the same pattern in other places and fixed them too:

• the packed-output store in weights.load
• absorbOf's identity rows, results and partials in graph.zig
• the engine's weights and graph when a later init step fails
• keepState's copy targets

If the drain itself fails, the memory stays allocated on purpose. Store.drop, Weights.drop and Graph.drop free the host lists and leak the device buffers.

Regression: fake_driver.zig now keeps async copies, memsets and launches pending until a successful synchronize. It counts every free made while work is pending, and it can fail every call from N on, drains included. ling-init-faults fails each driver call and host allocation of the open plus a kept state in turn, with transfers pending, and then every call from N on. No free may happen while work is pending, and memory may stay held only after a failed drain. Over bf16, FP8 and NVFP4, with and without MTP, that is 15,180 failure points (1,804 of them failing with a transfer pending), all clean.
I reverted each fix on its own (Stage.deinit, lane staging, NVFP4 lane staging, the load's store, absorbOf, the engine's weights, keepState), and each revert makes the test fail. The Stage.deinit revert gives 24 to 32 failing points per FP8/NVFP4 run.

Re-run at a78a7a49:

• Host and lint: zig build test, test-ling 30/31 (1 skipped), test-cuda-host, zig fmt --check, lean_check 527 (unchanged).
• GLM unchanged: glm-ref; glm-route-old and glm-mla-same (13 launches, 14.3 MB equal to 6380c8da); ling-route-old.
• Ling: glm-kda-ling; ling-slots; ling-ckpt-check (three synthetic checkpoints and the full FP8 release); ling-init-faults and ling-lanes for bf16/FP8/NVFP4 with and without --mtp (100 and 109 checks); ling-weights for the three formats; ling-pack layer 4.
• Real layer: ling-layer-real L4 at rel L2 3.126e-3, as before. Its output bytes equal 7c446062's.

The branch conflicts with main since 1.0.12 (#804 is in it). I can rebase it onto main if that helps the recheck.

[jschmied]

Rebased onto main 46a08c18 (1.0.12), now at 1395e267, since #804 is in main. Conflicts were only in wiring: build, the CUDA family registry in native/cuda.zig, the hub title and the test command table. Every Ling-only file is byte-identical to a78a7a49. Re-run: GLM unchanged (glm-route-old and glm-mla-same equal to 6380c8da); ling-lanes ids/picks byte-equal to before; ling-layer-real L4 output bytes equal (rel L2 3.126e-3). Hashes in the description are mapped to the new commits.

[jschmied]

We are pausing here. The branch stays as it is at e99e6505, and anyone is welcome to continue it.

Why: the whole model does not fit one GB10. FP8 is 574 GB and NVFP4 about 324 GB, so an end-to-end run, real MTP acceptance and TP all need several Sparks; this PR was checked on synthetic checkpoints and single real layers only. A one-Spark build would have to be around 1.25-1.7 bpw EXL3, which TensorFold reads only once #832 is in. We tried our own 1.25 bpw conversion and stopped it: with 1-bit experts, each layer's error rose about 1.5x over 1.5-bit experts and the hidden states drifted quickly with depth, so we didn't expect a usable model.

Next steps: someone with two or more Sparks can take TP and the end-to-end checks (docs/recipes/ling-3.1.md lists the open items and how to test). We will pick it up again if a quantization that fits one Spark and holds up in quality is published.

[ashhart]

Thank you, @jschmied, the scratch and failure controls leave useful work available even though the whole model exceeds one GB10.

We checked e99e650 and will keep the branch parked, following your pause. It stays parked until a quality-qualified pack fits one GB10 or the TP fabric in #636 lands, with resumption left to you.

The failed-drain teardown follow-up at engine.zig:149-163/199-204 can go in at landing. We will retain your synthetic, allocation and real-layer receipts with their scope; whole-model, real MTP and TP qualification still need their own runs when work resumes.

Posted by Ash's AI Assistant Synthena

[jschmied]

Ling-3.0-tiny (inclusionAI/Ling-3.0-tiny@9a98e35f, MIT, 7.9 B parameters, 1.1-1.4 B active, bf16 15.8 GB) has the same architecture and fits one GB10, so the size that paused this PR is gone. It runs on this PR's CUDA path at TP=1. Five commits on top of e99e6505 (branch ling-tiny; the PR branch still conflicts with main, a rebase is separate).

What 3.0-tiny changes in the config, one by one

• q_lora_rank 256 was the only refusal. It only changes q_a_layernorm's width: glue.rmsnorm had 512, 1536 and 4096, so there is a 256 instance now (2 warps, one tile). The 47 older functions of glm_glue and the 4 of ling_glue keep their sm_121 SASS and registers. It is registered with the kernel contract: gates 1-4 and gate 5 pass on the GB10.
• expert_swiglu_limit_list and share_expert_swiglu_limit_list are null: read as no clamp.
• Same code, no change: 24 layers in groups of 4 (MLA on 3, 7, ..., 23), dense layer 0, hidden 1536, 16 heads, 128 experts in 8 groups of 16 keeping 4, moe/shared width 512, no MTP head. Everything else still refuses with its named error; tiny_test.zig holds the real config and all 9,283 index names, and a one-edit copy of each refusal.
• bf16 has no grouped expert kernel, and the dense loop read the whole model per token (11.2 tok/s). block_ffn.picked runs each expert on just the rows that picked it. 480 prompt rows and 32 decode rows after them are byte-identical to the old loop (TF_LING_DENSE_EXPERTS=1 keeps it). Added tf_ling_gather_rows and tf_ling_scatter_rows.

Receipts (one GB10, bf16, files checked by lfs.sha256)

• zig build test from the root: 897 of 905 pass, 8 skipped; lean 0 new; zig fmt clean.
• ling-ckpt-check: 9,283 tensors, none missing or wrong. The index's total_size (3,869,925,683,584) is wrong; the shards hold 15,786,865,792 bytes and the check now goes by them.
• ling-lanes: 117 checks. Chunked == whole (512, 37, 16 rows), drafted == plain (greedy, sampled), resumed == fresh at marks 37 to 349, concurrent == solo at 2, 3 and 4 streams, cancels, capacity refusals. Also on synthetic tiny-shaped checkpoints (ling-synth --tiny) in bf16, FP8 and NVFP4.
• Logits of 480 tokens of real text against ling_forward.py's float64 on the same picks: rel L2 1.05e-2, where the HF bf16 rounding points are 1.04e-2 from float64; argmax 477 of 480 (the rest within 0.11 of the runner-up). Decode rows equal prompt rows byte for byte. The transformers model in bf16 on the GPU is 3.3e-2 from float64 and 3.5e-2 from this engine (475 of 480).
• Greedy chat through the server: 64 tokens, equal to transformers'.
• bench_concurrent.py --alone at 1, 2, 4 streams: every reply equals its solo run.

bf16	dense loop	picked
decode, 1 stream	11.2 tok/s	76.6 tok/s
decode, 4 streams		149-157 tok/s aggregate, 38-40 each
TTFT, 8,189-token prompt	4.55 s	1.74 s

One test threshold changed: ling-lanes against float64 bounded the distance to the bf16-points model by the float64 distance alone; independent roundings of real weights differ by the two errors in quadrature (5.13e-3 against 4.24e-3 and 3.80e-3 here), so the bound is now that. The float64 bound is as before.

Not covered: the official FP8 tiny on real weights (the loader's FP8 path is checked on synthetic tiny-shaped files), EXL3, TP > 1, MTP (tiny has none). The server needs the possessive-quantifier tokenizer change that main already has (79ba3f58c); against tokenizers it matched 2,784 of 2,784 encodes on tiny's file.

[ashhart]

Thanks @jschmied. The exactness work here is strong (float64 references, oracles pinned by sha256 to the HF source, fault injection, weights served in their own formats). It's about 1,800 commits behind (base beta.7) with 13 conflicts in the shared GLM CUDA kernels, so it needs a rebase onto current main, after which GLM's images, launch records and tokens must be shown unchanged. Please also register tf_glm_chunks_r16, tf_pack_fp8_experts, ling_glue and the EPI-4 instances with the kernel contract. Then we'd run the Ling oracles, drafted vs drafts-off, and a GLM token check on a GB10 at TP=1.

[ashhart]

Review against main e9b69ed. The exactness work here is strong: float64 references, oracles pinned by SHA-256 to the Hugging Face source, fault-injection tests, and weights served in the checkpoint's own formats. The blocker is age: the branch is based on beta.7, about 1,800 commits behind, with 13 conflicts in shared GLM CUDA kernels. The GLM changes use template defaults and new instances (RW=0, EPI 4), so GLM's paths look unchanged in source, but that has to be shown again after the rebase, with GLM's images and launch records identical. The new launches (tf_glm_chunks_r16, tf_pack_fp8_experts, ling_glue, the EPI-4 instances) also need contract records. Device plan once rebased: on a GB10 at TP=1, the Ling oracles, drafted output equal to drafts-off, and GLM's tokens unchanged.

[ashhart]

Thanks @jschmied for Ling on CUDA and the careful float64 references. We're closing this draft for now, as it has been parked since 16 Oct and our 26 Oct rebase request hasn't had a reply. You're very welcome to reopen it, or send a fresh PR for Ling-3.0-tiny, rebased on current main with GLM's images and tokens shown unchanged.