[jcscocca]

What this changes

Paired GLM-5.3-Flash can stall during startup when its Metal kernels poll a peer flag while the host writes it. Use Metal shared events for both directions of the handoff. Publish an arrival only after the peer transfer flushes. Bound each send with a short host deadline and the peer's arrival with a longer one, and use 64-bit epoch guards to release waiters safely when the link fails.

• Issue Fixes GLM-5.3-Flash --speed-up on two M5 Ultras (macOS 27.2): ranks connect, then rank 1 fails exchange 2: PeerSilent and rank 0 never serves #3158.
• Stacked on main at bafe3e07cb35f1c406a61d61fe3933a415c4f645. The rebased patch is byte-identical to the one used in the original Studio benchmark. Upstream has since advanced to 90b4fae6. Its changes are in CUDA and the changelog. A local three-way merge against that tip completed without conflicts.
• atc overlap --json The selected-PR check against 90b4fae6 on 2026-11-14 UTC found one conflict with glm metal: one arithmetic order for prompt windows, chunks and decode on M1-M4 (#743) #2128 in zig/src/families/glm/prompt.zig. The registry registration was moved away from metal row projection: windows of 4+ rows run 8 at a time, then 4, 2 and 1, each code unpacked once #3129's insertion, and the follow-up check found no conflict with metal row projection: windows of 4+ rows run 8 at a time, then 4, 2 and 1, each code unpacked once #3129, MoE route tap: optional TF_ROUTE_TAP trace of global top-k #1886, or metal families: refuse a token id at or past the embedding table before the lookup #3045. A separate three-way audit of 68 file overlaps among 361 open PRs found glm metal: one arithmetic order for prompt windows, chunks and decode on M1-M4 (#743) #2128 was the only new conflict.

Tested merge resolution for #2128

The trial merge used #2128 at a2b0943d97f15647210239ca0dc55654968420a5 projected onto main at 948e7485840834e5c78f2db1ce4d44322cf72e44. Keep the *Pass signatures and const e = pass.enc in prompt.zig for moe and mla. Restore #2128's x.k.ordered early returns, calling ordered_prompt.moe with pass and ordered_prompt.mla with e. In ordered_prompt.zig, accept *Pass in moe and forward it to fwd.moe. Ordered dispatch remains limited to the unpaired path.

zig build native test-glm-ordered-host tf-kernel-contract -Dcpu=apple_m1 -j2 passed 73/73 build steps and 10/10 ordered GLM host tests on the composed tree. The trial merge is separate from this draft. The same patch applied to the projection of #2128 onto current main at bafe3e07c. The resolved tree 8e7d9c9e1b5e9fb9a3cdbde1b30b4722048a119e passed all 10 ordered GLM host tests. An apply check also succeeded against the projection onto 90b4fae6. After applying the replay patch, apply the follow-up test diff from git diff --binary 5ec85fd74 27ddaf316 -- zig/tests/contract. The combined apply check on 90b4fae6 produced tree b54b59f876b5918dba265220f68eff8540b28461. Apply the resolution when rebasing the second PR after the first lands.

To reproduce on a clean disposable worktree containing those refs, project #2128 with git merge-tree --write-tree bafe3e07c a2b0943d, load that tree with git read-tree -mu <projected-tree>, and apply the replay with git apply --index merge-2128.patch. Download the tested replay patch.

Checks on this head

• Head commit cf2c65bd3e562b9cfd621f56b8af6c45a71bfc3f. The latest follow-up splits the relay's deadline in zig/src/families/glm/ep.zig. One native send and its flush keep the 1 s budget and fail with EpTransferTimeout. The wait for the peer to reach the same exchange and send it now has its own 10 s limit, the same as main's exchange wait, and fails with PeerSilent. Before this, the 1 s budget started at this rank's GPU post, so a peer running more than 1 s behind ended the pair. A GLM_EP_DELAY_US hold now runs before the send budget starts. A healthy exchange now also re-arms the deadline once after its flush. The fault, late-peer and restart checks were rerun on the Studios at this head. The decode, prefill and exactness tables were measured before this change.

• Checks at cf2c65bd3 zig build native test-glm-expiry test-glm-scheduling-host tf-kernel-contract -Dcpu=apple_m1 -j4 --summary all passed 75/75 build steps and 85/85 tests on the M4 Pro. tf-kernel-contract --only 'metal glm ep' exited 0 with 8/8 kernels passing, 8/8 controls caught and zero limit problems. zig fmt --check build.zig zig and git diff --check passed. The No GPU workflow's Zig steps also ran on the M4 Pro with -Dfixed-cpu: test-kernel-contract (205/205 tests), test-golden, test-gradient-serving test-qwen-serving-host, test-server-security test-pull-security (27/27 security checks) and fuzz-cortheon passed. test-host registers only on a Linux host, so all 48 test-glm-* and check-glm-* steps it collects ran instead with --test-timeout 20m: 185/185 build steps and 374/376 tests passed, with two GLM OCR tests skipped. The focused MCDMA tests were not rerun at this head. The Studio reruns are under Receipt.

• Previous head 27ddaf3161c07c4197c789aa92425b01496b4dfe removed 41 net lines from the Metal test scaffolding. The same eight kernel probes and eight planted controls use one production Metal library and one fault-injected library. It left the inference source and binary unchanged.

• Inference checks at 5ec85fd74 The native build, focused GLM host targets, both golden suites, and Metal contract build passed in zig build native tf-kernel-contract test-glm-expiry test-glm-scheduling-host test-golden -Dcpu=apple_m1 -j4 --summary all (80/80 build steps, including cached results for the focused host tests). The golden suites reported 322/335 and 55/55 matches with zero unexpected differences. The rebuilt native binary had the same SHA-256 as the Studio benchmark binary. A fresh smoke attempt on the rebuilt binary stopped before any exchange when rank 1 reported MemoryShort (7.7 GB free or reclaimable against the 8.6 GB admission floor). Both test processes were cleaned up. On the rerun with memory available, the same draft-head binary served on rank 0 in 15.421 s. All three requests returned the expected replies (BANANA, 42, and blue), each rank landed all 1,502 exchanges, and both exited cleanly. The focused M4 contract rerun passed 8/8 kernels with all 8 controls caught and zero limit problems. zig fmt --check build.zig zig and git diff --check passed.

• Lean checker python3 tools/zig/lean_check.py reported 341 inherited findings on bafe3e07c, 27ddaf316 and cf2c65bd3. tools/zig/lean_new.py reported zero new findings.

• Developer tools at 27ddaf316 zig build tf-kernel-contract dev-tools -Dcpu=apple_m1 -j3 --summary all passed all 264 build steps. Zig formatting and diff checks passed.

• Focused MCDMA tests at 5ec85fd74 19 passed, 0 failed, 0 skipped. These use fake endpoints for Thunderbolt and ConnectX, so they run without a native MCDMA library.

  zig test -lc --dep sys --dep verbs --dep bf16_round -Mroot=zig/src/fabric/mcdma.zig \
    --dep sys -Mverbs=zig/src/fabric/verbs.zig -Msys=zig/src/platform/sys.zig \
    -Mbf16_round=zig/src/core/bf16_round.zig

• Original checks at 948e74858 with the contract additions zig build test-kernel-contract -Dcpu=apple_m1 passed 229/229 tests. The Metal launch map resolved all 736 registrations and both limit controls. The focused Metal device results are below.

• Inference source checks before the contract-only additions zig build native -Dcpu=apple_m1 passed. zig build test-glm-expiry test-glm-scheduling-host -Dcpu=apple_m1 passed 85/85 tests, and the focused MCDMA unit run passed 19/19. zig build test-golden -Dcpu=apple_m1 had zero unexpected differences across both golden suites. zig build test -Dcpu=apple_m1 passed 4,949, skipped 57, and failed two local memory-floor tests (paced_read and GLM weights_fix). Both reproduced as MemoryShort on pristine main.

• CI GitHub created No GPU and Zig format for 27ddaf316. Both require maintainer approval for the fork. No CI jobs have started.

• Metal missing-device check at 27ddaf316 A process-local dyld interposer forced MTLCreateSystemDefaultDevice to return NULL. The prior runner exited 1 with NoDevice. The fixed runner prints SKIP Metal kernel contract: no Metal device and exits 0 in both focused and limits-only modes, before opening a command queue. Both runs emitted no gate summary. With a real M4 Pro device the rebuilt runner passed 8/8 kernels, caught 8/8 controls and had zero limit problems. Filters for a copy kernel, a reduction kernel, controls only and no match retained their expected counts. Limits-only mode checked all eight launches.

• Not run to completion The full Metal numerical suite was stopped after compiler calibration. The filtered device suite ran every changed handoff kernel. The local glm metal: one arithmetic order for prompt windows, chunks and decode on M1-M4 (#743) #2128 composition was built and host-tested. Its unpaired path still needs an on-device inference comparison if the changes land together.

Receipt

• Main against this branch Both ranks connected on pristine main at 948e74858. Rank 1 failed with PeerSilent at exchange 2, and rank 0 did not become ready within 65 seconds with tracing off. The patched rank 0 became ready in 21.126 seconds and served requests. Both runs used the same Studios, checkpoint, MCDMA library and 66,560-token context, with the prompt cache disabled and tracing off. The startup failure prevented measuring baseline decode throughput or cold prompt time to first token at any width.
• Boxes and paths Native Metal GLM-5.3-Flash over a two-rank Thunderbolt 5 link. The full matrix used the default paired split. A separate smoke run exercised the expert split at one and four streams. CUDA, HIP, and XPU are outside this change.
• Environment Two M5 Ultra Mac Studios, 256 GiB each, macOS 27.2 RC build 26B118, one Thunderbolt 5 link, and Zig 0.17.0. Checkpoint and integrated MTP weights from TensorFold/GLM-5.3-Flash-MLX-4bit-MTP revision d2b17f5253440224422b150705c1109a9a3a0368. Both ranks used the same MCDMA dylib SHA-256 ae63dd35624573a647d8fd23b0a29c436a5d3d3de97d40c29d880ce179ac7679. The original Studio inference benchmark used binary SHA-256 54594cc1ec9d2d46d4da484e7b64bd2b3747de58ac1118262a9fcbfaf50f7aa8, built on 948e74858 with this production diff.
• New switch and profile entry None.
• License of new weights or drafter N/A. Both ranks used the existing checkpoint and its integrated MTP weights.
• Sampling Benchmark requests used temperature 0 and seed 1234. A request at temperature 0.7 returned HTTP 400, as expected for this greedy-only path.
• Exactness All 201 independent full token-ID comparisons matched across 197 successful requests. The default split covered drafting on and off, and solo versus concurrent at every admitted width from 1 to 8. Cold runs covered 2,048 through 65,536 tokens. The expert split contributed 12 comparisons across four fixtures. Cache replay restored 11,178 of 11,226 prompt tokens, with identical 256-token replies in fresh, resumed draft, and resumed plain requests.
• Timeout and clean restart An injected 1.6 s stall inside a native MCDMA call during startup caused both ranks to report EpTransferTimeout at exchange 183. The stalled rank reported the timeout before the native call returned. Both ranks exited without a GPU watchdog. A fresh pair answered BANANA, 42, and blue and completed 1,502 exchanges on each rank. That run predates cf2c65bd3.
• Timeout at cf2c65bd3 Same Studios, checkpoint and MCDMA library, with a test-only shim that re-exports the MCDMA library and holds rank 1's 183rd payload mcdma_fabric_write_signal call for 1.6 s. Rank 1 reported EpTransferTimeout for exchange 181 after 1.001 s, before the stalled call returned. Rank 0 accepted that late payload, then reported PeerDown at exchange 182 when rank 1 closed the link. Both exited with status 1 about 2 s after the stall began, without a GPU watchdog. A fresh pair answered BANANA, 42, and Blue, each rank completed 29,460 exchanges, and both stopped with status 0 on SIGTERM.
• Late peer at cf2c65bd3 With GLM_EP_DELAY_US=1500000 on rank 1, each prompt-chunk send over 1 MiB waited 1.5 s. The pair completed all 180 held exchanges and answered a 392-token prompt with BANANA. The previous 1 s budget would have failed the first held exchange. With GLM_EP_DELAY_US=12000000, both ranks' command buffers failed with kIOGPUCommandBufferCallbackErrorTimeout about 5 s into the first hold, before the 10 s arrival limit. Both reported GpuFailed and exited with status 1. On these Macs a peer more than about 5 s late therefore ends the pair through the GPU timeout rather than PeerSilent.

The public receipt preserves exact prompt fixtures, full output IDs, cache counts, timing data and settings. Download both files into the same folder and run python3 verify_studio.py to verify the comparisons and reproduce the medians. Raw Studio receipt · Verification script

Decode and cold prefill

Median of three runs per cell with tracing off and the prompt cache disabled. Decode uses 256-token replies across public repository fixtures. Aggregate decode is the total tokens after each stream's first token divided by the time between the first and last emitted tokens across the group. Cold prefill reports client time to first token and uses 16-token replies. Every cold run reported zero cached prompt tokens.

Measurement	Drafting on	Drafting off
1 stream decode	107.61 tok/s	24.16 tok/s
4 streams aggregate decode	114.08 tok/s	73.79 tok/s
8 streams aggregate decode	151.31 tok/s	115.59 tok/s
2,048 prompt tokens	1.441 s	1.424 s
8,192 prompt tokens	4.421 s	4.380 s
32,768 prompt tokens	17.578 s	17.548 s
65,536 prompt tokens	35.545 s	35.451 s

To reproduce startup on each rank, build with zig build native -Dcpu=apple_m1, then use the same checkpoint and the respective rank's settings file. Start rank 1 before rank 0.

tensorfold-native serve "$CHECKPOINT" --name glm-paired --host 127.0.0.1 --port 8153 \
    --speed-up "$RANK_SETTINGS" --context 66560 --prompt-cache-gib 0 \
    --temperature 0 --no-thinking --no-update-check

For the timing and equality measurements, stream the same requests through rank 0 with draft set to true and false, and compare the full output IDs alongside the token SHA. For cold prefill, send exact-length token-ID prompts with a fresh KV cache for each request. For cache replay, enable the default prompt cache and repeat the same chat prefix.

Kernel contract

• Kernels ep_pack, ep_unpack, ep_rcombine, ep_rfinal, ep_rsend, ep_rsum, ep_amax_send, and ep_amax_merge replace their same-named flag-based implementations. Cases exercise live and stale epochs, repeatability, guarded buffers, non-finites, launch shapes, and FP32 accumulation and BF16 rounding where applicable.
• Controls Each kernel has a planted low-32-bit epoch comparison. All eight were caught on each machine.
• Commands zig build tf-kernel-contract -Dcpu=apple_m1 and tf-kernel-contract --only 'metal glm ep'. The M4 run also used --offline to check M1/M2 launches.
• Coverage Live M5 Ultra on both Studios and M4 Pro on macOS 26.7.1 build 25G241. M1 and M2 launch limits were checked offline with Xcode 26.6 and Apple Metal 32023.883. Live execution on M1, M2, and M3 was unavailable.

Studio rank 0 (M5 Ultra), exit 0
SUMMARY 8 kernels: 8 pass gates 1-4, 0 fail, 0 known findings; 8 controls: 8 red as planted, 0 missed; 0 limit problems; gates 4410 ms, limits 195 ms
Studio rank 1 (M5 Ultra), exit 0
SUMMARY 8 kernels: 8 pass gates 1-4, 0 fail, 0 known findings; 8 controls: 8 red as planted, 0 missed; 0 limit problems; gates 4374 ms, limits 195 ms
M4 Pro, exit 0
SUMMARY 8 kernels: 8 pass gates 1-4, 0 fail, 0 known findings; 8 controls: 8 red as planted, 0 missed; 0 limit problems; gates 5849 ms, limits 1318 ms

M4 Pro rerun at 27ddaf316, exit 0

SUMMARY 8 kernels: 8 pass gates 1-4, 0 fail, 0 known findings; 8 controls: 8 red as planted, 0 missed; 0 limit problems; gates 5552 ms, limits 1189 ms

M4 Pro rerun at cf2c65bd3, exit 0

SUMMARY 8 kernels: 8 pass gates 1-4, 0 fail, 0 known findings; 8 controls: 8 red as planted, 0 missed; 0 limit problems; gates 5611 ms, limits 227 ms

Checklist

• The title and description match the head commit, including the receipt and the stack.
• New tests fail before the change, pass after it, and skip cleanly without their dependency.
  • Before and after Pristine main reproduced the startup failure. On this patch the Studio smoke test passed, all eight Metal probes passed, and all eight low-word epoch mutations failed their intended checks. The new probes bind the new shader ABI, so their negative GPU evidence comes from the planted mutations.
  • Dependency behavior The 19 transport unit tests passed without the native MCDMA library using simulated endpoints. For the Metal runner, injected device absence now skips cleanly with exit 0. The prior runner failed the same check with exit 1.
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.
• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.
• No precision traded for speed. If the bits change, the description says which sums change and why.
• Decode and prompt processing are no slower than main at one stream and the widest admitted; with --drafter, no slower than without it, greedy and sampled
  • Main comparison Rechecked pristine main at 90b4fae6 on the same Studios and checkpoint. The default TP2 split at a 66,560-token context and GLM_EP_SPLIT=experts at an 8,192-token context both connected and then reported PeerSilent on rank 1 at exchange 2. Both ran with tracing off and the prompt cache disabled. Rank 0 reached neither HTTP readiness deadline of 65 s. The table reports candidate timings. A relative performance result needs a pair on which pristine main serves the same workload.
  • Drafter comparison Integrated MTP raised median greedy decode from 24.16 to 107.61 tok/s at one stream and from 115.59 to 151.31 aggregate tok/s at eight streams. Median server-side cold prefill at 65,536 tokens was 35.485 s with MTP and 35.402 s with drafting off, a measured difference of +0.23% over three runs per mode. Paired GLM Metal uses integrated MTP and supports greedy decoding. External --drafter and positive temperature do not apply.