[jcscocca]

What this changes

Full GLM-5.3 (EXLR8, --speed-up) never serves on two M5 Ultra Mac Studios: rank 0 hangs while loading and rank 1 then needs kill -9. The cause is the one confirmed in #3158: pair_post_wait in core/pair.metal waits for the peer inside a running command buffer by polling memory the host writes. This change removes that GPU-side wait. Each exchange now retires its producer command buffer, waits for the peer on the host, and encodes the consumer into a new command buffer only after the host has published the exact 64-bit receive epoch. A silent or dead peer ends in a controlled PairLinkFailed on both ranks.

• Issue Refs GLM-5.3-Flash --speed-up on two M5 Ultras (macOS 27.2): ranks connect, then rank 1 fails exchange 2: PeerSilent and rank 0 never serves #3158. That issue reports the GLM-5.3-Flash pair, which glm metal: use shared events for paired transfers #3293 fixes. This is the same root cause on the full GLM-5.3 path (families/glm_dsa, core/pair.zig, core/pair.metal), which glm metal: use shared events for paired transfers #3293 does not touch.
• Stacked on glm metal: use shared events for paired transfers #3293 at cf2c65bd3e562b9cfd621f56b8af6c45a71bfc3f. This pull request adds one commit, 786950bc62be9e9fd32c51fb700c0c56a1d55034 (25 files, +1768 −170). The four earlier commits on the branch belong to glm metal: use shared events for paired transfers #3293 and are not new scope here. The branch merges cleanly onto main at 2ba55ec7d32d0bec446bb012678550be7ef52895.
• atc overlap --json Run with atc 0.9.6 on 2026-11-14 UTC against the stack base. The repository-wide run and the selected run (--pulls) both returned complete: false (timeLimit, conflictChecksUnavailable) and listed 14 open pull requests sharing files, each with conflict unknown. I then test-merged this head and its base against those 14 plus qwen27: serve the 27B pair, refuse what a pair cannot keep, and die with the peer #1434, glm metal: one arithmetic order for prompt windows, chunks and decode on M1-M4 (#743) #2128 and metal families: refuse a token id at or past the embedding table before the lookup #3045 with git merge-tree. One new conflict appeared, with Implement GLP control vectors with a fused Metal kernel for Qwen3.8-27B #1835 in zig/tests/contract/metal_limits.zig; moving this change's branch of that else if chain below the neighbouring one removed it. On the pushed head no pull request in that set has a conflict that it does not already have with glm metal: use shared events for paired transfers #3293's head. glm metal: one arithmetic order for prompt windows, chunks and decode on M1-M4 (#743) #2128 keeps its known conflict in zig/src/families/glm/prompt.zig, inherited from glm metal: use shared events for paired transfers #3293.

Why not shared events inside one command buffer, as in #3293

The first build of this change did that, and it passed one start. On the second start a command buffer accumulated enough handoffs to sit in an event wait for about 5 s and the GPU watchdog killed it (kIOGPUCommandBufferCallbackErrorTimeout). The build in this pull request has no GPU peer waiter at all, so there is nothing for the watchdog to kill.

Shared code

core/pair.metal and core/pair.zig are used in production only by the full GLM-5.3 engine and model, and both callers are converted together. core/pair_rows.* and the existing metal pair contracts are unchanged.

Checks on this head

• Head commit 786950bc62be9e9fd32c51fb700c0c56a1d55034.
• Lean checker python3 tools/zig/lean_check.py with the head's checker: 341 findings on the base cf2c65bd3, 341 on this head. tools/zig/lean_new.py reports 0 new.
• Tests run on an M4 Pro (macOS 26.7.1), Zig 0.17.0:
  • zig build test-glm-dsa-pair -Dcpu=apple_m1 -j4 --summary all: 43/43 steps, 73/73 tests passed, 0 skipped.
  • zig build test test-golden -Dcpu=apple_m1 -j4 --summary all: 851/853 steps; 13,791 of 13,902 tests passed, 109 skipped, 2 failed. The same command on the base: 13,771 of 13,882 passed, 109 skipped, the same 2 failed (core.paced_read.test.aligned spans copy the bytes asked for, at any offset and length and families.glm.weights_fix_test.test.GLM copies and transforms join before shard eviction; errors and duplicate opens retain ownership). Both failures are inherited. This head adds 20 passing tests.
  • zig fmt --check build.zig zig and git diff --check pass.
  • python3 tools/pr_check.py --base cf2c65bd3: 25 changed files, lean 341 vs 341, 0 new; 0 errors, 1 advisory warning (the Unreleased line, which needs this pull request's number).
  • Kernel contract: see the section below.
• CI Not started. Fork workflows need maintainer approval.
• Not run, and why
  • The full Metal kernel contract (zig build kernel-contract -Dcpu=apple_m1) exits 1 on this M4 Pro with UnrecordedRotaryCompiler before reaching any kernel, on the base as well as on this head. The filtered device runs below cover every changed kernel and the other metal pair kernels.
  • Pristine main was not run on the Studios. The "before" run used the glm metal: use shared events for paired transfers #3293 head build; core/pair.metal, core/pair.zig, families/glm_dsa and native/glm_dsa_host.zig are identical between that commit and main at 2ba55ec7d.
  • CUDA, HIP and XPU: outside this change.
  • M1 to M4 pairs: not run. I have one M4 Pro and two M5 Ultras.

Receipt

• Main against branch Same two Studios, checkpoint and flags (--context 8192 --parallel 4).
  • Before (build of cf2c65bd3): rank 1 loaded in 396 s (159.6 GB) and reported serving. Rank 0 never finished loading; after 20 minutes its main thread was in families.glm_dsa.engine.Engine.chunk > Engine.finish > -[_MTLCommandBuffer waitUntilCompleted]. After rank 0 was stopped, rank 1 did not exit on SIGTERM within 30 s and needed kill -9.
  • After (this head): API ready after 519 to 531 s on seven separate starts of the same binary (SHA-256 0470bd50ca3ad67d88bf04076a5e7c0362a4afcb4b8dbbae20b2cdd0fdf4b4eb), every start stopped both ranks on SIGTERM with no kill -9 and no GPU watchdog message.
• Boxes it reaches Native Metal full GLM-5.3 over a two-rank Thunderbolt 5 MCDMA link. One-Mac serving of this family does not exist. Other families do not use core/pair.*.
• New switch? None. The host wait is the default and only behaviour in pair mode.
• Environment TensorFold 786950bc6 on cf2c65bd3; Zig 0.17.0; two Mac Studio M5 Ultra, 256 GB, macOS 27.2, rdma_en3 over Thunderbolt 5; checkpoint kindlingai/glm-5.3-exlr8-k2-k3-k4@f459f5d0272a529403dd9b3fd32f771f4250a0d4; greedy, integrated drafts.
• Paths and platforms Ran: M5 Ultra pair (serving, exactness, failure paths) and M4 Pro (host tests, filtered kernel contract). Could not run: M1 to M4 pairs, other backends.
• Exactness tools/bench_concurrent.py http://127.0.0.1:18792 glm-5.3 --tokens 256 --reps 3 --temperatures 0 --levels 1,4 --alone --serial: 0 failed. One stream: 3/3 replies equal to the request alone, and equal to "draft": false. Four streams: 12/12 equal to the request alone, and equal to "draft": false. The same holds with thinking off. Separately, two prompts sent with drafts on and off, two repeats each, returned identical 128-token visible replies and token hashes, and both ranks logged equal pick counts and pick hashes on every run (for example 17,273 picks, hash 08dd0396…d56a, on the 75-request run).
• Main against this branch, same machine and checkpoint See the first line: main's path does not serve on these machines, so there is no decode number to compare with.
• Decode speed tools/bench_openai.py … --tokens 256 --reps 3 --temperatures 0, median of three, one stream: 8.92 tok/s (fibonacci-raw), 7.85 tok/s (gpu-chat-no-think). bench_concurrent.py at four streams: 33.9 tok/s aggregate, 8.7 per stream; 30.4 aggregate with thinking off. Round gap p50 is 314 to 324 ms at about 2.5 to 2.7 tokens per round. This is below the 21 and 31 to 32 tok/s in docs/glm-5.3-exlr8.md. I do not know how much of the gap is this change (each exchange now pays a command-buffer boundary) and how much is these machines; there is no working baseline here to separate them. Server-reported decode on other prompts ranged from 6 to 31 tok/s and was episodic within a run, with no established cause.
• Prompt speed tools/bench_cache.py … --require-cold, 2,570 prompt tokens, zero cached tokens reported: ttft_s 13.21, 13.12 and 12.99 across the first and two repeated requests. No before number, for the same reason.
• Failure paths, on the Studios SIGTERM to rank 1 during a streamed request: rank 0 logged PeerDown and returned PairLinkFailed in 0.55 s, then stopped normally. SIGSTOP to rank 0 while rank 1 was idle: rank 1's follower returned PairLinkFailed 2 s later. A 75-request run including 60 consecutive mixed requests completed with no error.
• Licence of any new weights or drafter n/a.

Kernel contract

• Kernels pair_stage is new. pair_add_rms, pair_argmax and pair_pick replace their same-named versions, which called the polling pair_post_wait; that function is removed. Each consumer now takes the host-published 64-bit epoch and refuses a slot whose epoch does not match.
• Controls One planted control per kernel under metal glm dsa pair; each fails as planted.
• Commands zig build tf-kernel-contract -Dcpu=apple_m1, then tf-kernel-contract --only 'metal glm dsa pair' and tf-kernel-contract --only 'metal pair'.
• Apple M4 Pro (applegpu_g16), exit 0 each

SUMMARY 4 kernels: 4 pass gates 1-4, 0 fail, 0 known findings; 4 controls: 4 red as planted, 0 missed; 0 limit problems; gates 604 ms, limits 66 ms
SUMMARY 4 kernels: 4 pass gates 1-4, 0 fail, 0 known findings; 6 controls: 6 red as planted, 0 missed; 0 limit problems; gates 62 ms, limits 113 ms

• Apple M5 Ultra, both Studios, --only 'metal glm dsa pair', exit 0 each

SUMMARY 4 kernels: 4 pass gates 1-4, 0 fail, 0 known findings; 4 controls: 4 red as planted, 0 missed; 0 limit problems

Not done

• No explanation for the decode speed gap against the guide, or for the episodic slow requests.
• Six small allocations per rank (about 59 KB) are reported leaked at exit. They come from GLM DSA cleanup paths this change does not touch; I confirmed that by reading the source, not by running the base, which cannot reach a clean exit here.
• Raw benchmark JSON (decode.json, concurrent.json, cache.json) and the run logs are available on request; they are not attached yet.