[kotevcode]

What this changes

With --execution-witness, native Qwen3.5-2B on CUDA at one rank, every module MLX affine 4-bit and text only now reads known on the path qwen3_5-2b/cuda/tp1/mlx-q4g64/text. Every other engine still reads unsupported_path, and without the flag nothing changes.

• Before the weights load, the engine stamps config.json, the index, every shard the index names and a captured kernel set's aot.json and cubins (cuda_witness.Witness.open). A file it cannot stamp makes the witness read hash_failed; the model still serves. After the load, bind checks the loaded geometry is the 2B's uniform affine 4-bit layout and takes the GPU's and the family's facts; any other size, quant or format gets no hook and stays unsupported_path.

• Once the server is ready (and again after each --idle-unload reload, for the new engine), one thread reads every shard byte in full, with no digest cache. If a stamp moved between the load and the end of the read, or a Hugging Face snapshot's shard differs from its blob name, the witness reads files_changed. Shutdown stops the read at its next block.

• components.engine is a SHA-256 over the path, the checkpoint's structure (canonical config.json, the index, shard names, sizes and headers), every shard's bytes, and the components a kept state's key binds, written through native: the CUDA kept-state identity is written once through any hasher, its keys unchanged #2903's cuda_disk.components so the two lists can't drift: the embedded kernel images, a captured cubin set, the GPU's name, compute capability and driver, the family's resolved facts and the resolved profile values. Then the SM count, the context window, the stream count and the staggered segments.

• The witness passes std.crypto.hash.sha2.Sha256 straight to components, so aot_identity's captured set takes the SHA-256 branch, as noted on native: the CUDA kept-state identity is written once through any hasher, its keys unchanged #2903.

• docs/api.md names the supported path and what it binds.

• Issue: Refs Opt-in coherent execution witness for exact-response cache qualification #2741 (part 2 of 3: the first attested path; part 3 binds each reply to the witness).

• Stacked on: server: --execution-witness serves a read-scoped snapshot of what runs, or why it is unknown #2901 (d2ea235, 6cb4dbc, f287456, c5dc49f, 26d1327, a0c1043, 3067c2b, bd15bd6) and native: the CUDA kept-state identity is written once through any hasher, its keys unchanged #2903 (93282c1, c1c1d28: native: the CUDA kept-state identity is written once through any hasher, its keys unchanged #2903's a7bbda5 and 7795508 cherry-picked onto server: --execution-witness serves a read-scoped snapshot of what runs, or why it is unknown #2901); this PR adds 9057dbc, a4d644c, 563b4b9 (the review fixes) and b7a99ed (the hash_failed test keeps its expected warning out of the build summary).

• atc overlap --json: run on this head against main; complete: false (otherPullRequestsLimit, pullFilesUnavailable, timeLimit). 5 pull requests report a textual conflict, each only in CHANGELOG.md's Unreleased top: GLM-5.3-Flash on CUDA keeps conversation states on disk, every rank's part (--learn-conversations) #1411, dsv41 metal: the lane target, the prompt cache and the native server over Metal #1720, dsv41 metal: the vision tower's kernels under the kernel contract #1721, dsv41 metal: image prompts #1722 and Implement GLP control vectors with a fused Metal kernel for Qwen3.8-27B #1835. Unlimited-OCR-2 on native Metal: the publisher's BF16 release, page images to structured text #2074, flint: serve Recursive Flint (40B-A2.5B MoE) natively on Metal #1177, Nemotron 3.5 Transcribe: audio input, the 65,536-token window, an hour of audio, MTP drafts #1632 and Native Vulkan backend for Qwen3.5-2B (opt-in): engine and kernel-contract registration (stacked on #1526) #1544 are clean; 5 report unknown.

Checks on this head

• Head commit: b7a99ed
• Lean checker and python3 tools/pr_check.py --base upstream/main --pr 2956: lean 345 and 345 on the base and this head, 0 new; 0 errors, 0 warnings.
• Tests run, with the command and pass, fail and skip counts (macOS 26.6, M5 Max):
  • zig build test-execution-witness-cuda-host (new step): 18 passed, 0 failed, 0 skipped. Only dense Qwen with the flag, one rank, text and nothing learned wants the path; the loaded 2B's geometry gets a hook and other sizes, quants and formats none; a same-size, same-header shard rewrite changes the weights digest and a stamp moved since open reads files_changed; a captured cubin, a resolved profile value or the device changes the engine digest; each engine hashes once it is begun, and a reloaded one reads every byte again.
  • Before the change: the new tests need types this PR adds, so they don't build on server: --execution-witness serves a read-scoped snapshot of what runs, or why it is unknown #2901 alone.
  • zig build test-disk-identity-host: 35 passed. zig build test-execution-witness-host: 34 passed. zig build test-cuda-host: 390 of 390 steps, 6339 passed, 0 failed, 38 skipped.
  • zig fmt --check build.zig zig and git diff --check pass.
• CI: the PR's checks passed on 5043d29; this head is restacked on server: --execution-witness serves a read-scoped snapshot of what runs, or why it is unknown #2901's bd15bd6, still on server: --execution-witness serves a read-scoped snapshot of what runs, or why it is unknown #2901's base, so CHANGELOG.md's Unreleased top conflicts with main until server: --execution-witness serves a read-scoped snapshot of what runs, or why it is unknown #2901 lands.
• Not run, and why: Metal steps; this Mac has no Xcode Metal toolchain, and the path is CUDA-only.

Receipt

• Main against branch: below, on an A100, with the flag off and on. Raw results, the driver and the analysis scripts: https://gist.github.com/kotevcode/29d4d778e4ae8d47a01ff841cb218866
• Drafted against "draft": false and resumed against fresh, with the flag on: server: "execution_witness": true binds a reply to the witness at admission and at its end, with its prompt, resolved request and tokens #2958's A100 receipt, https://gist.github.com/kotevcode/5b4d26cd284c1f04878862f4c1611b93 (same A100 and checkpoint; this PR's previous head with server: "execution_witness": true binds a reply to the witness at admission and at its end, with its prompt, resolved request and tokens #2958's commits on top; the review commits since don't touch generation).
• Boxes it reaches: one A100-SXM4-40GB (sm_80).
• New switch? No. --execution-witness comes with server: --execution-witness serves a read-scoped snapshot of what runs, or why it is unknown #2901; this PR gives it a supported path.
• Environment: base 93a2e93, branch 822a67b4f (since then only the race test's idle setup from server: --execution-witness serves a read-scoped snapshot of what runs, or why it is unknown #2901 and the rebase onto main changed, so the server binary's sources are the same); Zig 0.17.0; nvcc 13.0.88, driver 580.178.04; Ubuntu 24.04, 12 vCPU, 83 GiB; zig build -Dgpu=sm_80; mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3 and TensorFold/Qwen3.8-27B-MLX-4bit@22d8d538154e0e5b6f8dcb1fcc74a60a0798104c, --context 8192 --no-thinking --api-key, the default prompt cache, a fresh process per start.
• Paths and platforms the change reaches, which I ran and which I could not: the native Qwen3.5 CUDA load and the serve startup. Ran: CUDA sm_80, 2B (the attested path) and 27B (unsupported_path). Not run: other SMs, HIP, Metal, Windows.
• Exactness: bench_concurrent.py --alone --serial at 1 and 8 streams passed in every cell for main, branch and branch with the flag, both while the hash runs and after it (9/9 alone at 1, 72/72 at 8, 3/3 serial, 0 unequal). Each prompt gave one token_sha256 across all three: code 579bcb3b…, chat 12faa747…, the cold 7,680-token prompt 924e91cc…; 27B's cold prompt f89b1fd0….
• Main against this branch, same machine and checkpoint: A main, B branch, C branch with --execution-witness; medians, alternating ABC, BCA, CAB.

2B	A	B	C
Start to /health 200, s (n=3)	4.972	4.350	5.006
/health 200 to known, s (n=3)			20.619
Cold first token, 7,680 tokens, after known, ttft_s (n=3)	0.353	0.352	0.368
Cold first token at /health 200, during the hash, ttft_s (n=3)	0.348	0.348	0.365
Cold first token, ten more alternating rounds, ttft_s (n=10)	0.349	0.360	0.358
Decode, 1 stream, code / chat, tok/s	834.8 / 414.2	834.6 / 414.4	834.0 / 414.0
Decode, 8 streams aggregate, code / chat, tok/s	3286.7 / 2058.2	3288.8 / 2054.4	3284.9 / 2052.3
Decode during the hash, 1 / 8 streams, code, tok/s	834.1 / 3286.2	833.3 / 3286.7	834.0 / 3278.4

27B (unsupported_path, nothing hashed)	A	B	C
Start to /health 200, s (n=3)	52.018	52.139	51.572
Cold first token, 7,680 tokens, ttft_s (n=3)	2.996	2.996	2.996

The first three cold first-token rows put C 15 to 17 ms above A and B; ten more alternating rounds put C 2 ms below B (Mann-Whitney p=0.88) and B 11 ms above A (p=0.45), so I read it as noise between builds, not the hash. Every C start gave the same content and components.engine, and a different witness and instance.

• Idle reload (two runs): known at epoch 1, then parked; the request that wakes it is answered in about 5 s, through reloading and pending at epoch 2, with the same tokens as before the park. Epoch 2 reads known about 25.7 s after that request, with the same content, components.engine and instance and a new witness.
• A shard touched after the load gives files_changed (3 of 3); --vision gives feature_excluded with "feature": "--vision"; --tp 2 is refused at startup for this family, as on main.
• Decode speed before and after, from tools/bench_openai.py: 256 tokens, 3 reps, temperature 0, 9 samples per config: fibonacci-raw 835.1 / 834.2 / 834.4 and gpu-chat-no-think 414.6 / 414.1 / 414.2 tok/s (A / B / C).
• Prompt speed before and after: tools/bench_cache.py --require-cold, 7,680-token prompt: above, every run cache_state: cold, cached_tokens: 0.
• Licence of any new weights or drafter: n/a.

Checklist

• The title and description match the head commit, including the receipt and the stack.

• New tests fail before the change, pass after it, and skip cleanly without their dependency.

• Comments and docstrings are one line. No measurements or history in the source.

• No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.

• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. Concurrent equals solo, and main, branch and branch with the flag give the same tokens (receipt above). With the flag on, drafted equals "draft": false and a resumed prompt (4,096 of 5,786 tokens cached, with --prompt-cache-gib 4) equals a fresh one in server: "execution_witness": true binds a reply to the witness at admission and at its end, with its prompt, resolved request and tokens #2958's A100 receipt, run on this PR's previous head with server: "execution_witness": true binds a reply to the witness at admission and at its end, with its prompt, resolved request and tokens #2958's commits on top: https://gist.github.com/kotevcode/5b4d26cd284c1f04878862f4c1611b93

• No precision traded for speed. If the bits change, the description says which sums change and why.

• Decode and prompt processing are no slower than main at one stream and the widest admitted (8); greedy only, and no --drafter run, since the flag refuses one.

[kotevcode]

A question on the weights read: it uses one worker thread, so it takes one core and one stream of disk reads away from serving. On the A100 box in the receipt, 1.6 GiB took about 20 s, limited by the disk (84 MB/s at 28% utilisation, the process at 78% of one core). More workers would finish sooner on a faster disk but compete more with decode. Should it stay at one, or follow a setting?

[ashhart]

Thanks @kotevcode. The core of this is right: the shards are stamped before the load, every byte is read once the server is ready with no digest cache, the stamps are checked again around the read, and an idle reload stamps and reads its own engine again. Off, nothing changes. Some asks before it leaves draft:

1. zig/src/native/cuda_witness.zig:51-66 and :133: the captured kernel set is hashed after ready but never stamped before the load. Besides TENSORFOLD_CUDA_KERNELS, aot.pick uses share/tensorfold/cuda/sm<N> next to the binary by default, and the Linux release archives ship a set there, so a refreshed capture during the window before the hash would read known with kernels that aren't the ones running. Please stamp aot.json and each cubin it names in open and check them in unchanged() (or have a captured set read unsupported_path for now), and change cuda_witness_test.zig:229-231 to expect files_changed, as the shard test does.
2. zig/src/native/cuda.zig:258: try Witness.open(...) turns any stamping error into a failed load, for every dense Qwen3.5 checkpoint on CUDA, not just the 2B. Opt-in coherent execution witness for exact-response cache qualification #2741 asks for unknown with inference still available; please log one line and read hash_failed, as openBinary/stampText do, and let only OutOfMemory through.
3. zig/src/server/witness.zig:150-174: start and bump begin the engine's read even when the snapshot can only be feature_excluded (--cortheon, --compact-at, --compact-memory), so those servers read every weight byte at ready and after each reload for nothing. Please skip it when excluded is set.
4. Device evidence for the path now that it reads known: on the A100 with the flag on, please show a resumed prompt equals a fresh one and drafted output equals "draft": false (both are on by default in your runs). --kv-cache fp8, --prompt-precision int8 and --rope-yarn also reach this path and read known; either run the same checks with them or have them read feature_excluded for now. The second is fine with us.

On your question: keep one worker and no setting. It's off the request path and bounded by the checkpoint, and your decode-during-hash rows are level. We'll run a GB10 check on our side once these are in.

• Synthena (Ash's Agent)

[kotevcode]

Thanks. All four are in 5043d29:

1. Witness.open takes the captured set's folder (aot.pick's result, so TENSORFOLD_CUDA_KERNELS or the share/tensorfold/cuda/sm<N> set beside the binary). It stamps aot.json and each cubins/<hash>.cubin it names, next to the shards, and unchanged() checks them all. The second check now runs after the kept components (which read the cubins) as well as the shards. cuda_witness_test.zig expects files_changed after the cubin rewrite, and a witness reopened over the new cubin gives a different digest.
2. cuda.zig calls Witness.openOrFailed. Only OutOfMemory fails the load. Any other stamping error logs one line and gives a witness that reads hash_failed and never reads anything. New test: a malformed aot.json reads hash_failed.
3. start and bump skip the engine's read when excluded is set. New test: the hook's begin count is 1 at ready and 2 after a reload on a plain server, and 0 and 0 with --cortheon.
4. --kv-cache other than bf16, --prompt-precision other than bf16 and --rope-yarn now read feature_excluded with the flag's name, and docs/api.md lists them.

One worker, no setting: kept. test-execution-witness-cuda-host 18/18, test-execution-witness-host 34/34, test-cuda-host, test-server-cpu, test-model-idle-host and test-disk-identity-host pass; pr_check --pr 2956: 0 errors, lean 345/345.

[ashhart]

Thanks @kotevcode, 5043d29 covers the code asks:

1. Witness.open stamps the captured set that aot.pick chose: aot.json and every cubin it names. unchanged() now runs again after the kept components have read those cubins, and the cubin test expects files_changed.
2. openOrFailed lets only OutOfMemory fail the load. A file it can't stamp logs one line and reads hash_failed, and the malformed aot.json test pins that.
3. start and bump skip the engine's read on an excluded server, and the begin counts (1, then 2 after a reload; 0 with --cortheon) show it.
4. --kv-cache, --prompt-precision and --rope-yarn read feature_excluded. All three default to bf16/unset, so a default server still takes the path.

The PR's own CI passed on this head. Merged onto today's main here, test-execution-witness-cuda-host and test-disk-identity-host pass. The only conflict was CHANGELOG.md.

Before it leaves draft:

1. Please restack it on server: --execution-witness serves a read-scoped snapshot of what runs, or why it is unknown #2901's current head, bd15bd6, and on native: the CUDA kept-state identity is written once through any hasher, its keys unchanged #2903's head. It still carries 3067c2b, which doesn't have the _TOKEN filter or its docs/api.md line.
2. The first half of ask 4 was device evidence on this path with the flag on: a resumed prompt equal to a fresh one, and drafted output equal to "draft": false. Your server: "execution_witness": true binds a reply to the witness at admission and at its end, with its prompt, resolved request and tokens #2958 receipt has both on 2B, run on this PR's previous head with server: "execution_witness": true binds a reply to the witness at admission and at its end, with its prompt, resolved request and tokens #2958's commits on top. The review commit here doesn't touch generation, so that's enough for us. Please link it from this description and update the checklist line that still says they weren't run.
3. Optional, your call: the malformed aot.json test passes, but its hash_failed warning goes to stderr, so zig build's summary lists test-execution-witness-cuda-host under "failed command" next to real failures. Keeping that warning out of the test's output would keep the summary clean.

Once it's restacked and out of draft, we'll run the GB10 check we promised.

• Synthena (Ash's Agent)

[kotevcode]

Restacked as asked, onto #2901's head bd15bd6 and #2903's current commits; it's still on #2901's base, not main. #2903's two commits (a7bbda5, 7795508) change the same lines, with the same messages, as the copies here, which are now 93282c1 and c1c1d28. The rebase had no conflicts, and the branch's diff over #2901 is unchanged.

I also took the stray warning out of the build summary (b7a99ed). The malformed aot.json test now lowers std.testing.log_level to .err around openOrFailed, as lane_activity_test and checkpoint_metal do. Production still logs the hash_failed warning, and test-execution-witness-cuda-host no longer shows up under "failed command".

The description now links the #2958 A100 receipt (https://gist.github.com/kotevcode/5b4d26cd284c1f04878862f4c1611b93) for resumed = fresh, and for drafted = "draft": false, with the flag on.

test-execution-witness-cuda-host 18/18, test-execution-witness-host 34/34, test-disk-identity-host 35/35, and test-cuda-host passes on the Mac host (6339 pass, 38 skip, across its 390 steps). fmt, git diff --check and pr_check are clean.

[ashhart]

Thanks @kotevcode. The restack is clean. #2903's two commits come over with the same changes, and #2901's _TOKEN filter is now included. The hash_failed warning is out of the build summary. The description now links the #2958 A100 receipt, which shows resumed equals fresh and drafted equals "draft": false with the flag on. No more asks from us.

As promised, we'll run the GB10 check on this head next: Qwen3.5-2B MLX 4-bit with --execution-witness, against main and against the flag off. We'll check for known on qwen3_5-2b/cuda/tp1/mlx-q4g64/text with a captured kernel set in place, files_changed after a shard is touched, equal token_sha256 at 1 and 8 streams, and level decode, including while the hash runs. We'll post the numbers here.

It lands after #2896, #2901 and #2903, in that order. GitHub won't start CI while the stack conflicts with main in CHANGELOG.md, so we'll resolve that when we land, as on #2901.

• Synthena (Ash's Agent)