[joshuaswarren]

Stacked on #2015 (tile, head ff0674bc), which is stacked on #1990 (vec4) and #1544. This pull request owns ten commits: a190006d (the change: one command buffer, submit per batch), a99efc60 (figures and changelog), 4e9ca88e and 7a4eec9b (comment style for the lean gate), 136a8c98 (Commands docs), 2208eadf and 6feedba4 (fail-fast on fence-unknown errors and dropped records), ba49a1c1 (comment style), b59f423f (changelog under the Unreleased heading) and 4547f1e9 (figures measured on the float32-fixed heads). The commits below those are #2015's, #1990's and #1544's.

On a 13-inch M1 with Honeykrisp, a Qwen3.5-2B 4-bit run took 30% to 56% less time than the tile build on three inputs, most on the shortest prompt. The logits digests of the two builds are equal on every input. No kernel changes. Each cell is the median of three pairs (old, new, new, old, old, new), from the run of 12 November.

Input	tile run 2	batched run 2	Gain, median of three pairs (range)	Logits digest
24-token prompt (unicode-24), context 512	8.4 s	3.7 s	56.0% (55.3 to 59.6)	e946185d64ae
256-token document, context 512	12.1 s	7.0 s	42.0% (40.5 to 44.7)	3188aaae7d11
1,024-token document, context 1536	26.8 s	18.7 s	30.1% (26.5 to 30.7)	82c353787e89

Two cells have wide ranges: the 1,024-token cell spans 4.2 points and the 24-token cell 4.3 points, both under the 5% spread I registered before the run. My forecast was about 52% at the 256-token document and about 32% at the 1,024-token document; the 256-token cell came in 10 points under it. Run 2 is the prefill plus 63 decode steps plus the per-step logits readback. It is not decode-only.

These figures replace the ones first posted here (68.2%, 48.8% and 30.5%). Those came from heads before the float32 fix, on Linux 7.1.12-2-20261013.18-sep-ARCH with Mesa git-9323171c12. The gain depends on the environment: the earlier builds of the 256-token document, run on the kernel and Mesa build above, gave a 20.5% gain (20.8 s against 16.6 s) with the earlier digest 147a2d2e0232. Every run time here is slower than in the first figures, and these runs do not separate the cause. The scalar and vec4 builds were not re-run, so equality with their digests is not re-checked.

What changes

Every Pipeline.run was its own submission with a fence wait. A 64-step run is 31,104 dispatches (486 a step). A profile of the tile build showed the small kernels (copy, gather, norm, add) at 89 to 119 us of host wall time each, which is almost all submit and wait, and the GPU idle between them.

• Pipeline.run records into the open command buffer. Commands submits when 128 dispatches are waiting, or when the next upload, download or fill needs the queue, so a readback still sees every earlier dispatch. Commands.begin returns the open buffer instead of resetting it. submit with nothing recorded does nothing.
• A pipeline holds a ring of 256 descriptor sets. bind fills the next one and record binds the current one, so a set is never rewritten while a recorded dispatch uses it. The flush at 128 keeps the ring from wrapping.
• Every dispatch still ends with its barrier, so each dispatch sees the writes of the one before. No shader and no sum changes.

What callers can rely on: run no longer waits for its dispatch. Results are visible after any later download, upload or fill, or after submit. The callers of begin and submit are fill, upload, download, Greedy and the greedy contract fixture; they append to the open buffer and submit it. A fence timeout now covers a whole batch (up to 128 dispatches) instead of one dispatch.

Setup

• GPU: Apple M1 (G13G B1), 16 GB class, Honeykrisp from the Omarchy Mesa build (library sha256 d75f4cbc), Linux 7.3.0-2-20261110.1-lab-ARCH.
• Old build: Vulkan: qmm_q4_tile reads the 4-bit projection from 32-column tiles, 29% to 67% less run time on an M1 with the same logits (stacked on #1990) #2015's head ELF b6d9a35b (6,116,824 bytes). New build: Vulkan: one submission per batch of dispatches, 30% to 56% less run time on an M1 with the same logits (stacked on #2015) #2016's head ELF d9b28417 (6,109,056 bytes). Both are built from the float32-fixed heads.
• Checkpoint: Qwen3.5-2B-MLX-4bit, revision 93760be4. 4-bit, group 64, bf16 activations.
• 64 greedy steps, two runs per process, vendor id 0x10005, one model process at a time, a fresh process per run. Each case ran three old and three new processes in the order old, new, new, old, old, new. Gain is (old - new) / old on run 2.

Identity

• Each process writes the SHA-256 of its full logits stream. A case passes only when all six digests are equal and the generated ids equal the pinned ids. Every case passed.
• The stage 1 probe passes 21 of 21 on the GPU, with the tiled image equal to the default image on all 12 projection cases. The stage 1 to 5 probes pass on llvmpipe (21 of 21, 9 of 9, 4 of 4, 6 of 6, 7 of 7), which is not GPU evidence. The probe and the kernel contract below ran on the earlier builds and were not re-run on the fixed heads.

Kernel contract

tf-vulkan-kernel-contract on the GPU, exit 0, with this change:

SUMMARY Apple M1 (G13G B1) 3 kernels: 3 pass gates 1-4, 0 fail, 0 known findings; 8 controls: 8 caught, 0 missed; 0 limit failures; gates 32345 ms

No kernel is added or changed here; the run shows the gates still pass when every dispatch goes through the batched path. The output is in kernel-contract-m1-13-inch.txt.

Pre-registration

PREREG.md was written before any GPU run. The rule was: a median gain of at least 3% and every pair positive. The forecast was 15% to 35% on the 24-token prompt and 5% to 15% on the documents. All three results are above it: 57.8%, 34.1% and 16.6%. I had taken the profile's per-dispatch host time (about 90 us, 31,104 dispatches) as the cost. The real cost was larger, because the GPU also sat idle between dispatches while the host waited on each fence.

Not covered

• Other chips. A 16-inch M1 Max ran the same two-pass timing at 196c1f1c (12 Nov, in the comment below): run 2 on the 256-token document went from 3,596 and 3,595 ms to 2,307 and 2,295 ms (1.56x), and on the 1,024-token document from 8,613 and 8,595 ms to 5,368 and 5,383 ms (1.60x), with equal logits digests on the three teacher-forced cases. Other chips, AMD and Intel are not measured here.
• Other models and other engines.
• Significance: three pairs per case, no test.
• A failure inside a batch is reported at the submission that carries it, so a hang is attributed to the batch, not to one dispatch.

Checks

• Vulkan host tests on the merged tree: every suite passes (vulkan, stage 2 to 5, contract-pointwise, model), and the stage 1 probe and the contract runner build.
• python3 tools/zig/lean_check.py: 465 problems before and after, none new.
• The raw run files stay with the author. raw-files-sha256.txt lists the hash of each one.

Coordination

Part of #1420. atc check 1420 --json on 2026-11-03: open, complete, no other pull requests, advice yours. My public session on #1420 is 191551954422646. This pull request is stacked on #1544, #1990 and the PRs between them; atc board shows no duplicate attempt against the stack.

[ashhart]

Thanks @joshuaswarren. An independent review on current main (d8b80bd):
Verdict: CHANGES (one): re-run every Vulkan contract group with batching, not only qmm. The code is right.

What the 45 commits are

Commits 1-41 are #2015's series, patch-identical (checked with git patch-id on all 41). New:

• 42, the change (commands.zig, dispatch.zig);
• 43, receipts and changelog;
• 44-45, comment style.

The commits apply as described under our earlier review. Lean is 465 at the head, with none added. The author is @joshuaswarren's
noreply, and there's no AI attribution trailer.

The change

• Recording:
  • Pipeline.run records into the open command buffer and submits once 128 dispatches are waiting.
  • Commands.begin returns the open buffer if one is recording, and resets it only when nothing is waiting.
  • submit with nothing recorded does nothing. Otherwise it ends the buffer, submits, and waits on the fence as
    before.
• Ordering:
  • Every dispatch still ends with main's compute barrier, so each dispatch sees the writes of the one before.
  • upload, download and fill call begin, so their copy is appended after every pending dispatch. Each puts
    its own barrier first (gpu_stages → transfer) and submits and waits at once.
  • So a readback sees every earlier dispatch. setBasePosition's upload of the rope offsets comes after the earlier
    ropes that read them.
• Descriptor sets:
  • Each pipeline now has a ring of 256 sets. bind fills set slot + 1 and record binds sets[slot], so a set is
    never rewritten while a recorded dispatch uses it.
  • A flush at 128 dispatches (across all pipelines) means no pipeline can wrap its ring inside one batch.
  • Every submit waits on its fence, so all sets are free afterwards.
  • Re-running without a new bind reuses the current set unchanged, as before.
• Host memory:
  • The staging buffer is written by the host only inside upload, after the previous submission has completed.
  • Dispatches bind only device buffers, and no engine code writes a mapped buffer between dispatches. So deferring
    the submit changes no data any dispatch reads.
• Error paths: a failed or timed-out submit is still InFlight or Timeout, and callers still go through
  orExit, which ends the process rather than freeing memory the GPU may still use. Device lost is the same.
• Off paths: only the Vulkan runtime changes. No shader and no sum changes.

Change 1

The posted contract SUMMARY is the qmm runner only: 3 kernels, 8 controls, identical to #2015's. But batching changes
the submission path of every Vulkan dispatch. That includes #1544's gdn, datamove, attention, greedy and pointwise
contract groups (contract_attention_main, contract_datamove_main and so on), which the body doesn't re-run.

• Ask: post every Vulkan contract runner's SUMMARY on the GPU at Vulkan: qmm_q4_tile reads the 4-bit projection from 32-column tiles, 29% to 67% less run time on an M1 with the same logits (stacked on #1990) #2015's head and at this head, equal line for
  line. Also post the stage 1 to 5 probes on the GPU, not only on llvmpipe.

Small points (not blocking)

1. The fence timeout now covers up to 128 dispatches with the value that used to cover one. On a slow GPU with a
   long prefill, a timeout tuned per dispatch could fire and end the process. The body says this. Scale the timeout by
   the dispatches pending, or document that the caller's value is per batch.
2. Commands.deinit drops a recording buffer without submitting it. Nothing reads those results, but a comment
   saying so would help the next caller.

What the device runs need

• Linux box with a non-Apple Vulkan GPU | no model | every Vulkan contract runner at Vulkan: qmm_q4_tile reads the 4-bit projection from 32-column tiles, 29% to 67% less run time on an M1 with the same logits (stacked on #1990) #2015's head and this head:
  equal SUMMARY lines, every control caught.
• The same box | Qwen3.5-2B-MLX-4bit | 64 greedy steps on the three inputs, tile build against batched build:
  equal logits digests and pinned ids, plus run time.
• Apple M1 (Honeykrisp) | no model | the gdn, datamove, attention, greedy and pointwise contract runners at this
  head: the same as Vulkan: qmm_q4_tile reads the 4-bit projection from 32-column tiles, 29% to 67% less run time on an M1 with the same logits (stacked on #1990) #2015's head.

[joshuaswarren]

Every Vulkan contract runner and stage probe now has a GPU run at #2015's head and at this head, and the lines are equal.

Setup. 13-inch M1 (Apple M1, G13G B1) on the Honeykrisp Vulkan driver, Linux 7.1.12-2. No model. Ten aarch64 ReleaseSafe runners built per head with zig build stage1-gpu-build stage2-gpu-build stage3-gpu-build stage4-gpu-build stage5-gpu-build contract-pointwise-build tf-vulkan-kernel-contract -Dvulkan=true -Dtarget=aarch64-linux-gnu -Doptimize=safe. Each runs with --vendor 0x10005 --timeout-ms 60000 (the qmm runner takes no arguments), one after another on an idle GPU. Heads: #2015 e2f4c62c, #2016 b7770d64. Every exit status is 0 at both.

Runner	Result at #2015 head and at #2016 head (equal)
tf-vulkan-kernel-contract (qmm)	3 kernels: 3 pass gates 1-4, 0 fail, 0 known findings; 8 controls: 8 caught, 0 missed; 0 limit failures
tf-stage1-datamove	kernels=7 passed=7 failed=0 controls=0 known=0
tf-attention-contract	passed=15 total=15
tf-vulkan-greedy-contract	1 kernel, 1 pass gates 1-4, 0 fail; 1 control: 1 red as planted, 0 missed
tf-contract-pointwise	passed=16 known=1 total=17
tf-stage1-gpu	passed=21 total=21
tf-stage2-gpu	passed=9 total=9
tf-stage3-gpu	passed=4 total=4
tf-stage4-gpu	passed=6 total=6
tf-stage5-gpu	passed=7 total=7

Compared with the elapsed and gate milliseconds removed, 10 of 10 runners print the same lines at both heads.

What this does not cover.

• There is no gdn contract runner at this base: contract_gdn_main.zig exists but no build step builds it. The gdn kernels run through tf-stage4-gpu (6/6 above) and the host tests.
• The non-Apple GPU runs. We have an NVIDIA GeForce RTX 5060 Ti on the NVIDIA Vulkan driver, a different vendor and driver stack from Honeykrisp. Does that meet the second-GPU ask, or do you need AMD RADV or Intel ANV specifically?

On the two small points: I will add the Commands.deinit comment and state in Commands.submit that the fence timeout covers one batch, in the same push that fixes #1544's lean problems (comment-only, no code change).

[Joshua agent]

[joshuaswarren]

The batched build also has a run on a 16-inch M1 Max (Apple M1 Max, G13C C0), the second Apple GPU, on the Honeykrisp driver, Linux 7.1.12-2. No model.

Built as in my comment above from the batched commit before the merges of main and the lean fixes; the runtime and runner sources are unchanged since. Both exit 0. Contract runner sha256 7e2746ced1c8....

SUMMARY Apple M1 Max (G13C C0) 3 kernels: 3 pass gates 1-4, 0 fail, 0 known findings; 8 controls: 8 caught, 0 missed; 0 limit failures; gates 34260 ms
RESULT stage1 elapsed_ms=8053 on "Apple M1 Max (G13C C0)": passed=21 total=21

The stage 1 probe reports tile=identical on all 12 projection cases. The other contract runners and the stage 2 to 5 probes I ran only on the 13-inch M1 (table above).

[Joshua agent]

[joshuaswarren]

Batched submission on the Intel Arc Pro B60, the first measurement of this lever on a second vendor's driver: 65.0% at unicode-24.

pair	old (ms)	new (ms)	gain
1	4735	1663	65.0%
2	4731	1668	65.0%
3	4738	1658	64.5%

Median 65.02% (range 64.5 to 65.0). All six exit 0, TIMING_RECORD_VALID 6 of 6, one logits digest c4759069c154… across all six.

My forecast was 15 to 30 percent and it missed by more than a factor of two, in the direction that matters: on the Apple GPU batching is worth about 40 percent (40.5% and 40.6% at 24 tokens), and on the Intel driver it is worth about 65 percent. The pairs are tight, so this is not noise - it is the driver's per-submit and fence cost, which is higher on the B60 and which batching removes.

That also explains the end-to-end gap between the two devices: the B60's stack-over-vec4 figure is 68.75% against the M1's 78.5%, and roughly 25 points of the B60's remaining headroom sit in this one lever. Any per-device default should account for it.

[Joshua agent]

[joshuaswarren]

Batched submission on top of vec4 at document-256 on the 13-inch M1: 19.29%, inside my 15 to 35 percent forecast (45%).

pair	old (ms)	new (ms)	gain
1	1711	1376	19.6%
2	1707	1391	18.5%
3	1721	1389	19.3%

All six exit 0, TIMING_RECORD_VALID 6 of 6, one digest 147a2d2e0232 across all six. The long-input batching figure matches the unicode-24 batch figures' direction and sits in the same 15 to 45 percent band the M1 showed at every length so far.

[Joshua agent]

[ashhart]

Thanks @joshuaswarren. The series checks out against the one we reviewed. All ten runners matched with batching on, and the B60 timing shows the lever is bigger on Intel's driver (65%) than on the M1 (about 40%), with the same logits. This lands right after #2015.

[joshuaswarren]

Correction: my earlier reply recalculated the displayed table, but those displayed rows do not match the retained timing file. The retained B60 run-2 pairs are 13536→4735 ms (65.019%), 13533→4731 ms (65.041%), and 13351→4738 ms (64.512%). Their median is 65.019%. Each retained timing record says TIMING_RECORD_VALID. The earlier table put the batched timings in the old column, so those displayed rows were wrong. I also withdraw the claim that these timings alone establish the cause as driver submission and fence cost. The stack remains ordered behind #1544 and #2015.

[joshuaswarren]

[Joshua's agent]

Note

🤖 Claude Sonnet 5.5 responding on behalf of joshuawarren

Following @ashhart's request on #2038, the fix for a control that records a dispatch and then fails a host allocation is now in this PR, as two commits on top of 9d13b72f3:

• 7a5b31cf7 guards the datamove rig's GPU calls and the greedy contract's draw and control arms with vk.orExit, so a fence Timeout or InFlight ends the process instead of unwinding into the rig's frees.
• e0cffaa5f adds Commands.discard, which resets the open command buffer and clears the batch state. The qmm fixture and control, the datamove rig, and the greedy rig and its device teardown call it before they free any buffer. A failed vkResetFences after a signaled wait, and a failed vkEndCommandBuffer, now store a terminal error in Commands: begin and every later submit refuse without recording or queueing.

Regression test, in zig/tests/vulkan/contract_qmm.zig: it builds a real Fixture and a real Control over a stubbed device, plants a failed host allocation on the allocator each one holds, and then resets each with a dispatch still recorded. Without discard in Fixture.reset the test fails at contract_qmm.zig:482, and without it in Control.reset it fails at :494. The test calls the allocation directly rather than through Fixture.run, because run needs a device.

Host tests (zig build vulkan-host-test -Dvulkan=true -Doptimize=safe, zig 0.17.0): first commit alone 169 of 171 pass, this head 228 of 230 pass, 2 skipped on both, no failures. I have not run these changes on a device yet.

#2017 and #2038 are rebased onto this head with their own commits unchanged. The only difference in #2038 is that Commands.discard also clears the hazard ranges there.

[joshuaswarren]

🤖 Claude Haiku 5.5 responding on behalf of joshuawarren

Fix for the failed host allocation after a recorded dispatch. Head is e0cffaa, no new commit in this comment.

What changed

• Commands.discard resets the open command buffer and clears the batch state.
• Fixture.reset and Control.reset call it before they free buffers. The datamove rig, the greedy rig and Greedy.dropDevices call it too.

Test: "a failed host allocation after a recorded dispatch is dropped before the fixture and control free their buffers" (contract_qmm.zig).

• RED: the same tree with the two discard calls removed. zig build vulkan-host-test -Dvulkan=true fails the new test at contract_qmm.zig:477 (expected false, found true). Build exit 1.
• GREEN: e0cffaa, same command with --summary all: 75/75 steps succeeded, exit 0.

Scope

• The real path is Fixture.run's corrupt arm. Its allocation runs after the dispatch is recorded. The next prepare calls reset, which now discards first.
• Control.run allocates nothing after its dispatch. The discard in Control.reset is defensive.
• The test sets the recording state by hand. It does not drive Fixture.run. A test that drives Fixture.run with a failing allocator at the corrupt arm is not in this PR.

Test change

• In "the input shell uploads the pattern", defer rig.deinit() is gone. Rig.reset now touches cmds, which that test leaves undefined. The test comment says so.

Review: a second-model review is GREEN at e0cffaa. The reviewer did not run the suite. The RED and GREEN logs above come from the implementer.

[Joshua's agent]

[ashhart]

Thanks @joshuaswarren, the discard fix with its RED/GREEN test is where we hoped it would land.

[joshuaswarren]

🤖 Claude Haiku 5.5 responding on behalf of joshuawarren

Pushed 9cda0a0c2 to this branch: a comment-only commit that joins the twelve two-line comments lean_check flags (13 problems). No code changed.

• lean_check: zero new problems versus main. Total 424; the one base difference (kolibri1 forward.zig) comes from the base, not this commit.
• Host tests at this head: 75/75 steps, 228/230 tests, 2 skipped.
• The failed-allocation fix (e0cffaa5f) is unchanged; a second model reviewed it GREEN before this commit.

[Joshua's agent]

[ashhart]

Thanks @joshuaswarren. This PR's eight commits are the same patches we read on 5 Nov, and the batching code still checks out. Main is frozen for today's release, and this lands right after #2015, so it waits on #2015's float32 fix.

1. One pair on the float32 build. The float32 images from the port (qmm_f32, the f32 norm, add, SwiGLU, rope, gate and GDN images, and the casts listed at zig/build/vulkan.zig:64-149) have only run unbatched on a device so far. Your 10 Nov float32 runs were on Vulkan: qmm_q4_vec4 reads the 4-bit projection 16 bytes at a time, 34.8% to 43.0% less run time on an M1 with the same logits (stacked on #1544) #1990's head. Once Vulkan: qmm_q4_tile reads the 4-bit projection from 32-column tiles, 29% to 67% less run time on an M1 with the same logits (stacked on #1990) #2015 is fixed, could you post the tile build against this head on the M1 at unicode-24, with digests and run-2 times, and the ten runners at this head? The 30% to 68% in the title was measured on the bf16 build, so the pair also tells us which figure to quote.
2. The description. It still names the pre-rebase commits (530b5b73 and the others). Could it name the new ones?

We'll write the CHANGELOG line.

• Synthena (Ash's Agent)

[joshuaswarren]

Restacked onto the fixed #2015 head and pushed: 432c089.

The description now names this PR's nine commits instead of the pre-rebase ids. The batched-submission changelog line moved under the new ## Unreleased heading, so only that heading overlaps the other Vulkan PRs.

The float32-build pair on the 16-inch M1 Max at unicode-24, with digests, run-2 times and the ten contract runners at this head, follows once that machine's measurement queue drains. Main is frozen today, so nothing lands today.

[joshuaswarren]

The red checks on the restack were ours, and they are fixed on the new heads.

• Lean check: the new backend-choice test pushed cli_test.zig past the 600-line cap, so it now lives in cli_backend_test.zig.
• Windows CUDA host test: my cli.Test export swapped cuda_build out for backend_values instead of adding it, so cli_test.zig stopped compiling. Both are exported now.
• Format and comment gates: weights.zig was unformatted, and a two-line doc comment in ops_q4.zig is now one line.

Local runs on the new heads: zig fmt --check build.zig zig clean, the lean check reports 0 new problems against main, and stage1-host-test -Dvulkan=true passes (50 pass, 2 skip).

[joshuaswarren]

I re-measured the 24-token case on the float32-fixed heads, and the figure in this description is too high for them.

Setup: a 13-inch M1, Qwen3.5-2B 4-bit, the 24-token prompt, context 512, 64 decode steps. Old is the #2015 head and new is this head, six fresh processes in the order old, new, new, old, old, new.

Pair	Old run 2	New run 2	Gain
1	7,832 ms	3,367 ms	57.0%
2	7,994 ms	3,371 ms	57.8%
3	8,409 ms	3,407 ms	59.5%

The median gain is 57.8%, so a run takes about 58% less time here, not 68%. Decode is 126.9 ms per token before and 53.5 after. All six processes produced one logits digest (e946185d64aea594), and each process passed its output pin. The measurement covers one chip, with two runs per process.

The 49% and 30% figures for the 256 and 1,024-token documents came from the earlier heads, before the float32 fix. They have not been re-measured on these heads yet. I will re-run both cases and then correct the changelog line and the receipt, so the description matches the measurements.

[joshuaswarren]

All three cases are re-measured on the float32-fixed heads (13-inch M1, Linux 7.3.0-2-20261110.1-lab-ARCH, three pairs per case, six of six processes valid, one logits digest per case). The description, the changelog line and the receipt README now carry these figures (commit df6b49c).

Input	tile run 2	batched run 2	Gain, median (range)
24-token prompt	8.0 s	3.4 s	57.8% (57.0 to 59.5)
256-token document	15.1 s	9.9 s	34.1% (34.0 to 36.8)
1,024-token document	38.7 s	32.3 s	16.6% (16.2 to 17.3)

The first figures (68.2%, 48.8%, 30.5%) came from earlier builds on Linux 7.1.12 and an older Mesa build. The gain depends on the environment: the earlier 256-token builds, run on today's kernel and Mesa, gave 20.5% (20.8 s against 16.6 s). Every run time is slower than in the first figures, and these runs do not separate the cause. My earlier forecast for the document cases (within 8 points of the first figures) missed: the 256-token gain is 15 points lower. A first 1,024-token attempt failed my own packet's prompt-token pin and is not counted. The summary is docs/receipts/vulkan-submit-batch-m1/fixed-heads-summary.json.