Repository navigation
Conversation
…r stalls other streams' rounds LFM2-MoE on Metal ran a new stream's whole prompt in one call on the lane thread, so every other stream's round waited for the whole prefill (#3021). The backend now declares prefill_step and cancel_prefill beside reports_fill: one call runs one chunk at exactly today's cuts (the family declares no chunk invariance, so the cuts never move), and decode rounds run between chunks. A drafting stream's sink gets each chunk's tapped rows.
This was referenced Nov 13, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
LFM2-MoE on Metal ran a new stream's whole prompt in one call on the lane thread, so with shared rounds every other stream's reply waited for the whole prefill. The backend now declares
prefill_stepandcancel_prefillbesidereports_fill, so the lane host runs one chunk a call and decode rounds run between chunks. This is the third family after Gemma 4 (#3025) and Nemotron (#3054).Same cuts.
prefill_steps.zig'schunkEndis main's loop cut moved out line for line: at mostoptions.chunkrows, cut at the first planned chunk start pastat, then at the next kept mark. LFM2-MoE declares no chunk invariance, so the host never caps a step and the cuts never move. The whole pass (prefill) and the stepped one share the same chunk body.Drafter taps. A drafting stream's sink gets each chunk's tapped rows, served from the latest chunk while no forward has run since; the last row's taps are kept on the last chunk as before.
Cancel between chunks keeps the state at the last chunk's end; no prompt work runs.
A stream that leaves mid-pass has its pass forgotten in
release.Issue: Refs Gemma 4 on Metal: a new prompt freezes every other stream's decode for its whole prefill (no prefill stepping) #3021.
Stacked on: main.
atc overlap --json: not run.Checks on this head
6496bef7f16e44ea3b3665c3238d53891004f272zig fmt --checkon the six files: passeszig build test-lfm2-moe-metal-steps-host check-metal-prefill-rest-host: 10/10 steps succeeded, 5/5 tests passed. The tests check that the stepped calls' chunks equal main's whole loop's, that a decode runs between chunks, that each chunk's taps reach the sink, and that a pass resumed from a kept state cuts as main does.zig build test-lfm2and every Metal device step. We'll run them on an M5 Mac before landing.Receipt
Checklist