Skip to content

lfm2_moe metal: prefill runs a chunk a call, so a new prompt no longer stalls other streams' rounds (#3021) - #3058

Open
ashhart wants to merge 2 commits into
mainfrom
lead/lfm2moe-metal-prefill-step
Open

ashhart wants to merge 2 commits into
mainfrom
lead/lfm2moe-metal-prefill-step

Conversation

@ashhart

@ashhart ashhart commented Nov 12, 2026

Copy link
Copy Markdown
Owner

What this changes

LFM2-MoE on Metal ran a new stream's whole prompt in one call on the lane thread, so with shared rounds every other stream's reply waited for the whole prefill. The backend now declares prefill_step and cancel_prefill beside reports_fill, so the lane host runs one chunk a call and decode rounds run between chunks. This is the third family after Gemma 4 (#3025) and Nemotron (#3054).

  • Same cuts. prefill_steps.zig's chunkEnd is main's loop cut moved out line for line: at most options.chunk rows, cut at the first planned chunk start past at, then at the next kept mark. LFM2-MoE declares no chunk invariance, so the host never caps a step and the cuts never move. The whole pass (prefill) and the stepped one share the same chunk body.

  • Drafter taps. A drafting stream's sink gets each chunk's tapped rows, served from the latest chunk while no forward has run since; the last row's taps are kept on the last chunk as before.

  • Cancel between chunks keeps the state at the last chunk's end; no prompt work runs.

  • A stream that leaves mid-pass has its pass forgotten in release.

  • Issue: Refs Gemma 4 on Metal: a new prompt freezes every other stream's decode for its whole prefill (no prefill stepping) #3021.

  • Stacked on: main.

  • atc overlap --json: not run.

Checks on this head

  • Head commit: 6496bef7f16e44ea3b3665c3238d53891004f272
  • Lean checker on the base and on this head: 345 and 345, 0 new.
  • Tests run:
    • zig fmt --check on the six files: passes
    • zig build test-lfm2-moe-metal-steps-host check-metal-prefill-rest-host: 10/10 steps succeeded, 5/5 tests passed. The tests check that the stepped calls' chunks equal main's whole loop's, that a decode runs between chunks, that each chunk's taps reach the sink, and that a pass resumed from a kept state cuts as main does.
  • CI: runs on this push.
  • Not run, and why: zig build test-lfm2 and every Metal device step. We'll run them on an M5 Mac before landing.

Receipt

  • Main against branch: owed. On an M5, one long prompt beside a decoding stream, greedy, with and without a drafter: both streams' replies must equal main's. The decoding stream's inter-token latency while a 32k-token prompt fills should stop stalling for the whole fill.
  • Paths and platforms: Metal only; CUDA's LFM2-MoE path is untouched.
  • Exactness: the cuts are main's (host test above); device token equality owed with the run.
  • Licence of any new weights or drafter: n/a.

Checklist

  • The title and description match the head commit.
  • New tests pass after the change. They use the new step driver, so they can't compile on main.
  • Comments and docstrings are one line.
  • No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.

…r stalls other streams' rounds

LFM2-MoE on Metal ran a new stream's whole prompt in one call on the lane thread, so every
other stream's round waited for the whole prefill (#3021). The backend now declares
prefill_step and cancel_prefill beside reports_fill: one call runs one chunk at exactly
today's cuts (the family declares no chunk invariance, so the cuts never move), and decode
rounds run between chunks. A drafting stream's sink gets each chunk's tapped rows.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant