Repository navigation
Conversation
…tate (#3133) A prompt resumed from a kept state never splices the rows of images before its resume point, yet every image went through the embedder. encodeImages now takes the pass's start and encodes only images whose rows end past it; the buffer keeps its layout for every image, so each chunk's splice offsets are unchanged. A resume never cuts an image (resumeAt refuses one), so an image is either wholly kept or encoded. The paged opening already knows its start before encoding. The legacy opening now restores first, then encodes. The image choice and each chunk's splices move to cuda_image_rows.zig, which both paths call, with a host test: a counting fake encoder sees one image of three on a pass resumed past two, and every chunk from the resume splices rows equal to a cold pass's.
…te holds (#3133) An image prompt resumed from a kept state went through the vision tower whole before the restore, though rows before the resume are never spliced. The opening now restores first and passes the resume point to Media.init. The tower's runs are formed over every image exactly as before (whole images, as many as one call's patches hold), and only a run that ends at or before the resume is skipped. Every run that is encoded keeps a cold pass's members and patch count, so its matrix multiplies keep their shapes (cuBLASLt picks the algorithm by shape) and their bits. A resume inside an image keeps that image's whole run. The positions and rope_delta are still laid out from every image, and a failed restore encodes everything, as before. The run loop moves to vision_prompt.runs with a host test: a counting fake tower sees one run of two on a prompt resumed past the first run, both runs on one resumed inside it, and every row a chunk from the resume splices equals a cold pass's.
…ast its kept state through the tower (#3133) On one rank, a prompt resumed from a kept state now restores first and sends only the images whose rows end past the resume through the tower. The image plan (splice rows, M-RoPE positions, delta) is still built from every image, and the tower takes one image per call, so the encoded images' rows are a cold pass's. A failed restore leaves the start at 0, and every image is encoded, as before. The restore never reads the media, and Draft.reset leaves its view alone, so the head's view is set after it. Tensor-parallel ranks keep today's order. Rank zero broadcasts every image's rows before the restore, because a follower takes them only while its sequence is unfed (cuda_rank_media.receive refuses once fed != 0, and a restore sets fed). Restoring first would desync the followers, so they encode every image as now. The restore moves into resumeKept, which both orders call. shared.media.Media.init takes the start; its callers pass it (tests 0). The family's own image-lane test (the real open, fake driver) now expects no tower call for a prefix kept past its image and one for a prefix kept before or inside it; main runs the tower in all three. A further host test with the fake driver: a counting tower sees one image of two on a prompt resumed past the first, from its own pixels, at the cold pass's feature row, with the same plan.
ashhart
force-pushed
the
lead/cuda-encoder-skip-cached
branch
from
November 14, 2026 03:51
c98f4b1 to
a64d0f1
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3133.
On CUDA, a request that resumes from a cached prefix ran the vision encoder over every image in the prompt, then spliced in only the rows past the resume point. Now only the images the restored state doesn't hold are encoded:
resumeAtalready refuses a kept state inside one.delta) are still built from all images.--tp 2it keeps main's order (every image encoded and broadcast before the restore), so followers can't fall out of step.A failed restore starts at 0 and encodes everything, as before. One visible change: a malformed image placeholder is now refused after the restore instead of before. It still fails before any reply, and the cache is given back.
Tests:
test-cuda-host393/393 steps. The Flash Next lane test now expects 0 tower calls for a prefix kept past the image, and 1 for before or inside it; main fails it (1 call). The new Gemma 4 and Qwen3.5 helper tests don't compile on main.compile-cuda-testsandcheck-native-cuda-hostpass for x86_64 and aarch64 Linux.Device check pending (GB10, against main): Gemma 4 (paged and legacy openings), Qwen3.5 with 3+ images (two tower runs), and Flash Next on one rank and at
--tp 2. Turn 1 with 2+ images, then a resumed turn 2, a turn adding an image after a kept prefix, and for Qwen3.5 a resume inside an image. Token hashes, greedy and seeded, drafted and not, must equal main's and a fresh server's cold turn 2, withcached_tokensequal. Turn-2 TTFT should drop, with turn 1 level.