Repository navigation
Conversation
A stream's words, a slot's descriptor and the rows its keep replays, one Plan a cache slot. No kernel launches yet.
…laying the kept rows A window's conv split, gates and delta rule read its rows from the plan, and the keep rebuilds the conv tail from the kept inputs and re-runs the scan over the kept rows, so the state is serial decode's while the round writes no snapshot a row.
The engine makes a plan a cache slot where the shapes are the plan kernels' and sends its words inside a round's graph, a window's linear layer goes through the plan, and a keep replays the kept rows where the plan took it and copies the snapshots where it did not. The plan's bytes count in a stream's admission.
… k, v, state and conv tail At Flash Next's shape: the plan's conv split against today's launch and the kept conv tail for one to four rows, the plan's delta rule y against serial and the state a keep replays for each kept row (a one-row window is its own keep; a longer one leaves the state as the round found it), and two streams each replayed from its own rows. Host: the words' layout, the kept sections' offsets, and the shape guard, which is the only choice now.
…paths The kernel tests now reach the family's model-free side through kernel_reach, so a conv split test can build a real plan and compare today's launch with the plan's q, k, v, gate and beta bit for bit.
… and keep the kernel-test wiring out of hip.zig
A long prompt keeps only its history, shared and rewind marks, so a later request that preserves a large prefix after compaction has no state inside that prefix and prefills from the last shared cut. Keep up to `intermediate_most` checkpoints at power-of-two multiples of the ladder's spacing under the history mark. They are admitted after the endpoint and rewind, inside the store's budget and slots, and are exempt from superseded eviction, so they stay across append turns while remaining evictable as ordinary oldest states. Flash Next on CUDA opts in at 8,192-token spacing with four checkpoints, behind `TF_PROMPT_INTERMEDIATE=1` until a served A/B shows the extra marks pay. Host tests cover the ladder plan, a compacted 160K-token prompt resuming at 64K against the 4K shared cut without it, an append turn, admission beside the endpoint, and eviction.
On one GB10 with nvidia/Qwen3.8-Flash-Next-NVFP4, --no-drafts and an 8 GiB cache, a 104K-token request that kept its first 100K resumed 65,536 tokens instead of 2 (TTFT 19.9 s against 50.7 s, same tokens). A cold 165K prompt paid 0.9 s more and the cache held two more states. The follow-up resumed 65,536 because the compacted turn had evicted the larger states.
prompt_intermediate used a copy of prompt_interior's same check from before the image-identity change, so it missed cache_images.matches. Make the interior check public and call it, state that any kept state at a rung position counts as intermediate, and test that passRules leaves the ladder off with TF_PROMPT_INTERMEDIATE unset or 0.
The lean rule caps the file at 600 lines and main is already there. Move the ladder's two values into cuda_snap.zig, expose them to the family through one alias, and drop one blank line between the weight-byte wrappers.
Since #2699 the victim order keeps each conversation's newest state before redundant copies. A ladder checkpoint was exempt from the superseded pick, so once the plain redundant copies were gone the oldest state went next, which could be another conversation's newest. A superseded checkpoint now goes after every other redundant copy and before any newest state. Under a reservation guard, fitting took the marks in position order, so the low ladder positions could take the room the rewind state needed. Ladder positions now fit last, after the endpoint and every other mark, in the room other passes' promises leave. Co-authored-by: javierbits <177527508+javierbits@users.noreply.github.com>
The switch is a CUDA-scope flag now, checked when the backend starts like the other prompt-cache switches, and documented. The test's env handle moves inside the test, since main's snap_tail test declares its own. Co-authored-by: javierbits <177527508+javierbits@users.noreply.github.com>
The rebase put it under 1.2.2. It now also says how its checkpoints are evicted and fitted. Co-authored-by: javierbits <177527508+javierbits@users.noreply.github.com>
The narrow tensor-unit default installs the existing greedy callback again. Sampled and shared rounds retain their lane path, and explicit lane overrides keep their meaning. Every copied token still goes through target verification. Refs #3013
Use the lane selector sibling policy for the default narrow callback, and require a confident suffix before appending copy rows. Explicit serial overrides retain their original head, admission and submissions. Keep dependent matching, taps and preparation of the same keep at the existing verify tail, preserving target operators, budgets and EOS. Refs #3013
…d cache, and #3000 for the numbers
…x kernels nothing launches q27_prompt_product, _silu, _decay, _qk, q27_prompt_conv and q27_prompt_recur were reached only through prompt_gpu.Kernels and prompt_state_gpu.Kernels, which nothing builds with pipelines. The host records in prompt_glue.zig, prompt_gpu.zig, prompt_state.zig and prompt_state_gpu.zig, the build and runtime source lists, the runtime analysis line and the prompt.metal write in tests/qwen27/run.zig go with them.
…see calls No family calls rotate(); the shared attention's runtime callers (Gemma 4 and Qwen27 DFlash) rope their rows elsewhere. Shape loses rotary_dim and rope, which only the kernel read, and lanes_see feeds raw K and Q straight to append and attention.
…passed to shared_attention; changelog line
…says dsv41_cuda.module pinned the family, kernel and runtime modules to Debug, and dsv41-assets built tf-dsv41-assets on it, so the AOT planner always ran unoptimized. Defilan measured one aot-needs enumeration at --context 524288 on a GB10: about 45 minutes as built, under 7 with the family modules at ReleaseSafe, planning the same 138 variants (#2270). module and runtime keep their Debug signatures for every other caller; moduleAt and runtimeAt take the mode, and dsv41-assets builds the tool on moduleAt(.ReleaseSafe), so its runtime and kernel modules go to ReleaseSafe too. ReleaseSafe keeps the safety checks, and no code under the tool reads the optimize mode, so the plan is unchanged. test-dsv41-assets-host stays on the Debug module. Refs #2270. Co-authored-by: Defilan <7034124+Defilan@users.noreply.github.com>
…nt blocks (relu^2 up, fp32 or bf16 down)
…and bf16 down, gates 1-5 with two controls
…, as pack does, with a test
…Apache-2.0, as its model card now says
Owner
Author
|
Restacking: the train this one was stacked on did not land as tested. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI for a landing train: #3065 #3007 #1630 #3084 stacked on main in that order. Nothing merges from this PR: each PR above lands on its own once this tip is green, and this PR is then closed.