Repository navigation
Conversation
|
Thanks @laudney. We've read the four commits on top of #2875. Every text user keeps today's path: the new schedule rule reduces to the old grid and page step, The CI runs on
|
C6 of ashhart#2444 makes reuse depend on more than the key: a request's image identities and frame, a namespace, its own chunk starts and the rows it prefills. The family's Query now carries them through kv_prompts (a core-side image identity, converted by the CUDA adapter) and GLM passes its stream's images. The store enforces none of them yet, so every store user is unchanged.
C6 of ashhart#2444 for the resume schedule: as in the copy store, a planned store cuts and resumes at a request's own chunk starts when it gives them and on its grid otherwise, so a grid point inside the request's chunk is never its resume point; the marks, keeps and both tree walks use one rule. A key of another length than its prompt now reuses and keeps nothing, and a keep past the rows a pass prefills is refused. Without starts every store user cuts where it did.
C6 of ashhart#2444 for content: a page matches only when its words, the words past it, the exact identities of the images over them, and its namespace and frame are all equal, as the copy store compares exact image identities beside its key. Two pictures under equal key words are two branches, text before them stays shared, and a split keeps an image on both sides. Eviction reads the same identity, so another namespace's longer prompt never counts as a conversation moving past a state. Text keys match as before.
A shared cut was recorded by its words alone, so a cut planned in one namespace or over one picture could flag another identity's state at the same words as shared, and keep it from its normal eviction order. The record now hashes the namespace, frame and image identities too (text in the default namespace hashes the same words as before). A test through the CUDA adapter shows a picture reaching the store: text pages shared, image pages not.
6b79b26 to
7d2aa5f
Compare
|
Rebased onto main 484437b now that #2875 is in (its commits landed patch-identical, so this is the same four commits on top): head 7d2aa5f. On this head: the store's tests (63) and the kv-pool checks (25) pass, |
|
Thanks @laudney, that is the run we wanted. We rebased our check onto today's main as well: the mbox applies cleanly, the store and kv-pool steps pass here too, and lean still adds nothing. No asks left. This can queue as it stands.
|
What this changes
P2 of #2444, its second step: reuse identity (C6) in the core radix store. A page matches only under the same identity as the copy store checks today, and every current store user stays byte-equal.
aheadwords past it{at, tokens, gh, gw, hash}of every image over its rows andaheadwords, and its namespace and frameframestays 0: GLM has noframe_key)Every text user passes no images, the default namespace and no starts, so its pages, marks and hits are as before. Namespace and frame are compared per node rather than kept in separate roots: no page is shared across them either way, and eviction and reclaim walk one tree.
atc overlap --json: run after the last rebase: the same as prompt radix: checkpoint parts per cache group, one hold a page, page-only stores (#2444 P2, C3) #2875 (conflicts with core: a bounded ladder of intermediate prompt-cache checkpoints #1953 and DeepSeek-V4.1-Flash on Metal: the EXL3 2.9 bpw lane #2373 inzig/src/core/prompt_cache.zig, where the stack only adds the optionalholdtoPages.VTable); clean with metal nemotron: TF_KV_SHARE keeps prompt states as core pool pages (stacked on #1837) #1850 and Prefill on a GB10, decode on a Mac: long prompts for Qwen3.8-27B handed over ConnectX (#1594, step 2b) #2648.Checks on this head
zig build test test-golden -Dhip=true: 7344 passed, 43 skipped, 2 failed:cuda_disk_testandlane_host_rows_test's close, the disk-timing tests that fail on main too on the same busy host;zig build test-kv-pool: 25 passed; the store's tests: 62 passed;zig fmt --check build.zig zigandgit diff --check: clean.Receipt
TF_QWEN_KV_SHARE)TF_KV_SHARE), TP=2TF_KV_SHARE)TF_KV_SHARE)GLM at TP=2 with
--vision, one long shared text and then picture A, picture B at the same position, A again: main and this branch give the same tokens and cached counts, 0 / 6144 / 8192 (B reuses the text before the picture; A again resumes past it and replies as the first A).TF_KV_SHAREfamilies, Qwen 3.5/3.6 on CUDA, Qwen 3.5 and Nemotron on HIP).donelines, resumed ttft 0.131 / 0.132 s (Nemotron HIP) and 0.109 / 0.107 s (Qwen 3.5 HIP), decode 226.6 / 225.1 and 82.4 / 82.4 tok/s, one run each. The change adds compares at a page edge, nothing per token.Host: the store's tests (62) and the kv-pool checks (24): two pictures under one key, an image at a page edge with a lookahead word, splits across and at an image, two namespaces, a request start off the grid and a grid point inside its chunk, a key of another length, a keep past the prefilled rows, a page-only store on a request's schedule, eviction across namespaces, a shared cut in another namespace, and a picture through the CUDA adapter. Planted removals (request starts ignored, images ignored, namespace ignored in eviction) each fail their tests.
Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (Resumed equals fresh, above; the rest not run.)