Repository navigation
CUDA (GB10): Flash Next KV in the display carveout, opt-in - #2763
bertholomus wants to merge 5 commits into
Conversation
|
Thanks @bertholomus. Counting the carved planes into the room the streams are planned in, rather than
|
|
Thanks @ashhart. 1. You read it correctly. It is a fault in this head. The Nov 4 runs had 2. We will add a host test for the 3. Receipts on this head with both sources are not run yet. Noted that the five-stream cell waits for #2753. 4. We will rebase on #2762 once the shared opener settles there. This stays a draft.
|
|
Thanks @bertholomus for checking so quickly and for finding exactly what the rebase dropped. The plan sounds right:
A few small things for the next push:
|
|
Pushed the fix for the fault you found: 468b35e on this branch (one commit on f271b73, fast-forward).
Not done yet, as you asked: the rebase waits until @BadAd84's fold in #2762 settles. After it, the carve opener becomes a call to the shared opener in |
|
Thanks @bertholomus, that's the fault fixed. Two small things for the next push:
The rest is as agreed: receipts on the new head with both sources (the five-stream drafts-on cell after #2753), then the rebase onto #2762's fold with the opener moving to the shared one and the room helper to its own file. It stays a draft until then, which is fine.
|
|
Pushed 35dbd00 on this branch (fast-forward from 468b35e), for your two points:
One file changed. |
|
Thanks @bertholomus, both are in on
|
TF_CUDA_CARVEOUT stays 0 or 1. TF_CUDA_CARVEOUT_SOURCE picks where the display reservation comes from: drm, the default and the existing dumb-buffer route, or daemon, where a display-memory daemon lends it in whole 2 MiB slices over a unix socket. Which daemon a box runs is a setting, not a code path: TF_CUDA_CARVEOUT_SOCKET (default /run/dispram/dispram.sock, kindling spark-os's dispramd, whose README publishes the default key), and TF_CUDA_CARVEOUT_KEY, or TF_CUDA_CARVEOUT_KEY_FILE for a daemon whose key is per box: spark-attic's atticd keeps /etc/spark-attic/key and listens on /run/spark-attic/attic.sock. The client, cuda/carveout_lend.zig, was written from a specification of the daemon's observed behaviour, by an agent working in a copy of this tree that held no client for the protocol. The specification, the probe scripts and their output are in the thread of pull request 2762. An ask is rounded up to a whole slice before it is sent, and a grant smaller than the ask is refused rather than fitted. The reply's granted size is the authority, ok must be true (a refusal carries a size field too), and a seekable descriptor's own length is checked as well. The slice is mapped through arena.zig's Vmm. One pair of calls serves every caller. carveout.requested decides whether the carveout is on and reads its rows through the settings table, where TF_DRM_CARD gains a row. Carveout.from opens the source the request names and logs a refusal once. The CLI and Nemotron make those two calls and nothing else. The key file is read only for the daemon source, as its claim is made. A file that is missing, not a regular file, empty on its first line or 256 bytes long there is refused with TF_CUDA_CARVEOUT_KEY_FILE named, and neither the path nor the file's bytes are shown. The startup walk validates the carveout's rows when it is on and skips them when it is off, as it already did for TF_CUDA_CARVEOUT_MIB, and reads TF_DRM_CARD only while the carveout is on and the card is its source. Off Linux the claim returns NoDaemon before any of the protocol code is analysed, so the Windows host tests build. The client gives the daemon 10 seconds to answer, and a daemon's rows are read only while the daemon is the source, as the card's are only while the card is. Host tests, in carveout.zig and carveout_lend_test.zig (the second runs on Linux only): the request's rows and their defaults, the key file's accepted and refused forms, the key file left unopened for the drm source, and a fake daemon on a real SEQPACKET socket for a granted claim, the descriptor by SCM_RIGHTS, a refused claim, a refusal that carries a size, a malformed reply, a short grant, an ask between two slices, a missing descriptor, a missing socket, a daemon that never answers, a daemon that has gone away, the key file's line on the wire, and the daemon getting the memory back when the connection closes. zig/tests/fixtures/gradient-serving.tsv takes carveout.zig's new hash and a row for carveout_lend.zig. docs/recipes/cuda.md says what each source costs, measured on a GB10, and CHANGELOG.md gets its Unreleased bullet. Co-authored-by: bertholomus <311723197+bertholomus@users.noreply.github.com>
TF_CUDA_CARVEOUT=1 puts Flash Next's KV caches in the display carveout too: a contiguous cache holds it one at a time, as many whole planes of a full window as fit (cuda_carve.zig) - the keys and values of the first attention layers (both or neither), then indexer keys, then pooled keys. A shorter cache takes the same planes, each smaller, so it fits too and never needs more device memory than a full window. The DeltaNet states, the n-gram tail, the MTP head's cache and the pool's pages stay in device memory. The memory budget counts the planes out of device memory: Loaded.carved_bytes joins the room cuda_streams.zig counts the streams in (kv_pool.streamsRoom), which the grown streams, the shared page pool, a KV ledger's plan and the kept prompt cache are sized from. So pages or streams admitted grow by about the lent bytes. Nothing is counted when another cache may hold the carveout while the first stream runs: --slide's learner, a page test, or every stream on pages. The carveout opens before the checkpoint load finishes, so a refusal keeps the page cache as any other refusal does. The startup log names the planes, their bytes and the bytes counted. Flash Next makes the same two calls the CLI and Nemotron make: carveout.zig's requested(Settings) reads every row through the settings table, TF_DRM_CARD included, and Carveout.from opens the source the request names and logs a refusal once. The daemon's rows are read only while the daemon is the source, and the card's only while the card is. cuda_carve.zig holds no opener of its own. Opt-in and off by default: without TF_CUDA_CARVEOUT nothing changes. One rank: --tp 2 with the carveout is refused (FlashNextCarveoutUnsupported). Nemotron's placement and the card source are unchanged. Only the planes' addresses change, not the math. Host tests: the plane sizes against seqBytes; the plan at 2,044 MiB, at 1,792 MiB, at a short window and with no room; every cache size up to the window takes the window's planes from an empty carveout, in order; a cache's planes by part; and the pool gaining the carved planes' whole pages (135 pages of 512 tokens for 2,032 MiB at 30,784 bytes a token) while the prompt cache stays at its budget. Test fixtures that build an Engine field by field set carve = null. cuda_lanes.zig and cuda_prompt.zig stay under the 600-line limit: an unused import goes, the layer-map test moves to cuda_prompt_test.zig, and the engine closes the carveout. docs/recipes/cuda.md and flash-next-cuda.md describe it. Credits: urtho wrote cuda.Carveout and its spans (ashhart#487), which this builds on; coolbho3k had the idea of the display carveout as KV room; ezoushen and BadAd84 put display KV into the GLM recipe; Kindling's dispramd introduced the lending protocol atticd speaks; plotarmordev gave receipts on a second Spark; Mia's Qwen3.8-Flash-Next work, her checkpoint and settings, is what this is for; Ash's TensorFold.
…es, as newSeqOn does With TF_FLASHNEXT_LANE_PARTS at its default, a server stream's contiguous cache came from seq_parts.newSeqIn, which passed no carved planes to its caches (an empty list), while cuda_lanes_load.zig still counted the carveout's planes into the room: the plan could count up to 2 GiB that no stream uses. newSeqIn takes the carved planes again under the same rule as newSeqOn (the branch before an earlier rebase had it right), so every constructor of a contiguous cache either claims the planes or the plan does not count them. cuda_seq_parts_test.zig: with lane parts on, a lane stream's first planes are the carveout's (key/value/index/pooled of the plan's counts, borrowed, the carveout no longer whole), a second stream finds the carveout held and takes nothing, and with no carve plan nothing is borrowed. It fails on the commit before this one (the caches get no planes) and passes here. cuda_lanes.zig is untouched by this commit; cuda_launch.zig registers the new test file. The serving pin for cuda_launch.zig is re-pinned to what this commit ships.
…rry no commit hashes Ash's review: the --tp 2 refusal (carveout.open checks tp before it touches the context) had no test. One now calls open() with tp = 2 and the carveout requested through the environment, and expects FlashNextCarveoutUnsupported - with a context pointer the refusal path never reads. The test's name carries no commit hash: names and comments stay free of history.
streamsRoom() took cuda_kv_pool.zig past its 600-line limit once the carveout credited the streams' room (589 lines before it, 614 with it). It moves to cuda_kv_room.zig with its test, and cuda.zig calls it there. No behavior change: the same room and the same page gain.
35dbd00 to
5bad6de
Compare
|
Thanks @bertholomus for the rebase onto #2762's new head. This head (
Still open from our 11 Nov note:
New on this head:
CI hasn't run on this head yet. It stays a draft until the receipts are in, and it lands after #2762. Once the receipts are posted, we'll repeat what we can on a GB10 with Qwen3.8-Flash-Next.
|
What this changes
Before: with
TF_CUDA_CARVEOUT=1, Flash Next keeps all of its attention caches in device memory.After: one contiguous Flash Next cache holds the display carveout and keeps there as many whole planes of a full window as fit, and the memory budget counts those bytes out of device memory. Without
TF_CUDA_CARVEOUTnothing changes.0e8576ee6a02a24b54d945fd45dcd3b72208f2cd). This pull request adds one commit,f271b73470f965b00fd81446daf2a2f7a9e873d2. The diff againstmainalso shows the parent's files; they are not new scope here.atc overlap --json: run 2026-11-10T15:19:53Z with atc 0.9.4 againstmainat8a7838a65a1f.completeis false; limitations:otherPullRequestsLimit,pullFilesUnavailable,timeLimit. 31 overlaps with open pull requests: 8clean, 21unknown, 2conflicts(1 file each).myPullRequestsis empty. The 2 conflicts are with other open pull requests, not withmain, and are not resolved here: both commits cherry-pick cleanly ontomainat8a7838a65a1f. The result covers the whole stack (23 changed files); the parent alone had 18 overlaps and 0conflicts.Checks on this head
f271b73470f965b00fd81446daf2a2f7a9e873d2python3 tools/zig/lean_check.py, on the base and on this head: 386 on the base, 386 on this head, 0 new.zig build test-cuda-host --summary all: base 365/365 steps, 5951 of 5979 tests passed, 0 failed, 28 skipped; this head 365/365 steps, 5969 of 5997 tests passed, 0 failed, 28 skipped. Both logs carry 8failed command:lines beside test binaries' stderr output, the same number on the base; the build summary counts every step as succeeded.zig build test --summary all: base 647/647 steps, 8617 of 8684 passed, 0 failed, 67 skipped; this head 647/647 steps, 8635 of 8702 passed, 0 failed, 67 skipped (15failed command:lines in each log, as above).zig build test-golden --summary all: 5/5 steps on both.zig build install -Doptimize=debug, empty cache: exit 0 on the base and on this head.check-native-cuda-host,compile-checkpoint-consumers,compile-cuda-tests,compile-modelopt-startup,compile-config-server,compile-omni-video-host,compile-nemotron-cuda-locate-host,compile-flashnext-learn,compile-flashnext-mtp-parity-host,compile-cuda-contract-consumers,compile-dsv41-cuda,compile-moss-teacher-cuda.zig fmt --check build.zig zigandgit diff --checkagainst the base: clean.python3 tools/pr_check.py --base 33ada77c: 0 errors, 1 advisory warning (it asks for--pr NUMBERto check the Unreleased line).atticd; the--tp 2refusal with the carveout on; the budget arithmetic on real 2,044 MiB and 1,792 MiB carveouts. Metal, HIP, XPU and Windows: not built; the checks ran on a Linux host with CUDA host tests only.zig build dev-tools: macOS only, not run. The CHANGELOG Unreleased line is not in this commit yet: it needs the pull request number.Receipt
8db1ee1a), not on this head:an ASUS Ascent GX10 (GB10, SM 121) on stock DGX OS 7.6.0, kernel 7.0.0-1024-nvidia-64k, driver 580.178.04, spark-attic 0.1.0 lending 2,044 MiB.
Qwen3.8 Flash Next, bf16 KV, text only,
--context 262144 --parallel 5, three arms on one boot: off, attic, off again.KV pool 224,256 to 293,376 tokens (+135 pages of 512): 2,032 of the 2,044 MiB held the keys and values of 3 attention layers, the indexer keys of 7 and the pooled keys of 3.
TF_CUDA_CARVEOUT=1.--tp 2with the carveout is refused.environment.settingandenvironment.pathinplatform/environment.zig.33ada77c3122fca844a6825316ce8bbabaca05beand headf271b73470f965b00fd81446daf2a2f7a9e873d2, Zig 0.17.0, x86-64 Linux host with no GPU. The node run above was on8db1ee1aplus this stack.tools/bench_concurrent.py --alone --serialortoken_shaagainst"draft": false: not run on this head. On the earlier base, replies equalled the off arm for 8 of 8 prompts at temperature 0 and 3 of 3 seeded at 1.0 (text, finish reason,token_sha256), and every stream that finished made the same tokens in all three arms.tools/bench_openai.py: not run on this head. On the earlier base, tok/s attic against off and off again: 1 stream 45.1 against 45.6 and 44.9; 5 streams with drafts off 77.7 against 78.9 and 77.4. Five streams with drafts on is not compared: aProposalPaddingPickstream error ended 8 of 9 such runs off, 9 of 9 attic and 5 of 9 off again.tools/bench_cache.py --require-cold. On the earlier base, prefill tok/s attic against off and off again: 8K 2,097.8 against 1,929.0 and 2,085.1; 32K 2,061.0 against 2,060.8 and 2,048.0.Checklist
The title and description match the head commit, including the receipt and the stack.
New tests fail before the change, pass after it, and skip cleanly without their dependency.
Not ticked: the new tests sit in files and rows that the parent commit does not have, so the parent has nothing to fail. They pass on this head.
Comments and docstrings are one line. No measurements or history in the source.
No personal data, machine names, internal hosts or local paths in the diff or its commits. No AI attribution lines.
Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
Drafted output equals
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.No precision traded for speed. If the bits change, the description says which sums change and why.
Decode and prompt processing are no slower than main at one stream and the widest admitted; with
--drafter, no slower than without it, greedy and sampledNot ticked: measured on an earlier base, not on this head.