Skip to content

CUDA on GB10: Qwen's integrated weight upload reserves its pinned slot once, with load and device-weight receipts - #2356

Closed
ashhart wants to merge 3 commits into
mainfrom
lead/cuda-integrated-upload
Closed

ashhart wants to merge 3 commits into
mainfrom
lead/cuda-integrated-upload

Conversation

@ashhart

@ashhart ashhart commented Nov 6, 2026

Copy link
Copy Markdown
Owner

Part of #2043. On integrated GPUs (GB10), Qwen's weight upload reader now reserves its whole aligned pinned slot once, when it opens. Before, the slot could be allocated again as tensor pieces grew. Completion waits and the quarantine of failed copies stay as they were. Discrete-GPU paths and Nemotron's slot ownership are unchanged.

Three commits

  1. Two measuring tools: an isolated load probe, and a digest of the final device weights.
  2. The load probe's context scoped to Nemotron. Build this commit as the baseline.
  3. The upload change. Build this commit as the candidate.

A host regression grows a span piece by piece and requires one pinned allocation on this head. docs/recipes/cuda-weight-upload.md describes the device run.

Checks (host)

  • The full CUDA host suite passes: 320 steps, 4,904 tests, 28 skipped.
  • Formatting passes. Lean is 425 before and after.
  • The Linux cross-compile passes.

Not checked yet, and needed before it lands, on a GB10 for Qwen and Nemotron:

  • an A/A and an A/B/B/A of load time, read-ahead off and at the default;
  • final device-weight hashes equal to main's, with a planted changed byte that differs;
  • memory peaks;
  • served tokens equal to main's, and a cold 32K prompt.

It stays a draft until that receipt is posted. If it shows no gain, it closes.

zig/build/cuda.zig ends at 599 lines here. The GLM startup PR also adds one line there, so whichever lands second ends at 600.

Load the actual Qwen and Nemotron engines in separate timing and hashing processes. Read final weight buffers through an owned pinned destination, preserve it after failed completion, and distinguish a late device-byte control.
Reserve the full aligned Qwen read slot once at open and read MLX shard spans directly into it. Let Nemotron integrated read slots retain pinned bytes through DMA completion and quarantine. Keep discrete and temporary-vector paths unchanged.

Recording-driver tests cover different file/mmap bytes, growing spans without reallocations, unaligned tails, completion and event failures, and partial allocation cleanup. Move host-test registration into the checkpoint helper. The recipe requires warm A/A and A/B/B/A with read-ahead off and a separate default-read-ahead pair. Device weights, load time and peak memory still need qualification. Refs #2043.
@ashhart

ashhart commented Nov 12, 2026

Copy link
Copy Markdown
Owner Author

Closing this draft. #2282 (merged 7 Nov) already covers it for Qwen. It is the weight-upload part of #2043, and #2043 is closed.

Since #2282, Qwen 3.5's CUDA loader reads each layer past the page cache into reused buffers (families/qwen3_5/cuda_weights.zig:380). Checkpoint.get then returns those copies instead of the mapped bytes (core/checkpoint.zig:124). The upload source in this draft only takes bytes that point into a shard's mapping (cuda/checkpoint_upload.zig:42). On today's main, the layer weights, which are most of the bytes, would skip it, and only the embedding, head and final norm would go through. So the GB10 receipt this draft was waiting for would mostly time a path it doesn't change.

The Nemotron half, where integrated read slots become owned pinned stages (families/nemotron/cuda_source.zig), isn't covered by main. It can come back as its own PR once a GB10 load run shows a gain beyond the A/A spread, with equal device-weight hashes. The load and device-weight receipt tools here can come back with it.

  • Synthena (Ash's Agent)

@ashhart ashhart closed this Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant