Skip to content

DeepSeek-V4.1: read the pack's weight spans past the page cache (core/direct_io) - #1924

Open
ashhart wants to merge 6 commits into
mainfrom
lead/s191-dsv41-direct
Open

ashhart wants to merge 6 commits into
mainfrom
lead/s191-dsv41-direct

Conversation

@ashhart

@ashhart ashhart commented Nov 2, 2026 •

Copy link
Copy Markdown
Owner

Part of #1857: DeepSeek-V4.1 reads its pack's weight spans past the page cache, so a load no longer fills the cache it then has to release.

What changes

  • Pack.read reads DeepSeek-V4.1's weight spans with O_DIRECT through core/direct_io, in at most 8 MiB of aligned staging, with exact strided gathers.
  • core/direct_io now owns the TF_LOAD_DIRECT switch and the chunk size. TF_LOAD_DIRECT=0 reads through the page cache as before. A file system that refuses O_DIRECT, at open or on a read, falls back the same way.
  • The prepared-image identity adds pack_read.zig, so existing prepared folders rebuild once, the same on every rank.

The pack hash and the prepared-image reads still go through the page cache, so this narrows #1857's minimum during load rather than fixing it outright.

Checks

  • Host: the loaded bytes equal a real mapping's across large tensors, unaligned starts and tails, chunk seams, strided, overlapping and repeated runs, and 65 shards, in both TF_LOAD_DIRECT modes. The lean check finds no new problems.
  • An independent review: LAND after one change, which this carries (the switch moved into core/direct_io, so it stands on main alone).
  • Linux host run (aarch64, ext4, head c6d1c720): test-dsv41-direct-host in both TF_LOAD_DIRECT modes, test-dsv41-env-host and core/direct_io's tests all pass, including the two that only run on Linux. The first run on the previous head failed one: under O_DIRECT, a pack cut short inside a span re-read the same aligned tail and reported ReadFailed instead of end-of-file, so the pack reader never said ShortRead. c6d1c720 fixes that and adds a test for it. A Mac never takes this path (no O_DIRECT).
  • Device: a load on a Linux box with a DeepSeek-V4.1 EXL3 pack, comparing load time, MemFree and Cached against main. None of our Linux boxes has the pack at the moment, so this waits for that run and doesn't land before it.

@ashhart
ashhart force-pushed the lead/s191-dsv41-direct branch from 8e8fc81 to eebcf48 Compare November 2, 2026 17:36
@ashhart
ashhart force-pushed the lead/s191-dsv41-direct branch 2 times, most recently from f88414b to 9905ddf Compare November 8, 2026 17:39
ashhart added a commit that referenced this pull request Nov 8, 2026
Preserve the reviewed exact-zero disabling rule while routing its new reader through the current environment module. Keep bounded pack reads independent of Gather and test the legacy spellings without extending the lean exemptions.

Refs #1924 and #1857.
@ashhart
ashhart force-pushed the lead/s191-dsv41-direct branch from 9905ddf to 14ea21e Compare November 8, 2026 17:40
ashhart added a commit that referenced this pull request Nov 8, 2026
Preserve the reviewed exact-zero disabling rule while routing its new reader through the current environment module. Keep bounded pack reads independent of Gather and test the legacy spellings without extending the lean exemptions.

Refs #1924 and #1857.
@ashhart
ashhart force-pushed the lead/s191-dsv41-direct branch from 14ea21e to c6d1c72 Compare November 8, 2026 18:00
Keep the direct-read switch and staging bound in core.direct_io so the pack reader applies to main alone. Preserve the mapped-byte comparisons, strided spans and shard-count coverage.

Refs #1857.
Keep the relocated pack-range advice under the shared load-readiness policy and update its host source guard. Put the reviewed loading note at the top of Unreleased.
Preserve the reviewed exact-zero disabling rule while routing its new reader through the current environment module. Keep bounded pack reads independent of Gather and test the legacy spellings without extending the lean exemptions.

Refs #1924 and #1857.
@ashhart
ashhart force-pushed the lead/s191-dsv41-direct branch from efdeaa5 to c2090ff Compare November 12, 2026 14:48
@ashhart

ashhart commented Nov 12, 2026

Copy link
Copy Markdown
Owner Author

Rebased onto today's main (c2090ff8c8). The changelog was the only conflict, and one commit dropped out because main already carries it. test-dsv41-direct-host passes in both TF_LOAD_DIRECT modes, with the environment and load-cache steps and the Linux DeepSeek server build. What's left is the GB10 run with the DeepSeek-V4.1-Flash EXL3 pack.

  • Synthena (Ash's Agent)

@ashhart

ashhart commented Nov 12, 2026

Copy link
Copy Markdown
Owner Author

Device check status: blocked for now. On main, the CUDA server refuses DeepSeek-V4.1 behind its whole-model gate (#827). Its prompt path still needs captured Triton kernels, so a served main-vs-branch run can't start. The EXL3 2.9 bpw checkpoint is 197 GB, so it needs two GB10s at TP=2; one box can't hold it. This PR waits for #827, or for a two-GB10 run once both boxes have room for the checkpoint. Host tests and CI stand as they are.

  • Synthena (Ash's Agent)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant