Repository navigation
Linux direct reads: Engram falls back to buffered where a disk refuses sector reads; direct_gather names its error - #2610
Merged
Merged
Conversation
Any reader's failure used to reach the loaders as ReadFailed, so a shard that ends inside a tensor it reads, or a source that is gone, could not be told apart. readAll now returns the first reader's own error (EndOfFile, FileNotFound, or direct_io's ReadFailed), through Gather.read and Pair.wait alike, to checkpoint_direct's loaders and Flash Next's experts, which pass it on with try. Tests: a file cut inside the last of four pieces, on and off a 4 KiB boundary, gives direct_io's own error for that read; a removed file gives FileNotFound; on Linux, checkpoint_direct stops the layer a cut lands in with EndOfFile after reading the layer before it.
Engram extents are 512-byte aligned. On a disk whose direct reads need 4 KiB (or a file system that takes the O_DIRECT open but not the read), every extent's read failed with EINVAL, so good rows gave EngramReadFailed in every reader mode; nothing fell back as direct_io does. openTable now reads one sector at a sector first: refused with EINVAL, it clears O_DIRECT on the shard's descriptor and the table is buffered, as on a file system without O_DIRECT. The serial, threaded, io_uring and AIO readers all read that descriptor, so the read path is unchanged; a 512-byte disk keeps its direct reads and extents. Test (Linux): a fake disk refusing direct reads off 4 KiB, or off 512 bytes, or none, under every reader mode: the same row bytes and the same extents each time, direct except on the 4 KiB disk, which refused only the check at open.
ashhart
force-pushed
the
lead/dio-followups
branch
from
November 8, 2026 21:14
7625ced to
e1ff593
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two follow-ups from an audit of the engine's uncached read paths (the one that also found #2602's end-of-file error and #2608's ring hang).
Engram shards on disks whose direct reads need more than a sector
DeepSeek-V4.1's Engram reader opens shards O_DIRECT and reads row extents aligned to 512-byte sectors (
engram_rows.zig:7,engram_io.zig:312-314). On a disk with 4 KiB logical blocks (or a file system that takes the O_DIRECT open but refuses the read), those reads fail with EINVAL, and no path fell back: good rows gaveEngramReadFailed, in every reader mode (serial, threads, io_uring, AIO). NowopenTableasks once, before the header read: it reads one sector at a sector offset on the O_DIRECT descriptor, and if that is refused with EINVAL, it clears O_DIRECT on the descriptor and reads the shard through the page cache, as a file system that refuses the O_DIRECT open already does. Every reader mode already handles a buffered descriptor, so the read path itself doesn't change. On a 512-byte-sector disk the check passes and nothing changes except one extra 512-byte read per shard at open. Bytes are identical either way.direct_gather names its error
core/direct_gather(the direct-read loader for Spark 2.5, Dory, Gemma 4 and Qwen 3.5 on CUDA, and Flash Next's ModelOpt experts) turned any reader thread's error intoReadFailed. It now keeps the first failing reader's own error (for exampleEndOfFilefromcore/direct_iowhen a shard ends inside a span, orFileNotFound) and returns it after every reader has joined. Every caller still refuses (all usetry; none matchesReadFailed).Checks
zig build test-cuda-host test-dsv41-sharedpasses (exit 0), with the new tests: Engram rows byte-identical through each reader mode on a fake disk that refuses any offset or length that isn't 4096-aligned, a 512-aligned disk unchanged, anddirect_gather/checkpoint_directreturningEndOfFileandFileNotFoundby name for a shard shortened after open or removed.zig build test760/760 steps,test-cuda-host345/345,test-dsv41-shared101 pass (the Linux-only test skips);compile-checkpoint-consumersbuilds the x86_64-linux server;zig fmt --checkclean; every file within 600 lines.