Repository navigation
Conversation
8 tasks done
Owner
|
Thanks @laudney, the CUDA page reads well and its limits match the code. One thing:
|
8 tasks done
Contributor
Author
|
Thanks. |
Owner
|
Thanks @laudney, the oracle paragraph now covers both GPUs, the fallback and the Pillow pin. The stack is ready from our side. It lands in order after #2673, and we'll write each part's CHANGELOG line when it lands, so there's nothing more to do on yours. (It currently cites #2732, which is closed; ours will name the PRs that land.)
|
14 of 26 tasks
7 of 8 tasks
4 tasks done
Contributor
Author
|
Sorry, @ashhart: at 14:28 I force-pushed The code did not change: my heads hold the same 26 commits as yours ( |
Owner
|
No harm done, @laudney, thanks for saying. Your heads carry the same 26 commits and merge cleanly with main, so we'll land the stack from them as they are; the changelog lines get added at landing, so nothing needs pushing back. The seven are queued in order. We'll take it from here and only ask if something needs your hand.
|
3 of 4 tasks
dots.mocr's image tower rotates its queries and keys by the patch's row and column with FP32 cosines and sines, and adds the patch embedding's bias after the projection has already rounded to BF16. Two CUDA kernels now do this in Transformers' rounding: the 2-D rotary in place over the q|k|v rows, and the patch bias as bf16(y + b). Each has a planted control (BF16 cosines and sines; one rounding of the FP32 sum) and a contract against an FP64 reference at up to 19,520 rows and positions to 3,393, with every BF16 input pattern and tie for the bias. Their launch records come from nvcc 13.0 images for the five SMs, which 13.2 and 13.3 compile the same.
The comment beside the dots_mocr entry in the CUDA kernel list named only the text q|k|v bias and RoPE, though dots_mocr.cu now also builds the vision tower's 2-D rotary and patch bias launches, so the comment names all three.
The dense engine read text prompts only, so a recipe with an image tower had nowhere to put its rows. A recipe may now declare a Tower, checked at compile time against the contract in recipe.zig; each lane owns a buffer of the request's image tokens, the first one at open and the others at attach, and these bytes join every stream's bytes and fixed part so the page planner shrinks an automatic count. The lane encodes its images in order when it begins, with a cancel check before each, and each prompt chunk copies the image rows that it cuts over its embedding rows. Native(R) is now one shared core with two variants: the text-only one keeps the old decl set exactly, and the tower one adds the vision options, field and bytes that the host reads. Host tests drive the tower variant with a fake tower, also through the host's own offer and options code, and pin that the image span refusal names --vision-image-tokens, the flag the CLI accepts.
Only the words of DenseTowerWidth were tested, so dropping the width check in openTower would pass the host suite. openTower is now public, and a host test opens the fake tower against a decoder one row wider and expects the refusal, closing the tower first if it opens, then opens it at the decoder's width and without --vision.
No test drove a chunk that cuts an image through the backend, so a chunk that passed no injects or a forward pass that skipped the splice would stay green. The fake device now records its device-to-device copies, and a host test runs two chunks cut inside an image on a quiet recipe (no layer work): each must copy that image's rows from the lane's feature rows to the chunk's embedding rows, at the right row, offset and length.
The engine sizes each lane's feature rows by the request's image tokens and copies image.tokens rows an image from them, so the Tower contract now says that encode writes exactly image.tokens rows of the tower's width at out.
Three trailing comments in images_test.zig ran past 120 columns; each now sits on its own line above the code it describes.
The tower's tensors load as plain BF16 [n, k] matrices for cuBLASLt: fc1 and fc3 stacked into one gate|up projection, and the patch weight flattened to 588 columns and zero-padded to 592 so its rows meet the library's alignment. Every vision_tower tensor is consumed, the device bytes count the padding, and a failed load drains its uploads before any buffer is freed.
The drain test called settle directly, so dropping the errdefer in load, or registering it before the tower's own errdefer, would stay green. A host test now runs load on a sparse checkpoint against a fake device whose first upload's wait fails: the buffers must go only after the context proved the copies done. The sparse checkpoint helper moves to its own test file so the tower's tests can share it.
The dense engine's Tower contract gets its first real tower: dots.mocr's 42-block image tower, which turns merge-ordered patches into the decoder's rows at the rounding points of the HIP port and of Transformers. The tower reuses the shared vit GEMMs, attention, LayerNorm and linears, the engine's norms and SwiGLU, and the new vision rotary and patch bias launches. It holds one call's scratch at the processor's 57,600 patches and taps every stage for the oracle's comparison. Host tests pin the contract, the exact weight, scratch and open bytes against a sparse checkpoint, the rotary inv_freq fixture, the refusals' words (the budget's naming --vision-image-tokens, the flag the CLI accepts), the grid checks and the patch rows; the recipe declares the tower in its own change.
The tower needs device checks that the host suite cannot run, so tf-cuda-test gains two dots commands behind one dispatch line and a dots-help that prints their usage. dots-stages encodes one image with a tap that writes every stage's rows as BF16 .npy files for the oracle's stages mode. dots-image encodes a generated page (the processor fixture's pixel pattern, no file read) and an optional image twice each, requires equal bytes, prints the merged rows' hash and the time, and fails unless the page's hash equals its anchor, so a tower drift fails without the Python oracle. The runner imports the served dots module and its engine.
Nothing failed if the tower stopped trimming its cuBLASLt plans, so each new image size could grow the cache without bound. A host test fills the plan map on a fake driver and cuBLASLt: below max_plans nothing drains and nothing goes, and at max_plans the stream drains once and every plan's handle, descriptor, layouts and preference go.
The vision-merged pin is a GB10 value, so another GPU's rows would read as a tower drift. dots-image still encodes twice and requires equal bytes everywhere, but on a device other than sm_121 it skips the pin comparison and names that device's SM.
Two trailing comments ran past 120 columns (the dots runner import in the CUDA build and a control line in the tower test); each now sits above its line. The tower test also takes the sparse checkpoint dir from the shared test helper instead of keeping its own copy.
cuda_tower.zig cites fixtures/README.md for vision_inv_freq.txt, but the entry arrived with the docs piece; it now lands with the tower that reads the fixture, so the reference holds at this head.
The recipe now declares the tower, so the dense engine's tower variant serves dots.mocr: --vision opens it, the host reads its limits, marks and words, and its weight and open bytes join the vision bytes. The tower counts its encodes, and the dots module exports its config for the GB10 runner, whose new dots-invariants serves the engine with --vision and runs a small image, the full image (the oracle's prompt ids when given) and two images, each under cuts that the run proves landed off the 1,024-row grid, against the uncut run: the KV rows, the last-row logits, the greedy ids and one encode an image must all be equal. Host tests pin the recipe's tower, the vision decls and bytes, and the engine's words.
ashhart#2732's runner gains dots-text, so a decoder drift fails without the Python oracle: three chat-templated prompts of our own (18, 578 and 2,537 ids, committed as fixtures/own-prompts.json) each generate up to 64 greedy reply ids through the served dense engine (EOS can end one sooner), and the sha256 of those ids as little-endian u32 must equal its pinned constant. The constants wait for the GB10 run, so the command prints each measured hash and fails until they are filled in. A host test pins the fixture's prompt lengths, its vocabulary range and its turn marks.
The description claims the text path with and without --vision, but dots-text served the engine only with --vision. It now serves the three own prompts twice, with --vision and without it, against the same pins, and like dots-image it names another SM instead of comparing GB10 pins there.
The fixture's ids could not be regenerated from the tree. The fixtures README now names their source text, our own ownprompts.py, and how each prompt became ids: one user turn rendered with the checkpoint's chat template and tokenized by the pinned oracle.
The GB10 has no flash-attn, so oracle, score and text install a stand-in for the tower's import, put the tower on SDPA and refuse a block that is not VisionSdpaAttention; oracle and score record the SDPA backend Torch picks, score teacher-forces image prompts with the reference's pixel_values and image_grid_thw, oracle saves BF16 tower taps (with the patch embed before its norm), the tower rotary inv_freq fixture text and cos/sin, and decides once or twice rounding of the patch bias, and a stages mode compares native taps by max abs and BF16-ulp histogram; image references now require pillow 12.2.0 and qwen-vl-utils 0.0.10, because Pillow 13.0.0 changes the demo pixels from the fixture's hash. The image modes (oracle, stages and score's pixels) move to dots_mocr_images.py, which the reference script loads by its path, so python -I runs it from any directory, and each file keeps room under the 600-line limit.
Gate 2 asks each changed reduction order to be no worse than Torch's BF16 matmul against FP64 on the same device inputs, so the dense pass and the dots tower gain an optional tap on each projection's input and output rows. Served passes leave the tap null. The new dots-projections runner command encodes one image and prefills the image prompt's first chunk with the taps set. It writes every input row, sampled output rows and each fused class's separate GEMMs for the tower's patch, blocks 0, 20 and 41 and merger, and for the decoder's first and last layers and head. tools/zig/ dots_projection_check.py computes FP64 products and Torch's layers as Transformers calls them, and fails a class past one BF16 step plus the FP32 sum's error, worse than Torch's worst column, or with a fused part off its own GEMM; a missing class also fails.
The oracle put the tower on SDPA on every GPU, so HIP image references made with flash-attn would differ in the tower from the ones behind the HIP numbers. The tower now follows --attention when flash-attn is installed and falls back to SDPA only without it, and only an SDPA tower refuses blocks that are not VisionSdpaAttention. oracle and score gain --sdpa-backend auto|math: math turns flash, memory-efficient and cuDNN SDPA off, as the math-backend reference needs, and both record the choice in their output.
A trailing comment ran past 120 columns; it now sits above its line.
The oracle's help and the image module's docstring still promised an SDPA tower, though the tower now follows --attention when flash-attn is installed; both now say so, with SDPA as the fallback.
The CUDA page now serves images with --vision: the per-request image budget, the largest image, the per-lane rows that lower the admitted lanes, the JPEG library, and the oracle's SDPA image references, the tower stage comparison and its Pillow pin, and the runner's anchored dots-image and dots-text checks; the README, the recipes index and the HIP page point to it, and the changelog credits the change under Unreleased.
The fixtures README still described the oracle as HIP-only with the flash command. It now says that the tower and the decoder follow --attention, that without flash-attn the tower falls back to SDPA and the decoder needs --attention sdpa, that --sdpa-backend math keeps SDPA on its math kernel, and that image references pin Pillow 12.2.0; it gives the HIP and the CUDA commands, and the CUDA page names the math-backend switch.
This was referenced Nov 13, 2026
Owner
|
@laudney main landed dots changes of its own under this stack (
|
6 tasks done
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
docs/recipes/dots-mocr-cuda.md--visionrefused--vision: the image budget, the largest image, per-lane rows, the JPEG library, the oracle's SDPA image references, the stage comparison, the Pillow pin, and the runner'sdots-imageanddots-textchecksfixtures/README.mdinv_freqtableCHANGELOG.mdWhat this changes
Documentation only, for the image path that pieces a to f add:
docs/recipes/dots-mocr-cuda.md: serving with--vision(--vision-image-tokenssets the request's total image budget, 14,400 merged tokens by default; the largest image is 57,600 patches; each lane holds its request's image rows, so--visionlowers the lanes a memory budget admits; JPEG needs libjpeg-turbo 3.1.4.1, PNG needs no library; video is refused), the code map, the oracle's SDPA image references withstages, and the GB10 runner'sdots-image,dots-textanddots-help.README.md,docs/recipes/README.md,docs/recipes/dots-mocr-hip.md: the CUDA row and links.zig/src/families/dots_mocr/fixtures/README.md:vision_inv_freq.txt, its source and the host test that checks it.CHANGELOG.md: "dots.mocr on CUDA:--visionserves images through the dense engine, with dots.mocr's 42-block image tower, its 2-D vision rotary and patch bias kernels, and a request's image budget set by--vision-image-tokens; on a GB10 every token of the model card's demo scores exact under HF's teacher-forced logits (cuda: dots.mocr's image tower on the dense engine #2732). Thanks to @laudney."Model licence: n/a, documentation only.
Issue: Refs Native OCR models on HIP: dots.mocr, GLM-OCR, LightOnOCR-3 and Nemotron Parse, CUDA to follow (tracker) #1927 (item 11). Remaining after this stack: Nemotron Parse and GLM-OCR on this engine.
Stacked on: dots: oracle image references and FP64 projection checks #2772 (piece 6,
5e04a9beb). This PR's commits:7eef428f5docs(dots): dots.mocr's images on native CUDA;cd1cb0baadocs(dots): the oracle's attention, SDPA fallback and Pillow pin. Piece 7 of 7.Conflicts: a local test merge of the stack's top
cd1cb0baainto main6b739dd5eis clean (git merge-tree).Checks on this head
cd1cb0baazig build test-cuda-host,check-native-cuda-host7/7,test-kernel-contract,zig fmt --checkandgit diff --checkpass; lean 386 / 386, 0 new against piece f. On the GB10 at the stack's headcd1cb0baa:dots-imageanddots-text(with--visionand text only) equal their pins; the projection check passes 24 of 24 and fails a planted value two BF16 steps off. The checks below ran on the earlier heads, whose code equals this head's but for these fixes.python3 tools/zig/lean_check.py, on the base (cuda: a shared engine for dense decoders, with dots.mocr's text decoder on it #2673's head05e5e5fdd) and on this head: 386 / 386, 0 new (also 0 new against piece f).zig build test-cuda-host: 378/378 steps; 6,028 of 6,055 tests passed, 27 skipped, 0 failed (uncached).zig build test-cuda-dense-host: 33/33 passed, uncached.zig build test-dots-mocr-cuda-host: 24/24 passed, uncached.zig build check-native-cuda-host: 7/7 steps.zig build test-kernel-contract: 32/32 steps, 0 failed.zig build compile-cuda-tests(the GB10 runner compiles): 10/10 steps.zig fmt --check build.zig zig,git diff --check: pass;python3 -I -m py_compileondots_mocr_reference.py,dots_mocr_images.py,dots_projection_check.py: pass.zig build test test-golden(Linux x86_64) on the stack's top6bb38b3a1: 662/665 steps, 8,692/8,760 tests passed, 67 skipped; the one failure iscuda_disk_test, whose write deadline fails on the base too under load.python3 tools/pr_check.py --base 05e5e5fdd --pr 2732: 0 errors; one advisory warning,dots-mocr-cuda.md: "added non-loopback IPv4 address", which is the version3.1.4.1of libjpeg-turbo.Receipt
n/a: documentation only; no code changes. The claims it makes are the receipts of pieces d, e and f.
Checklist