Repository navigation
Conversation
dots.mocr, Nemotron Parse and GLM-OCR need a dense decoder on CUDA, which had none, and every CUDA family carried its own copy of the lane plumbing. The cuda_dense module now holds that once: paged keys and values in core pages, prompt and decode attention picked from the head layout, the lane backend with copy drafts and cancel, device draws that refuse non-finite rows before any launch, host uploads kept alive to completion, failures drained before teardown, a recipe's own rotary table, and Native(R), which gives the CUDA host a family from a recipe of one layer's arithmetic. No family uses it yet.
dots.mocr's Qwen2 decoder runs on the dense engine: q|k|v and gate|up fused into packed BF16 lanes, the q|k|v biases and rotary in its own Torch-order kernel with Transformers' own float32 frequency table, attention through the engine, the untied head, and no reasoning block. The loader drains its streams before freeing on a failure. It is not registered yet; its host tests check the geometry, text bytes, frequencies, declarations and refusals.
The native CUDA server now lists dots_ocr with the publisher's BF16 weights and opens it through the dense engine, built once for the server's engines. Its capability entry is checked, and the host's reasoning test now expects only dots.mocr to declare replies without a reasoning block. Images stay refused until its tower lands.
The dots.mocr oracle only ran the model card's image example on one HIP GPU. It now runs text-only prompts on a HIP or CUDA GPU at the pinned Transformers, with the rotary values the native kernel's checks read, and a score mode that teacher-forces native ids through HF and classifies each as exact, one BF16 step or a failure, exiting nonzero on any failure or on inputs that do not line up. Without flash-attn, a refusing stand-in lets the tower's import load for text.
The dense engine also launches the BF16 embedding rows and the Torch-order SwiGLU, and neither had a launch record. Their register and shared-memory usage, read from nvcc 13.0 images for sm_80, sm_86, sm_89, sm_120 and sm_121, now sits with dots' rotary record; the contract's image table gains the lfm2 image the SwiGLU lives in.
dots.mocr's text decoder now serves on CUDA through the shared dense engine, so it gets a recipe page of its own beside the HIP one: how to build and serve it, which flags it refuses and why, where each part lives, and how the oracle's text and score modes check its replies. The HIP page, the recipe index and the README row point to it.
The declaration test's hook list carried its comment at the end of the line, past the column limit; the comment now sits on the line above it, and nothing else changes.
…air count (#2823) Grouped.run and bytes take the real pair count and the caller's tile (16 or 64; any other tile is refused with TileNotBuilt). maxm is the tile, grid.z is mats * sk * (tile / 16), and x0, x1 and z are spaced by pairs instead of items * 64. Every caller passes its own tile: decode and verify 16, wide prompts 64, MiniMax-M3's prefill 64. Qwen sizes its grouped storage once at max_rows * slots. No kernel source changes.
6 of 8 tasks
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI for a landing train: #2673 #2843 stacked on main in that order. Nothing merges from this PR: each PR above lands on its own once this tip is green, and this PR is then closed.