Repository navigation
Conversation
…e the rank stops Split's start-up warm-up sends and receives 4 MiB with every other rank, and runs before Engine.init's split probe, whose refusal names the fix. On a fabric where some ranks reach only their neighbours (a ring of four without a switch) the warm-up's group ends with an NCCL error first, and check stops every rank at once, so the probe's refusal never prints. From four ranks, where a ring has ranks that are not neighbours, the warm-up now prints that refusal, naming gather, when its group end fails under split, and hands check a name that starts with it, so a rank that stops on another rank's failure names the refusal as the cause too. Fewer ranks, ring and gather warm-ups, and a warm-up that finishes, print nothing new.
P2 of #2444 adds reuse identity (C6) beside the tree's words, so the store's begin and keep now take a Query instead of a bare prompt. The engine_api side carries every identity field with defaults and reads only the key for now; the core side carries the key and prompt length, which the CUDA adapter converts as it already does for Plan. Every caller passes the same words as before, so no store user changes behaviour.
The checkpoint manifest of #2444 (C3) lets several nodes and entries name one page, so the tree now holds its pages through Holds, a registry that takes one pool reference on a page's first name and gives it back on the last. The new prompt_radix_parts.zig also declares the manifest's types for the next steps. Node pages are distinct today, so the pool sees the same retains and releases in the same order and every store user is unchanged.
A family whose cache has several prefix groups (a target and a drafter's full attention on one pool, #2444 C3) needs one page id of each group for every logical page on the tree's path. The tree now keeps per ids a logical page, page-major, counts logical pages in its walks and ids in its limits, and initGroups declares the groups, refusing groups with other rows a page than the store's. With one group, today's case, every index and count is as before.
C3 of #2444: a family may declare checkpoint groups whose pages an entry holds beside its legacy snapshot. A copied part takes store-held pool pages its live bytes are copied into; a retained part names the stream's own pages and copies a partial last page. Entries are published only whole, with every copy fenced before a page goes back; a node is pinned while a keep builds it or a resume restores it, so reclaim never frees its own source; reclaim first frees idle parts. Also fixes a leak in insert when its second allocation failed. Stores without parts are unchanged.
C3 of #2444 defines the store for a family with no checkpoint at all (pure full attention, as vLLM's prefix cache): it keeps the path's pages with no snapshot slot, resumes at the deepest page edge a prompt repeats, inside a node too, and makes room by freeing idle leaves, never the path a pass resumes. No family on the store is such a family today, so every store user is unchanged; the first one (MiniMax M3, P7) needs no store change.
A mark whose path and checkpoint parts passed the page limit was planned anyway; making room for it evicted a kept checkpoint before the keep itself was refused. Planning now counts the part pages it knows before the keep (copied parts and partial retained pages), so such a mark is refused up front. Long trailing comments in the lines this branch added move above their lines.
The CUDA ring families' snapshot saves take pool pages, so a save can make the pool reclaim from the store in the middle of a keep. On a keep onto a leaf an earlier failed save left with no entry, that reclaim could prune the very leaf the keep then writes its entry to. The keep's node pin prevents it; this test holds a save that needs one page more than free over exactly that leaf, and it crashes with the pin removed.
…ckground pass's chunks narrow with depth A 20 GiB prompt cache holds two 128k states. The state kept at a reply's end pushed out the one at the history end, so a client whose replayed reply differs from the generated one resumed a turn further back. Slot 0's in-slot rewind buffer now exists whenever there is room, also beside a cache that keeps two full-context states, where needsRewind alone makes none, and its point is carried through passes that only extend slot 0, so the next turn rewinds in place. Beside such a wide cache, which keeps the reply's end, the warm pass over slot 0 runs when it carries that point; beside a smaller cache it still yields, as on main. Background chunks scale down with depth so a waiting request is held for less.
…'s choice are tested rules Standing.warmYields holds the warm pass's yield in opening, Standing.take holds takeStanding's use of the record and its carried rewind point, and makesRewind holds rewindInit's choice: the buffer wherever room holds it, tight where needsRewind says the cache cannot keep two full-context states, else wide. Each has a test that fails when its rule is reverted. The changelog takes the line for #2723 under a new Unreleased heading.
… new rewind point Slot 0 remembers the rows of the reply it generated. A follow-up whose prompt replays that reply as generated and resumes, by the slot or a cache state, inside it or at its end skips the extra rewind stop at its history end, so its fill is the same as main's. A client that drops the reply's reasoning, a cold turn and a resume that goes back still make a fresh rewind point.
check_cuda_sources.py failed 66 of its 918 checks on main: eleven units use SDK names that the fixed declarations in tools/zig/cuda_host lack. Declare each as CUDA 13.0 does: __syncthreads_or and __double2float_rz from sm_20_intrinsics.h, __float2int_rn from crt/device_functions.h, __float2bfloat16_rz from cuda_bf16.h, __hmul on half from cuda_fp16.h, ldexpf from crt/math_functions.h, and the __maxnreg__ macro from crt/host_defines.h, whose maxnreg attribute Clang ignores with a warning. Only the check reads these headers: no kernel, engine or test binary changes.
…er, its keys unchanged
The video byte gate, tf-glm-video with glm_video_fixtures.py, builds only on macOS and needs a checkpoint's processor. This tool needs only a libtfvideo build and clips: it decodes every frame, GLM's 2 frames a second, and the first three and last frames, hashes each frame's RGB rows, and exits nonzero when any frame or return code differs from PyAV's or from a second build's. A clip that will not open or decode fails and the run goes on. --time-only times the decoder alone. Nothing in the engine or the build calls it, so no served path changes.
…links FFmpeg's x86_64 assembly addresses tables defined in C, such as ff_pw_* and ff_h264_cabac_tables, with 32-bit PC-relative relocations. Linked from the static libav*.a into a shared library, those tables are exported and so preemptible, and the link stopped on R_X86_64_PC32 errors (5,046 with gcc 13.3) although build_ffmpeg.sh passes --enable-pic. On ELF targets the library now links with a version script that exports the four tf_video_* symbols core/tfvideo.zig looks up and makes every other symbol local, so the tables bind inside the library with no text relocations. check-video-exports-linux checks this for aarch64 and x86_64 Linux with a planted table. The loader, build_ffmpeg.sh and its FFmpeg pin, and Mach-O builds are unchanged.
engine_api.zig is at the 600-line limit, and the OCR reader types that follow add to it. Memory and MemorySource describe a backend's status, as HealthSource and Status do, so they move beside them in engine_status.zig and engine_api re-exports them under the same names; no caller changes.
The libjpeg-turbo loader and the BGRA page encoder move to their own module, jpeg_out.zig, with the optional-library loader that PDFium shares, so the OCR route can encode image crops and pdf.zig gains the room its page selection needs later; jpegPixels becomes jpegSize, which returns the width and height a frame header names, for the OCR page planner. Chat PDF input behaves exactly as before, and the new encodeRgb is covered by a round-trip test.
An engine now declares an optional OCR reader in Info.ocr: the prompt that follows the page image, the geometry it gives and a parse from generated text to Markdown blocks with quads in original pixels. PreparedImage gains the tower-input frame so readers can map boxes back, the HIP offer carries the reader through, and a bounded reader for the HTML table subset dots.mocr writes turns tables into GFM pipes, clean spanned HTML or text rows, escaping literal markup so Markdown never reinterprets recognized text.
The OCR route reuses chat.prepare and chat.generate but needs its own encoded-byte limits for page images, a reply that may fill the room the prompt leaves, and the frame each image was resized to so readers can map boxes back. chat.Input gains image_bytes and fill_room with defaults equal to chat's values, a route with its own limits decodes data URLs against them and its refusals name them while chat keeps its words, and the frame of each image part flows from PreparedImage through Expanded and Rendered to Prepared.
Clients need the route's contract before they can point the Mistral SDKs at the server, so docs/api.md gains an OCR section: the route and the reader, the Markdown the reader writes, page and image budgets, the refusals of PDF documents and pages, failure statuses and every Mistral field, and the inference scope lists /v1/ocr. The changelog names the route under Unreleased.
Step 1 reads page images through /v1/ocr only, so the code that later steps use moves to them: the plain-text line path in markdown.zig, the keep_math switch it needed, rowsText, PaddleX's polygon with its rounding helpers, OcrLine and fetch.https's PDF switch all return in the step that first calls them. The pipe-table tests that went through plainLines now read the cell back through pipeTable, which keeps the emphasis, code, link, entity and one-pass guards on the clean-up that step 1 still runs, and two comments now cite docs/api.md sections that exist.
The run decoded a page up to 32 Mi pixels, GLM-OCR's limit, but step 1 serves only dots.mocr, whose processor stops at 16 Mi pixels. The page limit is now picture.max_pixels, so a larger page is refused as bad input at decode with the picture's own message, before the reader runs; GLM-OCR's step raises it again.
Step 1 reads one page image a request, so the page budget, the planned page's source, index and DPI, the sink's source argument, Geometry with its none and line readers, Kind.page, the null quad, the no-geometry header and footer refusal and the Scanner's line payloads have no caller here; each returns with the step that first uses it. An image is page 0 at 72 DPI, every block has a box, and comments that named PDF, /ocr, PaddleX or visualisations now describe this step only.
A route test sends a PNG whose header names 4097 x 4096 pixels, 4096 past the 16 Mi decoded-pixel limit and under 8192 a side, and checks HTTP 400 with the decoder's own message before any generation. The server started without --vision now checks the refusal's text, not only its status, and that nothing was generated.
The reader prompt is the model card's text and the demo fixtures are the model's output on the publisher's image, so the fixture README and a THIRD_PARTY_NOTICES.md entry now name the publisher's MIT License, Copyright (c) 2025 rednote-hilab, and the dots.mocr License Agreement that adds terms to it. LICENSES/ holds both texts: the MIT file unchanged from the pinned commit and the agreement from the pinned release with three lines' trailing spaces removed.
Step 1 set a source's name but never read it; docTR in step 5 is the first reader that names its sources in replies. The field and its writers in the Mistral request and the run tests go, and step 5 brings the field back with its first reader, as the review asked.
The dots.mocr License Agreement covers the model card's prompt and the model's outputs, and its terms should not enter the tree. The reader now uses a prompt written for TensorFold (prompt.txt), the fixtures record oracle runs on two pages the tool draws, and they store token-id and text hashes, not the model's text. The tests read written block lists, and TF_DOTS_OCR_DIR checks real oracle runs by hash before it parses them. The licence copies and notice go.
Resolving docs/api.md during the rebase onto 33ada77 kept only the inference scope row and lost the OCR section (route and reader, Reader Markdown, pages and budgets, failures, Mistral fields) that code comments, the CHANGELOG line and the PR text cite; this puts it back, unchanged, between Rerank and System One.
The rebase onto 33ada77 joined #2253's image-only refusal with /v1/ocr in routes.zig by counting OCR as a text route; this test sends POST /v1/ocr to the images suite's image-only stub and expects the same 400 naming the images route, with no engine call. It fails when is_ocr is taken out of is_text.
…e, reads row exponents from the bits, and takes e^la once a row The prompt GDN's output kernel keeps every product and its order. The two rescales after the products multiply by 2^n when every lane's n is in [-124, 127], where ldexp is that one product, and run ldexp otherwise. The row exponent is the float's exponent field, equal to 14 - ilogb(m) under the same clamp for every float. e^la is computed once a row in the first half. Refs #2859
… run on pushes to main only A landing train pushes one commit a PR, and every push to main kept its own full No GPU run: seven runs in four minutes on 13 Nov, each a tree the train's own run had already tested, queued ahead of the next train's run. Main pushes now share one group that a newer push cancels, as a pull request's do; the nightly schedule keeps its own group. The app workflow's push trigger, which had no branch filter, now runs on main only, as its pull_request runs cover branches.
… bf16 logits on request tf_route_logits gave each lane four 32-wide slices of k, which covers a hidden width of exactly 4096 (GLM-5.3-Flash and Step-5); at 2048 it read past each row and past the packed router. Each lane now takes hidden / 1024 slices, the same four at 4096, so those routers' arithmetic is unchanged, and `source` refuses a width that is not a multiple of 1024. `round_logits` rounds each logit to bf16 before the select, as a bf16 router matmul (MLX's) gives it.
A Metal family for `model_type: flint` (RecursiveSI/flint), served from Recursive's MLX INT4 export (RecursiveSI/flint-int4: affine 4-bit groups of 64, an 8-bit embedding) by the lane host with the prompt cache, structured output, tools and several streams. - Layers: 35 sliding (513-token window, RoPE) and 14 full (no positional encoding) GQA layers, 32 query heads over 4 key heads, per-head q/k norms and a learned sink each head; two dense SwiGLU layers, then 256 routed experts (top 6 by sigmoid score, renormalized, no score bias) beside an ungated shared expert. - Rounding: MLX's points, as RecursiveSI's rec_mlx computes the model: a bf16 residual, rms_norm as w * bf16(x * inv), SiLU rounded op by op, bf16 router logits. - Attention: prompt chunks on the tensor units (gemma4's prompt kernel over a position-major cache, the sink in the denominator); decode rows on sdpa_vector's lanes over key blocks at absolute positions, merged with the sink by MLX's second pass, so a row's bits do not depend on the rows beside it. Sliding layers keep a ring of the window and one prompt chunk. - Projections: q|k|v and each MLP's gate|up stacked into one matrix at load (one launch, the same row sums); one launch norms and rotates q and k and writes k and v; residual adds fused with the next norm. - Routing: core moe_route's logits, then a top-k pick a simdgroup a row (the core select's arithmetic without its per-Mac lists). - Serving: several streams' windows share one forward, each reply equal to its run alone; a lone stream's steps run a step ahead, drawn on the GPU (argmax, or the keyed sampler of core lanes' gpu_rule and gpu_full at Flint's vocabulary) into a token ring, and the host draws with the same sampler's twin. - tf-flint-run: greedy tokens, every position's logits, forced steps and step timing, for a reference's check, and one-row steps against windows of any width, bit for bit (`exact`).
…ac's RAM is refused, not converted
…(the fp32 form missed MLX on 1,360 of 65,280 bf16 gates, and gave 0 below -17); the router keeps MLX's fp32 form
…al_main has room under 600 lines
…p, decode and prompt attention; gates 1-4 tests, 16 planted controls, gate-5 records), and core tf_route_logits at Flint's, GLM's and Step-5's widths
…he tensor units keep (fp32 cut to 10 mantissa bits, measured on the device before the gates run), not bf16's
A recorded one-row step copied each stream's whole GDN state and conv tail for a keep of no rows, and served traffic never keeps one. Part now carries undo (default on, so warm-up, qualification and host tests keep their copy); the served queue and verify parts set it only while learning. With undo off the staged segment reads no before, recurrent.prepare leases no copy, and the two kernels skip only their copy: the conv tail's in-place update still runs. An empty keep after such a step refuses with QwenNoRollbackCopy rather than restoring bytes the step never wrote. Host: qwen35 shared, cache, pending, rounds, verify, live, prefill and the recurrent workspace steps pass; compile-cuda-tests for Linux compiles.
Owner
Author
|
Conflicts with main (main moved after stacking); restacking. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI for a landing train: #3230 #1177 #2933 stacked on main in that order. Nothing merges from this PR: each PR above lands on its own once this tip is green, and this PR is then closed.