Repository navigation
Conversation
reply_text.Incremental decodes a reply's new tokens with the previous chunk as context and holds a character split across tokens until it is whole (reply_text.zig:453-485, used at chat_generation.zig:108). No test named either property; main's suite saw them only through other outputs (thinking-tail and wire-byte tests for the hold, two stream arena bounds at 7500 tokens for the window). Two tests over a byte tokenizer whose decode spells a trailing partial UTF-8 sequence as U+FFFD: a split euro sign streams only once whole and the result equals a full decode; over 300 chunks no decode reads more than four tokens. Each fails on its planted mutation (no hold, full re-decode) and passes on main.
#1449's two kernels give tf_glm_scores' and tf_glm_select_rows' bytes, so a prompt chunk takes them unless TF_GLM_INDEX_CUDA=0; scores still bisects. Shapes the kernels do not take keep the old launches either way: a recorded-launch test pins prompt chunks to the kernels and a decode row, an unaligned key cache and a small pool cap to tf_glm_scores.
…ine for the default With the kernels on by default, the warning a failed load prints reaches people who never set TF_GLM_INDEX_CUDA, so it starts glm:, as GLM's other start-up lines do, and says the fallback is what TF_GLM_INDEX_CUDA=0 asks for. The changelog line records the new default: unset takes the kernels, 0 keeps tf_glm_scores.
Exercise reordered real PSI fields and unknown readings without full PSI or MemAvailable through the planted-file reader. Document that the Linux opt-in reads host pressure inside containers and cgroups.
…m_86 SASS An exact match against the release list (80, 86, 120, 121) refused an RTX 4090, L4 or L40S (sm_89) and Jetson Orin (sm_87), which today's release server runs: the driver runs sm_80 and sm_86 SASS on any later 8.x GPU. Below sm_90 no image is arch-specific, so a target now serves a GPU of the same major and an equal or higher minor; from sm_120 on some images are sm_120a or sm_121a, so those still need the exact SM. The new host test fails on sm_87 with the exact match.
Head.chains drew each stream's draft a level at a time: a greedy level took fourteen launches a stream, and a sampled one took a draw a stream. When every live part shares a draw mode and head, the level now runs as one pass over all its rows: greedy as one toF32, one topk, one lookup and one keyed draw, sampled as one tf_draw_ids_rows launch with each row's rule and position. One scatter writes every row's token and probability back to its stream. The level's rows come in through one gather, and a level's tokens through one copy of the previous level's slab; the tail's last rows move down in one ordered gather. attentionSegs takes the spans in chunks of eight. Mixed modes, one part and the Triton captured set keep today's per-stream path. The new kernels are recorded as pending until a GPU measures their register usage. Refs #2910.
A shared window drew each stream's target token a stream at a time and downloaded each one. With every stream in the same mode, the rows now draw in one pass: one argmax over all rows, or one tf_draw_rows launch with each row's rule and position, then one scatter to each sequence's sampled buffer and one download of the slab. Grammar masks still run a stream at a time before the draw. Mixed modes and a single window keep today's path. The batch is built only for several streams and refused when their modes differ; each row carries its own rule and position, so a multi-row stream's rows do not share one rule. Its rules and positions go up with the window's other pinned words, before the event that lets the next window rewrite them, and each part's place in the drawn-token slab is kept on the engine rather than in the meta row the kernels read. The forward refuses a batch whose mode does not match its streams' instead of reading an unwritten slab. A device case (tf-cuda-test nemotron-draws) runs the batched draws, the target's rows draw and argmax, and the row kernels against the per-row path, byte for byte. Refs #2910.
…e gather rowKernels' last check read dst_a[1] for the scattered token, but the scatter it set up writes the token at tok_at = 2 (prob_at = 1, as the engine's level j writes j and j - 1, cuda_mtp.zig:415-416), so word 1 still held the last-row gather's ha[17] and one check failed: "1 row kernels differ". The kernels are right; the check now compares both destinations whole against the expected rows, the token at tok_at and the probability at prob_at. The gather's output was never read: the last-row gather overwrote both rows before the download. The test now reads it after the gather and checks part k's row at dst row k.
… the full kernel contract (sm_80/86/89/120/121)
…in.zig within 600 lines
The serial side of the eight-chain draft test panicked with incorrect alignment on Linux CI: candidates() reads the top-k columns as i32 from the draft's u16 host buffer, and a stack [N]u16 is only 2-aligned. Production host buffers come from the allocator at row scale.
… tool tests, so pytest from the root imports them
…a private journal, checked graph states The four fixes the review on this PR asked for, each with a test that fails before it: - transport: a loopback client builds its opener with an empty ProxyHandler, so an ambient http_proxy never receives a request or its Authorization header; --allow-remote keeps the environment's proxies. - transport: SSE lines are read with readline(1 MiB + 1), so an oversized line is refused without reading it whole, and one stream keeps at most 4 MiB of event data and 4096 events. - ledger: missing journal directories are created 0700 and the database 0600 (SQLite gives its WAL and SHM the same mode); an existing database, WAL or SHM open to other users is refused by name. An existing parent directory keeps its mode. - transport and controller: every graph read checks each fact's state against the native graph's states (queued, learning, learned, missed, replaced, deleting, deleted); a missing, non-string or unknown state stops the controller before any lesson.
…are the ones checked when SQLite opens them The re-check of the journal fix found a gap: a file could appear between the mode checks and SQLite opening it, and the journal accepted a 0666 database, WAL and SHM planted there. - The journal's own directory must be a real directory owned by this user with mode 0700, or the ledger is refused by name; missing directories are created 0700. Only this user can then add or replace files in it. - The database is created O_EXCL|O_NOFOLLOW at 0600; on a collision the existing file is opened O_NOFOLLOW and must be a regular file of ours with mode 0600. The lock file is opened O_NOFOLLOW too. - After SQLite opens the database, and again after WAL mode creates the WAL and SHM, the directory and database must still be the same files (device and inode), and all three files private; anything else is refused before a fact is written. - The default ledger moves to ~/.local/state/tensorfold/living-weights-v2/ledger.sqlite3, a directory of its own, and the docs say what the journal requires. Regressions: a database swapped in before SQLite opens, one created in the gap after the directory check, a WAL planted before the open, a linked database, and a shared or linked directory are each refused, with nothing written to the planted file.
…d, /api/tags, /api/show, /api/ps, /api/version) over the native engine (#3117)
…, and the refused-field bodies are JSON Every test in ollama_e2e_test.zig hung until its 30 s poll timed out, then panicked in deinit. The cause is the fixture, not the routes: the fake engine sent prefilled and its three tokens but never `finished`. The engine API says every request ends with `finished` (core/engine_api.zig:37 and :420), and the server's generation loop (server/chat_generation.zig:216) ends a reply only when it arrives, so the reply, its socket and the open connection never ended. The other loopback fixtures (auth_scope_http_test.zig) send it. With that fixed, "format, tools and images are mapped or refused by name" failed on its own bodies: `format":"xml` and the other two fragments lacked their opening quote, so each request was refused as malformed JSON and the reply never named the field. They are now whole JSON fields, and each refusal is checked to name its field. test-ollama-e2e-host runs these loopback tests alone. No route or server code changes.
baidu/Unlimited-OCR-2 at its released bf16 weights on M5-class GPUs: the publisher's preprocessing (bit-exact), the SAM and CLIP image encoder, the DASH MoE decoder with greedy generation, chunked prefill (bit-identical at any cut), serving through tensorfold-native --vision with the family's chat template, and every kernel under the contract. The SAM encoder, the decoder's attention, its page2d rotary and its MoE gate are written from the papers and from the release's modules measured through their inputs and outputs. tf-uocr-blocks gates each encoder step alone against float32, tf-uocr-vision and tf-uocr-decode gate a page's stages, and tools/zig/uocr_reference.py makes their cases from the release's own code. Fed torch bf16's reply, tf-uocr-decode accepts a departure only as a tie: the same two contenders on both sides, each margin within one bf16 step (the reference records every step's two highest logits); --reply-reported reports them instead. The server's built-in templates share one helper (hf_text_parse.zig): Nemotron Parse's replaces the checkpoint's, this family's fills in when the checkpoint ships none, and a --chat-template wins over both.
Under slots_only without --prompt-cache-gib, plan sets aside the slots that fit past the engine's own window and each other stream's fixed part and page, TF_KV_SHARE at most, before windows and held count the rest. The rest of the default cache still becomes pages after the windows. Plans that already kept all n are unchanged, so is every plan without slots_only or with an asked cache. GLM's advice under 4 states a stream names TF_KV_SHARE alone where its slots now fit, and --prompt-cache-gib only beside a named cache too small for them.
A row said which products its entry takes by rows, items and pairs, and nothing about the weights' width. Timings on gfx1151 show a choice that turns with it: of the two K-parallel tiles a plain prompt of 1 row can run on, one is ahead on 2-bit and 3-bit codes and the other from 4-bit up. `max_bits` is the most bits a weight code holds for the row to take a product (0, the default: any). The registry's cost check refuses wider codes for plain products and plans alike. A bound that does not parse refuses its row instead of being ignored: - a bound that is no code width (2, 3, 4, 5, 6 or 8) stops the build of the table that writes it, naming the entry and the bound, where 7 or 9 would have read as "up to 6" or "any"; a row made by hand with such a bound takes no product. The registry and the kernel cases read the widths from one list, beside the row type; - `tuning.build` copies an entry's fields by name and dropped the ones the row type lacks, so a misspelled bound left its row taking every product. Such a field now stops the build, naming the entry and the field. Every field the four tables carry is the row type's. `hip-affine-tune` prints the bound with a row it proposes. No table sets the bound yet, so no choice changes. Refs #2923
…-parallel tile A plain prompt product of 1 or 2 rows ran on `kp0` (8 columns a lane set, 1 row a pass) on every card. On gfx1151 `kp3` (4 columns, 2 rows a pass) is ahead at 2 rows at every width, and at 1 row from 4-bit codes up; at 1 row of 2-bit or 3-bit codes `kp0` stays ahead. A resumed prompt ends in such a piece whenever 1 or 2 tokens follow the kept state. gfx1151's table now gives `kp0` one row of codes of at most 3 bits and `kp3` the other plain products of up to 2 rows. Costs stay as they were, so `kp0` keeps the products its row still takes and no row is shadowed. Nothing else is chosen differently: products of 3 rows and more, routed plans, decode and the other three tables keep their tiles. Both tiles are one template and write the 128-row tile's bytes. Tests: - launches: at every width gfx1151 picks `kp3` at 2 rows and, at 1 row, `kp0` for 2-bit and 3-bit codes and `kp3` from 4-bit up, with the matrix cores on or off; it keeps `kp1` and `kp2` past 2 rows and the reference tile with the dot2 GEMM tiles off; gfx1030, gfx1100 and gfx1201 still pick `kp0`. - tuning: gfx1151's bound is set, so a table that lost it fails. - kernel suite: `kp0`, `kp3` and the 128-row dot2 tile, launched by id whatever the card's table picks, write the same bytes at 1 and 2 rows, every width and group, fp32 and rounded into out16; an output left unwritten fails too. Refs #2923
Owner
Author
|
Stopped: the train this one was stacked on did not land whole; its PRs go back to the queue. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI for a landing train: #2074 #3125 #3273 stacked on main in that order. Nothing merges from this PR: each PR above lands on its own once this tip is green, and this PR is then closed.