Repository navigation
docs: the model recipes' native-server claims and commands, checked against the code (fifteen fixes) - #2361
Merged
Conversation
Owner
Author
|
/atc exchange 65f0de9478ffeb5126de0fee2bd7dfb0596690a60563ba569cd3528fa5b038fb |
…l calls
docs/recipes/gemma-4-native.md:94 said M1 to M4 are "Refused: the lane
projection needs M5 tensor units". Pre-M5 chips load and serve through the
row path: zig/src/families/gemma4/model.zig:92-93 ("Before it the row path
serves the projections") and weights.zig:234 (packRows when the device has
no tensor units); only the drafters refuse (assistant.zig:95, dflash.zig:89).
The row now says served through the row path, drafters refuse.
:84-85 said tool_choice "required" and a named function get HTTP 400 because
the grammar for them reads <tool_call> markup. The engine has token-grammar
support (zig/src/native/gemma4.zig:132, .structures = true), the server
enforces required calls whenever the engine does (zig/src/server/server.zig:318-321),
and the grammar has a .gemma style writing Gemma's <|tool_call>call: markup
(zig/src/server/tool_grammar.zig:30,156). Now says enforced through the token
grammar, which writes Gemma's markup.
docs/recipes/gemma-4-12b-native.md:63 said M1 to M4 are "Refused: the lane
projection needs M5 tensor units". Pre-M5 chips load and serve through the
row path: zig/src/families/gemma4/model.zig:92-93 ("Before it the row path
serves the projections") and weights.zig:234 (packRows when the device has
no tensor units); only the drafters refuse (assistant.zig:95, dflash.zig:89).
Same wording as the 31B page's row.
docs/recipes/qwen3.5-2b-native.md:77 ran `zig build test test-native check-generated`, but no step named check-generated exists anywhere under zig/build/ (every b.step call grepped). The generated-ops check is the tf-qwen35-check flow the next two lines already run. The phantom step is dropped; test and test-native remain.
…rve it docs/recipes/qwen3.6-moe.md:10 said `tensorfold pull` and `tensorfold info` refuse the checkpoint on Metal. The hub registers the model type (zig/src/cli/hub.zig:17), and pull and info serve registered families (zig/src/cli/pull.zig:74-77, zig/src/cli/info.zig:35-36); only an unregistered model type refuses. What Metal lacks is the qwen3_5_moe family, so it cannot serve the checkpoint there. The sentence now says that, with pull and info still reading it.
…reserves +32 pages docs/recipes/nemotron-3.5.md:54-55 said the native binary refuses the Python engine's --tp, --rank, --master and --prefill-fp8. The CLI reads --tp, --rank and --master (zig/src/server/cli.zig:130-132), but Nemotron's CUDA family has no rank option, so --tp above 1 is refused by name (zig/src/native/cuda_refusal.zig:54-55, no_ranks); --prefill-fp8 has no .native and is refused as an unknown flag. The sentence now says so. :231 said a request reserves ceil((prompt + max_tokens + 16) / 512) pages. The reserve adds 32: zig/src/families/nemotron/cuda_pages.zig:15-16 and cuda_state.zig:14. The worked example still holds (ceil((2048+4096+32)/512) = 13 pages).
docs/recipes/exl3.md:10-13 said no model serves an EXL3 checkpoint in this release and the 27B on CUDA is the first planned. The page's own "Using it" section (:218) says On CUDA, qwen3_5 serves EXL3 packs of the Qwen3.5 MoE models and of the dense Qwen3.8-27B, and the family admits the format (zig/src/families/qwen3_5/cuda_native.zig:28 lists "exl3"). The bullet now points at that serving. :12 also listed #741 as in review. The grouped routed experts landed: zig/src/families/qwen3_5/cuda_exl3_experts.zig exists and cuda_native.zig:29 serves through exl3_experts. #741 moved to a landed note; #740 and #806 stay as in review (not checked either way).
docs/recipes/exl3-universal-experts.md:8 said the native engine's CUDA routed experts on this format are in review (#741). They landed: zig/src/families/qwen3_5/cuda_exl3_experts.zig exists and zig/src/families/qwen3_5/cuda_native.zig:29 serves EXL3 packs through exl3_experts. The sentence now says landed, with qwen3_5 serving packs through them.
docs/recipes/cuda.md:355 said the CUDA CLI does not apply --alias. It does: zig/src/server/cli.zig:51 parses it into repeated.alias and zig/src/server/serve.zig:141 applies the aliases to the served model IDs. The sentence now says the CUDA CLI applies --alias too, with --name setting the served model ID.
… the one that landed
docs/recipes/adding-living-weights-to-a-family.md:8 said Nemotron is the
only family listed in living/families.zig. The file lists four:
nemotron_h, nemotron_h_omni, qwen4_exp and qwen3_5
(zig/src/living/families.zig:7,32,40,56). The bullet now says Nemotron is
the first family and the only one whose learning has landed, and names the
other three listings. The learning claim itself stands: the Metal slide
registry serves Nemotron alone (zig/src/native/metal.zig:15,
metal_slide.Registry(.{nemotron})).
…--parallel, and the depth rule runs 3/6/8 docs/recipes/adding-a-zig-family.md said Flash Next on Metal "serves one reply at a time" unconditionally. That's the default only: batch_meta.zig's `shared` rule (an explicit --parallel above one, one Mac, no learning) sends Metal to native/flashnext_batch_host.zig's openConfigured, which serves it through core/lane_host.zig. Line 41 and the Serving row now say so. The depth rule said three levels, 3, 6 and 15, with MTP chains stopping at 6. families/flashnext/drafts.zig has 3, 6 and 8 on one Mac (3 or 7 on a pair): up to 6 above 0.8 and 8 above 0.74, back to 6 below 0.62 and 3 below 0.65, the widest while nearly all drafts land. Metal's batch host drafts 8 (batch_costs.zig:140); on CUDA cuda_mtp.zig defaults to 6, 8 for greedy streams, with a cap of 15. The measured receipts below the rule are unchanged.
api.md:425 said `zig build grammar` installs libtfgrammar into `zig-out/native/lib` on macOS and `zig-out/lib` on Linux. The install destination is `native/lib` under the prefix on every OS (zig/build/grammar.zig:44-46), and the server looks in the binary directory's ../lib (zig/src/platform/optional_libraries.zig:129), which is `zig-out/native/lib` on Linux too.
The recipe said `zig build grammar` installs under `<prefix>/lib` and named `zig-out/lib/libtfgrammar.so`. The install destination is `native/lib` under the prefix on every OS (zig/build/grammar.zig:44-46), so `--prefix zig-out/xpu` lands the library in `zig-out/xpu/native/lib`, not the `zig-out/xpu/lib` the XPU server's ../lib lookup reads (zig/src/platform/optional_libraries.zig:129); the copy from `zig-out/native/lib` is the working step.
- `pip install -e '.[test,tui]'`: setup.py:1-6 is a stub that refuses installs; the checks' Python dependency is mlx-lm, which qwen35_fidelity.py:12-14 and capture tools import - `check-generated`: no build file defines the step (dropped from docs before, 849812afa) - `tools/zig/capture_qwen35.py`: its `from gen_qwen35_kernels import specs` (capture_qwen35.py:15) has no module anywhere in the tree, so the capture command fails; tf-qwen35-check's fixtures argument is optional (zig/tests/qwen35_check.zig:8), so the check runs on the model alone - the closing `pytest -q tests`: no tests/ directory and no TENSORFOLD_QWEN35_MODEL reader exist
… it; the depth rule as drafts.zig moves it
ashhart
force-pushed
the
lead/docs-recipes-q28
branch
from
November 7, 2026 02:58
7527a4f to
db986ac
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The model recipes against today's code: twelve wrong claims about the native server fixed across ten pages. Each commit cites the code that settles it. Python 0.6.6 sections and measured numbers are unchanged.
gemma-4-native.md,gemma-4-12b-native.mdfamilies/gemma4/model.zig:92,weights.zig:234)gemma-4-native.mdtool_choice: "required"and a named function get HTTP 400server/server.zig:318,tool_grammar.zig's.gemmastyle)qwen3.5-2b-native.mdzig build ... check-generatedzig/build/; droppedqwen3.6-moe.mdtensorfold pullandinforefuse the checkpoint on Metalqwen3_5_moefamily, so it can't serve itnemotron-3.5.md--tp,--rankand--master--tpabove 1 is refused by name (native/cuda_refusal.zig:55)nemotron-3.5.md+16rows past its reply+32, the widest verify window (cuda_state.zig:14). The worked example still gives 13 pagesexl3.md,exl3-universal-experts.mdfamilies/glm/cuda_native.zig:22) andqwen3_5serves dense EXL3 packs (families/qwen3_5/cuda_native.zig:29)cuda.md--aliasserver/serve.zig:141)adding-a-zig-family.md--parallelabove one on one Mac serves it throughlane_host(families/flashnext/batch_meta.zig'sshared,native/flashnext_batch_host.zig)adding-a-zig-family.mdfamilies/flashnext/drafts.zig. Metal chains run up to 8; CUDA defaults to 6 (8 greedy), capped at 15adding-living-weights-to-a-family.mdliving/families.zignative/metal.zig:15)Three more, from a check of every command the docs tell a reader to run:
docs/api.mdand the XPU recipe: the grammar library installs intozig-out/native/libon every OS (zig/build/grammar.zig:44-46), notzig-out/lib. The server looks for it there, through../libfromzig-out/native/bin.qwen3.5-2b-native.md's check block loses four things that are gone: the editablepip install(the setup file is a stub that refuses), thecheck-generatedstep,capture_qwen35.py(its import has no module) andpytest tests. What it actually needs,mlx-lm, is installed instead.Docs only.
Revision (6 Nov)
An independent read found three more errors, now fixed:
exl3.mdcalled cuda: EXL3 projections behind qlinear (#593) #740 and cuda: EXL3 projections write fp32 too (qlinear decodeF32 and promptF32) #806 "in review"; both landed in 1.1.0 with cuda: EXL3 routed experts behind experts.Grouped (#593, stacked on #740) #741 and cuda: EXL3 linear and grouped routed experts, every codebook and width, a width per expert (#593) #837 (CHANGELOG's 1.1.0 EXL3 line).exl3-universal-experts.mdsaidqwen3_5runs routed experts through it.qwen3_5on CUDA is the dense family; it serves dense EXL3 packs.adding-a-zig-family.mdsaid the widest draft level "runs only while nearly all drafts land".flashnext/drafts.zigholds depth 8 while the moving average stays at 0.62 or above (0.8 for depth 7 on a pair); the page now gives each move in order.