Skip to content

docs: the model recipes' native-server claims and commands, checked against the code (fifteen fixes) - #2361

Merged
ashhart merged 15 commits into
mainfrom
lead/docs-recipes-q28
Nov 7, 2026
Merged

ashhart merged 15 commits into
mainfrom
lead/docs-recipes-q28

Conversation

@ashhart

@ashhart ashhart commented Nov 6, 2026 •

Copy link
Copy Markdown
Owner

The model recipes against today's code: twelve wrong claims about the native server fixed across ten pages. Each commit cites the code that settles it. Python 0.6.6 sections and measured numbers are unchanged.

Page It said The code
gemma-4-native.md, gemma-4-12b-native.md M1 to M4 are refused Served through the row path; only the drafters refuse (families/gemma4/model.zig:92, weights.zig:234)
gemma-4-native.md tool_choice: "required" and a named function get HTTP 400 Enforced through the token grammar, which writes Gemma's own call markup (server/server.zig:318, tool_grammar.zig's .gemma style)
qwen3.5-2b-native.md zig build ... check-generated No such step exists under zig/build/; dropped
qwen3.6-moe.md tensorfold pull and info refuse the checkpoint on Metal They read it, since the hub registers the model type. Metal has no qwen3_5_moe family, so it can't serve it
nemotron-3.5.md The native binary refuses --tp, --rank and --master The CLI reads them, but Nemotron has no tensor-parallel ranks on CUDA, so --tp above 1 is refused by name (native/cuda_refusal.zig:55)
nemotron-3.5.md A request reserves +16 rows past its reply +32, the widest verify window (cuda_state.zig:14). The worked example still gives 13 pages
exl3.md, exl3-universal-experts.md No model serves EXL3, and #741 is in review CUDA's EXL3 operators, projections and grouped routed experts landed in 1.1.0 (#740, #741, #806, #837); GLM-5.3-Flash runs its routed experts through them (families/glm/cuda_native.zig:22) and qwen3_5 serves dense EXL3 packs (families/qwen3_5/cuda_native.zig:29)
cuda.md The CUDA CLI doesn't apply --alias It does (server/serve.zig:141)
adding-a-zig-family.md Flash Next on Metal serves one reply at a time That's the default. An explicit --parallel above one on one Mac serves it through lane_host (families/flashnext/batch_meta.zig's shared, native/flashnext_batch_host.zig)
adding-a-zig-family.md Draft levels 3, 6 and 15; MTP chains stop at 6 3, 6 and 8 on one Mac (3 or 7 on a pair), with the thresholds in families/flashnext/drafts.zig. Metal chains run up to 8; CUDA defaults to 6 (8 greedy), capped at 15
adding-living-weights-to-a-family.md Nemotron is the only family in living/families.zig It lists four. Nemotron's is the learning that landed on Metal (native/metal.zig:15)

Three more, from a check of every command the docs tell a reader to run:

  • docs/api.md and the XPU recipe: the grammar library installs into zig-out/native/lib on every OS (zig/build/grammar.zig:44-46), not zig-out/lib. The server looks for it there, through ../lib from zig-out/native/bin.
  • qwen3.5-2b-native.md's check block loses four things that are gone: the editable pip install (the setup file is a stub that refuses), the check-generated step, capture_qwen35.py (its import has no module) and pytest tests. What it actually needs, mlx-lm, is installed instead.

Docs only.

Revision (6 Nov)

An independent read found three more errors, now fixed:

@ashhart ashhart changed the title docs: the model recipes' native-server claims, checked against the code (ten fixes) docs: the model recipes' native-server claims, checked against the code (twelve fixes) Nov 6, 2026
@ashhart ashhart changed the title docs: the model recipes' native-server claims, checked against the code (twelve fixes) docs: the model recipes' native-server claims and commands, checked against the code (fifteen fixes) Nov 6, 2026
@ashhart

ashhart commented Nov 6, 2026

Copy link
Copy Markdown
Owner Author

/atc exchange 65f0de9478ffeb5126de0fee2bd7dfb0596690a60563ba569cd3528fa5b038fb

…l calls

docs/recipes/gemma-4-native.md:94 said M1 to M4 are "Refused: the lane
projection needs M5 tensor units". Pre-M5 chips load and serve through the
row path: zig/src/families/gemma4/model.zig:92-93 ("Before it the row path
serves the projections") and weights.zig:234 (packRows when the device has
no tensor units); only the drafters refuse (assistant.zig:95, dflash.zig:89).
The row now says served through the row path, drafters refuse.

:84-85 said tool_choice "required" and a named function get HTTP 400 because
the grammar for them reads <tool_call> markup. The engine has token-grammar
support (zig/src/native/gemma4.zig:132, .structures = true), the server
enforces required calls whenever the engine does (zig/src/server/server.zig:318-321),
and the grammar has a .gemma style writing Gemma's <|tool_call>call: markup
(zig/src/server/tool_grammar.zig:30,156). Now says enforced through the token
grammar, which writes Gemma's markup.
docs/recipes/gemma-4-12b-native.md:63 said M1 to M4 are "Refused: the lane
projection needs M5 tensor units". Pre-M5 chips load and serve through the
row path: zig/src/families/gemma4/model.zig:92-93 ("Before it the row path
serves the projections") and weights.zig:234 (packRows when the device has
no tensor units); only the drafters refuse (assistant.zig:95, dflash.zig:89).
Same wording as the 31B page's row.
docs/recipes/qwen3.5-2b-native.md:77 ran `zig build test test-native
check-generated`, but no step named check-generated exists anywhere under
zig/build/ (every b.step call grepped). The generated-ops check is the
tf-qwen35-check flow the next two lines already run. The phantom step is
dropped; test and test-native remain.
…rve it

docs/recipes/qwen3.6-moe.md:10 said `tensorfold pull` and `tensorfold info`
refuse the checkpoint on Metal. The hub registers the model type
(zig/src/cli/hub.zig:17), and pull and info serve registered families
(zig/src/cli/pull.zig:74-77, zig/src/cli/info.zig:35-36); only an
unregistered model type refuses. What Metal lacks is the qwen3_5_moe family,
so it cannot serve the checkpoint there. The sentence now says that, with
pull and info still reading it.
…reserves +32 pages

docs/recipes/nemotron-3.5.md:54-55 said the native binary refuses the Python engine's
--tp, --rank, --master and --prefill-fp8. The CLI reads --tp, --rank and --master
(zig/src/server/cli.zig:130-132), but Nemotron's CUDA family has no rank option, so
--tp above 1 is refused by name (zig/src/native/cuda_refusal.zig:54-55, no_ranks);
--prefill-fp8 has no .native and is refused as an unknown flag. The sentence now says so.

:231 said a request reserves ceil((prompt + max_tokens + 16) / 512) pages.
The reserve adds 32: zig/src/families/nemotron/cuda_pages.zig:15-16 and
cuda_state.zig:14. The worked example still holds (ceil((2048+4096+32)/512)
= 13 pages).
docs/recipes/exl3.md:10-13 said no model serves an EXL3 checkpoint in this
release and the 27B on CUDA is the first planned. The page's own "Using it"
section (:218) says On CUDA, qwen3_5 serves EXL3 packs of the Qwen3.5 MoE
models and of the dense Qwen3.8-27B, and the family admits the format
(zig/src/families/qwen3_5/cuda_native.zig:28 lists "exl3"). The bullet now
points at that serving.

:12 also listed #741 as in review. The grouped routed experts landed:
zig/src/families/qwen3_5/cuda_exl3_experts.zig exists and cuda_native.zig:29
serves through exl3_experts. #741 moved to a landed note; #740 and #806 stay
as in review (not checked either way).
docs/recipes/exl3-universal-experts.md:8 said the native engine's CUDA
routed experts on this format are in review (#741). They landed:
zig/src/families/qwen3_5/cuda_exl3_experts.zig exists and
zig/src/families/qwen3_5/cuda_native.zig:29 serves EXL3 packs through
exl3_experts. The sentence now says landed, with qwen3_5 serving packs
through them.
docs/recipes/cuda.md:355 said the CUDA CLI does not apply --alias. It does:
zig/src/server/cli.zig:51 parses it into repeated.alias and
zig/src/server/serve.zig:141 applies the aliases to the served model IDs.
The sentence now says the CUDA CLI applies --alias too, with --name setting
the served model ID.
… the one that landed

docs/recipes/adding-living-weights-to-a-family.md:8 said Nemotron is the
only family listed in living/families.zig. The file lists four:
nemotron_h, nemotron_h_omni, qwen4_exp and qwen3_5
(zig/src/living/families.zig:7,32,40,56). The bullet now says Nemotron is
the first family and the only one whose learning has landed, and names the
other three listings. The learning claim itself stands: the Metal slide
registry serves Nemotron alone (zig/src/native/metal.zig:15,
metal_slide.Registry(.{nemotron})).
…--parallel, and the depth rule runs 3/6/8

docs/recipes/adding-a-zig-family.md said Flash Next on Metal "serves one reply at a time" unconditionally.
That's the default only: batch_meta.zig's `shared` rule (an explicit --parallel above one, one Mac, no
learning) sends Metal to native/flashnext_batch_host.zig's openConfigured, which serves it through
core/lane_host.zig. Line 41 and the Serving row now say so.

The depth rule said three levels, 3, 6 and 15, with MTP chains stopping at 6. families/flashnext/drafts.zig
has 3, 6 and 8 on one Mac (3 or 7 on a pair): up to 6 above 0.8 and 8 above 0.74, back to 6 below 0.62 and
3 below 0.65, the widest while nearly all drafts land. Metal's batch host drafts 8 (batch_costs.zig:140);
on CUDA cuda_mtp.zig defaults to 6, 8 for greedy streams, with a cap of 15. The measured receipts below the
rule are unchanged.
api.md:425 said `zig build grammar` installs libtfgrammar into
`zig-out/native/lib` on macOS and `zig-out/lib` on Linux. The install
destination is `native/lib` under the prefix on every OS
(zig/build/grammar.zig:44-46), and the server looks in the binary
directory's ../lib (zig/src/platform/optional_libraries.zig:129), which
is `zig-out/native/lib` on Linux too.
The recipe said `zig build grammar` installs under `<prefix>/lib` and
named `zig-out/lib/libtfgrammar.so`. The install destination is
`native/lib` under the prefix on every OS (zig/build/grammar.zig:44-46),
so `--prefix zig-out/xpu` lands the library in
`zig-out/xpu/native/lib`, not the `zig-out/xpu/lib` the XPU server's
../lib lookup reads (zig/src/platform/optional_libraries.zig:129); the
copy from `zig-out/native/lib` is the working step.
- `pip install -e '.[test,tui]'`: setup.py:1-6 is a stub that refuses
  installs; the checks' Python dependency is mlx-lm, which
  qwen35_fidelity.py:12-14 and capture tools import
- `check-generated`: no build file defines the step (dropped from
  docs before, 849812afa)
- `tools/zig/capture_qwen35.py`: its `from gen_qwen35_kernels import
  specs` (capture_qwen35.py:15) has no module anywhere in the tree, so
  the capture command fails; tf-qwen35-check's fixtures argument is
  optional (zig/tests/qwen35_check.zig:8), so the check runs on the
  model alone
- the closing `pytest -q tests`: no tests/ directory and no
  TENSORFOLD_QWEN35_MODEL reader exist
@ashhart
ashhart force-pushed the lead/docs-recipes-q28 branch from 7527a4f to db986ac Compare November 7, 2026 02:58
@ashhart
ashhart merged commit cc2dec7 into main Nov 7, 2026
0 of 7 checks passed
@ashhart
ashhart deleted the lead/docs-recipes-q28 branch November 10, 2026 16:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant