[cshintov]

What this changes

On Metal, tensorfold info printed a context-free "memory floor" (weights plus 2 GiB) for Qwen3.8-27B. tensorfold info MODEL [--context N] [--parallel N|auto] now prints the charge serve makes at startup, computed by the same code, and whether it fits this Mac.

• Issue: Refs Metal: serve has no memory admission, so a 262k-context start on a swapping Mac reserves 67 GB in seconds and the kernel watchdog panics #2761 (item 1; the startup warning is fix(qwen27): warn when Metal startup exceeds available memory #2886).
• Stacked on: main.
• atc overlap --json: not run (atc isn't installed here). By hand, open PRs that touch the same files: fix(qwen27): warn when Metal startup exceeds available memory #2886 adds lines right after the fit_prompt lines this moves in qwen27_served.zig and keeps working against reserve.fit (it still returns a prompt_memory.Fit); train 1113142824: #2921 (CI only) #3229, train 1113140943: #2868 #2875 #2723 #2902 #2903 #2906 #2861 #2570 (CI only) #3228, qwen27 metal: M5 takes TF_QWEN27_METAL_KV_PAGES; a paged prompt chunk finds its two pages once #3180, metal: presence and frequency penalties on Flash Next and Qwen3.8-27B, each row over the reply and its own path #3145, environment: Metal Qwen paging, slot resume and slide checks refused by name (#2370) #3018 and qwen27 served: the reply-end keep test names its history, and a step runs it #3012 touch other parts of qwen27_served.zig; Nemotron on Metal: warm turns, so a long follow-up resumes past the previous prompt #3190, cli: tensorfold device says what this machine runs, saved in ~/.tensorfold/device.json #3107, lanes: Nemotron plain streams share one exact Metal round (+20% at 2 streams, +77% at 4) #3005 and llama on Metal: Qwen2 checkpoints, Bimo-7B-CyberAgent's bf16 release with its q, k and v biases and YaRN at HF's rounding points #2890 touch other parts of metal.zig, and cli: tensorfold device says what this machine runs, saved in ~/.tensorfold/device.json #3107 a different command in cli.zig.

Per file:

• cli/info.zig, cli/cli.zig: the flags read as serve reads them (--context defaults to 32,768, 1 to 262,144 and within max_position_embeddings; --parallel defaults to auto, 8 that shrink to what fits, and a number asks for exactly that many). For a checkpoint the Metal engine opens on the 27B-class host, two lines replace the floor:

  $ tensorfold info TensorFold/Qwen3.8-27B-MLX-4bit --context 32768 --parallel 1
  ...
  startup budget: 21.00 GiB at --context 32768: weights, 1 stream of KV and state, 512-row prompt scratch (no drafter or images)
  this Mac: fits beside its 51.8 GiB Metal working set and an 8 GiB margin
  

  The second line also says when serve would start fewer streams, or refuse. A checkpoint the engine can't price (a shape the attention plan rejects, for example) prints startup budget: not priced (TargetAttentionShape) and keeps the floor.

• native/qwen27_reserve.zig (new): the fit serve charges (fit), the prompt rows it takes (rows, requested), the working-set cap, and budget, which prices a checkpoint unloaded. qwen27_served.zig calls fit, rows, requested and cap from here instead of building its own.

• native/metal.zig: the "opens on the 27B host" test and the context rule become ownsQwen27 and qwen27Context, used by open and by the new qwen27Budget.

• cli/hub.zig: modelType returned a slice into the JSON parse it had just freed; it now copies it into the caller's allocator (every caller passes an arena).

Design

Choice: info calls Fit.cost through the same fit and rows that serve calls. One input is a stand-in: unloaded, the model's bytes are its weight files plus the frame the deferred load allocates, where serve reads device.allocated(). Those bytes decide only the room the fit is checked against, never the charge itself.

Alternatives considered:

• A second formula in info (layers x context x KV width): the drift Metal: serve has no memory admission, so a 262k-context start on a swapping Mac reserves 67 GB in seconds and the kernel watchdog panics #2761 warns about, and it misses the GDN state, prompt scratch and tap planes that runnerBytes counts.
• Load the model in info and read device.allocated(): exact, but it costs the weights' memory and seconds, which defeats a pre-start check on a Mac already short of memory.
• One line for both chip classes without opening a device: frame rows, window and shared prefix differ by chip. Opening the default Metal device takes milliseconds and no weights.

Deferred: --drafter (the ring and tap planes already go through fit; info would need the drafter's config and size) and images (the tower loads before the slots). The line says "no drafter or images".

Checks on this head

• Head commit: b1f55379c on main aaac299.
• Lean checker on the base and on this head: 341 and 339, 0 new (the two getenvs that moved now read through platform/environment.zig).
• Tests: zig build test-cli 40 pass, 15 skip, 0 fail. New: "info --context and --parallel parse as serve reads them" and "info prices a Qwen3.8-27B start from serve's charge" (8,192 against 262,144 tokens, 1, 2 and auto streams, a capped working set that fits no stream, a context past 262,144, a shape the engine refuses). Both fail on main. With hub.modelType put back as on main, the pricing test fails too: the second info in the process reads a freed model_type. zig build compile-metal-residency-host builds; zig build native -Dcpu=apple_m1 builds.
• CI: pending.
• Not run, and why: no M5 here (tensor units: 2,048 prompt rows and the shared-prefix geometry). That path differs only in the device facts budget reads, the same ones serve reads.

Receipt

• Environment: this change on main c34ff56 (rebased since onto aaac299 with no conflict outside CHANGELOG; tests above rerun on the new head), Zig 0.17.0, Xcode Metal toolchain, Apple M4 Pro, 64 GB (51.8 GiB Metal working set), macOS 26.5. Checkpoint TensorFold/Qwen3.8-27B-MLX-4bit @ 22d8d538154e0e5b6f8dcb1fcc74a60a0798104c.

• info against real starts. serve ... --parallel 1 --context N --prompt-cache-gib 0: its qwen27: prompt capacity=... reserve=... line, and tensorfold:process_footprint_bytes from /metrics after one 4-token greedy request:

  --context	info budget	serve rows	serve reserve	serve footprint	info minus footprint
  8,192	19.50 GiB	512	4,636,573,696 B	20,529,655,400 B (19.12 GiB)	+0.38 GiB
  32,768	21.00 GiB	512	6,247,186,432 B	22,138,383,976 B (20.62 GiB)	+0.38 GiB
  262,144	not one stream fits			refuses at start	agrees

  The step from 8,192 to 32,768 tokens is 1.50 GiB in info, in the logged reserve and in the footprint. The 0.38 GiB gap doesn't move with the context, so it sits in the fixed stand-in, not in the charge, and errs high. At 262,144 tokens serve exits: "--parallel 1 needs 33.9 GiB for its streams and 512-row prompt scratch at a 262144-token context; 29.5 GiB remains beside the weights and an 8 GiB margin".

• Main against branch, serve ... --parallel 1 --context 8192: the same start on both, qwen27: prompt capacity=512 rows, startup stream and scratch reserve=4636573696 bytes and Qwen3.8-27B: 1 stream of 8192 tokens; footprint after one request 20,527,967,824 B on main and 20,526,722,616 B on the branch.

• Exactness, decode and prompt speed: n/a. Nothing after startup changes: serve's fit and rows are the same expressions, moved.

• New switch: info --context N and --parallel N|auto, defaults 32,768 and auto, as serve.

• Licence of any new weights: n/a.

[ashhart]

Hi @cshintov, thanks. Tying info to the same fit/rows/cost that serve charges is the right way to stop #2761 drifting, and the unloaded stand-in is clearly labelled. Host review found nothing to fix:

1. git diff --check, zig fmt --check build.zig zig, zig ast-check on all seven changed .zig files and zigb test-cli-memory-host are clean. Lean goes from 341 to 339 with no new problems (the two getenvs now read through sys.environment). The patch applies cleanly to main.
2. zigb test-cli and zigb compile-metal-residency-host are registered only for macOS (build.zig:175 and :130), so your two new tests did not run in our Linux pass. We'll run both on an M5 Ultra, which also covers the tensor-unit geometry (2,048 prompt rows, shared prefix); your receipt is from an M4 Pro.
3. One wording ask: please soften the hub.modelType evidence line in the body. With parseFromSlice's default .alloc_if_needed and every CLI caller passing init.arena.allocator(), the old t.string points into a buffer that is never freed on that path, and the parse's deinit() frees are arena no-ops. So restoring main's version does not by itself make the pricing test fail. The dupe is still right; a small test with a freeing allocator would back the claim if you want to keep it.
4. The flag parsing matches server/cli.zig:513-523 and int(), and the 1.50 GiB step from 8,192 to 32,768 tokens matches 16 attention layers × 4×256 KV. All files are under 600 lines.

Next step on our side: the M5 run of test-cli and compile-metal-residency-host, plus a spot check of the printed budget against a real serve start.

• Synthena (Ash's Agent)

[ashhart]

Follow-up on our side, at your head 6808133:

• zigb test-cli and zigb compile-metal-residency-host both pass on macOS, so your two new tests ran.
• The served A/B on an M5 Ultra (main against this PR, Qwen3.8-27B) is GREEN with the same tokens. Speed is level, as expected for a change to info alone.

It's queued to land. Softening the hub.modelType line in the body would still be welcome, but it doesn't block.

• Synthena (Ash's Agent)