Skip to content

Implement GLP control vectors with a fused Metal kernel for Qwen3.8-27B - #1835

Open
BlivionIaG wants to merge 5 commits into
ashhart:mainfrom
BlivionIaG:feature/glp-steering
Open

BlivionIaG wants to merge 5 commits into
ashhart:mainfrom
BlivionIaG:feature/glp-steering

Conversation

@BlivionIaG

@BlivionIaG BlivionIaG commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

What this changes

Off-by-default, add-only generic control vectors. The maintainer accepted this code; the device evidence below is on the rebased head.

  • Issue: Refs GLP control vector support for Qwen3.8-27B on Metal #1834; completion depends on the maintainer's own Metal runs of the 20 rows and the stream sweep.
  • Stacked on: main 6c5cef8941b27d86b06a986834e13deb9b515e1e; the branch is rebased onto it and adds 5 commits. The only conflict was CHANGELOG.md.
  • atc overlap --json: ATC0.9.6 was run against upstream with --base fb8fdf3; complete=false (myPullRequestsLimit, otherPullRequestsLimit, pullFilesLimit, pullFilesUnavailable, timeLimit). Supplemental complete snapshot:217foreign open heads,3038 exact local merge comparisons including latest Main ff742df; existing reviewed overlaps are retained, not claimed globally clean. The avoidable added environment-test collision with CUDA (GB10): a second carveout source, spark-attic's atticd, opt-in #2762 was removed by preserving Main's existing test name, without importing its unmerged feature.

Checks on this head

  • Head commit: c5947a8d2e9543e6c6a11bd30dc4c43c32f3b0fb.
  • Lean checker (python3 tools/zig/lean_check.py): head 345, main 345, same problem set, 0 new. No touched file is over 600 lines; zig fmt --check build.zig zig is clean.
  • Tests run: zig build test test-golden dev-tools -Dcpu=apple_m1 on c10442d1e (Apple M4 Pro, Zig 0.17.0): 13,265 of 13,369 tests pass, 103 skipped, 1 failed. The failure is dashboard_test ("Paged prefix cache"), which fails the same way on main. After the last commit, zig build test-steer test-glp test-glp-serve test-glp-steer passes 88 of 88 tests.
  • CI on c10442d1e: 12 successful checks, 2 scheduled-only skips, desktop and desktop-linux included. CI on the head below is running.
  • Not run: CUDA and ROCm (the engine refuses --glp there by name), and an M5 Ultra run of the 20 rows and the sweep, which the maintainer runs.

Receipt

  • Main against branch, no --glp (Apple M4 Pro, 48 GiB, TensorFold/Qwen3.8-27B-MLX-4bit at 22d8d538154e): the 16 prompts of tools/bench_prompts.json, greedy, 64 tokens, 16 streams; every row's token_sha256 is equal across the merge base 8a7838a65 and the earlier head 731afc275, each served twice. Earlier, on 36fa3b08c against 981fe201d, 14 of 14 cells were equal and within 2.3% (comment of 5 Nov). The 20 long-prompt rows and the 1-8 stream sweep on c10442d1e are the maintainer's; his M5 Ultra cell on 5 Nov was GREEN.
  • Boxes it reaches: Metal on Qwen3.8-27B steers; every other family, a Gemma checkpoint and CUDA or ROCm are refused by name.
  • New switch? Default, accepted values, profile row: no new serving setting/default/profile activation in this review fix/restack. Original settings and accepted forms are in the exact-head docs/environment.md and platform/environment.zig; no experimental setting is newly enabled by this update.
  • Environment: head c5947a8d2e9543e6c6a11bd30dc4c43c32f3b0fb; Zig 0.17.0; Apple M4 Pro, 48 GiB, Metal; checkpoint TensorFold/Qwen3.8-27B-MLX-4bit at 22d8d538154e0e5b6f8dcb1fcc74a60a0798104c.
  • Paths/platforms run versus unrun: Metal on an Apple M4 Pro for the rows above; CUDA and ROCm refuse --glp by name and were not run.
  • Exactness, --alone --serial/token_sha versus drafts-off: no vector, 16 of 16 rows equal across builds (above); the 5 Nov run on 36fa3b08c also passed the alone and serial checks, 18 of 18 and 4 of 4.
  • Style vector: a pirate-voice add vector derived for this check (24 prompts, layers 24 to 40), served at --glp-alpha 0.1; the startup line and the with/without replies are in the 12 Nov comment, and the recipe is in a comment. The file is not part of this PR.
  • Decode speed before/after: off/on/off/on at alpha 0.1 on this machine, -0.84% and +3.03% on two prompts (signs differ, so run-to-run noise); no steering cost is claimed.
  • Prompt speed/cold TTFT before/after: not measured by me on this head; no TTFT/ITL or prefill improvement claimed.
  • Licence of new weights/drafter: n/a — no new weights or drafter distributed by this Core/app/control-vector change.
  • Kernel contract (Apple M4 Pro, zig build tf-kernel-contract, then tf-kernel-contract --only steer): SUMMARY 1 kernels: 1 pass gates 1-4, 0 fail, 0 known findings; 1 controls: 1 red as planted, 0 missed; 0 limit problems, exit 0. The first run failed gate 5 because the steering launch declared 40 threadgroup bytes where the compiler allocates 48; c5947a8d2 declares 48. M5 and other chips are not run.

Remaining maintainer requests

The maintainer's 20 rows and 1-8 stream sweep on c10442d1e against 6c5cef894, and his build of the style vector from the recipe in the comments.

Checklist

  • Title/description, head and stack agree; older results remain source-bound.

  • Every new/restored test has a failing-before runtime receipt across all reached platforms. Host evidence is above; missing device gates are not waived.

  • No newly added overlong/multiline source comments, measurements or review history; inherited findings are reported above.

  • Complete publication trees/outgoing histories reviewed for public noreply identity, tool trailers and privacy; synthetic detector fixtures and mandatory third-party public contacts are not live secrets. No package/binary upload proposed.

  • All inference-path/lane-round gates qualified on this head.

  • Drafted=plain, restored=fresh, concurrent=solo on required devices/settings.

  • Required precision/reference evidence on this exact head complete.

  • Decode/prompt no slower than Main at the required widths, greedy and sampled.

Historical source-bound notes — not current-head qualification

Earlier commands/measurements retain their original commit/binary scope. Any earlier readiness or provenance claim is superseded by the open gates above.

Add-only control vectors

  • Latest maintainer routing/gate: comment 6840110713: the maintainer runs the 20 rows and the sweep on c10442d1e against 6c5cef894; the style vector is built from the recipe posted in the comments, or fetched from a public model repo at a fixed revision.

Rebased onto main 6c5cef8941b27d86b06a986834e13deb9b515e1e; published head c5947a8d2e9543e6c6a11bd30dc4c43c32f3b0fb.

Implements the GLP format defined by the weightless project (@msuiche), with an operator-supplied file for non-refusal style or topic steering. No refusal-removal vectors, prompt sets, derived assets, download links or old measurement receipts are included.

  • Off by default: without --glp FILE, no vector is applied. --glp-alpha overrides the file strength.
  • Add only. Missing mode means additive; both mode metadata keys independently reject project and every other non-add mode by name. No directional-ablation kernel or CLI mode remains.
  • Startup and in-force messages escape filenames and metadata, report SHA-256 of the exact already-read file bytes, and explicitly say replies are steered.
  • Unsupported families (including Gemma), CUDA/ROCm, malformed numeric metadata/strengths, mismatched width/checkpoint, duplicate/out-of-range layers and invalid tensor data are refused rather than silently ignored.
  • Persisted Qwen conversation identity includes the exact file hash and effective f32 strength; changing either cannot reuse a differently steered state. With no vector, the existing policy key is retained. The new kernel source causes older persisted states to start cold once after upgrade.
  • Direction ownership no longer performs a redundant allocation or risks a second free on an allocation failure.

Exercised qualification

  • Repository preflight against pinned Main: 0 errors, 0 warnings, 0 new lean findings; 43 changed files.
  • Latest affected build: 23/23 steps, 30 fresh startup tests; reader, contract and plan gates cached. Native CUDA host cross-compilation passed; this is not Metal qualification.
  • Current-head actual disk-cache regression passes1/1. The planted omission was run on prior eea372f: omitting the vector hash makes stale-state isolation fail, restoring it passes; that helper and regression are unchanged by this rebase.
  • NumPy-enabled comparator suite: 4/4 passed.
  • Real reader/startup/in-force CLI smoke accepts an additive synthetic GGUF, emits its exact file SHA and safely escapes embedded newlines. Project metadata and integer strength metadata are refused by name. These are format/logging probes, not semantic style/topic or model-output evidence.

Open maintainer gates

Still owed, and the maintainer's to run: the 20 long-prompt rows and the 1-8 stream sweep with no --glp on c10442d1e against 6c5cef894 (Metal, M5), and the style vector built from the recipe in the comments or fetched from a public model repo at a fixed revision. No CUDA or ROCm steering result is claimed; those backends refuse --glp by name.

Refs #1834 until those runs match main with level speed.

Rebased onto main 6c5cef894; the only conflict was CHANGELOG.md. zig fmt --check build.zig zig is clean. Old source-bound receipts are not relabeled.

CI on the current head: 12 successful checks and 2 scheduled-only skips, desktop and desktop-linux included. This does not waive the maintainer's 20 rows, the sweep or the style-vector check.

@BlivionIaG BlivionIaG changed the title GLP support: apply a control vector to Qwen3.8-27B on Metal at inference time (the model's effective weights change on the fly; the checkpoint does not) Implement GLP control vectors with a fused Metal kernel for Qwen3.8-27B Nov 1, 2026
@BlivionIaG
BlivionIaG force-pushed the feature/glp-steering branch 2 times, most recently from 51cc5e2 to f57cc5d Compare November 1, 2026 14:22
@ashhart

ashhart commented Nov 2, 2026

Copy link
Copy Markdown
Owner

Thanks @BlivionIaG. We reviewed this independently. We'd take the generic, opt-in --glp control-vector mechanism with its fused Metal norm-and-projection kernel. Three changes:

  1. Split out the app. The 34 files under app/ (the settings panel, IPC, preload and 13 locale packs) go in their own PR. App changes go through the app's own review.
  2. Refuse repeated layer ids. direction.5 and direction.05 both parse to 5, read() sorts but doesn't refuse, and steer.init keeps the later one. Please refuse a repeat in glp.read, with a test. A layer id past the model's depth is refused only after the 27B loads, though num_hidden_layers is readable at startup, so refuse it there.
  3. One GGUF reader. A second zig/src/core/gguf.zig collides with the GGUF reader now on main (core: a GGUF reader and an exact repack of GGUF's quantized blocks (#1920, steps 1-2) #1926). Please build on that one, or rename yours.

On scope, our decision: the engine ships the mechanism only. Please leave out the matched harmful and harmless prompt sets, the deriver, and the receipts measuring refusal removal on the 27B. Deriving vectors that remove a model's safety refusals isn't something we'll ship or document in TensorFold. A user can supply their own direction files.

BlivionIaG added a commit to BlivionIaG/TensorFold that referenced this pull request Nov 2, 2026
Review changes from ashhart#1835:
- the app's settings panel, IPC, preload and locale packs move to
  feature/glp-steering-app, so app changes go through the app's review
- glp.read refuses a file that names one layer twice (direction.5 and
  direction.05 both parse as 5), with a test
- the serve refuses a layer id past the checkpoint's num_hidden_layers at
  startup, before the model loads
- the GLP container is read by the engine's own GGUF reader; its writer stays
  in glp.zig, so there is one GGUF reader
- the deriver, the matched prompt sets, the refusal-measurement gate and their
  receipt are removed; a vector file is supplied by the operator
@BlivionIaG

Copy link
Copy Markdown
Contributor Author

All three changes are in, plus the scope cut.

  • App: the 32 app files moved to app: control vector picker and strength in Engine advanced #1951; app/ no longer appears in this PR's diff.
  • Repeated layer ids: glp.read now refuses a file that names one layer twice, with a test. A layer id past the checkpoint's num_hidden_layers is refused in glp_startup.refusal, at startup, before the model loads — two tests cover the boundary.
  • GGUF reader: the branch's zig/src/core/gguf.zig is gone; the GLP reader now parses with main's gguf.Document. Only the writer stays local to glp.zig.
  • Scope: the deriver, both prompt sets, the refusal-measurement gate and the receipt are removed; the guide and README describe the mechanism only.

The branch is merged with upstream/main at 16d95ccec and pushed as feature/glp-steering (head 7f83ebb0). zig build test-glp passes 57 tests and zig build test-glp-serve 25.

@BlivionIaG

Copy link
Copy Markdown
Contributor Author

CI caught one thing the local suites did not: after merging main, glp.zig importing the engine's gguf.zig put that file in two modules (engine_api and core), and every target linking both failed to compile.

Fixed in b81f6002: the GLP container reader is zig/src/core/glp_container.zig, named for its scope — the review's stated alternative — so the engine's gguf.zig stays the engine's reader. Repeated-layer and depth refusals are unchanged. zig build tf-qwen27-served-check compiles the server path, and the GLP suites still pass.

@ashhart

ashhart commented Nov 2, 2026

Copy link
Copy Markdown
Owner

Thanks @BlivionIaG. The 18:50 push answers every 17:58 ask:

Off by default is main (Model.steer null, both norm sites call main's norm), and the file is validated before use. Four small things left:

  1. The startup depth check reads num_hidden_layers from the config root only, while hidden_size and the engine read text_config. So a nested config skips the check until after load. Please read it from text_config too, with a nested-config test.
  2. Add-mode directions aren't checked for NaN or infinity. Please refuse non-finite values in both modes.
  3. Lean is 426 against main's 424 (core/glp.zig:161, core/glp_container.zig:1).
  4. The changelog and docs still say the engine's own GGUF reader is used, but it's the new container reader now.

It's 38 commits behind main, so a rebase with these would be ideal.

@BlivionIaG

Copy link
Copy Markdown
Contributor Author

CI is green on 2f90635ad (all 11 checks pass).

Two things came out of the first full run:

  • The lean gate counted three new problems: two comment lines and qwen27_served.zig past 600 lines. The serve-side steering helpers moved to qwen27_steer_served.zig (90 lines), qwen27_served.zig is back to 521, and the comments are one line each. lean_check now reports 424 problems, 424 on the base, 0 new.
  • zig and zig-windows failed on a regression already on main (linear_time.Fastest indexed as an array in cortheon/program.zig). Main's 5fe74ebb9 fixes it; this branch merges that tip.

@ashhart

ashhart commented Nov 3, 2026

Copy link
Copy Markdown
Owner

Thanks @BlivionIaG: the rebase applies on main and lean is back to 424 (the served helpers moved verbatim). Three asks are still open: (1) modelOf reads num_hidden_layers from the config root, not text_config; (2) glp.zig:read checks finiteness only in project mode, so add-mode directions aren't checked; (3) the changelog and docs still say the engine's own GGUF reader is used. Nothing cleared earlier changed.

@BlivionIaG

Copy link
Copy Markdown
Contributor Author

All three are in, fd0388546.

  • Depth from text_config. modelOf now reads num_hidden_layers from text_config, falling back to the config root. A nested-config test writes a config whose depth lives only in text_config and holds the depth refusal before load.
  • Non-finite directions refused in both modes. glp.read scans every direction component before the mode-specific rule and refuses NaN and ±infinity in project and add, with a test that writes each through both modes. The project-mode unit-norm check no longer carries its own isFinite guard, since the scan precedes it.
  • Reader named. The changelog and the guide now say core/glp_container.zig reads the GGUF v3 subset a vector needs, not the engine's gguf.zig. The guide's refusal table also lists non-finite directions.

zig build test-glp, zig build test-glp-serve and zig build tf-qwen27-served-check pass. lean_check reports 424 problems, 424 on the base, so nothing new.

@BlivionIaG

Copy link
Copy Markdown
Contributor Author

End-to-end run on an M4 Pro, macOS, native Metal server built from fd0388546 (zig build native, Zig 0.17.0).

The served checkpoint's config.json carries its depth only in text_config (text_config.num_hidden_layers = 64, text_config.hidden_size = 5120; no root num_hidden_layers/hidden_size), so this run exercises the nested-config path:

  • A vector naming layer 64 (direction.64, 5120 wide, project) is refused in 0.21 s, before the model loads: --glp: the vector steers layer 64 of a model with 64 layers: its deepest layer is 63.
  • An add-mode vector with one NaN component is refused in 0.02 s: argument --glp: layer 10's direction holds a value that is not finite.
  • A 63-layer vector naming layers 1-63 is accepted at the boundary and serves: glp on: ... (mode: project; alpha: 1.00; layers: 63 (1-63); base: Qwen/Qwen3.8-27B @ 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0; digest: d92fddd15d08 (verified)), then control vector in force: project mode, alpha 1.000, 63 layers (1-63).

Serving with the published 49-layer vector from msuiche/Qwen3.8-27B-abliterated-cyber-GLP-49 (mode: project, layers 10-58, digest verified) and the prompt "Write a haiku about the sea." at temperature 0.2, enable_thinking: false:

  • steered: "Waves crash on the shore, / White foam dissolves in the blue, / Silence takes the tide." (24 completion tokens)
  • the same build without --glp: "Salt spray on the wind, / Waves crash against the dark shore, / Moonlight on the foam." (23 completion tokens)

So the projection changes the served tokens on a benign prompt, and the fused path runs. Both servers were stopped after the requests; the throwaway generator used for the refusal files and the files themselves are not in the tree.

@ashhart

ashhart commented Nov 3, 2026

Copy link
Copy Markdown
Owner

Thanks @BlivionIaG. An independent re-review of fd0388546 finds all three open asks met. Without --glp, Qwen3.8-27B on Metal runs main's path: no steer, no new pipeline compiled, no extra report line. Every refusal names its cause.

One rebase is needed. On current main, 42 of the 43 files apply. The only conflict is the all_metal line in core/contract/registry.zig: main has since appended ling_route_records.zig and this PR appends steer_records.zig, so keep both. The other shared lists (core/contract.zig, build.zig, zig/build/qwen27.zig's parts) may need the same keep-both.

Once it's rebased, we'll run the Mac cells on an M5 Ultra and land it on a clean result:

  • off against main: token_sha256 equal, default and --no-drafts, at 1 and 2 streams;
  • on, with a project-mode vector at --glp-alpha 1.0: decode within 1% of off;
  • the four refusals by name;
  • norm_steer through the Metal kernel contract.

@BlivionIaG
BlivionIaG force-pushed the feature/glp-steering branch from fd03885 to b61f6fe Compare November 3, 2026 07:33
BlivionIaG added a commit to BlivionIaG/TensorFold that referenced this pull request Nov 3, 2026
Review changes from ashhart#1835:
- the app's settings panel, IPC, preload and locale packs move to
  feature/glp-steering-app, so app changes go through the app's review
- glp.read refuses a file that names one layer twice (direction.5 and
  direction.05 both parse as 5), with a test
- the serve refuses a layer id past the checkpoint's num_hidden_layers at
  startup, before the model loads
- the GLP container is read by the engine's own GGUF reader; its writer stays
  in glp.zig, so there is one GGUF reader
- the deriver, the matched prompt sets, the refusal-measurement gate and their
  receipt are removed; a vector file is supplied by the operator
@BlivionIaG

Copy link
Copy Markdown
Contributor Author

Rebased onto upstream/main at ef9fa700a; head is efd421d96, and the diff is 43 files.

The conflict was all_metal in core/contract/registry.zig: both sides kept, so the list now ends ... @import("ling_route_records.zig").records ++ @import("steer_records.zig").metal;. The only other conflicts were the branch's own core/gguf.zig add against main's GGUF reader, in two intermediate commits; resolved to main's reader, and the final tree has no branch copy — glp.zig reads glp_container.zig. The shared lists in core/contract.zig, build.zig and zig/build/qwen27.zig auto-merged, with both additions kept.

zig build test-glp and zig build test-glp-serve (26/26) pass, zig build tf-qwen27-served-check compiles, and lean_check reports 424 problems, 424 on the base. The one added commit is the guide naming the GLP release the tap now ships (v1.2.1-glp.1, upstream 1.2.1 plus this branch).

@BlivionIaG

Copy link
Copy Markdown
Contributor Author

All 11 checks pass on efd421d96: zig, zig-windows, zig-macos-dev-tools, fmt, desktop, desktop-linux, mobile, appimage-linux, release-paths and both benchmark-clients jobs.

@ashhart

ashhart commented Nov 3, 2026

Copy link
Copy Markdown
Owner

Thanks @BlivionIaG. Two of the four Mac checks are in, on an M5 Ultra at your rebased head:

  • Off against main (no --glp, Qwen3.8-27B-MLX-4bit, fresh servers, base-new-new-base): GREEN. Tokens are equal and every cell is level.
  • Metal kernel contract (--only qwen27): norm_steer passes gates 1-4 with its control red, and there are 0 limit problems.

Still owed, and these need your vector files:

  1. on, with a project-mode vector at --glp-alpha 1.0: the startup line, and decode within 1% of off;
  2. the four refusals by name before load (a nested config past depth, an add-mode NaN vector, --glp with no file, a CUDA build given --glp).

If you post those two receipts from your Mac, this lands on them.

@BlivionIaG

BlivionIaG commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor Author

Mac receipts on efd421d96

Apple M4 Pro, 48 GiB, macOS 26.5.1, Zig 0.17.0. Native Metal server built with zig build native (110 s), tensorfold-native 1.2.1.

On at --glp-alpha 1.0

Vector: msuiche/Qwen3.8-27B-abliterated-cyber-GLP-49, Qwen3.8-27B-abliterated-cyber-GLP-49-L10-58-a1.gguf — project, alpha 1.0, layers 10-58, digest da81a8b364de (verified).

[tensorfold] glp on: $V49 (mode: project; alpha: 1.00; layers: 49 (10-58); base: Qwen/Qwen3.8-27B @ 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0; digest: da81a8b364de (verified))
info: Qwen3.8-27B: control vector in force: project mode, alpha 1.000, 49 layers (10-58), base Qwen/Qwen3.8-27B @ 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0, digest da81a8b364de

The startup line is verbatim; $V49 is the resolved cache path, set in the recipe below.

Decode: same binary, fresh server per run, --context 2048, default drafts, tools/bench_openai.py --tokens 64 --reps 5 --temperatures 0, interleaved off / on / off / on so thermal drift cancels. (A single off run immediately after the build read 14.84 tok/s; the machine had drifted by the time of the paired runs, so only the four-run A/B is comparable.)

prompt off 1 on 1 off 2 on 2 off mean on mean delta
fibonacci-raw 13.9855 13.9951 13.8591 14.0267 13.9223 14.0109 +0.64%
gpu-chat-no-think 13.9507 13.8885 13.9249 13.9714 13.9378 13.9300 -0.06%

Both prompts are inside 1% of off; the spread is the run-to-run noise, not a steering cost.

Four refusals by name

1. Nested config past depth. /tmp/tf-glp-nested/config.json carries depth and width only under text_config (num_hidden_layers = 32, hidden_size = 5120), with no root num_hidden_layers/hidden_size. A 63-layer vector:

$ tensorfold-native serve /tmp/tf-glp-nested --glp .../Qwen3.8-27B-abliterated-cyber-GLP-63-L1-63-a1.gguf --context 1024
tensorfold: --glp: the vector steers layer 32 of a model with 32 layers: its deepest layer is 31
error: GlpRefused

Exit 1 in 0.19 s, before the model is touched.

2. Add-mode NaN. A one-layer add file with direction.10[0] = NaN:

$ tensorfold-native serve TensorFold/Qwen3.8-27B-MLX-4bit --glp /tmp/glp-nan-add.gguf --context 1024
tensorfold serve: error: argument --glp: layer 10's direction holds a value that is not finite

Exit 2 in 0.012 s, at argument parsing.

3. --glp with no file.

$ tensorfold-native serve TensorFold/Qwen3.8-27B-MLX-4bit --glp
tensorfold serve: error: argument --glp: expected one argument

Exit 2 in 0.010 s.

4. CUDA build given --glp. Cross-compiled from the same tree for aarch64-linux-gnu (zig build native -Dtarget=aarch64-linux-gnu -Doptimize=ReleaseSafe) and run in a Debian bookworm container; the Mac has no NVIDIA GPU. The model directory held the checkpoint's config.json and tokenizer files (no weights), and the vector was the same 49-layer file:

$ /tf serve /run/model --glp /run/vec/vector.gguf --context 1024
tensorfold: --glp: only the Metal engine steers: this server runs the CUDA engine
error: GlpRefused

Exit 1. The engine-open path refuses before any weights load; the text/tokenizer load runs alongside it, as startup.both always does. A refusal on a real CUDA host is still the one thing not exercised here.

Recipe: exact commands
git clone https://github.com/BlivionIaG/TensorFold.git
cd TensorFold
git checkout efd421d9615956bd3b414ab87e1305c59270c39f
# Zig 0.17.0 must be on PATH.
zig build native        # Metal server
zig build tf-glp-check  # tf-glp-check

Off/on decode (fresh server per run; only --glp/--glp-alpha differ):

V49=$(ls ~/.cache/huggingface/hub/models--msuiche--Qwen3.8-27B-abliterated-cyber-GLP-49/snapshots/*/Qwen3.8-27B-abliterated-cyber-GLP-49-L10-58-a1.gguf)

# off
./zig-out/native/bin/tensorfold-native serve TensorFold/Qwen3.8-27B-MLX-4bit \
  --context 2048 --port 8123 --host 127.0.0.1 --name bench --no-update-check

# on
./zig-out/native/bin/tensorfold-native serve TensorFold/Qwen3.8-27B-MLX-4bit \
  --glp "$V49" --glp-alpha 1.0 \
  --context 2048 --port 8123 --host 127.0.0.1 --name bench --no-update-check

# for each server, once it is listening:
python3 tools/bench_openai.py http://127.0.0.1:8123 Qwen3.8-27B-MLX-4bit \
  --tokens 64 --reps 5 --temperatures 0 --output /tmp/bench.json

Nested-config fixture:

import glob, json, os, shutil
snap = glob.glob(os.path.expanduser(
    "~/.cache/huggingface/hub/models--TensorFold--Qwen3.8-27B-MLX-4bit/snapshots/*/"))[0]
out = "/tmp/tf-glp-nested"
shutil.rmtree(out, ignore_errors=True)
os.makedirs(out)
for name in os.listdir(snap):
    os.symlink(os.path.realpath(os.path.join(snap, name)), os.path.join(out, name))
d = json.load(open(os.path.join(snap, "config.json")))
d.pop("num_hidden_layers", None)
d.pop("hidden_size", None)
tc = d.setdefault("text_config", {})
tc["num_hidden_layers"] = 32
tc["hidden_size"] = 5120
json.dump(d, open(os.path.join(out, "config.json"), "w"), indent=2)

Throwaway NaN-vector generator (zig/src/core/tmp_glp_bad.zig, deleted after the run):

const std = @import("std");
const glp = @import("glp.zig");
const gguf = @import("glp_container.zig");

const width = 5120;

pub fn main(init: std.process.Init) !void {
    const a = init.arena.allocator();
    const args = try init.minimal.args.toSlice(a);
    if (args.len != 3) return error.Usage;
    const kind = args[1];
    const out_path = args[2];

    var dir = try a.alloc(f32, width);
    for (dir) |*v| v.* = 0;
    dir[0] = 1;

    if (std.mem.eql(u8, kind, "deep")) {
        const layers = [_]glp.Layer{.{ .id = 64, .dir = dir }};
        const bytes = try glp.write(a, .{
            .mode = .project,
            .alpha = 1,
            .hook_point = glp.residual_site,
            .derived_at = null,
            .base_model = "TensorFold/Qwen3.8-27B-MLX-4bit",
            .base_revision = "",
            .method = "test",
            .contrast = "",
            .layers = &layers,
        });
        try std.Io.Dir.cwd().writeFile(init.io, .{ .sub_path = out_path, .data = bytes });
        return;
    }
    if (std.mem.eql(u8, kind, "nan-add")) {
        var raw = try a.alloc(u8, width * 4);
        std.mem.writeInt(u32, raw[0..4], @bitCast(std.math.nan(f32)), .little);
        @memset(raw[4..], 0);
        const entries = [_]gguf.Entry{
            .{ .key = "general.architecture", .value = .{ .string = "controlvector" } },
            .{ .key = "glp.spec_version", .value = .{ .u32v = 1 } },
            .{ .key = "glp.mode", .value = .{ .string = "add" } },
            .{ .key = "glp.rank", .value = .{ .u32v = 1 } },
            .{ .key = "glp.hook_point", .value = .{ .string = glp.residual_site } },
            .{ .key = "glp.alpha_default", .value = .{ .f32v = 1 } },
            .{ .key = "glp.base_model", .value = .{ .string = "TensorFold/Qwen3.8-27B-MLX-4bit" } },
        };
        const dims = [_]u64{width};
        const tensors = [_]gguf.OutTensor{.{ .name = "direction.10", .dims = &dims, .kind = gguf.f32_type, .data = raw }};
        const bytes = try gguf.write(a, &entries, &tensors);
        try std.Io.Dir.cwd().writeFile(init.io, .{ .sub_path = out_path, .data = bytes });
        return;
    }
    return error.UnknownKind;
}
zig run zig/src/core/tmp_glp_bad.zig -- nan-add /tmp/glp-nan-add.gguf
rm -f zig/src/core/tmp_glp_bad.zig

CUDA build and container run:

zig build native -Dtarget=aarch64-linux-gnu -Doptimize=ReleaseSafe
mkdir -p "$HOME/tf-glp-cuda/run/model" "$HOME/tf-glp-cuda/run/vec"
cp zig-out/native/bin/tensorfold-native "$HOME/tf-glp-cuda/tensorfold-native-cuda-aarch64"
# rebuild the Metal binary afterwards, since the cross-build writes the same path:
zig build native

# populate run/model with the checkpoint's config.json and tokenizer files (no weights),
# and run/vec with the 49-layer vector above.
python3 - <<'PY'
import glob, os, shutil
home = os.path.expanduser("~")
snap = glob.glob(home + "/.cache/huggingface/hub/models--TensorFold--Qwen3.8-27B-MLX-4bit/snapshots/*/")[0]
model = home + "/tf-glp-cuda/run/model"
for name in ("config.json", "generation_config.json", "tokenizer.json", "tokenizer_config.json",
             "vocab.json", "chat_template.jinja", "preprocessor_config.json",
             "processor_config.json", "model.safetensors.index.json"):
    src = os.path.join(snap, name)
    if os.path.exists(src):
        shutil.copy(os.path.realpath(src), os.path.join(model, name))
vec = glob.glob(home + "/.cache/huggingface/hub/models--msuiche--Qwen3.8-27B-abliterated-cyber-GLP-49/snapshots/*/Qwen3.8-27B-abliterated-cyber-GLP-49-L10-58-a1.gguf")[0]
shutil.copy(os.path.realpath(vec), home + "/tf-glp-cuda/run/vec/vector.gguf")
PY

podman run --rm --platform linux/arm64 \
  -v "$HOME/tf-glp-cuda/tensorfold-native-cuda-aarch64:/tf:ro" \
  -v "$HOME/tf-glp-cuda/run:/run:ro" \
  docker.io/library/debian:bookworm-slim \
  /tf serve /run/model --glp /run/vec/vector.gguf --context 1024 --port 8193

@ashhart

ashhart commented Nov 4, 2026

Copy link
Copy Markdown
Owner

Thanks @BlivionIaG for both receipts. On-mode decode is within 1% of off, and all four refusals stop by name before load. A few things before it lands:

  1. It conflicts with main again, so please rebase.
  2. Please remove the "The engine that steers" section from docs/control-vectors.md. Once this is merged, --glp ships in TensorFold itself, so the docs shouldn't point to another tap or release.
  3. In glp_publication_test.zig, please use a neutral contrast string in the sample metadata, in line with the scope we agreed.

We'll check the CUDA refusal ourselves on a GB10.

@BlivionIaG
BlivionIaG force-pushed the feature/glp-steering branch from efd421d to b721c88 Compare November 4, 2026 07:15
BlivionIaG added a commit to BlivionIaG/TensorFold that referenced this pull request Nov 4, 2026
Review changes from ashhart#1835:
- the app's settings panel, IPC, preload and locale packs move to
  feature/glp-steering-app, so app changes go through the app's review
- glp.read refuses a file that names one layer twice (direction.5 and
  direction.05 both parse as 5), with a test
- the serve refuses a layer id past the checkpoint's num_hidden_layers at
  startup, before the model loads
- the GLP container is read by the engine's own GGUF reader; its writer stays
  in glp.zig, so there is one GGUF reader
- the deriver, the matched prompt sets, the refusal-measurement gate and their
  receipt are removed; a vector file is supplied by the operator
@BlivionIaG

Copy link
Copy Markdown
Contributor Author

Rebased onto 8db1ee1a3; head is b721c88ae.

  • Removed the external-build section from docs/control-vectors.md. The guide now documents only the in-tree --glp surface and points at no tap or release.
  • Neutralized the sample metadata: the publication sample no longer names refusal or harm, and the contrast sample reads matched_prompts_unpaired.
  • Resolved the rebase conflicts: metal_main.zig keeps the attention and steering imports; engine_api.zig keeps upstream's engine_request_controls.zig, so the branch no longer duplicates those request shapes.

zig build test-glp, zig build test-glp-serve and zig build tf-qwen27-served-check pass locally. CI is running on the new head. The CUDA refusal is yours to check on the GB10.

@BlivionIaG

Copy link
Copy Markdown
Contributor Author

release-paths caught one rebase follow-on: the contract imported bf16_round.zig by path, and the release build also gives that file its own module. Fixed in 9c90e0956 by importing the module. Head is now 9c90e0956; test-glp, test-glp-serve, tf-qwen27-served-check and check-release-paths pass locally.

BlivionIaG added a commit to BlivionIaG/TensorFold that referenced this pull request Nov 4, 2026
Review changes from ashhart#1835:
- the app's settings panel, IPC, preload and locale packs move to
  feature/glp-steering-app, so app changes go through the app's review
- glp.read refuses a file that names one layer twice (direction.5 and
  direction.05 both parse as 5), with a test
- the serve refuses a layer id past the checkpoint's num_hidden_layers at
  startup, before the model loads
- the GLP container is read by the engine's own GGUF reader; its writer stays
  in glp.zig, so there is one GGUF reader
- the deriver, the matched prompt sets, the refusal-measurement gate and their
  receipt are removed; a vector file is supplied by the operator
@BlivionIaG

Copy link
Copy Markdown
Contributor Author

For the remaining no---glp gate on frozen 731afc275, could you link the harness/command (prompt IDs and request settings) for the 20 rows in your Nov 5 M5 receipt, and the main/head SHAs it compared? I haven't recovered that row-level protocol from the retained receipts/history. Any additional matrix will stay separately labelled, not substituted for those 20 rows.

@ashhart

ashhart commented Nov 12, 2026

Copy link
Copy Markdown
Owner

@BlivionIaG the script for those 20 rows isn't in the repository, so here is how it works:

  • Commits compared on 5 Nov: main 981fe201d50748bc748a3c0a116e395222592700, which was the branch's merge base then, against the head at the time, 59228664b081fcadfd2f8fa473d769ddc91fc3b6.
  • Checkpoint: TensorFold/Qwen3.8-27B-MLX-4bit at revision 70ae7fac63274ff2eac54152031433374cb80f2f.
  • Server, both arms: serve MODEL --context 73728 --parallel 8 --prompt-cache-gib 0, with no --glp. Each run gets a fresh server, in the order main, branch, branch, main, after one warm-up run of main that isn't counted.
  • Prompts: one fixed text, tokenized once through the server's /tokenize with add_special_tokens: false. There are five sizes: 2,048, 8,192, 16,384, 32,768 and 65,536 tokens. At each size we send three cold prompts cut at token offsets 0, 5,000 and 10,000, then the offset-0 prompt once more. That makes 4 rows a size and 20 rows in all. Any fixed text works, since only main and the branch on the same machine are compared.
  • Each request: POST /v1/completions with prompt as the list of token IDs (no chat template), max_tokens: 64, temperature: 0, seed: 1234 and ignore_eos: true, streamed with include_usage. That is the request stream() in tools/bench_concurrent.py sends. Each row records token_sha256, TTFT, prompt tok/s (size / TTFT) and decode tok/s. It must report 0 cached tokens and exactly the size in prompt tokens.
  • How rows are read: a row differs when its token hash is not the same in all four runs. Speed is compared row by row: prompt tok/s and decode tok/s, the median of main's two runs against the median of the branch's two.
  • Streams: on the same server, after the rows, the sweep is tools/bench_concurrent.py --levels 1,2,4,8 --temperatures 1.0,0 --tokens 256 --reps 3 --alone --serial. The 5 Nov note quoted its 1- and 4-stream rows.

The body says your machine can't run Metal right now, so we'll run these 20 rows and the sweep ourselves on a Metal Mac: 731afc275 against its merge base 8a7838a65. For the style or topic check we need a vector file. If you have one (a non-refusal direction for the 5120-wide 27B), please link it with its SHA-256 and the strength you used, and we'll run its startup line and replies with and without it in the same session.

You don't need to rebase. On today's main only CHANGELOG.md conflicts, and lean shows 0 new problems on the merged tree. We'll put the line under Unreleased when it lands.

  • Synthena (Ash's Agent)

@BlivionIaG

Copy link
Copy Markdown
Contributor Author

Device evidence for the two asks, on an Apple M4 Pro (48 GiB), Qwen3.8-27B (TensorFold/Qwen3.8-27B-MLX-4bit), zig build native. "Main" here is this branch's merge base 8a7838a65; the head is 731afc275.

No --glp: 16 of 16 rows equal. Rows are the 16 prompts in tools/bench_prompts.json, greedy, 64 tokens, run as one cell of 16 streams with tools/bench_concurrent.py (I don't have your 20-row protocol, so this is the closest harness in the tree). Each build was served twice; every row's token_sha256 is identical across all four runs (base/base, base/head, head/head). Aggregate rates were 51.8 and 50.8 tok/s on the base, and 52.2 and 45.8 on the head: the head's second run was slower than its first, so I won't claim a speed delta from this; the interleaved off/on pairs below are the decode check.

One vector, a style one. The repo carries none, so I derived a pirate-voice vector: the difference of means at the last prompt token between 12 pirate-voiced and 12 plain paragraph prompts, on the block outputs of layers 24 to 40, written with the branch's own glp.write as an add file. Startup line (alpha 0.1):

[tensorfold] glp on: replies are steered by the control vector in "pirate.gguf" (file sha256: 22e59384d51eb021e68663af3a14dae8a0d143b5163d8de4ac6a9cea83d48c04; alpha: 1.00; layers: 17 (24-40); base: TensorFold/Qwen3.8-27B-MLX-4bit; tensor digest: 87da5ed9022f (verified))

(That line is from the alpha 1.0 run; --glp-alpha 0.1 is the same line with alpha: 0.10.) Greedy replies, thinking off, first 90 tokens:

prompt without with, alpha 0.1
Short paragraph about tomorrow's weather "Since I don't have access to your specific location, I can't provide a real-time forecast. However…" "Oh, mate, grab yer umbrella and hold on tight! … a stormy beast up in the sky!"
What is a hash table, two sentences "A hash table is a data structure that uses a hash function to map keys to array indices…" "A hash table is a magical chest where you toss in your data using a special "hash" key…"
Three job interview tips "Here are three essential tips to help you succeed in your next job interview: 1. **Research the Company…" "Landing that dream job is all about the vibe and the story you tell!…"

Strength is the operator's, as the doc says: alpha 0.25 turns the replies to shouting pirate noise ("Ahoy, me Salty Sea Dogs… YARR"), and alpha 1.0 gives repeated O and g characters.

Decode, off/on/off/on, fresh server each, alpha 0.1, tools/bench_openai.py --tokens 64 --reps 5 --temperatures 0, median tok/s per run:

prompt off on off on delta of means
fibonacci-raw 13.95 13.40 13.96 14.27 -0.84%
gpu-chat-no-think 13.23 13.85 13.81 14.01 +3.03%

The signs differ, so this is run-to-run noise on this machine, not a steering cost.

I can attach the vector file or the derivation script if you want either.

BlivionIaG added a commit to BlivionIaG/TensorFold that referenced this pull request Nov 12, 2026
A settings panel beside the engine's other advanced options, the picker's
path taken through the OS picker and never as a renderer argument, the
engine's --glp/--glp-alpha arguments, and the audit recording the vector's
state and strength rather than its path. The panel is inert when the engine
does not advertise --glp.

Split out of ashhart#1835, whose engine change carries the mechanism.
@BlivionIaG

Copy link
Copy Markdown
Contributor Author

Rebased onto main 6c5cef894, now ccac1b833; GitHub shows it mergeable. The only conflict was CHANGELOG.md: your line and main's are both kept, and the GLP line now ends "Thanks to @BlivionIaG." like the other contributors' entries.

On this head: zig fmt --check build.zig zig is clean, lean reports 345 problems, the same as main, no touched file is over 600 lines, and zig build test test-golden dev-tools passes except dashboard_test ("Paged prefix cache"), which fails on main the same way. The desktop check on the PR is the inherited macOS bounded-output failure.

@ashhart

ashhart commented Nov 12, 2026

Copy link
Copy Markdown
Owner

Thanks @BlivionIaG. The rebase onto 6c5cef894 is clean, the CHANGELOG.md line credits you, and all checks are green on ccac1b833, desktop included.

On the device evidence:

  • No --glp: 16 of 16 rows have identical token hashes across all four runs. That covers short prompts at 16 streams. It doesn't cover the long prompts (2,048 to 65,536 tokens) or the 1 to 8 stream sweep, so, as we said, we'll run those 20 rows and the sweep ourselves on an M5 Mac, ccac1b833 against 6c5cef894.
  • The pirate vector: the startup line has everything we asked for (escaped file name, file SHA-256, alpha, layers and the steered note), and the replies with and without it show the mechanism plainly. The two prompts' off/on decode deltas differ in sign (-0.84% and +3.03%), so we read them as noise too.

One thing would help: please attach the vector file (SHA-256 22e59384d51eb021e68663af3a14dae8a0d143b5163d8de4ac6a9cea83d48c04) and the script that derived it. We'll run its startup line and a few replies with and without it, at alpha 0.1, in the same session as the 20 rows. Once the 20 rows match main and speed is level, it goes on the landing queue.

  • Synthena (Ash's Agent)

@BlivionIaG

BlivionIaG commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor Author

Attached as a gist: https://gist.github.com/a31f0965d77eb5aa225e8903074d22de

It has pirate.gguf.b64 (gists hold text only, so base64 -d gives the file; SHA-256 22e59384d51eb021e68663af3a14dae8a0d143b5163d8de4ac6a9cea83d48c04, checked after the round trip), derive.py (mlx-lm; difference of means at the last prompt token, 12 pirate against 12 plain prompts, layers 24 to 40), glp_make.zig (the throwaway writer that calls the branch's glp.write) and a README with the commands. I re-ran derive.py from the gist and it produces the same directions byte for byte on this machine. Strength is --glp-alpha 0.1.

Ready for your 20 rows and the sweep on ccac1b833 against 6c5cef894.

@ashhart

ashhart commented Nov 13, 2026

Copy link
Copy Markdown
Owner

Thanks @BlivionIaG for putting the file, the derivation and the writer together so quickly.

We asked you to attach the file, and we need to change that ask. Sorry for the extra round. We don't load a binary that arrives as base64 in a gist. A file like that can't be reviewed by reading it, so we only load vectors we build ourselves from a recipe, or fetch from a public model repo at a fixed revision. We won't decode or load pirate.gguf.b64.

What would let us build the same vector ourselves, posted here as text in a comment so it stays with the review:

  • the 24 prompts word for word (12 pirate-voiced, 12 plain), and whether they went through the chat template (thinking on or off) or in raw;
  • what you captured: which tensor counts as a block's output at layers 24 to 40, at the last prompt token, and in what precision;
  • how each layer's direction is formed from those: the difference of means, any scaling or normalisation, and the precision written;
  • the checkpoint revision and the mlx-lm version;
  • the metadata your writer passes to the branch's glp.write (zig/src/core/glp.zig:283): the base model, the default strength, and the layer ids.

If derive.py and the writer are short, pasting both in a comment does all of this at once. We'd then build the file with the branch's own glp.write and run its startup line and a few replies with and without it at alpha 0.1. Our file's SHA-256 may not match yours if the sums round differently on another Mac. That wouldn't block anything: the check is the startup line and the replies.

If instead a style or topic vector for this 5120-wide checkpoint is already published on Hugging Face in a format the branch reads, name the repo and revision and we'll fetch it from there.

Nothing else changes. We'll run the 20 rows and the sweep on ccac1b833 against 6c5cef894, and once the rows match main and speed is level, it goes on the landing queue. One small thing: the description still names 731afc275 as the head, so please update it to ccac1b833 when you next edit it.

  • Synthena (Ash's Agent)

@BlivionIaG

Copy link
Copy Markdown
Contributor Author

Understood, no binary. The full recipe as text, so you can build the vector with the branch's glp.write.

Prompts (12 topics in two forms, 24 in all), each sent as the user message through the checkpoint's chat template with add_generation_prompt=True and enable_thinking=False, so each ends <|im_start|>assistant\n<think>\n\n</think>\n\n. Pirate-voiced:

1. Write a short paragraph about the weather, talking like a pirate, with 'arr' and 'matey' and sea metaphors.
2. Write a short paragraph about a morning commute, talking like a pirate, with 'arr' and 'matey' and sea metaphors.
3. Write a short paragraph about learning to cook, talking like a pirate, with 'arr' and 'matey' and sea metaphors.
4. Write a short paragraph about a long train journey, talking like a pirate, with 'arr' and 'matey' and sea metaphors.
5. Write a short paragraph about choosing a laptop, talking like a pirate, with 'arr' and 'matey' and sea metaphors.
6. Write a short paragraph about planting a garden, talking like a pirate, with 'arr' and 'matey' and sea metaphors.
7. Write a short paragraph about the history of the telephone, talking like a pirate, with 'arr' and 'matey' and sea metaphors.
8. Write a short paragraph about saving money, talking like a pirate, with 'arr' and 'matey' and sea metaphors.
9. Write a short paragraph about a rainy afternoon, talking like a pirate, with 'arr' and 'matey' and sea metaphors.
10. Write a short paragraph about starting a new job, talking like a pirate, with 'arr' and 'matey' and sea metaphors.
11. Write a short paragraph about a trip to the market, talking like a pirate, with 'arr' and 'matey' and sea metaphors.
12. Write a short paragraph about fixing a bicycle, talking like a pirate, with 'arr' and 'matey' and sea metaphors.

Plain:

1. Write a short paragraph about the weather, in a plain, neutral, matter-of-fact tone.
2. Write a short paragraph about a morning commute, in a plain, neutral, matter-of-fact tone.
3. Write a short paragraph about learning to cook, in a plain, neutral, matter-of-fact tone.
4. Write a short paragraph about a long train journey, in a plain, neutral, matter-of-fact tone.
5. Write a short paragraph about choosing a laptop, in a plain, neutral, matter-of-fact tone.
6. Write a short paragraph about planting a garden, in a plain, neutral, matter-of-fact tone.
7. Write a short paragraph about the history of the telephone, in a plain, neutral, matter-of-fact tone.
8. Write a short paragraph about saving money, in a plain, neutral, matter-of-fact tone.
9. Write a short paragraph about a rainy afternoon, in a plain, neutral, matter-of-fact tone.
10. Write a short paragraph about starting a new job, in a plain, neutral, matter-of-fact tone.
11. Write a short paragraph about a trip to the market, in a plain, neutral, matter-of-fact tone.
12. Write a short paragraph about fixing a bicycle, in a plain, neutral, matter-of-fact tone.

What is captured. The output of each decoder layer (the residual stream after the block, which direction.N is added to) at the last prompt token, layers 24 to 40. mlx-lm runs the checkpoint in bfloat16, so each row is bfloat16, cast to float32 before anything else.

The direction. Per layer: mean over the 12 pirate prompts minus mean over the 12 plain prompts, in float32. No scaling, normalisation or centring. Written as fp32, one direction.N per layer.

Versions. Checkpoint TensorFold/Qwen3.8-27B-MLX-4bit at 22d8d538154e0e5b6f8dcb1fcc74a60a0798104c; mlx-lm 0.32.2, mlx 0.32.4, numpy 2.5.4; Apple M4 Pro.

What the writer passes to glp.write (zig/src/core/glp.zig:283): alpha = 1.0 (so glp.alpha_default is 1.0; the replies were at --glp-alpha 0.1), base_model = "TensorFold/Qwen3.8-27B-MLX-4bit", no base_revision, method = "difference_of_means_last_token_per_layer", contrast = "pirate-voiced paragraphs against plain neutral paragraphs, 12 topics each", name = "pirate-voice-test", created = "2026-11-12", and layer ids 24 to 40 (17 layers).

derive.py (writes pirate.bin: a u32 count, then a u32 id and 5120 little-endian fp32 per layer)
"""Derive a pirate-voice add vector for Qwen3.8-27B: difference of means at the last prompt token, per layer.

    pip install mlx-lm numpy
    python3 derive.py CHECKPOINT_DIR pirate.bin

CHECKPOINT_DIR is a local snapshot of TensorFold/Qwen3.8-27B-MLX-4bit. pirate.bin holds layers 24-40:
a u32 count, then for each layer a u32 id and 5120 little-endian fp32 values. glp_make.zig turns it into the GGUF.
"""
import struct
import sys

import mlx.core as mx
import numpy as np
from mlx_lm import load

LAYERS = range(24, 41)
TOPICS = ["the weather", "a morning commute", "learning to cook", "a long train journey", "choosing a laptop",
          "planting a garden", "the history of the telephone", "saving money", "a rainy afternoon",
          "starting a new job", "a trip to the market", "fixing a bicycle"]
PIRATE = [f"Write a short paragraph about {t}, talking like a pirate, with 'arr' and 'matey' and sea metaphors."
          for t in TOPICS]
PLAIN = [f"Write a short paragraph about {t}, in a plain, neutral, matter-of-fact tone." for t in TOPICS]

model, tok = load(sys.argv[1])
lm = model.language_model if hasattr(model, "language_model") else model
inner = lm.model
store = []


class Rec:
    """Wraps a decoder layer and keeps its output at the last token."""

    def __init__(self, layer):
        self.layer = layer
        self.is_linear = getattr(layer, "is_linear", False)

    def __call__(self, *a, **k):
        out = self.layer(*a, **k)
        store.append(out[0, -1].astype(mx.float32))
        return out


inner.layers = [Rec(layer) for layer in inner.layers]


def states(prompt):
    messages = [{"role": "user", "content": prompt}]
    try:
        text = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False, enable_thinking=False)
    except TypeError:
        text = tok.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
    store.clear()
    model(mx.array([tok.encode(text)]))
    mx.eval(store)
    return np.stack([np.array(s) for s in store])


diff = (np.mean([states(p) for p in PIRATE], axis=0) - np.mean([states(p) for p in PLAIN], axis=0)).astype(np.float32)
with open(sys.argv[2], "wb") as f:
    f.write(struct.pack("<I", len(LAYERS)))
    for i in LAYERS:
        f.write(struct.pack("<I", i))
        f.write(diff[i].astype("<f4").tobytes())
glp_make.zig (throwaway: copy to zig/src/core/tmp_glp_make.zig, zig run zig/src/core/tmp_glp_make.zig -Dcpu=apple_m1 -- pirate.bin pirate.gguf, then delete)
// Throwaway writer, not part of the PR. Copy it to zig/src/core/tmp_glp_make.zig on the #1835 branch, then
//   zig run zig/src/core/tmp_glp_make.zig -Dcpu=apple_m1 -- pirate.bin pirate.gguf
// and delete it. It writes the file with the branch's own glp.write, as an add vector.
const std = @import("std");
const glp = @import("glp.zig");

pub fn main(init: std.process.Init) !void {
    const a = init.arena.allocator();
    const io = init.io;
    const args = try init.minimal.args.toSlice(a);
    const raw = try std.Io.Dir.cwd().readFileAlloc(io, args[1], a, .limited(64 << 20));
    const n = std.mem.readInt(u32, raw[0..4], .little);
    const layers = try a.alloc(glp.Layer, n);
    var at: usize = 4;
    for (layers) |*l| {
        l.id = std.mem.readInt(u32, raw[at..][0..4], .little);
        at += 4;
        const dir = try a.alloc(f32, 5120);
        @memcpy(std.mem.sliceAsBytes(dir), raw[at..][0 .. 5120 * 4]);
        at += 5120 * 4;
        l.dir = dir;
    }
    const bytes = try glp.write(a, .{
        .alpha = 1.0,
        .base_model = "TensorFold/Qwen3.8-27B-MLX-4bit",
        .method = "difference_of_means_last_token_per_layer",
        .contrast = "pirate-voiced paragraphs against plain neutral paragraphs, 12 topics each",
        .publication = .{ .name = "pirate-voice-test", .created = "2026-11-12" },
        .layers = layers,
    });
    try std.Io.Dir.cwd().writeFile(io, .{ .sub_path = args[2], .data = bytes });
}

My file's SHA-256 was 22e59384d51eb021e68663af3a14dae8a0d143b5163d8de4ac6a9cea83d48c04; if yours differs in the last bits, the startup line and the replies at alpha 0.1 are the check. Please ignore the gist's base64 file.

@BlivionIaG

Copy link
Copy Markdown
Contributor Author

Kernel contract on the M4 Pro (tf-kernel-contract --only steer) found one real failure: gate 5 said the steering launch declares 40 threadgroup bytes where the compiler allocates 48 (10 floats, rounded to 16 bytes). 960448a9a declares 48. Now: SUMMARY 1 kernels: 1 pass gates 1-4, 0 fail; 1 controls: 1 red as planted, 0 missed; 0 limit problems. test-steer test-glp test-glp-serve test-glp-steer pass 88 of 88. Only steer.zig's constant changed.

Please run your 20 rows and sweep on 960448a9a (it is on 6c5cef894, and mergeable against main today). The description is updated to this head.

BlivionIaG added a commit to BlivionIaG/TensorFold that referenced this pull request Nov 13, 2026
A settings panel beside the engine's other advanced options, the picker's
path taken through the OS picker and never as a renderer argument, the
engine's --glp/--glp-alpha arguments, and the audit recording the vector's
state and strength rather than its path. The panel is inert when the engine
does not advertise --glp.

Split out of ashhart#1835, whose engine change carries the mechanism.
The reader implements the GLP format defined by the weightless project
(@msuiche): a GGUF v3 container of one fp32 direction per layer, applied by
adding it to the residual stream. A file with no mode is llama.cpp's additive
vector; a file that names any other mode, project included, is refused rather
than read as an add file. The reader also refuses a spec version, rank or hook
point it does not implement, a layer named twice, a direction that is not
finite, and a digest that does not match. It reports the SHA-256 of the exact
bytes it was handed, so a caller can name the file it loaded.

tf-glp-check shows the reader's view of a file for glp_parity.py to compare
with the reference reader, the file hash included.
…8-27B

A steered layer takes the dispatch an unsteered layer takes: the norm that
adds the block's output to the residual stream also adds alpha times the
layer's direction to the row, and the normalisation reads the result. The
plan holds one device buffer per layer the vector names, and a model without
one runs the plain norm. The steered norm clears the half-operand state the
plain norm leaves for the head, since its output has no such words.

The kernel contract has a float64 reference for the add, a bound for the
kernel's own fp32 product and sum, the launch's threads and threadgroup memory
for gate 5, and the GPU suite with a control that drops a NaN and one that
breaks the normalisation. The GPU half needs a Mac.
Off unless --glp is given: without it the server is as it was. The file is
read once at startup, before the model loads, and the startup line names the
file (escaped), the SHA-256 of its exact bytes and that replies are steered,
beside the strength, the layers and the checkpoint the vector came from. The
file hash is taken over the bytes already read; nothing is read twice.

A file the reader refuses, a width or base model that does not match the
checkpoint, a layer past the checkpoint's depth, a Gemma checkpoint, any other
family, and a CUDA or ROCm backend are all refused by name and the server does
not start. The engine reports the vector in force with the same file hash.

docs/control-vectors.md covers the format, the layer a direction names, the
refusals and how an operator brings a vector; the repository ships none.
@BlivionIaG

Copy link
Copy Markdown
Contributor Author

Commit identity fixed: author and committer on all 5 commits are now BlivionIaG <15323295+BlivionIaG@users.noreply.github.com> (the committer line had a malformed noreply address from a local git config). Main also moved: cli.zig conflicted where main added its connection flags and this PR adds the GLP flags, so the head is now c5947a8d2 on 0591c4e85, with both flag lists kept. Hashes changed, so please run the 20 rows and sweep on c5947a8d2 (the contract fix is in it).

Local, M4 Pro: zig fmt clean, lean 341 on the head and on main (same set), cli.zig is 583 lines. The full suite fails dashboard_test (also on main) and two tests outside this PR, core.paced_read and a GLM weights-load test that stopped "short of memory" while other jobs ran on the machine. CI on this head is still running; I'll treat a green zig job as the answer for those two.

@ashhart

ashhart commented Nov 14, 2026

Copy link
Copy Markdown
Owner

Thanks @BlivionIaG. The text recipe answers the revised vector request, and the contract now declares the compiler’s 48-byte allocation. The identity-only refresh is accounted for too.

Nothing further is needed from you for the recipe. We’ll build the style vector from it and finish the agreed 20-row and stream-sweep checks on our M5 Ultra, using c5947a8d2. The final host/CI checks still need to be clear before this goes into the landing queue.

  • Synthena (Ash's Agent)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants