Repository navigation
Implement GLP control vectors with a fused Metal kernel for Qwen3.8-27B - #1835
BlivionIaG wants to merge 5 commits into
Conversation
51cc5e2 to
f57cc5d
Compare
|
Thanks @BlivionIaG. We reviewed this independently. We'd take the generic, opt-in
On scope, our decision: the engine ships the mechanism only. Please leave out the matched harmful and harmless prompt sets, the deriver, and the receipts measuring refusal removal on the 27B. Deriving vectors that remove a model's safety refusals isn't something we'll ship or document in TensorFold. A user can supply their own direction files. |
Review changes from ashhart#1835: - the app's settings panel, IPC, preload and locale packs move to feature/glp-steering-app, so app changes go through the app's review - glp.read refuses a file that names one layer twice (direction.5 and direction.05 both parse as 5), with a test - the serve refuses a layer id past the checkpoint's num_hidden_layers at startup, before the model loads - the GLP container is read by the engine's own GGUF reader; its writer stays in glp.zig, so there is one GGUF reader - the deriver, the matched prompt sets, the refusal-measurement gate and their receipt are removed; a vector file is supplied by the operator
|
All three changes are in, plus the scope cut.
The branch is merged with |
|
CI caught one thing the local suites did not: after merging main, Fixed in |
|
Thanks @BlivionIaG. The 18:50 push answers every 17:58 ask:
Off by default is main (
It's 38 commits behind main, so a rebase with these would be ideal. |
|
CI is green on Two things came out of the first full run:
|
|
Thanks @BlivionIaG: the rebase applies on main and lean is back to 424 (the served helpers moved verbatim). Three asks are still open: (1) |
|
All three are in,
|
|
End-to-end run on an M4 Pro, macOS, native Metal server built from The served checkpoint's
Serving with the published 49-layer vector from
So the projection changes the served tokens on a benign prompt, and the fused path runs. Both servers were stopped after the requests; the throwaway generator used for the refusal files and the files themselves are not in the tree. |
|
Thanks @BlivionIaG. An independent re-review of One rebase is needed. On current main, 42 of the 43 files apply. The only conflict is the Once it's rebased, we'll run the Mac cells on an M5 Ultra and land it on a clean result:
|
fd03885 to
b61f6fe
Compare
Review changes from ashhart#1835: - the app's settings panel, IPC, preload and locale packs move to feature/glp-steering-app, so app changes go through the app's review - glp.read refuses a file that names one layer twice (direction.5 and direction.05 both parse as 5), with a test - the serve refuses a layer id past the checkpoint's num_hidden_layers at startup, before the model loads - the GLP container is read by the engine's own GGUF reader; its writer stays in glp.zig, so there is one GGUF reader - the deriver, the matched prompt sets, the refusal-measurement gate and their receipt are removed; a vector file is supplied by the operator
|
Rebased onto The conflict was
|
|
All 11 checks pass on |
|
Thanks @BlivionIaG. Two of the four Mac checks are in, on an M5 Ultra at your rebased head:
Still owed, and these need your vector files:
If you post those two receipts from your Mac, this lands on them. |
Mac receipts on
|
| prompt | off 1 | on 1 | off 2 | on 2 | off mean | on mean | delta |
|---|---|---|---|---|---|---|---|
| fibonacci-raw | 13.9855 | 13.9951 | 13.8591 | 14.0267 | 13.9223 | 14.0109 | +0.64% |
| gpu-chat-no-think | 13.9507 | 13.8885 | 13.9249 | 13.9714 | 13.9378 | 13.9300 | -0.06% |
Both prompts are inside 1% of off; the spread is the run-to-run noise, not a steering cost.
Four refusals by name
1. Nested config past depth. /tmp/tf-glp-nested/config.json carries depth and width only under text_config (num_hidden_layers = 32, hidden_size = 5120), with no root num_hidden_layers/hidden_size. A 63-layer vector:
$ tensorfold-native serve /tmp/tf-glp-nested --glp .../Qwen3.8-27B-abliterated-cyber-GLP-63-L1-63-a1.gguf --context 1024
tensorfold: --glp: the vector steers layer 32 of a model with 32 layers: its deepest layer is 31
error: GlpRefused
Exit 1 in 0.19 s, before the model is touched.
2. Add-mode NaN. A one-layer add file with direction.10[0] = NaN:
$ tensorfold-native serve TensorFold/Qwen3.8-27B-MLX-4bit --glp /tmp/glp-nan-add.gguf --context 1024
tensorfold serve: error: argument --glp: layer 10's direction holds a value that is not finite
Exit 2 in 0.012 s, at argument parsing.
3. --glp with no file.
$ tensorfold-native serve TensorFold/Qwen3.8-27B-MLX-4bit --glp
tensorfold serve: error: argument --glp: expected one argument
Exit 2 in 0.010 s.
4. CUDA build given --glp. Cross-compiled from the same tree for aarch64-linux-gnu (zig build native -Dtarget=aarch64-linux-gnu -Doptimize=ReleaseSafe) and run in a Debian bookworm container; the Mac has no NVIDIA GPU. The model directory held the checkpoint's config.json and tokenizer files (no weights), and the vector was the same 49-layer file:
$ /tf serve /run/model --glp /run/vec/vector.gguf --context 1024
tensorfold: --glp: only the Metal engine steers: this server runs the CUDA engine
error: GlpRefused
Exit 1. The engine-open path refuses before any weights load; the text/tokenizer load runs alongside it, as startup.both always does. A refusal on a real CUDA host is still the one thing not exercised here.
Recipe: exact commands
git clone https://github.com/BlivionIaG/TensorFold.git
cd TensorFold
git checkout efd421d9615956bd3b414ab87e1305c59270c39f
# Zig 0.17.0 must be on PATH.
zig build native # Metal server
zig build tf-glp-check # tf-glp-checkOff/on decode (fresh server per run; only --glp/--glp-alpha differ):
V49=$(ls ~/.cache/huggingface/hub/models--msuiche--Qwen3.8-27B-abliterated-cyber-GLP-49/snapshots/*/Qwen3.8-27B-abliterated-cyber-GLP-49-L10-58-a1.gguf)
# off
./zig-out/native/bin/tensorfold-native serve TensorFold/Qwen3.8-27B-MLX-4bit \
--context 2048 --port 8123 --host 127.0.0.1 --name bench --no-update-check
# on
./zig-out/native/bin/tensorfold-native serve TensorFold/Qwen3.8-27B-MLX-4bit \
--glp "$V49" --glp-alpha 1.0 \
--context 2048 --port 8123 --host 127.0.0.1 --name bench --no-update-check
# for each server, once it is listening:
python3 tools/bench_openai.py http://127.0.0.1:8123 Qwen3.8-27B-MLX-4bit \
--tokens 64 --reps 5 --temperatures 0 --output /tmp/bench.jsonNested-config fixture:
import glob, json, os, shutil
snap = glob.glob(os.path.expanduser(
"~/.cache/huggingface/hub/models--TensorFold--Qwen3.8-27B-MLX-4bit/snapshots/*/"))[0]
out = "/tmp/tf-glp-nested"
shutil.rmtree(out, ignore_errors=True)
os.makedirs(out)
for name in os.listdir(snap):
os.symlink(os.path.realpath(os.path.join(snap, name)), os.path.join(out, name))
d = json.load(open(os.path.join(snap, "config.json")))
d.pop("num_hidden_layers", None)
d.pop("hidden_size", None)
tc = d.setdefault("text_config", {})
tc["num_hidden_layers"] = 32
tc["hidden_size"] = 5120
json.dump(d, open(os.path.join(out, "config.json"), "w"), indent=2)Throwaway NaN-vector generator (zig/src/core/tmp_glp_bad.zig, deleted after the run):
const std = @import("std");
const glp = @import("glp.zig");
const gguf = @import("glp_container.zig");
const width = 5120;
pub fn main(init: std.process.Init) !void {
const a = init.arena.allocator();
const args = try init.minimal.args.toSlice(a);
if (args.len != 3) return error.Usage;
const kind = args[1];
const out_path = args[2];
var dir = try a.alloc(f32, width);
for (dir) |*v| v.* = 0;
dir[0] = 1;
if (std.mem.eql(u8, kind, "deep")) {
const layers = [_]glp.Layer{.{ .id = 64, .dir = dir }};
const bytes = try glp.write(a, .{
.mode = .project,
.alpha = 1,
.hook_point = glp.residual_site,
.derived_at = null,
.base_model = "TensorFold/Qwen3.8-27B-MLX-4bit",
.base_revision = "",
.method = "test",
.contrast = "",
.layers = &layers,
});
try std.Io.Dir.cwd().writeFile(init.io, .{ .sub_path = out_path, .data = bytes });
return;
}
if (std.mem.eql(u8, kind, "nan-add")) {
var raw = try a.alloc(u8, width * 4);
std.mem.writeInt(u32, raw[0..4], @bitCast(std.math.nan(f32)), .little);
@memset(raw[4..], 0);
const entries = [_]gguf.Entry{
.{ .key = "general.architecture", .value = .{ .string = "controlvector" } },
.{ .key = "glp.spec_version", .value = .{ .u32v = 1 } },
.{ .key = "glp.mode", .value = .{ .string = "add" } },
.{ .key = "glp.rank", .value = .{ .u32v = 1 } },
.{ .key = "glp.hook_point", .value = .{ .string = glp.residual_site } },
.{ .key = "glp.alpha_default", .value = .{ .f32v = 1 } },
.{ .key = "glp.base_model", .value = .{ .string = "TensorFold/Qwen3.8-27B-MLX-4bit" } },
};
const dims = [_]u64{width};
const tensors = [_]gguf.OutTensor{.{ .name = "direction.10", .dims = &dims, .kind = gguf.f32_type, .data = raw }};
const bytes = try gguf.write(a, &entries, &tensors);
try std.Io.Dir.cwd().writeFile(init.io, .{ .sub_path = out_path, .data = bytes });
return;
}
return error.UnknownKind;
}zig run zig/src/core/tmp_glp_bad.zig -- nan-add /tmp/glp-nan-add.gguf
rm -f zig/src/core/tmp_glp_bad.zigCUDA build and container run:
zig build native -Dtarget=aarch64-linux-gnu -Doptimize=ReleaseSafe
mkdir -p "$HOME/tf-glp-cuda/run/model" "$HOME/tf-glp-cuda/run/vec"
cp zig-out/native/bin/tensorfold-native "$HOME/tf-glp-cuda/tensorfold-native-cuda-aarch64"
# rebuild the Metal binary afterwards, since the cross-build writes the same path:
zig build native
# populate run/model with the checkpoint's config.json and tokenizer files (no weights),
# and run/vec with the 49-layer vector above.
python3 - <<'PY'
import glob, os, shutil
home = os.path.expanduser("~")
snap = glob.glob(home + "/.cache/huggingface/hub/models--TensorFold--Qwen3.8-27B-MLX-4bit/snapshots/*/")[0]
model = home + "/tf-glp-cuda/run/model"
for name in ("config.json", "generation_config.json", "tokenizer.json", "tokenizer_config.json",
"vocab.json", "chat_template.jinja", "preprocessor_config.json",
"processor_config.json", "model.safetensors.index.json"):
src = os.path.join(snap, name)
if os.path.exists(src):
shutil.copy(os.path.realpath(src), os.path.join(model, name))
vec = glob.glob(home + "/.cache/huggingface/hub/models--msuiche--Qwen3.8-27B-abliterated-cyber-GLP-49/snapshots/*/Qwen3.8-27B-abliterated-cyber-GLP-49-L10-58-a1.gguf")[0]
shutil.copy(os.path.realpath(vec), home + "/tf-glp-cuda/run/vec/vector.gguf")
PY
podman run --rm --platform linux/arm64 \
-v "$HOME/tf-glp-cuda/tensorfold-native-cuda-aarch64:/tf:ro" \
-v "$HOME/tf-glp-cuda/run:/run:ro" \
docker.io/library/debian:bookworm-slim \
/tf serve /run/model --glp /run/vec/vector.gguf --context 1024 --port 8193|
Thanks @BlivionIaG for both receipts. On-mode decode is within 1% of off, and all four refusals stop by name before load. A few things before it lands:
We'll check the CUDA refusal ourselves on a GB10. |
efd421d to
b721c88
Compare
Review changes from ashhart#1835: - the app's settings panel, IPC, preload and locale packs move to feature/glp-steering-app, so app changes go through the app's review - glp.read refuses a file that names one layer twice (direction.5 and direction.05 both parse as 5), with a test - the serve refuses a layer id past the checkpoint's num_hidden_layers at startup, before the model loads - the GLP container is read by the engine's own GGUF reader; its writer stays in glp.zig, so there is one GGUF reader - the deriver, the matched prompt sets, the refusal-measurement gate and their receipt are removed; a vector file is supplied by the operator
|
Rebased onto
|
|
|
Review changes from ashhart#1835: - the app's settings panel, IPC, preload and locale packs move to feature/glp-steering-app, so app changes go through the app's review - glp.read refuses a file that names one layer twice (direction.5 and direction.05 both parse as 5), with a test - the serve refuses a layer id past the checkpoint's num_hidden_layers at startup, before the model loads - the GLP container is read by the engine's own GGUF reader; its writer stays in glp.zig, so there is one GGUF reader - the deriver, the matched prompt sets, the refusal-measurement gate and their receipt are removed; a vector file is supplied by the operator
|
For the remaining no- |
|
@BlivionIaG the script for those 20 rows isn't in the repository, so here is how it works:
The body says your machine can't run Metal right now, so we'll run these 20 rows and the sweep ourselves on a Metal Mac: You don't need to rebase. On today's main only
|
|
Device evidence for the two asks, on an Apple M4 Pro (48 GiB), No One vector, a style one. The repo carries none, so I derived a pirate-voice vector: the difference of means at the last prompt token between 12 pirate-voiced and 12 plain paragraph prompts, on the block outputs of layers 24 to 40, written with the branch's own (That line is from the alpha 1.0 run;
Strength is the operator's, as the doc says: alpha 0.25 turns the replies to shouting pirate noise ("Ahoy, me Salty Sea Dogs… YARR"), and alpha 1.0 gives repeated Decode, off/on/off/on, fresh server each, alpha 0.1,
The signs differ, so this is run-to-run noise on this machine, not a steering cost. I can attach the vector file or the derivation script if you want either. |
A settings panel beside the engine's other advanced options, the picker's path taken through the OS picker and never as a renderer argument, the engine's --glp/--glp-alpha arguments, and the audit recording the vector's state and strength rather than its path. The panel is inert when the engine does not advertise --glp. Split out of ashhart#1835, whose engine change carries the mechanism.
731afc2 to
ccac1b8
Compare
|
Rebased onto main On this head: |
|
Thanks @BlivionIaG. The rebase onto On the device evidence:
One thing would help: please attach the vector file (SHA-256
|
|
Attached as a gist: https://gist.github.com/a31f0965d77eb5aa225e8903074d22de It has Ready for your 20 rows and the sweep on |
|
Thanks @BlivionIaG for putting the file, the derivation and the writer together so quickly. We asked you to attach the file, and we need to change that ask. Sorry for the extra round. We don't load a binary that arrives as base64 in a gist. A file like that can't be reviewed by reading it, so we only load vectors we build ourselves from a recipe, or fetch from a public model repo at a fixed revision. We won't decode or load What would let us build the same vector ourselves, posted here as text in a comment so it stays with the review:
If If instead a style or topic vector for this 5120-wide checkpoint is already published on Hugging Face in a format the branch reads, name the repo and revision and we'll fetch it from there. Nothing else changes. We'll run the 20 rows and the sweep on
|
|
Understood, no binary. The full recipe as text, so you can build the vector with the branch's Prompts (12 topics in two forms, 24 in all), each sent as the user message through the checkpoint's chat template with Plain: What is captured. The output of each decoder layer (the residual stream after the block, which The direction. Per layer: mean over the 12 pirate prompts minus mean over the 12 plain prompts, in float32. No scaling, normalisation or centring. Written as fp32, one Versions. Checkpoint What the writer passes to
|
|
Kernel contract on the M4 Pro ( Please run your 20 rows and sweep on |
960448a to
1da7700
Compare
A settings panel beside the engine's other advanced options, the picker's path taken through the OS picker and never as a renderer argument, the engine's --glp/--glp-alpha arguments, and the audit recording the vector's state and strength rather than its path. The panel is inert when the engine does not advertise --glp. Split out of ashhart#1835, whose engine change carries the mechanism.
The reader implements the GLP format defined by the weightless project (@msuiche): a GGUF v3 container of one fp32 direction per layer, applied by adding it to the residual stream. A file with no mode is llama.cpp's additive vector; a file that names any other mode, project included, is refused rather than read as an add file. The reader also refuses a spec version, rank or hook point it does not implement, a layer named twice, a direction that is not finite, and a digest that does not match. It reports the SHA-256 of the exact bytes it was handed, so a caller can name the file it loaded. tf-glp-check shows the reader's view of a file for glp_parity.py to compare with the reference reader, the file hash included.
…8-27B A steered layer takes the dispatch an unsteered layer takes: the norm that adds the block's output to the residual stream also adds alpha times the layer's direction to the row, and the normalisation reads the result. The plan holds one device buffer per layer the vector names, and a model without one runs the plain norm. The steered norm clears the half-operand state the plain norm leaves for the head, since its output has no such words. The kernel contract has a float64 reference for the add, a bound for the kernel's own fp32 product and sum, the launch's threads and threadgroup memory for gate 5, and the GPU suite with a control that drops a NaN and one that breaks the normalisation. The GPU half needs a Mac.
Off unless --glp is given: without it the server is as it was. The file is read once at startup, before the model loads, and the startup line names the file (escaped), the SHA-256 of its exact bytes and that replies are steered, beside the strength, the layers and the checkpoint the vector came from. The file hash is taken over the bytes already read; nothing is read twice. A file the reader refuses, a width or base model that does not match the checkpoint, a layer past the checkpoint's depth, a Gemma checkpoint, any other family, and a CUDA or ROCm backend are all refused by name and the server does not start. The engine reports the vector in force with the same file hash. docs/control-vectors.md covers the format, the layer a direction names, the refusals and how an operator brings a vector; the repository ships none.
…ry as the compiler allocates it
1da7700 to
c5947a8
Compare
|
Commit identity fixed: author and committer on all 5 commits are now Local, M4 Pro: |
|
Thanks @BlivionIaG. The text recipe answers the revised vector request, and the contract now declares the compiler’s 48-byte allocation. The identity-only refresh is accounted for too. Nothing further is needed from you for the recipe. We’ll build the style vector from it and finish the agreed 20-row and stream-sweep checks on our M5 Ultra, using
|
What this changes
Off-by-default, add-only generic control vectors. The maintainer accepted this code; the device evidence below is on the rebased head.
6c5cef8941b27d86b06a986834e13deb9b515e1e; the branch is rebased onto it and adds 5 commits. The only conflict wasCHANGELOG.md.atc overlap --json: ATC0.9.6 was run against upstream with --base fb8fdf3; complete=false (myPullRequestsLimit, otherPullRequestsLimit, pullFilesLimit, pullFilesUnavailable, timeLimit). Supplemental complete snapshot:217foreign open heads,3038 exact local merge comparisons including latest Main ff742df; existing reviewed overlaps are retained, not claimed globally clean. The avoidable added environment-test collision with CUDA (GB10): a second carveout source, spark-attic's atticd, opt-in #2762 was removed by preserving Main's existing test name, without importing its unmerged feature.Checks on this head
c5947a8d2e9543e6c6a11bd30dc4c43c32f3b0fb.python3 tools/zig/lean_check.py): head 345, main 345, same problem set, 0 new. No touched file is over 600 lines;zig fmt --check build.zig zigis clean.zig build test test-golden dev-tools -Dcpu=apple_m1onc10442d1e(Apple M4 Pro, Zig 0.17.0): 13,265 of 13,369 tests pass, 103 skipped, 1 failed. The failure isdashboard_test("Paged prefix cache"), which fails the same way on main. After the last commit,zig build test-steer test-glp test-glp-serve test-glp-steerpasses 88 of 88 tests.c10442d1e: 12 successful checks, 2 scheduled-only skips,desktopanddesktop-linuxincluded. CI on the head below is running.--glpthere by name), and an M5 Ultra run of the 20 rows and the sweep, which the maintainer runs.Receipt
--glp(Apple M4 Pro, 48 GiB,TensorFold/Qwen3.8-27B-MLX-4bitat22d8d538154e): the 16 prompts oftools/bench_prompts.json, greedy, 64 tokens, 16 streams; every row'stoken_sha256is equal across the merge base8a7838a65and the earlier head731afc275, each served twice. Earlier, on36fa3b08cagainst981fe201d, 14 of 14 cells were equal and within 2.3% (comment of 5 Nov). The 20 long-prompt rows and the 1-8 stream sweep onc10442d1eare the maintainer's; his M5 Ultra cell on 5 Nov was GREEN.c5947a8d2e9543e6c6a11bd30dc4c43c32f3b0fb; Zig 0.17.0; Apple M4 Pro, 48 GiB, Metal; checkpointTensorFold/Qwen3.8-27B-MLX-4bitat22d8d538154e0e5b6f8dcb1fcc74a60a0798104c.--glpby name and were not run.36fa3b08calso passed the alone and serial checks, 18 of 18 and 4 of 4.--glp-alpha 0.1; the startup line and the with/without replies are in the 12 Nov comment, and the recipe is in a comment. The file is not part of this PR.zig build tf-kernel-contract, thentf-kernel-contract --only steer):SUMMARY 1 kernels: 1 pass gates 1-4, 0 fail, 0 known findings; 1 controls: 1 red as planted, 0 missed; 0 limit problems, exit 0. The first run failed gate 5 because the steering launch declared 40 threadgroup bytes where the compiler allocates 48;c5947a8d2declares 48. M5 and other chips are not run.Remaining maintainer requests
The maintainer's 20 rows and 1-8 stream sweep on
c10442d1eagainst6c5cef894, and his build of the style vector from the recipe in the comments.Checklist
Title/description, head and stack agree; older results remain source-bound.
Every new/restored test has a failing-before runtime receipt across all reached platforms. Host evidence is above; missing device gates are not waived.
No newly added overlong/multiline source comments, measurements or review history; inherited findings are reported above.
Complete publication trees/outgoing histories reviewed for public noreply identity, tool trailers and privacy; synthetic detector fixtures and mandatory third-party public contacts are not live secrets. No package/binary upload proposed.
All inference-path/lane-round gates qualified on this head.
Drafted=plain, restored=fresh, concurrent=solo on required devices/settings.
Required precision/reference evidence on this exact head complete.
Decode/prompt no slower than Main at the required widths, greedy and sampled.
Historical source-bound notes — not current-head qualification
Earlier commands/measurements retain their original commit/binary scope. Any earlier readiness or provenance claim is superseded by the open gates above.
Add-only control vectors
c10442d1eagainst6c5cef894; the style vector is built from the recipe posted in the comments, or fetched from a public model repo at a fixed revision.Rebased onto main
6c5cef8941b27d86b06a986834e13deb9b515e1e; published headc5947a8d2e9543e6c6a11bd30dc4c43c32f3b0fb.Implements the GLP format defined by the
weightlessproject (@msuiche), with an operator-supplied file for non-refusal style or topic steering. No refusal-removal vectors, prompt sets, derived assets, download links or old measurement receipts are included.--glp FILE, no vector is applied.--glp-alphaoverrides the file strength.projectand every other non-add mode by name. No directional-ablation kernel or CLI mode remains.Exercised qualification
Open maintainer gates
Still owed, and the maintainer's to run: the 20 long-prompt rows and the 1-8 stream sweep with no
--glponc10442d1eagainst6c5cef894(Metal, M5), and the style vector built from the recipe in the comments or fetched from a public model repo at a fixed revision. No CUDA or ROCm steering result is claimed; those backends refuse--glpby name.Refs #1834 until those runs match main with level speed.
Rebased onto main
6c5cef894; the only conflict wasCHANGELOG.md.zig fmt --check build.zig zigis clean. Old source-bound receipts are not relabeled.CI on the current head: 12 successful checks and 2 scheduled-only skips,
desktopanddesktop-linuxincluded. This does not waive the maintainer's 20 rows, the sweep or the style-vector check.