Optimizer phase 3: the TUI and the web console

AGENTS.md

@@ -12,7 +12,7 @@ Sibyl is a Fedora bootc host image plus the container images that it runs. Read
 - `os/rootfs/` is copied to `/` of the host. Quadlet units are in `os/rootfs/etc/containers/systemd/users/1001/`, and they run rootless as the `sibyl` user (UID 1001).
 - `containers/` holds one directory per service image. `containers/system-api` builds from the repo root.
 - `cmd/` and `internal/` are the Go System API. The design is in the "System API and host agent" section of `docs/design/overview.md`, and every route is in `docs/reference/system-api.md`.
-- `os/rootfs/usr/libexec/sibyl/agent` is the host agent. It is the only path from a container to systemd. Keep its allowlist in step with `hostctl.Units`.
+- `os/rootfs/usr/libexec/sibyl/agent` is the host agent. It is the only path from a container to systemd and Podman. Keep its allowlist in step with `hostctl.Units`. Its `tune` request checks every optimizer argument against a pattern for each flag, and `internal/hostctl/agent_test.go` tests the refusals.
 - `cmd/sibyl-hostd` is the root agent for hostname, network and accounts. `os/Containerfile` builds it into the host image. Its checks in `internal/hostd` are the security boundary, so keep every new action in `hostd.Decode` and test the refusals. The one exception is `password-check` in `internal/hostd/auth.go`. It is kept out of `Decode` on purpose, so the disk queue refuses it, and only the memory queue `/run/sibyl-auth/req` takes it.
 - `internal/modelform` is the model settings form that the TUI and the console share.
 - `internal/console` is the web console at `/console/`. It calls the System API through `internal/client` in process. Keep every console change on an existing API route, and `docs/design/web-console.md` in step.

containers/llama-swap-cuda/Containerfile

@@ -22,7 +22,7 @@ SHELL ["/bin/bash", "-o", "pipefail", "-c"]
 # The upstream image keeps llama.cpp in /app, where llama-swap lives in the
 # other Sibyl images. Move it to /opt/llama, so every image has one layout.
 RUN mkdir -p /opt/llama/bin /opt/llama/lib \
-    && mv /app/llama-server /app/llama-bench /opt/llama/bin/ \
+    && mv /app/llama-server /app/llama-bench /app/llama-batched-bench /opt/llama/bin/ \
     && mv /app/*.so* /opt/llama/lib/ \
     && rm -rf /app \
     && mkdir -p /app/config \

containers/llama-swap/Containerfile.hip

@@ -29,12 +29,12 @@ RUN HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
         -DLLAMA_BUILD_EXAMPLES=OFF \
         -DLLAMA_BUILD_UI=OFF \
         -DLLAMA_USE_PREBUILT_UI=OFF \
-    && cmake --build build --config Release -j"$(nproc)" --target llama-server llama-bench
+    && cmake --build build --config Release -j"$(nproc)" --target llama-server llama-bench llama-batched-bench
 
 # Collect the binaries, their shared libraries, and only the rocBLAS and
 # hipBLASLt kernel files for GPU_TARGETS. The full ROCm tree is ~20 GB.
 COPY collect-rocm-libs.sh /usr/local/bin/collect-rocm-libs.sh
-RUN /usr/local/bin/collect-rocm-libs.sh /out "${GPU_TARGETS}" build/bin/llama-server build/bin/llama-bench
+RUN /usr/local/bin/collect-rocm-libs.sh /out "${GPU_TARGETS}" build/bin/llama-server build/bin/llama-bench build/bin/llama-batched-bench
 
 FROM docker.io/ubuntu:24.04 AS llama-swap-dl
 

containers/llama-swap/Containerfile.vulkan

@@ -21,9 +21,9 @@ RUN cmake -S . -B build \
         -DLLAMA_BUILD_EXAMPLES=OFF \
         -DLLAMA_BUILD_UI=OFF \
         -DLLAMA_USE_PREBUILT_UI=OFF \
-    && cmake --build build --config Release -j"$(nproc)" --target llama-server llama-bench \
+    && cmake --build build --config Release -j"$(nproc)" --target llama-server llama-bench llama-batched-bench \
     && mkdir -p /out/opt/llama/bin /out/opt/llama/lib \
-    && cp build/bin/llama-server build/bin/llama-bench /out/opt/llama/bin/ \
+    && cp build/bin/llama-server build/bin/llama-bench build/bin/llama-batched-bench /out/opt/llama/bin/ \
     && find build -name '*.so*' -exec cp -P {} /out/opt/llama/lib/ \;
 
 FROM docker.io/ubuntu:26.04 AS llama-swap-dl

docs/design/optimizer.md

@@ -0,0 +1,104 @@
+# Inference optimizer design
+
+This document describes the inference optimizer. It finds the llama.cpp settings that run a model fastest on the host's own hardware, by measurement. An upstream LLM can suggest more settings to try, but only a measurement decides. [The design overview](overview.md) holds the rest of the design.
+
+## Goals
+
+- Find faster settings for one model on this host, with numbers that the owner can check.
+- Never apply settings that were not measured, and never apply settings that do not load at the model's real context size.
+- Keep the owner in control: the optimizer recommends, and the owner applies with one action.
+- Send no prompts, no user data and no model files off the box. The optional advisor gets hardware facts, model facts and benchmark numbers only.
+
+## What already exists
+
+- `internal/models/fit.go` chooses the context size and the KV cache types from the free VRAM.
+- `POST /models/{id}/benchmark` runs `llama-bench` once in each backend image through the host agent (`os/rootfs/usr/libexec/sibyl/agent`, `bench`), and recommends the backend with the fastest generation.
+- `models.Meta` reads the layer count, the expert count and the expert tensor sizes from the GGUF file.
+
+The optimizer replaces the fixed benchmark with a search over more settings, and it keeps the same path to the host.
+
+## Profiles
+
+The owner picks a profile for each run.
+
+| Profile | Score | Tool |
+|---|---|---|
+| `single` | Generation tokens per second at a context depth, then prompt tokens per second | `llama-bench` |
+| `throughput` | Total tokens per second over parallel requests, with a floor on the speed of each request | `llama-batched-bench` |
+
+The context depth of the `single` profile is the number of tokens that are already in the KV cache when the test runs. It is 8,192 tokens, or half the model's context if that is smaller. An empty cache makes generation look faster than a real chat. The floor of the `throughput` profile is half of the single-request generation speed, so a high total with slow answers does not win.
+
+## Settings that the search covers
+
+| Setting | llama.cpp flag | Model setting | Note |
+|---|---|---|---|
+| Batch and micro-batch | `-b`, `-ub` | `batch`, `ubatch` (new) | Large effect on prompt speed. A larger micro-batch needs a larger compute buffer. |
+| KV cache type | `-ctk`, `-ctv` | `kv_cache` | `q8_0` and `q4_0` free VRAM for context, and lower quality a little. |
+| MoE experts in system RAM | `-ncmoe`, `-ot` | `moe.cpu_layers`, `args` | Only for a model whose experts do not fit in VRAM. |
+| Split mode | `-sm` | `split_mode` (new) | `layer` or `row`, only with more than one GPU. |
+| Tensor split | `-ts` | `tensor_split` | Only with more than one GPU. |
+| Parallel slots | not in `llama-bench` | `parallel` | The `throughput` profile only. |
+| Backend | the image | the hardware configuration | The backend applies to every model, so the optimizer reports it apart, and applying it restarts llama-swap. |
+
+Flash attention stays on, and all layers stay on the GPUs, as llama-swap runs them today. Threads matter only for layers on the CPU, so the search leaves them alone.
+
+## The search
+
+The search is a fixed sequence of stages. Each stage runs one tool once. `llama-bench` takes a comma list for each flag and tests every combination, so one run covers one stage.
+
+1. Baseline: the model's current settings.
+2. Backend: each backend that can run on the selected GPUs. With more than one GPU, split mode `layer` and `row`.
+3. KV cache: `f16` and `q8_0`. A cache type that does not fit the model's context in VRAM (`FitContext`) is left out.
+4. Batch: micro-batch 256, 512, 1024 and 2048, with batch 2048 and 4096, where the batch is at least the micro-batch.
+5. MoE: for a model whose experts do not fit in VRAM, the smallest `cpu_layers` value that fits, and two values above it.
+6. Advisor: the candidates that the advisor proposes, if the owner turned it on.
+7. Parallel: for the `throughput` profile, 1, 2, 4 and 8 parallel sequences at the model's context.
+8. Verify: the host agent starts `llama-server` with the best settings at the model's real context size, waits for `/health`, and stops it.
+
+Each stage keeps the best result so far and passes it to the next stage. A result wins only when it beats the current best by more than three standard deviations of the two runs, or by 3 percent, whichever is larger. `llama-bench` reports the standard deviation (`stddev_ts`). Noise therefore does not change a setting.
+
+`llama-bench` allocates a context only for the test, not for the model's full context. A micro-batch that fits in the test can still fail at the real context. The verify stage catches that case. If the verify fails, the optimizer drops the last changed setting and verifies again.
+
+## The path to the host
+
+The System API runs in a container and has no access to Podman. The optimizer uses the host agent, as the benchmark does.
+
+- A new agent request, `tune.<model>`, runs one tool from `$RUN/tune/<model>.*` files: the tool (`bench`, `batched` or `server`), the backend, the model path, the GPUs and the arguments.
+- The agent checks every argument against an allowlist of flags. Each flag has a pattern for its value. An argument that is not on the list fails the request. The model path, the pool and the GPUs get the same checks as the benchmark.
+- The tool runs in the backend image with no network, all capabilities dropped and `no-new-privileges`, as the benchmark does.
+- The images build `llama-batched-bench` as well as `llama-server` and `llama-bench`.
+
+The search itself runs in the System API (`internal/optimize`). It writes one request for each stage, and reads the result files.
+
+## The advisor
+
+The advisor is off by default. When the owner turns it on, the optimizer asks one provider model that the router already has.
+
+- A new router route, `POST /sibyl/advise`, takes a provider and a model. Only the System API key can call it, as with `/sibyl/classify`. The provider keys stay in the router.
+- The optimizer sends the GPU names and VRAM, the system RAM, the backend, the model architecture, the parameter count, the quantization, the layer and expert counts, the context size, and every measured result.
+- The advisor answers with JSON: up to six candidates, each one a set of the settings in the table above. The optimizer checks each candidate with the same allowlist as the agent, and drops what fails.
+- The advisor is most useful for `-ot` patterns on a new MoE architecture, and for the order of the stages on hardware that the fixed search does not know.
+
+## API
+
+| Route | Purpose |
+|---|---|
+| `POST /models/{id}/optimize` | Start a run: `{"profile": "single" or "throughput", "advisor": {"provider": "...", "model": "..."}}`. The advisor is optional. |
+| `GET /models/{id}/optimize` | The state: the stage, every measured result, and the recommendation when the run is done. |
+| `DELETE /models/{id}/optimize` | Cancel the run. |
+| `POST /models/{id}/optimize/apply` | Write the recommended model settings, and reload llama-swap. |
+
+One run goes at a time on the host, because each run unloads the models and uses every selected GPU.
+
+## Limits
+
+- A run unloads the running models, so inference stops for its length: about 5 to 15 minutes for one model.
+- `llama-bench` measures the model alone. It does not measure speculative decoding, so a model with a draft model keeps its speculative settings, and the optimizer measures without them.
+- A recommendation is for the hardware and the llama.cpp version of the run. An update of either one makes the result old, and the console says so.
+
+## Phases
+
+1. Model settings `batch`, `ubatch` and `split_mode`. The `tune` agent request and its allowlist. `llama-batched-bench` in the images.
+2. `internal/optimize`, the API routes, and the verify stage.
+3. The TUI and the console.
+4. The advisor and `/sibyl/advise`.

docs/design/web-console.md

@@ -88,7 +88,7 @@ The console has the pages Router, Models, Downloads, Hardware, Host, Status and
 |---|---|---|
 | Status | Version, update state, services, GPU memory, the loaded models. The loaded models refresh every 5 seconds. | Restart a service, unload a model, connect Open WebUI, check for an update, apply an update with a reboot, set the nightly update window, reboot |
 | Hardware | GPUs, the selected GPUs, the backend, memory | Select GPUs, select the backend |
-| Models | Installed models with size, fit, context, layout, benchmark and load state. Each drafter is under the model that uses it. | Load or unload, model settings (the TUI settings form), benchmark, delete |
+| Models | Installed models with size, fit, context, layout, benchmark and load state. Each drafter is under the model that uses it. | Load or unload, model settings (the TUI settings form), benchmark, the optimizer (start, cancel, apply), delete |
 | Drafters | The drafters on Hugging Face for one model, and its last drafter test. The test block refreshes while a test runs. | Test an installed drafter, open the download page of the others, turn the drafter on after a test |
 | Downloads | Hugging Face search, the queue with progress | Search, pick a file, start, cancel, retry, set the Hugging Face token |
 | Host | Hostname, network, Tailscale, users, SSH keys, model pools | Change each, with the same checks as `sibyl-hostd` |

docs/guides/consoles.md

@@ -33,7 +33,7 @@ The bottom of each screen shows its actions as buttons, such as `[n] new client`
 |---|---|---|
 | `1` | Status | `r` restarts the selected service. `o` connects Open WebUI with an admin key. `u` checks for an update. `U` applies it. `B` reboots into a staged update. |
 | `2` | Hardware | `e` changes the GPUs, the backend and the default layout. |
-| `3` | Models | The list shows each model with its size and whether it is loaded, each drafter under the model that uses it, and the unused drafters last. The details show the context, the settings, the drafter and the benchmark. `enter` edits the settings. `l` loads the model, or unloads it when it is loaded. `f` finds a drafter. `b` benchmarks the backends. `d` deletes the model. |
+| `3` | Models | The list shows each model with its size and whether it is loaded, each drafter under the model that uses it, and the unused drafters last. The details show the context, the settings, the drafter and the benchmark. `enter` edits the settings. `l` loads the model, or unloads it when it is loaded. `f` finds a drafter. `b` benchmarks the backends. `o` starts the optimizer, or cancels it. `a` applies the optimizer's settings. `d` deletes the model. |
 | `4` | Download | `/` searches Hugging Face. `t` sets a Hugging Face token. `x` cancels a download. `R` retries one. |
 | `5` | Host | `h` hostname. `n` network. `t` joins or leaves a tailnet. `a` adds an operator. `p` makes a model pool. `x` detaches a pool. For a selected operator, `s` changes the SSH keys and `d` deletes the operator. After a network change, `c` keeps the change. |
 | `6` | Router | Five sections: Clients, Providers, Profiles, Calibration and Activity. `←` and `→` change the section. Clients: `n` new client, `e` edit, `K` new key, `d` delete. Providers: `n` add, `e` change, `d` delete. Profiles: `n` new profile, `e` edit the selected profile or rule, `a` add a rule, `<` and `>` move a rule, `d` delete, `s` sensitive words and classifier. Calibration: `e` settings, `l` loads the model that the samples wait for, `a` applies a proposal, `C` deletes the samples. |
@@ -85,7 +85,7 @@ If a network change gives the host a new address, open the console at the new ad
 | Page | What you can do |
 |---|---|
 | Router | Clients and keys, providers, profiles and rules, calibration settings, sensitive words and the classifier, calibration results and proposals, and the last decisions. See [Use the router](router.md). |
-| Models | See the installed models. Edit a model's settings, run the benchmark, or delete a model. See [Models](models.md). |
+| Models | See the installed models. Edit a model's settings, run the benchmark or the optimizer, or delete a model. See [Models](models.md). |
 | Downloads | Search Hugging Face, start a download, cancel or retry one, and set or remove the Hugging Face token. See [Models](models.md). |
 | Hardware | Select the GPUs, the backend and the default layout. |
 | Host | Change the hostname and the network, join or leave a tailnet, add or delete operators and their SSH keys, and make or detach model pools. See [Host](host.md). |

docs/guides/models.md

@@ -180,6 +180,20 @@ A backend is the GPU library that llama.cpp uses: `hip` (ROCm), `cuda` or `vulka
 
 The backend applies to every model, because llama-swap runs one image.
 
+## Optimize a model
+
+The optimizer measures llama.cpp settings for one model on your hardware, and recommends the fastest that loads. It covers the backend, the split mode across GPUs, the KV cache type, the batch sizes, the MoE layers in system RAM and, for throughput, the number of parallel requests. [The optimizer design](../design/optimizer.md) describes each stage.
+
+1. On the Models screen, select a model and press `o`. In the web console, open the model's page and use the Optimizer section.
+2. Choose what to optimize for:
+   - One user: generation speed, then prompt speed. Use this for one person or one agent.
+   - Throughput: the total speed over parallel requests, where each request keeps at least half the speed of one alone.
+3. Wait 5 to 15 minutes. Sibyl unloads the running models while the optimizer runs. Press `o` again to cancel.
+4. Read the result in the model's details: the speed now, the best speed, and each setting that changes.
+5. Press `a` to apply the settings. In the web console, use "Apply these settings". If a different backend was faster, Sibyl asks whether to switch the backend too. The backend applies to every model.
+
+A setting wins only when it beats the settings now by more than 3 percent and by more than the noise of the runs. The optimizer applies nothing by itself. A result is for the hardware and the llama.cpp version of the run, so run it again after an update.
+
 ## Model pools
 
 A model pool is an extra disk for models. Sibyl encrypts it and mounts it under `/var/lib/sibyl-pools`. The disk must be at least 16 GiB. It cannot be the system disk, or a disk that is mounted or in use.

docs/index.md

@@ -39,6 +39,7 @@ These pages are for contributors. They explain how Sibyl works and why.
 | [Overview](design/overview.md) | The decisions, the host image, the services, storage, security and the phase plan |
 | [Router](design/router.md) | The model router, the classifier, calibration and their measurements |
 | [Web console](design/web-console.md) | The web console, its sign-in and its security |
+| [Inference optimizer](design/optimizer.md) | The measured search for faster llama.cpp settings, and the optional LLM advisor |
 | [Diagnostics](design/diagnostics.md) | The watchdog, metrics, incidents, opt-in upload and the fleet service plan |
 | [Development](development.md) | Building, testing, CI, releases and testing on hardware |