Optimizer phase 1: batch settings, the agent's tune request, batched-bench

AGENTS.md

@@ -12,7 +12,7 @@ Sibyl is a Fedora bootc host image plus the container images that it runs. Read
 - `os/rootfs/` is copied to `/` of the host. Quadlet units are in `os/rootfs/etc/containers/systemd/users/1001/`, and they run rootless as the `sibyl` user (UID 1001).
 - `containers/` holds one directory per service image. `containers/system-api` builds from the repo root.
 - `cmd/` and `internal/` are the Go System API. The design is in the "System API and host agent" section of `docs/design/overview.md`, and every route is in `docs/reference/system-api.md`.
-- `os/rootfs/usr/libexec/sibyl/agent` is the host agent. It is the only path from a container to systemd. Keep its allowlist in step with `hostctl.Units`.
+- `os/rootfs/usr/libexec/sibyl/agent` is the host agent. It is the only path from a container to systemd and Podman. Keep its allowlist in step with `hostctl.Units`. Its `tune` request checks every optimizer argument against a pattern for each flag, and `internal/hostctl/agent_test.go` tests the refusals.
 - `cmd/sibyl-hostd` is the root agent for hostname, network and accounts. `os/Containerfile` builds it into the host image. Its checks in `internal/hostd` are the security boundary, so keep every new action in `hostd.Decode` and test the refusals. The one exception is `password-check` in `internal/hostd/auth.go`. It is kept out of `Decode` on purpose, so the disk queue refuses it, and only the memory queue `/run/sibyl-auth/req` takes it.
 - `internal/modelform` is the model settings form that the TUI and the console share.
 - `internal/console` is the web console at `/console/`. It calls the System API through `internal/client` in process. Keep every console change on an existing API route, and `docs/design/web-console.md` in step.

containers/llama-swap-cuda/Containerfile

@@ -22,7 +22,7 @@ SHELL ["/bin/bash", "-o", "pipefail", "-c"]
 # The upstream image keeps llama.cpp in /app, where llama-swap lives in the
 # other Sibyl images. Move it to /opt/llama, so every image has one layout.
 RUN mkdir -p /opt/llama/bin /opt/llama/lib \
-    && mv /app/llama-server /app/llama-bench /opt/llama/bin/ \
+    && mv /app/llama-server /app/llama-bench /app/llama-batched-bench /opt/llama/bin/ \
     && mv /app/*.so* /opt/llama/lib/ \
     && rm -rf /app \
     && mkdir -p /app/config \

containers/llama-swap/Containerfile.hip

@@ -29,12 +29,12 @@ RUN HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
         -DLLAMA_BUILD_EXAMPLES=OFF \
         -DLLAMA_BUILD_UI=OFF \
         -DLLAMA_USE_PREBUILT_UI=OFF \
-    && cmake --build build --config Release -j"$(nproc)" --target llama-server llama-bench
+    && cmake --build build --config Release -j"$(nproc)" --target llama-server llama-bench llama-batched-bench
 
 # Collect the binaries, their shared libraries, and only the rocBLAS and
 # hipBLASLt kernel files for GPU_TARGETS. The full ROCm tree is ~20 GB.
 COPY collect-rocm-libs.sh /usr/local/bin/collect-rocm-libs.sh
-RUN /usr/local/bin/collect-rocm-libs.sh /out "${GPU_TARGETS}" build/bin/llama-server build/bin/llama-bench
+RUN /usr/local/bin/collect-rocm-libs.sh /out "${GPU_TARGETS}" build/bin/llama-server build/bin/llama-bench build/bin/llama-batched-bench
 
 FROM docker.io/ubuntu:24.04 AS llama-swap-dl
 

containers/llama-swap/Containerfile.vulkan

@@ -21,9 +21,9 @@ RUN cmake -S . -B build \
         -DLLAMA_BUILD_EXAMPLES=OFF \
         -DLLAMA_BUILD_UI=OFF \
         -DLLAMA_USE_PREBUILT_UI=OFF \
-    && cmake --build build --config Release -j"$(nproc)" --target llama-server llama-bench \
+    && cmake --build build --config Release -j"$(nproc)" --target llama-server llama-bench llama-batched-bench \
     && mkdir -p /out/opt/llama/bin /out/opt/llama/lib \
-    && cp build/bin/llama-server build/bin/llama-bench /out/opt/llama/bin/ \
+    && cp build/bin/llama-server build/bin/llama-bench build/bin/llama-batched-bench /out/opt/llama/bin/ \
     && find build -name '*.so*' -exec cp -P {} /out/opt/llama/lib/ \;
 
 FROM docker.io/ubuntu:26.04 AS llama-swap-dl

docs/design/optimizer.md

@@ -0,0 +1,104 @@
+# Inference optimizer design
+
+This document describes the inference optimizer. It finds the llama.cpp settings that run a model fastest on the host's own hardware, by measurement. An upstream LLM can suggest more settings to try, but only a measurement decides. [The design overview](overview.md) holds the rest of the design.
+
+## Goals
+
+- Find faster settings for one model on this host, with numbers that the owner can check.
+- Never apply settings that were not measured, and never apply settings that do not load at the model's real context size.
+- Keep the owner in control: the optimizer recommends, and the owner applies with one action.
+- Send no prompts, no user data and no model files off the box. The optional advisor gets hardware facts, model facts and benchmark numbers only.
+
+## What already exists
+
+- `internal/models/fit.go` chooses the context size and the KV cache types from the free VRAM.
+- `POST /models/{id}/benchmark` runs `llama-bench` once in each backend image through the host agent (`os/rootfs/usr/libexec/sibyl/agent`, `bench`), and recommends the backend with the fastest generation.
+- `models.Meta` reads the layer count, the expert count and the expert tensor sizes from the GGUF file.
+
+The optimizer replaces the fixed benchmark with a search over more settings, and it keeps the same path to the host.
+
+## Profiles
+
+The owner picks a profile for each run.
+
+| Profile | Score | Tool |
+|---|---|---|
+| `single` | Generation tokens per second at a context depth, then prompt tokens per second | `llama-bench` |
+| `throughput` | Total tokens per second over parallel requests, with a floor on the speed of each request | `llama-batched-bench` |
+
+The context depth of the `single` profile is the number of tokens that are already in the KV cache when the test runs. It is 8,192 tokens, or half the model's context if that is smaller. An empty cache makes generation look faster than a real chat. The floor of the `throughput` profile is half of the single-request generation speed, so a high total with slow answers does not win.
+
+## Settings that the search covers
+
+| Setting | llama.cpp flag | Model setting | Note |
+|---|---|---|---|
+| Batch and micro-batch | `-b`, `-ub` | `batch`, `ubatch` (new) | Large effect on prompt speed. A larger micro-batch needs a larger compute buffer. |
+| KV cache type | `-ctk`, `-ctv` | `kv_cache` | `q8_0` and `q4_0` free VRAM for context, and lower quality a little. |
+| MoE experts in system RAM | `-ncmoe`, `-ot` | `moe.cpu_layers`, `args` | Only for a model whose experts do not fit in VRAM. |
+| Split mode | `-sm` | `split_mode` (new) | `layer` or `row`, only with more than one GPU. |
+| Tensor split | `-ts` | `tensor_split` | Only with more than one GPU. |
+| Parallel slots | not in `llama-bench` | `parallel` | The `throughput` profile only. |
+| Backend | the image | the hardware configuration | The backend applies to every model, so the optimizer reports it apart, and applying it restarts llama-swap. |
+
+Flash attention stays on, and all layers stay on the GPUs, as llama-swap runs them today. Threads matter only for layers on the CPU, so the search leaves them alone.
+
+## The search
+
+The search is a fixed sequence of stages. Each stage runs one tool once. `llama-bench` takes a comma list for each flag and tests every combination, so one run covers one stage.
+
+1. Baseline: the model's current settings.
+2. Backend: each backend that can run on the selected GPUs. With more than one GPU, split mode `layer` and `row`.
+3. KV cache: `f16` and `q8_0`. A cache type that does not fit the model's context in VRAM (`FitContext`) is left out.
+4. Batch: micro-batch 256, 512, 1024 and 2048, with batch 2048 and 4096, where the batch is at least the micro-batch.
+5. MoE: for a model whose experts do not fit in VRAM, the smallest `cpu_layers` value that fits, and two values above it.
+6. Advisor: the candidates that the advisor proposes, if the owner turned it on.
+7. Parallel: for the `throughput` profile, 1, 2, 4 and 8 parallel sequences at the model's context.
+8. Verify: the host agent starts `llama-server` with the best settings at the model's real context size, waits for `/health`, and stops it.
+
+Each stage keeps the best result so far and passes it to the next stage. A result wins only when it beats the current best by more than three standard deviations of the two runs, or by 3 percent, whichever is larger. `llama-bench` reports the standard deviation (`stddev_ts`). Noise therefore does not change a setting.
+
+`llama-bench` allocates a context only for the test, not for the model's full context. A micro-batch that fits in the test can still fail at the real context. The verify stage catches that case. If the verify fails, the optimizer drops the last changed setting and verifies again.
+
+## The path to the host
+
+The System API runs in a container and has no access to Podman. The optimizer uses the host agent, as the benchmark does.
+
+- A new agent request, `tune.<model>`, runs one tool from `$RUN/tune/<model>.*` files: the tool (`bench`, `batched` or `server`), the backend, the model path, the GPUs and the arguments.
+- The agent checks every argument against an allowlist of flags. Each flag has a pattern for its value. An argument that is not on the list fails the request. The model path, the pool and the GPUs get the same checks as the benchmark.
+- The tool runs in the backend image with no network, all capabilities dropped and `no-new-privileges`, as the benchmark does.
+- The images build `llama-batched-bench` as well as `llama-server` and `llama-bench`.
+
+The search itself runs in the System API (`internal/optimize`). It writes one request for each stage, and reads the result files.
+
+## The advisor
+
+The advisor is off by default. When the owner turns it on, the optimizer asks one provider model that the router already has.
+
+- A new router route, `POST /sibyl/advise`, takes a provider and a model. Only the System API key can call it, as with `/sibyl/classify`. The provider keys stay in the router.
+- The optimizer sends the GPU names and VRAM, the system RAM, the backend, the model architecture, the parameter count, the quantization, the layer and expert counts, the context size, and every measured result.
+- The advisor answers with JSON: up to six candidates, each one a set of the settings in the table above. The optimizer checks each candidate with the same allowlist as the agent, and drops what fails.
+- The advisor is most useful for `-ot` patterns on a new MoE architecture, and for the order of the stages on hardware that the fixed search does not know.
+
+## API
+
+| Route | Purpose |
+|---|---|
+| `POST /models/{id}/optimize` | Start a run: `{"profile": "single" or "throughput", "advisor": {"provider": "...", "model": "..."}}`. The advisor is optional. |
+| `GET /models/{id}/optimize` | The state: the stage, every measured result, and the recommendation when the run is done. |
+| `DELETE /models/{id}/optimize` | Cancel the run. |
+| `POST /models/{id}/optimize/apply` | Write the recommended model settings, and reload llama-swap. |
+
+One run goes at a time on the host, because each run unloads the models and uses every selected GPU.
+
+## Limits
+
+- A run unloads the running models, so inference stops for its length: about 5 to 15 minutes for one model.
+- `llama-bench` measures the model alone. It does not measure speculative decoding, so a model with a draft model keeps its speculative settings, and the optimizer measures without them.
+- A recommendation is for the hardware and the llama.cpp version of the run. An update of either one makes the result old, and the console says so.
+
+## Phases
+
+1. Model settings `batch`, `ubatch` and `split_mode`. The `tune` agent request and its allowlist. `llama-batched-bench` in the images.
+2. `internal/optimize`, the API routes, and the verify stage.
+3. The TUI and the console.
+4. The advisor and `/sibyl/advise`.

docs/index.md

@@ -39,6 +39,7 @@ These pages are for contributors. They explain how Sibyl works and why.
 | [Overview](design/overview.md) | The decisions, the host image, the services, storage, security and the phase plan |
 | [Router](design/router.md) | The model router, the classifier, calibration and their measurements |
 | [Web console](design/web-console.md) | The web console, its sign-in and its security |
+| [Inference optimizer](design/optimizer.md) | The measured search for faster llama.cpp settings, and the optional LLM advisor |
 | [Diagnostics](design/diagnostics.md) | The watchdog, metrics, incidents, opt-in upload and the fleet service plan |
 | [Development](development.md) | Building, testing, CI, releases and testing on hardware |
 

docs/reference/configuration.md

@@ -61,6 +61,9 @@ qwen3-32b-q4_k_m:
 | `reasoning.value` | string | `on` or `off` for `enable_thinking` and `thinking`. For `reasoning_effort`, 1 to 32 letters, digits, `_` and `-`. | `--chat-template-kwargs` | The default reasoning. See [Reasoning settings](#reasoning-settings). |
 | `parallel` | integer | 0 to 64 | `--parallel` | The number of requests that the model answers at once. The requests share the context. 0 leaves llama-server's default. |
 | `tensor_split` | list of numbers | 0 or more each, at most one value per selected GPU | `--tensor-split` | The share of the layers on each GPU of a spanning model. Empty lets Sibyl split by the memory that each GPU has left. |
+| `batch` | integer | 0, or 32 to 65536 | `--batch-size` | The logical batch size. 0 leaves llama-server's default. |
+| `ubatch` | integer | 0, or 32 to 65536, and not larger than `batch` | `--ubatch-size` | The physical batch size. A larger value makes the prompt faster and needs a larger compute buffer on the GPU. 0 leaves llama-server's default. |
+| `split_mode` | string | `layer` or `row` | `--split-mode` | How a spanning model splits across the GPUs. Empty is `layer`. It has no effect on one GPU. |
 | `kv_cache.type_k` | string | `f32`, `f16`, `bf16`, `q8_0`, `q4_0`, `q4_1`, `iq4_nl`, `q5_0`, `q5_1` | `--cache-type-k` | The K cache type. |
 | `kv_cache.type_v` | string | same as `type_k` | `--cache-type-v` | The V cache type. |
 | `speculative.types` | list of strings | `none`, `draft-simple`, `draft-eagle3`, `draft-mtp`, `draft-dflash`, `draft-dspark`, `ngram-simple`, `ngram-map-k`, `ngram-map-k4v`, `ngram-mod`, `ngram-cache` | `--spec-type` | The speculative decoding types, tried in order. |

docs/reference/system-api.md

@@ -159,7 +159,7 @@ Each model in `GET /models` has these fields:
 | `drafter` | `true` for a DFlash or EAGLE-3 drafter. llama-swap, the router and the Ollama list do not serve it, and it has no `fit`. |
 | `drafters` | The number of drafters on Hugging Face, from the last search. Missing until a search finishes. `GET /models` starts the searches in the background. |
 
-The settings body of `PUT /models/{id}` has these fields: `layout`, `ctx_size`, `embedding`, `hidden`, `sampling`, `speculative`, `kv_cache`, `moe`, `fit`, `preset`, `reasoning`, `parallel`, `tensor_split` and `args`. [Configuration](configuration.md) describes each field and its range. The route returns `404` for a model that is not installed. It returns `422` in these cases:
+The settings body of `PUT /models/{id}` has these fields: `layout`, `ctx_size`, `embedding`, `hidden`, `sampling`, `speculative`, `kv_cache`, `moe`, `fit`, `preset`, `reasoning`, `parallel`, `tensor_split`, `batch`, `ubatch`, `split_mode` and `args`. [Configuration](configuration.md) describes each field and its range. The route returns `404` for a model that is not installed. It returns `422` in these cases:
 
 - A value is out of range.
 - `moe` is set, but the model file has no experts.

internal/console/models.go

@@ -194,6 +194,7 @@ func formFromPost(r *http.Request) modelform.Form {
 	v := func(name string) string { return strings.TrimSpace(r.PostFormValue(name)) }
 	return modelform.Form{
 		Layout: v("layout"), CtxSize: v("ctxsize"), Parallel: v("parallel"), TensorSplit: v("tensorsplit"),
+		Batch: v("batch"), UBatch: v("ubatch"), SplitMode: v("splitmode"),
 		Embedding: r.PostFormValue("embedding") == "on", Hidden: r.PostFormValue("hidden") == "on",
 		Temperature: v("temperature"), TopK: v("topk"), TopP: v("topp"), MinP: v("minp"),
 		RepeatPenalty: v("repeatpenalty"), PresencePenalty: v("presencepenalty"),

internal/console/templates/model.html

@@ -21,6 +21,12 @@ <h1 class="mono">{{.Model.ID}}</h1>
     <label>Tensor split <input name="tensorsplit" placeholder="by free memory" value="{{$f.TensorSplit}}"></label>
   </div>
   <p class="muted">Parallel slots share the context. The tensor split is the share of the layers on each GPU, such as 3,2.</p>
+  <div class="row">
+    <label>Batch size <input name="batch" inputmode="numeric" placeholder="llama.cpp default" value="{{$f.Batch}}"></label>
+    <label>Micro-batch size <input name="ubatch" inputmode="numeric" placeholder="llama.cpp default" value="{{$f.UBatch}}"></label>
+    <label>Split mode <select name="splitmode"><option value="">Layer (default)</option><option value="row"{{if eq $f.SplitMode "row"}} selected{{end}}>Row</option></select></label>
+  </div>
+  <p class="muted">A larger micro-batch makes the prompt faster and uses more GPU memory. The split mode applies only to a model that spans GPUs.</p>
   <div class="row">
     <label>KV cache K type <select name="kvtypek"><option value="">Default (f16)</option>{{range .CacheTypes}}<option{{if eq . $f.KVTypeK}} selected{{end}}>{{.}}</option>{{end}}</select></label>
     <label>KV cache V type <select name="kvtypev"><option value="">Default (f16)</option>{{range .CacheTypes}}<option{{if eq . $f.KVTypeV}} selected{{end}}>{{.}}</option>{{end}}</select></label>