Repository navigation
Optimizer phase 4: the LLM advisor - #46
Merged
Merged
Conversation
…bench The design is in docs/design/optimizer.md: a measured search for faster llama.cpp settings, with an optional LLM advisor that only proposes. - Model settings batch, ubatch and split_mode, with their llama-server flags, the shared form, the TUI and the console. Old --batch-size and --ubatch-size raw args move to the typed fields on load. - The host agent's tune.<model> request runs llama-bench, llama-batched-bench or a llama-server load check in a backend image, as the benchmark does. Every argument must match a pattern for its flag, so a container cannot pass a model path, a network flag or a device. - The hip, vulkan and cuda images ship llama-batched-bench. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
internal/optimize runs the staged search from docs/design/optimizer.md:
baseline, backend and split mode, KV cache, batch sizes, MoE layers in
system RAM, and parallel requests for the throughput profile. A result
wins only when it beats the best by more than 3 percent and 3 standard
deviations. The verify stage starts llama-server at the model's real
context. When the best does not load, it drops the memory settings one
at a time, and tries the measured smaller micro-batches first.
The API runs it through the host agent's tune request:
POST, GET and DELETE /models/{id}/optimize, and
POST /models/{id}/optimize/apply, which reuses the checks of
PUT /models/{id}. One run goes at a time.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- GET /models includes each model's last optimizer run, and the client has StartOptimize, CancelOptimize and ApplyOptimize. - TUI Models screen: o starts the optimizer (one user or throughput) or cancels the run, a applies the result and asks whether to switch the backend. The details show the stage while it runs, then the speed now, the best speed and each change. The list refreshes while a run goes. - Web console: an Optimizer card on the model page with the same actions, outside the settings form. - The models and consoles guides describe it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Router: POST /sibyl/advise sends one question to one provider model. Only the System API key can call it, the provider keys stay in the router, and the cost counts under @optimizer, apart from the calibration budget. - internal/optimize: an advisor stage after the fixed stages. The advisor sees the facts and every measurement, and proposes up to 6 candidates as JSON. Each candidate must pass the same limits as the host agent's allowlist, and unknown fields drop it. The search measures what passes, as any other candidate. A failed advisor does not stop the run. - API: the advisor in POST /models/{id}/optimize must be a router provider. The facts hold the GPUs, the system RAM and the model metadata, with no prompts, no user data and no file paths. - TUI and console: an advisor choice from the router's provider models with a key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # docs/design/optimizer.md # docs/guides/models.md # docs/reference/system-api.md # internal/api/optimize.go # internal/api/optimize_test.go # internal/client/client.go # internal/console/console_test.go # internal/console/models.go # internal/console/templates/model.html # internal/optimize/optimize.go # internal/tui/models.go # internal/tui/optimize.go # internal/tui/optimize_test.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 4 of the inference optimizer (
docs/design/optimizer.md). It is stacked on #45.POST /sibyl/advisesends one question to one provider model. Only the System API key can call it. The provider keys stay in the router. The cost counts inusage.jsonunder@optimizer, apart from the calibration budget.internal/optimize: an advisor stage after the fixed stages. The advisor sees the facts and every measurement, and proposes up to 6 candidates as JSON. Each candidate must pass the same limits as the host agent's allowlist (backend, split, tensor split, batch sizes, KV types that fit, MoE layers, parallel only for throughput). A candidate with an unknown field is dropped. The search measures what passes, like any other candidate. A failed advisor does not stop the run.POST /models/{id}/optimizemust be a router provider. The facts hold the GPUs, the system RAM and the model metadata. They hold no prompts, no user data and no file paths.Verification
/sibyl/advisewith the System API key, and the prompt names the model file, not its path.go vet,go test -race ./...andgolangci-lint(0 issues) pass.🤖 Generated with Claude Code