Repository navigation
Optimizer phase 3: the TUI and the web console - #45
Merged
Merged
Conversation
…bench The design is in docs/design/optimizer.md: a measured search for faster llama.cpp settings, with an optional LLM advisor that only proposes. - Model settings batch, ubatch and split_mode, with their llama-server flags, the shared form, the TUI and the console. Old --batch-size and --ubatch-size raw args move to the typed fields on load. - The host agent's tune.<model> request runs llama-bench, llama-batched-bench or a llama-server load check in a backend image, as the benchmark does. Every argument must match a pattern for its flag, so a container cannot pass a model path, a network flag or a device. - The hip, vulkan and cuda images ship llama-batched-bench. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
internal/optimize runs the staged search from docs/design/optimizer.md:
baseline, backend and split mode, KV cache, batch sizes, MoE layers in
system RAM, and parallel requests for the throughput profile. A result
wins only when it beats the best by more than 3 percent and 3 standard
deviations. The verify stage starts llama-server at the model's real
context. When the best does not load, it drops the memory settings one
at a time, and tries the measured smaller micro-batches first.
The API runs it through the host agent's tune request:
POST, GET and DELETE /models/{id}/optimize, and
POST /models/{id}/optimize/apply, which reuses the checks of
PUT /models/{id}. One run goes at a time.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- GET /models includes each model's last optimizer run, and the client has StartOptimize, CancelOptimize and ApplyOptimize. - TUI Models screen: o starts the optimizer (one user or throughput) or cancels the run, a applies the result and asks whether to switch the backend. The details show the stage while it runs, then the speed now, the best speed and each change. The list refreshes while a run goes. - Web console: an Optimizer card on the model page with the same actions, outside the settings form. - The models and consoles guides describe it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 3 of the inference optimizer (
docs/design/optimizer.md). It is stacked on #44.GET /modelsincludes each model's last optimizer run. The client hasStartOptimize,CancelOptimizeandApplyOptimize.ostarts the optimizer (one user or throughput) or cancels the run.aapplies the result, and asks whether to switch the backend when a different one was faster. The details show the stage while a run goes, then the speed now, the best speed and each change. The list refreshes every 5 seconds while a run goes.docs/design/web-console.md.Verification
escclears the optimizer action, so the save key still works on the settings form.Originheader is refused.go vet,go test -race ./...andgolangci-lint(0 issues) pass.🤖 Generated with Claude Code