diff --git a/docs/design/optimizer.md b/docs/design/optimizer.md
index 413a4bd..6b355e2 100644
--- a/docs/design/optimizer.md
+++ b/docs/design/optimizer.md
@@ -74,7 +74,7 @@ The search itself runs in the System API (`internal/optimize`). It writes one re
The advisor is off by default. When the owner turns it on, the optimizer asks one provider model that the router already has.
-- A new router route, `POST /sibyl/advise`, takes a provider and a model. Only the System API key can call it, as with `/sibyl/classify`. The provider keys stay in the router.
+- A new router route, `POST /sibyl/advise`, takes a provider and a model. Only the System API key can call it, as with `/sibyl/classify`. The provider keys stay in the router. The router counts the cost in `usage.json` under `@optimizer`, apart from the calibration budget.
- The optimizer sends the GPU names and VRAM, the system RAM, the backend, the model architecture, the parameter count, the quantization, the layer and expert counts, the context size, and every measured result.
- The advisor answers with JSON: up to six candidates, each one a set of the settings in the table above. The optimizer checks each candidate with the same allowlist as the agent, and drops what fails.
- The advisor is most useful for `-ot` patterns on a new MoE architecture, and for the order of the stages on hardware that the fixed search does not know.
diff --git a/docs/guides/models.md b/docs/guides/models.md
index 64a94ad..c89644c 100644
--- a/docs/guides/models.md
+++ b/docs/guides/models.md
@@ -192,6 +192,8 @@ The optimizer measures llama.cpp settings for one model on your hardware, and re
4. Read the result in the model's details: the speed now, the best speed, and each setting that changes.
5. Press `a` to apply the settings. In the web console, use "Apply these settings". If a different backend was faster, Sibyl asks whether to switch the backend too. The backend applies to every model.
+If the router has a provider with a key, you can choose an advisor when you start. The advisor is a provider model that proposes more settings after the fixed stages, and the optimizer measures them like the others. It gets the GPUs, the system RAM, the model facts and the measurements. It gets no prompts, no user data and no file paths. Each run makes one call, and the router counts its cost under `@optimizer`.
+
A setting wins only when it beats the settings now by more than 3 percent and by more than the noise of the runs. The optimizer applies nothing by itself. A result is for the hardware and the llama.cpp version of the run, so run it again after an update.
## Model pools
diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md
index 569b5c7..17beaf3 100644
--- a/docs/reference/configuration.md
+++ b/docs/reference/configuration.md
@@ -269,7 +269,7 @@ The `calibration` section sets up continuous calibration. The router keeps a sam
| `no_seed` | boolean | `false` | Stops the first measurement of a new local model with the labelled request set. |
| `auto_apply` | boolean | `false` | Lets the System API apply a rule proposal by itself, when the profile already names the proposal's route. |
-A provider route needs a price for calibration, because the calibration budget counts its cost. The router keeps the calibration spend in `usage.json` under the name `@calibration`.
+A provider route needs a price for calibration, because the calibration budget counts its cost. The router keeps the calibration spend in `usage.json` under the name `@calibration`, and the spend of the optimizer's advisor under `@optimizer`.
### Rule proposals
diff --git a/docs/reference/system-api.md b/docs/reference/system-api.md
index dfbcbc4..10314d2 100644
--- a/docs/reference/system-api.md
+++ b/docs/reference/system-api.md
@@ -140,7 +140,7 @@ A value that does not fit the variant or the detected GPUs returns `422`.
| `DELETE /models/{id}` | None | The apply result | Deletes the model file and all of its shards, removes its settings, and applies. |
| `POST /models/{id}/benchmark` | None | `202`, `{"status": "queued"}` | Asks the host agent to run `llama-bench` on the model with each backend image. |
| `GET /models/{id}/benchmark` | None | The benchmark state | The progress or the result of the last benchmark. `404` if no benchmark ran. |
-| `POST /models/{id}/optimize` | `{"profile": "single"}` or `"throughput"` | `202`, the optimizer state | Starts the optimizer for a chat model. It unloads the models while it runs. `409` while another run goes. |
+| `POST /models/{id}/optimize` | `{"profile": "single"}` or `"throughput"`, and an optional `"advisor": {"provider": "...", "model": "..."}` | `202`, the optimizer state | Starts the optimizer for a chat model. It unloads the models while it runs. The advisor must be a provider of the router. `409` while another run goes. |
| `GET /models/{id}/optimize` | None | The optimizer state | The stage, every measured result, and the recommendation when the run is done. `404` if no run took place. |
| `DELETE /models/{id}/optimize` | None | `202` | Cancels the run. The tool that runs on the host finishes first. |
| `POST /models/{id}/optimize/apply` | `{"backend": true}` or `false` | The apply result | Writes the recommended settings. With `backend` true, it also switches the backend of every model when the run found a faster one. `409` if the settings did not load at the model's context. |
@@ -430,4 +430,4 @@ Port 8443 also serves the inference APIs. They take a router client key, not the
| `/api/*` | System API (Ollama shim) | The Ollama-compatible API: `version`, `tags`, `show`, `ps`, `chat`, `generate`, `embed`, `embeddings` |
| `/console/*` | System API | The web console |
-The router also has two routes that only the System API key can call: `POST /sibyl/explain` and `POST /sibyl/classify`. Caddy does not expose them. Call `POST /router/explain` and `POST /router/classify` on the System API instead.
+The router also has three routes that only the System API key can call: `POST /sibyl/explain`, `POST /sibyl/classify` and `POST /sibyl/advise`. Caddy does not expose them. Call `POST /router/explain` and `POST /router/classify` on the System API instead. The optimizer's advisor uses `/sibyl/advise`.
diff --git a/internal/api/optimize.go b/internal/api/optimize.go
index 8a39c18..d6dd519 100644
--- a/internal/api/optimize.go
+++ b/internal/api/optimize.go
@@ -1,10 +1,12 @@
package api
import (
+ "bytes"
"context"
"encoding/json"
"errors"
"fmt"
+ "io"
"net/http"
"os"
"path/filepath"
@@ -18,17 +20,28 @@ import (
"github.com/diffsec/sibyl/internal/hostctl"
"github.com/diffsec/sibyl/internal/models"
"github.com/diffsec/sibyl/internal/optimize"
+ "github.com/diffsec/sibyl/internal/router"
)
// OptimizeRequest starts an optimizer run.
type OptimizeRequest struct {
Profile optimize.Profile `json:"profile"`
+ // Advisor is an optional router provider model that proposes more
+ // settings to measure.
+ Advisor *OptimizeAdvisor `json:"advisor,omitempty"`
+}
+
+// OptimizeAdvisor names a provider model of the router.
+type OptimizeAdvisor struct {
+ Provider string `json:"provider"`
+ Model string `json:"model"`
}
// OptimizeState is the state of a model's last optimizer run.
type OptimizeState struct {
Status string `json:"status"` // running, done, failed or cancelled
Profile optimize.Profile `json:"profile"`
+ Advisor string `json:"advisor,omitempty"` // provider/model
Stage string `json:"stage,omitempty"`
Started string `json:"started"`
Updated string `json:"updated"`
@@ -98,7 +111,7 @@ func (s *Server) startOptimize(w http.ResponseWriter, r *http.Request) {
writeError(w, http.StatusUnprocessableEntity, fmt.Errorf("profile %q: use %s or %s", req.Profile, optimize.Single, optimize.Throughput))
return
}
- in, runner, err := s.optimizeInput(id, req.Profile)
+ in, runner, err := s.optimizeInput(id, req.Profile, req.Advisor)
if err != nil {
code := http.StatusUnprocessableEntity
if errors.Is(err, os.ErrNotExist) {
@@ -120,6 +133,9 @@ func (s *Server) startOptimize(w http.ResponseWriter, r *http.Request) {
st := OptimizeState{Status: "running", Profile: req.Profile, Stage: "baseline",
Started: time.Now().UTC().Format(time.RFC3339), ImageTag: s.ImageTag}
+ if req.Advisor != nil {
+ st.Advisor = req.Advisor.Provider + "/" + req.Advisor.Model
+ }
if err := s.writeOptimize(id, st); err != nil {
s.endOptimize()
writeError(w, http.StatusInternalServerError, err)
@@ -164,7 +180,7 @@ func (s *Server) runOptimize(ctx context.Context, id string, st OptimizeState, i
}
// optimizeInput describes the model and the host for a run.
-func (s *Server) optimizeInput(id string, profile optimize.Profile) (optimize.Input, optimize.Runner, error) {
+func (s *Server) optimizeInput(id string, profile optimize.Profile, adv *OptimizeAdvisor) (optimize.Input, optimize.Runner, error) {
s.mu.Lock()
defer s.mu.Unlock()
m, err := s.findModel(id)
@@ -216,10 +232,96 @@ func (s *Server) optimizeInput(id string, profile optimize.Profile) (optimize.In
}
in.MoE = moeCandidates(set.MoE.CPULayers, in.Layers, meta.ExpertCount)
}
+ if adv != nil {
+ rc, err := s.loadRouter()
+ if err != nil {
+ return optimize.Input{}, nil, err
+ }
+ if _, ok := rc.Providers.Providers[adv.Provider]; !ok || adv.Model == "" || strings.ContainsAny(adv.Model, " \t\r\n") {
+ return optimize.Input{}, nil, fmt.Errorf("advisor: the router has no provider %q with a model %q", adv.Provider, adv.Model)
+ }
+ in.Advisor = optimize.LLMAdvisor{Ask: s.askAdvisor(adv.Provider, adv.Model), Facts: s.optimizeFacts(m, meta, used)}
+ }
runner := &agentRunner{s: s, id: id, rel: filepath.ToSlash(m.Path), pool: m.Pool, gpus: strings.Join(list, ","), poll: s.tunePoll}
return in, runner, nil
}
+// optimizeFacts describes the hardware and the model for the advisor. It
+// holds no prompts, no user data and no file paths.
+func (s *Server) optimizeFacts(m models.Model, meta *models.Meta, used []gpu.GPU) string {
+ var b strings.Builder
+ for i, g := range used {
+ parts := []string{g.Vendor}
+ if g.DeviceID != "" {
+ parts = append(parts, "device "+g.DeviceID)
+ }
+ if g.Driver != "" {
+ parts = append(parts, "driver "+g.Driver)
+ }
+ if g.VRAMTotal > 0 {
+ parts = append(parts, fmt.Sprintf("%.1f GiB VRAM", float64(g.VRAMTotal)/(1<<30)))
+ }
+ if g.Arch != "" {
+ parts = append(parts, "ROCm target "+g.Arch)
+ }
+ if g.LinkWidth != "" {
+ parts = append(parts, fmt.Sprintf("PCIe %s at %s", g.LinkWidth, g.LinkSpeed))
+ }
+ if g.VisibleVRAM > 0 && g.VisibleVRAM < g.VRAMTotal {
+ parts = append(parts, "Resizable BAR off")
+ }
+ fmt.Fprintf(&b, "GPU %d: %s\n", i, strings.Join(parts, ", "))
+ }
+ if _, ram, _ := s.memory(); ram > 0 {
+ fmt.Fprintf(&b, "System RAM: %.1f GiB\n", float64(ram)/(1<<30))
+ }
+ fmt.Fprintf(&b, "Model file: %s, %.1f GiB\n", filepath.Base(m.Path), float64(m.Size)/(1<<30))
+ if meta != nil {
+ fmt.Fprintf(&b, "Architecture: %s, native context %d tokens, %d attention layers", meta.Architecture, meta.ContextLength, meta.AttentionLayers)
+ if meta.ExpertCount > 0 {
+ fmt.Fprintf(&b, ", %d experts per MoE layer", meta.ExpertCount)
+ }
+ b.WriteString("\n")
+ }
+ return b.String()
+}
+
+// askAdvisor asks the router's /sibyl/advise route. The provider keys stay
+// in the router.
+func (s *Server) askAdvisor(provider, model string) func(ctx context.Context, system, prompt string) (string, error) {
+ return func(ctx context.Context, system, prompt string) (string, error) {
+ body, err := json.Marshal(router.AdviseRequest{Provider: provider, Model: model, System: system, Prompt: prompt})
+ if err != nil {
+ return "", err
+ }
+ ctx, cancel := context.WithTimeout(ctx, 4*time.Minute)
+ defer cancel()
+ req, err := http.NewRequestWithContext(ctx, http.MethodPost, s.RouterURL+"/sibyl/advise", bytes.NewReader(body))
+ if err != nil {
+ return "", err
+ }
+ req.Header.Set("Authorization", "Bearer "+s.Token)
+ req.Header.Set("Content-Type", "application/json")
+ resp, err := http.DefaultClient.Do(req)
+ if err != nil {
+ return "", fmt.Errorf("reach the router: %w", err)
+ }
+ defer func() { _ = resp.Body.Close() }()
+ raw, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
+ if err != nil {
+ return "", err
+ }
+ if resp.StatusCode != http.StatusOK {
+ return "", fmt.Errorf("the router answered %d: %.300s", resp.StatusCode, raw)
+ }
+ var ar router.AdviseResponse
+ if err := json.Unmarshal(raw, &ar); err != nil {
+ return "", err
+ }
+ return ar.Answer, nil
+ }
+}
+
// moeCandidates are the CPUMoE values to try: fewer layers with experts in
// system RAM than now. Fewer is faster, if it loads.
func moeCandidates(now, layers, experts int) []int {
diff --git a/internal/api/optimize_test.go b/internal/api/optimize_test.go
index 818a4cd..7e31189 100644
--- a/internal/api/optimize_test.go
+++ b/internal/api/optimize_test.go
@@ -5,14 +5,18 @@ import (
"encoding/json"
"fmt"
"net/http"
+ "net/http/httptest"
"os"
"path/filepath"
+ "slices"
"strings"
+ "sync"
"testing"
"time"
"github.com/diffsec/sibyl/internal/models"
"github.com/diffsec/sibyl/internal/optimize"
+ "github.com/diffsec/sibyl/internal/router"
)
// fakeAgent answers tune requests until the test ends: Vulkan generates
@@ -180,3 +184,46 @@ func TestOptimizeRefusals(t *testing.T) {
}
var optimizeRecommendationUnverified = optimize.Recommendation{}
+
+func TestOptimizeAdvisor(t *testing.T) {
+ f := newFixture(t)
+ f.server.tunePoll = 5 * time.Millisecond
+ f.do(t, "PUT", "/hardware", `{"backend":"hip","gpus":[1,0],"layout":"span"}`, true)
+ write(t, f.paths.routerFile(router.ProvidersName), "providers:\n cloud:\n type: openai\n base_url: https://x/v1\n")
+ var mu sync.Mutex
+ var got router.AdviseRequest
+ var auth string
+ rt := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ mu.Lock()
+ defer mu.Unlock()
+ auth = r.Header.Get("Authorization")
+ _ = json.NewDecoder(r.Body).Decode(&got)
+ _, _ = w.Write([]byte(`{"route":"cloud/big","answer":"{\"candidates\":[{\"ubatch\":1024,\"split_mode\":\"row\"}]}"}`))
+ }))
+ defer rt.Close()
+ f.server.RouterURL = rt.URL
+ fakeAgent(t, f, false)
+ if code, b := f.do(t, "POST", "/models/big/optimize", `{"profile":"single","advisor":{"provider":"nope","model":"big"}}`, true); code != 422 {
+ t.Errorf("unknown provider = %d %v", code, b)
+ }
+ if code, b := f.do(t, "POST", "/models/big/optimize", `{"profile":"single","advisor":{"provider":"cloud","model":"big"}}`, true); code != 202 {
+ t.Fatalf("start = %d %v", code, b)
+ }
+ b := waitOptimize(t, f, "big", "done")
+ if b["advisor"] != "cloud/big" {
+ t.Errorf("advisor %v", b["advisor"])
+ }
+ var stages []string
+ for _, r := range b["results"].([]any) {
+ stages = append(stages, r.(map[string]any)["stage"].(string))
+ }
+ if !slices.Contains(stages, "advisor") {
+ t.Errorf("no advisor result in %v", stages)
+ }
+ mu.Lock()
+ defer mu.Unlock()
+ if auth != "Bearer "+token || got.Provider != "cloud" || got.Model != "big" || !strings.Contains(got.Prompt, "GPU 0: amd\n") ||
+ !strings.Contains(got.Prompt, "big-00001-of-00002.gguf") || strings.Contains(got.Prompt, f.paths.Models) {
+ t.Errorf("advise request auth %q %+v", auth, got)
+ }
+}
diff --git a/internal/client/client.go b/internal/client/client.go
index 2d30890..24277e2 100644
--- a/internal/client/client.go
+++ b/internal/client/client.go
@@ -221,6 +221,7 @@ type Benchmark struct {
type Optimize struct {
Status string `json:"status"` // running, done, failed or cancelled
Profile optimize.Profile `json:"profile"`
+ Advisor string `json:"advisor,omitempty"`
Stage string `json:"stage,omitempty"`
Started string `json:"started"`
Updated string `json:"updated"`
@@ -293,9 +294,29 @@ func (c *Client) StartBenchmark(ctx context.Context, id string) error {
return c.do(ctx, http.MethodPost, "/models/"+url.PathEscape(id)+"/benchmark", nil, nil)
}
-// StartOptimize starts the optimizer for a model.
-func (c *Client) StartOptimize(ctx context.Context, id string, p optimize.Profile) error {
- return c.do(ctx, http.MethodPost, "/models/"+url.PathEscape(id)+"/optimize", map[string]any{"profile": p}, nil)
+// StartOptimize starts the optimizer for a model. advisor is a router
+// provider model as provider/model, or empty for none.
+func (c *Client) StartOptimize(ctx context.Context, id string, p optimize.Profile, advisor string) error {
+ body := map[string]any{"profile": p}
+ if prov, model, ok := strings.Cut(advisor, "/"); ok {
+ body["advisor"] = map[string]string{"provider": prov, "model": model}
+ }
+ return c.do(ctx, http.MethodPost, "/models/"+url.PathEscape(id)+"/optimize", body, nil)
+}
+
+// AdvisorChoices lists the router provider models that can advise the
+// optimizer, as provider/model. A provider needs its key.
+func AdvisorChoices(r Router) []string {
+ var out []string
+ for _, p := range r.Providers {
+ if !p.KeySet {
+ continue
+ }
+ for _, m := range p.Models {
+ out = append(out, p.Name+"/"+m)
+ }
+ }
+ return out
}
// CancelOptimize cancels a model's optimizer run.
diff --git a/internal/console/console_test.go b/internal/console/console_test.go
index 63a0ce4..0fa7fd5 100644
--- a/internal/console/console_test.go
+++ b/internal/console/console_test.go
@@ -774,7 +774,8 @@ func TestOptimizerCard(t *testing.T) {
}
body := w.Body.String()
for _, want := range []string{"
Optimizer
", ">48.2<", ">1310<", "ubatch: default to 1024",
- `action="/console/models/optimize/apply"`, "Switch the backend to vulkan too", `action="/console/models/optimize"`} {
+ `action="/console/models/optimize/apply"`, "Switch the backend to vulkan too", `action="/console/models/optimize"`,
+ ``} {
if !strings.Contains(body, want) {
t.Errorf("no %q in the model page", want)
}
@@ -791,6 +792,8 @@ func TestOptimizerCard(t *testing.T) {
want string
}{
{"/console/models/optimize", url.Values{"id": {"qwen"}, "profile": {"throughput"}}, "POST /models/qwen/optimize", `{"profile":"throughput"}`},
+ {"/console/models/optimize", url.Values{"id": {"qwen"}, "profile": {"single"}, "advisor": {"cloud/big"}}, "POST /models/qwen/optimize",
+ `{"advisor":{"model":"big","provider":"cloud"},"profile":"single"}`},
{"/console/models/optimize/apply", url.Values{"id": {"qwen"}, "backend": {"on"}}, "POST /models/qwen/optimize/apply", `{"backend":true}`},
{"/console/models/optimize/apply", url.Values{"id": {"qwen"}}, "POST /models/qwen/optimize/apply", `{"backend":false}`},
}
diff --git a/internal/console/models.go b/internal/console/models.go
index a6d078a..f831077 100644
--- a/internal/console/models.go
+++ b/internal/console/models.go
@@ -181,11 +181,15 @@ func (c *Console) modelPage(w http.ResponseWriter, r *http.Request) {
moe = m.Meta.ExpertCount > 0
}
spec := slices.DeleteFunc(slices.Clone(models.SpecTypes), func(s string) bool { return s == "none" })
+ var advisors []string
+ if rc, err := c.client.Router(r.Context()); err == nil {
+ advisors = client.AdvisorChoices(rc)
+ }
data := pageData(r)
for k, v := range map[string]any{
"Model": m, "Head": head, "Fit": fitText(m.Fit), "Form": modelform.FromSettings(m.Settings),
"Layouts": layouts, "Drafts": drafts, "SimpleDrafts": simple, "CacheTypes": models.CacheTypes, "Presets": models.Presets,
- "SpecTypes": spec, "MoE": moe, "Reasoning": reasoningControl(m),
+ "SpecTypes": spec, "MoE": moe, "Reasoning": reasoningControl(m), "Advisors": advisors,
} {
data[k] = v
}
@@ -251,7 +255,7 @@ func (c *Console) benchModel(w http.ResponseWriter, r *http.Request) {
func (c *Console) optimizeModel(w http.ResponseWriter, r *http.Request) {
id := r.PostFormValue("id")
- err := c.client.StartOptimize(r.Context(), id, optimize.Profile(r.PostFormValue("profile")))
+ err := c.client.StartOptimize(r.Context(), id, optimize.Profile(r.PostFormValue("profile")), r.PostFormValue("advisor"))
c.back(w, r, "models/edit?id="+url.QueryEscape(id), "the optimizer for "+id+" started", err)
}
diff --git a/internal/console/templates/model.html b/internal/console/templates/model.html
index 4802167..bb5b302 100644
--- a/internal/console/templates/model.html
+++ b/internal/console/templates/model.html
@@ -106,7 +106,7 @@
Optimizer
Running ({{.Profile}}), stage {{.Stage}}. The models are unloaded until it ends. Reload the page to see the progress.
{{else if .Recommendation}}{{$r := .Recommendation}}
-
Last run ({{.Profile}}), {{.Updated}}, llama.cpp image {{.ImageTag}}.
+
Last run ({{.Profile}}{{with .Advisor}}, advisor {{.}}{{end}}), {{.Updated}}, llama.cpp image {{.ImageTag}}.
Backend
Generation tok/s
Prompt tok/s
{{if $r.Best.TotalTPS}}
Total tok/s
Parallel
{{end}}
@@ -135,9 +135,10 @@
Optimizer
+ {{if .Advisors}}{{end}}
-
The optimizer measures settings with llama-bench on this hardware, and checks the best with llama-server at the model's context. The models are unloaded for 5 to 15 minutes. Nothing changes until you apply the result.
+
The optimizer measures settings with llama-bench on this hardware, and checks the best with llama-server at the model's context. The models are unloaded for 5 to 15 minutes. Nothing changes until you apply the result. An advisor gets the hardware, the model facts and the measurements, with no prompts and no user data, and proposes more settings to measure.
{{end}}
{{end}}
diff --git a/internal/optimize/advisor.go b/internal/optimize/advisor.go
new file mode 100644
index 0000000..93c9237
--- /dev/null
+++ b/internal/optimize/advisor.go
@@ -0,0 +1,164 @@
+package optimize
+
+import (
+ "context"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "regexp"
+ "slices"
+ "strings"
+)
+
+// Advisor proposes candidates from the measurements so far.
+type Advisor interface {
+ Propose(ctx context.Context, in Input, best Result, results []Result) ([]Candidate, error)
+}
+
+// maxProposals is the most advisor candidates that the search measures.
+const maxProposals = 6
+
+// Limits of a candidate. The host agent's tune allowlist holds the same.
+const (
+ minBatch = 32
+ maxBatch = 65536
+ maxParallel = 64
+)
+
+var (
+ kvTypes = []string{"", "f16", "q8_0", "q4_0"}
+ tensorSplit = regexp.MustCompile(`^[0-9]{1,3}(\.[0-9]{1,3})?(/[0-9]{1,3}(\.[0-9]{1,3})?){0,15}$`)
+)
+
+// valid checks a candidate against the host and the limits. The search
+// measures only valid candidates, whatever an advisor proposes.
+func (in Input) valid(c Candidate) bool {
+ backends := in.Backends
+ if len(backends) == 0 {
+ backends = []string{in.Backend}
+ }
+ if !slices.Contains(backends, c.Backend) {
+ return false
+ }
+ switch c.SplitMode {
+ case "":
+ case "layer", "row":
+ if in.GPUs < 2 {
+ return false
+ }
+ default:
+ return false
+ }
+ if c.TensorSplit != "" && (in.GPUs < 2 || !tensorSplit.MatchString(c.TensorSplit) ||
+ strings.Count(c.TensorSplit, "/")+1 != in.GPUs) {
+ return false
+ }
+ for _, b := range []int{c.Batch, c.UBatch} {
+ if b != 0 && (b < minBatch || b > maxBatch) {
+ return false
+ }
+ }
+ if c.Batch > 0 && c.UBatch > c.Batch {
+ return false
+ }
+ if !slices.Contains(kvTypes, c.TypeK) || !slices.Contains(kvTypes, c.TypeV) {
+ return false
+ }
+ if in.KVFits != nil && !in.KVFits(strings.TrimPrefix(c.TypeK, "f16"), strings.TrimPrefix(c.TypeV, "f16")) {
+ return false
+ }
+ if c.CPUMoE < 0 || c.CPUMoE > in.Layers {
+ return false
+ }
+ if c.Parallel < 0 || c.Parallel > maxParallel || (in.Profile != Throughput && c.Parallel > 0) {
+ return false
+ }
+ return true
+}
+
+// LLMAdvisor asks an upstream model for candidates. Ask sends one question
+// and returns the answer. Facts describe the hardware and the model, with
+// no prompts and no user data.
+type LLMAdvisor struct {
+ Ask func(ctx context.Context, system, prompt string) (string, error)
+ Facts string
+}
+
+// advisorSystem tells the model what to answer.
+const advisorSystem = `You tune llama.cpp inference for one model on one machine. You see the hardware, the model, and every setting that was measured with llama-bench, with its speed. Propose up to 6 new settings that are likely to be faster. Do not repeat a measured setting.
+
+You may set only these fields. Leave a field out to keep the best setting's value.
+- "backend": one of the backends listed under the hardware
+- "split_mode": "layer" or "row", only with more than one GPU
+- "tensor_split": the share of layers on each GPU, such as "3/2", only with more than one GPU
+- "batch" and "ubatch": 32 to 65536, ubatch not larger than batch
+- "type_k" and "type_v": "f16", "q8_0" or "q4_0"
+- "cpu_moe": the number of layers whose MoE experts stay in system RAM, 0 to the layer count, only for a MoE model
+- "parallel": 1 to 64 parallel requests, only for the throughput profile
+
+Answer with JSON only, and nothing after it: {"candidates": [{"ubatch": 1024, "split_mode": "row"}, ...], "reason": "one sentence"}`
+
+// Propose asks the model, and turns its answer into candidates from the
+// best setting.
+func (a LLMAdvisor) Propose(ctx context.Context, in Input, best Result, results []Result) ([]Candidate, error) {
+ var b strings.Builder
+ b.WriteString(a.Facts)
+ fmt.Fprintf(&b, "\nProfile: %s. Backends: %s. GPUs for the model: %d. MoE layers: %d. Context: %d tokens.\n",
+ in.Profile, strings.Join(in.Backends, ", "), in.GPUs, in.Layers, in.Ctx)
+ fmt.Fprintf(&b, "Each test: %d prompt tokens and %d generated tokens, after %d tokens in the KV cache.\n\n",
+ promptTokens, genTokens, in.depth())
+ b.WriteString("Measured settings (gen and prompt in tokens per second):\n")
+ for _, r := range results {
+ c, _ := json.Marshal(r.Candidate)
+ if r.Error != "" {
+ fmt.Fprintf(&b, "- %s %s: failed: %.200s\n", r.Stage, c, r.Error)
+ continue
+ }
+ fmt.Fprintf(&b, "- %s %s: gen %.1f, prompt %.0f", r.Stage, c, r.GenTPS, r.PromptTPS)
+ if r.TotalTPS > 0 {
+ fmt.Fprintf(&b, ", total %.1f over %d requests", r.TotalTPS, r.Parallel)
+ }
+ b.WriteString("\n")
+ }
+ bc, _ := json.Marshal(best.Candidate)
+ fmt.Fprintf(&b, "\nBest so far: %s\n", bc)
+ answer, err := a.Ask(ctx, advisorSystem, b.String())
+ if err != nil {
+ return nil, err
+ }
+ return parseProposals(answer, best.Candidate)
+}
+
+// parseProposals reads the advisor's JSON. Each candidate starts from the
+// best setting, and changes the fields that it names.
+func parseProposals(answer string, best Candidate) ([]Candidate, error) {
+ i, j := strings.Index(answer, "{"), strings.LastIndex(answer, "}")
+ if i < 0 || j < i {
+ return nil, errors.New("the advisor's answer has no JSON")
+ }
+ var raw struct {
+ Candidates []json.RawMessage `json:"candidates"`
+ }
+ if err := json.Unmarshal([]byte(answer[i:j+1]), &raw); err != nil {
+ return nil, fmt.Errorf("the advisor's answer: %w", err)
+ }
+ var out []Candidate
+ for _, r := range raw.Candidates {
+ c := best
+ // Unknown fields fail, so a proposal cannot carry a flag that the
+ // search does not know.
+ dec := json.NewDecoder(strings.NewReader(string(r)))
+ dec.DisallowUnknownFields()
+ if err := dec.Decode(&c); err != nil {
+ continue
+ }
+ if c.TypeK == "f16" {
+ c.TypeK = ""
+ }
+ if c.TypeV == "f16" {
+ c.TypeV = ""
+ }
+ out = append(out, c)
+ }
+ return out, nil
+}
diff --git a/internal/optimize/advisor_test.go b/internal/optimize/advisor_test.go
new file mode 100644
index 0000000..8089575
--- /dev/null
+++ b/internal/optimize/advisor_test.go
@@ -0,0 +1,120 @@
+package optimize
+
+import (
+ "context"
+ "errors"
+ "strings"
+ "testing"
+)
+
+type fixedAdvisor struct {
+ props []Candidate
+ err error
+ seen []Result
+}
+
+func (a *fixedAdvisor) Propose(_ context.Context, _ Input, _ Result, rs []Result) ([]Candidate, error) {
+ a.seen = rs
+ return a.props, a.err
+}
+
+func TestSearchAdvisor(t *testing.T) {
+ h := &fakeHost{}
+ adv := &fixedAdvisor{props: []Candidate{
+ {Backend: "hip", SplitMode: "row", UBatch: 1024}, // valid: row split is 10 faster
+ {Backend: "cuda"}, // not on this host
+ {Backend: "hip", Batch: 512, UBatch: 1024}, // ubatch over batch
+ {Backend: "hip", Parallel: 4}, // parallel in the single profile
+ {Backend: "hip", TensorSplit: "1/1/1"}, // three values for two GPUs
+ }}
+ rec, err := (&Search{Runner: h}).Run(context.Background(), Input{
+ Profile: Single, Backend: "hip", Backends: []string{"hip"}, GPUs: 2, Ctx: 8192, Advisor: adv,
+ })
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(adv.seen) == 0 {
+ t.Error("the advisor saw no results")
+ }
+ var measured int
+ for _, c := range h.calls {
+ if c.Tool == ToolBench && strings.Contains(strings.Join(c.Args, " "), "-ub 1024") && values(c.Args)["-sm"][0] == "row" {
+ measured++
+ }
+ }
+ if measured == 0 {
+ t.Error("the valid proposal was not measured")
+ }
+ for _, c := range h.calls {
+ if c.Backend == "cuda" || strings.Contains(strings.Join(c.Args, " "), "-b 512") {
+ t.Errorf("an invalid proposal was measured: %+v", c)
+ }
+ }
+ if rec.Best.SplitMode != "row" {
+ t.Errorf("best %+v", rec.Best.Candidate)
+ }
+ // A failed advisor does not stop the run.
+ if _, err := (&Search{Runner: &fakeHost{}}).Run(context.Background(), Input{
+ Profile: Single, Backend: "hip", Backends: []string{"hip"}, GPUs: 1, Ctx: 8192,
+ Advisor: &fixedAdvisor{err: errors.New("provider down")},
+ }); err != nil {
+ t.Errorf("err %v", err)
+ }
+}
+
+func TestValidCandidate(t *testing.T) {
+ in := Input{Profile: Throughput, Backend: "hip", Backends: []string{"hip", "vulkan"}, GPUs: 2, Layers: 40,
+ KVFits: func(k, v string) bool { return k != "q4_0" }}
+ for _, c := range []struct {
+ c Candidate
+ want bool
+ }{
+ {Candidate{Backend: "vulkan", SplitMode: "row", TensorSplit: "3/2", Batch: 4096, UBatch: 1024, TypeK: "q8_0", CPUMoE: 10, Parallel: 8}, true},
+ {Candidate{Backend: "rocm"}, false},
+ {Candidate{Backend: "hip", SplitMode: "tensor"}, false},
+ {Candidate{Backend: "hip", TensorSplit: "3/2/1"}, false},
+ {Candidate{Backend: "hip", TensorSplit: "3,2"}, false},
+ {Candidate{Backend: "hip", Batch: 16}, false},
+ {Candidate{Backend: "hip", UBatch: 100000}, false},
+ {Candidate{Backend: "hip", TypeK: "q4_0"}, false}, // does not fit
+ {Candidate{Backend: "hip", TypeV: "f32"}, false},
+ {Candidate{Backend: "hip", CPUMoE: 41}, false},
+ {Candidate{Backend: "hip", Parallel: 65}, false},
+ } {
+ if got := in.valid(c.c); got != c.want {
+ t.Errorf("%+v: %v", c.c, got)
+ }
+ }
+ one := Input{Profile: Single, Backend: "hip", GPUs: 1}
+ if one.valid(Candidate{Backend: "hip", SplitMode: "row"}) || one.valid(Candidate{Backend: "hip", Parallel: 2}) {
+ t.Error("a split on one GPU or parallel in the single profile passed")
+ }
+}
+
+func TestLLMAdvisor(t *testing.T) {
+ var gotSystem, gotPrompt string
+ a := LLMAdvisor{Facts: "GPUs: 2x Radeon RX 7900 XTX, 24 GiB each.", Ask: func(_ context.Context, system, prompt string) (string, error) {
+ gotSystem, gotPrompt = system, prompt
+ return "Here you go:\n```json\n" + `{"candidates":[{"ubatch":1024,"split_mode":"row"},{"type_k":"f16","cpu_moe":4},{"ubatch":512,"rpc":"10.0.0.1"}],"reason":"r"}` + "\n```", nil
+ }}
+ best := Result{Candidate: Candidate{Backend: "hip", Batch: 2048}, GenTPS: 50}
+ got, err := a.Propose(context.Background(), Input{Profile: Single, Backends: []string{"hip"}, GPUs: 2, Ctx: 8192},
+ best, []Result{{Candidate: best.Candidate, Stage: "baseline", GenTPS: 50, PromptTPS: 1000}, {Stage: "batch", Error: "oom"}})
+ if err != nil {
+ t.Fatal(err)
+ }
+ if len(got) != 2 || got[0].UBatch != 1024 || got[0].SplitMode != "row" || got[0].Batch != 2048 || got[1].TypeK != "" || got[1].CPUMoE != 4 {
+ t.Errorf("candidates %+v", got)
+ }
+ for _, want := range []string{"7900 XTX", "gen 50.0, prompt 1000", "failed: oom", "Best so far"} {
+ if !strings.Contains(gotPrompt, want) {
+ t.Errorf("no %q in the prompt:\n%s", want, gotPrompt)
+ }
+ }
+ if !strings.Contains(gotSystem, "JSON only") {
+ t.Error("the system prompt asks for no JSON")
+ }
+ if _, err := parseProposals("no json here", best.Candidate); err == nil {
+ t.Error("an answer with no JSON passed")
+ }
+}
diff --git a/internal/optimize/optimize.go b/internal/optimize/optimize.go
index 9ce2833..c717347 100644
--- a/internal/optimize/optimize.go
+++ b/internal/optimize/optimize.go
@@ -103,8 +103,9 @@ type Input struct {
// Layers is the model's block count. A model with every expert in
// system RAM (models.AllLayers) runs with CPUMoE at Layers.
Layers int
- // Proposals are extra candidates, from the advisor.
- Proposals []Candidate
+ // Advisor proposes more candidates after the fixed stages. Nil skips
+ // the stage.
+ Advisor Advisor
}
// Change is one setting that the recommendation changes.
@@ -247,9 +248,25 @@ func (s *Search) Run(ctx context.Context, in Input) (*Recommendation, error) {
if err := pick("moe", cands); err != nil {
return nil, err
}
- // The advisor's proposals.
- if err := pick("advisor", in.Proposals); err != nil {
- return nil, err
+ // The advisor's proposals. A failed advisor does not stop the run.
+ if in.Advisor != nil {
+ s.report("advisor")
+ props, err := in.Advisor.Propose(ctx, in, best, slices.Clone(s.results))
+ if ctx.Err() != nil {
+ return nil, ctx.Err()
+ }
+ if err != nil {
+ s.results = append(s.results, Result{Candidate: best.Candidate, Stage: "advisor", Error: "advisor: " + err.Error()})
+ }
+ cands = nil
+ for _, p := range props {
+ if p != best.Candidate && in.valid(p) && !slices.Contains(cands, p) && len(cands) < maxProposals {
+ cands = append(cands, p)
+ }
+ }
+ if err := pick("advisor", cands); err != nil {
+ return nil, err
+ }
}
if in.Profile == Throughput {
if best, err = s.parallel(ctx, best); err != nil {
@@ -500,15 +517,19 @@ func (s *Search) load(ctx context.Context, c Candidate) (bool, error) {
return false, nil
}
-// groupByBackend splits candidates by backend, in order.
+// groupByBackend splits candidates by backend and tensor split, in order.
+// One llama-bench run holds one of each: the backend is the image, and
+// llama-bench prints the tensor split rounded, so two splits in one run
+// cannot be told apart.
func groupByBackend(cands []Candidate) [][]Candidate {
var out [][]Candidate
- idx := map[string]int{}
+ idx := map[[2]string]int{}
for _, c := range cands {
- i, ok := idx[c.Backend]
+ k := [2]string{c.Backend, c.TensorSplit}
+ i, ok := idx[k]
if !ok {
i = len(out)
- idx[c.Backend] = i
+ idx[k] = i
out = append(out, nil)
}
out[i] = append(out[i], c)
diff --git a/internal/router/advise.go b/internal/router/advise.go
new file mode 100644
index 0000000..f24df0a
--- /dev/null
+++ b/internal/router/advise.go
@@ -0,0 +1,86 @@
+package router
+
+import (
+ "context"
+ "encoding/json"
+ "fmt"
+ "net/http"
+ "strings"
+ "time"
+)
+
+// OptimizerSpender is the name under which usage.json keeps the provider
+// spend of the optimizer's advisor. It is not a valid client name.
+const OptimizerSpender = "@optimizer"
+
+// AdviseRequest is the body of POST /sibyl/advise: one question to one
+// provider model, for the inference optimizer.
+type AdviseRequest struct {
+ Provider string `json:"provider"`
+ Model string `json:"model"`
+ System string `json:"system"`
+ Prompt string `json:"prompt"`
+}
+
+// AdviseResponse is the answer of /sibyl/advise.
+type AdviseResponse struct {
+ Route string `json:"route"`
+ Answer string `json:"answer"`
+ CostUSD float64 `json:"cost_usd,omitempty"`
+}
+
+// Limits of an advise request.
+const (
+ maxAdvisePrompt = 64 << 10
+ adviseTimeout = 3 * time.Minute
+ adviseMaxTokens = 4096
+)
+
+// advise asks a provider model a question for the optimizer. Only the
+// System API key can call it. The provider keys stay in the router.
+func (s *Server) advise(w http.ResponseWriter, r *http.Request) {
+ if !equal(clientKey(r), s.AdminKey) {
+ writeOpenAIError(w, http.StatusUnauthorized, "missing or invalid API key")
+ return
+ }
+ body, err := readBody(w, r)
+ if err != nil {
+ writeOpenAIError(w, errCode(err), err.Error())
+ return
+ }
+ var ar AdviseRequest
+ if err := json.Unmarshal(body, &ar); err != nil {
+ writeOpenAIError(w, http.StatusBadRequest, "the body is not valid JSON")
+ return
+ }
+ cfg := s.config()
+ if _, ok := cfg.Providers.Providers[ar.Provider]; !ok || ar.Model == "" || strings.ContainsAny(ar.Model, " \t\r\n") {
+ writeOpenAIError(w, http.StatusBadRequest, fmt.Sprintf("no provider %q with a model %q", ar.Provider, ar.Model))
+ return
+ }
+ if strings.TrimSpace(ar.Prompt) == "" || len(ar.Prompt)+len(ar.System) > maxAdvisePrompt {
+ writeOpenAIError(w, http.StatusBadRequest, fmt.Sprintf("the prompt must be 1 to %d bytes", maxAdvisePrompt))
+ return
+ }
+ rt := Route{Provider: ar.Provider, Model: ar.Model}
+ chat, err := json.Marshal(map[string]any{
+ "model": ar.Model,
+ "messages": []map[string]string{
+ {"role": "system", "content": ar.System},
+ {"role": "user", "content": ar.Prompt},
+ },
+ "max_tokens": adviseMaxTokens,
+ })
+ if err != nil {
+ writeOpenAIError(w, http.StatusInternalServerError, err.Error())
+ return
+ }
+ ctx, cancel := context.WithTimeout(r.Context(), adviseTimeout)
+ defer cancel()
+ side, err := s.completeAs(ctx, cfg, rt, chat, OptimizerSpender)
+ if err != nil {
+ writeOpenAIError(w, http.StatusBadGateway, err.Error())
+ return
+ }
+ writeJSON(w, http.StatusOK, AdviseResponse{Route: rt.String(), Answer: side.Answer, CostUSD: side.CostUSD})
+}
diff --git a/internal/router/advise_test.go b/internal/router/advise_test.go
new file mode 100644
index 0000000..e0c2e0c
--- /dev/null
+++ b/internal/router/advise_test.go
@@ -0,0 +1,38 @@
+package router
+
+import (
+ "encoding/json"
+ "testing"
+ "time"
+)
+
+func TestAdvise(t *testing.T) {
+ h := newHarness(t, testProfiles)
+ req := AdviseRequest{Provider: "cloud", Model: "big", System: "You tune llama.cpp.", Prompt: "Hardware: 2 GPUs."}
+ if resp := h.post(t, "/sibyl/advise", hybridKey, req); resp.StatusCode != 401 {
+ t.Errorf("a client key called advise: %d", resp.StatusCode)
+ }
+ for _, bad := range []AdviseRequest{{Provider: "nope", Model: "big", Prompt: "x"}, {Provider: "cloud", Model: "big"},
+ {Provider: "cloud", Model: "a b", Prompt: "x"}} {
+ if resp := h.post(t, "/sibyl/advise", adminKey, bad); resp.StatusCode != 400 {
+ t.Errorf("%+v = %d", bad, resp.StatusCode)
+ }
+ }
+ resp := h.post(t, "/sibyl/advise", adminKey, req)
+ var ar AdviseResponse
+ if err := json.NewDecoder(resp.Body).Decode(&ar); err != nil || resp.StatusCode != 200 {
+ t.Fatalf("advise %d %v", resp.StatusCode, err)
+ }
+ if ar.Route != "cloud/big" || ar.Answer == "" {
+ t.Errorf("answer %+v", ar)
+ }
+ got := h.provider.last()
+ msgs, _ := got["messages"].([]any)
+ if got["model"] != "big" || got["stream"] != false || len(msgs) != 2 {
+ t.Errorf("provider got %v", got)
+ }
+ // The spend counts for the optimizer, not for calibration.
+ if day, _ := h.router.Spend.Spent(CalibrationSpender, time.Now()); day != 0 {
+ t.Errorf("calibration spend %v", day)
+ }
+}
diff --git a/internal/router/calibrate.go b/internal/router/calibrate.go
index 7494609..62a8066 100644
--- a/internal/router/calibrate.go
+++ b/internal/router/calibrate.go
@@ -606,7 +606,13 @@ func (s *Server) compare(ctx context.Context, cfg *Config, smp *Sample, ready fu
}
// complete sends one chat request without streaming and reads the answer.
+// The calibration budget counts its cost.
func (s *Server) complete(ctx context.Context, cfg *Config, rt Route, body []byte) (*Side, error) {
+ return s.completeAs(ctx, cfg, rt, body, calibrationSpender)
+}
+
+// completeAs is complete, with the cost recorded for spender.
+func (s *Server) completeAs(ctx context.Context, cfg *Config, rt Route, body []byte, spender string) (*Side, error) {
var req map[string]any
if err := json.Unmarshal(body, &req); err != nil {
return nil, err
@@ -638,8 +644,8 @@ func (s *Server) complete(ctx context.Context, cfg *Config, rt Route, body []byt
if !rt.Local() {
if p, ok := cfg.price(rt); ok {
side.CostUSD = p.Cost(side.Usage)
- if err := s.Spend.Add(calibrationSpender, side.CostUSD, start); err != nil {
- s.logger().Error("record the calibration spend", "err", err)
+ if err := s.Spend.Add(spender, side.CostUSD, start); err != nil {
+ s.logger().Error("record the spend", "spender", spender, "err", err)
}
}
}
diff --git a/internal/router/server.go b/internal/router/server.go
index 7870f81..4a43bbc 100644
--- a/internal/router/server.go
+++ b/internal/router/server.go
@@ -39,7 +39,7 @@ type Server struct {
// LocalURL is llama-swap, such as http://llama-swap:8080.
LocalURL string
// AdminKey is the System API key. It is not an inference key: it can
- // only call /sibyl/explain.
+ // only call the /sibyl/ routes.
AdminKey string
// DefaultKey is the key of the client "default".
DefaultKey string
@@ -177,6 +177,7 @@ func (s *Server) Handler() http.Handler {
mux.HandleFunc("/v1/", s.authed(s.passthrough))
mux.HandleFunc("POST /sibyl/explain", s.explain)
mux.HandleFunc("POST /sibyl/classify", s.classifyText)
+ mux.HandleFunc("POST /sibyl/advise", s.advise)
return mux
}
diff --git a/internal/tui/models.go b/internal/tui/models.go
index fa4f85b..4b142ec 100644
--- a/internal/tui/models.go
+++ b/internal/tui/models.go
@@ -20,10 +20,11 @@ import (
)
type modelsLoaded struct {
- models []client.Model
- hw hardware.Config
- gpus []gpu.GPU
- err error
+ models []client.Model
+ hw hardware.Config
+ gpus []gpu.GPU
+ advisors []string
+ err error
}
type modelSaved struct {
@@ -55,7 +56,11 @@ type modelsView struct {
action string
optProfile string
optBackend bool
- fields settingsForm
+ optAdvisor string
+ // advisors are the router provider models that can advise the
+ // optimizer.
+ advisors []string
+ fields settingsForm
// orig is fields when the form opened, and secs the sections of the
// form, for the section bar.
orig settingsForm
@@ -96,7 +101,12 @@ func (v *modelsView) fetchModels() tea.Cmd {
}
// Without the GPU list the estimate leaves out the VRAM total.
gpus, _ := v.c.GPUs(ctx)
- return modelsLoaded{models: m, hw: hw, gpus: gpus}
+ // Without the router, the optimizer offers no advisor.
+ var advisors []string
+ if r, err := v.c.Router(ctx); err == nil {
+ advisors = client.AdvisorChoices(r)
+ }
+ return modelsLoaded{models: m, hw: hw, gpus: gpus, advisors: advisors}
}
}
@@ -345,7 +355,7 @@ func (v *modelsView) update(msg tea.Msg) tea.Cmd {
case modelsLoaded:
v.err = msg.err
if msg.err == nil {
- v.list, v.hw, v.gpus = client.OrderWithDrafters(msg.models), msg.hw, msg.gpus
+ v.list, v.hw, v.gpus, v.advisors = client.OrderWithDrafters(msg.models), msg.hw, msg.gpus, msg.advisors
}
if v.cursor >= len(v.list) {
v.cursor = max(0, len(v.list)-1)
diff --git a/internal/tui/optimize.go b/internal/tui/optimize.go
index 76ae443..e1217c1 100644
--- a/internal/tui/optimize.go
+++ b/internal/tui/optimize.go
@@ -19,7 +19,11 @@ func optimizeText(o *client.Optimize) string {
return ""
}
var b strings.Builder
- b.WriteString("Optimizer (" + string(o.Profile) + ")\n")
+ b.WriteString("Optimizer (" + string(o.Profile))
+ if o.Advisor != "" {
+ b.WriteString(", advisor " + o.Advisor)
+ }
+ b.WriteString(")\n")
switch o.Status {
case "running":
fmt.Fprintf(&b, " %s · stage %s · %d settings measured\n", goodSty.Render("running"), o.Stage, measured(o.Results))
@@ -83,17 +87,27 @@ func (v *modelsView) optimizeForm(m client.Model) tea.Cmd {
)).WithShowHelp(false)
return v.form.Init()
}
- v.action, v.optProfile = "optimize", string(optimize.Single)
- v.form = huh.NewForm(huh.NewGroup(
+ v.action, v.optProfile, v.optAdvisor = "optimize", string(optimize.Single), ""
+ fields := []huh.Field{
huh.NewSelect[string]().Title("Optimize "+m.ID+" for").Options(
huh.NewOption("One user: generation speed, then prompt speed", string(optimize.Single)),
huh.NewOption("Throughput: total speed over parallel requests", string(optimize.Throughput)),
).Value(&v.optProfile),
- huh.NewConfirm().Title("Start the optimizer?").
- Description("It measures settings with llama-bench and checks the best one with llama-server.\n"+
- "The models are unloaded for 5 to 15 minutes. Nothing changes until you apply the result.").
- Affirmative("Start").Negative("Back").Value(&v.confirm),
- )).WithShowHelp(false)
+ }
+ if len(v.advisors) > 0 {
+ opts := []huh.Option[string]{huh.NewOption("None: the fixed search only", "")}
+ for _, a := range v.advisors {
+ opts = append(opts, huh.NewOption(a, a))
+ }
+ fields = append(fields, huh.NewSelect[string]().Title("Advisor").Options(opts...).Value(&v.optAdvisor).
+ Description("A router provider model proposes more settings to measure. It gets the hardware,\n"+
+ "the model facts and the measurements: no prompts and no user data. One call per run."))
+ }
+ fields = append(fields, huh.NewConfirm().Title("Start the optimizer?").
+ Description("It measures settings with llama-bench and checks the best one with llama-server.\n"+
+ "The models are unloaded for 5 to 15 minutes. Nothing changes until you apply the result.").
+ Affirmative("Start").Negative("Back").Value(&v.confirm))
+ v.form = huh.NewForm(huh.NewGroup(fields...)).WithShowHelp(false)
return v.form.Init()
}
@@ -143,11 +157,11 @@ func (v *modelsView) submitOptimize() tea.Cmd {
return noteMsg{"the optimizer for " + id + " stops after its current tool", true}
}
case "optimize":
- profile := optimize.Profile(v.optProfile)
+ profile, advisor := optimize.Profile(v.optProfile), v.optAdvisor
return tea.Batch(func() tea.Msg {
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()
- if err := v.c.StartOptimize(ctx, id, profile); err != nil {
+ if err := v.c.StartOptimize(ctx, id, profile, advisor); err != nil {
return noteMsg{"optimize " + id + ": " + err.Error(), false}
}
return noteMsg{"the optimizer for " + id + " started; it takes 5 to 15 minutes", true}
diff --git a/internal/tui/optimize_test.go b/internal/tui/optimize_test.go
index a58b897..f88a886 100644
--- a/internal/tui/optimize_test.go
+++ b/internal/tui/optimize_test.go
@@ -66,6 +66,7 @@ func TestOptimizeForms(t *testing.T) {
t.Fatal(err)
}
v := newModelsView(c)
+ v.advisors = []string{"cloud/big"}
m := client.Model{}
m.ID = "big"
@@ -74,7 +75,7 @@ func TestOptimizeForms(t *testing.T) {
if v.action != "optimize" || v.saveable() {
t.Fatalf("action %q saveable %v", v.action, v.saveable())
}
- v.optProfile, v.confirm, v.form = string(optimize.Throughput), true, nil
+ v.optProfile, v.optAdvisor, v.confirm, v.form = string(optimize.Throughput), "cloud/big", true, nil
drain(v.submit())
// A running run offers to cancel.
m.Optimize = &client.Optimize{Status: "running"}
@@ -105,7 +106,7 @@ func TestOptimizeForms(t *testing.T) {
}
mu.Lock()
defer mu.Unlock()
- want := []string{`POST /models/big/optimize {"profile":"throughput"}`, `DELETE /models/big/optimize null`,
+ want := []string{`POST /models/big/optimize {"advisor":{"model":"big","provider":"cloud"},"profile":"throughput"}`, `DELETE /models/big/optimize null`,
`POST /models/big/optimize/apply {"backend":true}`}
if strings.Join(got, "\n") != strings.Join(want, "\n") {
t.Errorf("requests:\n%s", strings.Join(got, "\n"))