Optimizer phase 4: the LLM advisor

docs/design/optimizer.md

@@ -74,7 +74,7 @@ The search itself runs in the System API (`internal/optimize`). It writes one re
 
 The advisor is off by default. When the owner turns it on, the optimizer asks one provider model that the router already has.
 
-- A new router route, `POST /sibyl/advise`, takes a provider and a model. Only the System API key can call it, as with `/sibyl/classify`. The provider keys stay in the router.
+- A new router route, `POST /sibyl/advise`, takes a provider and a model. Only the System API key can call it, as with `/sibyl/classify`. The provider keys stay in the router. The router counts the cost in `usage.json` under `@optimizer`, apart from the calibration budget.
 - The optimizer sends the GPU names and VRAM, the system RAM, the backend, the model architecture, the parameter count, the quantization, the layer and expert counts, the context size, and every measured result.
 - The advisor answers with JSON: up to six candidates, each one a set of the settings in the table above. The optimizer checks each candidate with the same allowlist as the agent, and drops what fails.
 - The advisor is most useful for `-ot` patterns on a new MoE architecture, and for the order of the stages on hardware that the fixed search does not know.

docs/guides/models.md

@@ -192,6 +192,8 @@ The optimizer measures llama.cpp settings for one model on your hardware, and re
 4. Read the result in the model's details: the speed now, the best speed, and each setting that changes.
 5. Press `a` to apply the settings. In the web console, use "Apply these settings". If a different backend was faster, Sibyl asks whether to switch the backend too. The backend applies to every model.
 
+If the router has a provider with a key, you can choose an advisor when you start. The advisor is a provider model that proposes more settings after the fixed stages, and the optimizer measures them like the others. It gets the GPUs, the system RAM, the model facts and the measurements. It gets no prompts, no user data and no file paths. Each run makes one call, and the router counts its cost under `@optimizer`.
+
 A setting wins only when it beats the settings now by more than 3 percent and by more than the noise of the runs. The optimizer applies nothing by itself. A result is for the hardware and the llama.cpp version of the run, so run it again after an update.
 
 ## Model pools

docs/reference/configuration.md

@@ -269,7 +269,7 @@ The `calibration` section sets up continuous calibration. The router keeps a sam
 | `no_seed` | boolean | `false` | Stops the first measurement of a new local model with the labelled request set. |
 | `auto_apply` | boolean | `false` | Lets the System API apply a rule proposal by itself, when the profile already names the proposal's route. |
 
-A provider route needs a price for calibration, because the calibration budget counts its cost. The router keeps the calibration spend in `usage.json` under the name `@calibration`.
+A provider route needs a price for calibration, because the calibration budget counts its cost. The router keeps the calibration spend in `usage.json` under the name `@calibration`, and the spend of the optimizer's advisor under `@optimizer`.
 
 ### Rule proposals
 

docs/reference/system-api.md

@@ -140,7 +140,7 @@ A value that does not fit the variant or the detected GPUs returns `422`.
 | `DELETE /models/{id}` | None | The apply result | Deletes the model file and all of its shards, removes its settings, and applies. |
 | `POST /models/{id}/benchmark` | None | `202`, `{"status": "queued"}` | Asks the host agent to run `llama-bench` on the model with each backend image. |
 | `GET /models/{id}/benchmark` | None | The benchmark state | The progress or the result of the last benchmark. `404` if no benchmark ran. |
-| `POST /models/{id}/optimize` | `{"profile": "single"}` or `"throughput"` | `202`, the optimizer state | Starts the optimizer for a chat model. It unloads the models while it runs. `409` while another run goes. |
+| `POST /models/{id}/optimize` | `{"profile": "single"}` or `"throughput"`, and an optional `"advisor": {"provider": "...", "model": "..."}` | `202`, the optimizer state | Starts the optimizer for a chat model. It unloads the models while it runs. The advisor must be a provider of the router. `409` while another run goes. |
 | `GET /models/{id}/optimize` | None | The optimizer state | The stage, every measured result, and the recommendation when the run is done. `404` if no run took place. |
 | `DELETE /models/{id}/optimize` | None | `202` | Cancels the run. The tool that runs on the host finishes first. |
 | `POST /models/{id}/optimize/apply` | `{"backend": true}` or `false` | The apply result | Writes the recommended settings. With `backend` true, it also switches the backend of every model when the run found a faster one. `409` if the settings did not load at the model's context. |
@@ -430,4 +430,4 @@ Port 8443 also serves the inference APIs. They take a router client key, not the
 | `/api/*` | System API (Ollama shim) | The Ollama-compatible API: `version`, `tags`, `show`, `ps`, `chat`, `generate`, `embed`, `embeddings` |
 | `/console/*` | System API | The web console |
 
-The router also has two routes that only the System API key can call: `POST /sibyl/explain` and `POST /sibyl/classify`. Caddy does not expose them. Call `POST /router/explain` and `POST /router/classify` on the System API instead.
+The router also has three routes that only the System API key can call: `POST /sibyl/explain`, `POST /sibyl/classify` and `POST /sibyl/advise`. Caddy does not expose them. Call `POST /router/explain` and `POST /router/classify` on the System API instead. The optimizer's advisor uses `/sibyl/advise`.

internal/api/optimize.go

@@ -1,10 +1,12 @@
 package api
 
 import (
+	"bytes"
 	"context"
 	"encoding/json"
 	"errors"
 	"fmt"
+	"io"
 	"net/http"
 	"os"
 	"path/filepath"
@@ -18,17 +20,28 @@ import (
 	"github.com/diffsec/sibyl/internal/hostctl"
 	"github.com/diffsec/sibyl/internal/models"
 	"github.com/diffsec/sibyl/internal/optimize"
+	"github.com/diffsec/sibyl/internal/router"
 )
 
 // OptimizeRequest starts an optimizer run.
 type OptimizeRequest struct {
 	Profile optimize.Profile `json:"profile"`
+	// Advisor is an optional router provider model that proposes more
+	// settings to measure.
+	Advisor *OptimizeAdvisor `json:"advisor,omitempty"`
+}
+
+// OptimizeAdvisor names a provider model of the router.
+type OptimizeAdvisor struct {
+	Provider string `json:"provider"`
+	Model    string `json:"model"`
 }
 
 // OptimizeState is the state of a model's last optimizer run.
 type OptimizeState struct {
 	Status         string                   `json:"status"` // running, done, failed or cancelled
 	Profile        optimize.Profile         `json:"profile"`
+	Advisor        string                   `json:"advisor,omitempty"` // provider/model
 	Stage          string                   `json:"stage,omitempty"`
 	Started        string                   `json:"started"`
 	Updated        string                   `json:"updated"`
@@ -98,7 +111,7 @@ func (s *Server) startOptimize(w http.ResponseWriter, r *http.Request) {
 		writeError(w, http.StatusUnprocessableEntity, fmt.Errorf("profile %q: use %s or %s", req.Profile, optimize.Single, optimize.Throughput))
 		return
 	}
-	in, runner, err := s.optimizeInput(id, req.Profile)
+	in, runner, err := s.optimizeInput(id, req.Profile, req.Advisor)
 	if err != nil {
 		code := http.StatusUnprocessableEntity
 		if errors.Is(err, os.ErrNotExist) {
@@ -120,6 +133,9 @@ func (s *Server) startOptimize(w http.ResponseWriter, r *http.Request) {
 
 	st := OptimizeState{Status: "running", Profile: req.Profile, Stage: "baseline",
 		Started: time.Now().UTC().Format(time.RFC3339), ImageTag: s.ImageTag}
+	if req.Advisor != nil {
+		st.Advisor = req.Advisor.Provider + "/" + req.Advisor.Model
+	}
 	if err := s.writeOptimize(id, st); err != nil {
 		s.endOptimize()
 		writeError(w, http.StatusInternalServerError, err)
@@ -164,7 +180,7 @@ func (s *Server) runOptimize(ctx context.Context, id string, st OptimizeState, i
 }
 
 // optimizeInput describes the model and the host for a run.
-func (s *Server) optimizeInput(id string, profile optimize.Profile) (optimize.Input, optimize.Runner, error) {
+func (s *Server) optimizeInput(id string, profile optimize.Profile, adv *OptimizeAdvisor) (optimize.Input, optimize.Runner, error) {
 	s.mu.Lock()
 	defer s.mu.Unlock()
 	m, err := s.findModel(id)
@@ -216,10 +232,96 @@ func (s *Server) optimizeInput(id string, profile optimize.Profile) (optimize.In
 		}
 		in.MoE = moeCandidates(set.MoE.CPULayers, in.Layers, meta.ExpertCount)
 	}
+	if adv != nil {
+		rc, err := s.loadRouter()
+		if err != nil {
+			return optimize.Input{}, nil, err
+		}
+		if _, ok := rc.Providers.Providers[adv.Provider]; !ok || adv.Model == "" || strings.ContainsAny(adv.Model, " \t\r\n") {
+			return optimize.Input{}, nil, fmt.Errorf("advisor: the router has no provider %q with a model %q", adv.Provider, adv.Model)
+		}
+		in.Advisor = optimize.LLMAdvisor{Ask: s.askAdvisor(adv.Provider, adv.Model), Facts: s.optimizeFacts(m, meta, used)}
+	}
 	runner := &agentRunner{s: s, id: id, rel: filepath.ToSlash(m.Path), pool: m.Pool, gpus: strings.Join(list, ","), poll: s.tunePoll}
 	return in, runner, nil
 }
 
+// optimizeFacts describes the hardware and the model for the advisor. It
+// holds no prompts, no user data and no file paths.
+func (s *Server) optimizeFacts(m models.Model, meta *models.Meta, used []gpu.GPU) string {
+	var b strings.Builder
+	for i, g := range used {
+		parts := []string{g.Vendor}
+		if g.DeviceID != "" {
+			parts = append(parts, "device "+g.DeviceID)
+		}
+		if g.Driver != "" {
+			parts = append(parts, "driver "+g.Driver)
+		}
+		if g.VRAMTotal > 0 {
+			parts = append(parts, fmt.Sprintf("%.1f GiB VRAM", float64(g.VRAMTotal)/(1<<30)))
+		}
+		if g.Arch != "" {
+			parts = append(parts, "ROCm target "+g.Arch)
+		}
+		if g.LinkWidth != "" {
+			parts = append(parts, fmt.Sprintf("PCIe %s at %s", g.LinkWidth, g.LinkSpeed))
+		}
+		if g.VisibleVRAM > 0 && g.VisibleVRAM < g.VRAMTotal {
+			parts = append(parts, "Resizable BAR off")
+		}
+		fmt.Fprintf(&b, "GPU %d: %s\n", i, strings.Join(parts, ", "))
+	}
+	if _, ram, _ := s.memory(); ram > 0 {
+		fmt.Fprintf(&b, "System RAM: %.1f GiB\n", float64(ram)/(1<<30))
+	}
+	fmt.Fprintf(&b, "Model file: %s, %.1f GiB\n", filepath.Base(m.Path), float64(m.Size)/(1<<30))
+	if meta != nil {
+		fmt.Fprintf(&b, "Architecture: %s, native context %d tokens, %d attention layers", meta.Architecture, meta.ContextLength, meta.AttentionLayers)
+		if meta.ExpertCount > 0 {
+			fmt.Fprintf(&b, ", %d experts per MoE layer", meta.ExpertCount)
+		}
+		b.WriteString("\n")
+	}
+	return b.String()
+}
+
+// askAdvisor asks the router's /sibyl/advise route. The provider keys stay
+// in the router.
+func (s *Server) askAdvisor(provider, model string) func(ctx context.Context, system, prompt string) (string, error) {
+	return func(ctx context.Context, system, prompt string) (string, error) {
+		body, err := json.Marshal(router.AdviseRequest{Provider: provider, Model: model, System: system, Prompt: prompt})
+		if err != nil {
+			return "", err
+		}
+		ctx, cancel := context.WithTimeout(ctx, 4*time.Minute)
+		defer cancel()
+		req, err := http.NewRequestWithContext(ctx, http.MethodPost, s.RouterURL+"/sibyl/advise", bytes.NewReader(body))
+		if err != nil {
+			return "", err
+		}
+		req.Header.Set("Authorization", "Bearer "+s.Token)
+		req.Header.Set("Content-Type", "application/json")
+		resp, err := http.DefaultClient.Do(req)
+		if err != nil {
+			return "", fmt.Errorf("reach the router: %w", err)
+		}
+		defer func() { _ = resp.Body.Close() }()
+		raw, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
+		if err != nil {
+			return "", err
+		}
+		if resp.StatusCode != http.StatusOK {
+			return "", fmt.Errorf("the router answered %d: %.300s", resp.StatusCode, raw)
+		}
+		var ar router.AdviseResponse
+		if err := json.Unmarshal(raw, &ar); err != nil {
+			return "", err
+		}
+		return ar.Answer, nil
+	}
+}
+
 // moeCandidates are the CPUMoE values to try: fewer layers with experts in
 // system RAM than now. Fewer is faster, if it loads.
 func moeCandidates(now, layers, experts int) []int {

internal/api/optimize_test.go

@@ -5,14 +5,18 @@ import (
 	"encoding/json"
 	"fmt"
 	"net/http"
+	"net/http/httptest"
 	"os"
 	"path/filepath"
+	"slices"
 	"strings"
+	"sync"
 	"testing"
 	"time"
 
 	"github.com/diffsec/sibyl/internal/models"
 	"github.com/diffsec/sibyl/internal/optimize"
+	"github.com/diffsec/sibyl/internal/router"
 )
 
 // fakeAgent answers tune requests until the test ends: Vulkan generates
@@ -180,3 +184,46 @@ func TestOptimizeRefusals(t *testing.T) {
 }
 
 var optimizeRecommendationUnverified = optimize.Recommendation{}
+
+func TestOptimizeAdvisor(t *testing.T) {
+	f := newFixture(t)
+	f.server.tunePoll = 5 * time.Millisecond
+	f.do(t, "PUT", "/hardware", `{"backend":"hip","gpus":[1,0],"layout":"span"}`, true)
+	write(t, f.paths.routerFile(router.ProvidersName), "providers:\n  cloud:\n    type: openai\n    base_url: https://x/v1\n")
+	var mu sync.Mutex
+	var got router.AdviseRequest
+	var auth string
+	rt := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		mu.Lock()
+		defer mu.Unlock()
+		auth = r.Header.Get("Authorization")
+		_ = json.NewDecoder(r.Body).Decode(&got)
+		_, _ = w.Write([]byte(`{"route":"cloud/big","answer":"{\"candidates\":[{\"ubatch\":1024,\"split_mode\":\"row\"}]}"}`))
+	}))
+	defer rt.Close()
+	f.server.RouterURL = rt.URL
+	fakeAgent(t, f, false)
+	if code, b := f.do(t, "POST", "/models/big/optimize", `{"profile":"single","advisor":{"provider":"nope","model":"big"}}`, true); code != 422 {
+		t.Errorf("unknown provider = %d %v", code, b)
+	}
+	if code, b := f.do(t, "POST", "/models/big/optimize", `{"profile":"single","advisor":{"provider":"cloud","model":"big"}}`, true); code != 202 {
+		t.Fatalf("start = %d %v", code, b)
+	}
+	b := waitOptimize(t, f, "big", "done")
+	if b["advisor"] != "cloud/big" {
+		t.Errorf("advisor %v", b["advisor"])
+	}
+	var stages []string
+	for _, r := range b["results"].([]any) {
+		stages = append(stages, r.(map[string]any)["stage"].(string))
+	}
+	if !slices.Contains(stages, "advisor") {
+		t.Errorf("no advisor result in %v", stages)
+	}
+	mu.Lock()
+	defer mu.Unlock()
+	if auth != "Bearer "+token || got.Provider != "cloud" || got.Model != "big" || !strings.Contains(got.Prompt, "GPU 0: amd\n") ||
+		!strings.Contains(got.Prompt, "big-00001-of-00002.gguf") || strings.Contains(got.Prompt, f.paths.Models) {
+		t.Errorf("advise request auth %q %+v", auth, got)
+	}
+}

internal/client/client.go

@@ -221,6 +221,7 @@ type Benchmark struct {
 type Optimize struct {
 	Status         string                   `json:"status"` // running, done, failed or cancelled
 	Profile        optimize.Profile         `json:"profile"`
+	Advisor        string                   `json:"advisor,omitempty"`
 	Stage          string                   `json:"stage,omitempty"`
 	Started        string                   `json:"started"`
 	Updated        string                   `json:"updated"`
@@ -293,9 +294,29 @@ func (c *Client) StartBenchmark(ctx context.Context, id string) error {
 	return c.do(ctx, http.MethodPost, "/models/"+url.PathEscape(id)+"/benchmark", nil, nil)
 }
 
-// StartOptimize starts the optimizer for a model.
-func (c *Client) StartOptimize(ctx context.Context, id string, p optimize.Profile) error {
-	return c.do(ctx, http.MethodPost, "/models/"+url.PathEscape(id)+"/optimize", map[string]any{"profile": p}, nil)
+// StartOptimize starts the optimizer for a model. advisor is a router
+// provider model as provider/model, or empty for none.
+func (c *Client) StartOptimize(ctx context.Context, id string, p optimize.Profile, advisor string) error {
+	body := map[string]any{"profile": p}
+	if prov, model, ok := strings.Cut(advisor, "/"); ok {
+		body["advisor"] = map[string]string{"provider": prov, "model": model}
+	}
+	return c.do(ctx, http.MethodPost, "/models/"+url.PathEscape(id)+"/optimize", body, nil)
+}
+
+// AdvisorChoices lists the router provider models that can advise the
+// optimizer, as provider/model. A provider needs its key.
+func AdvisorChoices(r Router) []string {
+	var out []string
+	for _, p := range r.Providers {
+		if !p.KeySet {
+			continue
+		}
+		for _, m := range p.Models {
+			out = append(out, p.Name+"/"+m)
+		}
+	}
+	return out
 }
 
 // CancelOptimize cancels a model's optimizer run.

internal/console/console_test.go

@@ -774,7 +774,8 @@ func TestOptimizerCard(t *testing.T) {
 	}
 	body := w.Body.String()
 	for _, want := range []string{"<h2>Optimizer</h2>", ">48.2<", ">1310<", "ubatch</span>: default to 1024",
-		`action="/console/models/optimize/apply"`, "Switch the backend to vulkan too", `action="/console/models/optimize"`} {
+		`action="/console/models/optimize/apply"`, "Switch the backend to vulkan too", `action="/console/models/optimize"`,
+		`<option>cloud/big</option>`} {
 		if !strings.Contains(body, want) {
 			t.Errorf("no %q in the model page", want)
 		}
@@ -791,6 +792,8 @@ func TestOptimizerCard(t *testing.T) {
 		want string
 	}{
 		{"/console/models/optimize", url.Values{"id": {"qwen"}, "profile": {"throughput"}}, "POST /models/qwen/optimize", `{"profile":"throughput"}`},
+		{"/console/models/optimize", url.Values{"id": {"qwen"}, "profile": {"single"}, "advisor": {"cloud/big"}}, "POST /models/qwen/optimize",
+			`{"advisor":{"model":"big","provider":"cloud"},"profile":"single"}`},
 		{"/console/models/optimize/apply", url.Values{"id": {"qwen"}, "backend": {"on"}}, "POST /models/qwen/optimize/apply", `{"backend":true}`},
 		{"/console/models/optimize/apply", url.Values{"id": {"qwen"}}, "POST /models/qwen/optimize/apply", `{"backend":false}`},
 	}

internal/console/models.go

@@ -181,11 +181,15 @@ func (c *Console) modelPage(w http.ResponseWriter, r *http.Request) {
 		moe = m.Meta.ExpertCount > 0
 	}
 	spec := slices.DeleteFunc(slices.Clone(models.SpecTypes), func(s string) bool { return s == "none" })
+	var advisors []string
+	if rc, err := c.client.Router(r.Context()); err == nil {
+		advisors = client.AdvisorChoices(rc)
+	}
 	data := pageData(r)
 	for k, v := range map[string]any{
 		"Model": m, "Head": head, "Fit": fitText(m.Fit), "Form": modelform.FromSettings(m.Settings),
 		"Layouts": layouts, "Drafts": drafts, "SimpleDrafts": simple, "CacheTypes": models.CacheTypes, "Presets": models.Presets,
-		"SpecTypes": spec, "MoE": moe, "Reasoning": reasoningControl(m),
+		"SpecTypes": spec, "MoE": moe, "Reasoning": reasoningControl(m), "Advisors": advisors,
 	} {
 		data[k] = v
 	}
@@ -251,7 +255,7 @@ func (c *Console) benchModel(w http.ResponseWriter, r *http.Request) {
 
 func (c *Console) optimizeModel(w http.ResponseWriter, r *http.Request) {
 	id := r.PostFormValue("id")
-	err := c.client.StartOptimize(r.Context(), id, optimize.Profile(r.PostFormValue("profile")))
+	err := c.client.StartOptimize(r.Context(), id, optimize.Profile(r.PostFormValue("profile")), r.PostFormValue("advisor"))
 	c.back(w, r, "models/edit?id="+url.QueryEscape(id), "the optimizer for "+id+" started", err)
 }