[ihavespoons]

Summary

Phase 2 of the inference optimizer (docs/design/optimizer.md). It is stacked on #43.

• internal/optimize runs the staged search: baseline, backend and split mode, KV cache, batch sizes, MoE layers in system RAM, and parallel requests for the throughput profile. A result wins only when it beats the best by more than 3 percent and 3 standard deviations.
• The verify stage starts llama-server at the model's real context, because llama-bench allocates only its test context. When the best does not load, it drops the memory settings one at a time: the measured smaller micro-batches first, then the default batch, then the KV cache, then half the parallel requests. A variant that was not measured is measured first.
• API: POST, GET and DELETE /models/{id}/optimize, and POST /models/{id}/optimize/apply. Apply reuses the checks of PUT /models/{id} (now setModel). It refuses a recommendation that did not load. One run goes at a time.
• A KV cache type is tried only when it takes no more memory than the model's cache now, so the context still fits.

Verification

• internal/optimize tests run the search against a simulated host. The tests check the chosen backend and split, a q8_0 cache inside the noise that stays f16, a micro-batch that does not load with the measured smaller one winning, a KV type that does not fit and is never measured, MoE with every expert in system RAM, the throughput floor, and failures.
• internal/api tests run the whole API with a fake host agent: start, state, apply with the backend switch, a 409 for a second run, cancel, and the refusals.
• The tests found two flaws in my first verify fallback, which I fixed.
• go vet, go test -race ./... and golangci-lint (0 issues) pass.
• Not run on the box, because it runs a training job.

🤖 Generated with Claude Code