[cnuss]

What

The metal-agent no longer exits at startup when llama-server is missing, as long as --runtime selects another engine (ollama, omlx, mlx-server, vllm-swift, tensorfold). It logs a warning and registers no llama.cpp executor. An InferenceService that resolves to llamacpp is then refused with the new RuntimeNotAvailable reason, before its Model is fetched or a running engine is stopped, so its status.schedulingStatus and a Warning Event say llama-server was not found and how to fix it. With --runtime llama-server (the default) a missing binary is still fatal.

Also:

• /usr/bin/llama-server is added to the search paths after the two Homebrew ones.
• The "not found" error names brew install llama.cpp only on macOS; elsewhere it points at the distribution package or a source build.
• The same refusal covers any runtime whose binary was not configured (for example mlx-server without its binary). Before, such a service only logged "no executor registered" on every reconcile, after its running engine had already been stopped. RuntimeNotAvailable is in MetalAgentRefusalReasons, so the controller keeps the refusal.
• stopManagedProcess returns an error when no executor is registered for the process's runtime. (Corrected after merge:) on main the llama.cpp executor was always registered, so outside unit tests this path fell back to it and called StopProcess for a process of a runtime that executor doesn't own. With the llama.cpp executor now optional, that fallback can be nil, so the missing executor is an error instead, including for an adopted process with PID 0.

A Mac with llama-server installed behaves as before.

Why

A host that only runs Ollama or oMLX had to install llama.cpp anyway. On Linux the error was a Homebrew hint. With this change a Linux host gets as far as the Metal check and exits with "Metal support not detected"; the Linux platform check is #2520.

Fixes #2516

How

• cmd/metal-agent/main.go: new resolveDefaultLlamaServerBin wraps resolveLlamaServerBin. When the binary is missing, it returns nil (and logs a warning) when IsMetalOnlyRuntime(cfg.Runtime), otherwise the error. The startup log line "llama-server binary found" is printed only when a path resolved. llamaServerInstallHint(goos) supplies the hint.
• pkg/agent/agent.go: the llama.cpp executor setup moves into buildLlamaCPPExecutor, which buildExecutors calls only when LlamaServerBin != "". s3SizeProbe already falls back to a stand-in executor when none is registered, so it needs no change. stopManagedProcess checks for a nil executor after the Ollama, oMLX and adopted-with-PID branches, which handle nil safely.
• reconcileProcess looks up the executor for the resolved runtime before fetchModelThenStopStale and refuses through refuseStart when there is none. runtimeNotAvailableMessage in pkg/agent/refusal.go writes the llama.cpp-specific and the generic message.
• deployment/macos/README.md: the llama.cpp prerequisite says it can be skipped with another --runtime, "Choosing a Runtime per InferenceService" lists the search paths and the refusal, and RuntimeNotAvailable is in the refusal and Events lists and has a Troubleshooting entry.

llmkube metal join still refuses to run without llama-server. I left it unchanged here.

Tests:

• cmd/metal-agent/main_test.go: a table over runtime (llama-server, llamacpp, ollama, omlx, mlx-server) by binary present or missing by override, the /usr/bin path, and the install hint per OS.
• pkg/agent/llama_executor_test.go: buildExecutors registers neither llama.cpp key without a binary and both with one. stopManagedProcess returns an error, and does not panic, for a spawned llama.cpp process and for an adopted process without a PID when no executor is registered. TestEnsureProcessRefusesLlamaCPPWithoutLlamaServer (both runtime names) checks there is no start, status.schedulingStatus is RuntimeNotAvailable, the message names the fix, and there is exactly one Warning Event. TestEnsureProcessRuntimeNotAvailableKeepsRunningEngine checks that a healthy engine with an older spec is not stopped.
• The controller's "keep an agent refusal" table and the refusal-reason scan test include RuntimeNotAvailable.

Checks I ran on macOS (arm64):

• make gate BASE_REF=origin/main on the final commit: every step passed.
• GOOS=linux bin/golangci-lint run ./... and GOOS=darwin bin/golangci-lint run ./...: 0 issues each.
• make lint-deadcode, and the same config with GOOS=linux: 0 issues.
• GOOS=linux go vet ./cmd/metal-agent/ ./pkg/agent/: clean.

Assisted-by: Claude Code (Claude Opus 5.5) split this out of #2482, wrote the description and ran the checks above in my session.

Miscellaneous fixes

Shared code that assumed the host is a Mac now says "host": the AllowUnsafeExtraArgs and currentBackend comments in agent.go, and the --allow-unsafe-extra-args help ("for hosts whose InferenceService authors are fully trusted").

Checklist

• Tests added/updated
• make test passes locally
• make lint passes locally
• Commit messages follow conventional commits
• All commits are signed off (git commit -s) per DCO
• AI assistance (if any) is disclosed above, per CONTRIBUTING.md
• Documentation updated (if user-facing change)

[codecov]

Codecov Report

❌ Patch coverage is 82.85714% with 6 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
cmd/metal-agent/main.go	68.42%	6 Missing ⚠️

📢 Thoughts on this report? Let us know!

[cnuss]

Addressed the review finding in the squashed commit (477ceb9):

• Missing executor stopped the engine first and left no status. reconcileProcess now looks up the executor for the resolved runtime before it fetches the Model or stops a stale engine. A runtime with no executor is refused through refuseStart with a new RuntimeNotAvailable reason, so it shows in status.schedulingStatus/schedulingMessage and as a Warning Event, instead of a log line on every reconcile. For llamacpp/llama-server the message says llama-server was not found and to install llama.cpp or pass --llama-server and restart (or change spec.runtime). An engine already running for the service is left alone. RuntimeNotAvailable is in MetalAgentRefusalReasons so the controller keeps the refusal, and the macOS README covers it under Troubleshooting.

[Defilan]

I checked every lookup of the llama.cpp executor for a nil: s3SizeProbe falls back to a stand-in, warnIfDaemonExposed checks presence, and stopManagedProcess now errors instead of calling a nil executor. An explicit --llama-server path is returned as-is, so a typo there still builds the executor rather than silently degrading. Moving the refusal ahead of fetchModelThenStopStale is the right order, and it sits after the eviction, scale-to-zero and matching-hash checks, so none of those change.

A few small things, none blocking:

• The description (and the commit message) says stopManagedProcess used to call StopProcess on a nil executor. On main the llama.cpp executor is always registered, so outside unit tests it fell back to that executor and called StopProcess for a process of a runtime it doesn't own. Worth correcting in the description for the record.
• The starting metal agent log line still prints llamaServerBin as an empty string when it wasn't found; drop the key or say it's absent, as you did for the "found" line.
• No test covers the generic (non-llama.cpp) refusal message, e.g. spec.runtime: mlx-server on an agent started without --mlx-server-bin.
• docs/site/guides/macos-metal.md has a troubleshooting entry per refusal reason (ModelDownloadTooLarge, SourceSecretNotAllowed, ...); RuntimeNotAvailable should get one there too, not only in the README.
• llmkube metal join still says the agent "cannot start without" llama-server. Fine to leave here; could you file the follow-up?

[cnuss]

Thanks. The description now has the correct account of stopManagedProcess (commit messages can't change after the squash merge). The rest is filed for the next PRs:

• the empty llamaServerBin log key, a test for the generic RuntimeNotAvailable message, and a RuntimeNotAvailable entry in macos-metal.md: [Raspberry Pi] Run the metal-agent on Linux hosts (experimental) #2520 (comment)
• metal join still saying the agent can't start without llama-server: [Raspberry Pi] Run metal join and leave on Linux through a systemd user unit #2525 (comment)