[Defilan]

What

Two related metal agent fixes, one commit each, plus a third commit that keeps engine log content out of every Event they add. Three follow-ups came on 2026-11-13: a test (7909d4e6b), a comment and doc fix (339e31be8), and rounding the delay in Events (734c6380c), found in the live test below.

Rebased 2026-11-13 onto main e012e2e00, after #2529 (RuntimeNotAvailable), #2531 and #2462 (adopting a running engine that misses its probe as unhealthy) merged.

• Conflicts: all were from feat(metal-agent): start without llama-server when --runtime selects another engine #2529 adding RuntimeNotAvailable in the same places this PR adds StartFailed: the reason constants and MetalAgentRefusalReasons, the Events list in the macOS README, and the controller test entries. Both sides are kept.
• feat(metal-agent): start without llama-server when --runtime selects another engine #2529: a RuntimeNotAvailable refusal returns at the executor lookup, before the start. StartFailed is written only after a start fails, so a refusal is never reported or retried as StartFailed.
• fix: preserve unhealthy running engines during adoption #2462: 7909d4e6b extends TestEngineExit_AdoptedEngine with an engine adopted unhealthy. When it exits, its restart is queued at once and one EngineExited Event is emitted. The test fails if the exit watch skips unhealthy engines. The health monitor and the exit watch cannot both restart one engine: stopForRestart checks and removes the process in one critical section.
• Second review (qwen-review, on the rebase): it confirmed both interactions above. Its must-fixes were comments that still called MetalAgentRefusalReasons a list of refusals only, which StartFailed contradicts. Those comments, plus four nits from the same review, are fixed in 339e31be8:
  • markExited's comment now states the 1 s window a late probe really has;
  • the README now names the engine log file and says an exit restart backs off like StartFailed;
  • a comment now says the newline cut in summarizeForEvent is what keeps log content out of Events;
  • the test now reads the process under the lock.
• Its open questions, checked:
  • A failed record write cannot leak an engine: afterSpawn stops the child.
  • Ollama records carry no PID, and oMLX records are discarded at adoption, so no shared daemon PID is ever watched.

A failed start is reported and retried (#2404). A start that fails for a reason other than a refusal or a transient API read (the engine exits while loading, the download fails on a full disk or a bad status, the engine never answers its startup health check, the engine record cannot be written) now:

• writes status.schedulingStatus: StartFailed, with the error's first line as schedulingMessage, so the InferenceService stops saying "Waiting for the host metal-agent to fetch the model and register Endpoints";
• emits a StartFailed Warning Event with the error's first line, the retry delay and the path of the engine log on the Mac (<log> and <log>.1). A failed engine-record write keeps its existing EngineRecordFailed reason;
• is retried on the health-restart queue and backoff: after 10s, then 20s, 40s, up to 5 minutes, without any InferenceService change. A delete, scale to zero, suspend or agent shutdown cancels the retry, and a spec change starts the new spec at once, as for a health restart. Each retry that fails emits RestartFailed in the same shape. The status is cleared once an engine runs.

An engine exit is acted on at once (#2405). When an engine the agent is serving exits:

• its endpoint is withdrawn at once, not at the next 30s probe;
• the exit status is logged with the engine log tail, and an EngineExited Warning Event carries the exit status and the path of the crashed run's log;
• the engine is restarted without waiting for the three-probe threshold (at once for the first restart, then on the same backoff);
• opening the engine log for the restart moves the crashed run's log to <runtime>-<namespace>-<name>.log.1 instead of truncating it.

The probe threshold is unchanged for an engine that is alive but not answering. Engines adopted from a previous agent process are covered too on macOS (see How).

No engine log content in Events. None of these Events, and not the scheduling message, carries engine log content: only the error's first line (cut at the first newline and at 200 bytes, with any attached log tail removed), the exit status, the retry delay and where the log is. The last lines of the engine log go to the agent's own log.

Docs: the agent Events list in deployment/macos/README.md names StartFailed and EngineExited, says what each carries, and says why neither carries engine log content. docs/site/guides/macos-metal.md is not touched (another change restructures it).

Why

Fixes #2404
Fixes #2405

Before this change a failed start was log-only: no Event, no status, no retry, and the InferenceService kept claiming the agent was fetching the model until something else bumped its resourceVersion or the agent restarted (refusal.go said such failures are not requeued). An engine that exited kept its endpoint for up to 30s, was restarted only after about 90s, left no exit status anywhere, and its restart truncated the log that explained the crash.

How

This changes metal agent behavior: failed starts are now retried on a backoff and reported, engine exits trigger an immediate withdrawal and restart, and the previous engine log is kept as .log.1. Engine log content still never reaches an Event.

Failed starts. handleStartProcessError sends every non-refusal, non-transient StartProcess failure to the new failStart (pkg/agent/start_failure.go). It writes the status on a fresh copy of the InferenceService, schedules a retry on the service's existing restartBackoff (the delay starts at 10s), emits the Event, and logs the full error, tail included, to the agent log. StartFailed is added to api/v1alpha1.MetalAgentRefusalReasons, so the controller's determinePhase keeps it instead of overwriting it with WaitingForMetalAgent, and the agent's refusal-reason scan test and the controller's table test both cover it. It is not a refusal: it is not re-checked on the refusal backoff (5s to 1 min), because a retry may download or load the whole model again. runRestart marks its own attempt (restartBackoff.attempting), so a failed retry writes the status but is reported once, as RestartFailed, and rescheduled once. Memory admission already clears the status before a start; clearStartFailure covers memory-check-mode=warn, where admission can end without clearing it.

One consequence to know about: while a failed start's retry is pending, a watch event for the same spec (a label or annotation edit, or the agent's own status write) is left to the retry, exactly as deferToPendingRestart already does for a health restart. Before, such an edit was the only thing that retried a failed start at all; now it waits for the backoff (at most 5 minutes), and a spec change still starts at once. TestEnsureProcess_StartFailureWithdrawsThenRecovers relied on the immediate re-delivery and now recovers through the retry; TestEnsureProcess_PlainStartFailureNotRefused required no Event and an empty status (the #2404 bug) and now requires StartFailed without a refusal record.

Engine log content stays out of Events. Events are readable by anyone who can read the namespace and are commonly exported to central logging, and an engine log can carry request text (an engine run with verbose logging, through extraArgs, logs prompts). So summarizeForEvent (pkg/agent/child_exit.go) builds what an Event, or the scheduling message, may say about a start error: the first line, cut at 200 bytes, with any engine log tail removed. withLogTail now returns a typed error so the tail can be dropped exactly, and the first-line cut also covers errors that embed other multi-line text. Each Event names the log on the Mac instead (<log> for the failed run, <log>.1 for the run before it, or for EngineExited the crashed run's log and where the restart moves it) and says its last lines are in the agent log. The agent log keeps the full tail: failStart and runRestart log the whole error, and the exit handler logs the tail it reads before the restart rotates the log. runRestart's comment is back to saying the tail belongs in the agent log, not in an Event, with the reason. The first two commits put a bounded tail in these Events; the third removes it, so a squash merge never ships it.

Engine exits. The four executors that spawn one engine per service (llama-server, mlx-server, vllm-swift, tensorfold) hand the engine's reaper and log path to its ManagedProcess. After reconcileProcess stores the process, watchEngineExit (pkg/agent/engine_exit.go) waits on the reaper. On an exit it acts only if the agent still tracks that process: every stop the agent makes itself (delete, scale to zero, spec change, eviction, health restart) removes the process from the table before stopping it, so those exits are ignored. It reads the log tail first (the restart rotates the log) for the agent log, marks the engine unhealthy, withdraws the endpoint, and then reuses the health restart's stop-and-schedule body, which restartUnhealthy now shares as stopForRestart. The restart therefore goes through the same backoff, start guard, delete and scale-to-zero cancellation, and engine-record cleanup; stopping an exited child goes through its reaper and signals nothing, so a reused PID is never hit. If a start holds the service's key when the exit arrives, the handler retries every second while the engine is still tracked, and withdraws again before each retry, because that start may have registered the endpoint Ready after the first withdrawal. The heartbeat already withdraws instead of registering an unhealthy engine.

Adopted engines. An engine adopted from a previous agent process (#2079) is not this process's child, so it has no reaper. On macOS, Start sets up a kqueue EVFILT_PROC/NOTE_EXIT watch for each adopted engine (pkg/agent/procexit_darwin.go) after the registry is in its mode, then re-proves the engine's identity (its start token), so a PID reused between adoption and the watch is never treated as the engine; an engine already gone at either step is handled as exited. The kernel gives no exit status for a non-child, and the Event says so; it names the log path from the engine's record. The group stop of an adopted engine that is already gone reports an unclean stop, so its restart waits the existing 10s release floor rather than starting at once. On other platforms watchProcessExit is a stub and the probe stays the detector (the agent ships for macOS only). The shared oMLX daemon and Ollama serve many models from one process and are not covered: the probe stays their detector.

Log rotation. openEngineLog first renames a regular-file log to <log>.1 (one generation, replacing an older .1), then opens a new log with O_NOFOLLOW and mode 0600 as before; the rotated file keeps the mode the agent created it with. A symlink at the log path is not rotated, so the start still fails naming it, and a symlink planted at .1 is replaced by rename(2), never followed. Rotation is best effort: if the rename fails, the new run truncates the log as before. It applies to every start of an engine, so .1 is always the run before the current one.

Consumers. status.schedulingStatus is read by the controller's determinePhase (now keeps StartFailed) and shown in the InferenceService printer column. The new Event reasons are read by people (kubectl describe). The model store gains one .log.1 per engine. No CRD, RBAC, chart or wire format changes.

Tests. Each repro was written and run against unmodified upstream/main before any production edit. #2404 (first versions of the tests in pkg/agent/start_failure_test.go):

--- FAIL: TestEnsureProcess_FailedStartIsReportedAndRetried (0.01s)
    start_failure_test.go:135: events = [], want one Warning StartFailed
    start_failure_test.go:145: status = "" / "", want StartFailed naming the exit status
    start_failure_test.go:148: no restart pending after the failed start: it is never retried
    start_failure_test.go:172: at +10s: starts = 1, want 2
--- FAIL: TestEnsureProcess_FailedStartStatusClearedWithoutAdmission (0.01s)
    start_failure_test.go:211: status = "" after the failed start, want StartFailed
--- FAIL: TestHealthRestart_FailedRestartSaysStartFailed (0.00s)
    start_failure_test.go:307: status after a failed restart = "", want StartFailed

#2405 (pkg/agent/engine_exit_test.go; the agent-level test runs the real llama-server executor against a fake engine script the test spawns and SIGKILLs):

--- FAIL: TestEngineExit_WithdrawsAndRestartsAtOnce (9.68s)
    engine_exit_test.go:201: the endpoint still advertises Ready 3s after the engine exited
    engine_exit_test.go:204: no restart queued 3s after the engine exited
    engine_exit_test.go:207: EngineExited events = [], want one
--- FAIL: TestOpenEngineLog_KeepsOnePreviousGeneration (0.00s)
    engine_exit_test.go:297: llama-server-ns-svc.log.1 = "not the agent's\n" (<nil>), want "run 2\n"
    engine_exit_test.go:306: llama-server-ns-svc.log.1 mode = Lrwxr-xr-x, want a regular file with mode 0600

After review, TestEvents_NeverCarryEngineLogContent (pkg/agent/engine_log_privacy_test.go) drives StartFailed, RestartFailed and EngineExited with an engine log whose tail carries the string PROMPT-SENTINEL, and requires that no Event carries it, that each Event names the log, and that the agent log keeps it. Against the first two commits (c75fdc5c7) it failed:

--- FAIL: TestEvents_NeverCarryEngineLogContent (0.06s)
    --- FAIL: TestEvents_NeverCarryEngineLogContent/StartFailed (0.05s)
        engine_log_privacy_test.go:123: an Event carries engine log content: "Warning StartFailed Start failed: llama-server process exited before becoming healthy (exit status 1); retrying in 10s. Last lines of /var/folders/87/zq8by9j569ddkq9x3kvw2h8c0000gn/T/TestEvents_NeverCarryEngineLogContentStartFailed3538634903/001/llama-server-default-svc.log:\nsrv  log_server_r: request: {\"prompt\": \"PROMPT-SENTINEL tell me the launch codes\"}\nllama_model_load: error loading model: unknown model architecture: 'qwen9'"
    --- FAIL: TestEvents_NeverCarryEngineLogContent/RestartFailed (0.00s)
        engine_log_privacy_test.go:123: an Event carries engine log content: "Warning RestartFailed Restart did not bring the engine back: failed to start process: llama-server process exited before becoming healthy (exit status 1); retrying in 10s. Last lines of /var/folders/87/zq8by9j569ddkq9x3kvw2h8c0000gn/T/TestEvents_NeverCarryEngineLogContentRestartFailed2992024668/002/llama-server-default-svc.log:\nsrv  log_server_r: request: {\"prompt\": \"PROMPT-SENTINEL tell me the launch codes\"}\nllama_model_load: error loading model: unknown model architecture: 'qwen9'"
    --- FAIL: TestEvents_NeverCarryEngineLogContent/EngineExited (0.01s)
        engine_log_privacy_test.go:123: an Event carries engine log content: "Warning EngineExited Engine process 1111 exited (exit status 0); restarting in 0s. Last lines of /var/folders/87/zq8by9j569ddkq9x3kvw2h8c0000gn/T/TestEvents_NeverCarryEngineLogContentEngineExited160974992/002/llama-server-default-svc.log:\nsrv  log_server_r: request: {\"prompt\": \"PROMPT-SENTINEL tell me the launch codes\"}\nllama_model_load: error loading model: unknown model architecture: 'qwen9'"
        engine_log_privacy_test.go:130: the agent log does not keep the engine log tail

TestSummarizeForEvent pins the summary: the tail of a wrapped engine-crash error is dropped, a multi-line error keeps its first line, and a long line is cut at 200 bytes. The tests that used to require tail content in an Event now require the log path and forbid the content.

Other tests guard what must not change (written with the repros, they pass on main) or cover the fix's own edges (added with it): retry cancellation on delete, scale to zero and spec change; no report or retry for a start cut short by shutdown; the Event tail bound; one Event per failed restart; every spawning executor hands over its reaper and log path; an exit while a start holds the key (including that start's registration); an engine the agent stops itself is not a crash; a symlinked log is still refused; and, on darwin, the kqueue watch on a non-child, on a gone PID, after its context ends, and an adopted engine's exit end to end with the real process inspector.

Falsification: each piece of the fix was reverted on its own and the tests guarding it failed (the first twelve rows were run before the third commit, against the tests then; the Event-content assertions in those rows were replaced by the log-path assertions above):

Reverted	Failing test(s)
failStart call in handleStartProcessError	FailedStartIsReportedAndRetried (events = [], status "", no retry), PlainStartFailureNotRefused
StartFailed status write	FailedStartIsReportedAndRetried, FailedRestartSaysStartFailed
attempting mark in runRestart	FailedRestartSaysStartFailed (duplicate StartFailed Event), FailedStartIsReportedAndRetried (backoff skipped a step)
clearStartFailure call	FailedStartStatusClearedWithoutAdmission
StartFailed in MetalAgentRefusalReasons	TestMetalAgentRefusalReasons_CoversEveryAgentRefusalReason, controller determinePhase table entry
watchEngineExit call	WithdrawsAndRestartsAtOnce (Ready 3s after exit, no restart, no Event)
rotateEngineLog call	KeepsOnePreviousGeneration, WithdrawsAndRestartsAtOnce (no .log.1)
regular-file check in the rotation	SymlinkedLogStillRefused
retry loop while a start holds the key	WaitsForAStartInFlight
re-withdraw after that wait	WaitsForAStartInFlight (endpoint left Ready)
vllm-swift reaper hand-over	HandOverReaperAndLog/vllm-swift
adopted engine's watch assignment	TestEngineExit_AdoptedEngine
first-line cut in summarizeForEvent	TestSummarizeForEvent (multi-line error leaks its second line)
200-byte cap in summarizeForEvent	TestSummarizeForEvent (long line kept whole)

Risks.

• A start that keeps failing is now retried indefinitely, at most every 5 minutes, and each retry may load (or resume downloading) the model. Before, it was never retried.
• A crash loop restarts at once and then backs off, as a health restart does; the backoff resets after 10 minutes of healthy running.
• The Events give the error's first line and a log path, not the engine's own explanation, so diagnosing a failure takes a look at the agent log or the engine log on the Mac. The first line is cut at 200 bytes and could still name a model source URL or a local path, as the agent's existing Events and statuses already do.
• One goroutine per running engine waits on its reaper; on macOS each adopted engine also holds a kqueue descriptor and a goroutine that wakes once a second to check for shutdown.

Live test on an M6 Mac (2026-11-13):

• Setup: Apple M6, 32 GiB, macOS 27.2.1, llama.cpp 0.9.0 from Homebrew. The cluster was a local Apple container cluster running the released 0.10.17 operator from Helm. The agent was built from 339e31be8, then 734c6380c.
• Adopted engine killed with kill -9:
  • The agent's EndpointSlice read not-ready 0.4 s after the kill. Before this PR it waited for failed health probes, about 90 s.
  • The restart came after 10 s, and the service was Ready again at 11.5 s.
  • One EngineExited Event was emitted: "exit status unknown: the engine was adopted from a previous agent process". The crashed run's log was kept as .log.1.
• Agent-spawned engine killed: the same 0.4 s withdrawal, reporting signal: killed. A second kill a few seconds after its restart backed off to 20 s.
• Truncated GGUF (the first 100 MiB of the file): one StartFailed Event, then RestartFailed at 10, 20, 40, 80 and 160 s. No Event carried log content.
• Status: with the 0.10.17 controller in the cluster, schedulingStatus read WaitingForMetalAgent, as expected, since that controller predates StartFailed. With this branch's controller run in its place, it read StartFailed with the error's first line and kept it across reconciles.
• Found and fixed: Events printed delays such as "restarting in 9.999996916s" and "retrying in 1m19.99999675s". 734c6380c rounds them to whole seconds, with a test, and the live run then showed "restarting in 10s".
• Observed, not changed: the agent re-lists InferenceServices every 5 s by name. A delete and re-create with the same spec inside one poll is not noticed, so the old retry backoff carries over to the new object. This is the existing polling design.

AI assistance

Assisted-by: Claude Opus 5.5 (claude-opus-5-5, Anthropic, via Claude Code; roles: investigation, implementation, tests, documentation, PR description, and the 2026-11-13 rebase with its conflict resolution, test and review fixes, plus the live test on an M6 Mac)
Assisted-by: qwen38-flash-next (unsloth/Qwen3.8-Flash-Next-GGUF UD-Q4_K_XL, self-hosted with LLMKube on AMD Strix Halo; roles: read-only review of the rebased branch. Checked the #2529 and #2462 interactions and flagged the stale refusal-list comments and four nits, fixed in 339e31b)

Checklist

• Tests added/updated
• make test passes locally. On the rebased head 339e31be8, go test ./pkg/... ./api/... ./cmd/... -count=1 passes, and the full internal/controller envtest suite passes (198 s), all on darwin. go build and go vet pass for darwin and linux. Before the rebase, only ./pkg/agent/... ./cmd/metal-agent/... and the focused controller spec "should keep an agent refusal" (10 of 1222 specs) were run.
• make lint passes locally: 0 issues on the rebased head 339e31be8 and on 734c6380c, which changes only pkg/agent; go test ./pkg/agent/... passes on it too. Before the rebase, ran ./bin/golangci-lint run on ./bin/golangci-lint run on ./pkg/agent/... ./api/... ./cmd/metal-agent/... under darwin and GOOS=linux: 0 issues each, ./internal/controller/... likewise before the third commit, which does not touch it; plus -c .golangci-deadcode.yml on ./pkg/agent/... and go run ./cmd/check-copyright)
• Commit messages follow conventional commits
• All commits are signed off (git commit -s) per DCO
• AI assistance (if any) is disclosed above, per CONTRIBUTING.md
• Documentation updated (if user-facing change)

[codecov]

Codecov Report

❌ Patch coverage is 79.03226% with 39 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
pkg/agent/engine_exit.go	59.72%	29 Missing ⚠️
pkg/agent/start_failure.go	84.31%	8 Missing ⚠️
pkg/agent/child_exit.go	94.44%	1 Missing ⚠️
pkg/agent/procexit_other.go	0.00%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!