Repository navigation
Conversation
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Collaborator
Author
|
Holding this in draft until I can run |
This was referenced Nov 12, 2026
7 tasks
This was referenced Nov 13, 2026
cnuss
force-pushed
the
feat/linux-host-agent
branch
from
November 13, 2026 03:50
d510fb3 to
7df0d2b
Compare
cnuss
added a commit
to cnuss/LLMKube
that referenced
this pull request
Nov 13, 2026
…ning defilantech#2482 work) This is the part of defilantech#2481 not yet split into its own PR. It sits on the four split commits for defilantech#2516, defilantech#2517, defilantech#2519 and defilantech#2521 and holds the rest of defilantech#2482. API and controller: - A hailo:// Model source names a HEF in Hailo's GenAI model zoo (api/v1alpha1/hailo_source.go); the source pattern and the generated CRDs accept it. - The Model controller marks a valid hailo:// Model Ready without downloading it (MetalAgentManaged), and fails one whose tag is malformed or whose accelerator is not metal with InvalidSource. - Runtime placement refuses a hailo:// Model on a runtime other than ollama. metal-agent: - The agent starts on Linux (experimental) as well as on a host with Metal; on Linux it logs whether a Hailo device node is present (cmd/metal-agent/hailo.go). - The Ollama executor pulls a hailo:// source's tag, refuses it unless the daemon is hailo-ollama (GET /hailo/v1/list), and sends load and unload requests without a prompt field, which hailo-ollama needs. - The agent refuses a hailo:// Model on a runtime other than ollama. - Memory admission sizes a hailo:// source with a HEAD of its HEF, found through hailo-ollama's manifests, before anything is pulled. --hailo-manifests-dir and --hailo-library-url configure it, and metal join passes both through. - Host capacity on Linux reads the board or CPU name from /proc and memory through LinuxMemoryProvider; the sysctl reader moves to the darwin file with its test. e2e and CI: - hack/e2e-host-agent.sh runs two Linux host agents, host-a and host-b, in kind against the Hailo-10H simulator: dial-back, relay chat, per-namespace isolation, admission before pull, and InvalidSource for hailo:// on another accelerator. - make test-e2e-host-agent runs it; test-e2e.yml runs it as a non-blocking job, and the gate exempts the target. Docs: - docs/site/guides/linux-host-agent.md, linked from the nav and the macOS guide: any Linux host first (building the agent, joining by hand, keeping hosts apart, engines, memory and cgroups), then Hailo-10H hosts (hailo-ollama, its systemd unit, hailo:// sources, known limits), with the Raspberry Pi 5 and AI HAT+ 2 install in a subsection of its own. - The generated CRD reference lists the hailo:// scheme. Signed-off-by: Christian Nuss <cnuss@users.noreply.github.com>
cnuss
changed the title
test/hailosim builds Hailo's hailo-ollama 5.1.1 server against HailoRT 5.1.1's public headers and links it to fake_hailort.cpp, a stand-in libhailort that emulates a Hailo-10H. The HTTP layer is hailo-ollama's own code, so its quirks (404 on GET /v1/models, created in nanoseconds, 500 without a JSON Content-Type) carry over. Fake HEFs of exactly HAILO_SIM_HEF_SIZE bytes are seeded under the digests written into the manifests, so /api/pull is an offline cache hit; a small server answers HEAD /blob/sha256_<digest> like dev-public.hailo.ai. The entrypoint validates HAILO_SIM_LISTEN and HAILO_SIM_LIBRARY_LISTEN, exits if the library server cannot bind, and writes an edited config under its own XDG_CONFIG_HOME, so the image runs as any user. make docker-build-hailo-sim builds the image (HAILO_SIM_PLATFORM, default the host's arch). It is not published. hack/hailo-probe.sh checks every route the agent's Ollama executor and ingress allow for the ollama runtime against a hailo-ollama server, real or simulated, and prints each status. hailo-ollama 5.1.1 answers 404 on the embedding, /health, /metrics, /v1/models, /v1/completions and /v1/responses routes; the simulator matches it. Fixes defilantech#2521 Signed-off-by: Christian Nuss <cnuss@users.noreply.github.com>
cnuss
added a commit
to cnuss/LLMKube
that referenced
this pull request
Nov 13, 2026
…ning defilantech#2482 work) This is the part of defilantech#2481 not yet split into its own PR. It sits on the four split commits for defilantech#2516, defilantech#2517, defilantech#2519 and defilantech#2521 and holds the rest of defilantech#2482. API and controller: - A hailo:// Model source names a HEF in Hailo's GenAI model zoo (api/v1alpha1/hailo_source.go); the source pattern and the generated CRDs accept it. - The Model controller marks a valid hailo:// Model Ready without downloading it (MetalAgentManaged), and fails one whose tag is malformed or whose accelerator is not metal with InvalidSource. - Runtime placement refuses a hailo:// Model on a runtime other than ollama. metal-agent: - The agent starts on Linux (experimental) as well as on a host with Metal; on Linux it logs whether a Hailo device node is present (cmd/metal-agent/hailo.go). - The Ollama executor pulls a hailo:// source's tag, refuses it unless the daemon is hailo-ollama (GET /hailo/v1/list), and sends load and unload requests without a prompt field, which hailo-ollama needs. - The agent refuses a hailo:// Model on a runtime other than ollama. - Memory admission sizes a hailo:// source with a HEAD of its HEF, found through hailo-ollama's manifests, before anything is pulled. --hailo-manifests-dir and --hailo-library-url configure it, and metal join passes both through. - Host capacity on Linux reads the board or CPU name from /proc and memory through LinuxMemoryProvider; the sysctl reader moves to the darwin file with its test. e2e and CI: - hack/e2e-host-agent.sh runs two Linux host agents, host-a and host-b, in kind against the Hailo-10H simulator: dial-back, relay chat, per-namespace isolation, admission before pull, and InvalidSource for hailo:// on another accelerator. - make test-e2e-host-agent runs it; test-e2e.yml runs it as a non-blocking job, and the gate exempts the target. Docs: - docs/site/guides/linux-host-agent.md, linked from the nav and the macOS guide: any Linux host first (building the agent, joining by hand, keeping hosts apart, engines, memory and cgroups), then Hailo-10H hosts (hailo-ollama, its systemd unit, hailo:// sources, known limits), with the Raspberry Pi 5 and AI HAT+ 2 install in a subsection of its own. - The generated CRD reference lists the hailo:// scheme. Signed-off-by: Christian Nuss <cnuss@users.noreply.github.com>
cnuss
force-pushed
the
feat/linux-host-agent
branch
from
November 13, 2026 23:22
7df0d2b to
9d6af22
Compare
This was referenced Nov 13, 2026
DetectHostCapacity on Linux reads the chip from /proc/device-tree/model with its trailing NUL stripped, else the first "model name" or "Model" line in /proc/cpuinfo, else leaves it empty. It goes through the same printable-prefix and 128-byte limit as on darwin. Memory is NewHostMemoryProvider().TotalMemory(), so a cgroup limit below MemTotal is what the capacity reports. hostCapacityFromSysctl and its test move into the darwin files, and the stub for other OSes returns an empty HostCapacity, so the sysctl parser is not dead code on Linux. The MetalNodeCapacity field comments are now host-neutral: memory is unified memory on a Mac and RAM under the agent's cgroup limit on Linux, the chip is the CPU brand string on macOS and the device-tree or cpuinfo model name on Linux, and GPU cores and the Metal version are reported on macOS only. The generated CRDs, the chart's CRD and the CRD reference page carry the new descriptions. Fixes defilantech#2518 Signed-off-by: Christian Nuss <cnuss@users.noreply.github.com>
The agent no longer exits on a host without Metal when the host is
Linux: checkHostPlatform logs "Metal not available; Linux host support
is experimental" and whether a Hailo device node (/dev/hailo*) is
present, and goes on. Any other host without Metal still exits.
llamaServerInstallHint reads the same hostGOOS probe, and the probes are
replaced in platform_test.go.
docs/site/guides/linux-host-agent.md is the first version of the Linux
host guide: what has been tested, building the agent, joining by hand
with a minted ServiceAccount token (metal join runs only on macOS),
keeping hosts apart, engines and memory. The docs nav and the macOS
guide link to it.
Review follow-ups:
- Memory eviction skips a runtime that runs outside the agent's cgroup
(runsOutsideAgentCgroup), whatever its ModelID.
- An agent whose memory controller is on cgroup v1, as on a hybrid
v1/v2 host, logs once at startup that its memory figures are the
host's.
- llmkube_metal_agent_memory_budget_bytes says it is the host-wide
budget and that in-cgroup engines are admitted against the cgroup's;
docs/site/reference/metrics.md is regenerated.
- The /tmp model-store refusal gives Linux steps (change --model-store
where the supervisor sets it) off macOS, and the launchd steps on
macOS.
- A test covers the generic RuntimeNotAvailable message, and the macOS
guide has a troubleshooting entry for RuntimeNotAvailable.
- The "starting metal agent" log line drops the binary paths, which
were empty strings when a binary was not found; the "binary found"
lines report them.
- Shared code and messages say "host" instead of "Mac": --metal-node
and --inference-service-allowlist help, the MetalNode address reasons
and comments, the allowlist comments and log line, and the expired
token message ("join the host again").
- The StateDir, InstallRoot and stop-engines --state-dir docs give the
XDG_DATA_HOME default with its ~/.local/share fallback.
- memory_procfs.go says why it has no build tag.
Fixes defilantech#2520
Signed-off-by: Christian Nuss <cnuss@users.noreply.github.com>
…(remaining defilantech#2482 work) This is the part of defilantech#2481 not yet split into its own PR: hailo:// sources (defilantech#2522) and the host-agent e2e (defilantech#2523). It sits on the split commits for defilantech#2521, defilantech#2518 and defilantech#2520. API and controller: - A hailo:// Model source names a HEF in Hailo's GenAI model zoo (api/v1alpha1/hailo_source.go); the source pattern, the generated CRDs and the CRD reference accept it. - The Model controller marks a valid hailo:// Model Ready without downloading it (MetalAgentManaged), and fails one whose tag is malformed or whose accelerator is not metal with InvalidSource. - Runtime placement refuses a hailo:// Model on a runtime other than ollama. metal-agent: - The Ollama executor pulls a hailo:// source's tag, refuses it unless the daemon is hailo-ollama (GET /hailo/v1/list), and sends load and unload requests without a prompt field, which hailo-ollama needs. - The agent refuses a hailo:// Model on a runtime other than ollama. - Memory admission sizes a hailo:// source with a HEAD of its HEF, found through hailo-ollama's manifests, before anything is pulled. --hailo-manifests-dir and --hailo-library-url configure it, and metal join passes both through. e2e and CI: - hack/e2e-host-agent.sh runs two Linux host agents, host-a and host-b, in kind against the Hailo-10H simulator: dial-back, relay chat, per-namespace isolation, admission before pull, and InvalidSource for hailo:// on another accelerator. - make test-e2e-host-agent runs it; test-e2e.yml runs it as a non-blocking job, and the gate exempts the target. Docs: the Linux host agent guide gains a Hailo-10H hosts section (installing hailo-ollama, its systemd unit, hailo:// sources, known limits) with a Raspberry Pi 5 and AI HAT+ 2 subsection, and a section on testing with the simulator. Its status table now reports MetalNode hardware for Linux hosts (defilantech#2518) and the hailo-ollama results. Signed-off-by: Christian Nuss <cnuss@users.noreply.github.com>
Releases carry static (CGO_ENABLED=0) linux/arm64 and linux/amd64 metal-agent archives, LLMKube-metal-agent_<version>_linux_<arch>.tar.gz, next to the darwin ones. They come from their own build and archive id, llmkube-metal-agent-linux, so the Homebrew cask, the Info.plist embedding, codesigning and notarization keep seeing only the darwin binaries. A path-filtered workflow, goreleaser-check.yml, runs the release job's pinned GoReleaser (v2.19.0) on PRs and pushes to main that touch the config, the agent or go.mod: goreleaser check, then a snapshot build of the linux metal-agent targets only. It publishes nothing and is not a required check. The Linux host agent guide lists the release archives and how to get the agent from one. Part of defilantech#2481. Fixes defilantech#2524. Signed-off-by: Christian Nuss <cnuss@users.noreply.github.com>
metal join refused any OS but darwin, so a Linux host agent could only be joined by hand, although redeeming the join string does not depend on the OS. On Linux, join now installs the agent as the llmkube-metal-agent.service systemd user unit, with the state directory under $XDG_DATA_HOME/llmkube/metal-agent (the agent's --state-dir default, datadir.Dir) and the unit under $XDG_CONFIG_HOME/systemd/user. It runs the same checks, credential setup, argument handling and Reachable wait as on a Mac; the plist and unit share one argument planner. Instead of a GUI session the preflight checks that the user's systemd instance answers, and join only suggests loginctl enable-linger. leave stops, disables and removes the unit join wrote, reloads systemd --user and stops the agent's engines; a unit join did not write is left running. A missing llama-server no longer stops join, on either OS, when the agent's runtime is one the agent starts without it (the rule in cmd/metal-agent resolveDefaultLlamaServerBin); join says the agent will refuse services that select llama.cpp and writes no --llama-server. With the llama.cpp runtime, or a --llama-server that is not executable, join still refuses. The join and leave help, flag help and messages shared by both OSes say "host" instead of "Mac", and name a MetalNode <name> rather than <mac>. Tests run join and leave against a fake systemctl. A live test, opted into with LLMKUBE_TEST_SYSTEMD_USER=1, installs, starts and removes the unit against the user's systemd instance, and a path-filtered workflow, test-systemd-user.yml, runs it on ubuntu-latest with lingering on. The Linux host agent guide and the generated CLI reference describe join and leave on Linux; the guide records the end-to-end join of a Raspberry Pi 5 as a systemd user unit with lingering on. Part of defilantech#2481. Fixes defilantech#2525. Signed-off-by: Christian Nuss <cnuss@users.noreply.github.com>
cnuss
force-pushed
the
feat/linux-host-agent
branch
from
November 14, 2026 00:21
9d6af22 to
61d57dc
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Caution
DO NOT MERGE. This PR is the integration branch for Linux host support (#2481). The work lands upstream through the single-commit PRs below; this one only tracks them.
What this branch is
feat/linux-host-agentis upstreammainplus every change in the Linux host-agent effort, one commit per sub-issue of #2481. Its diff againstmainis the whole remaining feature: the Linux metal-agent,hailo://sources, the Hailo-10H simulator, the host-agent e2e, linux release builds andmetal joinon Linux.How it is maintained:
mainand force-pushed (with lease) whenevermainmoves or a split PR changes. A split PR's commit here is that PR's commit, byte for byte. Where the two conflict, the PR's version wins.main, the diff is empty and this PR is closed without merging.Sub-issues and PRs
--runtimeselects another enginemainmainXDG_DATA_HOMEoff macOSmain3dd771ef4559bc6d3f6f6704hailo://Model sources through hailo-ollamad47fdf5d(remaining work)d47fdf5d439c7004metal joinand leave on Linux through a systemd user unit61d57dc0The order follows the dependencies in #2481. #2522 and #2524 come next, once #2520 merges. #2523 needs #2521 and #2522, and #2525 needs #2520.
Separately: #2514 / #2515 adds the FederatedCluster CRD that
make deployleaves out. It isn't on this branch, but without it the controller-manager restarts every couple of minutes under kustomize.Tested
At
61d57dc0,make gate BASE_REF=origin/mainandmake test-e2e-host-agentpass. The Raspberry Pi results below were taken at7df0d2b9, which had the same code before #2530's, #2518's and #2520's review changes.make gate BASE_REF=origin/mainpasses, and so doesmake test-e2e-host-agent. That runs the operator in kind and two Linux agents next to the Hailo-10H simulator. It checks dial-back, chat through the relay, memory admission before any pull, per-namespace serving, and thathailo://is refused on other accelerators.llmkube metal joinas a systemd user unit:Reachable.hailo://qwen2.5-instruct:1.5bwas admitted at 3.1 GB of a 5.3 GB budget before anything was pulled, and answered chat completions through the relay.llama3.2:3band a power cut with recovery under systemd. Details are in [FEATURE] Run the metal-agent on Linux arm64 (Raspberry Pi 5 + AI HAT+ 2) #2481.CGO_ENABLED=0Linux builds can't resolve an mDNS (*.local) API server name, because Go's built-in resolver skips nsswitch. A cgo build works. This is tracked for [Raspberry Pi] Ship the metal-agent for linux/arm64 and linux/amd64 #2524.Assisted-by: Claude Code (Claude Opus 5.5) split the work into sub-issues and PRs, rebuilt this branch, ran the checks and the Pi test, and wrote this description in my session.