[cnuss]

What

Adds test/hailosim, an image that runs Hailo's own hailo-ollama 5.1.1 server against a fake libhailort that emulates a Hailo-10H. It also adds hack/hailo-probe.sh, which calls every route the agent's Ollama executor uses and the ingress allows for the ollama runtime (pkg/agent/ingress/paths.go) against a hailo-ollama server and prints each status, and a make docker-build-hailo-sim target (HAILO_SIM_PLATFORM, default the host's arch). The image is not published. This PR has no Go, workflow or docs/site changes.

Why

CI has no AI HAT+ 2. A hand-written mock would miss hailo-ollama's quirks: on a real Pi 5 it returns 404 for GET /v1/models, reports created in nanoseconds, returns 500 when the request has no Content-Type: application/json, and generates for an empty prompt. The simulator's HTTP layer is hailo-ollama's own code, so it keeps all of these. 5.1.1 is the version Raspberry Pi's apt repository ships for the AI HAT+ 2, the Hailo-10H board this was tested against. #2523 builds the host-agent e2e on top of this.

Fixes #2521

How

• The Dockerfile fetches pinned commits of hailo-ai/hailo_model_zoo_genai, the hailort public headers (sparse checkout), and oatpp/oatpp-openssl. It compiles fake_hailort.cpp into libhailort.so.5.1.1, which the CMake package config in cmake/ exposes as HailoRT::libhailort, and then builds the hailo-ollama target. Files are laid out the way hailo_gen_ai_model_zoo_5.1.1_arm64.deb lays them out.
• For each installed manifest, the build writes a fake HEF of exactly HAILO_SIM_HEF_SIZE bytes (a header line naming the manifest, then zeros), points the manifest's hef_h10h at that HEF's digest and seeds it in the blob cache, so /api/pull hits the cache and never downloads. The entrypoint starts a small HTTP server that answers HEAD /blob/sha256_<digest> the way dev-public.hailo.ai does.
• The entrypoint checks that HAILO_SIM_LISTEN and HAILO_SIM_LIBRARY_LISTEN are host:port with a port from 1 to 65535, and exits 1 naming the variable otherwise. The library server signals ready only after its own bind succeeds (no SO_REUSEPORT), and the container exits 1 if it dies first; its stderr goes to the container log. HAILO_SIM_LISTEN writes an edited copy of hailo-ollama.json under a new XDG_CONFIG_HOME, which hailo-ollama 5.1.1 searches before /etc/xdg, so the image runs as any user. The model cache is a+rwX.
• hack/hailo-probe.sh covers the executor's routes, the Ollama routes the ingress allows (/, /api/chat, /api/generate, /api/embed, /api/embeddings, /api/tags, /api/show, /api/version, /api/ps), and the routes it allows for every runtime (/health, /metrics, /v1/models, /v1/models/{id}, /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/responses). It reports statuses only and asserts nothing. It keeps curl's stderr out of the status column: with no response it shows 000 and puts curl's error where the body would go.
• Settings: HAILO_SIM_DEVICE, HAILO_SIM_DEVICE_MEMORY, HAILO_SIM_TOKEN_DELAY_MS, HAILO_SIM_LISTEN, HAILO_SIM_LIBRARY_LISTEN. They are documented in test/hailosim/README.md.

Open question from the issue: the build compiles third-party sources at pinned commits fetched from GitHub. hailo-ollama 5.1.1 itself fetches oatpp's default branch, so OATPP_REF/OATPP_OPENSSL_REF pin the current heads of those branches. If you would rather mirror these sources, I can change it.

What I ran (macOS arm64, Docker Desktop):

• make docker-build-hailo-sim built. A separate docker build --no-cache of test/hailosim also built from scratch, so the pinned fetches still resolve.
• docker run --rm -d -p 18000:8000 localhost/llmkube-hailo-sim:dev, then hack/hailo-probe.sh http://127.0.0.1:18000, and the same script against the real hailo-ollama 5.1.1 on a Raspberry Pi 5 with an AI HAT+ 2. Both gave the same status on every route. 200: /, /api/pull, /api/generate, /api/ps, /api/version, /api/tags, /api/show, /api/chat, /v1/chat/completions and /hailo/v1/list. 404 (No mapping for HTTP-method): /api/embed, /api/embeddings, /health, /metrics, /v1/models, /v1/models/{id}, /v1/completions, /v1/embeddings and /v1/responses. So hailo-ollama serves chat and generate only. /v1/chat/completions reported created in nanoseconds. /api/chat on the simulator answered hailo-sim: Say hi.
• All 5 seeded HEFs are exactly 67108864 bytes, and HEAD /blob/sha256_<digest> returns Content-Length: 67108864.
• As root and as --user 65534:65534, each with HAILO_SIM_LISTEN=0.0.0.0:9000: the server came up, and every probed route answered as above.
• HAILO_SIM_LIBRARY_LISTEN=8001 exits 1 with want host:port; HAILO_SIM_LISTEN=0.0.0.0:http exits 1 with bad port 'http'; with 127.0.0.1:8001 already taken, the container logs python's Address already in use and exits 1.
• The probe against a closed port prints 000 curl: (7) Failed to connect ... for each route.
• I checked each README setting against the running image:
  • HAILO_SIM_DEVICE=0: /api/chat returns 500 with Failed to create VDevice.
  • HAILO_SIM_DEVICE_MEMORY=1000: 500 with Failed to create LLM.
  • HAILO_SIM_TOKEN_DELAY_MS=200: a warm /api/chat takes 0.61s, against 0.008s with no delay.
  • HAILO_SIM_LISTEN=0.0.0.0:9000: the server answers on 9000.
  • HAILO_SIM_LIBRARY_LISTEN= (empty): nothing listens on 8001.
  • With the default library listener, HEAD /blob/sha256_<digest> returns 200 with the HEF's Content-Length.
  • A request body without Content-Type: application/json gets a 500 (No suitable mapper found to deserialize the request body).
• shellcheck hack/hailo-probe.sh test/hailosim/entrypoint.sh: clean.
• make gate BASE_REF=origin/main on the final commit: every step passed.
• GOOS=linux bin/golangci-lint run ./..., GOOS=darwin bin/golangci-lint run ./... and make lint-deadcode: 0 issues each.

The README now says the library server on 8001 is for the agent's --hailo-library-url size check, and that hailo-ollama itself still points at dev-public.hailo.ai.

Assisted-by: Claude Code (Claude Opus 5.5) split this out of #2482, wrote the description and ran the checks above in my session.

Checklist

• Tests added/updated
• make test passes locally
• make lint passes locally
• Commit messages follow conventional commits
• All commits are signed off (git commit -s) per DCO
• AI assistance (if any) is disclosed above, per CONTRIBUTING.md
• Documentation updated (if user-facing change)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[cnuss]

Addressed review findings in the squashed commit (25c673b):

• Library server failing silently. entrypoint.sh validates HAILO_SIM_LISTEN and HAILO_SIM_LIBRARY_LISTEN as host:port (port 1-65535) and exits 1 naming the variable otherwise. The library server signals ready only after its own bind, with no SO_REUSEPORT, and the container exits 1 if it dies first. Its stderr now goes to the container log.
• Non-root containers. HAILO_SIM_LISTEN writes the edited hailo-ollama.json under a new XDG_CONFIG_HOME, which hailo-ollama 5.1.1 searches before /etc/xdg, and the model cache is a+rwX. Checked as root and as --user 65534:65534.
• HEF size. Each fake HEF is exactly HAILO_SIM_HEF_SIZE bytes, header included.
• Probe status column. hailo-probe.sh keeps curl's stderr out of the status column: it shows 000 and puts curl's error in the detail column.

[Defilan]

The pinning holds: genai at the pinned commit fetches oatpp with fetchcontent_declare(oatpp ...), so FETCHCONTENT_SOURCE_DIR_OATPP and ..._OATPP-OPENSSL do take effect. Mirroring isn't needed for an unpublished test image.

One change: the PR body and the commit say the probe calls "every route the agent's Ollama executor and ingress use", but pkg/agent/ingress/paths.go also allows POST /api/embed and POST /api/embeddings for Ollama, and GET /health, GET /metrics, POST /v1/completions, POST /v1/embeddings and POST /v1/responses for every runtime. The probe calls none of them. Since #2523 forwards whatever the ingress allows, could you add those seven to hack/hailo-probe.sh (statuses only, no assertions)? Or, if you'd rather keep it to chat, narrow the claim in the body and commit message.

Nit: a sentence in the README saying the library server on 8001 is for the agent's --hailo-library-url size check (hailo-ollama itself still points at dev-public.hailo.ai) would save the next reader a search.

[cnuss]

Done in 3dd771e (still one commit, rebased on main).

• Probe routes. hack/hailo-probe.sh now also calls the seven you listed, plus GET /v1/models/{id}, which the ingress allows for every runtime too. Statuses only, no assertions. The simulator and the real hailo-ollama 5.1.1 on the Pi agree on every route. The new eight are all 404 on both (No mapping for HTTP-method), so hailo-ollama serves chat and generate only. Worth knowing for [Raspberry Pi] Run Linux host agents against the Hailo-10H simulator in e2e CI #2523: those routes pass the ingress and get a 404 from the engine.
• README. It now says the 8001 library server is for the agent's --hailo-library-url size check, and that hailo-ollama still points at dev-public.hailo.ai.

Commit message and description updated to match.