[ihavespoons]

What this changes

Sibyl can now download, load and serve an Edge0 model folder, such as Edge0/Edge0-145B-q3k. llama.cpp cannot run these models. Their engine is edge0-strata, and the fork github.com/diffsec/Edge0 carries its kernels for the Radeon RX 7900 matrix cores (diffsec/Edge0#1).

The engine runs as one more llama-swap model inside the existing HIP image. There is no second container and no new unit, so the swap rules, the router, the load routes and the watchdog do not change.

• containers/llama-swap/Containerfile.hip builds the engine in a stage of its own, for gfx1100 only, at the commit in EDGE0_REF. Its base image is pinned by digest.
• models.Scan lists a folder with an Edge0 model.json and an experts.bin as one model, with engine: edge0. The folder's GGUF files are parts of it.
• llamaswap.Generate gives the model the command of edge0-launch.py. A model that spans the selected GPUs runs as a layer split, in a matrix set of its own. A model pinned to one GPU shares the pinned set, so a GGUF model on another GPU runs beside it.
• The download picker offers an Edge0 repository as one variant, the whole folder. Files keep their subfolders. Hugging Face gives a SHA-256 for LFS files only, so the download checks each small file against its Git blob ID.
• POST /models/{id}/load and the launcher refuse a load when MemAvailable is lower than the model's RAM need. On one GPU that is the expert file plus 6 GiB, which is 43.3 GiB for Edge0 145B. A span counts 65% of the expert file plus 6 GiB, which is 30.3 GiB.
• On more than one GPU the launcher adds --resident-experts, the engine's low-RAM mode: RAM keeps only the experts that no GPU holds.
• The TUI and the console show only the layout, the context size and the hidden flag for an Edge0 model. The benchmark and the optimizer answer 422.

The launcher changes two things that stop the engine:

• It removes GPU_MAX_HW_QUEUES from the engine's environment. With the image's value of 1, the first request stops with verify: timed out at layer 1.
• It removes --pcie-mode dma on more than one GPU. That option is not tested on a layer split.

EDGE0_REF is 49d5def. Before that commit, the second card of a split was given pointers to the first card's latent weights and token embedding table. A card cannot read another card's VRAM, so the first request never finished.

How it was checked

• go vet ./..., go test -race ./... and golangci-lint run ./... pass, with new tests for the scanner, the generator, the download, the Hugging Face client, the load route and the model routes.
• actionlint .github/workflows/bootc.yml passes. hadolint is not installed here, so it did not run.
• The folder variant against the real repository at commit d7de835c: 75 files, 98.13 GiB, 43.32 GiB of RAM.
• The image built on the test host with podman build, and check-libs.sh passed.
• The launcher's RAM check, by hand: with a 1 MiB expert file it refuses at 4 GiB free and passes at 8 GiB.
• On the test host, llama-swap from that image ran with the Quadlet's restrictions (read-only root, no capabilities, UID 10001) and a configuration in the generator's format:
  • The first request took 66 s with the load.
  • Edge0 145B on the PCIe 4.0 x16 card decoded at 53.2 to 66.9 tokens per second.
  • An embedding model ran on the other card at the same time, and /running showed both as ready.
  • /upstream/<model>/health and /upstream/<model>/metrics, which the watchdog reads, answered 200.
  • The unload left no engine process, and it returned 22 GB of VRAM and 41 GB of RAM.
• A span through llama-swap from the same image, layers 0 to 21 on the PCIe 3.0 x1 card and 22 to 47 on the x16 card:
  • The first request took 137 s with the load.
  • It decoded at 32.3 to 52.3 tokens per second, and it read a 6089-token prompt at 290.6 tokens per second.
  • The host used 31 GB of RAM with the model loaded and 4 GB after the unload. Both cards were full.
  • The kernel log showed no GPU error.

Not checked

• A real 98 GiB download through the picker. The download tests use a fake hub.
• The image build in CI.
• The 65% figure for a span comes from one measurement on two 24 GB cards, where 57% of the expert file stayed in RAM.
• The watchdog's stall check does not cover an Edge0 model, because the engine has no llamacpp:* counters. The health check does cover it.
• The NVFP4 folders need kernels that the fork has for CUDA only. Sibyl does not stop a download of one.

🤖 Generated with Claude Code