[mtalvi]

Related ticket: APPENG-5954

Summary

GPU passthrough is opt in at both the VM and sandbox layers. This rebases the implementation onto current upstream/main and aligns sandbox GPU requests with typed profile validation and automatic migration.

• VM host devices have no hardware defaults. The HCO chart supports an explicit externalResourceProvider per device; a read-only preflight checks the HCO and node allocatable through either HCO API layout.
• GPU builds use separate openshell-gateway-gpu objects throughout, including deletion and CDI API fallback paths. Kernel headers match the image kernel, with an image-time DKMS attempt and first-boot retry.
• The per-boot device unit avoids the systemd target ordering cycle, creates both UVM nodes before CDI generation, and accepts GPUs without capability nodes.
• Sandbox GPU requests are strictly typed. GPU drift recreates the sandbox and carries its /sandbox files over. Missing labels mean GPU off, so existing CPU sandboxes survive upgrades. The shipped CUDA profile remains GPU off.
• Live validation exposed two compatibility failures: toolkit-generated supplementary groups violated the workload identity, and Landlock denied the NVIDIA device paths. The image now generates CDI without extra groups and owns its refresh; the governance policy explicitly permits the required nodes. CPU sandboxes receive no CDI devices. The guide covers multi-GPU policy paths, x86, Grace, and GitOps ownership.
• Fresh golden-disk cloning on NFS exposed a blocked CDI upload. DataSource mode now permits TCP 8443 only from CDI clone-source pods in the golden-image namespace. Registry/HTTP disk sources and VM network access keep their previous rules.

Validation

• Local full suite rerun: 2,025 passed, 3 skipped. The final GPU probe change also passed a fresh 55-test installer suite.
• GPU boot/image follow-up regressions: 20 passed; boot/preflight suite: 16 passed. Lint (including 84 checks) and catalog checks pass.
• Default CNV and gateway-image chart renders are byte-identical to main. Default SAW resources match except for the updated embedded installer code and its derived VM checksum.
• CLI 0.1.2 confirms sandbox create --gpu [COUNT].
• Fresh clone-policy follow-up: 169 chart/network-policy tests and 84 Make interface tests passed; chart/shell lint, rendered shell checks, and catalog checks pass.
• CI download follow-up: 22 focused verification/image tests passed, including transient HTTP 504 recovery, bounded failures, permanent HTTP 404 handling, and checksum rejection after retry. The official pinned ARM64 Cosign binary also downloaded successfully and passed checksum and architecture checks. The CI downloader now retries transient failures with finite time limits; checksum verification remains mandatory.
• Gitleaks scanned all eight new commits with no findings; the added-line audit contains no cluster identifiers or real credentials.

Live GPU validation

LaunchPad x86 L40S functional gates passed.

• GPU image built and pushed with the corrected boot helper using a KVM-enabled Buildah validation pod. The cluster's standard nested builder lacked KVM access. Public Cosign input was cached only after its pinned checksum was verified.
• VM Running/Ready with one host device and dedicated CPU placement. Guest driver 610.57.04, matching kernel headers, DKMS modules, successful boot services, both UVM nodes, and rootless Podman GPU access verified. No systemd ordering cycle.
• CUDA sandbox Ready with the existing workload identity and one L40S visible. A fresh CPU notebook remains Ready under the device-path policy.
• GPU on -> off -> on recreated the sandbox at both transitions and preserved its /sandbox marker. GPU-off exposed no NVIDIA device. VM restarts refreshed this cluster's virtiofs ConfigMap snapshots; GPU access and installer verification passed after restart (apply: Done).
• Offline Ollama 0.9.6 / qwen2.5:3b completed a prompt inside the sandbox; ollama ps reported 100% GPU, and guest NVML showed the Ollama process using 2,926 MiB. Sandbox egress stayed closed.
• The user confirmed WebUI login and workspace display through the SSH SOCKS browser path, with PR feat(routes): align custom web UI hosts with VM configuration #100's standard-role mapping fix. The legacy *-dashboard route still returns the pre-existing documented 503 on OpenShell 0.1.x; it is not a successful browser endpoint.
• Final checks confirm the CPU ImageStream identity/tag content, platform HCO spec, CNV 4.22.12 / KubeVirt 1.8.4, and cluster ingress spec are preserved. CNV has no Pattern tracking ID.

The corrected image was imported by immutable digest into a new versioned golden PVC. A fresh VM cloned from it passed first boot and a full restart without manual GPU fixes: exact image-helper hashes, driver/DKMS/CDI checks, actual CUDA workload execution as UID 998, a CPU notebook without GPU devices, and both installer steps Done. Offline inference passed before and after restart at 100% GPU / 2,926 MiB; its /sandbox marker survived. The canonical GPU DataSource and validation GitOps entry now select the refreshed template. The prior golden disk is retained, and the original VM stayed on the same running instance with its original root PVC.

The user requested retaining the environment for reuse. No teardown was performed. Temporary registry configuration, validation workloads, and the fork-only validation branch remain in place. Current validation requires the SSH/SOCKS path; public-internet OIDC remains a follow-up.

Route support remains separate in PR #100. The validation overlay stays on a temporary fork-only branch and is never proposed upstream.

Follow-ups: public internet OIDC with a matching issuer certificate and back-channel, sandbox UI custom hosts, portal host links, a Redis session store if edge header limits require it, and container-backend GPU support (#153).

[politerealism]

@sauagarwa — flagging something in onboard_nemoclaw()'s GPU wiring that's worth a decision before this merges.

NEMOCLAW_GPU_ENABLED/NEMOCLAW_GPU_COUNT are dead — NemoClaw never reads them

Checked against current NemoClaw main (post the runtime-provider refactor in #11014): grepped the entire source tree for NEMOCLAW_GPU_ENABLED and NEMOCLAW_GPU_COUNT — zero matches. NemoClaw's onboarding doesn't consume either variable. The doc's own "This could not be verified" caveat on this point turned out to be right to flag — it's now a confirmed gap, not just an unknown.

NemoClaw's actual onboarding-time GPU surface is CLI flags, not env vars:

• --sandbox-gpu / --no-sandbox-gpu (enable/disable — src/lib/onboard/command.ts)
• --sandbox-gpu-device <device> (pin a specific device)
• --gpu / --no-gpu are just aliases for the same enable/disable knob (resolveSandboxGpuFlagFromOptions() in sandbox-gpu-preflight.ts unifies them)

Important: there's no count-based flag at this layer at all — onboarding is enable/disable/device only, never a number. So NEMOCLAW_GPU_COUNT's multi-GPU semantic can't be "fixed" by passing an equivalent flag; the concept doesn't exist in nemoclaw onboard.

Two ways to resolve this — which one do you want?

Option A — delete the dead env vars. Every sandbox in this pattern, including the GPU one, already gets created through apply_bom.py's create_sandbox_generic(), which appends --gpu <count> directly to openshell sandbox create — confirmed live and working per this doc's own Phase 3 evidence. If nemoclaw onboard's GPU flag isn't standing in for anything the BOM-driven flow actually depends on, the env vars are just inert and should come out, along with the doc's "passed on a best-effort basis" line.

Option B — wire the real flags. Only worth it if nemoclaw onboard running its own GPU preflight/setup once (separate from per-sandbox creation) is something we actually want — e.g. validating the host is GPU-ready before any sandboxes exist. If so: replace the env vars with --sandbox-gpu (from the BOM's gpu.enabled) and drop the count entirely, since there's nowhere for it to go at this layer. Would need new test_apply_bom.py coverage either way.

I'd lean toward A since B adds behavior nobody's asked for yet, but you'd know better whether onboarding-time GPU preflight is something the pattern needs — happy to help implement whichever direction you pick.

[mtalvi]

@sauagarwa — flagging something in onboard_nemoclaw()'s GPU wiring that's worth a decision before this merges.

NEMOCLAW_GPU_ENABLED/NEMOCLAW_GPU_COUNT are dead — NemoClaw never reads them

Checked against current NemoClaw main (post the runtime-provider refactor in #11014): grepped the entire source tree for NEMOCLAW_GPU_ENABLED and NEMOCLAW_GPU_COUNT — zero matches. NemoClaw's onboarding doesn't consume either variable. The doc's own "This could not be verified" caveat on this point turned out to be right to flag — it's now a confirmed gap, not just an unknown.

NemoClaw's actual onboarding-time GPU surface is CLI flags, not env vars:

• --sandbox-gpu / --no-sandbox-gpu (enable/disable — src/lib/onboard/command.ts)
• --sandbox-gpu-device <device> (pin a specific device)
• --gpu / --no-gpu are just aliases for the same enable/disable knob (resolveSandboxGpuFlagFromOptions() in sandbox-gpu-preflight.ts unifies them)

Important: there's no count-based flag at this layer at all — onboarding is enable/disable/device only, never a number. So NEMOCLAW_GPU_COUNT's multi-GPU semantic can't be "fixed" by passing an equivalent flag; the concept doesn't exist in nemoclaw onboard.

Two ways to resolve this — which one do you want?

Option A — delete the dead env vars. Every sandbox in this pattern, including the GPU one, already gets created through apply_bom.py's create_sandbox_generic(), which appends --gpu <count> directly to openshell sandbox create — confirmed live and working per this doc's own Phase 3 evidence. If nemoclaw onboard's GPU flag isn't standing in for anything the BOM-driven flow actually depends on, the env vars are just inert and should come out, along with the doc's "passed on a best-effort basis" line.

Option B — wire the real flags. Only worth it if nemoclaw onboard running its own GPU preflight/setup once (separate from per-sandbox creation) is something we actually want — e.g. validating the host is GPU-ready before any sandboxes exist. If so: replace the env vars with --sandbox-gpu (from the BOM's gpu.enabled) and drop the count entirely, since there's nowhere for it to go at this layer. Would need new test_apply_bom.py coverage either way.

I'd lean toward A since B adds behavior nobody's asked for yet, but you'd know better whether onboarding-time GPU preflight is something the pattern needs — happy to help implement whichever direction you pick.

Good catch on NEMOCLAW_GPU_*! Yeah, totally agree-going with Option A. Since apply_bom.py already passes --gpu directly to openshell sandbox create, those env vars are dead weight. I’ll strip them out along with the best-effort docs line and keep the test coverage clean.

[Hadar301]

The GPU image build has a high-severity issue at image-builder-charts/helm/openshell-gateway-image/templates/buildconfig.yaml:252. The default package value nvidia-driver-610.57.04 does not match a package in NVIDIA's Fedora 44 repository. NVIDIA publishes nvidia-open-610.57.04 and documents installing nvidia-open. Because the package install runs with set -e, every GPU image build stops there. Please use a package name that exists in both configured architecture repositories and verify the resolved package versions. See NVIDIA's Fedora 44 package index and driver install guide.

[mtalvi]

The GPU image build has a high-severity issue at image-builder-charts/helm/openshell-gateway-image/templates/buildconfig.yaml:252. The default package value nvidia-driver-610.57.04 does not match a package in NVIDIA's Fedora 44 repository. NVIDIA publishes nvidia-open-610.57.04 and documents installing nvidia-open. Because the package install runs with set -e, every GPU image build stops there. Please use a package name that exists in both configured architecture repositories and verify the resolved package versions. See NVIDIA's Fedora 44 package index and driver install guide.

I checked the repository metadata with DNF5 for both configured architectures.
nvidia-driver-610.57.04 resolves to 3:610.57.04-1.fc44.x86_64 and
3:610.57.04-1.el10.aarch64, respectively. Both actual driver RPMs download and
match NVIDIA's metadata checksums. The pinned open-DKMS and CUDA/SMI packages also resolve to 610.57.04 in both repositories. nvidia-open is the documented
metapackage, and its RPM itself requires nvidia-driver = 3:610.57.04, alongside
the open kernel module and CUDA package that this build already requests.
Therefore the stated missing-package build failure is not reproducible; I kept
the existing defaults. See the direct RPM URLs and resolved versions above.

[Hadar301]

LGTM 🚀