diff --git a/examples/ngc-pytorch-gpu/README.md b/examples/ngc-pytorch-gpu/README.md new file mode 100644 index 0000000000..e6c359f2c1 --- /dev/null +++ b/examples/ngc-pytorch-gpu/README.md @@ -0,0 +1,72 @@ +# NGC PyTorch GPU Sandbox + +Run an NGC PyTorch image from `nvcr.io` in a GPU sandbox and check that CUDA +works from Python. + +NGC images keep libraries outside the paths that the +[default policy](../../docs/how-it-works/policies/default-policy.mdx) grants, +so `import torch` fails in a default sandbox. `policy.yaml` adds those paths. + +Run these commands from `examples/ngc-pytorch-gpu`. You need a gateway whose +compute driver can attach a GPU (Docker or Podman with the NVIDIA Container +Toolkit and CDI, Kubernetes with the NVIDIA device plugin, or the MicroVM +driver with one GPU passed through), and the `openshell` CLI on the host. + +## 1. Run the check + +```shell +openshell sandbox create \ + --name torch-check \ + --gpu \ + --from nvcr.io/nvidia/pytorch:25.09-py3 \ + --policy policy.yaml \ + --no-keep --no-tty \ + -- python -c 'import torch; print(torch.__version__, torch.cuda.is_available(), torch.cuda.device_count()); x = torch.ones(1024, device="cuda"); print("sum", (x * 2).sum().item())' +``` + +The first run pulls the image, which is several gigabytes. A working sandbox +prints the PyTorch version, `True`, the GPU count, and `sum 2048.0`. +`--no-keep` deletes the sandbox when the command exits. + +To use the policy for every sandbox you create, point +`OPENSHELL_SANDBOX_POLICY` at it instead of passing `--policy`: + +```shell +export OPENSHELL_SANDBOX_POLICY="$PWD/policy.yaml" +``` + +## 2. What fails without the policy + +Run the same command without `--policy` and the import fails: + +```text +OSError: libmpi.so.40: cannot open shared object file: No such file or directory +``` + +The library exists. NGC images ship HPC-X under `/opt/hpcx`, and the loader +cache points `libmpi.so.40` there. Landlock refuses the `open()` with `EACCES`, +and glibc reports every refused candidate as "No such file or directory". +`ls -l` and `ldconfig -p` still list the file because Landlock does not govern +`stat()`, so neither command reveals the denial. To confirm a suspected path, +grant it in a copy of the policy and run the command again. + +`policy.yaml` makes these additions to the default policy: + +| Path | Access | Needed by | +|---|---|---| +| `/opt` | Read-only | HPC-X, UCX, and other NGC libraries that PyTorch loads at import. | +| `/sys` | Read-only | PyTorch's cpuinfo (`/sys/devices/system/cpu`); NCCL, NVML, and hwloc topology discovery (`/sys/bus/pci`, `/sys/class`, `/sys/devices/system/node`). | +| `/dev/shm` | Read-write | PyTorch DataLoader workers and NCCL's shared-memory transport. | + +Without `/sys`, every PyTorch run starts with +`Error in cpuinfo: failed to parse the list of possible processors in /sys/devices/system/cpu/possible`. +The policy grants the whole tree because multi-GPU NCCL reads well beyond +that one directory. + +`/proc` stays read-only. CUDA writes thread names during initialization, +and the runtime brokers those writes without general `/proc` write access. +OpenShell adds the GPU device nodes at runtime when the sandbox requests a GPU, +so the policy does not list them. + +The policy defines no network rules, so the sandbox has no outbound access. +Add network rules if your workload downloads models or datasets. diff --git a/examples/ngc-pytorch-gpu/policy.yaml b/examples/ngc-pytorch-gpu/policy.yaml new file mode 100644 index 0000000000..51ca708c91 --- /dev/null +++ b/examples/ngc-pytorch-gpu/policy.yaml @@ -0,0 +1,40 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +# Filesystem policy for NGC (nvcr.io) GPU images such as +# nvcr.io/nvidia/pytorch. It extends the default policy with the paths that +# CUDA, PyTorch, NCCL, and HPC-X need. See README.md for the failures each +# addition fixes. + +version: 1 + +filesystem_policy: + include_workdir: true + read_only: + - /bin + - /usr + - /lib + - /etc + - /proc + - /var/log + - /dev/urandom + # NGC images install HPC-X, UCX, and other libraries under /opt. The + # loader cache resolves libmpi.so.40 there, so `import torch` fails + # without it. + - /opt + # Kernel-exported topology. PyTorch's cpuinfo reads + # /sys/devices/system/cpu; NCCL, NVML, and hwloc read /sys/bus/pci, + # /sys/class, and /sys/devices/system/node to discover PCI and NVLink + # topology. + - /sys + read_write: + - /tmp + - /dev/null + # PyTorch DataLoader workers and NCCL's shared-memory transport. + - /dev/shm + +landlock: + compatibility: best_effort + +# The check needs no outbound network access. +network_policies: {}