NVIDIA Container Toolkit: how /dev/nvidia* gets into your container

The reason you can docker run --gpus all nvidia/cuda nvidia-smi and see your H100 — without baking the CUDA stack into the image — is the NVIDIA Container Toolkit. It is not magic and it is not a runtime; it is a chain of small tools that hooks into your existing container runtime (containerd / Docker / CRI-O / podman) and injects the host's NVIDIA driver stack into containers at start time.

This page is about understanding the chain, the modern CDI vs legacy flow, and what to do when a container won't see GPUs.

Why you need this at all

Three things have to be true for a process inside a container to use a GPU:

1. The container has access to the device files /dev/nvidia0..N, /dev/nvidiactl, /dev/nvidia-uvm, /dev/nvidia-uvm-tools, /dev/nvidia-modeset (and on systems with GDR/MIG: /dev/nvidia-caps/*).
2. The container has the userspace libraries that match the host's kernel module: libcuda.so.<driver-version>, libnvidia-ml.so.<driver-version>, libnvidia-encode.so, etc. These come from the host, not the image — because they must match the host's nvidia.ko.
3. The container's cgroups allow access to those device major/minor numbers.

You could do all this manually with --device /dev/nvidia0 --volume /usr/lib/x86_64-linux-gnu/libcuda.so.570.124.06:/usr/lib/... style hand-rolling, and the first generation of GPU containers literally did that. It was awful. The toolkit automates it.

The components

┌────────────────────────┐
│  containerd / Docker   │   "run image X"
└──────────┬─────────────┘
           │
           v
┌────────────────────────┐
│ nvidia-container-      │   shim runtime: intercepts container creation,
│ runtime                │   inserts the OCI hook
└──────────┬─────────────┘
           │
           v
┌────────────────────────┐
│ nvidia-container-      │   the actual hook: parses what GPUs the container
│ toolkit (CLI)          │   asked for, builds the device/library list
└──────────┬─────────────┘
           │
           v
┌────────────────────────┐
│ libnvidia-container    │   does the heavy lifting: queries the driver,
│ (libnvc / nvidia-      │   bind-mounts /dev/nvidia* and host libs,
│  container-cli)        │   adjusts cgroups and the OCI spec
└────────────────────────┘

Concretely:

Versions of all of these should match each other and ideally track the driver branch you have installed.

Two flows: legacy --gpus vs modern CDI

There are now two ways the toolkit gets invoked. Knowing which your stack uses is critical for debugging.

Legacy flow (still default for Docker, deprecated for K8s)

When you run docker run --gpus all <image>:

1. Docker sees --gpus all and emits an nvidia.com/gpu request to its configured runtime.
2. If nvidia-container-runtime is configured as the runtime (or as nvidia in daemon.json), it intercepts.
3. The runtime injects an OCI prestart hook that points at nvidia-container-runtime-hook.
4. Just before container start, runc invokes the hook. The hook calls libnvidia-container, which reads env vars (NVIDIA_VISIBLE_DEVICES, NVIDIA_DRIVER_CAPABILITIES) and bind-mounts the appropriate /dev/nvidia* + /usr/lib/x86_64-linux-gnu/libcuda.so.* etc into the container rootfs.

In /etc/docker/daemon.json this typically looks like:

{
  "runtimes": {
    "nvidia": {
      "path": "/usr/bin/nvidia-container-runtime",
      "runtimeArgs": []
    }
  },
  "default-runtime": "nvidia"
}

The legacy flow has worked since 2017. It is being phased out in favor of CDI because:

• It relies on env-var conventions (NVIDIA_VISIBLE_DEVICES=all) that are NVIDIA-specific.
• It modifies the container at hook time, which is invisible to anything reading the OCI spec.
• It doesn't compose well with multi-vendor accelerator setups (e.g. node has GPUs and a DPU).

Modern flow: CDI (Container Device Interface)

CDI is a CNCF-blessed standard for declaring devices in a vendor-neutral way. It works like this:

1. A vendor produces a CDI spec — a JSON/YAML file at /etc/cdi/nvidia.yaml (or /var/run/cdi/) that lists every "device" the vendor supports and describes the mounts/devices/env each one needs.
2. The container runtime (containerd 1.7+, CRI-O, podman 4+, recent Docker) reads CDI specs and, when asked to attach nvidia.com/gpu=0, simply applies the recipe from the spec to the OCI bundle.
3. No prestart hook, no runtime shim. Just a regular runc doing exactly what the spec says.

You generate the CDI spec with nvidia-ctk:

$ sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
INFO[0000] Generating CDI spec for nvidia GPUs
INFO[0000] Auto-detected mode: legacy
INFO[0000] Selecting /dev/nvidia0 as /dev/nvidia0
INFO[0000] Selecting /dev/nvidia1 as /dev/nvidia1
... (etc for all GPUs)
INFO[0000] Generated CDI spec with version 0.5.0

$ ls /etc/cdi/
nvidia.yaml

$ head -20 /etc/cdi/nvidia.yaml
cdiVersion: 0.5.0
kind: nvidia.com/gpu
devices:
- name: "0"
  containerEdits:
    deviceNodes:
    - path: /dev/nvidia0
    env:
    - NVIDIA_VISIBLE_DEVICES=0
    mounts:
    - hostPath: /usr/lib/x86_64-linux-gnu/libcuda.so.570.124.06
      containerPath: /usr/lib/x86_64-linux-gnu/libcuda.so.570.124.06
...

You re-generate this any time the driver changes. Then in containerd:

# /etc/containerd/config.toml
version = 2

[plugins."io.containerd.grpc.v1.cri".containerd]
  default_runtime_name = "runc"

[plugins."io.containerd.grpc.v1.cri".cdi]
  enable_cdi = true
  cdi_spec_dirs = ["/etc/cdi", "/var/run/cdi"]

And in your pod spec:

spec:
  containers:
  - name: cuda
    image: nvidia/cuda:12.6.0-runtime-ubuntu22.04
    resources:
      limits:
        nvidia.com/gpu: 1   # K8s device plugin still does the scheduling
    # CDI annotations supplied by the device plugin in modern flow

In K8s, the GPU Operator's device plugin (since v0.14) emits CDI annotations rather than relying on NVIDIA_VISIBLE_DEVICES. This is the recommended path on RKE2, EKS, GKE, etc.

/etc/nvidia-container-runtime/config.toml

Whether you use legacy or CDI flow, the toolkit's behavior is configured in /etc/nvidia-container-runtime/config.toml. The fields you actually care about:

disable-require = false       # if true, skip CUDA version requirements check

[nvidia-container-cli]
no-cgroups = false            # if true, don't add cgroup device rules
                              # (set to true on rootless / Talos / restricted hosts)

ldconfig = "@/sbin/ldconfig.real"  # path to ldconfig in the container
                                   # the @ means "execute from container, not host"

debug = "/var/log/nvidia-container-toolkit.log"  # turn on for triage
                                                  # gigabytes if left on long-term

user = "root:video"           # who CDI mounts are owned by inside container

[nvidia-container-runtime]
debug = "/var/log/nvidia-container-runtime.log"

[nvidia-container-runtime.modes.cdi]
default-kind = "nvidia.com/gpu"
spec-dirs = ["/etc/cdi", "/var/run/cdi"]

The most common knobs you'll touch:

• no-cgroups=true — needed when the container runtime cannot manipulate the device cgroup (rootless podman, certain hardened K8s setups, OpenShift with restricted SCC). You give up cgroup-level GPU isolation and rely on NVIDIA_VISIBLE_DEVICES only.
• debug= paths — flip on when triaging "container starts but no GPU visible". Then flip them off afterwards because the logs are huge.

CDI vs cgroup v1/v2

Cgroup v2 (default on Ubuntu 22.04+, RHEL 9+) changed how device access is controlled — instead of devices.allow files, eBPF programs are attached to cgroups. Older libnvidia-container versions don't speak BPF and fail silently when running on cgroup v2 hosts. Versions ≥1.13 handle both. Verify with:

$ nvidia-container-cli --version
cli-version: 1.16.2
lib-version: 1.16.2

< 1.10 on a cgroup v2 host = "container starts but /dev/nvidia* permission denied inside" type symptoms. Upgrade.

GPU Operator vs hand-installed

For Kubernetes you have two paths:

NVIDIA GPU Operator (recommended for K8s)

A bunch of helm-managed DaemonSets that install on every GPU node:

• nvidia-driver-daemonset — installs the driver into a container with privileged kernel-module access (no host driver needed)
• nvidia-container-toolkit-daemonset — installs the toolkit + configures the container runtime
• nvidia-device-plugin-daemonset — exposes GPUs as nvidia.com/gpu resources
• nvidia-dcgm-exporter — Prometheus metrics
• gpu-feature-discovery — labels nodes with nvidia.com/gpu.product, …count, …memory
• nvidia-mig-manager — applies MIG profiles

Pros: declarative, repeatable, includes GPU Feature Discovery labels and DCGM out of the box. Survives node re-image as long as the operator is still in the cluster.

Cons: opinionated about runtime config; can clash with custom containerd/config.toml; driver-in-container (driver.enabled=true) requires the host kernel headers to be available inside the driver container (Ubuntu HWE kernels are a frequent pain point).

In practice, on RKE2 + GPU Operator, set driver.enabled=false if you've already provisioned the driver on the host (DKMS or GitOps), then let the operator manage everything else.

Hand-installed (for bare-metal / Slurm / non-K8s)

# Ubuntu 22.04
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

apt update
apt install -y nvidia-container-toolkit

# Configure for your runtime
nvidia-ctk runtime configure --runtime=containerd
systemctl restart containerd

# Or for Docker
nvidia-ctk runtime configure --runtime=docker
systemctl restart docker

After install, generate or refresh the CDI spec:

sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

Troubleshooting

Error response from daemon: could not select device driver "nvidia" with capabilities: [[gpu]]

The container runtime doesn't know about the nvidia runtime. Either:

1. nvidia-container-toolkit is not installed.
2. It's installed but nvidia-ctk runtime configure was never run (or didn't succeed).
3. You forgot to restart the runtime (systemctl restart docker / systemctl restart containerd).

Verify:

$ docker info | grep -i runtime
 Runtimes: io.containerd.runc.v2 nvidia runc
 Default Runtime: runc

If nvidia is missing from the Runtimes list, that's your problem.

Pod starts but nvidia-smi inside says "command not found" or "no devices"

CDI is enabled in containerd but the spec is missing or stale.

# is CDI on?
$ grep -A 3 "cdi" /etc/containerd/config.toml
        enable_cdi = true
        cdi_spec_dirs = ["/etc/cdi", "/var/run/cdi"]

# does the spec exist?
$ ls /etc/cdi/
nvidia.yaml

# is it current? compare driver version
$ grep "nvidia.com/gpu" /etc/cdi/nvidia.yaml | head -1
$ nvidia-smi --query-gpu=driver_version --format=csv,noheader
570.124.06
$ grep "libcuda.so" /etc/cdi/nvidia.yaml | head -1
    - hostPath: /usr/lib/x86_64-linux-gnu/libcuda.so.570.124.06     # ← matches?

If the driver was upgraded and the CDI spec wasn't regenerated, the spec references libcuda.so.OLD_VERSION which no longer exists on the host → the bind-mount silently fails or attaches a non-existent file.

Fix: re-generate the spec on every node after a driver upgrade.

sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

The GPU Operator's nvidia-container-toolkit-daemonset does this automatically — if you're hand-managing, automate it.

"OCI runtime exec failed: exec failed: unable to start container process: error setting cgroup config: write /sys/fs/cgroup/devices.allow: invalid argument"

Cgroup v1/v2 mismatch — old libnvidia-container. Upgrade.

apt install --only-upgrade libnvidia-container1 libnvidia-container-tools nvidia-container-toolkit

Container sees GPUs but nvidia-smi shows different driver/library version

Failed to initialize NVML: Driver/library version mismatch
NVML library version: 570.124

The container is using a libnvidia-ml.so that was either baked into the image (don't do that) or got injected from a different host than the one currently running the container. Almost always: image has bundled libs that override the bind-mounts. Fix the image (use runtime flavor of nvidia/cuda not devel, or remove bundled libs explicitly).

Permission denied on /dev/nvidia0 inside container

cgroup deny. Either:

• no-cgroups = true should be set in config.toml for your environment.
• The runtime didn't get the cgroup edit applied — common with rootless podman. nvidia-ctk runtime configure --runtime=podman to fix.

Turn on debug in config.toml and look at /var/log/nvidia-container-toolkit.log — it shows the exact device list libnvc tried to inject and any cgroup write that failed.

MIG / vGPU container can't see its slice

Make sure the device plugin and the toolkit agree on which "device" is which:

$ ls /dev/nvidia-caps/
nvidia-cap1  nvidia-cap2  ...

$ kubectl get pod -o yaml my-mig-pod | grep -A 5 resources
        limits:
          nvidia.com/mig-1g.10gb: "1"

If GPU Operator's mig-manager reconfigures GPUs while pods are running, those pods will see their MIG instance vanish. The nvidia.com/mig.config node label drives this — only change MIG profile on idle nodes.

See also

• NVIDIA driver stack — the host driver that the toolkit injects from
• NVIDIA GPU Operator — the K8s-native way to manage all of this
• MIG partitioning — how GPU slicing interacts with the toolkit
• DCGM monitoring — what to scrape from the GPUs that the toolkit exposed
• Triage playbook — broader debugging flow when GPUs aren't behaving