NVIDIA GPU Operator — what it deploys, how to install it, what to validate

The NVIDIA GPU Operator turns a multi-step manual install (driver, container runtime hooks, device plugin, monitoring, feature labels) into one Helm chart. On a clean RKE2 cluster, you go from "GPUs are visible to the kernel" to "Kubernetes pods can request nvidia.com/gpu and run CUDA" in about ten minutes.

This page is the install flow, the values you'll change, and how to confirm each piece is working.

What the operator deploys

The validator's exit code is the cleanest health check: if kubectl get pod -n gpu-operator | grep validator shows Completed, the GPU stack is end-to-end functional on that node.

Driver: container vs preinstalled

Two install modes, mutually exclusive per cluster.

Driver as container (driver.enabled=true)

The operator ships a privileged DaemonSet that compiles or installs the NVIDIA driver kernel modules at pod startup. Pros: no host-level installs, easier to upgrade (change the operator chart, rollout restart), reproducible across nodes.

Cons: requires kernel-headers package available in the driver-container image for your specific kernel (or an "open-gpu-kernel-modules" build). On bleeding-edge kernels you may have to wait for a matching driver image to be published.

driver:
  enabled: true
  version: "550.90.07"
  # Optional, override registry / image
  repository: "nvcr.io/nvidia"
  image: "driver"

Driver preinstalled (driver.enabled=false)

The kernel module is on the host already (DKMS install, .run install, OS package). The operator skips driver-container deployment and reuses what's there. Pros: full control over driver version and timing, works on any kernel. Cons: you have to maintain the driver yourself across kernel upgrades.

driver:
  enabled: false
toolkit:
  env:
    - name: NVIDIA_DRIVER_ROOT
      value: "/"          # The host driver is in /, not the operator's installer path

For most production clusters the preinstalled path is the norm: drivers are baked into the OS image at build time, the kernel is pinned, and the operator handles only the userspace pieces (toolkit, device plugin, monitoring).

Helm install flow

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

# A single namespace, with the gpu-operator running in it
kubectl create namespace gpu-operator

# Install with a values file
helm install gpu-operator nvidia/gpu-operator \
  -n gpu-operator \
  --version v24.6.1 \
  -f values.yaml

Minimal values.yaml for a cluster with host-installed drivers:

driver:
  enabled: false
toolkit:
  enabled: true
  env:
    - name: CONTAINERD_CONFIG
      value: /var/lib/rancher/rke2/agent/etc/containerd/config.toml.tmpl
    - name: CONTAINERD_SOCKET
      value: /run/k3s/containerd/containerd.sock
    - name: CONTAINERD_RUNTIME_CLASS
      value: nvidia
    - name: CONTAINERD_SET_AS_DEFAULT
      value: "true"
devicePlugin:
  enabled: true
dcgmExporter:
  enabled: true
gfd:
  enabled: true
migManager:
  enabled: true
nodeStatusExporter:
  enabled: true
operator:
  defaultRuntime: containerd

The CONTAINERD_* env vars are the RKE2-specific bit. RKE2's containerd config lives at a different path than the upstream defaults; the operator's toolkit DaemonSet needs to know where to write the runtime hook.

Validation — the four checks

1. All operator pods Running, validator Completed

kubectl -n gpu-operator get pods
# NAME                                                          READY   STATUS      RESTARTS   AGE
# gpu-operator-1234567890-abcde                                 1/1     Running     0          5m
# nvidia-container-toolkit-daemonset-aaaaa                      1/1     Running     0          5m
# nvidia-dcgm-exporter-bbbbb                                    1/1     Running     0          5m
# nvidia-device-plugin-daemonset-ccccc                          1/1     Running     0          5m
# gpu-feature-discovery-ddddd                                   1/1     Running     0          5m
# nvidia-mig-manager-eeeee                                      1/1     Running     0          5m
# nvidia-operator-validator-fffff                               1/1     Running     0          5m
# nvidia-cuda-validator-99999                                   0/1     Completed   0          5m
# nvidia-device-plugin-validator-88888                          0/1     Completed   0          5m

Validator pods that are Completed (not CrashLoopBackOff) are the green light.

2. GPU resources visible to the scheduler

kubectl get nodes -o json | jq '.items[].status.capacity | {gpu: ."nvidia.com/gpu"}'
# {"gpu": "8"}
# {"gpu": "8"}
# ...

kubectl describe node gpu-01 | grep -A 10 'Capacity\|Allocatable' | head -25
# Capacity:
#   nvidia.com/gpu:   8
# Allocatable:
#   nvidia.com/gpu:   8

nvidia.com/gpu: 8 means the device plugin successfully advertised eight GPUs from this node.

3. GFD labels on the node

kubectl get nodes -o json | jq -r '.items[0].metadata.labels' | grep nvidia.com
# "nvidia.com/cuda.driver.major": "550",
# "nvidia.com/cuda.driver.minor": "90",
# "nvidia.com/gpu.compute.major": "9",
# "nvidia.com/gpu.compute.minor": "0",
# "nvidia.com/gpu.count": "8",
# "nvidia.com/gpu.family": "hopper",
# "nvidia.com/gpu.memory": "81920",
# "nvidia.com/gpu.product": "NVIDIA-H100-80GB-HBM3",
# "nvidia.com/mig.capable": "true",
# "nvidia.com/mig.strategy": "single",

These labels are what your scheduling rules will match on (nodeAffinity, nodeSelector).

4. End-to-end CUDA test pod

kubectl run cuda-vec-add --rm -it --restart=Never --image=nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda11.7.1 \
  --overrides='{"spec":{"containers":[{"name":"cuda-vec-add","image":"nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda11.7.1","resources":{"limits":{"nvidia.com/gpu":"1"}}}]}}'

# [Vector addition of 50000 elements]
# Copy input data from the host memory to the CUDA device
# CUDA kernel launch with 196 blocks of 256 threads
# Copy output data from the CUDA device to the host memory
# Test PASSED
# Done

If this exits 0, the entire stack is healthy.

DCGM and Prometheus integration

Once the DCGM exporter is up:

kubectl -n gpu-operator port-forward ds/nvidia-dcgm-exporter 9400 &
curl -s localhost:9400/metrics | grep DCGM_FI | head -10
# DCGM_FI_DEV_GPU_TEMP{gpu="0",UUID="GPU-...",pci_bus_id="00000000:17:00.0",...} 38
# DCGM_FI_DEV_POWER_USAGE{gpu="0",...} 75.342
# DCGM_FI_DEV_GPU_UTIL{gpu="0",...} 0
# DCGM_FI_DEV_MEM_COPY_UTIL{gpu="0",...} 0
# DCGM_FI_DEV_FB_USED{gpu="0",...} 0
# DCGM_FI_DEV_FB_FREE{gpu="0",...} 81559

Standard ServiceMonitor + Grafana NVIDIA dashboard (ID 12239) gives you the operator's view of fleet health.

MIG configuration

If the cluster runs MIG-capable GPUs (A100, H100, GB200), the MIG Manager reconfigures the slicing whenever a node label changes:

kubectl label node gpu-01 nvidia.com/mig.config=all-1g.10gb --overwrite
# MIG manager picks this up, drains GPUs, reconfigures, restarts the device plugin

Available profiles in kubectl describe configmap default-mig-parted-config -n gpu-operator — common ones: all-1g.10gb, all-2g.20gb, all-3g.40gb, all-7g.80gb, all-disabled, plus mixed configurations.

The reconfigure causes a brief disruption (5-30s) where pods using the GPU are evicted. Don't change MIG config under load.

Common failure modes

nvidia-driver-daemonset Init:CrashLoopBackOff with "kernel headers not found" — the driver-container image doesn't have headers for your running kernel. Either pin a kernel that's supported, switch to driver.enabled=false and install on the host, or wait for a matching driver image.

Validator stuck Init — the validator runs after every other component reports ready. If it's stuck, run kubectl logs -n gpu-operator <validator-pod> -c <init-container>. Common: toolkit-validation failing because containerd hasn't restarted with the new config; restart containerd manually.

nvidia.com/gpu shows 0 capacity — device plugin can't see the driver. Check kubectl logs -n gpu-operator nvidia-device-plugin-... — if it says "could not load NVML", the host driver is broken or the nvidia-container-toolkit hook isn't installed in containerd's config.

Pods schedule but nvidia-smi inside fails with "no devices" — the runtime class nvidia isn't being applied. Either set runtimeClassName: nvidia on the pod or set CONTAINERD_SET_AS_DEFAULT=true in the toolkit values (default since v24.x).

GPU node shows wrong driver version label — the GFD labeler caches; restart the GFD DaemonSet on that node: kubectl -n gpu-operator delete pod -l app=gpu-feature-discovery --field-selector spec.nodeName=gpu-01.

Upgrading the operator

helm upgrade gpu-operator nvidia/gpu-operator \
  -n gpu-operator \
  --version v24.9.0 \
  -f values.yaml

Component DaemonSets roll one pod at a time per node. The driver DaemonSet is the disruptive one — replacing it unloads nvidia.ko, which kills any GPU process on that node. Drain GPU workloads off a node first if you're upgrading driver versions:

kubectl drain gpu-01 --ignore-daemonsets
helm upgrade ...
# wait for the new driver pod to come up
kubectl uncordon gpu-01

For toolkit/device-plugin/DCGM-only upgrades, the disruption is shorter (seconds) and rarely worth a drain.

See also

• RKE2 — host platform; containerd path matters
• Network Operator — sister stack for RDMA/IB
• Reservations — how tenants get scheduled to GPU nodes
• ArgoCD — install via GitOps with values files
• Operations: troubleshooting — GPU-stack triage flow

External:

• docs.nvidia.com/datacenter/cloud-native/gpu-operator/
• github.com/NVIDIA/gpu-operator
• nvcr.io/nvidia/k8s — image registry