CUDA model essentials for operators

You don't need to write CUDA kernels to operate a GPU cluster, but you do need to understand the model well enough to read a stack trace, judge a "the toolkit needs version X" claim from a developer, and figure out which of driver / toolkit / cuDNN / NCCL is actually wrong when a job fails. This page is the operator's mental model of CUDA: just enough programming-model awareness to debug things, plus the version-compatibility rules that bite in production.

The programming model in 5 minutes

CUDA exposes the GPU as a massively parallel processor with a fixed hierarchy:

Grid       (a launch — what the kernel call dispatches)
 └─ Block  (group of threads that can share memory + sync)
     └─ Warp  (32 threads, lock-step on Tensor Core / SM)
         └─ Thread  (a single execution lane)

A kernel is a function that runs on the GPU. The host (CPU) launches it with a configuration <<<grid, block>>> that says how many blocks and how many threads per block. The driver schedules blocks onto SMs (Streaming Multiprocessors) — H100 has 132 SMs, B200 has ~148 per die.

Why this matters for ops

• Block size → occupancy → real utilization. A kernel launched with too-small blocks under-utilizes the SM even when nvidia-smi reports "100% GPU util". nvidia-smi's utilization counter says "did the GPU do anything" not "did the GPU do everything it could". For real metrics use DCGM SM_ACTIVE / TENSOR_ACTIVE (see DCGM).
• Warp = 32 threads. Branch divergence within a warp serializes; profiles often blame "warp divergence" for poor throughput. You'll see this in Nsight Compute reports developers send you.
• A kernel that hangs blocks all kernels in the same stream. GPU timeouts (TDR on Windows, watchdog on Linux for display GPUs) and "kernel timed out" errors are usually one runaway kernel pinning a stream. On compute-only datacenter GPUs there is no watchdog so a runaway kernel hangs forever — visible as a process at 100% util with no progress.

Memory hierarchy — what lives where

Unified Memory (managed memory)

cudaMallocManaged allocates a single virtual address that's valid on host and device. The driver migrates pages on demand using GPU page faults — convenient for development, often a performance trap in production. A workload that "looks fine" on a single GPU can collapse on multi-GPU because pages thrash between devices over PCIe.

When operators care: a developer reports their job is 10× slower than expected. Running the workload under nvidia-smi dmon -s pu and seeing high PCIe traffic + low SM utilization → almost always managed-memory thrash. Fix is on the dev side (explicit allocations or cudaMemAdvise hints), but you'll be the one diagnosing it.

Pinned (page-locked) host memory

cudaMallocHost / cudaHostAlloc allocates host RAM that the kernel can DMA directly. Required for fastest H2D / D2H transfers. Excessive use pins all of system RAM and triggers OOM-killer events on shared boxes. If dmesg shows OOM kills from a CUDA process despite RAM looking free, suspect pinned memory exhaustion.

CUDA toolkit components

The "CUDA toolkit" you install (apt install cuda-toolkit-12-6 or the .run installer) is a bundle of:

Important separation:

• libcuda.so ships with the driver, not the toolkit. The nvidia-smi you run, the libcuda your container links — those come from the host driver.
• Everything else (libcudart, libcublas, libcudnn) ships with the toolkit / library packages and is normally bundled inside the application container.

When a container says "CUDA 12.6", it means the toolkit + libs inside the container are 12.6. The driver on the host is independent — it can be older as long as it satisfies the minimum.

Compatibility — the version game that bites operators

Three layers, each with its own compatibility rules:

your application code  (PyTorch wheel, custom kernel)
       │
       ▼
CUDA toolkit + libs    (12.4 / 12.6 / 13.0 — usually inside container)
       │  links libcuda.so dynamically
       ▼
NVIDIA driver          (535 / 550 / 570 / 580 — on the host)

Rule 1 — minimum driver per CUDA major

Each CUDA major.minor has a minimum driver version. Drivers newer than the minimum are fine.

Authoritative table: https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html.

Rule 2 — Forward Compatibility

If you're stuck on an old driver but want to run a newer CUDA toolkit, NVIDIA ships a cuda-compat-<major>-<minor> package that provides a newer libcuda.so shim. The shim talks to the older kernel module via a stable interface. Caveats:

• Only supported for same or newer minor version of the same major branch.
• Datacenter GPUs only (Tesla SKUs).
• You set LD_LIBRARY_PATH to point at the compat libs ahead of system libcuda.so.

# example: run CUDA 12.6 against a host with driver 535 (which would normally only support 12.2)
$ ls /usr/local/cuda-12.6/compat/
libcuda.so.1
libnvidia-ptxjitcompiler.so.1

$ LD_LIBRARY_PATH=/usr/local/cuda-12.6/compat:$LD_LIBRARY_PATH ./my_app

This is what container images use to insulate themselves from host driver versions. The CUDA Toolkit container images on NGC have this baked in.

Reference: https://docs.nvidia.com/deploy/cuda-compatibility/.

Rule 3 — Minor Version Compatibility (MVC)

Within a CUDA major (say 12.x), an application built with toolkit X.Y can run on libraries from toolkit X.Z where Z > Y. This means:

• Container with PyTorch built against CUDA 12.4 will run on a host with libcuda.so from driver 580 (which serves CUDA 12.8).
• You don't need to rebuild PyTorch every time the driver bumps.

Limitations: features added after X.Y won't be available; PTX-JIT may be required for some kernels (slow first launch).

Rule 4 — NCCL / cuDNN are mostly forward

NCCL versions follow the same pattern: newer NCCL works on older CUDA toolkits within the same major. cuDNN 9 needs CUDA 12+; cuDNN 8 supports CUDA 11/12. Pip-installed PyTorch wheels bundle their own NCCL and cuDNN — that's what's actually running, not whatever you apt install'ed system-wide.

# from inside a Python env, see what's actually loaded
$ python -c "import torch; print(torch.version.cuda, torch.backends.cudnn.version(), torch.cuda.nccl.version())"
12.4 90100 (2, 21, 5)

Streams, events, async — why libraries pipeline GPU work

A CUDA stream is a sequence of GPU operations that run in order. Operations on different streams can overlap. The default stream (stream 0) is special: it serializes against everything unless you opt into the per-thread default-stream model.

Why operators care:

• A multi-stream program can fully utilize a GPU even though no single kernel saturates it. Profiling tools (Nsight Systems) show this as horizontally stacked stream timelines.
• Many libraries (cuDNN, NCCL, custom kernels) take a stream argument. Mixing streams across libraries without careful event synchronization causes hard-to-reproduce bugs that look like "the model produces NaN once every 1000 iterations" — a developer issue, but you'll see the bug report.
• CUDA Graphs are stream snapshots replayed cheaply. They reduce launch overhead for small kernels (training inner loop, inference). PyTorch 2 + torch.compile and TensorRT both use graphs heavily.

# coarse-grained: see if a process is using multiple streams
$ nsys profile --stats=true python train.py
# look for ≥ 2 distinct stream rows in the timeline

Why "Driver/library version mismatch" is so common

The libcuda.so your application loaded does not match the kernel module currently in memory. Three setups produce this:

1. Apt-upgraded the driver while pods were running. Userspace was replaced; kernel module can't unload because /dev/nvidia* is open. Next process picks up the new userspace and the old kernel module. Reboot or force-unload (see driver troubleshooting).
2. Container mounted a host libcuda.so that doesn't match the host driver. Usually means the NVIDIA Container Toolkit injected one set of libs and the image had stale ones cached. ldconfig -p | grep libcuda inside the container resolves it.
3. Two different drivers installed on the host, one prevails on LD_LIBRARY_PATH. Common when someone manually cuda*.run-installed on a node that also has the apt-package driver.

Fix order: confirm host driver version (nvidia-smi), confirm container's libcuda.so (strings /usr/lib/x86_64-linux-gnu/libcuda.so.1 | grep -i version), align them.

What goes into a container image — and why operators should care

A typical CUDA container image (e.g. nvcr.io/nvidia/pytorch:24.10-py3) has:

/usr/local/cuda-12.6/    ← toolkit (cuBLAS, cuDNN, etc.)
/opt/pytorch/            ← PyTorch built against 12.6
/usr/lib/x86_64-linux-gnu/libcuda.so   ← stub or compat lib

When the container launches under the NVIDIA Container Toolkit, the host driver's libcuda.so is bind-mounted into the container at /usr/lib/x86_64-linux-gnu/libcuda.so.1, replacing the stub. This is the magic that lets container CUDA versions float independently from host driver — within compat rules.

Operator-relevant consequences:

• You don't need to install CUDA toolkit on the host. Only the driver. The toolkit lives in containers.
• Don't bake libcuda.so into your image. Ship the stub from the toolkit; let the runtime mount the host one.
• When debugging "this works on dev box but not in K8s", check that the NVIDIA Container Toolkit + runtime class are configured on the node — without them, libcuda.so is the toolkit stub that has no actual driver attached.

Quick-reference debugging commands

# what driver does the host have
$ nvidia-smi --query-gpu=driver_version --format=csv,noheader
570.124.06

# what CUDA does that driver claim to support
$ nvidia-smi | grep "CUDA Version"
| NVIDIA-SMI 570.124.06             Driver Version: 570.124.06   CUDA Version: 12.8 |

# from inside a container, what does the app see
$ python -c "import torch; print(torch.cuda.is_available(), torch.version.cuda, torch.cuda.get_device_name(0))"
True 12.4 NVIDIA H100 80GB HBM3

# enumerate CUDA devices without Python
$ /usr/local/cuda/extras/demo_suite/deviceQuery | grep "CUDA Capability"
  CUDA Capability Major/Minor version number:    9.0

# show which libcuda the app ended up with (run from inside the app container)
$ ldd $(python -c "import torch; import os; print(os.path.dirname(torch.__file__))")/lib/libtorch_cuda.so | grep libcuda
        libcuda.so.1 => /usr/lib/x86_64-linux-gnu/libcuda.so.1 (0x...)

See also

• NVIDIA driver stack — what's actually under the toolkit
• GPU generations and compute capabilities
• DCGM monitoring — real per-SM metrics
• MIG partitioning
• NCCL multi-node tuning