NVIDIA Network Operator — RDMA, MOFED, SR-IOV, IPoIB on Kubernetes

The Network Operator is the GPU Operator's sibling for the network side of an HPC cluster: it deploys MOFED kernel drivers, SR-IOV/IPoIB CNIs, and the RDMA device plugin. It is the canonical way to get GPUDirect RDMA, RoCE, and IB networking working in Kubernetes without manually rebuilding mlx5 per kernel.

This page covers what it deploys, the install flow, when to not use it, and the single most common failure mode (kernel-version mismatch with mofed-precompiled images).

What the Network Operator deploys

You don't deploy all of these — you compose them via the NicClusterPolicy CRD based on what the cluster actually needs.

When to use it vs hand-installed OFED

The choice usually reduces to: do you trust the operator to find a matching mofed-precompiled image for your kernel? If yes, use the operator and let it handle driver updates as part of GitOps. If no, install OFED in the OS image build and tell the operator to skip the driver step.

Install flow

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

kubectl create namespace nvidia-network-operator

helm install network-operator nvidia/network-operator \
  -n nvidia-network-operator \
  --version v24.7.0 \
  -f values.yaml

Minimal values.yaml (just the operator, no policy yet):

operator:
  replicas: 1
sriovNetworkOperator:
  enabled: true
deployCR: false   # We'll create the NicClusterPolicy explicitly

Then a NicClusterPolicy says what to deploy:

apiVersion: mellanox.com/v1alpha1
kind: NicClusterPolicy
metadata:
  name: nic-cluster-policy
spec:
  ofedDriver:
    image: mofed
    repository: nvcr.io/nvidia/mellanox
    version: 24.07-0.6.1.0-0       # match your kernel — see below
    forcePrecompiled: true
  rdmaSharedDevicePlugin:
    image: k8s-rdma-shared-dev-plugin
    repository: nvcr.io/nvidia/cloud-native
    version: v1.5.1
    config: |
      {
        "configList": [
          {
            "resourceName": "rdma_shared_device_a",
            "rdmaHcaMax": 63,
            "selectors": {
              "vendors": ["15b3"],
              "deviceIDs": ["1021"]
            }
          }
        ]
      }
  secondaryNetwork:
    cniPlugins:
      image: plugins
      repository: nvcr.io/nvidia/mellanox
      version: v1.4.0
    multus:
      image: multus-cni
      repository: nvcr.io/nvidia/mellanox
      version: v3.9.3
    ipoib:
      image: ipoib-cni
      repository: nvcr.io/nvidia/mellanox
      version: v1.2.0

forcePrecompiled: true tells the operator to use the mofed-precompiled-<kernel> image rather than a runtime-compiling driver. This is where the matching pitfall happens — see below.

The mofed-precompiled kernel-mismatch pitfall

mofed-precompiled images are built per kernel version. The image tag looks like:

nvcr.io/nvidia/mellanox/mofed:24.07-0.6.1.0-0-5.15.0-101-generic-ubuntu22.04-amd64
                              ^MOFED         ^kernel              ^OS  ^arch

If your nodes run kernel 5.15.0-105-generic and only 5.15.0-101-generic is published, the driver pod's init container will fail with:

ERROR: kernel version mismatch: image=5.15.0-101 host=5.15.0-105
ERROR: precompiled MOFED not available for this kernel

Three escape routes:

1. Pin the kernel. Best practice. GPU/HPC nodes should be on a long-supported kernel that matches a published MOFED image. apt-mark hold linux-image-5.15.0-101-generic and don't auto-upgrade.

2. Switch to runtime-compiling. Set forcePrecompiled: false. The driver-container compiles mlx5 modules at first start using kernel headers in the image. Slower startup (3-5 min the first time), works on any kernel as long as headers are present.

3. Hand-install OFED on the host. Skip the operator's driver entirely:

   spec:
     ofedDriver:
       enabled: false
   

   The operator's other components (device plugin, CNIs) still work; they just don't manage the driver. Pair this with OFED installed via package or mlnxofedinstall.sh in your OS image build.

SR-IOV setup

Once the operator and sriov-network-operator are running, you create per-node policies:

apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetworkNodePolicy
metadata:
  name: sriov-policy-h100-rocev2
  namespace: nvidia-network-operator
spec:
  nodeSelector:
    node.coreweave.cloud/gpu-class: h100
  resourceName: rocev2_a
  numVfs: 8                    # 8 VFs per PF
  nicSelector:
    vendor: "15b3"
    pfNames: ["ens14f0np0"]
  deviceType: netdevice
  isRdma: true
  linkType: eth                # 'eth' for RoCE, 'ib' for InfiniBand

The operator drains the node, applies the SR-IOV config (writes sriov_numvfs), and uncordons. Each VF becomes schedulable as nvidia.com/rocev2_a.

A SriovNetwork then defines the network attachment a pod can request:

apiVersion: sriovnetwork.openshift.io/v1
kind: SriovNetwork
metadata:
  name: sriov-rocev2-net
  namespace: nvidia-network-operator
spec:
  resourceName: rocev2_a
  networkNamespace: tenant-foo
  ipam: |
    {
      "type": "whereabouts",
      "range": "10.0.0.0/16",
      "exclude": ["10.0.0.0/24"]
    }

A pod requests it via Multus annotation:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    k8s.v1.cni.cncf.io/networks: sriov-rocev2-net
spec:
  containers:
  - name: trainer
    resources:
      limits:
        nvidia.com/gpu: 1
        nvidia.com/rocev2_a: 1

The pod gets eth0 (cluster default) plus a high-bandwidth RoCE VF as a secondary interface.

RDMA shared device plugin (alternative to SR-IOV)

For workloads where each pod doesn't need an isolated VF — e.g., one tenant per node, sharing the host's PF — the rdma-shared-device-plugin advertises a single resource that multiple pods can claim:

resources:
  limits:
    rdma/rdma_shared_device_a: 1

All pods see the same RDMA device under /dev/infiniband/. Cheaper than SR-IOV but no isolation between pods.

Validation

# OFED kernel modules loaded
kubectl exec -n nvidia-network-operator -it ds/mofed-ubuntu22.04-ds -- lsmod | head
# Module                  Size  Used by
# mlx5_ib               401408  0
# ib_uverbs             184320  3 mlx5_ib,...
# mlx5_core            2105344  1 mlx5_ib

# Or from the host
lsmod | grep -E 'mlx5|ib_'

# RDMA device visible
ibv_devinfo
# hca_id: mlx5_0
#         transport:                      InfiniBand (0)
#         fw_ver:                         28.40.1000
#         node_guid:                      ...
#         port:   1
#                 state:                  PORT_ACTIVE (4)
#                 max_mtu:                4096 (5)
#                 active_mtu:             4096 (5)
#                 rate:                   400 Gb/sec (4X NDR)
#                 link_layer:             Ethernet

# nvidia_peermem loaded for GPUDirect RDMA
lsmod | grep nvidia_peermem
# nvidia_peermem        16384  0

# Pod can ping over its secondary network
kubectl exec -it <test-pod> -- ip a show net1
kubectl exec -it <test-pod> -- ibv_devinfo

If nvidia_peermem isn't loaded, GPUDirect RDMA falls back to host-staging-buffer mode and bandwidth halves. The Network Operator's MOFED build includes it; on hand-installed OFED, you load it explicitly: modprobe nvidia_peermem plus /etc/modules-load.d/nvidia_peermem.conf.

Common failure modes

mofed pod CrashLoopBackOff with "kernel version mismatch" — see the pitfall section above.

mofed pod stuck Init — the runtime-compiling path is downloading kernel headers; first start can be 3-5 min on big NICs. kubectl logs -n nvidia-network-operator <pod> -c mofed-container -f to watch progress.

SriovNetworkNodePolicy says Configured: True but no VFs visible on host — check dmesg for SR-IOV errors. Often "BIOS does not support SR-IOV" — fix in BIOS first. Also check cat /sys/class/net/<pf>/device/sriov_totalvfs is non-zero.

Pods get IPs from the SR-IOV network but can't talk — RoCE doesn't work without flow steering. Verify mlx5_core is loaded with roce_lag enabled. Or fall back to InfiniBand with linkType: ib if your fabric is IB.

ibv_devinfo shows state: INIT instead of ACTIVE — the link is down at L1. ethtool ens14f0np0 (Ethernet) or ibstat (IB) on the host. Fix the cable / switch port before troubleshooting K8s.

See also

• GPU Operator — sibling stack
• Operations: troubleshooting — RDMA/IB triage flow
• Sysctl tuning and Ulimits — MEMLOCK is required for ibv_reg_mr to succeed

External:

• docs.nvidia.com/networking/display/COKAN10/Network+Operator
• github.com/Mellanox/network-operator
• github.com/k8snetworkplumbingwg/sriov-network-operator