DCGM exporter: turning the GPU into a Prometheus target

The DCGM exporter DaemonSet, what each DCGM_FI_DEV_* metric actually measures, custom counters for XID and NVLink, integration with the GPU Operator, and the alert rules every operator should have.

▶Try the commands on this page in the command emulator — type help for the full list, or solutions for copy-paste fix recipes.

DCGM exporter is the bridge between NVIDIA's nv-hostengine (the userspace daemon that talks to the kernel driver and reads GPU telemetry) and Prometheus. It runs as a DaemonSet on every GPU node, scrapes DCGM at a configurable interval, formats the result as Prometheus text, and serves it on port 9400. Everything you see on a GPU dashboard — temperatures, power, ECC counters, NVLink throughput, XID errors — comes from DCGM exporter.

The thing nobody tells you is that the default counter list is incomplete for production GPU operations. The XID error counter is not in the default. NVLink throughput is not in the default. Several profiling counters that you absolutely want for a training cluster are off by default because they cost a small amount of GPU runtime to sample. This page covers what to enable, how to enable it, and the alert rules that consume the output.

For deployment context (where DCGM exporter sits in the operator stack) see drivers/nvidia and the NVIDIA DCGM page.

What it actually does

DCGM exporter is a thin Go process that:

  1. Reads dcgm-exporter.csv (the counter file) at startup. Each row is DCGM_FIELD_NAME, prom_type, help_string.
  2. Connects to the in-process libdcgm.so (or, in Kubernetes, talks over a Unix socket to the nvidia-dcgm DaemonSet's nv-hostengine).
  3. On every /metrics scrape, calls dcgmFieldGroupGet and emits the values for every (GPU, field) pair, with labels gpu, UUID, device, modelName, Hostname, pci_bus_id, and on Kubernetes container, namespace, pod.
  4. Returns Prometheus text format on port 9400 at /metrics.

The integration with kubernetes is via the kubernetes-device-plugin socket, which DCGM exporter queries to map GPU UUID → pod. This is how you get pod="train-foo-7d4" as a label on DCGM_FI_DEV_GPU_TEMP. The mapping is not free — it costs an extra syscall per scrape — but you cannot do per-pod GPU attribution without it.

Installing

There are three reasonable ways to install DCGM exporter, in order of operator simplicity:

1. Via GPU Operator (recommended)

If you already deploy the NVIDIA GPU Operator (and you should — see drivers/nvidia), DCGM exporter is one of the operator's components and is enabled by default:

# values.yaml for gpu-operator helm chart
dcgmExporter:
  enabled: true
  image:
    repository: nvcr.io/nvidia/k8s/dcgm-exporter
    version: 4.5.2-4.8.1-distroless
  serviceMonitor:
    enabled: false   # we manage PodMonitor ourselves
  args:
    - -f
    - /etc/dcgm-exporter/counters.csv
    - --collectors
    - /etc/dcgm-exporter/counters.csv
  config:
    name: dcgm-exporter-counters     # configMap with our custom counters.csv

The operator wires up:

  • nvidia-dcgm DaemonSet (the nv-hostengine socket provider).
  • nvidia-dcgm-exporter DaemonSet (the exporter itself, talks to nvidia-dcgm).
  • Both on every node with nvidia.com/gpu.present: true.

2. Standalone Helm chart

For clusters without GPU Operator, deploy directly:

helm repo add gpu-helm-charts https://nvidia.github.io/dcgm-exporter/helm-charts
helm repo update
helm install dcgm-exporter gpu-helm-charts/dcgm-exporter \
  --namespace monitoring \
  --create-namespace \
  --set serviceMonitor.enabled=false \
  --set extraConfigMapVolumes[0].name=counters \
  --set extraConfigMapVolumes[0].configMap.name=dcgm-counters

You will need to provide the kernel driver and nvidia-container-toolkit already installed on the host. The DCGM exporter image only ships the exporter binary and its userspace dependencies.

3. Vanilla DaemonSet YAML

For full control, hand-write the DaemonSet:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: dcgm-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: dcgm-exporter
  template:
    metadata:
      labels:
        app: dcgm-exporter
    spec:
      nodeSelector:
        nvidia.com/gpu.present: "true"
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      hostPID: true       # needed for pod attribution
      containers:
        - name: dcgm-exporter
          image: nvcr.io/nvidia/k8s/dcgm-exporter:4.5.2-4.8.1-distroless
          args:
            - -f
            - /etc/dcgm-exporter/counters.csv
            - --kubernetes
            - "true"
            - --kubernetes-gpu-id-type
            - device-name
          ports:
            - name: gpu-metrics
              containerPort: 9400
          securityContext:
            runAsNonRoot: false
            runAsUser: 0
            capabilities:
              add: [SYS_ADMIN]
          volumeMounts:
            - name: counters
              mountPath: /etc/dcgm-exporter
            - name: pod-gpu-resources
              mountPath: /var/lib/kubelet/pod-resources
              readOnly: true
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              memory: 256Mi
      volumes:
        - name: counters
          configMap:
            name: dcgm-counters
        - name: pod-gpu-resources
          hostPath:
            path: /var/lib/kubelet/pod-resources

The pod-resources mount is the link to kubelet for pod-to-GPU attribution. Without it you still get metrics, but the pod and namespace labels will be empty.

The counter list (counters.csv)

This is the file that determines what the exporter actually emits. The default ships with 35 fields. The list below is what we run in production — about 60 fields — and it is what every alert rule in this repository assumes is available.

# Format: DCGM FIELD ID, Prometheus metric type, help message
# (each row needs exactly two commas)

# Clocks
DCGM_FI_DEV_SM_CLOCK,                   gauge, SM clock frequency in MHz.
DCGM_FI_DEV_MEM_CLOCK,                  gauge, Memory clock frequency in MHz.

# Temperatures
DCGM_FI_DEV_GPU_TEMP,                   gauge, GPU temperature in C.
DCGM_FI_DEV_MEMORY_TEMP,                gauge, HBM memory temperature in C.

# Power
DCGM_FI_DEV_POWER_USAGE,                gauge, Power draw in W.
DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION,   counter, Total energy consumed in mJ.
DCGM_FI_DEV_POWER_VIOLATION,            counter, Throttle time due to power violation in us.
DCGM_FI_DEV_THERMAL_VIOLATION,          counter, Throttle time due to thermal violation in us.

# Utilization
DCGM_FI_DEV_GPU_UTIL,                   gauge, GPU utilization percent.
DCGM_FI_DEV_MEM_COPY_UTIL,              gauge, Memory copy utilization percent.

# Profiling counters (DCP)
DCGM_FI_PROF_GR_ENGINE_ACTIVE,          gauge, Graphics engine active fraction.
DCGM_FI_PROF_SM_ACTIVE,                 gauge, SM active fraction.
DCGM_FI_PROF_SM_OCCUPANCY,              gauge, SM occupancy fraction.
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE,        gauge, Tensor pipe active fraction.
DCGM_FI_PROF_DRAM_ACTIVE,               gauge, DRAM active fraction.
DCGM_FI_PROF_PIPE_FP64_ACTIVE,          gauge, FP64 pipe active fraction.
DCGM_FI_PROF_PIPE_FP32_ACTIVE,          gauge, FP32 pipe active fraction.
DCGM_FI_PROF_PIPE_FP16_ACTIVE,          gauge, FP16 pipe active fraction.
DCGM_FI_PROF_PCIE_TX_BYTES,             counter, PCIe TX bytes.
DCGM_FI_PROF_PCIE_RX_BYTES,             counter, PCIe RX bytes.
DCGM_FI_PROF_NVLINK_TX_BYTES,           counter, NVLink TX bytes.
DCGM_FI_PROF_NVLINK_RX_BYTES,           counter, NVLink RX bytes.

# Memory
DCGM_FI_DEV_FB_FREE,                    gauge, Frame buffer memory free in MiB.
DCGM_FI_DEV_FB_USED,                    gauge, Frame buffer memory used in MiB.
DCGM_FI_DEV_FB_TOTAL,                   gauge, Total frame buffer memory in MiB.
DCGM_FI_DEV_FB_RESERVED,                gauge, Reserved frame buffer memory in MiB.

# Throttle reasons (bitmask)
DCGM_FI_DEV_CLOCK_THROTTLE_REASONS,     gauge, Clock throttle reasons bitmask.

# ECC
DCGM_FI_DEV_ECC_SBE_VOL_TOTAL,          counter, Volatile single-bit ECC errors.
DCGM_FI_DEV_ECC_DBE_VOL_TOTAL,          counter, Volatile double-bit ECC errors.
DCGM_FI_DEV_ECC_SBE_AGG_TOTAL,          counter, Aggregate single-bit ECC errors.
DCGM_FI_DEV_ECC_DBE_AGG_TOTAL,          counter, Aggregate double-bit ECC errors.
DCGM_FI_DEV_RETIRED_SBE,                counter, Pages retired due to single-bit errors.
DCGM_FI_DEV_RETIRED_DBE,                counter, Pages retired due to double-bit errors.
DCGM_FI_DEV_RETIRED_PENDING,            counter, Pages pending retirement.
DCGM_FI_DEV_ROW_REMAP_FAILURE,          counter, Row remap failures.
DCGM_FI_DEV_UNCORRECTABLE_REMAPPED_ROWS, counter, Uncorrectable rows remapped.
DCGM_FI_DEV_CORRECTABLE_REMAPPED_ROWS,   counter, Correctable rows remapped.

# XID
DCGM_FI_DEV_XID_ERRORS,                 gauge, Last seen XID error code.

# NVLink
DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL,     counter, Total NVLink bandwidth in KB.
DCGM_FI_DEV_NVLINK_REPLAY_ERROR_COUNT_TOTAL,   counter, Total NVLink replay errors.
DCGM_FI_DEV_NVLINK_RECOVERY_ERROR_COUNT_TOTAL, counter, Total NVLink recovery errors.
DCGM_FI_DEV_NVLINK_CRC_FLIT_ERROR_COUNT_TOTAL, counter, Total NVLink CRC flit errors.
DCGM_FI_DEV_NVLINK_CRC_DATA_ERROR_COUNT_TOTAL, counter, Total NVLink CRC data errors.

# PCIe
DCGM_FI_DEV_PCIE_REPLAY_COUNTER,        counter, PCIe replay counter.

# Fan & system
DCGM_FI_DEV_FAN_SPEED,                  gauge, Fan speed percent.
DCGM_FI_DEV_VGPU_LICENSE_STATUS,        gauge, vGPU license status (irrelevant on bare metal).

Mount this as a ConfigMap and point the exporter at it via -f /etc/dcgm-exporter/counters.csv.

A few notes on what these mean operationally:

  • DCGM_FI_PROF_* require profiling permissions on the GPU and cost ~1–3% of compute when enabled. Worth it.
  • DCGM_FI_DEV_XID_ERRORS reports the last seen XID code as a gauge value. Zero means no XID since the last clear. To detect a new XID, you alert on changes(DCGM_FI_DEV_XID_ERRORS[5m]) > 0 and inspect the value to know which XID.
  • DCGM_FI_DEV_CLOCK_THROTTLE_REASONS is a 64-bit bitmask. Bits worth knowing: 0x1 GPU idle, 0x2 software power cap, 0x4 HW slowdown, 0x8 sync boost, 0x10 SW thermal, 0x20 HW thermal, 0x40 HW power-brake, 0x80 display clock setting. A value of 0 means no throttle.

For the full DCGM field ID list, see the NVIDIA DCGM API documentation; the field names in our CSV are the same as the symbols in dcgm_fields.h.

Per-pod attribution

Once --kubernetes true is on and /var/lib/kubelet/pod-resources is mounted, each metric carries pod, namespace, container labels:

DCGM_FI_DEV_GPU_TEMP{
  Hostname="<node>",
  UUID="GPU-abc123...",
  device="nvidia0",
  gpu="0",
  modelName="NVIDIA H100 80GB HBM3",
  pci_bus_id="00000000:01:00.0",
  pod="train-resnet50-7d4f8",
  namespace="ml-team",
  container="trainer"
} 67

This is the join key you use for everything user-facing. To get "GPU temperature per training run":

avg by (training_run) (
  DCGM_FI_DEV_GPU_TEMP
  * on (pod, namespace) group_left(label_training_run)
  kube_pod_labels{label_training_run!=""}
)

The kube_pod_labels allowlist must include training_run for this to work — see prometheus-stack.

For workloads using MIG, the gpu label becomes gpu="0" and a separate label GPU_I_ID shows the MIG instance. See nvidia/mig for what each MIG slice looks like and nvidia/gpu-sharing for how kubelet sees them.

Custom counters: tracking XID error rate per GPU

The default DCGM_FI_DEV_XID_ERRORS is a gauge of the last XID. If you want a counter ("how many XID 79 events has GPU 3 on this node had in the last hour"), DCGM does not give that to you directly. Two ways to get there:

1. Recording rule on top of the gauge. Use changes():

groups:
  - name: gpu-xid-recording
    interval: 30s
    rules:
      - record: gpu:xid_errors:rate5m
        expr: |
          changes(DCGM_FI_DEV_XID_ERRORS[5m])
      - record: gpu:xid_79_events:total
        expr: |
          (DCGM_FI_DEV_XID_ERRORS == 79)
          and ignoring(__name__) (changes(DCGM_FI_DEV_XID_ERRORS[1m]) > 0)

This is fragile because two XID 79 events back-to-back will both register as the same value and changes() will only see one.

2. Custom XID exporter that tails dmesg and counts XIDs by code per GPU:

#!/usr/bin/env bash
# /usr/local/bin/xid-exporter.sh - simple Prometheus textfile exporter
set -euo pipefail
TEXTFILE_DIR=/var/lib/node_exporter/textfile
mkdir -p "$TEXTFILE_DIR"

declare -A XID_COUNT

journalctl -k -f -o cat | while IFS= read -r line; do
  if [[ "$line" =~ NVRM:\ Xid\ \(PCI:([0-9a-f:.]+)\):\ ([0-9]+) ]]; then
    bus="${BASH_REMATCH[1]}"
    xid="${BASH_REMATCH[2]}"
    key="${bus}|${xid}"
    XID_COUNT[$key]=$((${XID_COUNT[$key]:-0} + 1))

    {
      echo "# HELP gpu_xid_total Total XID events seen in dmesg by PCI bus and code."
      echo "# TYPE gpu_xid_total counter"
      for k in "${!XID_COUNT[@]}"; do
        b="${k%|*}"; x="${k##*|}"
        echo "gpu_xid_total{pci_bus_id=\"$b\",xid=\"$x\"} ${XID_COUNT[$k]}"
      done
    } > "$TEXTFILE_DIR/xid.prom.tmp"
    mv "$TEXTFILE_DIR/xid.prom.tmp" "$TEXTFILE_DIR/xid.prom"
  fi
done

Run it as a systemd unit on every GPU node, with node-exporter's --collector.textfile --collector.textfile.directory=/var/lib/node_exporter/textfile flag set. Result is a real counter:

gpu_xid_total{pci_bus_id="00000000:01:00.0",xid="79"} 3
gpu_xid_total{pci_bus_id="00000000:01:00.0",xid="48"} 0

This is more reliable than changes() and lets you alert on rate(gpu_xid_total{xid="79"}[10m]) > 0.

Prometheus alert rules for DCGM

Now the meat. These are the alert rules every GPU cluster should have. They go into a PrometheusRule CR (operator) or rules.yml (vanilla). The full Alertmanager config is in alerting.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: gpu-dcgm-rules
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
    role: gpu-alerts
spec:
  groups:
    - name: gpu.dcgm.fatal
      interval: 30s
      rules:

        - alert: GPUFellOffTheBus
          expr: DCGM_FI_DEV_XID_ERRORS == 79
          for: 0s
          labels:
            severity: critical
            team: gpu-ops
            page: "true"
          annotations:
            summary: "GPU {{ $labels.gpu }} on {{ $labels.Hostname }} fell off the bus (XID 79)"
            description: |
              GPU {{ $labels.gpu }} (UUID {{ $labels.UUID }}) on node {{ $labels.Hostname }}
              reported XID 79: GPU has fallen off the bus. The GPU is no longer responsive.
              Affected pod: {{ $labels.namespace }}/{{ $labels.pod }}
            runbook_url: https://runbooks.internal/gpu/xid-79

        - alert: GPUDoubleBitECCError
          expr: increase(DCGM_FI_DEV_ECC_DBE_VOL_TOTAL[5m]) > 0
          for: 0s
          labels:
            severity: critical
            team: gpu-ops
            page: "true"
          annotations:
            summary: "Uncorrectable ECC error on GPU {{ $labels.gpu }} ({{ $labels.Hostname }})"
            description: |
              GPU {{ $labels.gpu }} (UUID {{ $labels.UUID }}) on {{ $labels.Hostname }}
              recorded an uncorrectable (double-bit) ECC error. Memory corruption is possible.
              Drain the node and reset the GPU. Any training output produced on this GPU
              since the last clean reset is suspect.
            runbook_url: https://runbooks.internal/gpu/ecc-dbe

        - alert: GPUXID48
          expr: DCGM_FI_DEV_XID_ERRORS == 48
          for: 0s
          labels:
            severity: critical
            team: gpu-ops
            page: "true"
          annotations:
            summary: "XID 48 (DBE memory error) on GPU {{ $labels.gpu }} ({{ $labels.Hostname }})"
            description: |
              GPU {{ $labels.gpu }} reported XID 48: double-bit ECC error.
              Equivalent to ECC_DBE_VOL increase but reported as kernel XID.
              Drain immediately.
            runbook_url: https://runbooks.internal/gpu/xid-48

        - alert: GPUXID74NVLinkError
          expr: DCGM_FI_DEV_XID_ERRORS == 74
          for: 0s
          labels:
            severity: warning
            team: gpu-ops
          annotations:
            summary: "NVLink error (XID 74) on GPU {{ $labels.gpu }} ({{ $labels.Hostname }})"
            description: |
              NVLink error reported on GPU {{ $labels.gpu }}. Check fabric-manager log,
              run nvidia-smi nvlink -e to inspect link state, and dcgmi diag -r 3 to
              run NVLink diagnostics.
            runbook_url: https://runbooks.internal/gpu/xid-74

        - alert: GPUXID64MemoryRetirement
          expr: DCGM_FI_DEV_XID_ERRORS == 64
          for: 0s
          labels:
            severity: warning
            team: gpu-ops
          annotations:
            summary: "Memory page retirement (XID 64) on GPU {{ $labels.gpu }} ({{ $labels.Hostname }})"
            description: |
              GPU {{ $labels.gpu }} retired a memory page. Pending retirements:
              {{ $value }}. GPU is degrading; schedule a power cycle.
            runbook_url: https://runbooks.internal/gpu/xid-64

    - name: gpu.dcgm.warning
      interval: 30s
      rules:

        - alert: GPUSustainedHighTemp
          expr: avg_over_time(DCGM_FI_DEV_GPU_TEMP[5m]) > 87
          for: 5m
          labels:
            severity: warning
            team: gpu-ops
          annotations:
            summary: "GPU {{ $labels.gpu }} on {{ $labels.Hostname }} sustained > 87 C"
            description: |
              GPU {{ $labels.gpu }} averaged {{ $value | printf "%.1f" }} C over 5 minutes.
              Investigate cooling: chassis fans, intake temps, power consumption,
              dust filters. Check DCGM_FI_DEV_CLOCK_THROTTLE_REASONS for thermal throttle.
            runbook_url: https://runbooks.internal/gpu/thermal

        - alert: GPUSustainedHighHBMTemp
          expr: avg_over_time(DCGM_FI_DEV_MEMORY_TEMP[5m]) > 95
          for: 5m
          labels:
            severity: warning
            team: gpu-ops
          annotations:
            summary: "GPU {{ $labels.gpu }} HBM > 95 C on {{ $labels.Hostname }}"
            description: |
              HBM memory temperature {{ $value | printf "%.1f" }} C exceeds 95 C
              average over 5m. HBM thermal limit is approached;
              memory clock will throttle imminently.

        - alert: GPUHWThermalThrottle
          # Bit 0x20 = HW slowdown thermal.
          expr: (DCGM_FI_DEV_CLOCK_THROTTLE_REASONS / 32) % 2 == 1
          for: 1m
          labels:
            severity: warning
            team: gpu-ops
          annotations:
            summary: "GPU {{ $labels.gpu }} hardware thermal throttle on {{ $labels.Hostname }}"
            description: |
              HW thermal throttle bit set on GPU {{ $labels.gpu }}. Silicon hit Tjmax.
              Drain or reduce workload.

        - alert: GPUNVLinkRecoveryErrorRate
          expr: |
            rate(DCGM_FI_DEV_NVLINK_RECOVERY_ERROR_COUNT_TOTAL[5m]) > 0
          for: 2m
          labels:
            severity: warning
            team: gpu-ops
          annotations:
            summary: "NVLink recovery errors on GPU {{ $labels.gpu }} ({{ $labels.Hostname }})"
            description: |
              NVLink is retraining links on GPU {{ $labels.gpu }}.
              Rate: {{ $value | printf "%.3f" }}/s. Inspect with:
              nvidia-smi nvlink -e
              dcgmi diag -r 3
            runbook_url: https://runbooks.internal/gpu/nvlink-recovery

        - alert: GPUNVLinkReplayErrorRateHigh
          expr: |
            rate(DCGM_FI_DEV_NVLINK_REPLAY_ERROR_COUNT_TOTAL[5m]) > 1
          for: 5m
          labels:
            severity: warning
            team: gpu-ops
          annotations:
            summary: "Sustained NVLink replay error rate on GPU {{ $labels.gpu }}"
            description: |
              GPU {{ $labels.gpu }} replay rate {{ $value | printf "%.2f" }}/s.
              CRC errors on NVLink causing retransmission. Cable or connector issue likely.

        - alert: GPUMemoryRetirementPending
          expr: DCGM_FI_DEV_RETIRED_PENDING > 0
          for: 5m
          labels:
            severity: warning
            team: gpu-ops
          annotations:
            summary: "GPU {{ $labels.gpu }} on {{ $labels.Hostname }} has pending memory retirements"
            description: |
              {{ $value }} memory pages pending retirement. GPU requires a power cycle
              to apply. Schedule a maintenance drain.

        - alert: GPUFBNearlyFull
          expr: |
            (DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL) > 0.97
          for: 10m
          labels:
            severity: info
            team: ml-platform
          annotations:
            summary: "GPU {{ $labels.gpu }} memory > 97% used on {{ $labels.Hostname }}"
            description: |
              Workload {{ $labels.namespace }}/{{ $labels.pod }} is using
              {{ $value | humanizePercentage }} of GPU memory. OOM is likely on
              spikes. Consider larger GPU or smaller batch.

    - name: gpu.dcgm.fleet
      interval: 1m
      rules:

        - alert: GPUFleetXIDStorm
          # More than 5 XID events across the fleet in 10 minutes
          expr: |
            sum(rate(gpu_xid_total[10m])) * 600 > 5
          for: 5m
          labels:
            severity: warning
            team: gpu-ops
          annotations:
            summary: "Cluster-wide XID storm: {{ $value | printf "%.1f" }} XIDs in 10m"
            description: |
              The cluster is seeing an unusually high rate of XID errors across multiple nodes.
              Likely cluster-wide cause: power event, recent driver upgrade, kernel mismatch,
              or fabric-manager issue. Investigate before silencing.

        - alert: GPUFabricManagerDown
          expr: |
            node_systemd_unit_state{name="nvidia-fabricmanager.service",state="active"} != 1
            and on (instance) (count by (instance) (DCGM_FI_DEV_GPU_TEMP) >= 8)
          for: 1m
          labels:
            severity: critical
            team: gpu-ops
            page: "true"
          annotations:
            summary: "fabric-manager down on {{ $labels.instance }}"
            description: |
              nvidia-fabricmanager.service is not active on a GPU node.
              The NVSwitch fabric is unconfigured; multi-GPU jobs will fail.
              See /drivers/fabric-manager for triage.
            runbook_url: https://runbooks.internal/gpu/fabricmanager-down

A few subtleties:

  • The for: 0s on fatal alerts (XID 79, ECC DBE) is deliberate. These are single-event alerts — by the time the metric registers, the damage is done. Don't wait.
  • The throttle-reason bit-mask alert uses integer division to extract bit 5 (0x20, hardware thermal). This is fragile — if NVIDIA changes the bit layout, your alert breaks silently. Test it after every DCGM upgrade.
  • GPUFabricManagerDown is gated by count(DCGM_FI_DEV_GPU_TEMP) >= 8. This makes sure we only fire on nodes with >= 8 GPUs (i.e., HGX/DGX). PCIe-only single-GPU nodes do not need fabric-manager and would otherwise fire constantly.

Dashboard hooks

The DCGM exporter project ships a reference Grafana dashboard at github.com/NVIDIA/dcgm-exporter/grafana/. Import it as a starting point but expect to fork it — the default panels do not surface XID errors, throttle reasons, or NVLink errors prominently. See grafana-dashboards for the panel set we run.

Operational notes

DCGM exporter and MIG

When MIG is enabled, DCGM exporter emits two sets of metrics: per-physical-GPU and per-MIG-instance. The MIG metrics carry an extra GPU_I_ID label. Some metrics (clock, temperature) are only emitted at the physical level; others (memory, profiling counters) at the instance level. Be careful with PromQL aggregation that does not split on GPU_I_ID — you can double-count.

NVML semaphore contention

DCGM and nvidia-smi both contend on the same NVML semaphore. If you have a script that runs nvidia-smi in a tight loop on the same node DCGM is sampling, you can see DCGM scrape latency spike. The fix is to either rate-limit the smi calls or migrate them to use DCGM directly via dcgmi / Python bindings.

Restart behavior

The exporter does not persist counters across restarts. On a pod restart, DCGM_FI_DEV_NVLINK_REPLAY_ERROR_COUNT_TOTAL resets to whatever value nv-hostengine reports (which is itself "since driver load"). Prometheus's rate() and increase() correctly handle counter resets, but if you build dashboards on absolute values you will see spurious drops on every redeploy.

Updating the counter list

Edit the ConfigMap, then either restart the DaemonSet or send SIGHUP to the exporter:

kubectl rollout restart -n gpu-operator daemonset/nvidia-dcgm-exporter

A rolling restart on a 200-node fleet takes ~5 minutes and produces a brief gap in metrics per node. Schedule it during low-traffic windows.

See also