DCGM: NVIDIA's datacenter GPU monitoring stack

nvidia-smi is for humans poking at one node. DCGM (Data Center GPU Manager) is the monitoring API NVIDIA gives you for fleets — it exposes counters that nvidia-smi doesn't, has stable metric names, runs as a daemon, and ships a Prometheus exporter. If you're operating more than ~5 GPUs in production and you're not running DCGM, you are flying blind. This page is what to install, what to scrape, and what to alert on.

Why DCGM exists

nvidia-smi's output is a snapshot serialized as text. It works, but:

• It can only be invoked one node at a time. No remote access without your own SSH fan-out.
• Counters reset every time you call it; deltas need to be computed externally.
• The most operationally interesting counters — XID error history, NVLink lane errors with recovery counts, ECC remap pool state, profiler-grade SM occupancy — aren't exposed.
• Output is human-readable, not machine-stable. Field rename-related breakage hits monitoring scripts on every driver bump.

DCGM solves all four. It's a long-running daemon (nv-hostengine) that uses NVML + a profiler API to track everything the GPU exposes, with a stable field-ID API and a few CLIs / exporters on top.

Architecture

                     ┌─────────────────────────────┐
                     │       nv-hostengine         │   the daemon (one per node)
                     │  ─ NVML access              │
                     │  ─ profiler counters        │
                     │  ─ XID watcher              │
                     │  ─ field-ID API             │
                     └─────────────────────────────┘
                              │   │   │
            TCP 5555 / IPC    │   │   │
        ┌─────────────────────┘   │   └────────────────────┐
        ▼                         ▼                        ▼
   dcgmi (CLI)            dcgm-exporter            Custom apps
   diag, dmon, group       Prometheus              libdcgm.so

Modes:

• Embedded mode — daemon runs as a library inside another process (older Bright Cluster Manager, custom apps).
• Standalone mode — nv-hostengine runs as a systemd service. This is what every modern deployment uses.

Components

Install on Ubuntu:

$ apt install datacenter-gpu-manager-4
$ systemctl enable --now nvidia-dcgm
$ systemctl status nvidia-dcgm --no-pager
● nvidia-dcgm.service - NVIDIA DCGM service
     Active: active (running)

If you run the GPU Operator on Kubernetes, DCGM + dcgm-exporter come bundled — you don't install them by hand.

dcgmi quick tour

# enumerate GPUs as DCGM sees them
$ dcgmi discovery -l
1 GPU found.
+--------+----------------------------------------------------------------------+
| GPU ID | Device Information                                                  |
+========+======================================================================+
| 0      | Name: NVIDIA H100 80GB HBM3                                         |
|        | PCI Bus ID: 00000000:1B:00.0                                        |
|        | Device UUID: GPU-1234abcd-…                                          |
+--------+----------------------------------------------------------------------+

# live monitor — like `nvidia-smi dmon` but with stable fields
$ dcgmi dmon -e 100,140,150,155,156,1004,1005 -d 1000
# GPU   POWER  GPUTL  MCUTL  TMPTR  TMEM  TENSO  DRAMA
    0   145    96     78     54     54    72     61

# health snapshot of a group of GPUs
$ dcgmi health -g 0 -c
Health Monitor Report
+------------------+------------+----------------------------------+
| GPU ID           | Status     | Recommendation                  |
+==================+============+==================================+
| 0                | Healthy    |                                  |
| 1                | Healthy    |                                  |
+------------------+------------+----------------------------------+

DCGM-Diag — active health checks

dcgmi diag -r N runs synthetic load and validates outputs. Run levels:

# typical pre-job check: "does this node deserve a 24h training run"
$ dcgmi diag -r 2

Successfully ran diagnostic for group.
+---------------------------+---------------------------------------------+
| Test Name                 | Result                                      |
+===========================+=============================================+
| Deployment                | Pass                                        |
| Memory                    | Pass                                        |
| Hardware                  | Pass                                        |
| Stress                    | Pass - All                                  |
+---------------------------+---------------------------------------------+

Failures point you at a specific subsystem. A Memory test failure that complains about ECC uncorrectable events on GPU 3 → that GPU is suspect; pull it from the schedulable pool, run -r 3, decide RMA.

Field IDs you actually want to scrape

DCGM uses numeric field IDs. The full list lives in /usr/include/dcgm_fields.h — it's long. Here are the ones that pay rent in production:

The 1xxx block is profiler-grade and gives you the truth that 155 (DCGM_FI_DEV_GPU_UTIL) only approximates. For training jobs, watch 1004 SM_ACTIVE and 1006 TENSOR_ACTIVE — if both are low while util is "100%", the kernel is launch-bound or memory-bound.

XID codes — what to alert on

XID is an NVRM kernel log code. DCGM exposes the latest one as field 310; the kernel log keeps the full history. Critical XIDs operators should page on:

Reference: https://docs.nvidia.com/deploy/xid-errors/index.html.

A reasonable alerting matrix:

• Page: 48, 62, 74, 79, 95, 119, 120.
• Ticket: 43, 44, 63, 64, 69, 92, 94 — accumulate, escalate on repeat.
• Log only: 13, 31, 45 — usually app issues, not infra.

dcgm-exporter + Prometheus

dcgm-exporter is the bridge to Prometheus. It runs as a sidecar to nv-hostengine and serves /metrics.

# trimmed view of the metric output
DCGM_FI_DEV_GPU_TEMP{gpu="0",UUID="GPU-…"}        54
DCGM_FI_DEV_POWER_USAGE{gpu="0"}                  148
DCGM_FI_DEV_FB_USED{gpu="0"}                      512
DCGM_FI_PROF_SM_ACTIVE{gpu="0"}                   0.96
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE{gpu="0"}          0.71
DCGM_FI_DEV_XID_ERRORS{gpu="0"}                   0
DCGM_FI_DEV_NVLINK_REPLAY_ERROR_COUNT_TOTAL{...}  0
DCGM_FI_DEV_ECC_DBE_AGG_TOTAL{gpu="0"}            0

Customize which fields are scraped via dcgm-exporter's csv config. The default set is fine for 90% of cases; add the 1xxx profiler IDs explicitly because they're not all on by default (they have a small overhead).

A starter rule set

# Prometheus alerting — minimal viable
groups:
- name: gpu.rules
  rules:
  - alert: GpuFallenOffBus
    expr: changes(DCGM_FI_DEV_XID_ERRORS{xid="79"}[5m]) > 0
    for: 0m
    labels: { severity: page }
    annotations: { summary: "GPU {{$labels.gpu}} on {{$labels.instance}} fell off the bus (XID 79)" }

  - alert: GpuDoubleBitECC
    expr: increase(DCGM_FI_DEV_ECC_DBE_VOL_TOTAL[10m]) > 0
    labels: { severity: page }

  - alert: GpuOverTemp
    expr: DCGM_FI_DEV_GPU_TEMP > 85
    for: 5m
    labels: { severity: warn }

  - alert: GpuMemTempCritical
    expr: DCGM_FI_DEV_MEMORY_TEMP > 95
    for: 1m
    labels: { severity: page }

  - alert: GpuNvlinkReplayCountIncreasing
    expr: increase(DCGM_FI_DEV_NVLINK_REPLAY_ERROR_COUNT_TOTAL[1h]) > 0
    labels: { severity: warn }

  - alert: GpuPcieReplayIncreasing
    expr: increase(DCGM_FI_DEV_PCIE_REPLAY_COUNTER[1h]) > 5
    labels: { severity: warn }

Integration with K8s GPU Operator

The GPU Operator deploys a DaemonSet that includes:

• NVIDIA driver containerized (or relies on host driver).
• nv-hostengine running on each GPU node.
• dcgm-exporter exposed as a service on :9400.
• ServiceMonitor (if Prometheus Operator is present) to scrape it.

Verifying after install:

$ kubectl -n gpu-operator get pods -l app=nvidia-dcgm-exporter
NAME                              READY   STATUS    RESTARTS   AGE
nvidia-dcgm-exporter-abc12        1/1     Running   0          3d

$ kubectl -n gpu-operator port-forward svc/nvidia-dcgm-exporter 9400 &
$ curl -s localhost:9400/metrics | grep -E "DCGM_FI_DEV_(GPU_TEMP|POWER_USAGE)" | head

Profiling — the more advanced use

dcgmi profile runs the GPU profiler counters more aggressively than the default scrape. Useful for capacity planning ("are these GPUs really being used") but adds non-trivial overhead if all 1xxx fields are sampled at 100ms intervals on a busy GPU. Don't enable everything everywhere; pick a handful and sample at 1s+ in steady state.

$ dcgmi profile --pause
$ dcgmi profile --resume
# pausing useful when running latency-sensitive benchmarks

Common gotchas

• nv-hostengine and nvidia-fabricmanager race on Hopper/Blackwell. If DCGM starts before the fabric is up, NVLink fields will read zero/error for a few seconds. Set After=nvidia-fabricmanager.service in the DCGM unit override.
• Persistence mode interacts with DCGM. With persistence off, between-pod GPU resets clear ECC volatile counters, fragmenting your time series. Always run nvidia-persistenced or -pm 1 (see driver).
• MIG and DCGM. When MIG is enabled, you scrape per-instance metrics. The GPU-level fields are mostly NaN. Configure dcgm-exporter with the MIG-aware label so gpu= and GPU_I_ID= are both present.
• DCGM diag killed my workload. dcgmi diag -r 3 allocates GPU memory; can't run alongside another big workload. Schedule it to drain windows.

See also

• NVIDIA driver stack — what DCGM sits on top of
• GPU generations — temp/power baselines per arch
• MIG partitioning — DCGM behaviour with MIG
• Kubernetes GPU operator — bundled DCGM
• NVIDIA XID error reference: https://docs.nvidia.com/deploy/xid-errors/index.html