Grafana dashboards for GPU clusters: what to actually build

The default Grafana dashboards from the kube-prometheus-stack chart are fine for kubelet, kube-apiserver, and node-exporter generic stuff. They are useless for GPU clusters because they were not built for them. The DCGM exporter project ships a reference dashboard that is closer, but it presents one GPU at a time and that is not how operators think — when a node has eight GPUs, you want to see all eight on one screen.

This page describes the dashboard hierarchy I run on every GPU cluster, the upstream dashboards worth importing as starting points, and a handful of panel patterns (template variables, override transforms, value mappings) that turn raw DCGM/Prometheus metrics into something an operator can actually use during an incident.

Dashboard hierarchy

Six dashboards, roughly. They link to each other (Grafana data links + drilldown URLs) so you can navigate cluster → node → GPU → pod in three clicks.

Fleet overview
   ├─→ Per-cluster summary (XID heatmap, error fleet view)
   │      ├─→ Per-node detail (8-GPU mosaic)
   │      │      └─→ Per-GPU detail (single GPU deep dive)
   │      │             └─→ Per-pod GPU view
   │      └─→ Per-namespace / per-team usage
   │
   ├─→ Network fabric (IB / RoCE port state, PFC, ECN)
   │      └─→ Per-link errors
   │
   ├─→ Slurm queue & partitions
   ├─→ Kubernetes pod state
   └─→ Storage (Weka / NFS)

Data source layout

Every dashboard panel uses one of these:

Prometheus     -- DCGM, node-exporter, kube-state-metrics, cAdvisor
Loki           -- pod logs, dmesg, slurm prolog/epilog, fabric-manager log
Tempo          -- traces from control-plane services (rare; mostly inference)
Alertmanager   -- live alert annotations on time-series panels

Define them under additionalDataSources in the kube-prometheus-stack values — see prometheus-stack.

Template variables every dashboard needs

These four template variables ($cluster, $node, $gpu, $pod) appear on every GPU dashboard. They are populated from Prometheus label queries:

# Variable: cluster
Type:    Query
Source:  Prometheus
Query:   label_values(DCGM_FI_DEV_GPU_TEMP, cluster)
Sort:    alphabetical asc

# Variable: node
Type:    Query
Source:  Prometheus
Query:   label_values(DCGM_FI_DEV_GPU_TEMP{cluster="$cluster"}, Hostname)
Multi:   true
Include All:  true

# Variable: gpu
Type:    Query
Source:  Prometheus
Query:   label_values(DCGM_FI_DEV_GPU_TEMP{cluster="$cluster",Hostname=~"$node"}, gpu)
Multi:   true
Include All:  true

# Variable: pod
Type:    Query
Source:  Prometheus
Query:   label_values(DCGM_FI_DEV_GPU_TEMP{cluster="$cluster",namespace!=""}, pod)
Multi:   true
Include All:  true

The cluster label only exists if you set externalLabels on the Prometheus CR, which you should — see prometheus-stack.

Dashboard 1: per-GPU deep dive

One GPU. All the relevant counters. Used during an active incident.

Eight panels arranged in two rows of four:

The throttle-reasons panel is worth detailing. The metric is a 64-bit integer bitmask where each bit corresponds to a throttle cause:

A clean way to surface this in Grafana is a State timeline panel with one query per bit:

A: (DCGM_FI_DEV_CLOCK_THROTTLE_REASONS / 4) % 2          # HW slowdown
B: (DCGM_FI_DEV_CLOCK_THROTTLE_REASONS / 16) % 2         # SW thermal
C: (DCGM_FI_DEV_CLOCK_THROTTLE_REASONS / 32) % 2         # HW thermal
D: (DCGM_FI_DEV_CLOCK_THROTTLE_REASONS / 64) % 2         # HW power brake

with legendFormat: "{{ throttle_type }}" and override colors red for HW thermal, orange for SW thermal, yellow for power-brake.

Dashboard 2: per-node 8-GPU mosaic

The most useful dashboard during an incident. One node, all eight GPUs side by side, identical panels per GPU. The point is pattern recognition: if seven GPUs are flat at 70 C and one is at 89 C, you see it instantly.

Layout: 8 rows, 4 columns. Each row is one GPU. Columns are temp, power, SM active, FB used. PromQL uses gpu label as the row dimension:

Row template:
- variable: gpu, values: 0..7
  panels:
    - title: "GPU $gpu temp"
      query: DCGM_FI_DEV_GPU_TEMP{Hostname="$node", gpu="$gpu"}
    - title: "GPU $gpu power"
      query: DCGM_FI_DEV_POWER_USAGE{Hostname="$node", gpu="$gpu"}
    - title: "GPU $gpu SM"
      query: DCGM_FI_PROF_SM_ACTIVE{Hostname="$node", gpu="$gpu"}
    - title: "GPU $gpu FB"
      query: DCGM_FI_DEV_FB_USED{Hostname="$node", gpu="$gpu"} / 1024

Grafana does not have great support for "repeat row by variable" with consistent layout, so we typically generate this dashboard via Jsonnet (with grafonnet) or via a Python script that emits dashboard JSON. Hand-writing 32 panels is a recipe for inconsistency.

A header row at the top of the dashboard with node-level info: total power, total NVLink TX, total ECC, fabric-manager state.

# Total node power
sum(DCGM_FI_DEV_POWER_USAGE{Hostname="$node"})

# Total NVLink TX gigabits/s
sum(rate(DCGM_FI_PROF_NVLINK_TX_BYTES{Hostname="$node"}[1m])) * 8 / 1e9

# Total volatile DBE since boot
sum(DCGM_FI_DEV_ECC_DBE_VOL_TOTAL{Hostname="$node"})

# Fabric-manager state (1 = active, 0 = bad)
node_systemd_unit_state{name="nvidia-fabricmanager.service",state="active",instance=~"$node.*"}

Dashboard 3: fleet overview

This is the dashboard you put on the wall in the NOC. One row per cluster, key health indicators in stat panels with green/yellow/red thresholds.

Per cluster:

A heatmap below: GPUs (rows) × time (columns), color = max temperature in window. A clogged filter shows up as a vertical stripe across all eight GPUs in one chassis going from green to yellow to red.

max_over_time(DCGM_FI_DEV_GPU_TEMP{cluster="$cluster"}[$__interval])

with Hostname + gpu as the row dimension. With ~2000 GPUs this gets crowded; consider grouping by chassis (relabel to extract a rack label from the hostname) for the wall view, and drill-down to per-GPU on click.

Dashboard 4: NCCL traffic and IB fabric

This is the dashboard you look at when someone says "training is slow but I do not know why".

Top row — aggregate fabric throughput across the cluster:

# Total IB TX bandwidth, gigabits/s
sum by (cluster) (rate(node_infiniband_port_data_xmit_bytes_total{cluster="$cluster"}[1m])) * 8 / 1e9

# Total IB RX bandwidth, gigabits/s
sum by (cluster) (rate(node_infiniband_port_data_rcv_bytes_total{cluster="$cluster"}[1m])) * 8 / 1e9

# Total NVLink TX across the cluster
sum by (cluster) (rate(DCGM_FI_PROF_NVLINK_TX_BYTES{cluster="$cluster"}[1m])) * 8 / 1e9

Second row — fabric error rates. Use Prometheus topk to surface the worst N ports:

# Top 10 ports by symbol error rate
topk(10, rate(node_infiniband_port_xmit_wait_total{cluster="$cluster"}[5m]))

# Top 10 ports by link-down events
topk(10, increase(node_infiniband_link_downed_total{cluster="$cluster"}[1h]))

Third row — PFC pause frame rate (RoCE clusters):

sum by (instance) (rate(mlx5_priority_pause_storm_error[5m]))
sum by (instance) (rate(mlx5_priority_3_xoff[5m]))

The mlx5 metrics come from the Mellanox-provided mlx5_metrics exporter or from node_exporter's native --collector.ethtool collector if you are running a recent enough version.

Fourth row — port state heatmap. Use a state-timeline:

node_infiniband_state_id{cluster="$cluster"}

with value mappings: 1 → "Down" (red), 2 → "Initializing" (yellow), 3 → "Armed" (yellow), 4 → "Active" (green). Anything other than green for more than a few seconds is a problem.

For deeper context on what these metrics mean physically, see networking/perf-tuning, networking/ib-vs-roce, and networking/ib-architecture.

Dashboard 5: Slurm queue depth and partitions

If you run Slurm-on-Kubernetes (sunk), the slurm-exporter emits per-partition and per-state metrics. The dashboard:

# Pending jobs per partition
slurm_partition_jobs_pending

# Running jobs per partition
slurm_partition_jobs_running

# Wait time at the head of the queue (oldest pending job)
slurm_oldest_pending_job_age_seconds

# Nodes per state
slurm_nodes_idle
slurm_nodes_alloc
slurm_nodes_drain
slurm_nodes_down
slurm_nodes_fail

Per-partition stat panels for the first three, time-series for node-state counts. A "drain" or "down" count rising suddenly is the early warning of a fleet-wide issue (driver mismatch, cgroup config drift, fabric-manager flapping).

Dashboard 6: Kubernetes pod state

# Pods in CrashLoopBackOff in the last 24h
sum by (namespace) (
  changes(kube_pod_container_status_restarts_total[24h]) > 5
)

# OOMKilled in the last 1h
sum by (namespace, pod) (
  kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}
  == on (pod, namespace) (kube_pod_container_status_terminated_reason offset 1h)
)

# Pending pods older than 5m
count by (namespace) (
  kube_pod_status_phase{phase="Pending"}
  and on (pod, namespace) (
    (time() - kube_pod_created) > 300
  )
)

Important: a pending pod is not always wrong. The right alert threshold is "pending for longer than the queue-time SLO of that namespace." Don't page on a pod that is pending because the queue is full — that is the system working as intended.

Custom panel: throttle-reason flag

A useful panel to embed on the per-node dashboard: list all GPUs in the cluster currently running with non-trivial throttle reasons. Use a Table panel:

# All GPUs with thermal/power throttle currently active (any of bits 4, 5, 6)
DCGM_FI_DEV_CLOCK_THROTTLE_REASONS{cluster="$cluster"}
  unless on (Hostname, gpu)
    (DCGM_FI_DEV_CLOCK_THROTTLE_REASONS{cluster="$cluster"}
      < bool 0 + 16 + 32 + 64)
# this isn't exactly clean PromQL; we use a recording rule

In practice we use a recording rule:

- record: gpu:throttle_active:any_thermal
  expr: |
    (DCGM_FI_DEV_CLOCK_THROTTLE_REASONS / 16) % 2 == 1
    or (DCGM_FI_DEV_CLOCK_THROTTLE_REASONS / 32) % 2 == 1
    or (DCGM_FI_DEV_CLOCK_THROTTLE_REASONS / 64) % 2 == 1

then the Grafana table queries gpu:throttle_active:any_thermal == 1. Columns: cluster, Hostname, gpu, pod, value. Filter the value column to only show rows that are 1. Sort by Hostname.

We also include a "% of fleet throttling" stat next to it:

count(gpu:throttle_active:any_thermal{cluster="$cluster"} == 1)
  /
count(DCGM_FI_DEV_CLOCK_THROTTLE_REASONS{cluster="$cluster"})

with thresholds: < 1% green, < 5% yellow, > 5% red. A cluster with > 5% of GPUs throttling has a real cooling/power problem and you want to see it immediately.

Loki integration: log panels with metric correlation

A pattern we use heavily on the per-GPU dashboard: a Logs panel below each metric panel, scoped by node and by time range:

Data source:  Loki
Query:       {job="kmsg", node="$node"} |~ "Xid|nvidia|mlx5"
Time range:  Same as panel (linked)

When you click a spike on the temperature panel, the time range narrows and the logs panel shows you exactly what dmesg said in that window. This is the metrics → logs investigation loop, condensed into one screen.

The matching Loki ingestion config is in logging-loki.

Upstream dashboards worth importing

The following dashboards are reasonable starting points. Import them into Grafana, then fork them — never run upstream dashboards as-is, because the panel layouts assume a specific data shape that yours will not match exactly.

Import path: Grafana → Dashboards → Import → paste JSON or Grafana ID → select Prometheus data source.

For automation, dashboard provisioning is via Kubernetes ConfigMaps with the label grafana_dashboard: "1" (this is what the chart's sidecar discovery picks up; see the values block in prometheus-stack):

apiVersion: v1
kind: ConfigMap
metadata:
  name: gpu-fleet-overview
  namespace: monitoring
  labels:
    grafana_dashboard: "1"
data:
  fleet-overview.json: |
    {
      "title": "GPU Fleet Overview",
      "uid": "gpu-fleet",
      "panels": [...]
    }

The Grafana sidecar watches the namespace and re-imports any ConfigMap matching the label. That is how you keep dashboards in Git.

Annotations: alerts on graphs

Configure Grafana to fetch firing/resolved alerts from Alertmanager and render them as annotations on every time-series panel. You see the orange line right at the moment the GPU started throttling. Critical for post-incident review.

Dashboard settings → Annotations → New
  Name:   Alerts
  Source: Alertmanager (the data source)
  Filter: severity=~"warning|critical"

The Alertmanager data source needs to be configured in additionalDataSources. See alerting for the Alertmanager side.

Operational gotchas

instance vs Hostname

DCGM exporter labels metrics with Hostname (the in-pod hostname, which equals the K8s node name with our PodMonitor relabeling). node-exporter labels with instance (the scrape endpoint, by default node-ip:9100). They do not naturally join.

Two ways to fix this:

1. In your PodMonitor for DCGM (and your ServiceMonitor for node-exporter), relabel both to a common node label using __meta_kubernetes_pod_node_name. We do this; see prometheus-stack.
2. Alternatively, use a Prometheus join query with label_replace:

DCGM_FI_DEV_GPU_TEMP
* on (Hostname) group_left(instance)
label_replace(node_load1, "Hostname", "$1", "instance", "([^.]+).*")

The first option is cleaner.

Scrape-staleness on GPU teardowns

When a GPU pod is terminated, the per-pod labels on DCGM metrics persist in the TSDB until the next scrape interval. Range queries that group by pod will see the dead pod for up to one scrape cycle past its termination. Not a correctness issue but it confuses dashboards. The Grafana fix is interval > 1m on aggregating panels so the stale data is averaged out.

Counter resets on driver reload

Every counter in DCGM is "since driver load". Driver upgrades and nvidia-smi -r reset all counters. Prometheus's rate() and increase() handle resets correctly, but sum_over_time of a counter does not. Use idelta() for cumulative metrics across resets if you need the actual lifetime sum.

Panel cardinality

A repeat panel "by node" on a 256-node cluster creates 256 panels. Grafana renders fine but loading is slow. Use a Hostname=~"$node" template var with a single panel that gets the multi-select, and configure the panel to render one series per node.

See also

• monitoring/overview — what the dashboards exist to surface.
• monitoring/prometheus-stack — the metric source.
• monitoring/dcgm-exporter — DCGM counters that drive the GPU panels.
• monitoring/logging-loki — the logs panels described here.
• monitoring/alerting — how Alertmanager annotates these graphs.
• networking/perf-tuning — interpreting the IB fabric panels.
• networking/ib-vs-roce — RoCE-specific PFC panels.
• slurm-sunk/intro — Slurm-side metrics.
• operations/health-check-runbook — running queries during an incident.