Prometheus on a GPU cluster: the deployment that actually scales

kube-prometheus-stack, ServiceMonitor and PodMonitor CRDs, scrape config for DCGM and node-exporter on GPU nodes, TSDB sizing, retention, federation and remote-write for multi-cluster fleets.

▶Try the commands on this page in the command emulator — type help for the full list, or solutions for copy-paste fix recipes.

Prometheus is the metrics layer on every GPU cluster I have run. It is also, by a comfortable margin, the component people configure most badly the first time. The default prometheus-community/kube-prometheus-stack chart will install fine, scrape kubelet, scrape kube-state-metrics, and feed Grafana — and then the first time you tile up DCGM exporter on 200 H100 nodes you will discover that the default 5 GiB PVC is full in three days, or that the Prometheus resource is OOMing because nobody set retention and resources.limits.memory at the same time, or that you cannot federate to your central long-term store because the scrape config drops the labels you care about.

This page is the operator's guide: vanilla vs operator, the CRDs that matter, scrape config for the GPU-specific exporters, real TSDB sizing, retention math, and federation for fleet-wide observability. The next page covers DCGM exporter specifically — see dcgm-exporter.

Vanilla Prometheus vs prometheus-operator

Two ways to run Prometheus on Kubernetes:

Vanilla — a single Deployment (or StatefulSet) running prom/prometheus, with a ConfigMap mounted at /etc/prometheus/prometheus.yml. You write the entire scrape config by hand, you reload Prometheus when the config changes (sidecar or HUP signal). Simple, transparent, and reasonable for a single small cluster.

prometheus-operator — a CRD-based abstraction. You install the operator (which watches Prometheus, ServiceMonitor, PodMonitor, PrometheusRule, Alertmanager, AlertmanagerConfig CRDs), and instead of editing a flat YAML you create CR objects. The operator generates the scrape config and reloads Prometheus.

For a multi-tenant cluster, especially one where multiple teams want to add their own scrape targets without filing a ticket, the operator is the right answer. ServiceMonitor scopes scrape config to a namespace, RBAC scopes who can create them, and you stop being the bottleneck.

We use kube-prometheus-stack (the Helm chart that bundles operator + Prometheus + Alertmanager + Grafana + node-exporter + kube-state-metrics) on every cluster. For the rest of this page, "Prometheus" means a kind: Prometheus CR managed by the operator unless I say otherwise.

ServiceMonitor and PodMonitor

These two CRDs are how you tell prometheus-operator to scrape something.

ServiceMonitor — selects a Service by labels and scrapes the endpoints behind it. Use this for anything fronted by a service: kube-state-metrics, your own apps, ingress controllers. It scrapes the pods behind the service, not the service itself.

PodMonitor — selects pods directly by labels. Use this for things that do not have a service and should not, like DaemonSet-deployed exporters where every pod is a scrape target. DCGM exporter is the canonical example.

A minimal ServiceMonitor for kube-state-metrics:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kube-state-metrics
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: kube-state-metrics
  namespaceSelector:
    matchNames:
      - kube-system
  endpoints:
    - port: http-metrics
      interval: 30s
      scrapeTimeout: 10s

The release: kube-prometheus-stack label is critical. The Prometheus CR is configured (by the chart's defaults) to only pick up ServiceMonitors with that label. If you forget it, your ServiceMonitor is silently ignored. This is the single most common confusion when onboarding a new cluster.

A PodMonitor for DCGM exporter (DaemonSet, no service):

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: dcgm-exporter
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app: nvidia-dcgm-exporter
  namespaceSelector:
    matchNames:
      - gpu-operator
  podMetricsEndpoints:
    - port: gpu-metrics
      interval: 15s
      scrapeTimeout: 10s
      honorLabels: true
      relabelings:
        - sourceLabels: [__meta_kubernetes_pod_node_name]
          action: replace
          targetLabel: instance
        - sourceLabels: [__meta_kubernetes_pod_node_name]
          action: replace
          targetLabel: node

Two relabel rules worth understanding here:

  1. __meta_kubernetes_pod_node_name → instance — by default, instance is set to the pod IP. That is useless on a DaemonSet because the IP changes on pod restart. You want instance to be the node name so dashboards stay stable.
  2. honorLabels: true — DCGM exporter sets Hostname, gpu, UUID, device labels on every metric. We want those preserved, not relabeled away.

For the equivalent in vanilla Prometheus scrape config, see networking/nccl which uses a similar pattern for the NCCL exporter.

The Prometheus CR

The Prometheus resource itself controls the actual server. The fields that matter on a GPU cluster:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: kube-prometheus-stack-prometheus
  namespace: monitoring
spec:
  replicas: 2
  shards: 1
  retention: 30d
  retentionSize: 800GiB
  resources:
    requests:
      cpu: "4"
      memory: "32Gi"
    limits:
      memory: "48Gi"
  storage:
    volumeClaimTemplate:
      spec:
        storageClassName: local-nvme
        accessModes: [ReadWriteOnce]
        resources:
          requests:
            storage: 1000Gi
  serviceMonitorSelector:
    matchLabels:
      release: kube-prometheus-stack
  serviceMonitorNamespaceSelector: {}
  podMonitorSelector:
    matchLabels:
      release: kube-prometheus-stack
  podMonitorNamespaceSelector: {}
  ruleSelector:
    matchLabels:
      release: kube-prometheus-stack
  ruleNamespaceSelector: {}
  externalLabels:
    cluster: gpu-cluster-1
    region: dc-1
  remoteWrite:
    - url: https://mimir.observability.internal/api/v1/push
      writeRelabelConfigs:
        - sourceLabels: [__name__]
          regex: 'DCGM_FI_DEV_(GPU_TEMP|POWER_USAGE|XID_ERRORS|ECC_DBE_VOL_TOTAL|FB_USED|NVLINK_RECOVERY_ERROR_COUNT_TOTAL)'
          action: keep
      queueConfig:
        capacity: 10000
        maxSamplesPerSend: 2000
        maxShards: 30
  alerting:
    alertmanagers:
      - namespace: monitoring
        name: alertmanager-operated
        port: web

A few things in there are worth dwelling on.

replicas: 2 — two Prometheus pods, both scraping the same targets. Not a HA pair in the strict sense (Prometheus has no leader election; both write independently to local TSDB) but enough to survive one pod crashing or one PVC being slow. Alertmanager is what dedupes alerts.

shards: 1 — leave at 1 unless you are above ~5M active series. Sharding splits scrape targets across N Prometheus instances by hash. Once you turn it on, you also need a query layer (Thanos query, Mimir, Cortex) to fan out queries.

retention: 30d and retentionSize: 800GiB — both. Whichever triggers first wins. On a busy cluster, time-based retention keeps you from running out of disk during a metric explosion (someone deploys an app that exports millions of series); size-based keeps you sane on the upper bound.

storage — local NVMe is the right choice. TSDB is mmap-heavy and you want low-latency local IO. Network-attached block storage works but you will feel it on long range queries. Never put TSDB on NFS.

resources.requests.memory — Prometheus's RSS roughly equals (active series × ~3 KiB) + (head-block size in memory) + queries in flight. For 850k series, ~12 GiB working set is normal. Set request below that to leave headroom for kubelet but limit well above so OOMKill is rare. Limit and request must not be equal if your cluster has the bin-packing scheduler, you want some slack.

externalLabels — these get added to every metric on remote-write. This is how a multi-cluster Mimir/Thanos can tell which cluster a metric came from. Without them, federated dashboards become useless.

remoteWrite.writeRelabelConfigs — only push the metrics that matter. The full DCGM exporter output is ~80 metrics per GPU. For long-term retention you almost certainly do not need all of them. Drop the noisy ones at the remote-write boundary, not at the scrape boundary, so you keep them locally for short-term debugging.

TSDB sizing in practice

Prometheus's storage is straightforward but the math gets people. The reference formula from the Prometheus docs is:

needed_disk_space = retention_time_seconds * ingested_samples_per_second * bytes_per_sample

With Prometheus's compression, bytes_per_sample is typically 1.3 to 2 bytes. For a 256-node H100 cluster:

  • Active series: ~850k (rough estimate from the overview page).
  • Default scrape interval: 30 s, but DCGM exporter at 15 s.
  • Effective ingestion: ~40k samples/s.
  • 30-day retention: 30 * 86400 * 40000 * 1.5 bytes ≈ 155 GiB of compressed blocks.
  • Plus WAL: ~5–10 GiB depending on compaction state.
  • Plus head block: ~3 GiB in memory, mirrored to disk.
  • Round up for safety: provision 300 GiB for 30 days.

For a 1024-node cluster, roughly 4× the series, the math gives ~600 GiB. We provision 1 TiB and set retentionSize: 800GiB so we have a hard ceiling that triggers before the PVC actually fills.

The block layout on disk:

/prometheus
├── 01HQXR5T9P2N3K4M5Y6Z7A8B9C/   # 2-hour or compacted block
│   ├── chunks/
│   │   ├── 000001                # up to 512 MiB chunk segment
│   │   └── 000002
│   ├── index                     # series → chunk offsets
│   ├── meta.json
│   └── tombstones                # deletion markers
├── 01HQXR5T9P2N3K4M5Y6Z7A8B9D/
└── wal/                          # write-ahead log, 128 MiB segments
    ├── 00000123
    ├── 00000124
    └── checkpoint.000122/

Initial 2-hour blocks compact up over time into longer blocks, up to ~10% of the configured retention or 31 days, whichever is smaller. On 30-day retention the largest block on disk is roughly 3 days. On 90-day retention it is 9 days. This matters because compaction is IO-heavy and you can see latency spikes on queries during a big compaction — keep an eye on prometheus_tsdb_compaction_duration_seconds.

Scrape configs for GPU cluster targets

Here is a set of ServiceMonitor / PodMonitor objects that cover the bulk of what a GPU cluster needs scraped, beyond the chart defaults.

DCGM exporter

Already shown above. The DaemonSet is deployed by the GPU Operator under gpu-operator/nvidia-dcgm-exporter. See dcgm-exporter for the full counter list and custom-counters config.

Node exporter (for IB / NIC counters)

The chart's bundled node-exporter on a GPU node will pick up the InfiniBand counters from /sys/class/infiniband/*/ports/*/counters/* automatically when the infiniband collector is enabled. Make sure it is:

prometheus-node-exporter:
  extraArgs:
    - --collector.infiniband
    - --collector.systemd
    - --collector.systemd.unit-include=^(nvidia-fabricmanager|nvidia-persistenced|kubelet|containerd|chrony|sshd)\.service$
    - --collector.processes
    - --collector.nvme
    - --no-collector.fibrechannel
    - --no-collector.ipvs

The systemd unit-include pattern is important. If you let node-exporter scrape every systemd unit, you blow up cardinality with one series per unit per state. Restrict to the dozen units you actually care about.

Fabric-manager (via systemd collector)

No extra exporter needed — node-exporter's systemd collector exposes:

node_systemd_unit_state{name="nvidia-fabricmanager.service",state="active"} 1
node_systemd_unit_state{name="nvidia-fabricmanager.service",state="failed"} 0

The alert rule lives in alerting.

Kubelet (cAdvisor metrics)

kubelet exposes container metrics on its own /metrics/cadvisor endpoint, scraped by default by kube-prometheus-stack. The metrics that matter for GPU pods are:

  • container_cpu_usage_seconds_total
  • container_memory_working_set_bytes
  • container_network_receive_bytes_total / transmit_bytes_total
  • container_fs_reads_bytes_total / writes_bytes_total

There is no GPU information from cAdvisor. cAdvisor does not see CUDA contexts or GPU memory. The only source for per-pod GPU metrics is DCGM exporter joined to kube-state-metrics on pod and namespace labels — there is a recipe in dcgm-exporter.

Slurm exporter

If you run Slurm-on-Kubernetes (sunk — see slurm-sunk/intro), there is a slurm-exporter from the same operator that exposes per-partition queue depth, per-node state, and per-account utilization. ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: slurm-exporter
  namespace: slurm
  labels:
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: slurm-exporter
  endpoints:
    - port: metrics
      interval: 30s

Weka exporter

Weka ships its own Prometheus exporter, deployed as a sidecar inside one of the cluster pods (or standalone). Scrape it with a static additionalScrapeConfigs block (since the endpoint is outside Kubernetes):

additionalScrapeConfigs:
  - job_name: weka
    scrape_interval: 30s
    static_configs:
      - targets:
          - weka-exporter.storage.internal:8080
        labels:
          cluster: gpu-cluster-1
          storage: weka

additionalScrapeConfigs goes into the Prometheus CR's spec field of the same name; it is a Secret reference under the hood. See the Weka overview for what each Weka metric means.

Federation and multi-cluster

There are two fundamentally different patterns for "I have N clusters and want to see them in one place".

Federation (/federate)

A central Prometheus periodically scrapes a curated subset of metrics from each cluster's Prometheus via /federate?match[]=.... Cheap to set up, terrible at scale beyond ~5 clusters because:

  • The central Prometheus pulls everything matched on every scrape interval, full samples.
  • Range queries on the central instance are accurate only for the federated subset.
  • Loss of network connectivity between cluster and center loses data — federation is not durable.

Reasonable use case: a fleet view dashboard that needs only cluster_*_up, node_count, gpu_count, gpu_temperature_max aggregates. Not a long-term retention strategy.

- job_name: federate-cluster-1
  scrape_interval: 30s
  honor_labels: true
  metrics_path: /federate
  params:
    match[]:
      - '{__name__=~"DCGM_FI_DEV_(XID_ERRORS|ECC_DBE_VOL_TOTAL|GPU_TEMP)"}'
      - '{__name__=~"node_(load1|memory_MemAvailable_bytes|infiniband_state_id)"}'
      - '{__name__=~"kube_(pod_status_phase|node_status_condition)"}'
  static_configs:
    - targets:
        - prometheus-cluster-1.observability.internal:9090
      labels:
        cluster: gpu-cluster-1

Remote-write to long-term storage

The pattern that actually scales. Each cluster Prometheus runs remoteWrite to a central Mimir / Thanos / VictoriaMetrics / managed service. The central system handles:

  • Long-term storage on object storage (S3 / GCS / Azure Blob).
  • Query fanout via a central API.
  • Deduplication when multiple Prometheus replicas remote-write the same metrics.
  • Per-tenant isolation via X-Scope-OrgID (Mimir / Cortex multi-tenancy).

The example remoteWrite block in the Prometheus CR above shows the shape. For Mimir specifically:

remoteWrite:
  - url: https://mimir.observability.internal/api/v1/push
    headers:
      X-Scope-OrgID: gpu-cluster-1
    writeRelabelConfigs:
      - sourceLabels: [__name__]
        regex: '(DCGM_FI_DEV_.*|node_.*|kube_.*|up|prometheus_remote_storage_.*)'
        action: keep
    queueConfig:
      capacity: 10000
      maxSamplesPerSend: 2000
      maxShards: 30
      minBackoff: 30ms
      maxBackoff: 5s
    metadataConfig:
      send: true
      sendInterval: 1m

Tune queueConfig.maxShards upward if you see prometheus_remote_storage_samples_pending rising. Tune downward if your central system is getting overwhelmed.

You keep both: local Prometheus with 30-day retention for fast queries and dashboards, central long-term store for "what did the cluster look like 6 months ago when this customer's training run was working".

values.yaml: the parts that actually matter

For the kube-prometheus-stack Helm chart, here are the bits I always override on a fresh GPU cluster install:

fullnameOverride: kube-prom

defaultRules:
  create: true
  rules:
    etcd: true
    kubeApiserverAvailability: true
    kubeApiserverBurnrate: true
    kubeApiserverHistogram: true
    kubeApiserverSlos: true
    kubelet: true
    kubernetesApps: true
    kubernetesResources: true
    kubernetesStorage: true
    kubernetesSystem: true
    network: true
    node: true
    nodeExporterAlerting: true
    nodeExporterRecording: true
    prometheus: true
    prometheusOperator: true
    # Drop these, they're noise on a GPU cluster:
    general: false
    k8s: true
    kubeControllerManager: false
    kubeProxy: false
    kubeSchedulerAlerting: false
    kubeSchedulerRecording: false
    kubeStateMetrics: true

alertmanager:
  enabled: true
  config:
    # see /monitoring/alerting for the real config
    route:
      receiver: default
      group_by: [alertname, cluster, namespace]
  alertmanagerSpec:
    replicas: 3
    storage:
      volumeClaimTemplate:
        spec:
          storageClassName: local-nvme
          accessModes: [ReadWriteOnce]
          resources:
            requests:
              storage: 20Gi
    resources:
      requests:
        cpu: 100m
        memory: 256Mi
      limits:
        memory: 512Mi

grafana:
  enabled: true
  adminPassword: ""  # set via existingSecret
  admin:
    existingSecret: grafana-admin
    userKey: username
    passwordKey: password
  persistence:
    enabled: true
    storageClassName: local-nvme
    size: 20Gi
  defaultDashboardsTimezone: UTC
  sidecar:
    dashboards:
      enabled: true
      label: grafana_dashboard
      labelValue: "1"
      searchNamespace: ALL
    datasources:
      enabled: true
  additionalDataSources:
    - name: Loki
      type: loki
      url: http://loki-gateway.monitoring.svc:80
      access: proxy
    - name: Tempo
      type: tempo
      url: http://tempo-query-frontend.monitoring.svc:3100
      access: proxy

prometheus:
  prometheusSpec:
    replicas: 2
    shards: 1
    retention: 30d
    retentionSize: 800GiB
    walCompression: true
    enableRemoteWriteReceiver: false
    scrapeInterval: 30s
    evaluationInterval: 30s
    resources:
      requests:
        cpu: "4"
        memory: "32Gi"
      limits:
        memory: "48Gi"
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: local-nvme
          accessModes: [ReadWriteOnce]
          resources:
            requests:
              storage: 1000Gi
    externalLabels:
      cluster: gpu-cluster-1
      region: dc-1
    podMonitorSelectorNilUsesHelmValues: false
    serviceMonitorSelectorNilUsesHelmValues: false
    ruleSelectorNilUsesHelmValues: false
    podMonitorSelector: {}
    serviceMonitorSelector: {}
    ruleSelector: {}
    additionalScrapeConfigs:
      name: additional-scrape-configs
      key: prometheus-additional.yaml

prometheus-node-exporter:
  hostNetwork: true
  extraArgs:
    - --collector.infiniband
    - --collector.systemd
    - --collector.systemd.unit-include=^(nvidia-fabricmanager|nvidia-persistenced|kubelet|containerd|chrony|sshd|nv_peer_mem)\.service$
    - --collector.processes
    - --collector.nvme
    - --no-collector.fibrechannel
    - --no-collector.ipvs
    - --no-collector.btrfs

kube-state-metrics:
  metricLabelsAllowlist:
    - "pods=[app,team,job_id,training_run]"
    - "nodes=[topology.kubernetes.io/zone,nvidia.com/gpu.product]"

Two settings in there that catch people: *SelectorNilUsesHelmValues: false and the empty *Selector: {}. By default, the chart sets serviceMonitorSelector to a strict label match on the chart's own release label, which means anything you create without that label gets dropped. Setting both *NilUsesHelmValues: false and empty selectors makes Prometheus pick up every ServiceMonitor / PodMonitor / PrometheusRule in the cluster, regardless of label. That is what you want once you have multiple teams creating their own.

The metricLabelsAllowlist for kube-state-metrics is how you get useful pod labels — team, job_id, training_run — into your metrics for slicing. Without the allowlist they are dropped to control cardinality. Pick a small allowlist, deliberately.

Operational notes

Scrape interval and CPU on the target

DCGM exporter at 15 s scrape interval costs about 2% of one CPU core on the target node. node-exporter at 30 s is essentially free. Do not set scrape intervals below 10 s on DCGM — DCGM samples some counters off the GPU itself, and tighter intervals can interact with NVML semaphore contention with workloads. 15 s is the right answer.

Scrape timeout vs interval

scrapeTimeout must be less than scrapeInterval or the operator rejects the config. We use scrapeInterval: 15s, scrapeTimeout: 10s for DCGM. If a DCGM scrape is timing out, the GPU has bigger problems than monitoring.

Out-of-order ingestion

By default, Prometheus rejects samples with timestamps older than the head block. If you remote-write through a flaky link, samples can arrive out of order and get dropped. On Prometheus 2.39+, set tsdb.out_of_order_time_window: 30m in --config.file (or via the operator's tsdb.outOfOrderTimeWindow) to accept up to 30 minutes of late samples.

Reload mechanism

prometheus-operator watches the Prometheus's secret/configmap and when it changes, it sends SIGHUP to the Prometheus process via a sidecar called config-reloader. If you make a change and Prometheus does not pick it up:

kubectl logs -n monitoring prometheus-kube-prom-prometheus-0 -c config-reloader --tail=50

The reloader logs show the config hash and any reload errors. The most common cause of "my new ServiceMonitor is not being scraped" is the missing release: label, second is a typo in the port: name, third is a namespace selector that excludes the target namespace.

Validating the scrape

kubectl port-forward -n monitoring svc/prometheus-operated 9090:9090

# In another shell:
curl -s 'http://localhost:9090/api/v1/targets' | jq '.data.activeTargets[] | select(.labels.job=="dcgm-exporter") | {endpoint, health, lastError}'

Or in the Prometheus UI: Status → Targets. Anything in down state with a non-empty lastError tells you exactly what is wrong (DNS, connection refused, 403 from RBAC, etc.).

See also