Alertmanager for GPU clusters: routing, severity, and runbook discipline

The point of Alertmanager is not to send notifications. It is to send the right notifications to the right people, dedup them, suppress the noise, and stop sending when the on-call has acknowledged. Get this right and the on-call rotation is sustainable. Get this wrong and your team burns out in three months.

This page is the operator's Alertmanager: a route tree we actually use on GPU clusters, the inhibition rules that turn "control-plane is down" from N pages into 1 page, the severity ladder that determines who gets called at 03:00, and the concrete PrometheusRule definitions that map onto it. The metric and log alert expressions live in dcgm-exporter and logging-loki; this page focuses on what happens after an alert fires.

Severity ladder

Five levels. They map cleanly to how we route and how on-call responds.

The labels on every alert rule must include severity matching one of these:

labels:
  severity: critical    # → P0 or P1
  severity: warning     # → P2 or P3
  severity: info        # → P4

We split P0/P1 with an extra pager: "true" or pager: "team" label. The route tree handles the rest.

Alertmanager config: the full picture

Here is a working alertmanager.yaml. Replace the receiver credentials with secret references in your real deployment.

global:
  resolve_timeout: 5m
  smtp_smarthost: 'mail.observability.internal:587'
  smtp_from: 'alerts@observability.internal'
  slack_api_url_file: /etc/alertmanager/secrets/slack-url
  pagerduty_url: 'https://events.pagerduty.com/v2/enqueue'

# ---- Templates --------------------------------------------------------------
templates:
  - /etc/alertmanager/templates/*.tmpl

# ---- Route tree -------------------------------------------------------------
route:
  receiver: default-slack
  group_by: ['alertname', 'cluster', 'namespace']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h

  routes:
    # P0: anything tagged as cluster-wide outage
    - matchers:
        - severity = critical
        - scope = cluster
      receiver: p0-pagerduty
      group_wait: 0s
      group_interval: 1m
      repeat_interval: 30m
      continue: true

    # GPU ops team — XID, ECC, fabric-manager, NVLink
    - matchers:
        - team = gpu-ops
        - severity = critical
      receiver: gpu-ops-pagerduty
      group_by: ['alertname', 'Hostname', 'gpu']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 1h

    - matchers:
        - team = gpu-ops
        - severity = warning
      receiver: gpu-ops-slack
      group_by: ['alertname', 'cluster']
      group_wait: 1m
      group_interval: 10m
      repeat_interval: 12h

    # Storage team — Weka
    - matchers:
        - team = storage
        - severity = critical
      receiver: storage-pagerduty
      repeat_interval: 1h

    - matchers:
        - team = storage
        - severity =~ "warning|info"
      receiver: storage-slack
      repeat_interval: 12h

    # Network team — IB / RoCE / NIC
    - matchers:
        - team = net
        - severity = critical
      receiver: net-pagerduty
      group_by: ['alertname', 'cluster']
      repeat_interval: 1h

    - matchers:
        - team = net
        - severity = warning
      receiver: net-slack
      repeat_interval: 12h

    # ML platform — user-impacting but not infra
    - matchers:
        - team = ml-platform
      receiver: ml-platform-slack
      repeat_interval: 24h

    # Daily digest only — never page, never even Slack-in-the-night
    - matchers:
        - severity = info
      receiver: digest
      group_wait: 5m
      group_interval: 1h
      repeat_interval: 24h

    # Watchdog — ensures Alertmanager itself is alive
    - matchers:
        - alertname = Watchdog
      receiver: watchdog
      group_wait: 0s
      group_interval: 30s
      repeat_interval: 1m

# ---- Inhibition rules -------------------------------------------------------
inhibit_rules:

  # If the entire cluster is down, don't page the GPU team for individual XIDs.
  - source_matchers:
      - severity = critical
      - scope = cluster
    target_matchers:
      - severity =~ "warning|critical"
      - scope != cluster
    equal: [cluster]

  # If kubelet is down on a node, don't fire pod-level alerts for that node.
  - source_matchers:
      - alertname = KubeletDown
    target_matchers:
      - alertname =~ "KubePodCrashLooping|KubePodNotReady|GPUFellOffTheBus"
    equal: [cluster, instance]

  # If fabric-manager is down on a node, don't also page for NVLink errors
  # on that node — they are downstream effects.
  - source_matchers:
      - alertname = GPUFabricManagerDown
    target_matchers:
      - alertname =~ "GPUNVLinkRecoveryErrorRate|GPUXID74NVLinkError"
    equal: [cluster, Hostname]

  # If the Weka cluster is degraded, don't page for individual node IO latency.
  - source_matchers:
      - alertname = WekaClusterDegraded
    target_matchers:
      - alertname =~ "WekaIOLatencyHigh|WekaPodSlow"
    equal: [cluster]

  # If a node is in maintenance, suppress everything from it.
  - source_matchers:
      - alertname = NodeMaintenance
    target_matchers:
      - severity =~ "warning|critical"
    equal: [cluster, Hostname]

  # Critical inhibits warning for the same alert/labels.
  - source_matchers:
      - severity = critical
    target_matchers:
      - severity = warning
    equal: [alertname, cluster, namespace]

# ---- Mute time intervals ----------------------------------------------------
time_intervals:
  - name: business-hours
    time_intervals:
      - weekdays: [monday:friday]
        times:
          - start_time: '08:00'
            end_time:   '18:00'
        location: 'Etc/UTC'
  - name: nights-and-weekends
    time_intervals:
      - weekdays: [saturday, sunday]
      - weekdays: [monday:friday]
        times:
          - start_time: '18:00'
            end_time:   '08:00'
        location: 'Etc/UTC'
  - name: maintenance-window
    time_intervals:
      - weekdays: [tuesday]
        times:
          - start_time: '02:00'
            end_time:   '04:00'
        location: 'Etc/UTC'

# ---- Receivers --------------------------------------------------------------
receivers:

  - name: default-slack
    slack_configs:
      - channel: '#alerts-default'
        send_resolved: true
        title: '{{ template "slack.title" . }}'
        text: '{{ template "slack.text" . }}'

  - name: p0-pagerduty
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/pd-p0-key
        severity: critical
        description: '[P0] {{ .GroupLabels.alertname }} on {{ .GroupLabels.cluster }}'
        details:
          runbook:    '{{ (index .Alerts 0).Annotations.runbook_url }}'
          summary:    '{{ (index .Alerts 0).Annotations.summary }}'
          dashboard:  '{{ (index .Alerts 0).Annotations.dashboard_url }}'
          description: '{{ (index .Alerts 0).Annotations.description }}'
    slack_configs:
      - channel: '#incident-active'
        send_resolved: true
        title: '[P0] {{ .GroupLabels.alertname }} on {{ .GroupLabels.cluster }}'
        text:  '{{ template "slack.p0.text" . }}'

  - name: gpu-ops-pagerduty
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/pd-gpu-ops-key
        severity: '{{ if eq .CommonLabels.severity "critical" }}critical{{ else }}warning{{ end }}'
        description: '{{ .GroupLabels.alertname }} on {{ .CommonLabels.Hostname }} (GPU {{ .CommonLabels.gpu }})'
    slack_configs:
      - channel: '#alerts-gpu-ops'
        send_resolved: true

  - name: gpu-ops-slack
    slack_configs:
      - channel: '#alerts-gpu-ops'
        send_resolved: true
        title: '{{ template "slack.title" . }}'
        text:  '{{ template "slack.text" . }}'

  - name: storage-pagerduty
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/pd-storage-key
    slack_configs:
      - channel: '#alerts-storage'

  - name: storage-slack
    slack_configs:
      - channel: '#alerts-storage'
        send_resolved: true

  - name: net-pagerduty
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/pd-net-key
    slack_configs:
      - channel: '#alerts-network'

  - name: net-slack
    slack_configs:
      - channel: '#alerts-network'
        send_resolved: true

  - name: ml-platform-slack
    slack_configs:
      - channel: '#alerts-ml-platform'
        send_resolved: true
        title: '[ml-platform] {{ .GroupLabels.alertname }}'

  - name: digest
    email_configs:
      - to: 'gpu-ops-digest@observability.internal'
        send_resolved: false
        headers:
          Subject: 'Alert digest: {{ len .Alerts }} alerts'

  - name: watchdog
    webhook_configs:
      - url: 'http://watchdog-receiver.monitoring.svc:8080/webhook'
        send_resolved: false

What each part of the config does

group_by, group_wait, group_interval, repeat_interval

These four parameters control the dedup and notification cadence:

• group_by — alerts with identical values for these labels become one notification. We use [alertname, cluster, namespace] at the top, then narrow per-route. For GPU-ops we use [alertname, Hostname, gpu] so simultaneous failures on different GPUs each get their own page.
• group_wait — when a new group is created, wait this long before sending the first notification. Lets a "storm" of alerts in the same group arrive together. 30 s is good for warnings; 0 s for P0 (don't wait).
• group_interval — once a group is sent, this is how long we wait before sending an update for that group (new alerts joining or alerts resolving). 5 m default.
• repeat_interval — if the group is still firing, re-notify after this. 4 h default; 1 h for criticals so on-call doesn't lose the thread; 30 m for P0.

continue: true

Without continue, an alert that matches a route is delivered there and stops. With continue, it also continues evaluating subsequent routes. We use it on the P0 route so a P0 fires both PagerDuty and the team's normal channel — the team needs to know its area was hit even if the incident commander already has the page.

Inhibition

Inhibition rules say: "if A is firing, suppress B". The classic example is "if the cluster is down, don't page for individual pods being down". Concrete rules from the config above:

1. Cluster-down kills everything — the broadest. If a P0 cluster-scope alert fires, suppress everything else in that cluster.
2. Kubelet-down on a node kills pod-level alerts on that node — by instance (the node identifier).
3. Fabric-manager-down kills NVLink alerts on that node — they are downstream symptoms.
4. Weka degraded kills per-node IO latency alerts — cause vs symptom.
5. severity=critical on the same alertname inhibits severity=warning — if a critical version is firing, don't double-page on the warning version.

The equal: field is critical: it specifies which labels must match between source and target. Without it, an inhibit rule on cluster A would also suppress alerts on cluster B, which would be a disaster.

Mute time intervals

Use mute_time_intervals (in a route) to silence non-critical alerts during a maintenance window:

route:
  receiver: gpu-ops-slack
  matchers:
    - team = gpu-ops
    - severity = warning
  mute_time_intervals: [maintenance-window]

For ad-hoc maintenance, use amtool silence instead — that is what it is for.

Watchdog alert

Always have one alert that always fires (the "Watchdog"). It is a vector(1) always-on alert in Prometheus:

- alert: Watchdog
  expr: vector(1)
  labels:
    severity: none
  annotations:
    summary: "Alertmanager-prometheus pipe is alive"

Route it to a webhook receiver that just notes the timestamp. If the watchdog stops arriving, your alerting stack itself is broken — and you alert on that via a separate dead-man's-switch (a CronJob that pings deadmanssnitch.com or similar).

On-call rotation: PagerDuty / OpsGenie

We use PagerDuty per team — separate services for gpu-ops, storage, net, and p0. Each service has:

• Primary on-call.
• Secondary on-call (escalation in 15 min if primary doesn't ack).
• Manager on-call (escalation in 30 min).

Routing keys live in /etc/alertmanager/secrets/* mounted as a Secret. A team gets paged via Alertmanager → PagerDuty → SMS/phone/push. Slack notifications go alongside but never replace PD pages — Slack is for context, PD is for waking people up.

OpsGenie is structurally similar; the receiver type is opsgenie_configs instead of pagerduty_configs. Same fields apply.

Slack templates

A clean Slack message saves seconds when you are rolling out of bed. Define a template:

{{ define "slack.title" -}}
[{{ if eq .Status "firing" }}🔥 FIRING{{ else }}✅ RESOLVED{{ end }}]
{{ .GroupLabels.alertname }} ({{ .GroupLabels.cluster | default "any" }})
{{- end }}

{{ define "slack.text" -}}
{{ range .Alerts }}
*Severity:* {{ .Labels.severity }}
*Cluster:* {{ .Labels.cluster }}
{{ if .Labels.Hostname }}*Node:* `{{ .Labels.Hostname }}`{{ end }}
{{ if .Labels.gpu }}*GPU:* `{{ .Labels.gpu }}`{{ end }}
{{ if .Labels.namespace }}*Namespace:* `{{ .Labels.namespace }}`{{ end }}
{{ if .Labels.pod }}*Pod:* `{{ .Labels.pod }}`{{ end }}
*Summary:* {{ .Annotations.summary }}
{{ if .Annotations.description }}*Detail:* {{ .Annotations.description }}{{ end }}
{{ if .Annotations.runbook_url }}<{{ .Annotations.runbook_url }}|Runbook>{{ end }}
{{ if .Annotations.dashboard_url }} • <{{ .Annotations.dashboard_url }}|Dashboard>{{ end }}
{{ end }}
{{- end }}

runbook_url and dashboard_url are critical. Every page links to a runbook page. The runbook page has a literal command-line procedure ("SSH to the node, run X, if Y do Z"). Without this, on-call wastes ten minutes locating documentation while the customer's training job sits idle.

Alert rules: the canonical list

These are the alerts I run. Most expressions live in dcgm-exporter and logging-loki; the ones below are the ones not yet shown.

Fabric-manager dead

- alert: GPUFabricManagerDown
  expr: |
    node_systemd_unit_state{name="nvidia-fabricmanager.service",state="active"} != 1
    and on (instance) (count by (instance) (DCGM_FI_DEV_GPU_TEMP) >= 8)
  for: 1m
  labels:
    severity: critical
    team: gpu-ops
    pager: "true"
  annotations:
    summary: "fabric-manager down on {{ $labels.instance }}"
    description: |
      nvidia-fabricmanager.service is not active on a multi-GPU node.
      The NVSwitch fabric is unconfigured. Multi-GPU jobs on this node
      will fail or fall back to PCIe.
    runbook_url: https://runbooks.internal/gpu/fabricmanager-down
    dashboard_url: https://grafana.internal/d/gpu-node?var-node={{ $labels.instance }}

Repo-server CrashLoop (ArgoCD)

- alert: ArgoCDRepoServerCrashLoop
  expr: |
    kube_pod_container_status_waiting_reason{
      namespace="argocd",
      container="argocd-repo-server",
      reason="CrashLoopBackOff"
    } == 1
  for: 5m
  labels:
    severity: warning
    team: platform
  annotations:
    summary: "argocd-repo-server stuck in CrashLoopBackOff"
    description: |
      ArgoCD repo-server is unable to render manifests. New deployments
      will fail to sync until this is resolved. Common causes: OOMKill
      under large repo size, transient git auth failure, helm dependency
      timeout.
    runbook_url: https://runbooks.internal/argocd/repo-server-crash

Weka cluster degraded

- alert: WekaClusterDegraded
  expr: weka_cluster_status != 1
  for: 1m
  labels:
    severity: critical
    team: storage
    pager: "true"
  annotations:
    summary: "Weka cluster {{ $labels.cluster }} is not healthy"
    description: |
      weka_cluster_status reports {{ $value }} (expected 1 = HEALTHY).
      Investigate via `weka status` from a Weka client. All training
      jobs reading from this Weka cluster will see degraded IO.
    runbook_url: https://runbooks.internal/weka/cluster-degraded

NCCL bandwidth dropping below baseline

This one is subtle. We record a baseline for each cluster's typical NCCL all-reduce bandwidth and alert on regression. The recording rule:

- record: cluster:nccl_bandwidth_baseline:7d_p50
  expr: |
    quantile_over_time(0.5,
      sum by (cluster) (
        rate(DCGM_FI_PROF_NVLINK_TX_BYTES{cluster!=""}[5m])
        + rate(node_infiniband_port_data_xmit_bytes_total{cluster!=""}[5m])
      )[7d:5m]
    )

And the alert:

- alert: NCCLBandwidthBelowBaseline
  expr: |
    (
      sum by (cluster) (
        rate(DCGM_FI_PROF_NVLINK_TX_BYTES[5m])
        + rate(node_infiniband_port_data_xmit_bytes_total[5m])
      )
    )
    /
    cluster:nccl_bandwidth_baseline:7d_p50
    < 0.7
  for: 30m
  labels:
    severity: warning
    team: gpu-ops
  annotations:
    summary: "Cluster {{ $labels.cluster }} aggregate fabric throughput < 70% of 7d baseline"
    description: |
      Sustained for 30+ minutes. Possible causes: a flaky cable, a switch
      saturation issue, a rogue job hogging the fabric, or a fabric-manager
      issue affecting NVLink throughput.
    runbook_url: https://runbooks.internal/gpu/nccl-bandwidth-regression

This kind of "compare against my own baseline" alert is the only way to catch slow degradation. Threshold-based alerts ("bandwidth < 200 Gbps") miss the case where a cluster used to do 380 Gbps and is now doing 240 Gbps but neither value is "low enough" to trip a static threshold.

Repeat OOMKills in a system namespace

- alert: SystemNamespaceOOMKills
  expr: |
    sum by (namespace, pod) (
      increase(kube_pod_container_status_terminated_reason{
        reason="OOMKilled",
        namespace=~"kube-.*|gpu-.*|nvidia-.*|monitoring|argocd"
      }[1h])
    ) > 2
  for: 0s
  labels:
    severity: critical
    team: platform
    pager: "true"
  annotations:
    summary: "Multiple OOMKills in system namespace {{ $labels.namespace }} ({{ $labels.pod }})"
    description: |
      Pod {{ $labels.namespace }}/{{ $labels.pod }} OOMKilled
      {{ $value }} times in the last hour. Either a memory leak or
      under-sized limits.
    runbook_url: https://runbooks.internal/k8s/oom-system

The same alert for non-system namespaces is severity:info → digest, so users get notified but on-call doesn't.

Anti-patterns

The mistakes that erode trust in alerts:

1. No for: clause on a noisy metric. A single 1 s spike pages you. Always set a window appropriate to the metric's natural variance.
2. Absolute thresholds on counter values. Counters reset on driver reload; you'll see a fake < threshold event on every restart. Use rate() or increase() for counters.
3. Alerting on dashboard panels. Grafana managed alerts are fine for tactical use, but the source of truth should be PrometheusRule CRDs in Git. Otherwise you cannot review what is firing.
4. Vague alert names. HighGPUUsage tells you nothing. GPUSustainedHighTemp tells you what to look at. The alertname is the single most-read field.
5. Empty annotations. A page with summary: "fired" and no description, no runbook, no dashboard — useless. The runbook link is mandatory.
6. Identical alert at two severities. Don't have a WarningHighTemp and a CriticalHighTemp with different thresholds — use one alert with severity-by-value:

- alert: GPUTempHigh
  expr: avg_over_time(DCGM_FI_DEV_GPU_TEMP[5m]) > 87
  for: 5m
  labels:
    severity: '{{ if (gt (avg_over_time(DCGM_FI_DEV_GPU_TEMP[5m])) 95.0) }}critical{{ else }}warning{{ end }}'

(this is a contrived example; cleaner is two separate alerts with inhibit rules between them — but the principle is to avoid duplicate logic).

7. Pager on flapping signal. If an alert fires-resolves-fires-resolves in a 30-minute window, on-call gets paged 4×. Use keep_firing_for: 10m to hold the firing state and dedup.

Validating the config

Always test before applying:

# Validate alertmanager.yaml syntax + routing
amtool config check /etc/alertmanager/alertmanager.yaml

# Test routing for a hypothetical alert
amtool config routes test \
  --config.file=/etc/alertmanager/alertmanager.yaml \
  alertname=GPUFellOffTheBus team=gpu-ops severity=critical Hostname=node-X gpu=3
# → returns: gpu-ops-pagerduty

# Test that an inhibit rule works
amtool config check --alertmanager.url http://localhost:9093

# Send a fake alert
amtool alert add alertname=Test team=gpu-ops severity=warning \
  cluster=lab Hostname=test-node-1 \
  --start=$(date -u +%FT%TZ) --end=$(date -u -v+10M +%FT%TZ)

amtool ships with the Alertmanager binary. Do not skip the routing test — a typo in matchers silently routes alerts to the default receiver, which usually is not where on-call is looking.

Silencing during maintenance

When draining a node or doing planned maintenance:

# Silence everything from a single node for 2 hours
amtool silence add Hostname=node-X \
  --duration=2h \
  --comment="planned drain - ticket OPS-1234" \
  --author="$USER"

# Or by namespace
amtool silence add namespace=ml-team-staging \
  --duration=24h \
  --comment="staging environment maintenance"

Use the comment field. Six months from now, somebody will want to know why an alert was silenced.

For Kubernetes-managed silences (declared in YAML, applied by an operator like silence-controller), the Silence CR is the way to do this from a CI pipeline.

Operational notes

Replication and HA

Alertmanager runs in a 3-replica gossip mesh by default. The replicas use the gossip protocol to dedupe alerts (so you don't get 3 PagerDuty pages for the same alert). Configure with --cluster.peer=alertmanager-0.alertmanager-headless,alertmanager-1.alertmanager-headless,alertmanager-2.alertmanager-headless.

If gossip is misconfigured, you get duplicate pages. The kube-prometheus-stack chart wires it up correctly by default; if you change the StatefulSet name or service, double-check.

Notifications during a Prometheus outage

If Prometheus dies, no ALERTS go to Alertmanager and no pages fire — including ones that should ("Prometheus is down"). The dead-man's-switch is mandatory: the Watchdog alert always fires, and a separate external system (deadmanssnitch.com, healthchecks.io, or a small CronJob in another cluster) pages you when the Watchdog stops arriving.

Notification fatigue check

Every quarter, run a query like:

sum by (alertname, severity) (
  ALERTS_FOR_STATE{alertstate="firing"}
)

and review the top 20. Anything that fires more than ~10 times a week, on average, is either a real recurring problem (fix it) or a bad alert (raise the threshold, add for:, or delete it). This is the practice that keeps the alert list lean.

Alert annotations: runbook_url

Every alert in our repo has a runbook_url pointing to a real document. Runbook structure:

# <Alert name>

## What this means
One paragraph.

## Investigation steps
1. SSH to <node>.
2. Run `<command>`.
3. If <output>, then <action>.

## Mitigation
- <command>
- <command>

## Escalation
- After 30m without resolution, escalate to <team>.
- Ticket template: <link>

## Related
- /monitoring/<page>
- /drivers/<page>

A runbook for drivers/fabric-manager failures, for example, has the literal journalctl -u nvidia-fabricmanager and nv-fabricmanager --version commands an on-call should run. See operations/health-check-runbook and operations/incident-response for the runbook house style.

See also

• monitoring/overview — the alert short-list this config implements.
• monitoring/prometheus-stack — where PrometheusRule CRDs live.
• monitoring/dcgm-exporter — XID, ECC, NVLink alert expressions.
• monitoring/logging-loki — log-driven alerts via Loki ruler.
• monitoring/grafana-dashboards — dashboard_url annotation targets.
• drivers/fabric-manager — what the FM-down alert is telling you.
• operations/incident-response — the on-call procedure once an alert fires.
• operations/health-check-runbook — runbook conventions and template.
• operations/runbook-template — starter template for new runbooks.