monitoring a GPU cluster: what to watch and what to ignore

If you operate GPU clusters long enough, you will eventually learn that the difference between a healthy on-call rotation and a miserable one is not the dashboards you have. It is the alerts you do not have. A GPU node has roughly an order of magnitude more telemetry surface than a CPU node — eight GPUs, four NVSwitches, eight ConnectX-7 NICs, two CPUs, twelve memory channels per CPU, NVMe, BMC, PSU — and if you wire all of that into PagerDuty you get woken up every night for nothing. Then nobody trusts the alerts and the one that actually matters slips through.

This page lays out what is worth monitoring on a GPU cluster, what each signal actually tells you, and the very short list of conditions that justify paging a human in the middle of the night. The follow-on pages cover the implementation: Prometheus, DCGM exporter, Grafana, Loki, Alertmanager.

The four-layer pyramid

Observability on a GPU cluster splits cleanly into four layers, in roughly decreasing volume and increasing cost-to-emit:

                       events
                       /    \
                    traces   \
                   /          \
                logs           \
              /                 \
        metrics  <-- 95% of your value lives here

1. Metrics — numeric time-series scraped on a fixed interval. GPU temperature, power draw, ECC counters, NIC drops, pod state, kubelet up/down. Cheap to store (Prometheus is good at this), cheap to query, cheap to alert on. This is where you spend most of your effort.

2. Logs — semi-structured text with timestamps. dmesg, kubelet logs, slurmd logs, fabric-manager log, NCCL debug output, training job stdout/stderr. Higher volume, harder to query, but indispensable when a metric tells you something is broken and you need to know what. Loki is the right tool because labels are cheap and full-text search is fine when you already filtered to the relevant pod.

3. Traces — request-scoped spans for distributed systems. On a GPU cluster these matter for the control plane (API server, scheduler, etcd, ArgoCD, Slurm REST) and for inference serving (Triton, vLLM). They almost never matter for training jobs themselves — a training job is one big monolithic loop and tracing its CUDA kernels is the job of Nsight, not Tempo.

4. Events — discrete, irregular things that happened. kubectl get events, DCGM dcgmi diag results, XID errors out of the kernel, MOFED link state transitions, switch syslog. Lower volume than logs but each event is heavy with meaning. You ingest these into Loki with a separate label set and you alert on patterns of them, not on individual occurrences (with a small list of exceptions — see below).

The mistake everyone makes early on is treating these layers as interchangeable. They are not. You do not alert on a log line. You alert on a metric that counts how often that log line appears. You do not page on a single event. You page on the rate of events.

What to monitor on a GPU node

Per node — and on a real cluster you have hundreds of these — the signals worth scraping are:

GPU itself (DCGM)

• Utilization — DCGM_FI_DEV_GPU_UTIL, DCGM_FI_PROF_SM_ACTIVE, DCGM_FI_PROF_PIPE_TENSOR_ACTIVE. Not for alerting. For capacity planning and for catching jobs that allocate 8 GPUs and use 0.3 of them.
• Memory — DCGM_FI_DEV_FB_USED, DCGM_FI_DEV_FB_FREE, DCGM_FI_DEV_FB_TOTAL. Useful for OOM forensics. Almost never useful for alerting because the workload is supposed to fill the GPU.
• Temperature — DCGM_FI_DEV_GPU_TEMP, DCGM_FI_DEV_MEMORY_TEMP. Alert above sustained 87 C GPU or 95 C HBM. Not on a single sample — on a 5-minute average. GPUs spike to 90 C transiently all the time and it does not mean anything.
• Power — DCGM_FI_DEV_POWER_USAGE. Per-GPU and summed per node. Worth a dashboard, rarely worth an alert except when it goes to zero (GPU fell off the bus).
• Throttling — DCGM_FI_DEV_CLOCK_THROTTLE_REASONS. Bitmask. Bits set for thermal, power-cap, sync-boost, HW-slowdown, HW-thermal-slowdown, SW-thermal-slowdown. Alert if HW-thermal-slowdown is sustained for more than 60 seconds — that means the silicon hit Tjmax.
• ECC — DCGM_FI_DEV_ECC_SBE_VOL_TOTAL (correctable) and DCGM_FI_DEV_ECC_DBE_VOL_TOTAL (uncorrectable). Uncorrectable ECC is page-now. A double-bit error means the GPU silently corrupted memory. Every job on that GPU since the last reset is suspect.
• Retired pages — DCGM_FI_DEV_RETIRED_SBE, DCGM_FI_DEV_RETIRED_DBE, DCGM_FI_DEV_RETIRED_PENDING. Pending retirements above zero means the GPU wants a power cycle. Schedule it.
• NVLink — DCGM_FI_DEV_NVLINK_REPLAY_ERROR_COUNT_TOTAL, DCGM_FI_DEV_NVLINK_RECOVERY_ERROR_COUNT_TOTAL, DCGM_FI_DEV_NVLINK_CRC_FLIT_ERROR_COUNT_TOTAL. Replay rate above 1/min/link is a warning, recovery events are pages. CRC flit errors are physical-layer noise — clean the connector, swap the cable, or RMA the GPU.
• XID errors — DCGM_FI_DEV_XID_ERRORS. The single most important GPU signal. See "What to alert on" below.

NVSwitch fabric

• Fabric-manager service state — scrape node_systemd_unit_state{name="nvidia-fabricmanager.service"} from node-exporter. If FM is dead, the node is dead, even if every other metric looks healthy. Page immediately.
• NVLink throughput per GPU per direction — DCGM_FI_PROF_NVLINK_TX_BYTES, DCGM_FI_PROF_NVLINK_RX_BYTES. Useful for confirming NCCL is actually using NVLink and not falling back to PCIe.

NIC and IB fabric

• Link state — node_infiniband_state_id (1 = down, 4 = active). Page on transitions to anything other than 4.
• Port counters — node_infiniband_port_rcv_errors_total, node_infiniband_port_xmit_discards_total, node_infiniband_link_downed_total, node_infiniband_symbol_errors_total. Alert on rate, never on raw value (counters are cumulative since boot).
• PFC pause frames — RoCE only. node_infiniband_port_xmit_wait_total (IB equivalent of pause). Sustained pause frames mean the fabric is congested upstream and your training tail latency is being eaten by the switch.
• NIC drops at the OS — node_network_receive_drop_total, node_network_transmit_drop_total. Per-NIC. Drops on a backend NIC during training mean MTU/jumbo-frame mismatch, ring-buffer too small, or the NIC firmware is too old.
• Packet loss to peers — out-of-band ping/iperf checks. A counter at the OS will not catch a lossy path through the fabric.

Host kernel and OS

• Kernel ring buffer rate — node_kmsg_lines_total if you scrape dmesg, or just count log lines per second from a dmesg Promtail tail. A node spewing dmesg lines is a node that is about to crash.
• CPU steal / iowait — usually not interesting on bare-metal GPU nodes but worth knowing about on virtualized ones.
• Memory pressure — node_memory_MemAvailable_bytes, node_vmstat_oom_kill. OOM kills on a GPU node almost always mean a training job allocated more pinned memory than it should.
• NVMe SMART — node_nvme_* from nvme-cli exporter. Wear-leveling, media errors, temperature. Slow data plane disk under sustained read = wasted GPU cycles.

Cluster control plane

• Kubelet up — up{job="kubelet"} per node. Two minutes down = page.
• Pod state — kube_pod_status_phase, kube_pod_container_status_waiting_reason. CrashLoopBackOff and ImagePullBackOff on a system namespace = page. Same on a user namespace = ticket.
• Node conditions — kube_node_status_condition. MemoryPressure, DiskPressure, PIDPressure, Ready. NotReady for more than 5 minutes = page.
• API server latency — apiserver_request_duration_seconds. p99 above 1s sustained = something is wrong with etcd or the apiserver.

Slurm (for the slurm-on-k8s deployments)

• slurmctld up — page.
• slurmd per node — counts of nodes in idle, alloc, drain, down, fail. Sudden jump in drain count is a fabric-manager or DCGM issue rolling across the fleet.
• Queue depth per partition — slurm_partition_jobs_pending. For capacity planning, not for paging.

Storage (Weka, in our case)

• Cluster status — weka_cluster_status (1 = healthy, 0 = degraded). Anything other than healthy is a page.
• Drive failures — weka_drive_status. One drive in a 100-node Weka cluster failing is normal. Three failures in five minutes is a power event or a controller failure — page.
• IO latency p99 — alert on sustained > 5 ms read latency, training jobs notice.
• Free capacity — alert at 85% used.

For the operator's perspective on what these knobs mean, see the weka overview and drivers/fabric-manager. For the network-layer pieces — PFC, ECN, port state, IB vs RoCE — see networking/perf-tuning and networking/ib-vs-roce.

The metrics → logs → traces → events split, in practice

A concrete worked example. A user reports "my training job is slow". Here is the layered investigation:

1. Metrics first — open the per-job dashboard. SM active is 30%, NVLink TX is high but uneven, IB TX bandwidth is fluctuating between 50 and 200 Gbps when it should be flat at 380. There is the symptom.

2. Logs second — pull NCCL debug output from the pod via Loki: {namespace="train", pod=~"job-foo.*"} |= "NCCL". You see NET/IB : Got completion with error 12 (transport retry counter exceeded). There is the cause: one IB link is flaky.

3. Events third — query Loki for syslog from the node: {job="kmsg", node="<node>"} |= "mlx5". You see mlx5_core: ... port_module_event: module 4, Cable error, Bad cable. There is the root cause: the cable in port 4 is bad.

4. Traces — irrelevant here. Tracing does not help with NCCL all-reduce.

The lesson is that each layer narrows the scope. You go from "the cluster is slow" to "this job is slow on these GPUs" to "NCCL is retrying on this NIC" to "the cable in port 4 is bad" — and then you tell the data center to swap the optic. None of the upper layers told you about the cable directly. The cable showed up in events, but you would never have looked at events without the metrics first.

What to alert on (the short list)

These are the conditions that justify paging an on-call human. The list is short by design:

Everything else is a Slack notification, a ticket, a daily digest, or just a row on a dashboard you look at on Mondays.

XID error codes worth memorizing

XIDs are kernel-level GPU error codes printed by nvidia.ko to dmesg in the form:

NVRM: Xid (PCI:0000:01:00): 79, pid=0, GPU has fallen off the bus.

There are about 140 defined XID codes. Most are operationally irrelevant. The ones every operator should recognize on sight:

Anything not on this list is either rare or harmless. The full list is in nv_xid.h in the open-gpu-kernel-modules source. For deeper context on what the GPU subsystems behind these errors are doing, see drivers/nvidia.

Per-NIC and per-fabric signals worth knowing

The host-side networking surface on a GPU node is large — usually 8 ConnectX-7 NICs (one per GPU on a typical HGX board), each with one or two physical ports, plus a separate management NIC. Each port exposes hundreds of counters via /sys/class/infiniband/<dev>/ports/<n>/counters/. The signals worth scraping (most are picked up by node-exporter's infiniband collector automatically):

• port_data_xmit_bytes, port_data_rcv_bytes — bandwidth in/out. Per-port. Plot rate-of-change.
• port_xmit_packets, port_rcv_packets — packet count. Useful sanity check (do bytes-per-packet make sense? if they shrink, the workload is hitting tiny messages).
• port_rcv_errors — receive errors. Any nonzero rate is a flag.
• port_xmit_discards — outgoing discards, usually due to subnet manager re-routing or congestion. Sustained nonzero rate = fabric problem.
• link_downed — counts how many times the link transitioned to Down. Goes up by 1 per flap. Flap rate > 1/hour = bad cable / bad optic.
• symbol_errors — physical-layer bit errors. Slow rise = degrading optic. Sudden burst = a transient (cosmic ray, EMI). Does not on its own signal a problem unless rate > a few per minute.
• port_xmit_wait — IB equivalent of pause time. Sustained nonzero = the port is waiting on credit, i.e., the fabric ahead of it is congested.
• local_link_integrity_errors — physical-layer issue, usually a connector seating problem.
• port_rcv_remote_physical_errors — bit errors signaled by the remote end. Same as symbol errors but counted on the receive side.

For RoCE / Ethernet (mlx5):

• rx_pause_storm_warning_events — PFC pause storm. The fabric is congested and the priority is being throttled.
• rx_priority<N>_pause, tx_priority<N>_pause — PFC frames per priority. RoCE traffic typically uses priority 3.
• rx_buffer_full, rx_out_of_buffer — switch fabric is dropping into your NIC buffer. Bump the ring size (ethtool -G).
• fec_corrected_blocks, fec_uncorrected_blocks — forward error correction. Uncorrected blocks → packet loss.

What to alert on, what to dashboard:

For deeper context: networking/perf-tuning, networking/ib-architecture, networking/roce.

A worked triage scenario

User pages: "training job foo just crashed, can you look?"

# Step 1: who owns the pod, where is it running?
kubectl get pod -n ml-team train-foo-7d4f8 -o wide
# NAME             READY   STATUS    RESTARTS   AGE   IP            NODE        ...
# train-foo-7d4f8  0/1     Error     0          12m   <pod-ip>      <node>     ...

# Step 2: container exit reason
kubectl describe pod -n ml-team train-foo-7d4f8 | grep -A 3 "Last State"
# Last State:    Terminated
# Reason:        Error
# Exit Code:     1
# Started:       ...
# Finished:      ...

# Step 3: container logs (last 200 lines around the crash)
kubectl logs -n ml-team train-foo-7d4f8 --previous --tail=200
# ... NCCL stack trace ...
# RuntimeError: NCCL communicator was aborted on rank 0

So far you know: pod crashed with NCCL abort on rank 0. Now correlate with cluster-side events.

# Step 4: query DCGM for XID on that node, that pod, that time window
# In Grafana: open "GPU per-node" dashboard, set node=<node>,
# set time = ±5m around the crash time.
# You see DCGM_FI_DEV_XID_ERRORS{Hostname="<node>",gpu="3"} == 79 at the crash moment.

# Step 5: confirm in Loki
{job="kmsg", node="<node>"} |~ "Xid"
# (in Grafana Loki panel, scoped to the same time)
# → "Xid (PCI:0000:c1:00): 79, pid=12847, GPU has fallen off the bus."

# Step 6: you also notice fabric-manager log spammed in the same window
{job="fabric-manager", node="<node>"}
# → "GPU lost from fabric, removing from routing"

Now you have the full causal chain: GPU 3 on <node> fell off the bus (XID 79), fabric-manager lost it, NCCL all-reduce on rank corresponding to that GPU timed out, the trainer aborted.

The action: drain <node>, schedule a power cycle, RMA the GPU if it does not come back clean. The user's training job needs to be restarted. None of this required SSH-ing to the node or guessing — it all came out of the metrics and logs you already had.

This is what good observability looks like: 90 seconds from "user paged" to "I know exactly what happened and what to do". For the runbook on how to actually drive a node reset, see operations/health-check-runbook.

What NOT to alert on

This list is longer than the list above and that is the point.

• Per-GPU utilization. The user's job is supposed to use the GPU.
• Per-GPU memory used. Same logic.
• Per-GPU temperature spike below 90 C. Transient, normal, ignore.
• Single XID 31, 43, 13 errors on a user pod. These are user bugs (illegal memory access, application crash). Aggregate across pods, alert on a rate, not a count.
• Network bandwidth utilization. Spiky during all-reduce, idle during forward pass — meaningless without context.
• Node CPU usage. GPU nodes are mostly CPU-idle.
• Single-flap IB port (recovers within 10 s). It is going to flap a few times. The fabric handles it.
• Kubernetes pod restarts in Evicted state on a draining node — that is the operator doing its job.
• Log volume going up. Sometimes the workload is just chatty.
• TLS cert expiring in 30 days. Send to a ticket, not a pager.
• Disk usage above 70%. Trigger cleanup at 80%, page at 90%.

The core principle: if the alert does not come with an action a human will take in the next 30 minutes, it is not a page. It is a ticket, a Slack message, or a row on a dashboard. Anything else is alert fatigue, and alert fatigue is how the real outage gets missed at 03:14 on a Tuesday.

Cardinality budget

A note on cost. Prometheus pricing is roughly linear in active time-series. A 256-node H100 cluster with DCGM exporter at default counters and one scrape per 15 s is roughly:

• 256 nodes × 8 GPUs × ~80 metrics/GPU = ~164k series from DCGM.
• 256 nodes × ~200 node-exporter metrics × 12 NICs = ~600k series from node-exporter (NIC dimension is the killer here; consider relabel-dropping unused interfaces).
• ~50k series from kube-state-metrics on a normally-loaded cluster.
• ~20k series from kubelet.

Total ~850k active series. That is fine on a single Prometheus with 64 GiB RAM and TSDB on local NVMe, with 30-day retention and remote-write to a long-term backend (Mimir, Thanos, or VictoriaMetrics) for anything older.

If you are at thousands of nodes you federate, downsample, or shard — see prometheus-stack.

Data-flow at a glance

The signal flow on a typical GPU cluster, to make the layering concrete:

GPU silicon
   │ NVML / DCGM API
   ▼
nv-hostengine (DaemonSet)            <─── nvidia-fabricmanager.service
   │                                       │ /var/log/fabricmanager.log
   ▼                                       │
dcgm-exporter (DaemonSet)                  │
   │ :9400/metrics                         │
   ▼                                       ▼
Prometheus (StatefulSet, 30d)         Promtail (DaemonSet)
   │   │                                   │
   │   └── remote_write ──┐                ▼
   │                      ▼              Loki gateway
   ▼                  Mimir / Thanos         │
Alertmanager (3 replicas)  (long term)       ▼
   │                                       Loki ingesters
   ├── PagerDuty                              │
   ├── Slack #alerts-gpu-ops                  ▼
   └── e-mail digest                       Object storage (S3/MinIO)
                                              │
   ┌──────────────────────────────────────────┘
   ▼
Grafana
   - Prometheus DS (metrics)
   - Loki DS (logs)
   - Tempo DS (traces, optional)
   - Alertmanager DS (annotations on graphs)

Every box in that diagram has a corresponding page later in this section. Read in order: prometheus-stack → dcgm-exporter → grafana-dashboards → logging-loki → alerting.

Events as a first-class layer

Events are not logs. They are discrete, irregular things-that-happened with high information content per occurrence. The four kinds of events that matter on a GPU cluster:

1. Kernel events — XID, mlx5 link transitions, OOM-kill, soft-lockup, machine-check. Originate in /dev/kmsg. Get them into Loki as {job="kmsg"}. See logging-loki.

2. DCGM events — dcgmi diag results, dcgmi health -c status, GPU policy violations. The dcgm-exporter does not emit these directly; you scrape them via a sidecar that runs dcgmi diag -j -r 1 periodically and posts the JSON to Loki. Worth doing on a 1-hour cadence as a "cluster-wide synthetic check".

3. Kubernetes events — kubectl get events. These are stored in etcd by default with a 1-hour TTL, so you have to ship them out if you want history. Tools: event-exporter (bitnami) → Loki, or kube-events-exporter → Prometheus (as counts).

4. Switch syslog — Mellanox switches export syslog; SN3700/SN4700/SN5600 all support logging host configuration. Ship to Loki as {job="switch-syslog", switch="..."}. Critical for catching link-down events on the switch side that the host might not see.

The way you use events is not to alert on a single one (with the small list of exceptions in the alert table above) but to alert on patterns:

• "Five XIDs of any kind across the fleet in 10 minutes" — see dcgm-exporter.
• "More than 3 mlx5 cable events in 1 hour" — fabric is unstable.
• "More than 2 OOM-kills in a system namespace in 1 hour" — see alerting.
• "Pod evicted by kubelet for MemoryPressure more than once on the same node in 24h" — node has a memory leak.

SLOs that actually mean something

Three SLOs we track on every GPU cluster. They map cleanly to what customers care about and they survive contact with reality.

1. Useful GPU hours per calendar hour

Definition:

sum(DCGM_FI_PROF_SM_ACTIVE > 0.5) / count(DCGM_FI_DEV_GPU_TEMP)

Read as: "what fraction of the fleet's GPUs are doing real work right now". A cluster running a healthy training schedule averages 70–85%. Below 50% sustained means the queue is empty or jobs are scheduled but stuck (GPU allocated but not utilizing — a frequent failure mode on misconfigured pods).

This is not a real-time alert. It is a weekly review metric. Customers ask "what is my cluster utilization" and this is the number.

2. Fabric availability

1 - (
  count(node_infiniband_state_id != 4)
  /
  count(node_infiniband_state_id)
)

Fraction of IB ports in Active state. Typical: 99.9% during normal ops, dipping to 99% during a known-bad-cable replacement. Below 99% sustained = a real fabric problem. Reported as a 99.x% over a rolling 7-day window.

3. NCCL goodput vs baseline

The most operationally meaningful SLO and the hardest to define correctly. We compute:

goodput_ratio = aggregate_fabric_throughput_now / 7d_p50_baseline

(See the recording rule in alerting.) When goodput_ratio < 0.7 for > 30 min, something has changed: a flaky cable, a misbehaving job hogging the fabric, a switch buffer issue. We track the time-integrated goodput as a daily SLO and alert on long-form regressions.

These are cluster-level SLOs. They are not what you put on the on-call dashboard — that is for tactical alerts. They are what you put on the weekly health-of-cluster review and what you report to the customer.

Common mistakes the first time you wire this up

A list, in rough order of how often I see them:

1. One Prometheus, one Grafana, no Loki. You can debug metric problems but not log problems. The first incident where you need to correlate "GPU thermal throttle" with "what was the pod doing in the dmesg ring buffer" sends you to the affected node with ssh and dmesg | grep -i nvrm. Loki should be there from day 1.

2. Default scrape interval at 60 s. DCGM samples some counters at 1 Hz internally. Scraping at 60 s gives you 60 s of latency on a thermal alarm, which can be 60 s of training output corrupted in HBM. Use 15 s for DCGM and node-exporter on GPU nodes.

3. No cluster: external label. You cannot tell metrics from cluster A apart from cluster B. Federation breaks. Multi-cluster Mimir / Loki returns ambiguous results. Set externalLabels: {cluster: "..."} on every Prometheus.

4. All alerts go to the same Slack channel. No severity routing, no team routing. A storm of info-level alerts buries the one P1. Use Alertmanager routing properly — see alerting.

5. No PrometheusRule in Git. Alert rules live in the Prometheus pod's local config, hand-edited. Nobody knows what is supposed to fire. Move them to PrometheusRule CRDs in a Git-managed repo.

6. Loki labels include pod_uid or request_id. Cardinality blows up. Loki gets slow. See logging-loki.

7. Default node-exporter on GPU nodes. No infiniband, no systemd collector. You miss every fabric event and every fabric-manager state transition. Configure both — see prometheus-stack.

8. No remote-write for long-term retention. Local TSDB at 30 days. The next incident review needs data from 45 days ago. Set up Mimir / Thanos / VictoriaMetrics from day 1, even if it is just a single-binary instance pointed at a small S3 bucket.

9. No watchdog for the alerting stack. Prometheus dies, Alertmanager dies, the on-call rotation goes silent and nobody notices for 4 hours. Always have a dead-man's-switch — see alerting.

10. Running NCCL_DEBUG=TRACE in production. A single training job at TRACE level produces 200k log lines/s and hits Loki's per_stream_rate_limit. Use INFO at most, or scope DEBUG via env var on a single test pod. See networking/nccl.

Operational hygiene checklist

A short list of the practices that separate a sustainable on-call from a burning one:

• One source of truth for alerts. PrometheusRule CRDs in Git, deployed via GitOps. Not Grafana managed alerts, not hand-edited YAML on the Prometheus pod.
• Every alert links to a runbook. No exceptions. If you cannot write a runbook, the alert is not ready.
• Quarterly noise audit. Top-20 firing alerts of the quarter; for each, decide: real, fix the underlying issue / bad threshold, raise it / bad alert, delete it.
• Dead-man's-switch. External system pages you when the alerting stack itself is down. See alerting.
• Recording rules for everything you query more than once. Grafana dashboards that re-derive rate() and topk() on every render are slow and burn Prometheus CPU.
• Storage on local NVMe. Both Prometheus TSDB and Loki ingester WAL.
• Externalize long-term storage to object store. Local SSD is for the working set; cold data goes to S3/Mimir/Thanos.
• Tag every cluster with cluster=... external labels. Otherwise multi-cluster Mimir / Loki dashboards become useless.
• Test alert routing. amtool config routes test before every change to Alertmanager config.
• Keep dashboards in Git too. ConfigMaps with grafana_dashboard: "1" label, picked up by Grafana sidecar. See grafana-dashboards.

See also

• monitoring/prometheus-stack — the metrics implementation.
• monitoring/dcgm-exporter — the GPU metric source.
• monitoring/grafana-dashboards — how to look at all of this.
• monitoring/logging-loki — pod logs, kmsg, slurm prolog/epilog.
• monitoring/alerting — Alertmanager, routing, and runbook discipline.
• drivers/fabric-manager — what FM is and why its health is a page.
• networking/perf-tuning — fabric-side knobs that show up in metrics.
• weka/overview — storage health signals worth watching.