Loki for GPU clusters: pod logs, kmsg, slurm prolog/epilog, DCGM events

A GPU cluster generates several distinct kinds of log streams: pod stdout/stderr, kernel ring buffer (where XID errors land), slurmd / slurmctld with their prolog and epilog scripts, fabric-manager's own log file, NCCL debug output, and the smart events from DCGM and the device plugin. They live in different places on the host, have wildly different volume profiles, and you need them all in one place when debugging a flaky training job.

Loki is the right tool for this. It is built around the same label model as Prometheus — small set of labels, full-text search inside each stream — which is exactly the model you want for HPC logs where you know in advance how you want to slice things (by node, by pod, by job, by namespace) and the rest is unstructured text.

This page covers the operator's deployment: how to ingest each log source, what labels to attach, how to query them with LogQL, retention policy, and the object-storage backend that keeps it from costing a fortune.

What lives where on a GPU node

Before you can ship logs you need to know what to ship.

On Kubernetes, container stdout/stderr is captured automatically by kubelet and ends up under /var/log/pods/. Everything else needs an agent that knows where to look.

Architecture choice: Promtail vs fluent-bit vs Vector

Three reasonable agents:

• Promtail — Grafana's reference Loki agent. Tightly integrated, supports the same label model, includes journal and syslog scrapes natively. Less performant than fluent-bit at high throughput but plenty for a 256-node fleet.
• fluent-bit — C-based, very low memory, good Kubernetes service discovery, supports many backends. Good if you also want to forward logs elsewhere (S3, Elasticsearch).
• Vector — Rust-based, expressive transforms (VRL). Newer, smaller community footprint, but solid.

We use Promtail. The ServiceMonitor / Helm-chart story is mature, the labels match Prometheus naturally, and the maintenance burden is lower. For very high-volume edge cases (a node spitting 200k log lines/sec — a misbehaving training job in NCCL_DEBUG=TRACE mode) we have considered fluent-bit but in practice the issue is the source, not the agent.

Loki components and storage

Loki runs in three flavors of deployment:

1. Monolithic / single-binary — one process, runs all components. Good for small clusters / dev / lab environments.
2. Simple Scalable Deployment (SSD) — read path and write path scale separately. Three replica sets: read, write, backend. Recommended for most production deployments.
3. Microservices — one StatefulSet/Deployment per component (distributor, ingester, querier, query-frontend, compactor, index-gateway, ruler, etc.). For large multi-tenant deployments where you need to tune each piece.

For a single GPU cluster, SSD mode is right.

Storage is object-storage-only since Loki 2.0. You drop in S3-compatible (MinIO, Ceph RGW, native S3) and Loki shards index and chunk objects there. Reasonable layout:

loki-bucket/
├── index/              (TSDB index files, ~10 GiB per month for our scale)
│   └── index_19782/
│       └── tenant1/
│           └── ...
├── chunks/             (compressed log blocks, the bulk of storage)
│   └── tenant1/
│       └── <shard>/
│           └── <chunk-id>
└── compactor-marker/

A typical sizing for a 256-node GPU cluster running training:

• ~20 GiB/day of pod stdout (assuming bounded NCCL_DEBUG)
• ~500 MiB/day of kmsg
• ~2 GiB/day of slurm logs
• Compressed in object storage at ~5×, that is ~5 GiB/day total.

With 30-day Loki retention you have ~150 GiB. With 90-day for compliance, ~450 GiB. Cheap on S3 standard, cheaper on Glacier-Instant for the older blocks.

Loki Helm install (SSD mode)

Install the chart:

helm repo add grafana https://grafana.github.io/helm-charts
helm install loki grafana/loki -f loki-values.yaml -n monitoring

A working loki-values.yaml for SSD mode against S3-compatible storage:

deploymentMode: SimpleScalable

loki:
  schemaConfig:
    configs:
      - from: 2024-01-01
        store: tsdb
        object_store: s3
        schema: v13
        index:
          prefix: index_
          period: 24h
  storage:
    type: s3
    bucketNames:
      chunks: loki-chunks
      ruler:  loki-ruler
      admin:  loki-admin
    s3:
      endpoint: s3.observability.internal:9000
      region: us-east-1
      accessKeyId: ${S3_ACCESS_KEY}
      secretAccessKey: ${S3_SECRET_KEY}
      s3ForcePathStyle: true
      insecure: false
  limits_config:
    retention_period: 30d
    reject_old_samples: true
    reject_old_samples_max_age: 168h
    max_query_length: 721h
    max_query_parallelism: 32
    max_streams_per_user: 50000
    ingestion_rate_mb: 50
    ingestion_burst_size_mb: 75
    per_stream_rate_limit: 10MB
    per_stream_rate_limit_burst: 30MB
  compactor:
    retention_enabled: true
    retention_delete_delay: 2h
    delete_request_store: s3

write:
  replicas: 3
  persistence:
    storageClass: local-nvme
    size: 50Gi
  resources:
    requests:
      cpu: "1"
      memory: 6Gi
    limits:
      memory: 8Gi

read:
  replicas: 3
  resources:
    requests:
      cpu: "1"
      memory: 4Gi
    limits:
      memory: 6Gi

backend:
  replicas: 3
  persistence:
    storageClass: local-nvme
    size: 20Gi

gateway:
  enabled: true
  replicas: 2

monitoring:
  selfMonitoring:
    enabled: false
  serviceMonitor:
    enabled: true
    labels:
      release: kube-prometheus-stack

A few things people miss the first time:

• schema: v13 and store: tsdb is the modern path. Older docs reference store: boltdb-shipper — don't.
• compactor.retention_enabled: true is what actually deletes old chunks. Without it, retention is "ignore old data on query" and chunks accumulate forever in S3.
• max_query_length: 721h (30 days + 1) is required; without it queries on 30-day windows fail.
• Both per_stream_rate_limit and ingestion_rate_mb matter. The first is per-stream (per unique label set), the second is global for the tenant. A single misbehaving pod hits the per-stream limit first; a fleet-wide spike hits the global limit.

Promtail install: pod logs

Promtail as a DaemonSet that mounts /var/log/pods and /var/lib/docker/containers (containerd runtimes use the former only):

# promtail-values.yaml
config:
  clients:
    - url: http://loki-gateway.monitoring.svc:80/loki/api/v1/push

  positions:
    filename: /run/promtail/positions.yaml

  scrapeConfigs: |
    # 1. Pod logs from /var/log/pods, with full kubernetes_sd
    - job_name: kubernetes-pods
      kubernetes_sd_configs:
        - role: pod
      pipeline_stages:
        - cri: {}
        - drop:
            expression: '^\s*$'
        # Drop NCCL_DEBUG INFO lines unless DEBUG=TRACE on the pod
        - match:
            selector: '{nccl_debug!="TRACE"}'
            stages:
              - drop:
                  expression: 'NCCL INFO (Net|Channel|Comm)'
                  drop_counter_reason: nccl_info_noise
      relabel_configs:
        - source_labels: [__meta_kubernetes_pod_controller_name]
          target_label: controller
        - source_labels: [__meta_kubernetes_namespace]
          target_label: namespace
        - source_labels: [__meta_kubernetes_pod_name]
          target_label: pod
        - source_labels: [__meta_kubernetes_pod_container_name]
          target_label: container
        - source_labels: [__meta_kubernetes_pod_node_name]
          target_label: node
        - source_labels: [__meta_kubernetes_pod_label_app]
          target_label: app
        - source_labels: [__meta_kubernetes_pod_label_team]
          target_label: team
        - source_labels: [__meta_kubernetes_pod_label_job_id]
          target_label: job_id
        - action: replace
          replacement: /var/log/pods/*$1/*.log
          source_labels: [__meta_kubernetes_pod_uid, __meta_kubernetes_pod_container_name]
          regex: (.+);(.+)
          target_label: __path__

    # 2. Kernel ring buffer (XID, mlx5, OOM)
    - job_name: kmsg
      static_configs:
        - targets:
            - localhost
          labels:
            job: kmsg
            node: ${HOSTNAME}
            __path__: /dev/kmsg
      pipeline_stages:
        - regex:
            expression: '^(?P<priority>\d+),(?P<seq>\d+),(?P<usec>\d+),(?P<flags>[a-z-]+);(?P<message>.*)$'
        - labels:
            priority:
        - timestamp:
            source: usec
            format: UnixMs
        - output:
            source: message

    # 3. Systemd journal (kubelet, containerd, fabric-manager, slurmd)
    - job_name: journal
      journal:
        json: false
        max_age: 12h
        path: /var/log/journal
        labels:
          job: systemd-journal
          node: ${HOSTNAME}
      relabel_configs:
        - source_labels: [__journal__systemd_unit]
          target_label: unit
        - source_labels: [__journal_priority]
          target_label: priority

    # 4. Fabric-manager log file
    - job_name: fabric-manager
      static_configs:
        - targets:
            - localhost
          labels:
            job: fabric-manager
            node: ${HOSTNAME}
            __path__: /var/log/fabricmanager.log

    # 5. Slurm prolog/epilog stdout
    - job_name: slurm-prolog-epilog
      static_configs:
        - targets:
            - localhost
          labels:
            job: slurm-prolog-epilog
            node: ${HOSTNAME}
            __path__: /var/log/slurm/{prolog,epilog}/*.log

extraVolumes:
  - name: dev-kmsg
    hostPath:
      path: /dev/kmsg
  - name: var-log-fm
    hostPath:
      path: /var/log/fabricmanager.log
  - name: var-log-slurm
    hostPath:
      path: /var/log/slurm
  - name: var-log-journal
    hostPath:
      path: /var/log/journal

extraVolumeMounts:
  - name: dev-kmsg
    mountPath: /dev/kmsg
    readOnly: true
  - name: var-log-fm
    mountPath: /var/log/fabricmanager.log
    readOnly: true
  - name: var-log-slurm
    mountPath: /var/log/slurm
    readOnly: true
  - name: var-log-journal
    mountPath: /var/log/journal
    readOnly: true

env:
  - name: HOSTNAME
    valueFrom:
      fieldRef:
        fieldPath: spec.nodeName

tolerations:
  - operator: Exists

Notes:

• The cri: {} pipeline stage parses the CRI log format that containerd writes. Without it you get raw 2024-01-01T12:00:00Z stdout F your log line entries.
• The kmsg stage uses the kernel's /dev/kmsg interface, which is plain text with a structured prefix. Tailing this rather than journalctl -k gives you faster ingestion and lower CPU on the agent.
• Slurm prolog/epilog logs are vital. When a job fails at startup with "permission denied" or "GPU device not found", the kmsg trace is in /var/log/slurm/prolog/. Ship them.

Cardinality discipline

Resist the temptation to add labels. Every unique combination of label values creates a new stream in Loki, and stream count is the number-one driver of Loki cost and slow queries. Specifically:

• Do not label by pod_uid. UIDs are unique per pod creation and explode cardinality.
• Do not label by request_id or anything per-request.
• Do not label by user error message text.

The labels we use, in total: cluster, namespace, pod, container, node, app, team, job_id, unit, job (the Promtail scrape job), priority. That is ~11 labels and the cross-product is bounded.

Anything dynamic — log levels parsed from JSON, request IDs, error codes — goes inside the log line and you query it with |= or | json. Labels are for streams, not for facts.

LogQL patterns for incident response

LogQL syntax: {stream selector} | log pipeline | aggregations. The stream selector is mandatory; everything else is optional.

Find a specific XID across the fleet

{job="kmsg"} |~ "Xid \\(PCI:[0-9a-f:.]+\\): 79"

This streams every XID 79 line from every node. Good for "did anyone else see this in the last hour?".

Pod logs filtered to NCCL errors

{namespace="ml-team",pod=~"train-foo.*"} |~ "(?i)nccl.*(error|fail|abort|warn)"

The (?i) is regex case-insensitive, important since NCCL is inconsistent about casing.

Slurm job failures correlated with kmsg

{job="systemd-journal",unit="slurmd.service",node=~"$node"}
  |~ "JobId=$jobid"
  |= "FAILED"

Then a second panel:

{job="kmsg",node=~"$node"} |~ "Xid|mlx5|nvidia"

Side by side in Grafana with the same time range tells you in 30 seconds whether the slurm failure correlated with a kernel-level GPU/NIC event.

Aggregations: alert-feeding metrics from logs

You can convert log streams into Prometheus metrics:

sum by (node) (
  count_over_time(
    {job="kmsg"} |~ "Xid \\(PCI:[0-9a-f:.]+\\): 79"
    [5m]
  )
)

This becomes the basis for an alert that fires when XID 79 appears in dmesg even if DCGM exporter didn't catch it (e.g., the GPU is so dead DCGM lost the connection). Define it as a Loki recording rule or as a Grafana managed alert.

JSON-structured logs from training scripts

If your training scripts log JSON:

{namespace="ml-team",app="trainer"} | json | level="ERROR"

The | json parser extracts JSON keys into temporary labels (not stream labels), so you can filter on level without it being part of the stream identity. This is the right way to handle high-cardinality fields.

LogQL with Prometheus-style rate

rate({job="kmsg",node=~"$node"} |~ "mlx5.*event"[5m])

Returns events-per-second of mlx5 events on that node. Plot it next to NIC error counters from Prometheus and you have a fully correlated view of fabric health.

Recording rules and alerts in Loki

Loki has its own ruler that evaluates LogQL rules just like Prometheus. Use it for log-driven alerts:

# loki-rules.yaml, mounted into the ruler pod or written to S3 ruler bucket
groups:
  - name: gpu-log-alerts
    interval: 1m
    rules:
      - alert: KernelXIDInDmesg
        expr: |
          sum by (node) (
            count_over_time(
              {job="kmsg"} |~ "NVRM: Xid \\(PCI:[0-9a-f:.]+\\): (79|48|74)"[5m]
            )
          ) > 0
        for: 0s
        labels:
          severity: critical
          team: gpu-ops
          source: loki
        annotations:
          summary: "Critical XID seen in kmsg on {{ $labels.node }}"

      - alert: FabricManagerErrors
        expr: |
          sum by (node) (
            count_over_time({job="fabric-manager"} |~ "ERROR"[5m])
          ) > 5
        for: 2m
        labels:
          severity: warning
          team: gpu-ops

      - alert: NCCLAbortStorm
        expr: |
          sum by (namespace) (
            count_over_time(
              {namespace=~".+",container="trainer"}
              |~ "NCCL.*(abort|hang|timeout)"
              [10m]
            )
          ) > 20
        for: 5m
        labels:
          severity: warning
          team: ml-platform
        annotations:
          summary: "NCCL aborts in {{ $labels.namespace }} > 20 in 10m"

The Loki ruler sends alerts to Alertmanager (you configure it the same way as Prometheus). Routing is in alerting.

Retention strategy

Three classes of retention typically:

Configure per-stream retention with Loki's retention_stream:

limits_config:
  retention_period: 30d         # default
  retention_stream:
    - selector: '{job="kmsg"}'
      priority: 1
      period: 90d
    - selector: '{job="fabric-manager"}'
      priority: 1
      period: 90d
    - selector: '{namespace=~"ml-team|customer-.*"}'
      priority: 2
      period: 14d

Higher priority wins ties. The default applies if no selector matches.

Multi-tenancy

If you run multiple teams or a managed-multi-tenant cluster, set Loki to multi-tenant mode and have each Promtail / agent set the X-Scope-OrgID header per push:

clients:
  - url: http://loki-gateway.monitoring.svc:80/loki/api/v1/push
    tenant_id: gpu-cluster-1

Prometheus, Grafana, and Loki all support multi-tenancy via the same header. Tenants can be:

• One per cluster (so a multi-cluster Mimir/Loki has tenant-per-cluster).
• One per business unit (when you want hard isolation between teams).
• One per environment (prod/staging).

We use one per cluster; per-team isolation is enforced via Grafana role-based dashboards.

Operational notes

Promtail position file

Promtail tracks file offsets in /run/promtail/positions.yaml. If the file is on tmpfs and the pod restarts, you re-read every log file from the beginning. This is normally fine (kmsg has a small ring buffer; pod logs rotate at 10 MiB) but at scale it can produce a brief spike of duplicate ingestion. Mount the positions file on a hostPath PV if you want continuity across restarts.

Out-of-order log entries

Loki's default rejects entries with timestamps older than the last entry in a stream. NCCL prints multi-line stack traces where each line carries the original timestamp, which can confuse parsers. The fix is accept_out_of_order_writes: true (Loki ≥ 2.4) but it costs a bit on the indexer. We leave it on for {job="kmsg"} only.

Label parser order

Promtail's pipeline stages apply in order. cri: {} must come before any regex: stage that operates on the message body. If you reverse them you get the raw CRI envelope and the regex never matches.

Large log lines

Loki truncates lines > 256 KiB by default. NCCL with NCCL_DEBUG=TRACE produces lines that hit this. Bump max_line_size to 1 MiB in limits_config:

limits_config:
  max_line_size: 1048576
  max_line_size_truncate: false

Or, better, do not run NCCL TRACE in production. See networking/nccl for what to actually set.

Object storage cost

S3 PUT pricing is the surprising line item. Loki batches writes per chunk (default ~5 minutes per stream, ~1 MiB per chunk), so writes are bounded — but if you have lots of low-volume streams you can end up with many small chunks. Tune chunk_target_size and chunk_idle_period based on your actual stream count.

Compactor lag

The compactor merges and compacts old index files. If it falls behind, queries on older time ranges get slow. Watch loki_compactor_pending_delete_requests and the compactor pod's CPU. We give it a single dedicated pod with 2 CPU and 4 GiB.

See also

• monitoring/overview — what to log and what to alert on.
• monitoring/prometheus-stack — the metrics companion.
• monitoring/dcgm-exporter — DCGM metrics, for log-metric correlation.
• monitoring/grafana-dashboards — embedding Loki panels next to metric panels.
• monitoring/alerting — Alertmanager config that consumes both Loki and Prometheus alerts.
• networking/nccl — what NCCL_DEBUG knob to use without flooding Loki.
• drivers/fabric-manager — what /var/log/fabricmanager.log actually contains.
• slurm-sunk/sunk-troubleshooting — where slurm prolog/epilog logs become forensic evidence.