Dataloader runaway + Weka TCP fallback: GPUs idle for 24h

Symptom

Two PSI alerts fired on a DGX H100 GPU node:

HostMemoryPressureHigh
   memory pressure: wait 86.6% · stall 82.2% · avail 45.8% · majfault 85.8/s · swap 0.0/s
HostCPUPressureHigh
   CPU pressure: 82.8% waiting · load 1027

GPU node count and ready states were fine — arc: 32 assigned · 32 ready · 0 degraded. So at first glance, this looked like a noisy host metric, not a workload problem. It wasn't.

What the metrics tell you before logging in

The combination "no RAM pressure, no swap, but 85 majfaults/s and load = 5x core count" is page-cache thrashing: the kernel is constantly re-faulting file-backed pages because reads aren't keeping up with the working set. On a GPU node, the only backing store with that latency profile is the parallel filesystem (here: Weka).

Layer-by-layer triage

1. K8s view — node Ready, conditions clean

kubectl describe node <gpu-N> | sed -n '/Conditions:/,/^Addresses:/p'

MemoryPressure       False   KubeletHasSufficientMemory
DiskPressure         False   KubeletHasNoDiskPressure
PIDPressure          False   KubeletHasSufficientPID
Ready                True    KubeletReady
SlurmRunning         True    BecameActive

Important: kubelet does not report MemoryPressure, even with PSI memory at 86%. Kubelet's eviction signal uses MemoryAvailable, which here is 45.8% — far above any eviction threshold. That's why no pods got evicted; they're just slow.

2. What's running on the node

kubectl get pods -A --field-selector spec.nodeName=<gpu-N> -o wide

A single tenant pod from tenant-foo was holding the 8-GPU reservation; everything else was system DaemonSets (gpu-operator, weka-client, monitoring).

3. Process-level snapshot

kubectl debug node/<gpu-N> --image=ubuntu --profile=sysadmin gives you a privileged container in the host namespace. Then chroot /host bash:

cat /proc/loadavg
# 1051.67 1039.17 1021.87 772/8713 1636495

Load near 1000+ on a 224-core box, with 8713 total tasks. That's the cluster equivalent of "the office is on fire."

ps -eo state,pid,uid,pcpu,rss,comm --sort=-pcpu --no-headers | head -20

R   67855     0  100 2962048 wekanode
R   67863     0  100 2958636 wekanode
R   67858     0  100 2964384 wekanode
R   67862     0  98.1 2963968 wekanode
S 1066219  3723 88.5 9351836 python3
S 1066215  3723 88.4 9515580 python3
...
R 1087254  3723 51.8 10034708 pt_data_worker
R 1087245  3723 51.6 9885620 pt_data_worker
R 1087283  3723 51.4 9891920 pt_data_worker
...

Two patterns emerge:

• 4 wekanode threads pinned at 100% CPU. That's the user-mode Weka client. Saturated.
• 8 master Python procs at ~88% CPU each — one per GPU rank (torchrun --nproc_per_node 8).
• Many pt_data_worker at ~51% CPU each — PyTorch DataLoader workers.

Counts:

ps -eo uid --no-headers | sort | uniq -c | sort -rn | head

   2519     0          # root system processes
   1004  3723          # tenant user — !!!
     28 65535

1004 processes belonging to one tenant user. That's not normal.

ps -eo state | awk '$1=="D"' | wc -l
# 284

284 processes in D-state (uninterruptible I/O wait). All of them stuck in folio_wait_bit_common — the kernel function that waits for a page-cache folio writeback bit to clear. They're all blocked on the same Weka filesystem.

4. GPU activity — pinned but idle

nvidia-smi --query-gpu=index,utilization.gpu,memory.used,power.draw --format=csv

0, 0 %, 76029 MiB, 119.72 W
1, 0 %, 80577 MiB, 119.69 W
2, 0 %, 71229 MiB, 118.25 W
3, 0 %, 78793 MiB, 117.43 W
... (all 8 GPUs identical pattern)

Zero util on every GPU, but each is holding 71-80 GB of model weights and consuming 117-122 W (well above the ~70 W deep-idle baseline). The model is loaded; CUDA contexts are attached; nothing is actually running compute.

nvidia-smi pmon -c 3 -s u

# gpu  pid       type sm mem  command
   0   1066213    C   0  0   python3
   1   1066214    C   0  0   python3
   2   1066215    C   0  0   python3
   ...

sm=0 mem=0 across three samples = the GPUs really are doing nothing. The training step never advances because the dataloader pipeline never feeds it.

5. Weka transport — the second bug

grep -E "rdma_(read|write)" /proc/wekafs/stat
#       rdma_read:    0    0 (0 IOPS)
#       rdma_write:   0    0 (0 IOPS)

Lifetime RDMA reads/writes = 0. This client has been running for 20+ days and has never used RDMA. All Weka traffic is going via the kernel/userspace fallback path. The client is configured with an mlx5_8/vlan1032 RDMA device, but the data plane never picked it up. This multiplies read latency by 5-10x.

Read-latency average from /proc/wekafs/stat: 42 ms — versus the ~1 ms you'd expect on RDMA. Tail (max) read latency: 9.6 seconds.

Root cause

Two compounding failures:

Failure 1 — Tenant dataloader fan-out

Process tree per master rank:

ps -eo uid,pid,ppid,comm --no-headers | awk '$1==3723 && $4=="pt_data_worker"' | awk '{print $3}' | sort | uniq -c
   124 1066213
   124 1066214
   124 1066215
   124 1066216
   124 1066217
   124 1066218
   124 1066219
   124 1066220

Each master rank spawned 124 dataloader workers. 8 ranks × 124 = 992 workers, plus 8 masters = 1000 procs.

But the YAML config (entered the process's mount namespace via nsenter -t <pid> -m) explicitly says:

trainer:
  batch_size: 1
  num_workers: 4

num_workers: 4 per dataloader. So 124 workers per rank is way over what the user asked for. The multiplicative blow-up is from the framework's data pipeline — likely a combination of:

• preload_cache_into_memory: true triggering a one-shot parallel cache priming with multiprocessing.Pool(cpu_count // 2) (≈112 workers on a 224-core box)
• Plus the configured 4 DataLoader workers per dataloader
• Plus persistent workers held across train/val/test iterators

Whatever the exact mechanism, the observable is that one user task produced 1000 processes on a node sized for one user task.

Failure 2 — Weka client never used RDMA

Independently of the tenant's dataloader, the per-host Weka client should have been using RDMA over mlx5_8. It wasn't. With 1000 readers hammering through the slower kernel/userspace path, the four wekanode threads handling Weka I/O saturated immediately. Once the client side saturated, every dataloader worker stalled in folio_wait_bit_common waiting for page-cache writeback.

Compounding: GPUs starved for data → step-time → ∞ → the job sits at 0% util with the model loaded → tenant pays for 8 H100s reserved for 19 days while doing zero useful work.

How to reproduce the diagnosis on any node

This is the playbook for "PSI alert fires but kubelet says node is healthy":

# 1. Get into the host namespace
kubectl debug node/<gpu-N> --image=ubuntu --profile=sysadmin -- chroot /host bash

# 2. Confirm load + PSI
cat /proc/loadavg
cat /proc/pressure/{cpu,memory,io}

# 3. Find who's burning the box
ps -eo uid --no-headers | sort | uniq -c | sort -rn | head
ps -eo pid,uid,pcpu,pmem,rss,comm --sort=-pcpu --no-headers | head -20

# 4. D-state inventory — which kernel function are they stuck on?
ps -eo state,wchan --no-headers | awk '$1=="D"' | sort | uniq -c | sort -rn | head

# 5. GPU "are they actually doing anything" check
nvidia-smi --query-gpu=index,utilization.gpu,memory.used,power.draw --format=csv
nvidia-smi pmon -c 3 -s u

# 6. Distributed FS transport mode
grep -E "rdma_(read|write)" /proc/wekafs/stat 2>/dev/null
mount | grep -E "weka|nfs|lustre"

# 7. Read the user's actual config (mount-namespace-aware)
PID=<one-of-the-master-pids>
nsenter -t $PID -m -- cat /<path-to-yaml>

What to communicate

Two separate conversations, because the two bugs have different owners:

To the tenant (low risk, no impact on running job):

Your training job has been running for 24h with all 8 GPUs at 0% util while consuming a full node. The job has spawned ~1000 dataloader worker processes on a 224-core node — far more than the num_workers: 4 your config requests. The masters are pinned to GPUs but the dataloader pipeline never delivers a batch, so steps don't advance. Likely a preload_cache_into_memory interaction with your custom dataset class. Could you check the data pipeline before re-launching?

To the storage provider (low risk):

Weka client on <host> has lifetime rdma_read=0 / rdma_write=0 since deployment. Client configured with mlx5_8/vlan1032 as RDMA device but never used it. Data plane has been on the kernel/userspace fallback path for 20+ days, with read p99 latency ~9.6s and avg 42 ms. This will affect every tenant on this fabric. Could you verify the RDMA initialization path?

Lessons

1. Kubelet's MemoryPressure ≠ PSI memory. Kubelet uses MemoryAvailable; PSI captures the more interesting "are processes actually waiting" signal. PSI alerts are necessary because kubelet alone won't flag this state.

2. GPU 0% util + memory holding ≠ idle GPU. It means a process is resident but starving. nvidia-smi alone doesn't distinguish "I'm working" from "I'm waiting for data forever." nvidia-smi pmon -c N -s u over multiple samples does.

3. Process count by uid is the cheapest "is the tenant misbehaving" signal. One pipeline showing 1000+ procs of a single user uid on one node = something multiplied that shouldn't.

4. D-state wchan analysis tells you where the bottleneck is. All 284 D-state procs in folio_wait_bit_common = they're all waiting on the page cache. Combined with majfault 85.8/s = the page cache can't keep up with the read demand, which means the underlying filesystem is slow, which means dig into the FS transport.

5. Distributed FS transport mode is invisible until it goes wrong. lifetime rdma_read=0 on a Weka client is the kind of silent misconfig that survives weeks of "everything looks fine" because nobody opens /proc/wekafs/stat until performance falls apart.

6. The right action here is NOT to drain the node. Cordoning + draining would migrate the misbehaving job to another node and reproduce the problem there. It's the workload that's wrong, not the host.

See also

• Triage decision tree
• GPU pod failures
• RDMA debugging
• Weka troubleshooting
• Weka performance tuning
• PSI / pressure stall information (kernel.org)