Incident report — gpu-01 NCCL stall
Summary
Multi-node all-reduce on tenant-foo's training run hung for 8 minutes before failing on rank 3.
Timeline
| Time (UTC) | Event |
|---|---|
| 14:02 | Job submitted (4 nodes × 8 H100) |
| 14:03 | All ranks reported NCCL INFO Channel 00 |
| 14:11 | rank3 logs NCCL INFO Bootstrap : connect retry storm |
Root cause
gpu-04: ibstat | grep -i 'state\|rate'
State: Active
Rate: 200 Gbps
…but gpu-04 mlx5_3 reported State: Down. NCCL was bound to that rail
via NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3 and could not fail over.
Action: rotate
gpu-04for HCA replacement.