gpu-knowledge

A working operator's reference for GPU/HPC clusters. Plus an interactive command emulator that teaches you how to read real diagnostic output.

Featured

Command Emulator

The fastest way to learn this material. Type real commands — nvidia-smi, ibstat, dcgmi diag, nccl-tests — and see annotated outputs from real H100 nodes. Trigger failure modes with !broken prefixes.

scenarios
100
multi-step diagnostic walkthroughs — hypothesis → command → output → takeaway
playbooks
100
symptom → commands cheat sheets, layered from cheapest to deepest check
solutions
100
copy-paste fix recipes, each with the rationale of why it works
Open the emulator →
$ scenario nccl-vendor-err-81
# Symptom: 16-GPU multi-node hangs on first all_reduce.

$ next
# Step 1/5 — Hypothesis: Is the IB fabric OK?
$ ibstat | grep -E "State:|Rate:"
        State: Active
        Rate: 400
# Takeaway: fabric fine at L2.

$ next
# Step 2/5 — peermem loaded?
$ lsmod | grep peermem
(no output)
# Takeaway: peermem missing. GDR cannot work.

What's inside

Drivers
NVIDIA kernel/runtime stack, fabric-manager, peermem, container-toolkit, Mellanox OFED. AMD ROCm coverage too.
Networking
RDMA / RoCE / InfiniBand / GPUDirect / NCCL multi-node tuning, NVLink/NVSwitch, PCIe topology, IB switch architecture.
Kernel & Tuning
GRUB cmdline, ACS, IOMMU, NUMA, hugepages, sysctl, ulimits, jitter isolation, transparent hugepages.
Kubernetes
RKE2, GPU Operator, Network Operator, Reservations, ArgoCD operations and SMP gotchas.
Slurm + SUNK
Controller / dbd / login pods, enroot + pyxis, multi-tenant, scheduling, job failure cookbook.
Storage / Weka
Architecture, CSI, troubleshooting, performance tuning, day-2 operations.
NVIDIA Stack
GPU generations (V100→H100→H200→B100/B200/B300), DCGM, MIG, NIM/NeMo/Triton, DOCA, NIXL, GPU sharing strategies.
AMD Stack
AMD Instinct lineup (MI300X / MI355X), ROCm, RCCL vs NCCL, K8s AMD GPU Operator.
Monitoring
Prometheus + DCGM exporter + Grafana + Loki + Alertmanager. Real configs, real alert rules.
Datacenter
Power, cooling, rack design, physical network. Real numbers from public reference architectures.
Operations
Triage decision tree, NCCL failure cookbook, incident response, tenant onboarding/offboarding, capacity, cost, vendor management.
Glossary
200+ acronyms decoded — what the letters mean and where they fit.

Tools

19 interactive utilities for triage and authoring: GPU node health checks, Weka mount diagnosis, ulimit analyzer, NCCL log parser, dmesg classifier, NVIDIA driver compatibility matrix, fio recipe generator, kubectl helper, Slurm job debugger, Grafana link builder, and more.

Browse tools →

Why this exists

New SREs joining an HPC team face a wide surface: NVIDIA driver compatibility matrices, Mellanox firmware/OFED pairings, kernel cmdline flags, multiple network fabrics (IB, RoCE, NVLink, NVSwitch), distributed schedulers (Slurm + SUNK + Kubernetes), and multi-tenant isolation. Most of that knowledge lives in tribal memory, vendor PDFs, or post-incident notes. This site consolidates it into a single readable reference, with an interactive environment for hands-on practice.

Every page leads with the why (what problem does this solve, what failure mode does it prevent) before the how (commands, configs, fixes).