Tools
Interactive utilities for triage, validation, and authoring. Inspired by an internal support-tooling site, ported and generalized.
GPU node health
Multi-check on a GPU node: nvidia-smi, NVLink, ECC, PCIe link speed/width, fan/thermal, persistence mode, fabric-manager status.
Weka mount checker
Validate a Weka mount: cluster reachability, drivers-loader, wekafsio refcount, /proc/mounts entries, common stale-mount fixes.
ulimit analyzer
Walk through /etc/security/limits.conf, systemd unit overrides, and the effective ulimits for kubelet, containerd, slurmd, and user processes.
Pod inspector
Cheat sheet for diagnosing a stuck pod: events, scheduler reasons, init containers, volume mount errors, scheduler webhook denials.
Slurm job debugger
Step-by-step recipe to debug a stuck or failed Slurm job: scontrol show, NodeFail/NodeNotResponding, prolog/epilog hooks, enroot/pyxis errors.
Node speed benchmark
Standard mini-benchmark suite for a fresh GPU node: nccl-tests all-reduce, ib_send_bw / ib_read_bw / ib_write_bw across rails, fio against local NVMe and shared FS.
kubectl helper
Common kubectl one-liners: drain, cordon, taint, debug ephemeral container, evict, exec into init container, port-forward, cp, etc.
fio VAST tool
fio recipes tuned for VAST: throughput, IOPS, mixed workloads, multi-job, target queue depth, AIO vs libaio vs io_uring.
Cross-DC rsync
rsync with multi-stream, bandwidth limits, partial-resume, checksum-mode, sshpass alternatives, performance profiling for inter-datacenter copies.
Grafana link generator
Build deep links to Grafana dashboards with time range, variables, and node selectors prefilled — useful for paging on-call to the right view.
Slurm commands
sinfo / squeue / scontrol / scancel / sacct cheat sheet with the most useful flag combinations.
Slurm job submission
Templates for sbatch scripts: GPU jobs, multi-node MPI, enroot+pyxis, Slurm + container runtime, common pitfalls.
Speed test commands
curl, iperf3, ib_send_bw, perftest one-liners for measuring network performance from inside a pod or on the bare metal node.
Tailscale
Tailscale on bare metal hosts: subnet routes, exit nodes, ACLs, MagicDNS, troubleshooting.
Markdown viewer
Paste a markdown blob and render it. Useful when ops runbooks live in private repos / Slack and you want a quick clean view.
Prompt library
Curated prompts for LLMs to assist with operational tasks: incident summary, runbook drafts, diff explanation, postmortem structure.