Prompt library

Investigate pod restarts for the pods listed below and determine root cause and whether resources need to be increased.

Pods: <paste pod names>

For each pod:
1. Run `kubectl get pods <pod> -o wide` to find the node and restart count.
2. Run `kubectl describe pod <pod>` to find which container is restarting, exit code, and reason (OOMKilled, Error, Completed, etc.).
3. Get previous container logs with `kubectl logs <pod> -c <container> --previous` to find the actual error.
4. Check node headroom with `kubectl describe node <node>` — look at Allocated resources for CPU/memory requests and limits vs capacity.

Then summarize:
- Per pod: which container restarted, why (OOMKilled / DNS failure / liveness probe / cascade), what fix is needed.
- Node headroom: is there enough free memory and CPU to safely raise limits?
  - Memory: safe to raise if requests < 80 % of allocatable.
  - CPU: risky to raise requests if already > 85 % utilized.
- Recommended action: resource bump, infra fix (DNS, dependency), or no action.

A multi-node training job is hanging on NCCL. I will paste the job's stdout/stderr and a few diagnostics. Diagnose the root cause and propose the fix.

Inputs I'll provide:
- Slurm/k8s job spec (NCCL_* env vars + nodelist)
- The first 200 lines of NCCL_DEBUG=INFO output
- `ibstat` from one node
- `nvidia-smi topo -m` from one node
- `ip link show` from one node

For your diagnosis, walk through:
1. Did NCCL pick the right rails? Look for `NCCL INFO NET/IB : Using` lines — confirm one IB device per GPU.
2. Are the IB devices Active at the right rate?
3. Did the job get stuck in Bootstrap (TCP) vs in a collective (RDMA)? Bootstrap stalls usually mean OOB hostname/DNS issues, collective stalls usually mean PFC/ECN or wrong topology.
4. Is `NCCL_IB_HCA` filtered to only the GPU-rails (no mgmt iface)?
5. Is `NCCL_TOPO_FILE` set if topology is non-default?

Output a short summary with the most likely cause first and a one-line fix.

I need to upgrade NVIDIA driver from <current> to <target> on <N> nodes in cluster <cluster>. Help me write a change-mgmt plan.

Cover:
1. Pre-flight: verify target driver supports our kernel and GPU SKU, list any DKMS modules that will rebuild (peermem, nvidia-fabricmanager, gpfs, etc), confirm fabric-manager version matches driver.
2. Rollout strategy: rolling vs all-at-once, drain order, max-surge, batch size.
3. Per-node steps: cordon, drain, run installer, verify, uncordon.
4. Verification: nvidia-smi, dcgmi diag -r 1, fabric-manager status, NCCL all-reduce on a single node.
5. Rollback: how to pin previous version with apt-mark hold; what to do if DKMS fails.
6. Comms: what to put in #infra-changes channel, expected duration.

Keep it terse — bullet points, no fluff.

I will paste a Slack thread and timestamps from an incident. Produce a concise incident summary in this format:

**Title**: <short>
**Detected**: <timestamp + how>
**Resolved**: <timestamp>
**Customer impact**: <what they saw>
**Root cause**: <one paragraph, technical>
**Fix**: <what we did, including blast radius>
**Action items**: <bullets — owner, ETA>

Tone: factual, no blame, no marketing fluff. Skip unrelated chatter.

I'll paste a unified diff. Explain it as if briefing an on-call engineer who has to approve it.

Output:
1. **What changes** — 1-2 sentences, in plain English.
2. **Why** — best inferred reason from context.
3. **Blast radius** — does this touch hot paths, RBAC, networking, storage?
4. **Reversibility** — is this a config flip (cheap) or a state migration (expensive)?
5. **What to verify post-deploy** — concrete signals.

Skip nice-to-have nits. Focus on whether it's safe to merge.

Generate a blameless postmortem from the notes I paste. Use this structure:

# <Incident title>

## Summary
<2-3 lines>

## Impact
<who saw what, for how long>

## Detection
<how we found out, MTTD>

## Timeline (UTC)
<bullet list>

## Root cause
<technical, no blame, contributing factors as nested bullets>

## What went well
<bullets>

## What went badly
<bullets, no names>

## Action items
| Item | Owner | Due | Type |
|------|-------|-----|------|
<rows; type = mitigation / detection / process>

Keep it under 500 words.