Incident response — cadence, communication templates, postmortems

A GPU cluster incident is not a software outage in the SaaS sense. The blast radius is a tenant's training run, which represents real-money compute time and a deadline. The communication needs to be precise, the containment needs to be fast, and the postmortem needs to give the customer something useful.

This page is the cadence we follow, with the templates we send to customers and providers, and the structure of a postmortem that's actually worth writing.

The cadence

1. Identify         (T+0 to T+5 min)    — what's broken? blast radius?
2. Contain          (T+5 to T+30 min)   — stop new damage; ringfence the bad component
3. Communicate      (continuous)        — first ack within 15 min, updates every hour
4. Fix              (T+30 to T+resolve) — root cause + remediation
5. Postmortem       (within 5 days)     — what happened, why, prevention

Steps 1-3 happen in parallel. Don't block fixing on writing the perfect comms — drop a "we're investigating, will update in 30 min" and get back to the keyboard.

Step 1: Identify

Five questions in order:

The fifth question is the most often skipped and the most often the answer. "Nothing changed" is sometimes true, often not. Audit recent ArgoCD syncs, recent kernel/driver bumps, recent firmware updates, recent customer onboardings (which can reshape Reservation labels).

Step 2: Contain

Containment is preventing the issue from spreading or destroying more data. Concrete actions:

Containment buys time for the actual fix. It does not need to be permanent. Cordoning a single node for 30 minutes while you decide whether it's a software or hardware issue is fine.

Step 3: Communicate — customer comms template

The first comms goes out within 15 minutes of the incident being declared, even if you don't know the cause. Be terse, factual, and specific about what's affected.

First ack template

We've identified an issue affecting <specific symptom — e.g., GPU node gpu-01
in your cluster> as of <UTC time>. We're investigating now.

Affected: <namespaces / job IDs / specific resources>
Impact: <what the customer sees>
Workaround: <if any — usually "we're routing around"; otherwise "none, please pause">
Next update: <time, no later than 1h from now>

Hourly update template

Status update on the <issue> from <time>:

What we know: <the actual cause, if found, or what's been ruled out>
Current state: <what's working / not working>
ETA: <a real estimate or "still investigating">
Next update: <time>

Resolution template

The issue is resolved as of <UTC time>.

Cause: <one-paragraph plain-English summary>
Fix: <what we did>
Affected window: <T-start to T-resolved>
Affected jobs: <list, or "your jobs <IDs>">

We'll send a postmortem within 5 business days with full timeline and remediation.

Tone notes: no marketing copy, no "we apologize for the inconvenience" boilerplate, no AI-generated padding. The customer wants the facts; deliver them. Include actual job IDs and timestamps the customer can verify against their own logs.

Step 3 continued: Provider comms template

When the cause is upstream (Weka, NVIDIA, vendor, datacenter), open a ticket with the provider. Plain English prose, no markdown headers, IPs + servers + ask in the first paragraph.

Hi,

We're seeing <symptom> on <cluster> in <DC> starting <UTC time>.
The affected hosts are <gpu-01, gpu-02, gpu-03> on <vlan/subnet>, talking to
<your service> at <endpoint>. Tenant <X> is unable to <action>.

We've already verified <what we ruled out — driver versions, link state, etc.>.
Logs from <client>, <backend>, <switch> attached.

Could you (a) confirm whether you're seeing the issue from your side,
and (b) suggest the next diagnostic step?

Cluster ID: <id>, support contract: <id>, urgency: P1 / P2.

Thanks,
<name>, <role>

Avoid bullet lists, headers, "we are seeing the following symptoms:". The provider's engineer reads dozens of these a day; prose with the right facts inline is faster.

Step 4: Fix

The fix should:

1. Stop the bleed — bring service back to working, even if not optimal.
2. Address the immediate cause — patch the bug, swap the bad disk, roll back the chart.
3. Be reversible — if the fix turns out to be wrong, the rollback is one command.

Avoid the temptation to also fix everything else you see while you're in there. Scope creep during incident response is how you turn a 30-minute outage into a 6-hour one. Note the other problems for the postmortem; fix them in a planned change window.

If the fix requires a rolling restart of something tenants depend on (kubelet, containerd), schedule it explicitly, give tenants notice, and proceed only when notified is acknowledged. Surprise restarts are how you turn an incident into a customer-trust incident.

Step 5: Postmortem

Postmortem within five business days, sent to the affected tenant.

Structure

Title: Incident postmortem — <one-line summary> — <date>

Summary
A 2-3 sentence plain-English summary. What broke, how long, who was affected.

Timeline (UTC)
HH:MM   First customer report
HH:MM   On-call ack
HH:MM   Initial triage; identified <component>
HH:MM   Containment applied (<what>)
HH:MM   Root cause identified
HH:MM   Fix applied
HH:MM   Tenant verified service restored

Affected
- Tenant(s): <names>
- Resources: <pods, jobs, namespaces>
- Window: T-start to T-resolved (Y minutes)
- Direct impact: <jobs killed, data loss if any, billing window>

Root cause
The actual technical cause, in detail. Include logs, diff snippets, and version pins.
Make it useful to a future engineer reading this in 6 months.

Why we didn't catch it sooner
What monitoring/alerting/process should have caught this. Be honest.

Remediation
- Immediate (already done): <action> 
- Short-term (within 30 days): <action>, <action>
- Long-term (this quarter): <action>

Customer action required
Usually "none". If the customer needs to redo work or change something, list explicitly.

What a postmortem should not be

• A blamestorming document. The cause is "we did X" not "Alice did X".
• A checklist of "we will do better". Each remediation must have an owner and a date.
• A wall of jargon. The customer often forwards this to their non-technical leadership.

Severity classification

P0 incidents trigger a 24/7 page; P1 wakes on-call; P2/P3 wait for business hours.

Hand-off / shift change during a long incident

If an incident crosses shifts, the leaving engineer writes a hand-off:

Incident: <short name>
Started: <UTC>
Status: <where we are>
Last update sent: <time>, <to whom>

What we know
- <facts confirmed>

What we suspect
- <hypotheses being tested>

What's been tried
- <what didn't work>

Next steps for incoming on-call
1. <action>
2. <action>

Customer expects an update at: <time>

Logs, screenshots, ticket links: <paste>

The incoming engineer reads this before picking up the keyboard.

Don't let the customer be the monitor

If a customer is the first to notice an outage, your monitoring failed. Add an alert. The postmortem should explicitly call this out under "Why we didn't catch it sooner" with a remediation item.

On-call runbook structure

Every cluster should have an on-call runbook the engineer can navigate in 30 seconds. Sections, in order:

1. How to get paged — escalation policy, secondary on-call, manager contact.
2. Cluster facts — region, hostname pattern, bastion, kubeconfig location, 1Password vault refs, who pays for it, who the customer contacts are.
3. Triage entry points — links to the decision tree, the health-check runbook, the dashboards.
4. Severity definitions and response times — see severity table below.
5. Communication cadence — see "comms cadence" section below.
6. Customer / provider comms templates — see templates below.
7. Common failure modes & known issues — link to the dedicated playbooks (NCCL failures, k8s pod failures, RDMA debugging).
8. Postmortem template — see the postmortem section above.

The runbook lives in the team's docs repo. Update it after every incident.

Severity definitions (P0..P4)

The table from "Severity classification" earlier in this page is the four-level model. Add a P4 for hygiene:

A given symptom can be different severities depending on context. "One node down" is P2 in a 100-node cluster with idle capacity; it's P1 if that node is part of an active large training job that has to restart from a checkpoint. The on-call engineer assigns severity at incident open and revises as scope becomes clearer.

Comms cadence — by audience

The earlier section covers "first ack / hourly update / resolution". Different audiences need different cadences; don't overcommunicate to one and undercommunicate to another.

Internal (engineering, manager, support)

• Incident open: post in incident channel within 5 min. State the symptom, suspected layer, severity.
• Updates: every 30 min for P0, every 60 min for P1, on-resolution for P2+.
• Resolution: one-line summary + ETA for postmortem.

Customer (the affected tenant)

• Initial ack within 30 min for P0, 1h for P1.
• Status updates every 2-4h for P0/P1, 1-2x/day for P2.
• Resolution with summary + one-sentence root cause.
• Postmortem within 5 business days for P0/P1.

Provider / vendor

• Initial report: factual prose. Hostname + IP + symptom + diagnostic data attached + the ask.
• Updates as you find more data — don't make them ask.
• Closure: confirmation when resolved, including what they changed.

Customer comms — sanitized examples

These are the templates we actually send. Direct, brief, specific origin. Plain English; no marketing copy; no "we apologize for the inconvenience" filler.

Initial acknowledgment, narrowed scope

We're seeing failures in NCCL multi-node training jobs landing on nodes gpu-{14,15,16}. Initial diagnostic points to a fabric issue on leaf switch L3. We're investigating. Could you pause new launches that target those nodes? We'll update by 14:00 UTC.

Asking to take affected nodes for triage

We've identified that nodes gpu-{14,15,16} are producing NCCL vendor err 81 under load. The pattern matches a PFC misconfiguration on the leaf switch — non-destructive to fix but requires those three nodes drained. Can we cordon and drain them now? Estimated 2 hours to fix and revalidate. We'll re-add them once nccl-tests passes 95% of baseline busbw.

Mid-incident status update

Update at 13:30 UTC. Identified the PFC issue is on leaf switch L3 port 17, affecting 3 hosts. Switch vendor case opened. Workaround: routing tenant-foo's jobs around L3 by setting nodeSelector to other leaves. ETA on root fix: tomorrow morning UTC.

Resolution

Resolved at 16:45 UTC. Root cause was a PFC config drift on leaf switch L3 — the priority class for RoCE wasn't preserved across a switch firmware update. Reverted the switch config; nccl-tests passing at 99% baseline busbw on the affected nodes. Postmortem with prevention items by Friday.

The pattern: state what's broken, where, what was done, what you're asking from them, and when next.

Provider comms — sanitized examples

Plain English prose, factual, no markdown headers. Paste output as quoted blocks instead of summarizing it.

Opening a vendor case (HCA suspected bad)

We're seeing repeated link errors on a ConnectX-7 NIC in host gpu-21 (mgmt IP 10.0.21.10), specifically port mlx5_0. Errors started at 03:14 UTC after no configuration change on our side.

mlxlink output (run at 09:15 UTC):

Effective Physical Errors          : 4521
Effective Physical BER             : 8E-9
Raw Physical Errors Per Lane       : 12, 0, 0, 0
Height Eye Opening [mV]            : 8, 22, 23, 23

Lane 0 has degraded eye opening (8 mV vs 22 mV on the other three lanes) and is producing nearly all of the physical errors. Cable was replaced at 08:30 UTC; same symptom returned. Switch port (leaf-3 port 14) was swapped at 08:50 UTC; same symptom. We suspect the HCA itself.

Could you ship a replacement HCA and an RMA label? Hostname is gpu-21, serial number 1234567890, in rack R12 of DC alpha-1.

Closing a vendor case

Replacement HCA installed today. mlxlink output post-replacement:

Effective Physical Errors          : 0
Raw Physical Errors Per Lane       : 0, 0, 0, 0
Height Eye Opening [mV]            : 24, 24, 25, 24

Bandwidth at line rate, no errors after 4 hours of NCCL training load. Issue resolved on our side. Closing the case from our end.

The form: facts → output → ask. Or facts → output → confirmation.

When to escalate vs. continue solo

A heuristic, not a rule:

• Escalate when:
  • You've been on the same problem for > 1 hour and aren't narrowing it.
  • The blast radius is growing, not shrinking.
  • You're outside your area of expertise (storage, switch firmware, an unfamiliar workload framework).
  • You're considering a change that could make things worse (rolling driver upgrade, switch config change, force-restart).
• Continue solo when:
  • Each diagnostic is narrowing the issue.
  • The problem is contained (one node, one tenant) and not growing.
  • You have a working workaround in place that gives you breathing room.

In doubt, escalate. A second pair of eyes on a P1 catches mistakes; ego costs nothing in retrospect, costs hours in the moment.

"It's working again — should I stop debugging?"

The hardest call in incident response. Three cases:

Stop investigating

• Transient with a known cause: switch port flap, an ECC blip, a one-off cable jiggle. Document, watch the telemetry, but don't waste hours.
• Root cause found and durable fix applied: e.g. driver downgrade resolved it, you understand why, and the fix won't regress.

Keep investigating even though it's "working"

• Self-healing without explanation: NCCL was failing across multiple nodes and now it isn't, with no change on your side. It will happen again — usually at the worst time. Keep the incident open and chase it.
• Hardware that 'recovered': a GPU that was throwing XID 79 ("fallen off the bus") and is now back. The fault didn't go away; it became intermittent. Cordon the node for a deeper diag run.
• Customer reports a symptom you couldn't reproduce: customers usually aren't lying. If your tooling didn't show the issue but they hit it, your tooling has a blind spot — find it.

The rule: transient + understood = OK. Transient + unexplained = come back to it. Run a follow-up health-check pass on the affected nodes before un-cordoning, and watch the error rate for 24h before declaring closed.

Communication anti-patterns

Things that erode trust with customers / providers / your own team. Don't:

• Promise an ETA you don't have data for. "Should be fixed in an hour" with no diagnostic = lying. "Investigating, will update in 30 minutes with our current understanding" = honest.
• Speculate publicly. "It might be a kernel bug" gets repeated as "they think it's a kernel bug" gets repeated as "they confirmed kernel bug". Speculate internally; communicate facts externally.
• Disappear during long investigations. Even if there's nothing new, post "still investigating, no new data, next update in 2h" so people aren't wondering.
• Bundle unrelated context. "While debugging this we also noticed X" → it'll get conflated with the incident. Open a separate ticket.
• Pretend you understand vendor jargon. If the switch vendor says "the LAG hashing is suboptimal" and you don't know what that means, ask. Pretending leads to bad decisions.

See also

• Operations: troubleshooting — the layered triage that feeds incident detection
• Operations: runbook template — for documenting the fix after the postmortem
• Triage decision tree — fast top-down isolation
• Health check runbook — daily / weekly / pre-deploy health
• NCCL multi-node failures
• K8s GPU pod failures
• RDMA debugging
• Driver / firmware mismatch

External:

• Google SRE Book Ch. 14 (Managing Incidents): sre.google/sre-book/managing-incidents/
• Atlassian incident communication templates: atlassian.com/incident-management/incident-communication
• The Incident Command System (ICS) — origin of role-based incident structure