Runbook template — WHY / HOW / VALIDATE
A skeleton for writing a new operational runbook: frontmatter, sections, and the WHY-first structure that makes runbooks actually useful at 3 AM.
help for the full list, or solutions for copy-paste fix recipes.A good runbook is one you can follow successfully at 3 AM, possibly hung over, while a customer is asking for ETA every five minutes. That means: the why is upfront so you don't blindly run a destructive command; the how is exact commands with expected output; and the validate step proves the fix worked before you tell anyone it did.
This page is the template. Copy the skeleton, fill in the blanks, replace examples with your specific scenario.
The skeleton
---
title: "<Action> on <subject> — <one-line outcome>"
description: "<Concise outcome-oriented description, ~150 chars>"
---
## What this is
<One paragraph: what scenario triggers this runbook. Be specific. The reader should
recognize their situation in two sentences.>
## Why we do it this way
<Two-three paragraphs of context. The dependencies, the constraints, the tradeoffs.
What's the blast radius if we do it wrong? What's the alternative we considered
and rejected, and why? This is the section that prevents the operator from
"improving" the recipe under pressure.>
## Prerequisites
- Access: <SSH? kubectl? Weka API? whose creds?>
- Tools: <required CLIs and minimum versions>
- State checks: <"the cluster must be in X state before running this">
- Permissions: <RBAC roles, sudo, etc>
## Procedure
### Step 1 — <short imperative>
<Why this step exists. One sentence.>
```bash
# the actual command
<command>
# expected output (sanitized to remove identifiers):
<expected stdout>
```
If the output differs: <what to check, when to abort>.
### Step 2 — <short imperative>
<Why.>
```bash
<command>
```
Expected: <what>.
### Step N — <short imperative>
...
## Validation
<How to confirm the procedure worked. This is NOT the same as "Step N completed successfully";
it's an independent check that would have caught the original problem if it had been there.>
```bash
<validation command>
# expected output:
<output>
```
Specifically check for:
- <observable 1>
- <observable 2>
## Rollback
<What to do if the procedure fails partway through. Be honest if there isn't a clean
rollback — say so, and what the recovery looks like instead.>
```bash
<rollback command>
```
## Common failure modes
| Symptom | Likely cause | Action |
|----------------------------------------|---------------------------------------|----------------------------------------|
| <visible thing> | <cause> | <next step> |
| ... | ... | ... |
## When NOT to run this
<Scenarios that look similar but are actually a different problem requiring a different
runbook. List them. The reader should bail out and consult the right doc instead.>
## See also
- [<related runbook>](/section/page)
- [<background concept>](/section/page)
External:
- <link to upstream docs>
Why this structure
Frontmatter
The Next.js MDX site uses title and description for the page header and search snippets. A bad description here is a runbook that's hard to find when you need it.
"What this is" before "Why we do it this way"
The first thing a reader checks is "is this the right runbook?". Then "is this safe to do?". Then the procedure. Reorder these and you'll have operators running the wrong runbook because they assumed it was theirs.
Why-first
Every step has a # why comment. If the operator doesn't know why a step exists, they don't know whether to skip it when it errors. The most common operational mistake is "this step failed but the next one worked, so I just kept going" — the why-first pattern combats it.
Expected output
Most runbooks omit this. Put it in. The operator should be able to compare verbatim and know whether a step is on track. Strip identifiers from real output before pasting.
Validation as independent check
A validation step that's just "the previous command exited 0" is not validation. The validation should be something a fresh operator could run on a running system and confirm "yes, this is the desired state". Often this is a different command than what was used to make the change.
Rollback honestly
Many real-world fixes don't have clean rollbacks. Saying so explicitly is better than pretending one exists. If the only rollback is "restore from backup and replay 12 hours of work", say that.
"When NOT to run this"
Runbooks attract use beyond their intended scope. The reader who almost-fits this runbook needs a clear "no, this isn't your problem" exit.
Worked example: skeleton applied
Here's the same template filled in for a small example — disabling PCIe ACS on a freshly provisioned GPU node.
---
title: "Disable PCIe ACS on a fresh GPU node"
description: "Apply the runtime setpci-loop ACS disable + persist via /etc/rc.local for GPUDirect P2P."
---
## What this is
You've just provisioned a single-tenant GPU node and `nvidia-smi topo -p2p r` is showing
NS for same-baseboard GPU pairs. The kernel does not have the `pcie_acs_override` patch.
## Why we do it this way
PCIe ACS forces P2P traffic through the root complex, halving GPUDirect bandwidth.
On a single-tenant node we can disable it. We use the runtime setpci method (rather
than the boot param) because the stock kernel lacks the patch. We persist via
/etc/rc.local because that runs after PCI device enumeration is fully complete.
## Prerequisites
- Root SSH access to the node
- Single-tenant node (don't run on a multi-tenant hypervisor — see "When NOT to run")
- `lspci`, `setpci` installed (default on every distro)
## Procedure
### Step 1 — Identify ACS-capable PCIe bridges
We need to enumerate the bridges before patching them.
```bash
lspci -d "::0604" | head
# 00:01.0 PCI bridge: ...
# 17:00.0 PCI bridge: ...
# ...
```
### Step 2 — Write the disable script
```bash
sudo tee /usr/local/sbin/acs_disable.sh <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
for bdf in $(lspci -d "::0604" | awk '{print $1}'); do
sudo lspci -s "$bdf" -vvv 2>/dev/null | grep -q "Access Control Services" || continue
setpci -s "$bdf" ECAP_ACS+0x6.w=0000
done
EOF
sudo chmod +x /usr/local/sbin/acs_disable.sh
```
### Step 3 — Run it once and persist via rc.local
```bash
sudo /usr/local/sbin/acs_disable.sh
sudo tee /etc/rc.local <<'EOF'
#!/usr/bin/env bash
/usr/local/sbin/acs_disable.sh
exit 0
EOF
sudo chmod +x /etc/rc.local
sudo systemctl enable rc-local
sudo systemctl start rc-local
```
## Validation
```bash
# ACS Ctl bits should be all '-'
for bdf in $(lspci -d "::0604" | awk '{print $1}'); do
sudo lspci -vvv -s "$bdf" 2>/dev/null | awk '/ACSCtl/{print}'
done | grep -E 'RR\+|CR\+|UF\+' && echo "STILL ON" || echo "OK"
# P2P should now show OK between same-baseboard pairs
nvidia-smi topo -p2p r | head
```
## Rollback
ACS state is reset on reboot. If you don't want it persisted, remove rc.local:
```bash
sudo systemctl disable rc-local
sudo rm /etc/rc.local /usr/local/sbin/acs_disable.sh
sudo reboot
```
## When NOT to run this
- Multi-tenant hypervisor (VFIO pass-through to VMs). Disabling ACS breaks tenant isolation.
- Nodes where BIOS-level ACS is enforced — see [ACS deep-dive](/kernel-tuning/acs).
## See also
- [ACS deep-dive](/kernel-tuning/acs)
- [GRUB cmdline](/kernel-tuning/grub) for the patched-kernel alternative
Style notes
- Past-tense in the title is wrong — "Disabled ACS" implies a status report. Use imperative ("Disable ACS").
- No emoji. They look unprofessional in customer-facing runbooks and they break some terminals.
- No "we will" / "you should". Use imperative for instructions, plain present for explanation.
- Sanitize all identifiers in expected output: hostnames, IPs, customer names, internal codenames. Use generic placeholders (gpu-01, tenant-foo).
- Date the file at top if the procedure has been validated only on a specific kernel/version. "Validated on RKE2 v1.30.4 / kernel 5.15.0-101 / 2026-04".
See also
- Operations: troubleshooting — the triage flow runbooks plug into
- Operations: incident response — postmortems often spawn new runbooks
External:
- Google SRE Workbook Ch. 8 — On-Call rotation and runbook discipline