Pod inspector

Diagnostic shortcuts and a searchable list of common stuck-pod causes.

When a pod is stuck — Pending, CrashLoopBackOff, ImagePullBackOff, CreateContainerError — most of the diagnosis comes from kubectl describe pod and kubectl events. The shortcut commands below are the first three to run; the table underneath maps the symptom you see to cause and fix.

Quick queries

Pod summary + recent events

kubectl describe pod <pod> -n <ns>

Logs (current + previous)

kubectl logs -n <ns> <pod> --tail=200
kubectl logs -n <ns> <pod> --previous --tail=200

Pod-scoped events stream

kubectl events -n <ns> --for pod/<pod> --watch

Why is it Pending? (scheduler reason)

kubectl get pod <pod> -n <ns> -o jsonpath='{.status.conditions[?(@.type=="PodScheduled")].message}'

Node-scoped Loki / journal — OOM-killer

# Loki
{node="<node>"} |~ "(?i)Out of memory|Killed process|oom_reaper|invoked oom-killer"

# journal on the node
journalctl -k -g 'oom_reaper|killed process' --since '1 hour ago'

Symptom → cause → fix

Pending — Insufficient cpu/memory

cause

Scheduler can't find a node with enough free CPU/memory.

fix

Check `kubectl describe nodes` for Allocatable vs Allocated. If a hot tenant is using everything, evict the offender or scale up.

Pending — node(s) didn't match Pod's node affinity/selector

cause

Pod nodeAffinity, nodeSelector or topology spread can't be satisfied. Often a label drift between what the workload expects and what nodes carry.

fix

Compare `pod.spec.nodeSelector` and `pod.spec.affinity` against `kubectl get nodes --show-labels`. Look especially for stale `accelerator=h100` or generation labels.

Pending — taint <key>=<value>: NoSchedule

cause

Node has a taint the pod doesn't tolerate. Common culprits: repair taint, GPU operator install taint, dedicated tenant taint.

fix

Either add the matching toleration to the workload or untaint the node (`kubectl taint nodes <node> <key>-`).

CrashLoopBackOff — exit code 137 / OOMKilled

cause

Container was killed because it exceeded its memory limit.

fix

`kubectl describe pod` shows OOMKilled. Either bump `resources.limits.memory` or fix the leak. Confirm node has the memory headroom first.

CrashLoopBackOff — exit code 1, no log

cause

Common with init containers — the binary segfaulted before producing output.

fix

Run `kubectl logs --previous`. Fall back to `kubectl debug` with a tools image to inspect the same volume mounts.

CreateContainerError — failed to start container … no such device

cause

GPU/IB device not present on the node when runtime tried to bind it.

fix

Check Container Toolkit (`nvidia-container-cli info`) and the device plugin pod on that node. NIC restarts often cause this transient state.

CreateContainerError — failed to mount … permission denied

cause

Volume mount source has wrong perms (commonly hostPath or PVC migrated between security contexts).

fix

Check `runAsUser` / `fsGroup` on the pod, and the actual chown on the host path.

Init:Error — initContainer exit code != 0

cause

Init container failed before the main container could start.

fix

`kubectl describe pod` lists each init container's exit code. Tail logs for the failing one with `kubectl logs <pod> -c <init-container>`.

ImagePullBackOff — auth required

cause

Pull secret missing or token expired.

fix

Check the imagePullSecret on the pod (`kubectl get pod -o jsonpath='{.spec.imagePullSecrets}'`) and verify it exists in the same namespace.

NotReady — kubelet not posting status

cause

kubelet on the node is dead or partitioned from the apiserver. Pods stay scheduled but no events flow.

fix

SSH the node, `systemctl status kubelet`, `journalctl -u kubelet -n 200`. Common: kubelet OOMed, certs expired, containerd lost.

Webhook denied — admission webhook failed

cause

A validating/mutating webhook (Gatekeeper, Kyverno, CSI snapshot, sidecar injector) rejected the spec.

fix

Look at the error message — it names the webhook. Check the webhook pod's logs and the policy CR if it's a policy engine.