RKE2 — why we use it, how it's structured, how to operate it

RKE2 is a Kubernetes distribution from Rancher (now SUSE) that ships as a single static Go binary, runs each control-plane component as a static pod, and embeds etcd. It targets the cluster lifecycle that vanilla kubeadm makes painful: air-gapped installs, FIPS-compliant builds, deterministic version pinning, and operator-friendly upgrades.

This page is what an operator needs to know to install, expand, troubleshoot, and upgrade RKE2.

Why RKE2 (vs k3s, vs vanilla kubeadm)

Choose RKE2 when: you need air-gapped or supply-chain-verified Kubernetes, you want a single config file per node, and you want predictable upgrades. Choose k3s when you're running edge / IoT and don't need HA etcd. Choose kubeadm when you have a custom control plane (e.g., external etcd, custom auth webhooks) that doesn't fit either.

Architecture

Two node roles:

• rke2-server — runs the control plane: kube-apiserver, kube-controller-manager, kube-scheduler, kube-proxy (optional), embedded etcd, and the rke2-agent workload as well. Server nodes are also worker nodes.
• rke2-agent — kubelet, kube-proxy (or replacement), containerd. Pure worker nodes.

Each component runs as a static pod sourced from /var/lib/rancher/rke2/agent/pod-manifests/ (well, the server's manifests live there; the kubelet on each node spawns them). The supervisor process is rke2-server.service or rke2-agent.service — that's all you systemctl against.

        rke2-server.service
        ├── containerd
        ├── kubelet
        │   ├── (static pod) kube-apiserver
        │   ├── (static pod) etcd
        │   ├── (static pod) kube-controller-manager
        │   ├── (static pod) kube-scheduler
        │   └── (static pod) kube-proxy   [unless disabled]
        └── (calls home → rke2 supervisor port for cluster bootstrap)

Embedded etcd is the default. For HA you need 3 (or 5) server nodes, each running its own etcd member, joined together via the supervisor token.

Tokens — the truncated-token failure mode

RKE2 has two tokens:

Token file content looks like:

K10abcdef0123456789...::server:fedcba9876543210...

The K10 prefix, the ::server: separator, and the suffix are all required. The suffix is a base64 secret.

The classic failure: an operator copies the token from a screenshot, a Slack paste, or a half-loaded terminal, and the suffix gets truncated. The agent install completes, the agent starts, then endlessly fails to join with cryptic 401s in the logs. The server has no idea this is happening.

# Symptoms in /var/lib/rancher/rke2/agent/logs/...
Sep 12 14:30:21 gpu-01 rke2[12345]: time="..." level=error msg="failed to retrieve agent config: bad response code: 401"

Triage: copy the token directly from the file with wc -c to confirm length:

# On a server node
wc -c /var/lib/rancher/rke2/server/node-token
# 174 /var/lib/rancher/rke2/server/node-token

# On the failing agent
wc -c /etc/rancher/rke2/config.yaml
grep token /etc/rancher/rke2/config.yaml

If the token in the agent config is shorter than what the server file has, the truncation happened on copy. Re-copy with scp or paste from cat into a heredoc that you paste in one go.

A defensive habit: scrape the token via SSH into a config:

ssh cp-01 'sudo cat /var/lib/rancher/rke2/server/node-token' \
  | sudo tee -a /etc/rancher/rke2/config.yaml > /dev/null

/etc/rancher/rke2/config.yaml — common options

This is the only config file you should be editing routinely. It applies to both server and agent (different keys are honored on each side).

Server node example

# /etc/rancher/rke2/config.yaml on a control-plane node
write-kubeconfig-mode: "0640"
tls-san:
  - "k8s.tenant-foo.example.internal"
  - "10.0.0.10"
node-label:
  - "node-role.kubernetes.io/control-plane=true"
  - "topology.kubernetes.io/zone=rack-a"
node-taint:
  - "CriticalAddonsOnly=true:NoExecute"
disable:
  - rke2-ingress-nginx        # we provide our own
  - rke2-snapshot-controller
cni:
  - multus
  - calico
cluster-cidr: "10.42.0.0/16"
service-cidr: "10.43.0.0/16"
cluster-dns: "10.43.0.10"
disable-cloud-controller: true
kube-apiserver-arg:
  - "audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log"
  - "audit-log-maxage=7"
  - "anonymous-auth=false"
kubelet-arg:
  - "system-reserved=cpu=2,memory=8Gi"
  - "kube-reserved=cpu=2,memory=4Gi"
  - "eviction-hard=memory.available<2Gi,nodefs.available<10%"

Agent node example

# /etc/rancher/rke2/config.yaml on a GPU worker
server: "https://cp-01:9345"
token: "K10abcdef0123456789...::server:fedcba9876543210..."
node-label:
  - "node.coreweave.cloud/gpu-class=h100"
  - "topology.kubernetes.io/zone=rack-a"
node-taint:
  - "nvidia.com/gpu=present:NoSchedule"
kubelet-arg:
  - "system-reserved=cpu=4,memory=16Gi"
  - "kube-reserved=cpu=4,memory=16Gi"
  - "feature-gates=HugePages=true"

Keys you'll touch most often

After editing, restart the service: systemctl restart rke2-server or rke2-agent.

Installation flow

Single-server bootstrap:

# On the first server
curl -sfL https://get.rke2.io | INSTALL_RKE2_VERSION=v1.30.4+rke2r1 sh -

mkdir -p /etc/rancher/rke2
cat > /etc/rancher/rke2/config.yaml <<'EOF'
write-kubeconfig-mode: "0640"
tls-san:
  - cp-01
EOF

systemctl enable --now rke2-server.service

# Wait until the API is up — 30-90s typically
journalctl -u rke2-server -f | grep "rke2 is up"

# kubectl from the server itself
export KUBECONFIG=/etc/rancher/rke2/rke2.yaml
ln -s /var/lib/rancher/rke2/bin/kubectl /usr/local/bin/kubectl
kubectl get nodes

Add an agent:

# Get the token from the server
ssh cp-01 'sudo cat /var/lib/rancher/rke2/server/node-token'

# On the new node
curl -sfL https://get.rke2.io | INSTALL_RKE2_VERSION=v1.30.4+rke2r1 \
  INSTALL_RKE2_TYPE=agent sh -

cat > /etc/rancher/rke2/config.yaml <<'EOF'
server: https://cp-01:9345
token: K10...::server:...
EOF

systemctl enable --now rke2-agent.service
journalctl -u rke2-agent -f

Add another server (HA etcd):

curl -sfL https://get.rke2.io | INSTALL_RKE2_VERSION=v1.30.4+rke2r1 sh -

cat > /etc/rancher/rke2/config.yaml <<'EOF'
server: https://cp-01:9345
token: K10...::server:...
tls-san:
  - cp-02
EOF

systemctl enable --now rke2-server.service

Once you have 3 servers, kubectl -n kube-system get pods | grep etcd should show three healthy etcd static pods.

Upgrade flow

Two paths.

Manual rolling upgrade

# Drain a node
kubectl drain cp-02 --ignore-daemonsets --delete-emptydir-data

# Reinstall RKE2 at the new version (same script, new version pin)
curl -sfL https://get.rke2.io | INSTALL_RKE2_VERSION=v1.30.5+rke2r1 sh -

# The installer detects existing install and updates the binaries
systemctl restart rke2-server   # or rke2-agent

# Wait for it to come back, uncordon
kubectl wait --for=condition=Ready node/cp-02 --timeout=10m
kubectl uncordon cp-02

Order matters: upgrade all servers first, one at a time, then agents. Skipping versions (e.g., 1.28 → 1.30) is unsupported — go through the minor versions.

Automated upgrade via system-upgrade-controller

RKE2 ships with system-upgrade-controller. You write Plan CRDs:

apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: rke2-server-upgrade
  namespace: system-upgrade
spec:
  concurrency: 1
  nodeSelector:
    matchExpressions:
      - {key: node-role.kubernetes.io/control-plane, operator: In, values: ["true"]}
  serviceAccountName: system-upgrade
  cordon: true
  upgrade:
    image: rancher/rke2-upgrade
  version: v1.30.5+rke2r1

Apply, the controller drains/upgrades/uncordons each matching node serially. Repeat with a separate Plan for agents.

Common operational issues

Agent 401 on join — almost always a truncated token. See above.

rke2-server won't start, journalctl says "address already in use" — port 6443 or 9345 is held by a previous run that didn't clean up. ss -tlnp | grep -E '6443|9345'. Often a stuck kube-apiserver static pod from before the restart; systemctl reset-failed rke2-server && systemctl restart rke2-server after killing the leftover.

etcd member fails to join after a reinstall — the new node has the same hostname but a different cert/identity. Either remove the stale member from etcd first (etcdctl member remove <id>) or fully reset the new node:

rke2-killall.sh
rke2-uninstall.sh    # nukes /var/lib/rancher/rke2 and configs
# Then reinstall

Stale tokens after a security rotation — rke2 token rotate regenerates. New agents must use the new token; existing agents' kubelet certs are still valid until expiry.

node-token file changed unexpectedly — RKE2 keeps the bootstrap secret deterministic across restarts; if the file content changed, somebody ran rke2 token rotate or restored from backup with a different cluster ID. Audit journalctl -u rke2-server | grep -i token.

Backup of etcd

# Snapshot
rke2 etcd-snapshot save --name pre-upgrade-2026-05-04 \
  --dir /var/lib/rancher/rke2/server/db/snapshots

# List
ls /var/lib/rancher/rke2/server/db/snapshots/
# pre-upgrade-2026-05-04-2026-05-04T12-30-00Z.zip

# Restore (DESTRUCTIVE — wipes the cluster, single-node only)
systemctl stop rke2-server
rke2 server --cluster-reset \
  --cluster-reset-restore-path=/var/lib/rancher/rke2/server/db/snapshots/<file>
systemctl start rke2-server

cluster-reset-restore-path rebuilds a single-server cluster from the snapshot. To restore HA, start from this single server and re-add the others as fresh joins.

See also

• GPU Operator — how the GPU stack lands on RKE2
• Network Operator — RDMA + SR-IOV + IPoIB on RKE2
• Reservations — how multi-tenant binding works
• ArgoCD — bootstrapping cluster addons via GitOps
• Operations: troubleshooting — generic K8s pod debug

External:

• docs.rke2.io — official RKE2 docs
• rke2 --help, rke2 server --help, rke2 etcd-snapshot --help
• github.com/rancher/system-upgrade-controller