Cross-datacentre rsync runbook

Overview

Transfer data between two same-tenant disk-access pods in different clusters/datacentres using rsync over SSH via LoadBalancer services. Covers network validation, SSH auth, a tmux-resilient launch, and post-transfer sanity checks.

Step 1 — identify the pods and LB IPs

kubectl --context `<src-cluster>` get pods -n `<namespace>`
kubectl --context `<dst-cluster>` get pods -n `<namespace>`

kubectl --context `<src-cluster>` get svc -n `<namespace>`
kubectl --context `<dst-cluster>` get svc -n `<namespace>`

Also grab the node IPs (the DC team will need these, not the LB IPs):

kubectl --context `<src-cluster>` get pod `<src-pod>` -n `<namespace>` -o jsonpath='{.status.hostIP}'
kubectl --context `<dst-cluster>` get pod `<dst-pod>` -n `<namespace>` -o jsonpath='{.status.hostIP}'

LoadBalancer IPs handle inbound traffic only. Outbound from the pod sources from the node IP. Always give the DC team node IPs.

Step 2 — request the DC firewall rule

• Source node IP → ``<dst-lb-ip>:22 (src → dst)
• Destination node IP → ``<src-lb-ip>:22 (reverse direction)
• Protocol: TCP, port 22

Step 3 — verify connectivity

# TCP reachability
kubectl --context `<src-cluster>` exec -n `<namespace>` `<src-pod>` -- \
  bash -c "timeout 5 bash -c 'echo > /dev/tcp/`<dst-lb-ip>`/22' && echo OK || echo FAILED"

Outcomes:

• Connection timed out → route still closed
• Permission denied → route open, proceed to step 4
• TCP succeeds → fully working

Step 4 — verify SSH key auth

SRC_PUBKEY=$(kubectl --context `<src-cluster>` exec -n `<namespace>` `<src-pod>` -- \
  cat `<home-path>`/.ssh/id_ed25519.pub)

kubectl --context `<dst-cluster>` exec -n `<namespace>` `<dst-pod>` -- \
  bash -c "echo '$SRC_PUBKEY' >> `<home-path>`/.ssh/authorized_keys"

# end-to-end test
kubectl --context `<src-cluster>` exec -n `<namespace>` `<src-pod>` -- \
  bash -c "ssh -i `<home-path>`/.ssh/id_ed25519 \
    -o ConnectTimeout=5 -o StrictHostKeyChecking=no \
    -l `<username>` `<dst-lb-ip>` 'echo connected; hostname'"

Step 5 — dry run

kubectl --context `<src-cluster>` exec -n `<namespace>` `<src-pod>` -- bash -c "
rsync --dry-run -rltvhP \
  --partial --append-verify \
  --no-perms --no-owner --no-group --omit-dir-times \
  -e 'ssh -i `<home-path>`/.ssh/id_ed25519 -o StrictHostKeyChecking=no' \
  `<src-path>`/ \
  `<username>`@`<dst-lb-ip>`:`<dst-path>`/ 2>&1 | tail -5"

Step 6 — write a retry script onto the PVC

Cross-DC SSH connections drop frequently. --partial --append-verify lets rsync resume safely; the loop catches transient drops.

kubectl --context `<src-cluster>` exec -n `<namespace>` `<src-pod>` -- bash -c "cat > `<home-path>`/rsync-retry.sh << 'EOF'
#!/bin/bash
while true; do
  rsync -rltvhP \
    --partial --append-verify \
    --no-perms --no-owner --no-group --omit-dir-times \
    --timeout=60 \
    -e 'ssh -i `<home-path>`/.ssh/id_ed25519 -o StrictHostKeyChecking=no \
        -o ServerAliveInterval=30 -o ServerAliveCountMax=20' \
    `<src-path>`/ \
    `<username>`@`<dst-lb-ip>`:`<dst-path>`/ 2>&1 | tee -a `<home-path>`/rsync.log
  EXIT=${PIPESTATUS[0]}
  [ $EXIT -eq 0 ] && { echo 'rsync complete!'; break; }
  echo "rsync exited $EXIT, retrying in 10s..."; sleep 10
done
EOF
chmod +x `<home-path>`/rsync-retry.sh"

Step 7 — launch in tmux inside the pod

tmux on your laptop dies when the laptop sleeps. tmux inside the pod survives. Run it as the user (per-UID socket):

kubectl --context `<src-cluster>` exec -n `<namespace>` `<src-pod>` -- \
  bash -c "su -s /bin/bash `<username>` -c 'tmux new-session -d -s rsync-dst \
    \"`<home-path>`/rsync-retry.sh\"'"

# verify
kubectl --context `<src-cluster>` exec -n `<namespace>` `<src-pod>` -- \
  bash -c "su -s /bin/bash `<username>` -c 'tmux capture-pane -t rsync-dst -p'" | tail -15

Step 8 — monitor

# rsync output
kubectl --context `<src-cluster>` exec -n `<namespace>` `<src-pod>` -- \
  bash -c "su -s /bin/bash `<username>` -c 'tmux capture-pane -t rsync-dst -p'" | tail -15

# disk usage on the destination
kubectl --context `<dst-cluster>` exec -n `<namespace>` `<dst-pod>` -- du -sh `<dst-path>`

Step 9 — sanity checks and cleanup

# sizes
kubectl --context `<src-cluster>` exec -n `<namespace>` `<src-pod>` -- du -sh `<src-path>`
kubectl --context `<dst-cluster>` exec -n `<namespace>` `<dst-pod>` -- du -sh `<dst-path>`

# file counts
kubectl --context `<src-cluster>` exec -n `<namespace>` `<src-pod>` -- \
  bash -c "find `<src-path>` -type f | wc -l"
kubectl --context `<dst-cluster>` exec -n `<namespace>` `<dst-pod>` -- \
  bash -c "find `<dst-path>` -type f | wc -l"

# kill the tmux session
kubectl --context `<src-cluster>` exec -n `<namespace>` `<src-pod>` -- \
  bash -c "su -s /bin/bash `<username>` -c 'tmux kill-session -t rsync-dst'"

Reference — rsync exit codes