Node speed benchmark

Generate a fleet-wide curl benchmark and visualize the result.

Two stages. First, build a script that runs a download / upload / git-clone test inside a pod on every GPU node. Second, paste the output back here to spot which nodes are outliers (the script prefixes every block with === node | pod ===).

Outliers in download speed almost always mean the node is on the wrong egress path — either NAT-bound or stuck on a slow rail. Slow git-clone alone usually means a Tailscale/proxy egress, since git has its own protocol overhead.

Step 1 — generate script

Generated script
for node in $(kubectl get nodes --selector='!node-role.kubernetes.io/login' \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'); do

  pod=$(kubectl get pods -n tenant-foo \
    --field-selector spec.nodeName=$node \
    --no-headers -o custom-columns=":metadata.name" 2>/dev/null \
    | grep "^gpu-job-")
  if [ -z "$pod" ]; then continue; fi

  echo "=== $node | $pod ==="

  kubectl exec -n tenant-foo "$pod" -- bash -c '
    echo "--- Download (Cloudflare 1GB) ---"
    curl -o /dev/null -s "https://speed.cloudflare.com/__down?bytes=1000000000" \
      --write-out "DNS: %{time_namelookup}s | Connect: %{time_connect}s | Total: %{time_total}s | Speed: %{speed_download} bytes/s\n"
    echo "--- Upload (Cloudflare 200MB) ---"
    dd if=/dev/zero bs=1M count=200 2>/dev/null \
      | curl -sX POST https://speed.cloudflare.com/__up \
          -H "Content-Type: application/octet-stream" \
          --data-binary @- \
          --write-out "Upload Speed: %{speed_upload} bytes/s\n" \
          -o /dev/null
    echo "--- Git Clone Throughput ---"
    rm -rf /tmp/linux-test
    time git clone --depth=1 https://github.com/torvalds/linux /tmp/linux-test
    rm -rf /tmp/linux-test
  ' 2>&1
done

Step 2 — paste output

No data yet. Paste output and the per-node table will appear.