Weka troubleshooting: stale mounts, transport fallback, and the errors that block upgrades

Weka has good days and bad days. On a bad day the symptoms are misleading — a slow ls, a Pending PVC, a CSI pod stuck in Init:CrashLoopBackOff with a kernel module error — and the actual cause is two layers below where the error message points. This page collects the patterns we hit repeatedly and the diagnostic commands that get to root cause fast.

Stale mount blocking driver upgrade

This is the single most common Weka-related ticket on a fleet that migrates nodes between deployments.

Symptom

A node's csi-wekafs-node DaemonSet pod sits in Init:CrashLoopBackOff, logs show:

$ kubectl -n csi-wekafs logs csi-wekafs-node-mz -c drivers-loader
Loading wekafs kernel module for kernel 5.15.0-91-generic
insmod: ERROR: could not insert module wekafsio.ko: Module wekafsio is loaded but not unloadable
FATAL: drivers-loader cannot proceed

Or, on a host being upgraded manually:

$ sudo /opt/weka/bin/upgrade.sh
Removing old kernel modules...
rmmod: ERROR: Module wekafsio is in use
Upgrade aborted.

Why

wekafsio.ko has refcount > 0 because a wekafs mount from a previous Weka cluster registration is still active. The mount-helper container that owned that mount may be long gone, but the kernel still has the FS structure pinned. rmmod will not remove a module with refcount > 0, and the new version can't load alongside the old (kernel symbol conflict), so the upgrade hangs forever.

The most common trigger: the host was migrated from one Weka deployment to another (different cluster, different org) without explicitly unmounting the old /wekafs/<old-cluster> path first. The old container went away, the org credentials were rotated, but the kernel mount lingered.

Triage

# 1. Confirm wekafsio refcount > 0
$ lsmod | grep weka
wekafsio              655360  3
wekafs               1130496  1 wekafsio

# 2. List every wekafs mount on the host
$ grep wekafs /proc/mounts
cluster-old-poc /wekafs/cluster-old-poc wekafs rw,relatime ...
cluster-foo /wekafs/cluster-foo wekafs rw,relatime ...

# 3. Filter out the current production cluster — anything else is stale
$ grep wekafs /proc/mounts | grep -v cluster-foo
cluster-old-poc /wekafs/cluster-old-poc wekafs rw,relatime ...

If grep -v <current-prod-cluster> returns mounts, those are leftovers. If lsmod shows refcount > 0 but /proc/mounts is clean, look for processes holding files open:

$ for pid in $(ls /proc | grep -E '^[0-9]+$'); do
    if grep -q wekafs /proc/$pid/maps 2>/dev/null; then
      echo "PID $pid ($(cat /proc/$pid/comm 2>/dev/null)) has wekafs in maps"
    fi
  done

Fix

# Unmount each stale mount. The mount-helper container may auto-stop;
# umount is safe even if the helper is gone.
$ sudo umount /wekafs/cluster-old-poc

# If the umount hangs because helper isn't responding, lazy-unmount:
$ sudo umount -l /wekafs/cluster-old-poc

# After all stale mounts are gone, refcount should drop
$ lsmod | grep wekafsio
wekafsio              655360  0     # <-- 0, ready to rmmod

# Now drivers-loader can proceed
$ kubectl -n csi-wekafs delete pod csi-wekafs-node-mz
# pod recreates, drivers-loader succeeds

Prevention

When migrating any host between Weka clusters or orgs, audit ALL hosts for leftover registrations as part of the migration runbook:

# Run across the fleet (e.g. via SSH parallel or a DaemonSet)
$ for h in $(cat hosts.txt); do
    ssh $h "grep wekafs /proc/mounts | grep -v $CURRENT_PROD_CLUSTER" \
      && echo "  ^ stale on $h"
  done

Treat any stale mount as blocking before the migration is "done". Discovering it three weeks later when a kernel upgrade lands is the painful path.

Common error patterns

"wrong fs type, bad option, bad superblock"

mount: /wekafs/foo: wrong fs type, bad option, bad superblock on cluster-foo/foo, missing codepage or helper program, or other error.

Diagnosis: usually a kernel module / userspace version mismatch.

$ lsmod | grep weka                      # both modules loaded?
$ systemctl status weka-agent            # agent up?
$ cat /opt/weka/version                  # userspace version
$ dmesg | tail -50 | grep -i weka        # kernel-side errors

Fix: reinstall the client matching the cluster version. If the host was apt upgrade'd and got a new kernel, the existing wekafs.ko won't load against the new kernel — let drivers-loader (or the userspace installer) recompile.

"Cluster <X> not found"

$ sudo mount -t wekafs cluster-foo/data /wekafs/data
mount.wekafs: cluster cluster-foo not found

Diagnosis: cluster ID mismatch in /etc/wekaio/.

$ ls /etc/wekaio/clusters/
$ cat /etc/wekaio/current
$ weka cluster list                       # what does the agent see?

Fix: re-join the cluster. If you're targeting a new cluster ID with the same name, the old registration must be forgotten first:

$ weka cluster forget cluster-foo
$ weka cluster join <new-mgmt-ip>:14000 --auth-token-file <token>

"Module wekafsio is loaded but not unloadable"

See stale mount blocking driver upgrade — same root cause every time.

Slow IO with low CPU usage

Symptom: iostat/top show low CPU, but dd if=/wekafs/x of=/dev/null bs=1M runs at 200 MB/s instead of the 3+ GB/s you expect.

Diagnosis: RDMA path broken, fell back to TCP. Confirm:

$ weka cluster network --host $(hostname)
HOST       NIC         STATE  TRANSPORT  IP          MTU
gpu-42     mlx5_0      UP     TCP        10.10.20.4  1500    <-- not RDMA!

$ weka stats --category client --node-ids self --gauge \
    iops_read,iops_write,latency_avg_us,bandwidth_read

# Compare to a healthy peer:
$ weka cluster network --host gpu-43
HOST       NIC         STATE  TRANSPORT  IP          MTU
gpu-43     mlx5_0      UP     RoCE       10.10.20.5  4200    <-- RDMA, expected

Fix paths (in order):

1. NIC link state: ethtool mlx5_0 | grep "Link detected"
2. RDMA modules loaded: lsmod | grep -E 'rdma_ucm|ib_uverbs|mlx5_ib'. If missing, see RDMA modules-load.d — the universal fix is to load rdma_ucm, ib_uverbs, ib_umad, and friends at boot.
3. Fabric reachability: ibping or ib_send_bw to a known-good peer.
4. If RoCE: PFC / DCQCN configuration on switch and host.
5. Once fabric works, weka local restart to renegotiate transport. Verify with weka cluster network again.

"Stale file handle" after backend restart

$ ls /wekafs/foo
ls: cannot access '/wekafs/foo': Stale file handle

Diagnosis: a backend container restarted (rolling upgrade, OOM, ESS) and the client's connection state is out of sync.

$ weka cluster status
$ weka events list --severity ERROR --num-results 50 \
    | grep -E 'restart|disconnect|reconnect'

Fix: usually clears itself within 30s. If it doesn't:

$ sudo umount /wekafs/foo && sudo mount -t wekafs cluster-foo/foo /wekafs/foo

If umount hangs ("target is busy"), kill processes holding files open then retry. Do not force-remove the mount-helper container as a first response — that orphans the kernel-side state and you're back to the stale-mount problem.

Snapshot quiesce timeouts

$ kubectl get volumesnapshot
NAME            READYTOUSE  ...   ERROR
my-snap         false       ...   "snapshot quiesce timeout exceeded"

Diagnosis: the snapshot driver tried to quiesce IO on the source volume and didn't get quiet within the timeout. Usually the consumer pod has heavy in-flight writes.

Fix: either (a) accept the snapshot is crash-consistent (not app-consistent) and retry with a longer timeout in the VolumeSnapshotClass, or (b) coordinate with the app — fsfreeze -f /wekafs/foo from the consumer side, snapshot, fsfreeze -u, app continues.

weka cluster status shows containers STAGED

$ weka cluster container
CONTAINER ID  HOSTNAME  MODE     STATUS    ...
0             stor-01   COMPUTE  STAGED    <-- not UP

Diagnosis: container was created but never finished joining. Common after a chaotic backend restart, or a pod-level Weka deployment where the operator scaled containers up but the cluster didn't accept them.

$ weka events list --num-results 100 | grep -E 'staged|join|reject'
$ kubectl get wekacontainer -A    # if running operator-managed

Fix: depends on cause. If the backend simply needs a kick: weka cluster container start <id>. If the operator created mismatched containers (wrong CR generation): kubectl delete wekacontainer <name> and let the operator recreate. If multiple containers stuck: open a support ticket — don't improvise, you can lose data.

Diagnosis commands

Cluster-level (run from any client pod or host with weka CLI)

# Overall health
$ weka cluster status

# All container processes (UP/STAGED/DOWN)
$ weka cluster container

# Just clients
$ weka cluster container --clients -b --output id,hostname,status,mode,mounts

# Per-host network state and transport
$ weka cluster network

# Live stats
$ weka stats --category cluster --gauge \
    iops_read,iops_write,bandwidth_read,bandwidth_write,latency_avg_us

# Recent events
$ weka events list --severity ERROR --num-results 50

# Filesystems and capacity
$ weka fs

# Per-org usage
$ weka cluster org

Client-side (on the host)

# What's the local client doing?
$ weka local status

# Which kernel modules are loaded
$ lsmod | grep weka

# Active mounts
$ grep wekafs /proc/mounts

# Memory mappings (find processes holding wekafs files)
$ for pid in $(ls /proc | grep -E '^[0-9]+$'); do
    grep -l wekafs /proc/$pid/maps 2>/dev/null \
      | xargs -I{} sh -c 'echo "PID $(echo {} | cut -d/ -f3):"; head -1 {}'
  done

# Mount-helper container logs
$ journalctl -u weka-agent.service -n 200
$ sudo crictl ps | grep weka                # or docker/containerd equivalent
$ sudo crictl logs <weka-container-id>

Kubernetes / CSI

# CSI controller logs (provisioning failures)
$ kubectl -n csi-wekafs logs deploy/csi-wekafs-controller -c csi-wekafs-plugin

# Per-node DaemonSet, drivers-loader logs (kernel module load failures)
$ kubectl -n csi-wekafs logs csi-wekafs-node-<id> -c drivers-loader
$ kubectl -n csi-wekafs logs csi-wekafs-node-<id> -c csi-wekafs-plugin

# Operator-managed deployments (if applicable)
$ kubectl get wekacluster -A
$ kubectl get wekacontainer -A
$ kubectl get wekaclient -A

# PVC status
$ kubectl describe pvc <name> -n <ns>

Quick reference: which command for which question

See also

• Architecture — the model behind these commands
• Clients — mount-helper, multi-cluster pitfalls
• CSI — drivers-loader, PVC stuck Pending
• Performance tuning — when slow IO is a tuning issue, not a fault
• Operations — clean upgrade and node-removal workflows that prevent these issues