Weka CSI: how csi-wekafs actually provisions storage in Kubernetes

The csi-wekafs driver lets a Kubernetes pod mount wekafs like any other PVC, but the moving parts are unusual: a node-side DaemonSet that compiles and loads the kernel module on demand, a controller-side Deployment that talks to Weka's REST API to carve out filesystems, and a per-tenant org model that doesn't map cleanly to K8s namespaces. This page is the working operator's view.

What gets installed

A typical csi-wekafs install drops two workloads:

Plus a StorageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: weka
provisioner: csi.weka.io
parameters:
  volumeType: dir/v1
  filesystemName: cluster-foo-shared
  capacityEnforcement: HARD
reclaimPolicy: Retain
allowVolumeExpansion: true
volumeBindingMode: Immediate

The provisioner name is csi.weka.io for the upstream driver. Some legacy installs use weka.io/<cluster>.<namespace> (the old per-cluster operator-managed flavor). Both still exist; modern deployments use the upstream driver.

The drivers-loader init container

This is the part that surprises people. The csi-wekafs-node DaemonSet runs an init container that:

1. Detects the host kernel version.
2. Downloads (or builds) the matching wekafs.ko and wekafsio.ko.
3. insmods them into the host kernel.
4. Exits 0 so the main container can start.

$ kubectl -n csi-wekafs get pods -o wide
NAME                READY  STATUS   NODE        
csi-wekafs-node-2x  3/3    Running  gpu-01
csi-wekafs-node-7q  3/3    Running  gpu-02
csi-wekafs-node-mz  0/3    Init:CrashLoopBackOff  gpu-03   <-- module load failing

$ kubectl -n csi-wekafs logs csi-wekafs-node-mz -c drivers-loader
[2026-05-04 10:11:32] Loading wekafs kernel module for kernel 5.15.0-91-generic
[2026-05-04 10:11:33] insmod: ERROR: could not insert module wekafsio.ko: Module wekafsio is loaded but not unloadable
[2026-05-04 10:11:33] FATAL: drivers-loader cannot proceed

That Module wekafsio is loaded but not unloadable message is the killer. The module is already in the kernel (from a previous version), and the loader can't rmmod it because something is still holding a reference — almost always a leftover wekafs mount from a previous Weka cluster registration. See stale mount blocking driver upgrade.

When drivers-loader fails on a node, every PVC scheduled to that node stays Pending with a NodeAffinity error or a MountVolume.SetUp failed event. Recover by clearing the stale mount, then kubectl delete pod -n csi-wekafs csi-wekafs-node-<id> to force a re-init.

Dynamic provisioning: dir/v1 and friends

Weka exposes filesystems and directories as separately provisionable units:

For typical GPU workloads, dir/v1 is what you want. The controller creates a directory under the parent filesystem, sets a hard quota, and binds the PVC. Provisioning is fast (milliseconds — no actual filesystem create) and capacity is enforced at the directory level.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: training-data
  namespace: tenant-bar
spec:
  storageClassName: weka
  accessModes: [ReadWriteMany]
  resources:
    requests:
      storage: 100Gi

When this PVC binds, the controller creates /<filesystemName>/<pvc-uid>/ on the cluster, sets a 100 GiB hard quota, and exposes the directory as the volume. Pods that mount the PVC see only that subdirectory, not the parent filesystem.

ReadWriteMany vs ReadWriteOnce

Weka is a true parallel filesystem — multiple clients can hold the same file open for read+write simultaneously, with the cluster handling consistency. So RWX is the default and recommended access mode.

If a Helm chart insists on ReadWriteOnce, that's fine — Weka serves it. The chart probably wants RWX though, and you should patch the chart.

Static provisioning

Sometimes you want a PV pointing at an existing Weka directory (e.g., a dataset that lives outside K8s and predates the cluster). Static PV form:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: shared-imagenet
spec:
  capacity:
    storage: 1Ti
  accessModes: [ReadWriteMany]
  persistentVolumeReclaimPolicy: Retain
  storageClassName: ""              # empty = static
  csi:
    driver: csi.weka.io
    volumeHandle: dir/v1/cluster-foo/datasets/imagenet
    volumeAttributes:
      filesystemName: datasets

Then a PVC binds by name:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: imagenet
spec:
  storageClassName: ""
  volumeName: shared-imagenet
  accessModes: [ReadWriteMany]
  resources:
    requests:
      storage: 1Ti

When to use which:

Static is the right call for canonical datasets shared across tenants (ImageNet, LAION, your house image cache). Dynamic for everything else.

Per-tenant orgs

This is the bit that doesn't map cleanly to K8s. Weka has its own multi-tenancy concept — orgs — and each org has separate:

• Filesystems
• Quotas
• User accounts
• Auth credentials

The CSI driver authenticates with one org per StorageClass. So if you want strict isolation between tenant-foo and tenant-bar, you typically:

1. Create one org per tenant on the Weka cluster.
2. Create one StorageClass per tenant, each pointing at the right org.
3. Lock down which namespaces can use which StorageClass via ResourceQuota or admission webhooks.

# StorageClass for tenant-foo
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: weka-tenant-foo
provisioner: csi.weka.io
parameters:
  volumeType: dir/v1
  filesystemName: tenant-foo-fs
  csi.storage.k8s.io/provisioner-secret-name: weka-creds-tenant-foo
  csi.storage.k8s.io/provisioner-secret-namespace: csi-wekafs

The weka-creds-tenant-foo secret holds the org username + password. Each tenant gets their own secret, their own filesystem, and their own quota. Cross-tenant access is impossible because the credentials don't authenticate to the other org.

The simpler model — one org, RBAC at the K8s layer only — works for cooperating tenants. Strict tenancy needs orgs.

Volume expansion

Online expansion is supported when the StorageClass has allowVolumeExpansion: true:

$ kubectl edit pvc training-data
# bump spec.resources.requests.storage from 100Gi to 500Gi
$ kubectl get pvc training-data
NAME            STATUS   CAPACITY  ...
training-data   Bound    500Gi     ...   # quota updated, pods see new size

For dir/v1 volumes, expansion is a quota change — instant, no data movement. For weka/v2 (new filesystem per volume), expansion may involve real allocation and can take longer.

Shrinking is not supported by the upstream driver (and Kubernetes itself doesn't really support it).

Snapshots

Weka has cluster-native snapshots, exposed via the standard K8s VolumeSnapshot API:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: weka-snap
driver: csi.weka.io
deletionPolicy: Retain
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: training-data-2026-05-04
  namespace: tenant-bar
spec:
  volumeSnapshotClassName: weka-snap
  source:
    persistentVolumeClaimName: training-data

The snapshot is copy-on-write — instant creation, no data move. Reads from the snapshot pull from the original location until the original is overwritten, at which point the old block is preserved.

To restore, create a new PVC with dataSource pointing at the snapshot:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: training-data-restored
spec:
  storageClassName: weka
  accessModes: [ReadWriteMany]
  dataSource:
    name: training-data-2026-05-04
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  resources:
    requests:
      storage: 100Gi

Snapshots are point-in-time consistent across all files in the directory. They are NOT crash-consistent for application state — if your DB had transactions in flight, the snapshot will reflect mid-transaction state. Coordinate with the app (fsync, freeze) for app-consistent snapshots.

Common: PVC stuck Pending

This is the most-frequent CSI ticket. Diagnostic order:

$ kubectl describe pvc training-data
Events:
  Warning  ProvisioningFailed   ...   failed to provision volume with StorageClass "weka": rpc error: code = Unauthenticated desc = ...

The last one is the nasty case. The PVC binds to a node-specific PV, but the chosen node has stale wekafsio and the new module can't load. K8s won't reschedule because the PV's nodeAffinity points at exactly that node.

Fix:

# 1. Find the failing node
$ kubectl describe pod <consumer-pod> | grep -A3 "node affinity"

# 2. Check drivers-loader on that node
$ kubectl -n csi-wekafs logs csi-wekafs-node-<id> -c drivers-loader

# 3. If "loaded but not unloadable" — clear the stale mount on the node
$ ssh <node>
$ grep wekafs /proc/mounts                     # find leftover mounts
$ sudo umount <each-leftover>                  # unmount them all
$ sudo rmmod wekafs wekafsio                   # drop the modules

# 4. Bounce the daemonset pod so drivers-loader retries
$ kubectl -n csi-wekafs delete pod csi-wekafs-node-<id>

NodeAffinity from Reservation operator

If you use a Reservation operator that holds nodes for specific tenants, a Weka PVC may bind in the wrong order: the PVC's PV gets nodeAffinity for a node that the Reservation operator subsequently fences off, and pods can't schedule. Resolution:

• Use volumeBindingMode: WaitForFirstConsumer on the StorageClass — the PV is created only when a consumer pod is scheduled, so node affinity is set after reservation decisions.
• Or, if you have to use Immediate, scope tenants to specific nodes via NodeSelector and ensure the StorageClass's PV creation respects that.

See also

• Architecture — what the controller is talking to
• Clients — what the node-side daemonset wraps
• Operations — org management, upgrades
• Troubleshooting — stale mounts, drivers-loader failures

Troubleshooting

For drivers-loader failures, stale mounts, and "Module wekafsio is loaded but not unloadable", see the stale mount blocking driver upgrade section.