Weka clients: the mount-helper, kernel module, and host integration

Every Weka client is the same shape: a kernel module (wekafs.ko/wekafsio.ko), a mount-helper container running as a systemd unit, a small bit of state under /etc/wekaio/, and a weka-agent.service that gateways CLI calls to the running container. Get any of those wrong and the mount silently fails, falls back to TCP, or blocks node upgrades. This page covers the client side end-to-end.

What runs on a client

+-------------------------------------------------+
| GPU host (Linux)                                |
|                                                 |
|  systemd                                        |
|   |                                             |
|   +-- weka-agent.service                        |
|   |     (UDS gateway: weka CLI -> container)    |
|   |                                             |
|   +-- mount-helper container (one per host)     |
|         |                                       |
|         +-- userspace data path (DPDK/UDP/RDMA) |
|         +-- shared-memory ring with kernel      |
|                                                 |
|  kernel                                         |
|   +-- wekafs.ko    (POSIX VFS)                  |
|   +-- wekafsio.ko  (IO ring transport)          |
|                                                 |
|  /wekafs/<fs-name>  <-- POSIX mount             |
+-------------------------------------------------+

Three things must be true for mount -t wekafs to succeed:

1. Kernel modules loaded (lsmod | grep weka shows both).
2. weka-agent.service is up.
3. The mount-helper container is running and has a working transport (RDMA/UDP) to the cluster.

If any one is missing, the mount errors. The error message often points at the wrong layer — see troubleshooting.

The mount-helper container

When you mount -t wekafs cluster-foo /wekafs/foo, the kernel's mount(2) syscall hits wekafs.ko. The kernel module doesn't actually know how to talk to the cluster — it punts to userspace via a Unix-domain socket to the mount-helper container, which runs as a privileged container under containerd or Docker (depending on Weka version and install method).

The helper:

1. Validates the mount target exists in /etc/wekaio/clusters/<name>/.
2. Connects to a quorum of MGMT processes on the backend.
3. Authenticates with the org credentials.
4. Allocates client cores (DPDK) and hugepages, if configured.
5. Sets up shared-memory rings to wekafsio.ko.
6. Tells the kernel "OK, mount is live."

A single mount-helper container handles all wekafs mounts on the host, even across multiple clusters. You don't get one helper per mount.

weka-agent.service

This is a thin systemd unit that translates weka <subcommand> CLI calls into RPCs against the running mount-helper container's local API.

$ systemctl status weka-agent
weka-agent.service - Weka agent for local CLI access
     Loaded: loaded (/etc/systemd/system/weka-agent.service; enabled)
     Active: active (running) since ...
     Memory: 18.4M

If weka-agent is dead, every weka <anything> command times out with no useful error. The container can still be running and the mount can still be working — weka-agent is purely the local CLI gateway. Restart with systemctl restart weka-agent; if it keeps dying, check the helper container logs.

Stateless vs stateful clients

Weka clients are stateless by design. The host has no persistent data of its own — everything lives on the backend. This means:

• Wipe and reinstall a GPU node: as long as you re-install the Weka client, the mounts come back identical.
• The mount-helper container doesn't keep cache between reboots (page cache is repopulated; DPDK/RDMA state is rebuilt).
• /etc/wekaio/clusters/<name>/ holds (cluster-id, fingerprint, org-token) — restoring this from any backup re-establishes the trust relationship without re-running weka cluster join.

A typical GPU node is the simplest case: pure consumer, never serves data. The mount is read-write but the host has nothing the cluster needs.

Multi-cluster on the same host

You can mount filesystems from multiple Weka clusters on the same client. Each cluster is identified by name in /etc/wekaio/clusters/:

$ ls /etc/wekaio/clusters/
cluster-foo/  cluster-bar/

$ ls /etc/wekaio/clusters/cluster-foo/
cluster.json  ca.crt  agent.token

The mount-helper container handles all of them. To switch which cluster weka CLI defaults to:

$ weka cluster list
NAME          UUID                                  STATUS
cluster-foo   11111111-...                          ACTIVE
cluster-bar   22222222-...                          ACTIVE

$ weka cluster set cluster-bar
$ weka status   # now reports cluster-bar

Common pitfall: leftover cluster from a previous tenant

If you migrate a host between Weka deployments (e.g., decommissioning one cluster and joining another), the old cluster registration sticks around in /etc/wekaio/clusters/ unless you explicitly clean it up. A leftover registration with a stale mount under /wekafs/<old-name> will pin wekafsio.ko and prevent driver upgrades.

Audit and clean:

# List every wekafs mount and which cluster it's from
$ grep wekafs /proc/mounts
cluster-bar /wekafs/bar wekafs rw,relatime ...

# Should match what's in /etc/wekaio/current
$ cat /etc/wekaio/current
cluster-foo                  # <-- mismatch! bar is mounted but foo is "current"

# Unmount the leftover, then de-register
$ sudo umount /wekafs/bar
$ weka cluster forget cluster-bar

See stale mount blocking driver upgrade for the K8s flavor of this problem.

Client-side caching

Weka uses three caches on the client:

For most AI workloads, the kernel page cache does the heavy lifting — training datasets that fit in RAM get re-read at memory speed after the first epoch. The Weka client cache helps for mixed read/write and helps absorb burst writes.

To bypass caching entirely (for benchmarking real backend throughput), open files with O_DIRECT:

$ fio --name=t --rw=read --bs=1M --size=10G --filename=/wekafs/foo/test --direct=1

--direct=1 skips the page cache. Don't run real workloads with O_DIRECT — your re-reads will all hit the network.

Client-side compute resources

Weka's userspace data path is CPU-hungry. The default install dedicates cores to the mount-helper container — those cores are no longer available to your training job.

$ weka local resources
CONTAINER     MEMORY     CORES        HUGEPAGES
client0       2.0 GB     0,1          2 GB

Default behavior depends on the install:

• Auto-detect (coresNum=auto): Weka picks based on host topology and NIC count. Typical: 2–4 cores on a CPU node, 4–8 on a high-throughput client.
• Fixed (coresNum=N): you specify, Weka pins. Best for reproducibility.
• No pinning (coresNum=0): UDP-only, no DPDK, no hugepages. Lowest performance but works on hosts without huge pages or with awkward NIC layouts. Common on CPU worker nodes.

For an 8x H100 node where you don't want Weka stealing cores from nvidia-smi and dataloader workers, coresNum=2 with explicit core pinning to two NUMA-local non-GPU-adjacent CPUs is a reasonable starting point.

Mount options

mount -t wekafs <cluster>/<filesystem> <mountpoint> -o <opts> accepts options that change how the helper sets things up. The important ones:

# Manual mount with explicit DPDK + 4 cores + 4 GB hugepages
$ sudo mount -t wekafs cluster-foo/datasets /wekafs/datasets \
    -o net=dpdk,num_cores=4,hugepages=4g

# Or via /etc/fstab
cluster-foo/datasets  /wekafs/datasets  wekafs  net=udp,num_cores=2  0 0

For UDP-only (no hugepages, simpler setup), the install creates a WekaClient with coresNum: 0 and udpMode: true. The mount works on any host with the kernel modules loaded, no extra prereqs.

Common mount failures

"wrong fs type, bad option, bad superblock"

$ sudo mount -t wekafs cluster-foo/data /wekafs/data
mount: /wekafs/data: wrong fs type, bad option, bad superblock on cluster-foo/data, missing codepage or helper program, or other error.

Usually one of:

• wekafs.ko not loaded → modprobe wekafs
• wekafsio.ko version mismatch with mount-helper → reinstall client
• weka-agent.service not running → systemctl start weka-agent

Check dmesg | tail -50 and the mount-helper container logs first.

"Cluster not found"

$ sudo mount -t wekafs cluster-foo/data /wekafs/data
mount.wekafs: cluster cluster-foo not found

The host doesn't have /etc/wekaio/clusters/cluster-foo/. Either the join never ran, or the directory was wiped, or you're spelling the cluster name differently than what was registered.

$ ls /etc/wekaio/clusters/
$ weka cluster list
$ weka cluster join <mgmt-ip>:14000  # if missing

Mount succeeds but every IO is slow

Almost always the transport. Check:

$ weka cluster network --host $(hostname)
$ weka stats --category client --node-ids <self> --gauge ops_per_second,latency_avg

If TRANSPORT is TCP and you expected RDMA or RoCE, your fabric is broken or the NIC isn't being picked up. See performance tuning.

"Module wekafsio is loaded but not unloadable"

Refcount > 0 from a stale mount or process holding files open. Find it:

$ lsmod | grep wekafsio
wekafsio              655360  3                 # 3 references

$ grep wekafs /proc/mounts                      # active mounts
$ for pid in $(pgrep -a . | awk '{print $1}'); do
    grep -l wekafs /proc/$pid/maps 2>/dev/null
  done | xargs -I{} cat {} | grep -B0 wekafs   # processes mmap'ing wekafs

Kill or unmount, then rmmod. See troubleshooting.

Installing / removing a client

Day-2 ops are covered separately on operations. Short version:

# Install (downloads from cluster MGMT API)
$ curl -s http://<mgmt-ip>:14000/dist/v1/install | sudo sh

# Join
$ weka cluster join <mgmt-ip>:14000 --auth-token-file <token>

# Verify
$ weka local status
$ weka cluster status

# Mount
$ mkdir -p /wekafs/foo
$ sudo mount -t wekafs cluster-foo/foo /wekafs/foo

To remove cleanly:

$ sudo umount /wekafs/foo            # unmount everything wekafs first
$ weka cluster forget cluster-foo    # de-register
$ sudo systemctl stop weka-agent
$ sudo /opt/weka/bin/uninstall.sh    # actual binary removal
$ sudo rmmod wekafs wekafsio         # drop kernel modules

If umount fails with "target is busy", a process is holding files open or you have a stale mount. lsof /wekafs/foo and kill or umount -l (lazy) as a last resort.

See also

• Architecture — what's on the backend that the client talks to
• CSI — the Kubernetes flavor of all of the above
• Performance tuning — DPDK vs UDP, choosing core counts
• Operations — install, upgrade, retire workflows

Troubleshooting

For mount errors, transport fallback, and the "stale mount blocks driver upgrade" pattern, see troubleshooting.