Weka — distributed POSIX FS for HPC

What Weka is, how front-end clients and back-end storage hosts interact, clusters vs orgs, and the wekafs kernel module / mount-helper container.

▶Try the commands on this page in the command emulator — type help for the full list, or solutions for copy-paste fix recipes.

Weka is a distributed POSIX filesystem aimed at HPC and AI training workloads. It looks like NFS to a user (mount -t wekafs, ls /weka/..., regular file semantics), but underneath it shards data and metadata across many storage hosts and stripes I/O across all of them in parallel. On a tuned 8-host cluster with 100 Gbps networking you can sustain double-digit GB/s reads from a single client.

This page is what you need to know about the architecture before you start operating it: front-end vs back-end, clusters vs orgs, and how the client-side kernel module + mount-helper container fit together.

How Weka differs from NFS, Lustre, GPFS

Filesystem	Architecture	POSIX	Strengths	Weaknesses
NFS	Single server (NFSv3) or scale-out (NFSv4.x)	Yes	Simple, ubiquitous	Single-server bottleneck, slow lookups
Lustre	MDS + OSS, separate metadata/data servers	Yes	Massive scale, decades of HPC use	Operationally heavy, MDS = SPOF if so
GPFS / Spectrum Scale	Symmetric, NSDs across all servers	Yes	Mature, widely deployed	Licensed, complex
Ceph (CephFS)	RADOS object store with FS layer	Yes	Open source, unified block/obj/file	Latency, slower than dedicated FS
Weka	All-flash, parallel data + parallel metadata, RDMA-native	Yes	Very fast, snapshots, S3 tiering, GPUDirect	Proprietary, expensive, all-flash only

The architectural distinction that matters for operators: Weka has no central MDS. Metadata is sharded across all back-end hosts, the same way data is. There is no single server that bottlenecks stat() calls — and on a real HPC workload (millions of small file lookups during checkpoint loads), that single design choice is why Weka beats Lustre on the same hardware.

Front-end vs back-end

A Weka deployment has two roles:

Role	Where it runs	What it does
Back-end (storage host)	Dedicated nodes with NVMe drives + 100/200 Gbps NICs	Holds data, runs metadata service, replicates to peers
Front-end (client)	GPU/HPC compute nodes	Mounts the FS via `wekafs` kernel module, talks to back-ends over RDMA

Storage hosts in a Weka cluster are typically 6-12 dense NVMe servers (e.g., 24× U.2 NVMe per host). They run a Weka container that owns the drives and the user-space data path. The client side is much lighter: a kernel module + a small container.

Some deployments collocate front-end and back-end on the same nodes — the Weka container runs on every GPU node, exposing local NVMe to the cluster while the GPUs use the FS. This is the "hyperconverged" mode. It saves hardware but ties storage availability to compute availability.

Clusters vs orgs

A Weka cluster is one storage fabric — one set of back-ends with one capacity pool. A Weka org (organization) is a logical tenant boundary inside a cluster: separate filesystems, separate auth, separate quotas. One cluster can host many orgs.

This matters because clients mount one org at a time:

mount -t wekafs cluster1.example.internal/tenant-foo-fs /weka -o net=ib0

The path cluster1.example.internal/tenant-foo-fs includes both the cluster identity and the org/filesystem. A node that needs to access a different org has to mount it separately. This becomes a problem when nodes are reassigned between orgs — see the Weka troubleshooting page for the stale-mount issue.

Mounting Weka — the wekafs kernel module + mount-helper

There are two pieces on the client:

1. `wekafs` kernel module

A loadable kernel module (wekafs.ko) that registers a filesystem type. mount -t wekafs ... invokes it. The module talks to user-space via a Unix socket to the mount-helper container (a small persistent container the Weka client provides).

lsmod | grep weka
# wekafsio              999424  3        <-- refcount = 3, meaning 3 active mounts
# wekafs                  4096  0

The wekafsio refcount is the number of active mounts. Critical for the stale-mount troubleshooting — if the module is in use, it can't be unloaded, which blocks driver upgrades.

2. mount-helper container

The Weka client deploys a container per node that holds the user-space portion of the data path: connection to back-ends, request multiplexing, RDMA setup. The kernel module forwards requests to it via a Unix socket.

When the mount-helper container exits, mounts that depended on it become "stale" — the kernel module sees the socket disappear, marks the mount dead, and any further I/O returns ENOTCONN ("Transport endpoint is not connected").

You can see it:

weka local status
# Containers: 1 running                       <-- the mount-helper
# Mounts: 3 active

ps -ef | grep weka
# weka  12345  ...  /usr/bin/weka_mount_helper ...

ls /sys/fs/wekafs/
# tenant-foo-fs   tenant-bar-fs

Mounting on a Kubernetes node — CSI driver

Manual mounts are fine for non-K8s hosts. Inside K8s, the Weka CSI driver does it:

PVC is created in a tenant namespace.
CSI controller talks to Weka cluster, ensures filesystem/quota exists.
CSI node-plugin (DaemonSet) mounts the filesystem on the host node when a pod needs it, exposes it to the pod via bind mount.

See the CSI page for the Kubernetes-side integration.

Authentication

Weka clients authenticate to the cluster via organization tokens. Each org has a token; the client passes it at mount time:

weka user login --org tenant-foo --token <token>
mount -t wekafs cluster1/tenant-foo-fs /weka -o net=ib0

In K8s, the CSI driver pulls the token from a Secret. Stored in the Weka cluster's API responses, scoped to the org, can be rotated.

Snapshots and S3 tiering

Two operational features worth knowing about:

Snapshots — point-in-time copy of an entire filesystem. Cheap (CoW), instant. weka fs snapshot create ... or via the UI. Used heavily for "back up before this destructive op" and for tenant data branching.

S3 tiering — Weka can offload cold data to S3 (or any S3-compatible object store), keeping a stub on the cluster. Reads of cold data trigger a transparent rehydration. Cuts storage cost dramatically for long-tail data, at the cost of unpredictable rehydration latency on first access.

Configured per-filesystem:

weka fs tier add tenant-foo-fs --object-store s3-bucket --endpoint ...
weka fs tier policy set tenant-foo-fs --hot-days 7 --warm-days 30

What an operator usually touches

For day-to-day ops:

# On a client node
mount | grep wekafs                # what's mounted
weka local status                  # mount-helper container status
weka local stop / start            # restart the helper (use rarely; remounts everything)

# Cluster-side (logged into a back-end or via API)
weka cluster status
weka fs                            # list filesystems
weka org                           # list organizations
weka stats                         # live IOPS / throughput
weka events list -n 50             # event log
weka alerts                        # active alerts

# Per-host
weka cluster host list
weka cluster host info <id>
weka cluster drive list             # all NVMe drives across the cluster

Performance characteristics

A healthy 8-host all-NVMe Weka cluster on 200 Gbps NICs:

Single-client read: 10-25 GB/s sequential
Single-client write: 8-15 GB/s sequential
Aggregate: 100+ GB/s (limited by NIC count × bandwidth)
IOPS: millions, depending on metadata workload
Metadata ops/sec: hundreds of thousands

If you're seeing significantly less, the usual suspects are:

Network path issue (RDMA not actually being used; falling back to TCP)
One back-end host saturated (uneven data placement)
Client-side network bottleneck (single 25 Gbps NIC trying to push 25 GB/s)

weka stats throughput and per-host weka cluster host info show whether the load is balanced.