Weka architecture: how the filesystem actually works

Weka is marketed as a parallel filesystem for AI, but under the hood it's an unusual beast — a userspace data-plane (DPDK + RDMA) bolted to a kernel module (wekafs.ko / wekafsio.ko) that exposes a POSIX mount, with metadata sharded across the cluster and data placed via erasure coding across hundreds of NVMe drives. If you operate Weka at scale you will eventually need to debug all of those layers at once. This page is the working operator's mental model.

Two halves: front-end and back-end

Every Weka deployment has two roles, and a host can be one, the other, or both:

A typical AI cluster has a small dedicated BE (8–32 storage hosts in a rack) and many FE clients (every GPU node and every CPU worker). The clients are stateless — wipe one, reboot it, install the client again, and it picks up exactly where it left off because the data lives on the BE.

+--------------------------------+      +--------------------------------+
|  GPU node (front-end client)   |      |  Backend host                  |
|                                |      |                                |
|  app -> wekafs.ko (kernel) --+ |      |  +-> COMPUTE proc (port 14000) |
|                              | |      |  +-> COMPUTE proc (port 14001) |
|  mount-helper container -----+ |      |  +-> DRIVE proc   (port 14002) |
|        |                       |      |  +-> DRIVE proc   (port 14003) |
|        +-> userspace data path |      |  +-> FRONTEND proc(port 14004) |
|            (DPDK/RDMA/UDP) ----+------+--+-> MGMT proc    (port 14005) |
|                                |  net |                                |
+--------------------------------+      +--------------------------------+
                          (RDMA preferred, UDP/TCP fallback)

Container ports model

Each backend host runs multiple Weka container processes, not one. The split exists so that a single host failure or single-process crash doesn't take the cluster down, and so each process can be CPU-pinned and NUMA-local to specific NVMes.

A typical backend layout per host:

Each container listens on its own TCP/UDP port starting at 14000 and going up. So a single backend host typically occupies 14000 through ~14010. When you see weka cluster container output, that's what you're looking at — every row is one userspace process.

$ weka cluster container
CONTAINER ID  HOSTNAME      MODE       STATUS  IPS                IPV4 PORT
0             stor-01       MANAGEMENT UP      10.10.10.1         14000
1             stor-01       FRONTEND   UP      10.10.10.1         14001
2             stor-01       COMPUTE    UP      10.10.10.1         14002
3             stor-01       COMPUTE    UP      10.10.10.1         14003
4             stor-01       DRIVE      UP      10.10.10.1         14004
...

On the client side, the host runs a single client container (the mount-helper), which in turn brokers all wekafs mounts on that host. One container per host, regardless of how many filesystems are mounted.

The kernel module: wekafs vs wekafsio

The client side has two kernel modules and they confuse people:

When wekafsio is loaded but lsmod shows non-zero refcount, every mount and every process with an open file on /wekafs is holding a reference. If you can't rmmod it, you can't upgrade the client — see troubleshooting: stale mounts.

$ lsmod | grep weka
wekafsio              655360  3
wekafs               1130496  1 wekafsio

The 3 means three mounts (or active opens) hold a reference. To drop them: unmount everything wekafs, kill processes still holding files on it, then rmmod.

Cluster bootstrap: identity and trust

A Weka cluster is identified by:

• Cluster ID — UUID generated at format time. Burned into every backend container's config.
• Cluster fingerprint — public key derived from the cluster cert. Clients verify this on join.
• Org — multi-tenant boundary inside the cluster. Each org has its own users, filesystems, quotas, and credentials.
• License — capacity-based, signed by Weka. Without it, IO degrades or stops.

Clients that join a cluster store the (cluster-id, fingerprint, org, token) tuple in /etc/wekaio/. If the cluster ID doesn't match what they have cached, the mount fails with Cluster <X> not found.

# Inspect what a client is configured to talk to
$ ls /etc/wekaio/
clusters/  current/  global/

$ cat /etc/wekaio/current
cluster-foo

$ cat /etc/wekaio/clusters/cluster-foo/cluster.json | jq '{name, uuid, mgmtIPs}'
{
  "name": "cluster-foo",
  "uuid": "11111111-2222-3333-4444-555555555555",
  "mgmtIPs": ["10.10.10.1:14000", "10.10.10.2:14000", "10.10.10.3:14000"]
}

Data placement: chunks, stripes, and erasure coding

Weka's data layout has three tiers of granularity.

Chunks

Every file is split into fixed-size chunks, default 1 MiB. Small files (< 4 KiB) live inline in the metadata DB; everything bigger gets chunked.

Stripes and erasure coding

Each chunk is encoded into a stripe of K + P blocks: K data blocks plus P parity blocks. Common schemes:

The K + P value cannot exceed the failure domain count — if you have 6 backends and want 4+2, that's at the edge; you can't do 8+2 on 6 backends. Failure domains can be hosts, racks, or zones depending on configuration.

Placement groups

A placement group is a set of K + P backend processes that own a stripe. Weka spreads stripes across all available placement groups using a deterministic hash, so adding capacity rebalances automatically (slowly — Weka throttles rebalance to avoid impacting client IO).

When you read a file:

1. Client looks up the inode in the metadata DB → gets stripe locations.
2. Client opens connections (RDMA QPs or UDP sockets) to the relevant DRIVE processes.
3. Reads K data blocks in parallel; if any are missing/slow, decodes from parity.
4. Returns bytes to the kernel wekafs.ko, which copies to user buffer.

The "any K of K+P" property is why Weka can survive a backend reboot without IO pausing — the surviving N-1 backends still have at least K blocks for every stripe.

Metadata: the cluster-wide DB

File metadata (inodes, dirents, xattrs, ACLs, snapshots) lives in a distributed key-value store sharded across the COMPUTE processes. It's a custom B-tree-ish structure with strong consistency — stat() of a file from any client returns the same answer at the same instant. There is no NFS-style attribute caching weirdness.

This is also why Weka handles small-file workloads well that NFS does poorly: a find /wekafs -type f doesn't melt the cluster the way it does on a single-server NFS export — the metadata work is parallelized.

The cost: metadata operations (open/stat/getdents) require a network round-trip. On a fast RDMA fabric, this is ~10–30 µs per op. On TCP fallback, it can climb to 200+ µs, which is when "ls is slow" complaints start. Always check the network path first.

Network: RDMA-first, then UDP/DPDK, then TCP

Weka's data path can run over three transports:

The cluster auto-negotiates per-link. A single broken NIC can silently push a client into TCP fallback, which is the #1 cause of "Weka is slow today" tickets. Always check transport with weka cluster network and weka stats before tuning anything else.

$ weka cluster network
HOST       NIC         STATE  TRANSPORT  IP          MTU
stor-01    mlx5_0      UP     IB         10.10.10.1  4092
stor-01    mlx5_1      UP     IB         10.10.11.1  4092
client-42  mlx5_0      UP     RoCE       10.10.20.4  4200
client-43  mlx5_0      UP     TCP        10.10.20.5  1500   <-- fell back!

Ports used:

• 14000–14100 TCP/UDP — userspace container processes (data + control)
• 14000 TCP — REST API and CLI default
• RDMA — uses standard RoCE/IB ports (no special UDP port mapping)

Tiering: SSD tier and S3 tier

Weka can transparently tier cold data to S3 (or S3-compatible object storage). The hot tier lives on cluster NVMe; the cold tier lives in object storage.

+--------------------+        +-------------------+
|   SSD tier (hot)   |        |  S3 tier (cold)   |
|   NVMe in cluster  |  <-->  |  S3 / R2 / OSS    |
|   ~100 µs latency  |        |  ~50 ms latency   |
+--------------------+        +-------------------+

Lifecycle policies decide what tiers to S3:

• Time-based: files not accessed in N days → tier
• Size-based: files > X MB → eligible
• Path-based: /wekafs/cold/ always tiers, /wekafs/hot/ never tiers

When a client reads a tiered file, Weka transparently fetches from S3, caches on SSD, and serves the read. The first read is slow (object-storage latency); subsequent reads are SSD-fast. Random access to cold-tier files is genuinely slow — there is no magic.

For most AI workloads with tight latency budgets, S3 tiering is off for training data and on for checkpoints/datasets that get re-used months apart.

See also

• Clients — mount-helper container, multi-cluster, mount options
• CSI — Kubernetes integration, drivers-loader, dynamic provisioning
• Performance tuning — DPDK vs UDP, fio recipes, expected numbers
• Troubleshooting — stale mounts, transport fallback, error patterns

Troubleshooting

If something doesn't match this model — for example, a backend container is STAGED instead of UP, or weka cluster network shows a client on TCP when it should be RDMA — see the troubleshooting page.