Weka performance tuning: transport, cores, hugepages, and what 3 GB/s actually means

A healthy Weka client on a healthy fabric should saturate either the client's NIC or the cluster's per-client provisioning, whichever is smaller. In practice we routinely see GPU nodes at ~3 GB/s sustained read against a well-sized cluster — that's a real number, not a brochure number. When you see substantially less, it's almost always one of three things: transport fell back to TCP, the client doesn't have enough cores/hugepages dedicated, or you're benchmarking with the wrong access pattern. This page is the diagnostic and tuning checklist.

The performance hierarchy

Before you tune anything, know which layer is bottlenecked. There are three:

Tune in that order. If the network is broken (transport fallback), nothing you do at the client layer helps. If the client is saturated, expanding the cluster doesn't help.

Transport: prefer DPDK or RDMA over TCP

The single biggest performance variable. Confirm transport before anything else:

$ weka cluster network --host $(hostname)
HOST     NIC      STATE  TRANSPORT  IP          MTU
gpu-42   mlx5_0   UP     RoCE       10.10.20.4  4200    # good
gpu-43   mlx5_0   UP     IB         10.10.20.5  4092    # good
gpu-44   mlx5_0   UP     UDP        10.10.20.6  9000    # OK if no DPDK
gpu-45   mlx5_0   UP     TCP        10.10.20.7  1500    # bad, fell back

Order of preference:

1. RDMA (RoCE or IB) — line rate, lowest latency. Default when NICs and switches support it.
2. UDP/DPDK — userspace zero-copy with kernel bypass. Excellent throughput, requires hugepages.
3. UDP (kernel) — the udpMode: true path. Works without hugepages, lower per-client throughput than DPDK but no setup gotchas.
4. TCP — emergency fallback. Avoid in production. Caps you at ~1 GB/s per client and adds latency spikes.

If a client is on TCP unexpectedly, the fabric or NIC is broken — fix it before doing any other tuning. See troubleshooting: slow IO with low CPU.

Client cores

Weka's userspace data path runs in dedicated polling threads — the cores it owns are running at 100% all the time, even when idle, because that's how DPDK / RDMA polling works. More cores = more parallel IO requests in flight.

# What does this client have?
$ weka local resources
CONTAINER     MEMORY     CORES        HUGEPAGES
client0       2.0 GB     0,1          2 GB

# How saturated are they?
$ weka stats --category client --node-ids self --gauge cpu_usage_pct

If cpu_usage_pct is consistently 90%+ on the Weka cores during high IO, you're client-CPU-bound. Options:

1. Add more cores — re-deploy the client with coresNum: 4 or 8. Each added core costs you one CPU permanently from the host's allocatable pool.
2. Switch to DPDK if you're on UDP (kernel) — same number of cores, much higher throughput per core.
3. Reduce concurrent IO — if the workload is doing 64 parallel readers and you only have 2 client cores, the cores will be the bottleneck.

For an 8x H100 GPU node:

Pinning the cores to NUMA-local non-GPU-adjacent CPUs is worth doing on systems with 8+ NUMA nodes. The Weka installer can do this automatically with --cores-pinning auto.

Hugepages

DPDK uses hugepages for its packet rings and shared memory with the kernel. The default ask is 1–2 GiB per client core; 2 MB hugepages are fine, 1 GiB hugepages are slightly better.

Reserve hugepages at boot via the kernel command line:

# /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="... default_hugepagesz=2M hugepagesz=2M hugepages=4096"
# 4096 * 2 MB = 8 GB reserved

Or at runtime (best-effort):

$ echo 4096 | sudo tee /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
$ cat /proc/meminfo | grep -i huge
HugePages_Total:    4096
HugePages_Free:     2048      # client0 already grabbed 2048

If hugepages aren't available, DPDK won't initialize — the client either falls back to UDP-without-DPDK or fails to start. Logs will say EAL: FATAL: Cannot get hugepage information.

Easiest hugepage-free path: coresNum: 0 + udpMode: true. No DPDK, no hugepages, no per-host interface naming gotchas. This is the right choice for CPU worker nodes and for clusters where you can't reliably reserve hugepages at boot. Performance is lower than DPDK but consistent.

Access patterns: large sequential vs random small

Weka is genuinely good at both extremes. The tuning differs:

Large sequential (training data, video, model weights)

Default settings are fine. Bandwidth-limited workload. Use:

# 1 MB blocks, single-stream read
$ fio --name=seqread --filename=/wekafs/foo/test --rw=read \
      --bs=1M --size=10G --direct=0 --numjobs=1

Expect ~3 GB/s on a healthy H100 node. Multi-stream (--numjobs=4) can push higher if the backend has provisioning headroom.

Random small (databases, lots of tiny files)

Latency-limited. The default Weka client settings are tuned for this case. Use:

# 4K random reads
$ fio --name=randread --filename=/wekafs/foo/test --rw=randread \
      --bs=4k --size=10G --direct=1 --iodepth=32 --numjobs=4

Expect O(100k) IOPS per client on a healthy fabric. The --direct=1 is essential — without it, the page cache makes the test meaningless.

Don't mix these mentally

A "Weka is slow" report needs to specify access pattern. 200 MB/s on a 4K random read at iodepth=1 is not slow — that's ~50k IOPS, exactly what one would expect. 200 MB/s on a 1M sequential read is slow — should be 10x that.

O_DIRECT: when to use, when to avoid

O_DIRECT (POSIX open(... O_DIRECT), fio --direct=1) bypasses the Linux page cache. Use it for:

• Benchmarking real backend performance, not cache performance.
• Databases that manage their own buffer pool (MySQL InnoDB, RocksDB) and don't want the OS to double-cache.
• Streaming workloads where you read each block once and never come back.

Avoid it for:

• Re-read workloads — multi-epoch training over a dataset that fits in RAM. With O_DIRECT you go to the backend every epoch; without it, epochs 2+ are at memory speed.
• Random access in apps that don't have their own buffer pool. The page cache will substantially help.

The default for fio is buffered (no O_DIRECT). The default for typical training dataloaders (PyTorch, TF) is buffered. Most apps are fine with the default; the question is whether to explicitly disable caching.

mmap performance

mmap-based file access works on wekafs but has nuances. The wekafs driver maps file pages on demand — the first access faults a page in, subsequent accesses are page-cache fast. Sequential mmap reads are fine. Random mmap access patterns can cause more page-fault churn than you'd expect, because each page-fault is a network round-trip on the first access.

For very large files accessed mmap-style (e.g., parquet readers, embedding tables), measure both ways:

# Buffered read benchmark
$ time python -c "f=open('/wekafs/foo/data.bin','rb'); f.read()"

# mmap benchmark
$ time python -c "import mmap; f=open('/wekafs/foo/data.bin','rb'); m=mmap.mmap(f.fileno(),0,prot=mmap.PROT_READ); _=m[:]"

If mmap is much slower, fall back to read(2). If they're comparable, mmap saves the user-buffer copy and is the right call.

fio recipes for Weka characterization

A small battery to run on a fresh client to validate health:

# Common: dedicated test directory, never run on shared production data
$ TESTDIR=/wekafs/foo/perftest-$(hostname)-$(date +%s)
$ mkdir -p $TESTDIR

# 1. Sequential read, single stream — should hit ~3 GB/s on H100/H200 with RDMA
fio --name=seqread1 --rw=read --bs=1M --size=20G --direct=0 \
    --numjobs=1 --filename=$TESTDIR/seq.bin

# 2. Sequential read, 4 streams — should saturate a 100Gb NIC, push 8x more on 400Gb
fio --name=seqread4 --rw=read --bs=1M --size=20G --direct=0 \
    --numjobs=4 --group_reporting --filename=$TESTDIR/seq.bin

# 3. Sequential write
fio --name=seqwrite --rw=write --bs=1M --size=20G --direct=1 \
    --numjobs=1 --filename=$TESTDIR/seqw.bin

# 4. Random 4K read at queue depth — IOPS-limited
fio --name=randread --rw=randread --bs=4k --size=10G --direct=1 \
    --iodepth=32 --numjobs=4 --runtime=60 --time_based \
    --group_reporting --filename=$TESTDIR/rand.bin

# 5. Mixed 70/30 read/write
fio --name=mixed --rw=randrw --rwmixread=70 --bs=4k --size=10G --direct=1 \
    --iodepth=32 --numjobs=4 --runtime=60 --time_based \
    --group_reporting --filename=$TESTDIR/mixed.bin

# Cleanup
$ rm -rf $TESTDIR

Save outputs and use them as a baseline. Re-run after fabric or driver changes to confirm no regression.

Real-world reference numbers

From healthy H100-class GPU nodes hitting a well-sized Weka cluster on RoCE or IB:

These are per-client. Cluster-aggregate is much higher and is bounded by how many drives are participating and how the backend is provisioned. A 16-host cluster with 8 NVMes per host can comfortably push 100+ GB/s aggregate read.

Telling client vs backend vs network

When throughput is below expected, run all three checks in parallel:

# 1. NETWORK: am I on the right transport?
$ weka cluster network --host $(hostname)
# If TCP -> fix fabric, stop here

# 2. CLIENT: are my Weka cores saturated?
$ weka stats --category client --node-ids self --gauge cpu_usage_pct
# If > 90% sustained -> add cores or switch to DPDK

# 3. BACKEND: is the cluster busy?
$ weka stats --category cluster --gauge \
    iops_read,bandwidth_read,latency_avg_us
$ weka cluster status
# If backend is at its IOPS limit, multiple clients see the slowdown.
# If only YOUR client is slow, backend is not the issue.

Cross-reference with cluster-wide events:

$ weka events list --severity WARNING --num-results 100

Backend-side warnings about disk failures, rebalancing, or capacity pressure can quietly degrade per-client performance even when no individual component is "down".

See also

• Architecture — what RDMA-first really means at the wire level
• Clients — mount options that affect performance
• Network performance tuning — host-side network stack tuning that Weka inherits
• Troubleshooting — when slow IO is a fault, not a tuning issue

Troubleshooting

If weka cluster network shows TCP fallback, that's the slow IO with low CPU pattern — fix transport before tuning anything else.