Weka sizing for GPU clusters — IOPS, bandwidth, capacity, and tier layout

Every GPU cluster eventually discovers that compute is easy to scale and storage is hard. A 1024-GPU cluster doing pretraining at full speed reads training data at 50–100 GB/s and writes checkpoints at 10–30 GB/s — concurrently, from hundreds of nodes simultaneously. A single NFS server cannot do this. Weka can, but only if you size it correctly from the start.

This page is the sizing reference for Weka WEKA Data Platform (formerly WekaFS) deployments behind GPU clusters. For how Weka works internally, see Weka architecture.

What Weka is and isn't

Weka is a distributed parallel filesystem running a userspace data plane (DPDK + RDMA). Key properties for sizing:

What Weka does not absorb:

• Single-host bottlenecks from misconfiguration: if all compute nodes mount through one backend host's frontend process, throughput is capped at that host's NIC bandwidth.
• Metadata-heavy workloads: small-file workloads (millions of 1 KB files) are bounded by metadata IOPS, not bandwidth. Weka metadata is fast but not unlimited.
• Object storage failures: if the S3 backend is unreachable and data has been tiered there, reads from cold data will fail or degrade.

Bandwidth math

Training data loader (read path)

Each GPU's training dataloader reads samples sequentially (for shuffled large datasets pre-shuffled at dataset prep time). The read bandwidth per GPU depends on sample size and step rate:

Read BW per GPU = sample_size_bytes × batch_size_per_gpu × steps_per_second

Example: LLM pretraining
  Token sequence: 4096 tokens × 2 bytes (BF16 stored as int16 tokens) = 8 KB per sample
  Batch size per GPU: 4
  Steps per second: 2
  Read BW per GPU = 8 KB × 4 × 2 = 64 KB/s   ← trivially small per GPU

Example: Vision model (ImageNet-scale)
  Image: JPEG ~200 KB average
  Batch size per GPU: 64
  Steps per second: 10
  Read BW per GPU = 200 KB × 64 × 10 = 128 MB/s

Example: Video model (dense pre-extracted frames)
  Frame tensor: 224×224×3 FP32 = 602 KB
  Frames per clip: 32
  Batch per GPU: 4
  Steps per second: 3
  Read BW per GPU = 602 KB × 32 × 4 × 3 = 231 MB/s

Vision and video models are bandwidth-hungry. LLM token datasets are compute-bound, not storage-bound, once loaded into RAM.

Cluster aggregate read BW target (operator rule of thumb): multiply per-GPU read BW by GPU count and add 20% margin for metadata, OS buffering, and burst behavior:

Total read BW target = per_gpu_read_bw × num_gpus × 1.2

Checkpoint write path

Checkpoints are written by all GPUs simultaneously at checkpoint intervals (typically every 100–500 training steps). For a 70B parameter model in BF16:

Checkpoint size = model_params × dtype_bytes × shards
               = 70 × 10^9 × 2 bytes = 140 GB base model
               
With optimizer state (AdamW: 2× model params): 140 × 3 = 420 GB total
With gradient accumulation buffers: add 20%, ~504 GB

At 1024 GPUs, each GPU writes its shard:
  Per-GPU write: 504 GB / 1024 = ~492 MB
  Checkpoint interval: 200 steps at 1 step/sec = 200 seconds
  Sustained write BW per GPU: 492 MB / 200 s = ~2.5 MB/s per GPU
  
Cluster aggregate write during checkpoint burst (all GPUs write simultaneously):
  1024 GPUs × 2.5 MB/s = 2,560 MB/s = 2.5 GB/s sustained average
  
  BUT checkpoints often happen in a burst (all at once): 
  1024 GPUs × 492 MB written in 30–60 seconds (practical write window)
  Burst write BW: 1024 × 492 MB / 30 s = ~16.8 GB/s

Size the write bandwidth for the burst, not the average. 16–30 GB/s aggregate write is a typical target for 1024-GPU LLM pretraining.

Capacity sizing

Tier layout

Weka organizes capacity into:

• Tier 1 (NVMe hot tier): NVMe SSDs inside the backend hosts. All active IO lands here first. Cost ~$0.10–0.30 per GB (NVMe).
• Tier 2 (object cold tier): S3-compatible object storage (AWS S3, MinIO, Pure FlashBlade S3, etc.). Cheaper but higher latency. Cost ~$0.01–0.03 per GB.

Data is tiered from Tier 1 to Tier 2 based on access time (LRU eviction from hot tier). The hot-tier capacity should hold the working set — the data accessed in the current training run.

Typical ratios by workload type:

Rule of thumb: hot tier = active_training_dataset_for_one_epoch + current_checkpoint_rolling_window + 25% headroom.

Cluster topology

Backend hosts

A Weka backend host has:

• Dense NVMe bays: typically 10–24 NVMe SSDs per host (2U or 4U servers)
• One or two HCAs (ConnectX-7 NDR) for fabric connectivity
• 2× 10G management network (out-of-band)

Weka's data placement spreads stripes across drives in the cluster. Minimum recommended: 6 backend hosts per cluster for meaningful stripe width and protection. Typical deployments: 8–32 backend hosts.

WEKA BACKEND HOST LAYOUT (2U, 24-drive NVMe)

┌───────────────────────────────────────────────────────┐
│  Backend Host (stor-01)                               │
│                                                       │
│  NVMe[0..23]: 24× 7.68 TB U.2 NVMe = 184 TB raw      │
│                                                       │
│  HCA-0: CX-7 NDR 400G (storage rail)                 │
│  HCA-1: CX-7 NDR 400G (optional second rail)         │
│                                                       │
│  Processes: 2× COMPUTE + 24× DRIVE + 1× FRONTEND     │
│             + 1× MGMT (on first host only, or all)   │
│                                                       │
│  CPU: 2× 32-core Xeon (NUMA-aligned with NVMe slots) │
│  RAM: 512 GB (Weka needs ~4–8 GB per DRIVE process)   │
└───────────────────────────────────────────────────────┘

Host-to-GPU ratio

Operator rule of thumb: 1 storage host per 4–8 compute hosts (GPU servers), depending on read bandwidth target.

More precisely: each backend host with 2× CX-7 NDR (400 Gbps each) can sustain approximately 20–30 GB/s read bandwidth to clients. A 1024-GPU cluster with compute hosts at 256 servers and a 30 GB/s aggregate read target needs:

Required storage hosts = target_read_bw / per_host_read_bw
                       = 30 GB/s / 25 GB/s per host
                       = ~1.2 hosts → minimum 2 hosts (redundancy)

For a 100 GB/s aggregate read target (heavier vision workload):

Required storage hosts = 100 GB/s / 25 GB/s = 4 hosts minimum → size to 6-8 for safety margin

The 25 GB/s per host is an approximate operator rule of thumb; actual depends on NVMe model, WEKA version, and stripe width. Profile with weka cluster bandwidth --test on your actual hardware before finalizing sizing.

HCA per Weka backend host

For read-heavy workloads:

• Single CX-7 NDR (400G): adequate for up to ~20–25 GB/s per host
• Dual CX-7 NDR (2× 400G): for up to ~40 GB/s per host; required if the host serves both training data and checkpoint workloads

For write-heavy checkpointing:

• Write path in Weka is striped across hosts; peak write BW per host is similar to read.
• Dual NICs recommended when checkpoint write BW target is > 15 GB/s per host.

Storage NIC placement: use a dedicated storage rail, not a compute rail. See NIC sizing for the rationale (checkpoint bursts can saturate shared rail uplinks during allreduce-heavy training steps).

Failure domains

Weka uses erasure coding to protect against drive, host, and optionally rack failures. The protection scheme determines the minimum host and drive count.

Protection scheme options

Laying out hosts across racks

For rack-level failure tolerance:

RACK LAYOUT FOR RACK-AWARE WEKA (minimum viable)

  Rack-A             Rack-B             Rack-C
  ┌──────────┐       ┌──────────┐       ┌──────────┐
  │stor-01   │       │stor-03   │       │stor-05   │
  │stor-02   │       │stor-04   │       │stor-06   │
  └──────────┘       └──────────┘       └──────────┘

  Configuration: rack-aware CRUSH (Weka calls this "availability zones")
  - Each stripe has drives from all 3 racks
  - If Rack-A loses power: stor-01 and stor-02 are unavailable
  - Weka can reconstruct all data from Rack-B + Rack-C
  - Requires: ≥ 3 racks, hosts balanced across racks
  
  For 12 backend hosts across 3 racks:
    Rack-A: 4 hosts, Rack-B: 4 hosts, Rack-C: 4 hosts
    Each rack failure leaves 8 of 12 hosts — sufficient for 6+2 protection

Operator note: if you're in a single-rack deployment (common for small clusters), Weka still provides drive-level protection. The risk is that a single-rack power or cooling event takes the entire storage cluster with the entire compute cluster.

Performance tuning

Key settings for Weka clients on GPU nodes:

# Mount options (wekafs POSIX mount)
mount -t wekafs <cluster-name>/<fs-name> /mnt/weka \
  -o num_cores=4,net=<nic-device>,mode=rdma,io_timeout=60

# Recommended:
#   num_cores: match to NIC count (1 per CX-7 port is a good baseline)
#   net=<nic>: bind to a specific NIC (storage rail NIC, not compute rail)
#   mode=rdma: force RDMA path (falls back to UDP if unavailable)
#   io_timeout: 60s is safer for long-running training jobs

# Hugepages — Weka DPDK requires 2 MB hugepages
echo 2048 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
# Or persist in /etc/sysctl.conf:
# vm.nr_hugepages = 2048

# CPU affinity for Weka container: WEKA pins its I/O threads to cores
# Ensure these cores are not also used by NCCL or PyTorch DataLoader workers
# Check which cores Weka is using:
$ weka status --container-detail | grep cpu

# MTU: set storage NIC MTU to 4200 (Weka recommends 4200 over 9000 for DPDK path)
$ ip link set <storage-nic> mtu 4200

# Prefetch: Weka's read-ahead is configurable per mount
$ weka fs update --read-ahead=256m <fs-name>   # larger for sequential workloads

Worked sizing examples

Example 1: 64-GPU H100 cluster, 200 TB training dataset, ~10 GB/s aggregate read

Workload: LLM pretraining + vision fine-tuning
Training data: 200 TB (mixed token + image datasets)
Aggregate read target: 10 GB/s (64 GPUs × ~150 MB/s each)
Checkpoint write burst: ~5 GB/s (small model, fast checkpoint)

Hot tier sizing:
  Working set = 1 epoch of current training run
  Assume 20 TB active at any time → hot tier: 30 TB (with 50% headroom)
  
  NVMe drives: 30 TB / 7.68 TB per drive = 4 drives minimum
  In practice: start with 8× 7.68 TB NVMe = 61 TB across 2 hosts
  
Backend hosts:
  Read target: 10 GB/s / 25 GB/s per host = 0.4 → minimum 2 hosts
  Recommended: 4 hosts for fault tolerance and headroom
  
  4 hosts × 8 NVMe drives each = 32 drives × 7.68 TB = 245 TB raw NVMe
  With 6+2 protection overhead (~20% erasure): ~196 TB usable
  Hot tier budget: 196 TB >> 30 TB needed ✓
  
HCA per backend host:
  10 GB/s / 4 hosts = 2.5 GB/s per host → single CX-7 NDR is adequate
  Use single CX-7 NDR per host, dedicated storage rail

Cold tier: 200 TB total dataset - 30 TB hot = 170 TB → S3 or object backend
  S3 policy: tier to cold after 7 days of no access

Cluster: 4 backend hosts, single CX-7 NDR each, 8× 7.68 TB NVMe each
         8 Weka processes per host (4 DRIVE + 2 COMPUTE + 1 FRONTEND + 1 MGMT)

Example 2: 256-GPU H200 cluster, multi-tenant, 1 PB warm + 100 TB hot

Workload: 4 concurrent tenant teams, mixed LLM and vision pretraining
Hot tier needed: 100 TB (active working sets for 4 teams)
Warm tier: 1 PB (all tenant datasets, 6-month retention)
Read BW target: 50 GB/s (256 GPUs × ~200 MB/s average across workloads)
Write target: 15 GB/s checkpoint burst

Backend hosts sizing:
  Read: 50 GB/s / 25 GB/s per host = 2 hosts minimum → use 8 for margin + HA
  Hot NVMe: 100 TB / (8 hosts × 7.68 TB × 4 drives) = need ~13 NVMe per host
  Use 16× 7.68 TB = 122 TB raw per host × 8 hosts = 978 TB raw
  With 6+2 protection: ~782 TB usable NVMe hot tier
  → 100 TB hot tier fits ✓

  Warm tier: 1 PB in S3 backend (MinIO cluster or cloud S3)
  
HCA per backend host:
  50 GB/s / 8 hosts = 6.25 GB/s per host → single CX-7 NDR (25 GB/s) is fine
  Dual CX-7 NDR recommended for headroom during checkpoint bursts on top of read

Failure domains:
  8 hosts across 3 racks: 3 in Rack-A, 3 in Rack-B, 2 in Rack-C
  Rack-aware protection: any single rack failure leaves 5–6 hosts → 6+2 scheme survives

Total: 8 backend hosts, 2× CX-7 NDR each, 16× 7.68 TB NVMe each, rack-spread

Example 3: Inference cluster, model-weights-only pattern

Workload: LLM inference serving — load model weights at startup, then read-only
Model: 70B parameters, BF16 = 140 GB per replica
Replicas: 16 (across 16 inference nodes)
Access pattern: 16 × 140 GB loaded in parallel at startup (bursty), then idle

Startup read BW: 16 × 140 GB loaded in, say, 60 seconds
  = 2,240 GB / 60 s = ~37 GB/s burst read

During serving: near-zero storage traffic (everything in GPU HBM)

Sizing:
  Hot tier: 140 GB × 16 replicas × 2 (headroom for model updates) = 4.5 TB
  → 2 backend hosts with 4× NVMe each is grossly over-provisioned but minimum viable
  → In practice: NFS or even a single fast NFS server works for inference weight serving
  
  Read BW: 37 GB/s burst / 25 GB/s per host = 1.5 → 2 hosts minimum
  
For inference weight-only access: 
  Weka is over-engineered. A ZFS-on-Linux NFS server with 4× NVMe can deliver
  20–30 GB/s and serve this workload at a fraction of the cost.
  Only use Weka for inference if:
  (a) The cluster is already Weka-managed (avoid managing two storage systems), OR
  (b) You need concurrent model-weight serving to a large number of nodes simultaneously
      that would saturate a single NFS server.

When NOT to use Weka

Weka is a serious system with real operational overhead. Don't use it when:

• Cluster ≤ 32 GPUs, single team: a ZFS-on-Linux NFS box with 4–8 NVMe drives delivers 10–20 GB/s at a fraction of the cost and complexity. Simple is better.
• All NVMe local disk: if training is entirely on local NVMe (not recommended for multi-node, but valid for single-node jobs), Weka adds nothing.
• Budget under $500K for the full cluster: Weka licensing adds significant cost. For small clusters, consider Lustre (open source) or NFS if the workload allows.
• Operator team unfamiliar with DPDK/RDMA: Weka requires DPDK-capable NICs, hugepages, kernel version compatibility, and RDMA. If the team doesn't operate these, consider a managed alternative.
• Write-heavy workloads with random small IOs: Weka excels at sequential bandwidth. Very small random writes (like log aggregation, timeseries) can stress the metadata tier without delivering meaningful throughput.

For clusters ≥ 64 GPUs doing continuous pretraining: Weka is typically the right answer and the operational overhead is justified by the throughput headroom.