Ceph for GPU/HPC — when it fits, sizing, and the trade-offs vs Weka

Ceph is the most widely deployed open-source distributed storage system in HPC and cloud-adjacent environments. It runs on commodity hardware, supports three access protocols (CephFS, RBD, RGW), and integrates with almost every Linux tool. The question for GPU cluster operators is not whether Ceph can do the job — it can — but whether it should, and under what conditions.

Why Ceph in HPC at all

The case for Ceph in GPU/HPC clusters:

• Open source with no licensing cost: for operators who can absorb the operational overhead, eliminating per-TB licensing is a real budget advantage over Weka.
• Multi-protocol: the same Ceph cluster can provide CephFS (POSIX), RBD (block), and RGW (S3-compatible object) simultaneously. One system instead of three.
• Ecosystem: deep integration with OpenStack, Rook (Kubernetes), CSI drivers, Prometheus metrics. If your infra is Rook/K8s-based, Ceph is natural.
• Hardware flexibility: any commodity server with NVMe/SSD/HDD works. No NIC-level DPDK requirements (though RDMA messaging is available and recommended for performance).

The case against Ceph in high-throughput training clusters:

• Latency on the read path: Ceph's POSIX metadata (MDS) and data path have p99 latency typically 3–5× higher than Weka for small random reads and metadata operations. For sequential read-intensive training dataloaders, this is less critical — sustained bandwidth matters more than p99 latency.
• Parallel client scaling: CephFS's metadata server (MDS) is a bottleneck under very high concurrency (thousands of clients opening new files simultaneously). Weka's distributed metadata handles this better.
• Operational complexity: tuning BlueStore, PG counts, CRUSH maps, and OSD recovery simultaneously requires deep expertise.

The honest comparison

These are operator rules of thumb. Real numbers depend heavily on hardware, cluster size, and tuning level.

CephFS vs RBD vs RGW

CephFS

Use for: training datasets (POSIX semantics required by dataloader), shared model repositories, shared scratch space.

CephFS exposes a POSIX filesystem. GPU nodes mount it with the kernel CephFS client or the FUSE client. For training, use the kernel client (lower latency, higher throughput).

# Kernel CephFS mount (recommended for GPU nodes)
mount -t ceph <mon-ip>:/ /mnt/cephfs -o name=admin,secret=<keyring>,rsize=65536,wsize=65536

# Better performance options:
mount -t ceph <mon-ip>:/ /mnt/cephfs \
  -o name=<client-name>,secretfile=/etc/ceph/client.keyring,\
     rsize=1048576,wsize=1048576,noshare

Key CephFS limitations for GPU clusters:

• MDS is a single-threaded metadata bottleneck per active MDS daemon. Run MDS in HA pairs (active + standby) and consider multiple active MDS with directory pinning for very large namespaces (>50M files).
• Open-many concurrency: when all 1024 GPU processes open their training shard files simultaneously at job start, MDS is hammered. Pre-open or pre-stat files in the dataloader's __init__ to warm the MDS cache.

RBD (RADOS Block Device)

Use for: VM disks (not typical in GPU clusters), databases, or when you need a block device interface and have a single writer.

RBD in GPU clusters: used occasionally for OS images or container overlay storage, not for training data or checkpoints (single-writer limitation makes it wrong for parallel distributed access).

RGW (RADOS Gateway)

Use for: S3-compatible object storage for cold-tier data, dataset long-term archival, model artifact storage.

RGW pairs naturally with Weka's S3 tier: Weka backends tier to an RGW endpoint, giving you a unified Ceph + Weka story at lower cost than a dedicated S3 service. It also works standalone as a dataset archive for the GPU cluster.

# Configure a Weka filesystem to tier to RGW
weka fs tier s3 add \
  --filesystem <fs-name> \
  --obs-name ceph-rgw-tier \
  --bucket training-data \
  --access-key <key> \
  --secret-key <secret> \
  --obs-endpoint http://rgw.internal:7480

BlueStore tuning

BlueStore is Ceph's on-OSD storage engine (replaced Filestore in Nautilus). All modern Ceph deployments use BlueStore.

WAL and DB sizing

BlueStore separates data, WAL (write-ahead log), and DB (RocksDB metadata). The WAL and DB can be placed on faster storage (NVMe) while the bulk data stays on cheaper media.

For all-NVMe OSD deployments (typical for GPU cluster performance):

• Put WAL and DB on the same NVMe as data (co-located): simplest, avoids complexity.
• Only split WAL/DB to a separate faster NVMe tier if you have mixed NVMe + HDD and need to accelerate metadata.

Recommended WAL/DB sizing for all-NVMe OSD:
  DB size: 4% of OSD capacity (RocksDB metadata grows with object count)
         = 7.68 TB × 0.04 = ~307 GB per OSD (use 512 GB for headroom)
  WAL size: 1–2 GB per OSD (write burst buffer; small and fast)
  
  On a 7.68 TB NVMe OSD with co-located BlueStore:
    bluestore_block_size = 7,680 GB (the OSD data device)
    bluestore_block_db_size = 512 GB
    bluestore_block_wal_size = 2 GB

For an all-NVMe cluster, co-location is preferred — splitting to a separate device adds a second device failure mode per OSD.

OSD-per-NVMe

The common Ceph pattern: one OSD per NVMe. Splitting one NVMe into multiple OSDs can improve parallelism in some workloads but adds management overhead and is rarely needed for training data access patterns.

bluestore_min_alloc_size = 65536  # 64 KB for NVMe (default 4K is too small)
# 64 KB aligns with typical HPC sequential IO and reduces RocksDB write amplification

PG count

PG (Placement Group) count is one of the most common misconfiguration points in Ceph. Too few PGs: uneven data distribution and slow recovery. Too many: wasted memory.

Formula (operator rule of thumb):

PGs per pool = (OSDs × 100) / replication_factor

Example: 32 OSDs, 3× replication:
  PGs = (32 × 100) / 3 = ~1067 → round to nearest power of 2 = 1024

Example: 64 OSDs, erasure-coded 6+2 (k=6, m=2):
  Effective replication = k + m = 8
  PGs = (64 × 100) / 8 = 800 → use 1024 (round up to power of 2)

Ceph's pg_autoscaler can manage this automatically in recent Ceph releases:

ceph config set global mon_target_pg_per_osd 100
ceph mgr module enable pg_autoscaler
ceph osd pool set <pool> pg_autoscale_mode on

Prefer autoscaler for new deployments; it removes the guesswork.

Networking

Cluster network vs. public network

Ceph uses two networks:

• Public network: clients (GPU nodes, applications) connect here. OSD → MDS traffic also uses this.
• Cluster network: OSD-to-OSD replication and recovery traffic flows here only.

Separating them prevents recovery storms from impacting client reads. For GPU clusters:

Public network:  25 GbE or 100 GbE per OSD host (connects to same fabric GPU nodes use)
Cluster network: 25 GbE (replication is typically lower BW than client reads)
                 OR use the same 100 GbE but with separate VLAN + QoS

Rule of thumb: if OSD host has 2 NICs, dedicate one per network.
               If only 1 NIC: use VLAN separation and QoS DSCP marking to protect client traffic.

MTU

Set MTU 9000 (jumbo frames) on both networks. Ceph's default 1500 MTU is a real performance limiter at 25G+.

# Set on OSD host NICs
ip link set <nic> mtu 9000

# Verify Ceph is using jumbo frames
ceph config set global ms_osd_msg_size 65536

RDMA messaging

Ceph supports RDMA via the msgr2 RDMA backend (ms_type = async+rdma). For GPU clusters where both the Ceph OSD hosts and the GPU clients have ConnectX-7 NICs on the same fabric, RDMA messaging gives 20–30% lower latency on CephFS reads.

Enable with care: RDMA messaging in Ceph requires all nodes to have RDMA-capable NICs on the same fabric. Mixed RDMA/TCP clusters require careful routing.

[global]
ms_type = async+rdma
ms_async_rdma_device_name = mlx5_0   # match to actual IB/RoCE device
ms_async_rdma_port_num = 1

CRUSH map for failure domains

CRUSH (Controlled Replication Under Scalable Hashing) determines how Ceph places data across failure domains.

Basic rack-aware CRUSH

CRUSH HIERARCHY (host + rack + root)

root default
  rack rack-a
    host stor-01  [OSD.0, OSD.1, OSD.2, OSD.3]
    host stor-02  [OSD.4, OSD.5, OSD.6, OSD.7]
  rack rack-b
    host stor-03  [OSD.8, OSD.9, OSD.10, OSD.11]
    host stor-04  [OSD.12, OSD.13, OSD.14, OSD.15]
  rack rack-c
    host stor-05  [OSD.16, OSD.17, OSD.18, OSD.19]
    host stor-06  [OSD.20, OSD.21, OSD.22, OSD.23]

Replication rule: 3 replicas, one per rack
  Data is replicated to 3 OSDs, each in a different rack.
  A single rack failure loses one replica; data remains accessible.

Define rules in CRUSH:

# Create a rule that places across racks
ceph osd crush rule create-replicated rule-rack-aware default rack
ceph osd pool set <pool> crush_rule rule-rack-aware

For erasure-coded pools:

# k=4+m=2 with rack-level failure domain
ceph osd erasure-code-profile set ec-rack-profile \
  k=4 m=2 \
  crush-failure-domain=rack
ceph osd pool create <pool-name> erasure ec-rack-profile

Worked sizing

Dataset + checkpoint workload

Cluster: 128-GPU H100 cluster (16 servers × 8 GPUs)
Workload: LLM pretraining, 50 TB training dataset, checkpointing every 200 steps
Target aggregate read BW: 15 GB/s (128 × ~120 MB/s)
Target checkpoint write burst: 8 GB/s (small model, tight checkpoint window)

OSD sizing:
  Drive: 7.68 TB NVMe
  Raw capacity needed: 50 TB + checkpoint rolling window (~5 TB) = 55 TB
  With 3× replication: 55 × 3 = 165 TB raw
  OSDs needed: 165 TB / 7.68 TB = 21.5 → 24 OSDs (round to even number)
  OSD hosts at 4× NVMe each: 24 / 4 = 6 hosts

Read BW check:
  Per-OSD read BW on NVMe (sequential): ~4–5 GB/s (conservative estimate)
  24 OSDs × 4 GB/s = 96 GB/s aggregate raw OSD BW
  CephFS client overhead: ~60–70% efficiency → ~58–67 GB/s available to clients
  Target: 15 GB/s — well within capacity ✓

Write BW check:
  Checkpoint burst: 8 GB/s → 8 × 3 (replication) = 24 GB/s write to OSDs
  24 OSDs × 4 GB/s write = 96 GB/s OSD capacity → 24 GB/s within budget ✓

OSD per host: 4× 7.68 TB NVMe per host
HCA per host: 1× 25GbE (public) + 1× 25GbE (cluster), or 1× 100GbE dual-port
PG count: (24 × 100) / 3 = 800 → use 1024

Final: 6 OSD hosts, 4× NVMe each, 2× 25 GbE each, MDS on 2 dedicated hosts

Realistic latency expectations

These are operator rules of thumb for a well-tuned Ceph cluster on all-NVMe OSDs:

Latency differences matter for metadata-heavy workloads (many stat() and open() calls). For training dataloader patterns with large sequential reads and a warm page cache, the difference between Ceph and Weka is smaller.

When to use Ceph in a GPU cluster

Use Ceph when:

• Multi-protocol requirement: you need CephFS + S3 + block from the same system. Weka's tiering handles part of this but not all.
• Budget constraints: eliminating Weka licensing saves $50K–$200K+ for mid-size deployments. If the team can operate Ceph, the saving is real.
• Existing Ceph expertise: if the ops team already runs Ceph for other services, extending it to GPU storage is lower incremental overhead than introducing Weka.
• Kubernetes integration: Rook-Ceph is mature and provides Kubernetes CSI, PVC lifecycle management, and operator-based management. Better native K8s integration than Weka's CSI driver in some environments.
• Heterogeneous workloads: GPU training shares the cluster with CPU HPC jobs, databases, and object storage consumers. Ceph handles all of them; Weka is tuned for AI/ML.

Skip Ceph (prefer Weka) when:

• Peak training throughput is the primary constraint and budget is available.
• The cluster is single-tenant and uses exclusively Slurm or a single Kubernetes namespace.
• The operations team doesn't have Ceph expertise and no budget to hire it.

Operations

PG balancer

Enable the balancer module to keep PGs distributed evenly across OSDs:

ceph mgr module enable balancer
ceph balancer mode upmap
ceph balancer on

Check balance:

ceph osd df   # Look for variance in % columns; target <10% imbalance

Scrub schedule

Ceph scrubs verify data integrity in the background. In a training cluster, uncontrolled scrubs can cause latency spikes:

# Restrict scrubs to off-peak hours
ceph config set osd osd_scrub_begin_hour 2    # Start at 2 AM
ceph config set osd osd_scrub_end_hour 6      # End at 6 AM
ceph config set osd osd_scrub_chunk_max 5     # Limit concurrent scrub IO

Deep scrub (verifies checksums, more IO intensive) — weekly is typical:

ceph config set osd osd_deep_scrub_interval 604800   # 7 days in seconds

Recovery throttling under load

If an OSD fails and recovery starts during an active training job, it can double the IO load on the remaining OSDs:

# Throttle recovery bandwidth to protect client IO
ceph config set osd osd_recovery_max_active 2          # max concurrent recovery operations
ceph config set osd osd_recovery_op_priority 3         # low priority vs. client IO (default 10)
ceph config set osd osd_max_backfills 1                # backfill one PG at a time

# Monitor recovery speed
ceph -s   # Look for "recovery" line in output, GB/h rate

bluestore_min_alloc_size

One of the most impactful BlueStore settings:

# For NVMe with large sequential IO (training data, checkpoints)
ceph config set osd bluestore_min_alloc_size 65536   # 64 KB

# Default is 4096 (4 KB) — good for small random IO, adds RocksDB overhead for large IO
# 64 KB reduces write amplification for > 64 KB IOs (typical for training data)

This setting takes effect on new data written; existing data is not converted. Apply before first data writes (at cluster deployment time).