Weka day-2 operations: clients, orgs, snapshots, S3, capacity

The Weka steady-state runbook. Once a cluster is healthy and serving production traffic, day-2 work is mostly: bringing new clients online, retiring old ones, managing tenant boundaries, scheduling snapshots, watching capacity. Each section is a self-contained recipe with the commands you actually run.

Adding a new client

Three things have to happen on the new host: kernel modules, mount-helper, and cluster join.

# 1. Pre-flight: kernel matches an available Weka build, RDMA modules loaded
$ uname -r
$ lsmod | grep -E 'mlx5_core|rdma_ucm|ib_uverbs'

# 2. Install (downloads from a backend MGMT node)
$ curl -s http://<mgmt-ip>:14000/dist/v1/install \
    | sudo sh -s -- --no-start

# 3. Join the cluster (org credentials required)
$ weka cluster join <mgmt-ip>:14000 \
    --auth-token-file /etc/wekaio/auth-token

# 4. Start the client container
$ sudo systemctl enable --now weka-agent
$ weka local start

# 5. Verify
$ weka local status
$ weka cluster status

# 6. Mount the filesystem(s) you need
$ sudo mkdir -p /wekafs/datasets
$ sudo mount -t wekafs cluster-foo/datasets /wekafs/datasets

# 7. Persist via /etc/fstab so it survives reboot
$ echo "cluster-foo/datasets /wekafs/datasets wekafs net=udp,num_cores=2 0 0" \
    | sudo tee -a /etc/fstab

In Kubernetes-managed deployments, this is all wrapped: deploy a WekaClient CR (or just label the node weka.io/supports-clients=true if a single template covers all clients) and the operator's DaemonSet handles the install, join, and mount.

Removing a client cleanly

Order matters. Skipping the umount step is what causes the stale mount problem.

# 1. Drain workload off the node (k8s)
$ kubectl cordon <node>
$ kubectl drain <node> --ignore-daemonsets --delete-emptydir-data

# 2. Unmount EVERY wekafs mount on the host
$ for m in $(grep wekafs /proc/mounts | awk '{print $2}'); do
    sudo umount "$m"
  done

# 3. Confirm wekafsio refcount is 0
$ lsmod | grep wekafsio
wekafsio              655360  0       # ready

# 4. Forget the cluster registration
$ weka cluster forget cluster-foo

# 5. Stop the local client
$ weka local stop
$ sudo systemctl disable --now weka-agent

# 6. Uninstall the userspace
$ sudo /opt/weka/bin/uninstall.sh

# 7. Remove kernel modules
$ sudo rmmod wekafs wekafsio

# 8. Audit / clear state
$ sudo rm -rf /etc/wekaio /opt/weka /var/log/weka

Now the node is clean. If you skip the umount in step 2, you'll discover the stale mount weeks later when someone tries to upgrade and rmmod fails.

Upgrading client version

Client and backend versions don't have to be identical, but they have to be in a supported compatibility window — typically client can be one minor version older or newer than the backend.

# 1. Check current versions
$ weka version
client: 4.3.2
cluster: 4.3.4

# 2. Verify the target version is in the support matrix (Weka docs / KB)

# 3. On each client, stage the upgrade BUT do not yet bounce
$ sudo /opt/weka/bin/upgrade.sh --to 4.3.4 --no-restart

# 4. Pick a maintenance window. For each client:
$ sudo umount -a -t wekafs           # umount all wekafs
$ lsmod | grep wekafsio              # confirm refcount 0
$ weka local stop
$ sudo /opt/weka/bin/upgrade.sh --finalize
$ weka local start
$ sudo mount -a                      # re-mount via fstab
$ weka version                       # confirm new version

If umount fails in step 4, stop. There's a process holding files open or a leftover mount from a previous registration. Resolve before continuing — see troubleshooting. Forcing the upgrade past a stuck wekafsio puts the node in an unbootable-into-Weka state.

Org management

Orgs are Weka's tenancy primitive. Each org has its own filesystems, users, quotas, and credentials. Use them when you need actual data isolation between teams or customers; skip them when RBAC at a higher layer is enough.

Creating an org

# Run on a backend or an admin client
$ weka org create tenant-bar
$ weka org tenant-bar user create admin-bar --role admin
Password for admin-bar: ********

# Bind a filesystem to the org
$ weka fs create tenant-bar-fs --org tenant-bar --capacity 10TB

# Hand off credentials
$ weka org tenant-bar token create --description "tenant-bar service account"

The token is one-shot — copy it now or rotate. Orgs cannot see each other's filesystems; even an org admin can't list other orgs.

Sharing data across orgs

By default, no. If you have a dataset every team needs, you have two options:

1. Make it a shared filesystem in a "common" org that every tenant has read-only credentials to. Manageable for one or two shared datasets.
2. Single-org cluster with namespace/RBAC isolation at the consumer layer (K8s, Slurm). Simpler operationally; weaker tenancy guarantees.

For typical AI labs with cooperating teams, option 2 is fine. For multi-customer hosting, option 1 (or a per-customer cluster) is required.

Snapshot management

Manual one-off

# Snapshot a directory inside a filesystem
$ weka fs snapshot create cluster-foo/datasets pre-cleanup-2026-05-04 \
    --description "before deleting old runs"

# List snapshots on a filesystem
$ weka fs snapshot --filesystem cluster-foo/datasets

# Restore — typically: clone the snapshot to a new directory, validate, swap
$ weka fs snapshot clone pre-cleanup-2026-05-04 /datasets/restored-copy

# Delete a snapshot
$ weka fs snapshot delete cluster-foo/datasets pre-cleanup-2026-05-04

Snapshots are copy-on-write: instant create, near-zero space until the source diverges.

Scheduled

# Create a snapshot policy
$ weka fs snapshot policy create daily-backup \
    --schedule "0 2 * * *" \
    --filesystem cluster-foo/datasets \
    --retention 14d \
    --name-template "auto-{{date}}"

# View policies
$ weka fs snapshot policy

# Detach
$ weka fs snapshot policy delete daily-backup

Daily snapshots with 14-day retention is a sensible default for production data. For datasets that change frequently and have an external golden source (e.g., re-downloadable), snapshots may be unnecessary.

In Kubernetes, prefer the standard VolumeSnapshot API — see CSI. The K8s-native API and the CLI snapshots both end up in the same backend store; you can list/manage either way.

S3 integration: lifecycle to cold tier

Weka can transparently tier directories to S3-compatible object storage. Configure once per filesystem.

# 1. Register the S3 bucket as a tier target
$ weka cloud s3 register cold-tier \
    --endpoint https://s3.example.com \
    --bucket weka-cold-cluster-foo \
    --access-key AKIA... \
    --secret-key ... \
    --region us-east-1

# 2. Attach the tier to a filesystem
$ weka fs tier attach cluster-foo/datasets --target cold-tier

# 3. Set lifecycle: tier files not accessed in 30 days
$ weka fs tier policy create cluster-foo/datasets \
    --rule "atime > 30d" \
    --action tier

Lifecycle rules can combine path, size, and age:

Tiered files are still in the namespace — ls shows them, stat works, but read() pulls from S3 the first time. Plan for the latency increase when designing workloads against tiered data.

To force-rehydrate a directory back to SSD:

$ weka fs tier rehydrate cluster-foo/datasets/needed-now

License management

Weka licenses are capacity-based and signed by Weka. Without a valid license, the cluster degrades — read-only mode, then IO stop.

# Current license state
$ weka cluster license

# Apply a new license
$ weka cluster license set --file /tmp/license.json

# Capacity used vs licensed
$ weka cluster status | grep -i license

Renewals are usually annual. Set a calendar reminder 30 days before expiry — losing license mid-incident is the kind of compounding failure that ruins a weekend.

Capacity planning: what to monitor

Three signals matter:

# 1. Raw capacity
$ weka cluster status | grep -i capacity
Used:      210.4 TB / 350.0 TB (60%)
Free:      139.6 TB

# 2. Per-filesystem usage
$ weka fs
NAME                 CAPACITY  USED      USED%
cluster-foo-shared   100.0 TB  78.2 TB   78%
cluster-foo-cold     200.0 TB  120.0 TB  60%
tenant-bar-fs        50.0 TB    8.4 TB   17%

# 3. Per-org quotas
$ weka cluster org quota
ORG          USED      QUOTA      USED%
default      78.2 TB   100.0 TB   78%
tenant-bar   8.4 TB    50.0 TB    17%

# 4. Drive-level health
$ weka cluster drive
ID    HOST     STATUS  CAPACITY  USED%   FAILURES
0     stor-01  ACTIVE  3.5 TB    62%     0
1     stor-01  ACTIVE  3.5 TB    61%     0
...

Triggers for action

Expanding capacity

Two paths:

1. Add drives to existing backends — minimal disruption, rebalance kicks in automatically (slow, throttled), capacity available within hours.
2. Add new backend hosts — more substantial. New hosts join, take their share of placement groups, rebalance happens.

Both are online operations but rebalance throttles client IO modestly during the move. Plan for a window of 5–15% throughput overhead on clients while rebalance runs, possibly for hours.

Day-2 monitoring: what alerts to wire up

Recommended alert set:

Wire those into Prometheus via the Weka exporter, or scrape the REST API directly.

See also

• Architecture — what's actually being managed
• Clients — install/remove/upgrade detail
• CSI — Kubernetes-flavored versions of these ops
• Performance tuning — what changes after capacity expansion
• Troubleshooting — when day-2 ops go sideways

Troubleshooting

The most common day-2 mistake is removing a client without unmounting first, leading to stale mounts that block future driver upgrades. The umount-first rule applies to every client retire, every node migration, and every tenant decommission.