enroot + pyxis — containerized Slurm jobs without Docker

Running containerized workloads from Slurm jobs has been an unsolved-feeling problem for years. Docker doesn't fit Slurm's process model — the daemon owns processes, not the user, and Slurm wants to track CPU time, GPU usage, memory, and PIDs as belonging to the job. The combination NVIDIA pushes (and what most HPC sites converge on) is enroot + pyxis.

This page explains what each does, the canonical sbatch flow, and the operational issues you'll hit (image-pull credentials, /etc/enroot/enroot.conf, OCI imports).

Why Docker is a bad fit for Slurm

Singularity/Apptainer addresses some of this but has its own quirks (SIF format, suid mode). NVIDIA's answer is enroot.

enroot: chroot-based, daemonless containers

enroot extracts an OCI/Docker image into a directory tree (a "rootfs"), then unshares into it as the requesting user. There is no daemon, no privileged operation in the steady state, and the container's process tree is just children of the user's shell.

# Pull an OCI image to ~/.local/share/enroot/<image>.sqsh
enroot import docker://nvcr.io/nvidia/pytorch:24.07-py3

# Mount and start it interactively
enroot create --name pyt nvidia+pytorch+24.07-py3.sqsh
enroot start --rw pyt

# Inside: it's a normal shell with the image's libs
nvidia-smi
python -c "import torch; print(torch.cuda.is_available())"

The .sqsh is a SquashFS archive — single file, mountable read-only, very fast to start. Most sites cache them on a shared FS (Weka, Lustre) so every node sees the same image without re-importing.

pyxis: the Slurm SPANK plugin that wires it together

pyxis is a SPANK (Slurm Plug-in Architecture for Node Kontroll) plugin. SPANK plugins extend Slurm's job-launch path with custom logic — pyxis adds container-aware flags to sbatch/srun:

sbatch --container-image=nvcr.io#nvidia/pytorch:24.07-py3 \
       --container-mounts=/scratch:/scratch,/data:/data \
       --container-name=mytrain \
       --container-workdir=/workspace \
       train.sh

When this job runs on a worker, pyxis (loaded by slurmd) intercepts the launch:

1. Resolves the image (downloads via enroot import if not cached)
2. Creates an enroot container scoped to the job
3. Mounts the requested paths
4. unshares into it
5. Hands off to the user's script (train.sh)

The container is the job's leaf process tree, and Slurm tracks everything (CPU/GPU/memory accounting) normally — to Slurm it's just one more user process.

A real sbatch wrapper

#!/bin/bash
#SBATCH --job-name=train-tenant-foo
#SBATCH --partition=gpu
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=8
#SBATCH --gres=gpu:8
#SBATCH --cpus-per-task=24
#SBATCH --time=24:00:00
#SBATCH --output=/scratch/%u/logs/%j.out

# Job script (train.sh)
srun --container-image=registry.tenant-foo.example/trainer:v1.2.3 \
     --container-mounts=/scratch:/scratch,/weka/tenant-foo:/data \
     --container-workdir=/workspace \
     --container-writable \
     bash -c "
       set -euo pipefail
       export NCCL_DEBUG=INFO
       export NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3
       torchrun \
         --nnodes=\${SLURM_NNODES} \
         --nproc-per-node=\${SLURM_NTASKS_PER_NODE} \
         --rdzv-backend=c10d \
         --rdzv-endpoint=\${SLURMD_NODENAME}:29500 \
         /workspace/train.py
     "

Notes:

• --container-writable mounts the rootfs rw; without it, every container is read-only (which is fine for most reproducible-training cases).
• --container-name=foo reuses a previously-imported sqsh by name across job steps in the same allocation. Saves re-import time.
• The image reference uses the registry#path:tag form that enroot/pyxis prefer; it also accepts docker:// URLs.

/etc/enroot/enroot.conf — the operator-side config

This file controls every node-side enroot operation. Critical fields:

# /etc/enroot/enroot.conf

# Where to extract images
ENROOT_RUNTIME_PATH         ${HOME}/.local/share/enroot/${USER}
ENROOT_CACHE_PATH           /shared/enroot/cache
ENROOT_DATA_PATH            /scratch/${USER}/enroot

# Don't extract individual layers — keep .sqsh and mount via squashfuse or
# kernel squashfs. Saves time and disk.
ENROOT_SQUASH_OPTIONS       -comp xz -noD -noI -noF -noX

# Mount squashfs via kernel module (faster than squashfuse). Requires
# squashfs.ko loaded on host.
ENROOT_MOUNT_HOME           y
ENROOT_RESTRICT_DEV         y
ENROOT_ALLOW_SUPERUSER      n

# Image registry credentials (per-user .docker/config.json works too)
# Or set ENROOT_LOGIN_PATH for shared credential dir
ENROOT_LOGIN_PATH           /etc/enroot/credentials/${USER}

ENROOT_CACHE_PATH on a shared filesystem (Weka, NFS) means every node imports an image once across the cluster.

OCI image import

# From a Docker registry
enroot import docker://nvcr.io/nvidia/pytorch:24.07-py3
# Resolves -> downloads -> extracts -> creates nvidia+pytorch+24.07-py3.sqsh

# From a local OCI bundle
enroot import oci-archive://./trainer.tar

# With explicit credentials
echo "machine nvcr.io login \$oauthtoken password <token>" >> ~/.netrc
enroot import docker://nvcr.io/...

# Or via Docker auth
docker login nvcr.io
enroot import docker://nvcr.io/...   # picks up ~/.docker/config.json

Import is the slow step (network + decompression + sqsh build). For a 10 GB image it can take 5-15 min. Pre-importing during quiet hours and caching on Weka avoids per-job startup tax.

Common failure modes

sbatch --container-image=... returns "image pull failed" — credentials. Tenant doesn't have a valid auth for that registry. Either set up ~/.netrc or use ENROOT_LOGIN_PATH to point at a per-user credentials dir managed by the platform.

Job runs but nvidia-smi inside the container shows nothing — pyxis didn't pass through the GPU devices. Check the slurmd config: Gres=gpu:... must be set on the node, and pyxis SPANK must be loaded. slurmd -C shows the parsed gres config.

enroot import succeeds but enroot start fails with "permission denied" — ENROOT_ALLOW_SUPERUSER=n and the container is trying to do something privileged. Switch to ENROOT_RESTRICT_DEV=n if the workload needs it (rare), or fix the image.

Container starts but NCCL multi-node fails with bind: cannot assign requested address — pyxis defaults give the container its own network namespace. NCCL multi-node needs host networking. Add --container-mount-home and --container-readonly=false, plus ensure NCCL_SOCKET_IFNAME is set to a host iface, not a virtual one. Or run pyxis with --no-container-remap-root and host-network mode (pyxis doesn't have a clean flag for this; usually requires not using --container-image and instead launching enroot manually inside the job script).

enroot.conf changes don't take effect — pyxis picks up the conf at slurmd startup, not per-job. Restart slurmd on the affected nodes.

Out of disk on ENROOT_DATA_PATH — every job creates a per-job rootfs scratch. If ENROOT_DATA_PATH lives on the root partition and the host has small /, you fill up fast. Move to a big scratch volume (Weka, local NVMe scratch, large ephemeral PV).

"Permission denied" on enroot start — typically one of:

• ENROOT_ALLOW_SUPERUSER=n and the image's entrypoint expects to run as uid 0 with capabilities. Either fix the image or set ENROOT_REMAP_ROOT=y (apparent-only-root, safer).
• Squashfs file is owned by a different uid; chmod 644 the .sqsh on the shared cache.
• User namespaces disabled in kernel: cat /proc/sys/user/max_user_namespaces. Should be > 0. If 0, add kernel.unprivileged_userns_clone=1 to sysctl.

Pyxis hooks not loaded — srun --container-image=... returns srun: unrecognized option '--container-image'. Steps:

1. ldconfig -p | grep -i nvidia on the slurmd host. If missing the libnvidia-container path, pyxis can't find its dependency.
2. ls /usr/lib/spank/pyxis.so. Missing → reinstall the SUNK slurmd image.
3. cat /etc/slurm/plugstack.conf.d/pyxis.conf. Missing the required line → ConfigMap drift.
4. kubectl rollout restart the NodeSet's slurmd Pods after fixing.

Image pull failed: 401 unauthorized — credentials. Check ~/.docker/config.json exists for the user, or ENROOT_LOGIN_PATH/${USER} is populated. Test outside Slurm: enroot import docker://....

GPU not visible inside container even though --gres=gpu:8 — three possible causes:

1. gres.conf on the node has Files=/dev/nvidia[0-7] but cgroup constraint isn't passing them. Check cat /sys/fs/cgroup/devices/slurm/uid_*/job_*/devices.list on the node.
2. NVIDIA Container Toolkit's CDI not configured. nvidia-ctk cdi list should show H100 entries. CDI specs live at /etc/cdi/nvidia.yaml.
3. pyxis not handed the GPU env: NVIDIA_VISIBLE_DEVICES should be set in the job's environment. Check via env | grep NVIDIA inside an interactive srun.

enroot.conf changes don't take effect — pyxis picks up the conf at slurmd startup, not per-job. Restart slurmd on the affected nodes.

Image lifecycle: import → squashfs cache → run

The full path of an OCI image into a running job:

docker:// or oci-archive://         (network)
        │
        ▼
   enroot import                    (pulls layers, dedups, decompresses)
        │
        ▼
   squashfs build (-comp xz)        (single .sqsh file in $ENROOT_CACHE_PATH)
        │
        ▼
   ┌───────────────────────────┐
   │  Job submitted: pyxis     │
   │  --container-image=...    │
   └────────────┬──────────────┘
                ▼
   enroot create               (instance dir in $ENROOT_DATA_PATH/<jobid>)
                │
                ▼
   enroot start --rw           (unshare into rootfs)
                │
                ▼
   user CMD runs (torchrun, etc.)
                │
                ▼
   exit → cleanup epilog       (rm -rf instance dir; cache .sqsh kept)

Key insight: the .sqsh cache file is shared, the runtime instance directory is per-job and discardable. If your runtime path is on local NVMe (fast) and your cache is on Weka (shared), you get instant warm starts and zero re-import per node.

Pyxis flags worth knowing

Pyxis takes these as either CLI flags to srun/sbatch or as #SBATCH directives.

/etc/enroot/enroot.conf — full reference

The fields that matter most in production:

# /etc/enroot/enroot.conf

# --- Path layout ---
ENROOT_LIBRARY_PATH       /usr/lib/enroot
ENROOT_RUNTIME_PATH       /var/run/enroot/user-${UID}
ENROOT_CONFIG_PATH        ${HOME}/.config/enroot
ENROOT_CACHE_PATH         /shared/enroot/cache         # ← put on Weka/NFS
ENROOT_DATA_PATH          /scratch/enroot/${UID}       # ← per-user scratch
ENROOT_TEMP_PATH          /tmp

# --- Image format ---
ENROOT_SQUASH_OPTIONS     -comp lz4 -noD -noI -noF -noX
ENROOT_GZIP_PROGRAM       pigz                          # parallel gzip

# --- Mount + namespace ---
ENROOT_MOUNT_HOME         y                             # bind $HOME
ENROOT_RESTRICT_DEV       y                             # hide /dev/* not allow-listed
ENROOT_ALLOW_SUPERUSER    n                             # disable real-uid-0
ENROOT_REMAP_ROOT         y                             # appear as root inside

# --- Auth ---
ENROOT_LOGIN_PATH         /etc/enroot/credentials/${USER}
                                                         # used to find ~/.docker/config.json equivalents

Critical pairings:

• ENROOT_CACHE_PATH on a shared FS = every node imports an image once across the cluster. Without this, every node re-imports — for a 12 GB image on 64 nodes, that's 768 GB of pull traffic.
• ENROOT_DATA_PATH on local NVMe scratch = job rootfs unpack stays off the shared FS hot path. If you put it on Weka, every job's enroot create thrashes the same volume.
• ENROOT_RESTRICT_DEV=y + ConstrainDevices=yes in cgroup.conf = container only sees the GPUs Slurm allocated. Without it, a user with --gres=gpu:1 can nvidia-smi 8 GPUs.

ENROOT_SQUASH_OPTIONS=-comp lz4 builds faster but produces larger .sqsh than -comp xz. For sites with fast import / slow boot, lz4 wins. Re-evaluate on your network.

Pyxis hooks installation

Pyxis registers itself with Slurm via SPANK. Verify it's loaded:

# On a slurmd node
ls -la /usr/lib/spank/
# -rwxr-xr-x ... pyxis.so

# Check plugstack.conf points at it
cat /etc/slurm/plugstack.conf
# include /etc/slurm/plugstack.conf.d/*.conf

cat /etc/slurm/plugstack.conf.d/pyxis.conf
# required /usr/lib/spank/pyxis.so

# Verify slurmd loaded it
journalctl -u slurmd | grep -i pyxis
# pyxis: version 0.20.0 starting

If pyxis isn't loaded, srun --container-image=... will fail with "unrecognized option" because the SPANK flag wasn't registered.

Image-pull credentials

Two patterns in production:

Per-user ~/.docker/config.json — works out of the box if ENROOT_LOGIN_PATH is unset; enroot reads $HOME/.docker/config.json. Tenants run docker login once on the login pod (the docker CLI doesn't have to work — it just writes the config), then enroot import works.

Site-wide credential dir — set ENROOT_LOGIN_PATH=/etc/enroot/credentials/${USER} and pre-populate per-user credentials managed by the platform team. Useful when researchers shouldn't need to know the registry password.

For NVIDIA NGC specifically:

# In login pod
docker login nvcr.io
# Username: $oauthtoken
# Password: <your NGC API key>

# Or via netrc
cat >> ~/.netrc <<EOF
machine nvcr.io
login \$oauthtoken
password <NGC_API_KEY>
EOF
chmod 600 ~/.netrc

PYXIS_DOCKERFILE_AUTH=1 (set in slurmd env) makes pyxis prefer Docker-style auth chains. Useful for compatibility with multi-registry workflows.

Cache management

Images accumulate. Periodically:

# List
ls -lah /shared/enroot/cache/
# nvidia+pytorch+24.07-py3.sqsh    9.8G
# tenant-foo+trainer+v1.2.3.sqsh   12G
# ...

# Clean ones older than 30 days
find /shared/enroot/cache/ -name "*.sqsh" -mtime +30 -delete

Build this into a CronJob on K8s or a systemd timer on the storage host.

Diagnostic toolkit

# Show the version + build flags
enroot --version

# What the cache + runtime config look like at runtime
enroot list -f                            # all containers (created instances)
ls -lh $(enroot config | grep CACHE | awk '{print $2}')

# Verify a sqsh
unsquashfs -s /shared/enroot/cache/nvidia+pytorch+24.07-py3.sqsh
unsquashfs -ll /shared/enroot/cache/nvidia+pytorch+24.07-py3.sqsh | head

# Inspect the runtime instance dir of a stuck job
ls -la /scratch/enroot/${UID}/<jobid>/
cat /scratch/enroot/${UID}/<jobid>/config

# enroot-aux: helper logs (if the wrapper splits stdout)
journalctl -u slurmd | grep enroot-aux

# pyxis-side log (verbose)
srun --container-image=... -v ...         # add -v for SPANK debug

A useful trick: run srun -N1 --container-image=foo --pty bash to get a shell inside the container in interactive mode. Lets you confirm GPU visibility, mounts, and entrypoint behavior without launching a full job.

See also

• SUNK intro — how Slurm runs as K8s pods
• Login pods — where users submit jobs from
• Job failures — what to look at when a containerized job fails
• SUNK troubleshooting — when slurmd / pyxis won't load
• GPU Operator — provides NVIDIA Container Toolkit/CDI for the host
• Weka overview — typical home for the enroot cache and data

External:

• github.com/NVIDIA/enroot
• github.com/NVIDIA/pyxis
• enroot.conf(5)