Ulimits — limits.conf, systemd, PAM, and the layers that bite

Ulimits are the most-misunderstood-piece of Linux production tuning. The reason is layering: there are at least four places a process gets its limits from depending on how it was started, and limits.conf is the least effective of them on a systemd box.

This page tells you which file to edit for which process, and how to verify it actually took.

Why one file isn't enough

A process inherits its RLIMIT_* values from its parent. Where the limits come from depends on the parent:

The trap: kubelet, containerd, slurmd, nvidia-persistenced, and almost everything else operationally important on a GPU node is started by systemd, which does not read limits.conf. Editing only that file gives you values for interactive SSH sessions and nothing else.

What the critical processes need

MEMLOCK=infinity matters specifically because RDMA verbs need to pin DMA-able memory; without it ibv_reg_mr() returns ENOMEM and you get cryptic NCCL/Slurm failures.

The four places to edit

1. PAM / limits.conf (interactive SSH sessions)

# /etc/security/limits.conf
*       soft    nofile      1048576
*       hard    nofile      1048576
*       soft    nproc       unlimited
*       hard    nproc       unlimited
*       soft    memlock     unlimited
*       hard    memlock     unlimited
*       soft    stack       unlimited
*       hard    stack       unlimited
root    soft    nofile      1048576
root    hard    nofile      1048576
root    soft    nproc       unlimited
root    hard    nproc       unlimited
root    soft    memlock     unlimited
root    hard    memlock     unlimited

A drop-in file under /etc/security/limits.d/ works the same way. Last-loaded wins.

Confirm PAM is wired up:

grep -E 'pam_limits|pam_limits.so' /etc/pam.d/sshd /etc/pam.d/login /etc/pam.d/system-auth 2>/dev/null
# /etc/pam.d/sshd:session    required     pam_limits.so

If pam_limits.so is not in the SSH session stack, limits.conf is silently ignored even for SSH. RHEL has this on by default; some hardened Debian/Ubuntu builds have it disabled — check.

2. systemd-wide default (the safety net)

# /etc/systemd/system.conf.d/90-ulimits.conf
[Manager]
DefaultLimitNOFILE=1048576:1048576
DefaultLimitMEMLOCK=infinity
DefaultLimitSTACK=infinity

Then:

systemctl daemon-reexec    # re-exec PID 1 to pick up the new defaults

DefaultLimit* only applies to services that don't override. It's a safety net — newly added services without explicit limits will inherit these.

3. systemd unit drop-ins (the real fix)

For each critical service, drop in a .conf:

# /etc/systemd/system/kubelet.service.d/ulimits.conf
[Service]
LimitNOFILE=1048576
LimitNPROC=infinity
LimitMEMLOCK=infinity
LimitSTACK=infinity

# /etc/systemd/system/containerd.service.d/ulimits.conf
[Service]
LimitNOFILE=1048576
LimitNPROC=infinity
LimitMEMLOCK=infinity
LimitSTACK=infinity

# /etc/systemd/system/slurmd.service.d/ulimits.conf
[Service]
LimitNOFILE=1048576
LimitNPROC=infinity
LimitMEMLOCK=infinity
LimitSTACK=infinity

Apply:

systemctl daemon-reload
systemctl restart kubelet containerd slurmd

4. Container runtime defaults (limits inside pods)

containerd's CRI config has a per-container default ulimit list. The kubelet ulimit set above gives the runtime more room; this controls what containers it spawns inherit.

/etc/containerd/cri-base.json:

{
  "linux": {
    "resources": {
      "ulimits": [
        { "type": "RLIMIT_NOFILE",  "hard": 1048576,             "soft": 1048576 },
        { "type": "RLIMIT_NPROC",   "hard": 4294967295,          "soft": 4294967295 },
        { "type": "RLIMIT_MEMLOCK", "hard": 9223372036854771712, "soft": 9223372036854771712 },
        { "type": "RLIMIT_STACK",   "hard": 9223372036854771712, "soft": 9223372036854771712 }
      ]
    }
  }
}

Restart containerd. New pods inherit these; existing pods do not until they restart.

9223372036854771712 is RLIM_INFINITY ((1ULL << 63) - 1024, near INT64_MAX).

Verifying limits actually took

The single source of truth is /proc/<pid>/limits. Don't trust ulimit -a from a shell — that only shows the shell's own limits.

# kubelet
cat /proc/$(pgrep -x kubelet)/limits | grep -E 'open files|locked memory|stack'
# Max open files            1048576              1048576              files
# Max locked memory         unlimited            unlimited            bytes
# Max stack size            unlimited            unlimited            bytes

# containerd
cat /proc/$(pgrep -x containerd)/limits | grep -E 'open files|locked memory'

# slurmd
cat /proc/$(pgrep -x slurmd)/limits | grep -E 'open files|locked memory'

# Inside a running pod — confirm the pod-side limit
kubectl exec -it <pod> -- sh -c 'ulimit -n; ulimit -l'
# 1048576
# unlimited

# Or for a specific container PID via crictl
crictl exec <container-id> sh -c "ulimit -n; cat /proc/self/limits"

The thing to look for: soft and hard both at the target value, and Max locked memory showing unlimited. If Max locked memory is 65536 (the kernel default), MEMLOCK didn't propagate and RDMA pinning will fail.

The order operations should happen in (greenfield node)

1. Drop /etc/security/limits.conf and /etc/security/limits.d/* files
2. Drop /etc/systemd/system.conf.d/90-ulimits.conf and daemon-reexec
3. Drop unit drop-ins for kubelet, containerd, slurmd, and daemon-reload
4. Update /etc/containerd/cri-base.json (back it up first)
5. Bump fs.file-max and fs.nr_open in /etc/sysctl.d/30-fs.conf and sysctl --system
6. Restart containerd then kubelet then slurmd
7. Verify each PID's /proc/<pid>/limits

Step 5 is non-obvious: fs.file-max is the system-wide ceiling, and fs.nr_open is the per-process ceiling on what a process can request via setrlimit(RLIMIT_NOFILE). If fs.nr_open is lower than your LimitNOFILE, systemd silently caps you at fs.nr_open. Set both to 1048576 or higher.

Why people end up here

The single most common operational failure: a fresh GPU node passes basic NCCL tests, but the moment a tenant runs a real training job with hundreds of DataLoader workers, allocations start failing. Symptoms:

• OSError: [Errno 24] Too many open files
• ibv_reg_mr failed: Cannot allocate memory
• ulimit -n inside the pod shows 1024 (Linux ancient default)

Tracing back: kubelet was started with LimitNOFILE=4096 (some distros' shipped value), so containerd inherited 4096, so containers got 4096. The host's /etc/security/limits.conf had the right values, but they never reached the runtime.

The fix is always the same four places.

Rollback

If a config-management push breaks things:

rm /etc/systemd/system/kubelet.service.d/ulimits.conf
rm /etc/systemd/system/containerd.service.d/ulimits.conf
rm /etc/systemd/system/slurmd.service.d/ulimits.conf
mv /etc/containerd/cri-base.json.bak-YYYYMMDD /etc/containerd/cri-base.json
systemctl daemon-reload
systemctl restart containerd kubelet slurmd

See also

• Sysctl tuning — fs.file-max and fs.nr_open are paired with these
• GRUB cmdline — boot-time vs runtime kernel knobs
• Operations: troubleshooting — ibv_reg_mr failures triage

External:

• man 5 limits.conf, man 5 systemd.exec
• man 2 setrlimit, man 2 prlimit
• containerd cri-base.json reference: containerd.io/docs/cri/config/