Login pods — tenant SSH entry, SOCKS proxies, and PAM/LDAP

Login pods are the tenant's only entry point to a SUNK Slurm cluster. They run an SSH server, have the Slurm clients (sbatch, srun, squeue, sacct), and expose a SOCKS proxy so tenants can ssh further into running compute pods if needed. They are the canonical "where does the customer log in?" answer.

This page covers the design, the PAM/LDAP wiring, the SOCKS proxy pattern, and the onboarding flow that ties an SSH key + 1Password + Authentik group together.

Why a separate login pod (and not direct cluster access)

The alternative — giving tenants kubectl against your cluster — fails on three axes:

1. Multi-tenancy. kubectl is RBAC, but namespace-level isolation is hard to harden against. SSH is process-level isolation, much simpler.
2. HPC user expectation. The user knows sbatch, not kubectl exec. Force-feeding K8s to a researcher fails.
3. Provisioning surface. kubectl requires a kubeconfig per user, certificates that rotate, and a webhook for SSO. SSH-with-key is solved.

A login pod gives the tenant exactly one IP and port (ssh tenant-user@login.tenant.example.internal), and inside that pod they have Slurm + a shell and nothing else.

What a login pod runs

A typical login pod manifest:

apiVersion: v1
kind: Pod
metadata:
  name: login-0
  namespace: tenant-foo
spec:
  containers:
    - name: login
      image: ghcr.io/coreweave/sunk-login:23.11.7
      ports:
        - { name: ssh, containerPort: 22 }
      volumeMounts:
        - { name: home,        mountPath: /home }
        - { name: scratch,     mountPath: /scratch }
        - { name: slurm-conf,  mountPath: /etc/slurm }
        - { name: munge-key,   mountPath: /etc/munge }
        - { name: ldap-conf,   mountPath: /etc/openldap }
        - { name: authorized-keys, mountPath: /etc/ssh/authorized_keys.d }
      resources:
        requests: { cpu: 1, memory: 2Gi }
  volumes:
    - { name: home,    persistentVolumeClaim: { claimName: weka-home } }
    - { name: scratch, persistentVolumeClaim: { claimName: weka-scratch } }
    - { name: slurm-conf,    configMap: { name: slurm-conf } }
    - { name: munge-key,     secret:    { secretName: munge-key, defaultMode: 0o400 } }
    - { name: ldap-conf,     configMap: { name: ldap-config } }
    - { name: authorized-keys, secret:  { secretName: tenant-foo-authorized-keys } }

Exposed via a Service of type LoadBalancer (or NodePort + an external reverse proxy):

apiVersion: v1
kind: Service
metadata:
  name: login
  namespace: tenant-foo
spec:
  type: LoadBalancer
  selector: { app.kubernetes.io/name: sunk-login }
  ports: [{ port: 22, targetPort: 22, name: ssh }]

Tenants get the LB IP / DNS and ssh tenant-user@login.tenant-foo.example.internal.

PAM + LDAP for shared user auth

The login pod's sshd is configured to use PAM, and PAM is configured to consult LDAP (the Authentik LDAP outpost — see authentik).

/etc/nsswitch.conf inside the pod:

passwd:         files ldap
group:          files ldap
shadow:         files ldap

/etc/pam.d/sshd:

auth       required     pam_sepermit.so
auth       substack     password-auth
account    required     pam_nologin.so
account    include      password-auth
session    required     pam_selinux.so close
session    required     pam_loginuid.so
session    required     pam_selinux.so open env_params
session    optional     pam_keyinit.so force revoke
session    include      password-auth

/etc/pam.d/password-auth:

auth        required      pam_env.so
auth        sufficient    pam_unix.so try_first_pass nullok
auth        sufficient    pam_ldap.so use_first_pass
auth        required      pam_deny.so

account     required      pam_unix.so broken_shadow
account     sufficient    pam_localuser.so
account     sufficient    pam_succeed_if.so uid < 1000 quiet
account     [default=bad success=ok user_unknown=ignore] pam_ldap.so
account     required      pam_permit.so

session     required      pam_limits.so
session     sufficient    pam_unix.so
session     optional      pam_ldap.so

The relevant points:

• SSH key auth doesn't actually need PAM password modules to succeed — sshd checks authorized_keys first. PAM is consulted for the account and session stages.
• pam_limits.so is what reads /etc/security/limits.conf — see ulimits for what those values should be.
• nslcd or sssd runs as a sidecar/daemon in the login pod; without it, getent passwd alice returns nothing and sshd denies the user even if the key matches.

getent passwd alice should resolve to something like:

alice:x:5042:5042:Alice Researcher:/home/alice:/bin/bash

UID 5042 and GID 5042 come from Authentik (or whatever directory). Both must be consistent across login pods, slurmd pods, and the shared FS — otherwise file ownership gets confused. See authentik for the gidNumber convention.

SSH key authentication

authorized_keys for each user is mounted in via a Secret:

apiVersion: v1
kind: Secret
metadata:
  name: tenant-foo-authorized-keys
type: Opaque
stringData:
  alice: |
    ssh-ed25519 AAAA...alice's key... alice@laptop
  bob: |
    ssh-rsa AAAA...bob's key... bob@workstation

sshd_config reads them via AuthorizedKeysFile:

AuthorizedKeysFile  /etc/ssh/authorized_keys.d/%u
PasswordAuthentication no
PubkeyAuthentication yes
ChallengeResponseAuthentication no
PermitRootLogin no
AllowGroups tenant-foo-users

The mounted path is /etc/ssh/authorized_keys.d/alice. Because the Secret keys map to filenames, you get one file per user. Synced from Authentik via an operator that watches the IdP and renders this Secret.

SOCKS proxy pattern for inner pod access

Login pods are not compute. Tenants who want to connect a Jupyter kernel, debug a running training pod, or rsync data from a running worker need network reach into the cluster. SOCKS is the simplest way:

# On the laptop, open a SOCKS5 proxy on localhost:1080 over the SSH tunnel
ssh -D 1080 -N tenant-user@login.tenant-foo.example.internal &

# Then point a tool at it
curl --socks5 localhost:1080 http://gpu-01.tenant-foo.svc.cluster.local:8888/

The tenant's traffic exits the SSH tunnel inside the login pod's network namespace, which sees cluster DNS and pod CIDRs. From there it can reach any service the tenant's namespace exposes.

For browser use:

# Firefox/Chrome with SOCKS5 manual proxy: localhost:1080
# Make sure DNS-over-SOCKS is enabled (otherwise DNS leaks happen)
# In Firefox: about:config -> network.proxy.socks_remote_dns = true

This avoids exposing every internal service via Ingress / LoadBalancer.

Customer onboarding flow

The end-to-end flow when a new tenant user joins:

1. SRE creates user in Authentik UI (or via API)
   - sets username, email, full name
   - assigns to tenant-foo group
   - Authentik computes UID and gidNumber per convention (e.g., gidNumber = pk + 4000)

2. SRE asks the user for their SSH public key
   - they paste it into a 1Password item, or upload directly to Authentik

3. Authentik or sync operator:
   - reads keys from the user object
   - renders to per-cluster authorized_keys Secret
   - kubectl apply syncs it to the tenant namespace

4. Login pod's projected secret refreshes (kubelet does this within ~60s)

5. User can ssh in:
   ssh -i ~/.ssh/their-key tenant-user@login.tenant-foo.example.internal

6. SRE confirms by running:
   getent passwd tenant-user        # resolves
   id tenant-user                   # uid/gid match
   sudo -u tenant-user sbatch --test-only ...   # Slurm sees them

For SSH keys: the user's personal key, never an SRE-shared one. Tenant users own their SSH key; SREs do not paste their personal keys into tenant 1Password vaults.

SSH server hardening

The login pod's sshd_config is the front door. The minimum-acceptable hardened version:

# /etc/ssh/sshd_config

# --- protocol + crypto ---
Protocol 2
HostKey /etc/ssh/host_keys/ssh_host_ed25519_key
HostKey /etc/ssh/host_keys/ssh_host_rsa_key
KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com
HostKeyAlgorithms ssh-ed25519,rsa-sha2-512,rsa-sha2-256

# --- auth ---
PasswordAuthentication no
ChallengeResponseAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
PermitRootLogin no
PermitEmptyPasswords no
MaxAuthTries 3
MaxSessions 4
LoginGraceTime 30
AllowUsers tenant-foo-* admin
AllowGroups tenant-foo-users platform-admins

# --- session ---
ClientAliveInterval 60
ClientAliveCountMax 3
TCPKeepAlive yes
X11Forwarding no
AllowTcpForwarding yes        # SOCKS proxy needs this
GatewayPorts no
PermitTunnel no
AllowAgentForwarding no
PrintMotd no                  # PAM will print the motd

# --- forensic ---
LogLevel VERBOSE              # records key fingerprints
SyslogFacility AUTHPRIV

AuthorizedKeysFile /etc/ssh/authorized_keys.d/%u

Notes:

• PubkeyAuthentication yes + PasswordAuthentication no is the only acceptable mode. Always disable passwords.
• AllowUsers / AllowGroups is critical for shared LDAP — a bug in pam_ldap could let an unintended user resolve and authenticate. Hard-list at the sshd layer.
• MaxAuthTries 3 slows brute-force without breaking key+passphrase auth.
• LogLevel VERBOSE makes sshd log the fingerprint of the accepted key, so audit logs can tie a session to a specific key (not just a user).
• AllowTcpForwarding yes is required for the SOCKS proxy pattern below. If you don't allow that, set it to local (allows -L only) or remote (-R only) as appropriate.

End-user SSH config for SOCKS proxies

For tenants who want a one-line SSH config:

# ~/.ssh/config (on the tenant's laptop)

Host login-tenant-foo
    HostName login.tenant-foo.example.internal
    User alice
    Port 22
    IdentityFile ~/.ssh/tenant-foo-key
    ServerAliveInterval 30
    ServerAliveCountMax 3

Host *.tenant-foo.svc.cluster.local
    ProxyJump login-tenant-foo
    User alice
    IdentityFile ~/.ssh/tenant-foo-key

The first stanza is the login pod itself. The second uses ProxyJump so ssh gpu-01.tenant-foo.svc.cluster.local transparently jumps through the login pod. Useful for rsync, scp, and Jupyter port-forwards.

For a SOCKS proxy in the background:

# Start a backgrounded SOCKS proxy on localhost:1080
ssh -fN -D 127.0.0.1:1080 login-tenant-foo

# Browser: SOCKS5 proxy = 127.0.0.1:1080, "Proxy DNS" = on
# Or with curl:
curl --socks5-hostname 127.0.0.1:1080 http://gpu-01.tenant-foo.svc.cluster.local:8888/

--socks5-hostname (not --socks5) sends DNS through the tunnel, which is required because gpu-01.tenant-foo.svc.cluster.local doesn't resolve outside the cluster.

PAM stack details

The PAM stack on a login pod is layered. From outermost to innermost:

pam_slurm_adopt is the hidden-but-critical one for HPC clusters. When enabled, an SSH session to a compute node (not the login pod itself) is "adopted" into the cgroup of the user's currently-running Slurm job on that node. Two consequences:

1. The interactive shell is bounded by the job's cgroup limits — the user can't accidentally htop their way past their --mem= allocation.
2. When the job ends, the SSH session is killed. No leaked debug shells holding GPUs.

Configuration in /etc/pam.d/sshd on a slurmd-running node:

account    required     pam_slurm_adopt.so

Combined with pam_access.so to deny SSH to nodes where the user has no running job:

account    required     pam_access.so accessfile=/etc/security/access.conf
account    required     pam_slurm_adopt.so action_no_jobs=deny action_unknown=deny action_adopt_failure=deny

Effect: a user who ssh's to gpu-05 while they have no job there gets refused. While they have a job there, they get adopted into its cgroup and limits.

This is the canonical "interactive debug" pattern: srun --pty bash to get a shell on the allocated node, or a separate ssh in to attach to py-spy / nvidia-smi against the running training process — both end up in the same cgroup.

Common failure modes

User can SSH but Slurm says "Invalid account":

• Account not added to slurmdbd. sacctmgr add user alice Account=tenant-foo.

getent passwd alice returns nothing:

• nslcd/sssd not running or LDAP unreachable.
• LDAP base DN wrong in nslcd.conf.
• Authentik LDAP outpost down. kubectl -n auth logs -l app=ldap-outpost.

SSH succeeds but user has no $HOME:

• Weka/NFS PVC not mounted in the login pod.
• /home/alice doesn't exist; mkdir it (or have nslcd's pam_mkhomedir do it on first login):

  session    optional     pam_mkhomedir.so skel=/etc/skel umask=0077
  

Files written by alice on login show wrong owner on slurmd nodes:

• gidNumber differs between login pod's LDAP view and slurmd's view. Both must consult the same LDAP. Check id alice on both.

SOCKS works for HTTP but DNS leaks:

• Browser/tool using local DNS instead of SOCKS-tunneled DNS. Enable "remote DNS" in the SOCKS settings.

SSH connection drops after a few minutes idle:

• ServerAliveInterval / ClientAliveInterval not set. Add to sshd_config:

  ClientAliveInterval 60
  ClientAliveCountMax 3
  

NSS lookup hangs for 30+ seconds, then getent passwd returns nothing:

• LDAP outpost is down or unreachable. Login pod's nslcd/sssd retries with backoff before giving up.
• Test from inside the pod:

  ldapsearch -H ldap://authentik-ldap.auth.svc.cluster.local:389 \
    -D "cn=ldap-bind,ou=service,dc=tenant-foo,dc=internal" \
    -w "$BIND_PW" \
    -b "dc=tenant-foo,dc=internal" "(uid=alice)"
  

• If it times out: kubectl -n auth get pods -l app=ldap-outpost and check the outpost is healthy.
• Workaround until LDAP recovers: stop nslcd (kill -STOP $(pidof nslcd)) so NSS lookups fail-fast against files only and root can still SSH in.

Home dir mount failed (Weka stale mount):

• Login pod boots fine but /home/alice is empty or read-only.
• This is the stale wekafs mount problem: the login pod was scheduled on a node where wekafs is in a transitional state.
• mount | grep weka inside the pod, then df will show Transport endpoint not connected or similar.
• Fix: kill the login pod, let kubelet remount on Pod restart. If wekafs on the host is genuinely broken, drain the node first.

/etc/security/limits.conf not applied for non-root sessions:

• pam_limits.so not in the session stack of /etc/pam.d/sshd (or it's commented). NCCL workloads need at least nofile=1048576, memlock=unlimited. Without pam_limits, sessions inherit container defaults (often nofile=1024).
• Verify post-login: ulimit -n should be 1048576+, ulimit -l should be unlimited.
• Fix: ensure session required pam_limits.so is in /etc/pam.d/sshd and /etc/security/limits.conf has the right values, and /etc/security/limits.d/ has no overriding files.

pam_slurm_adopt denies SSH to a compute node even when the user has a job there:

• Slurm cgroup not yet created (race after srun start).
• The user's job is on a different compute node — pam_slurm_adopt only adopts on the local node.
• slurmd not running on the target → no cgroup to adopt to. Check systemctl status slurmd on that node.
• Verify with pam_slurm_adopt's debug log: add log_level=debug5 to the PAM line and tail /var/log/secure.

Multiple login pods (HA)

For tenants with many users, run two or three login pods behind one LB. SSH host keys must be the same across pods (otherwise users see "host key changed!" warnings) — mount the same Secret containing ssh_host_*_key files into every pod.

volumes:
  - name: ssh-host-keys
    secret:
      secretName: tenant-foo-ssh-host-keys
      defaultMode: 0o400

Generate the host keys once at tenant onboarding, store in the Secret, never rotate without warning users.

See also

• SUNK intro — full SUNK architecture
• Multi-tenant Slurm — onboarding flow, account hierarchy
• SUNK troubleshooting — login pod inside K8s, not standalone
• Authentik — LDAP outpost + gidNumber convention
• Reservations — login pod and slurmd both run inside the tenant's reservation

External:

• sshd_config(5), pam(8), nslcd.conf(5)
• pam_slurm_adopt documentation