4. Identity and access management for multi-team HPC

On a shared automotive CAE cluster, the hardest security question is rarely "can we authenticate users?" It is "can we prove that the right engineer reached the right solver, on the right data, for the right project, and nothing else?" A crash-simulation team, a CFD team, and an external supplier may all submit jobs to the same Slurm controller, yet their entitlements must never bleed into one another. Identity and access management (IAM) is the control plane that makes this separation enforceable rather than aspirational.

Directory services as the identity source of truth

Most HPC sites anchor identity in an LDAP directory, typically FreeIPA or Active Directory Domain Services. The directory stores the authoritative user object: UID, primary GID, supplementary groups, SSH public keys, and account expiry. Compute nodes should never hold local accounts for real users; instead they resolve identities through SSSD, which caches credentials and group memberships so that a node losing contact with the directory can still authenticate an already-known user.

A practical FreeIPA enrollment on a compute node looks like this:

ipa-client-install --domain=cae.example.com \
  --server=ipa01.cae.example.com \
  --mkhomedir --enable-dns-updates
systemctl enable --now sssd
getent passwd jdoe
id jdoe   # verify UID, GID and supplementary groups resolve

The id output is the first verification checkpoint. If supplementary groups are missing, RBAC and filesystem permissions will silently fail later, so confirm group resolution before trusting any downstream policy.

Federated identity for cross-site and supplier access

Automotive programs frequently span OEM sites, tier-one suppliers, and cloud burst capacity. Replicating accounts across every domain does not scale. Federated identity solves this by letting a home identity provider (IdP) assert a user's identity to a service provider via SAML 2.0 or OpenID Connect. In practice, a supplier engineer authenticates against their own corporate IdP, and the HPC portal accepts the signed assertion, mapping it to a local project role.

Two rules keep federation safe. First, never auto-provision cluster accounts from an assertion alone; require an explicit project entitlement record. Second, bind the federated session to a short-lived credential rather than a permanent SSH key. A typical mapping table might look like:

RBAC and project-based entitlements

Role-based access control assigns permissions to roles, and roles to users, so that policy is expressed once and reused. On an HPC platform the useful roles are coarse: submitter, project-admin, data-steward, and platform-operator. Project-based entitlements add the finer dimension: a user may be a submitter globally but only entitled to the crash and nvh projects.

Enforcement happens at three layers that must agree. The scheduler enforces queue and account limits through Slurm associations; the filesystem enforces data boundaries through POSIX groups or ACLs; the portal enforces what a user can see and request. A mismatch between layers is the most common source of both over-permission and mysterious "permission denied" failures.

# Slurm association: user jdoe may use account crash, partition main
sacctmgr add user jdoe account=crash partition=main \
  defaultaccount=crash
sacctmgr show assoc user=jdoe format=Account,Partition,GrpTRES%20

# Filesystem boundary for project data
chgrp -R crash /scratch/projects/crash
chmod 2770 /scratch/projects/crash   # setgid keeps group inheritance

The setgid bit on the project directory is deliberate: new files inherit the crash group, so a user who belongs to two projects cannot accidentally write crash data into an NVH-readable location.

Privileged access management

Administrators and support engineers need elevated rights, but standing root on a login node is unacceptable. Privileged access management (PAM in the operational sense) means just-in-time elevation, full session recording, and no shared passwords. Tools such as sudo with centralized policy, or a bastion with session audit, provide this. A minimal sudoers policy grants a named operator the ability to restart a specific service without a shell:

# /etc/sudoers.d/hpc-ops
Cmnd_Alias SLURM_CTL = /usr/bin/systemctl restart slurmd, \
                       /usr/bin/systemctl status slurmd
hpcops ALL=(root) NOPASSWD: SLURM_CTL
Defaults:hpcops log_output, iolog_dir="/var/log/sudo-io/%u"

Session I/O logging turns every privileged action into an auditable artifact, which matters when a supplier disputes who modified a shared dataset.

How the layers connect

The diagram below shows how an identity flows from the directory and federated IdP through role and entitlement evaluation to the three enforcement points. The key insight is that authentication and authorization are separate stages: a valid identity still passes through entitlement checks before any resource is touched.

Verification and common failure symptoms

After any IAM change, verify from the user's perspective rather than the admin's. Log in as a test account and confirm four things: id shows the expected groups, sacctmgr show assoc lists the correct account, a job submitted to an unauthorized partition is rejected, and a file in another project returns permission denied. If a user can submit to a queue they should not reach, the Slurm association is too broad. If they can read another project's data, the setgid or ACL layer is wrong. If privileged actions leave no trace, session logging is not enabled.