2. HPC environment architecture for crash simulation

Crash simulation is one of the most demanding workloads in CAE. A single full-vehicle frontal impact model can contain 8–15 million shell and solid elements, run for 120–150 ms of physical time, and require hundreds of CPU cores to complete overnight. The HPC environment that hosts these jobs is not a generic compute cluster; it is a tightly coupled system of interconnect topology, job scheduler policy, and shared storage performance. Getting any one of these layers wrong turns a 6-hour solve into a 30-hour one — or into a failed job at hour 12. This section describes the architectural layers you must understand before you can reproduce a crash simulation reliably.

Cluster Topology and Interconnect

A typical crash-simulation cluster is built from three node classes. Login/head nodes handle job submission, file staging, and license checkout. Compute nodes carry the solver processes (LS-DYNA, Pam-Crash, Radioss, Abaqus/Explicit). Storage nodes export the parallel filesystem. Compute nodes are usually dual-socket, 64–128 cores each, with 256–512 GB RAM. For crash models, memory is rarely the binding constraint; interconnect latency and bandwidth are.

Crash solvers use domain decomposition: the mesh is split into N domains, one per MPI rank, and each rank exchanges boundary-node forces with its neighbors every time step. A 10-million-element model on 256 ranks produces roughly 2,000–5,000 MPI messages per time step, and a typical explicit time step is 0.5–1.5 µs. That means the interconnect must sustain millions of small messages per second with sub-10 µs latency. InfiniBand HDR (200 Gb/s) or NDR (400 Gb/s) with a fat-tree or dragonfly+ topology is the norm. Omni-Path and 100 GbE RoCEv2 appear in cost-sensitive clusters, but RoCEv2 requires careful PFC/ECN tuning to avoid pause storms under all-to-all communication.

The practical consequence: rank placement matters. If the scheduler scatters ranks across racks connected by oversubscribed uplinks, communication time can double. Most sites therefore use topology-aware placement (Slurm's topology/tree or topology/block plugin) so that ranks of one job land on nodes within the same leaf switch whenever possible.

Job Schedulers: Slurm as the Reference

Slurm is the dominant scheduler in crash CAE environments. The relevant configuration for crash workloads is not the scheduler itself but the policy encoded in partitions, QOS, and resource limits. A representative partition layout:

# slurm.conf (excerpt)
PartitionName=crash   Nodes=cn[001-128] MaxTime=48:00:00 MaxNodes=64 \
                      Default=NO State=UP
PartitionName=debug   Nodes=cn[001-008] MaxTime=00:30:00 MaxNodes=8  \
                      Default=YES State=UP
PartitionName=post    Nodes=cn[129-136] MaxTime=04:00:00 MaxNodes=8  \
                      Default=NO State=UP

# QOS limits
QOSName=crash_qos Priority=100 MaxTRESPerUser=cpu=2048 MaxJobsPerUser=20

Three policy decisions dominate reproducibility. First, exclusive vs. shared nodes: crash jobs should run exclusively (--exclusive) because memory bandwidth contention from a co-scheduled job can slow the solver by 15–30%. Second, core specialization: reserve one core per node for the OS and MPI progress threads (--core-spec=2), otherwise the solver's own threads compete with system daemons. Third, deterministic rank mapping: use --ntasks-per-node and --distribution=block:block so that the same job always produces the same rank-to-node mapping. Without this, two runs of the same deck can differ in wall time by 20% or more, which destroys any attempt at performance reproducibility.

A minimal but complete submission script for a crash job:

#!/bin/bash
#SBATCH --job-name=front_impact
#SBATCH --partition=crash
#SBATCH --nodes=8
#SBATCH --ntasks-per-node=32
#SBATCH --exclusive
#SBATCH --core-spec=2
#SBATCH --distribution=block:block
#SBATCH --time=24:00:00
#SBATCH --output=%x_%j.out

module load intel/2023.2 openmpi/4.1.5 ls-dyna/13.1
export LSTC_LICENSE=network
export LSTC_LICENSE_SERVER=27000@license01

srun --mpi=pmix --cpu-bind=cores \
     ls-dyna i=front_impact.k ncpu=256 memory=2000m \
     d3plot=50ms binout=10ms

The ncpu=256 must match nodes × ntasks-per-node. A mismatch is the single most common cause of "job runs but is 3× slower than expected" — the solver silently falls back to fewer ranks or oversubscribes cores.

Shared Storage: The Hidden Bottleneck

Crash solvers write d3plot state files at fixed intervals. A 10-million-element model produces roughly 1.5–3 GB per state, and a 150 ms event with 50 ms output intervals yields 3–4 states plus a binout file. During the solve, each rank writes its own portion of the state file; at the end, the master rank assembles them. This is a many-writer, then single-reader pattern that stresses both metadata and bandwidth.

Lustre and GPFS/Spectrum Scale are the standard choices. The key parameters are stripe count and stripe size. For crash output directories, a stripe count of 4–8 across OSTs and a stripe size of 1–4 MB works well. A common mistake is to leave the default stripe count of 1, which serializes all writes to a single OST and caps throughput at ~500 MB/s regardless of how many ranks are writing.

# Set stripe on the output directory before the job runs
lfs setstripe -c 8 -S 4M /scratch/$USER/front_impact
# Verify
lfs getstripe /scratch/$USER/front_impact

Equally important is scratch hygiene. Crash jobs should write to node-local NVMe or a burst buffer, then stage results to the parallel filesystem after the solve. Writing d3plot directly to a shared Lustre filesystem during the solve is the second most common cause of poor scaling. A representative staging pattern:

# Stage input to node-local scratch
sbcast front_impact.k /local/scratch/front_impact.k
cd /local/scratch
srun ls-dyna i=front_impact.k ncpu=256
# Stage results back
cp d3plot* binout* /scratch/$USER/front_impact/

Putting the Layers Together

The three layers interact. A topology-unaware scheduler placement increases MPI latency, which increases time-step cost, which increases the number of output writes, which increases storage pressure. Conversely, a well-tuned Lustre filesystem cannot compensate for a job that scatters ranks across oversubscribed links. The diagram below shows the data and control flow through a typical crash-simulation HPC environment, with the three layers and their key tuning points.

Reproducibility Checklist

Before you can claim a crash simulation is reproducible, verify the following on every run. This is not a generic checklist; each item maps to a specific failure mode observed in crash CAE environments.

With the architecture understood — scheduler policy, interconnect topology, and storage layout — you can now treat the HPC environment as a reproducible component rather than a black box. The next step is to encode these settings into a version-controlled environment definition so that every team member gets the same cluster behavior from the same job script.