2. Linux HPC cluster architecture for crash simulation

Crash CAE workloads impose a very specific set of demands on a Linux HPC cluster. A typical full-vehicle explicit crash model contains 8–20 million elements, runs for 20–60 hours of wall-clock time, and is decomposed across 200–2000 MPI ranks. The solver — LS-DYNA, Pam-Crash, Radioss, or Abaqus/Explicit — is memory-bandwidth bound, writes large restart and d3plot files at regular intervals, and is extremely sensitive to interconnect latency during the domain-decomposition exchange phase. The cluster architecture must therefore be designed around the solver's execution model, not around generic HPC assumptions.

Node Roles and Their Separation

A production crash cluster is organized into four functional tiers. The head node (also called the management or master node) runs the cluster control plane: the resource manager (Slurm slurmctld, PBS pbs_server), the scheduler, the configuration database (LDAP or SSSD), the provisioning service (Warewulf, xCAT, or Confluent), and monitoring (Prometheus, Grafana, Nagios). It must never run user jobs. Its failure stops scheduling for the entire cluster, so it is usually deployed as an active/passive pair with a shared VIP.

Login nodes are the user-facing entry points. CAE engineers SSH here to prepare decks, submit jobs with sbatch or qsub, run small pre-processing tasks (HyperMesh, ANSA, Primer), and post-process results with LS-PrePost or Meta. Login nodes need fast local scratch, a full CAE toolchain, and a modest core count (16–32 cores) — but they must be isolated from compute traffic so that a runaway post-processing session cannot starve the solver fabric.

Compute nodes are where the solver actually executes. For crash simulation, the dominant configuration is a dual-socket node with 64–128 physical cores, 256–512 GB DDR5 per node, and local NVMe scratch of 1–4 TB. The memory-to-core ratio matters: explicit crash solvers typically need 1.5–4 GB per core depending on element count and contact complexity. A node with 128 cores and 256 GB gives only 2 GB/core, which is adequate for medium models but will page on large ones. Local NVMe is essential because the solver writes d3plot, binout, and restart files continuously; writing these directly to the parallel filesystem creates metadata storms.

Storage tier is split by access pattern. A parallel filesystem (Lustre, GPFS/Storage Scale, or BeeGFS) provides the shared /home and /project namespace with 5–20 PB capacity and 100–400 GB/s aggregate throughput. A smaller, faster tier — often NVMe-backed Lustre OSTs or a dedicated WekaFS — handles the hot working set: input decks, restart files, and the first hours of d3plot output. Archive storage (tape or object store) holds completed runs for regulatory retention, which in automotive crash can be 15+ years.

Interconnect Topology and Solver Communication

The interconnect is the single most consequential architectural choice for crash CAE. Explicit solvers communicate in two patterns: nearest-neighbor halo exchange during the time-step loop, and collective operations (reductions, broadcasts) during contact detection and time-step calculation. Halo exchange is latency-sensitive and bandwidth-hungry; collectives are topology-sensitive.

InfiniBand HDR (200 Gb/s) or NDR (400 Gb/s) in a fat-tree topology is the standard for medium-to-large clusters. A 2-level fat-tree with 1:1 or 2:1 oversubscription handles 500–2000 nodes well. For smaller clusters (under 128 nodes), a single-level fat-tree or a dragonfly variant may be sufficient. RoCEv2 over 100/200 GbE is a viable lower-cost alternative, but requires careful PFC/ECN tuning to avoid congestion collapse during collective phases — a common cause of "the job ran fine on 64 ranks but slowed down at 512 ranks" symptoms.

GPU acceleration is increasingly relevant. LS-DYNA and Radioss both offer GPU offload, and a node with 4–8 NVIDIA H100 or A100 GPUs connected via NVLink can replace 2–4 CPU nodes for the same model. The interconnect then must carry GPU-direct RDMA traffic, so NIC-to-GPU affinity (PCIe topology, NUMA alignment) becomes a first-order design concern.

Mapping the Solver Execution Model to Hardware

The architectural choices above only pay off if the solver's execution model is mapped correctly onto them. Three mappings matter most in crash CAE.

Rank placement and NUMA alignment. On a dual-socket node, each socket has its own memory controller and LLC. If MPI ranks are placed without NUMA awareness, half the memory accesses cross the UPI link, and effective bandwidth drops by 30–50%. Slurm's --cpu-bind=cores with --distribution=block:block pins ranks to physical cores within a socket. For a 128-core node running 128 ranks, the correct binding is one rank per physical core, with the rank-to-socket mapping matching the memory allocation. Verify with numactl --hardware and lstopo before submitting production jobs.

Halo exchange and interconnect oversubscription. Explicit crash solvers exchange boundary data every time step. For a model decomposed into 512 domains, each rank communicates with 5–20 neighbors. If the fat-tree is oversubscribed 4:1, the halo exchange stalls behind other jobs' traffic, and time-per-step grows non-linearly with rank count. The practical rule: keep oversubscription at or below 2:1 for crash workloads, and prefer 1:1 for jobs above 256 ranks. Monitor with ibstat, perfquery, and the solver's own timing breakdown (LS-DYNA's d3hsp file reports communication time per phase).

I/O staging. The solver writes d3plot files every 1–10 ms of simulated time. For a 40 ms crash simulation, that is 4–40 writes of 10–100 GB each. Writing these directly to Lustre creates lock contention and metadata pressure that slows every other job on the filesystem. The standard pattern is: solver writes to local NVMe scratch (/scratch/$SLURM_JOB_ID), and a post-processing step copies the completed d3plot series to the shared filesystem. This decouples solver I/O from filesystem load and is the single most effective tuning step for crash clusters.

Failure Modes and Verification

Two failure modes dominate crash cluster operations. First, rank-to-core misalignment: the job runs but time-per-step is 1.5–2× higher than expected. Diagnose with srun --cpu-bind=verbose and compare against a known-good binding. Second, interconnect congestion during collectives: the job runs fine at 64 ranks but degrades sharply at 512. Diagnose with ibmonitor or fabric-level counters, and check whether the job's rank placement crosses spine switches unnecessarily.

Verification of the architecture itself is straightforward. Run a representative crash deck at 64, 128, 256, and 512 ranks. Plot time-per-step against rank count. A well-designed cluster shows near-linear scaling up to the point where communication dominates; a poorly designed one shows a knee at 128 or 256 ranks. The location of that knee tells you whether the bottleneck is interconnect, memory bandwidth, or I/O — and therefore which tier of the architecture needs attention.