3. High-performance computing infrastructure for crash simulation

An explicit LS-DYNA crash model of a full vehicle typically contains 8–15 million shell and solid elements, runs for 120–150 ms of physical time, and requires a stable time step on the order of 0.5–1.0 µs. That translates into roughly 150,000–300,000 explicit integration cycles. On a single modern CPU core, such a job would take weeks. The entire purpose of the HPC infrastructure described in this section is to compress that wall-clock time to a few hours while preserving bit-level reproducibility across runs. Four subsystems must work together: the compute cluster, the interconnect, the parallel file system, and the job scheduler. A weakness in any one of them becomes the bottleneck for the whole simulation workflow.

Compute Cluster Architecture

A crash-simulation cluster is organized as a set of compute nodes connected by a high-speed fabric, with a small number of login nodes and management nodes handling user access and orchestration. Each compute node carries two or more sockets of server CPUs (for example, 2 × 64-core AMD EPYC or 2 × 32-core Intel Xeon), 256–512 GB of DDR5 memory, and a local NVMe scratch disk. Memory capacity is not arbitrary: LS-DYNA's implicit and explicit solvers both hold the full model in RAM, and a practical rule is to budget 1.5–2.0 GB of RAM per million elements for explicit crash models, plus headroom for contact and airbag particle data.

Two design choices dominate cost and performance. First, core count per node: more cores per node reduce the number of MPI ranks that must cross the network, but they also increase memory bandwidth contention. Crash models are memory-bandwidth-bound during element processing, so a node with 128 cores sharing 12 memory channels will scale sub-linearly beyond roughly 64 ranks. Second, CPU frequency versus core count: the explicit time step is set by the smallest element, and the solver is largely latency-tolerant, so higher core counts usually win over higher clock speeds for large models.

Accelerators (GPUs) are increasingly used for the contact and element-processing kernels of LS-DYNA and Radioss, but the crash workflow still depends heavily on CPU nodes for preprocessing, penalty-based contact setup, and post-processing. A realistic 2024-era cluster for automotive OEM work might contain 200–800 nodes, delivering 50,000–200,000 cores.

Interconnect: Why Latency and Bandwidth Both Matter

Explicit crash solvers decompose the mesh into domains, one per MPI rank. Every integration cycle, each rank exchanges ghost-node data with its neighbors across domain boundaries. The volume of this exchange scales with the surface area of the domain, while computation scales with its volume — so as rank count grows, communication becomes proportionally more expensive. This is the classic surface-to-volume problem, and it is why interconnect choice directly caps scalability.

InfiniBand HDR (200 Gb/s) and NDR (400 Gb/s) are the dominant fabrics in crash-simulation clusters, with Omni-Path and RoCEv2 as alternatives. The relevant metrics are latency (typically 1–2 µs for small messages) and message rate, not just peak bandwidth. A crash job with 512 ranks may exchange millions of small messages per second; a fabric with high bandwidth but poor message rate will stall. Non-blocking fat-tree or dragonfly+ topologies are standard, and oversubscription should be avoided for crash workloads.

Storage: Parallel File Systems and Scratch Strategy

Crash simulations are I/O-intensive in bursts. During a run, the solver writes d3plot state files at fixed intervals (often every 1–5 ms of physical time), plus restart files, interface force files, and material energy summaries. A single full-vehicle run can produce 50–500 GB of output. Reading the input deck and writing results both stress the file system.

Production clusters use a parallel file system — Lustre, GPFS/Spectrum Scale, or BeeGFS — with separate metadata and object storage targets. The key performance metric is aggregate throughput under many-client concurrency, not single-stream bandwidth. A 500-node cluster typically needs 100–300 GB/s of aggregate read/write to avoid I/O stalls during synchronized d3plot writes.

A common and effective strategy is tiered scratch: stage the input deck and solver executable to node-local NVMe, run the simulation writing d3plot to local scratch, then copy results to the parallel file system after the run completes. This avoids hammering the shared file system with thousands of concurrent writers. The trade-off is that local scratch is not persistent — if a node fails, results are lost — so restart files should still be written periodically to the shared file system.

Job Schedulers: Slurm in Practice

The job scheduler allocates nodes, enforces fair-share policies, and manages the queue. Slurm is the de facto standard in automotive HPC. A typical crash-simulation submission script looks like this:

#!/bin/bash
#SBATCH --job-name=fullveh_frontal
#SBATCH --partition=crash
#SBATCH --nodes=8
#SBATCH --ntasks-per-node=64
#SBATCH --mem=0
#SBATCH --time=12:00:00
#SBATCH --output=slurm-%j.out

module load ls-dyna/13.1.0
export LSTC_LICENSE_SERVER=27000@license01

srun --mpi=pmi2 ls-dyna i=fullveh_frontal.k ncpu=512 memory=400m \
     d3plot=1.0e-3 endtim=0.15

Three parameters deserve attention. ncpu must match the total MPI rank count (nodes × ntasks-per-node), or the solver will either under-utilize the allocation or fail to start. memory=400m tells LS-DYNA to allocate 400 MB per rank for the solver's internal arrays; setting this too low causes a "memory required" abort, while setting it too high wastes RAM and can trigger node-level OOM kills. d3plot=1.0e-3 sets the output interval to 1 ms of physical time — a reasonable default for frontal impact, but coarse for airbag deployment studies, which may need 0.1 ms.

Scheduler policies also matter. Backfill scheduling lets small jobs jump ahead of large ones without delaying them, improving cluster utilization. Fair-share prevents one team from monopolizing the cluster. Reservations are used for overnight regression runs that must complete before morning design reviews.

Putting It Together: A Reference Topology

The diagram below shows how the four subsystems connect in a typical crash-simulation cluster. Login nodes submit jobs to the scheduler, which dispatches MPI ranks across compute nodes. Compute nodes exchange ghost-node data over InfiniBand and read/write d3plot files to the parallel file system, with local NVMe used as a staging tier.

Practical Sizing and Failure Symptoms

When a crash job underperforms, the symptom usually points to one subsystem. If wall-clock time barely improves when you double the rank count, the interconnect or domain decomposition is the limit — check the LS-DYNA message file for the communication-to-computation ratio. If the job stalls periodically and then resumes, the parallel file system is likely saturated during d3plot writes; reduce output frequency or stage to local NVMe. If the job aborts with "insufficient memory," increase memory= per rank or reduce ranks per node. If nodes sit idle while the scheduler reports the job as running, check for MPI rank mismatch between ncpu and the Slurm allocation.

A useful sizing heuristic for frontal impact models: 1 million elements per 8–16 cores, with 2 GB RAM per million elements. A 10-million-element full vehicle therefore needs roughly 640–1,280 cores and 20 GB of aggregate solver memory, spread across 10–20 nodes. Validate any new configuration with a short 20 ms run before committing to a full 150 ms production job — the scaling curve you measure there will predict your production turnaround within about 10%.