2. HPC cluster architecture for crash simulation

Crash simulation is one of the most demanding workloads in industrial CAE. A single full-vehicle frontal impact model can contain 15–25 million shell and solid elements, run for 120–150 ms of physical time, and require 12–48 hours of wall-clock time on several hundred CPU cores. The cluster that hosts this workload is not a generic "big computer" — it is a layered system in which compute nodes, the interconnect fabric, the parallel filesystem, and the job scheduler each impose distinct constraints. Understanding how these four layers interact is the prerequisite for diagnosing slow jobs, planning capacity, and writing scheduler policies that actually match solver behavior.

Compute Nodes: The Unit of Allocation

In LS-DYNA, Pam-Crash, or Radioss workflows, the compute node is the smallest schedulable unit. A typical crash-simulation node in a modern cluster is a dual-socket server with 64–128 physical cores, 256–512 GB of DDR5 memory, and a local NVMe scratch disk of 1–4 TB. The memory-to-core ratio matters more than raw core count: explicit crash solvers are memory-bandwidth bound, and a node with 128 cores but only 256 GB of RAM will thrash when a 20-million-element model needs roughly 2–4 GB per million elements for the working arrays alone.

Two node classes usually coexist. Thin nodes (64–96 cores, 256 GB) handle large batches of medium models and post-processing. Fat nodes (128 cores, 1–2 TB) are reserved for the largest full-vehicle models and for implicit or hybrid stages that assemble a global stiffness matrix. Mixing them in one partition without node-feature constraints is a common source of "job ran 3× slower than yesterday" complaints — the scheduler silently placed the job on a thin node.

Local NVMe scratch deserves explicit attention. Crash solvers write restart files, d3plot state files, and temporary element data continuously. Writing these to the shared parallel filesystem instead of node-local scratch can multiply I/O wait by an order of magnitude. The standard pattern is: stage input decks from the parallel filesystem to local scratch, run the solver with its working directory on scratch, then copy results back.

Interconnect: Where Parallel Efficiency Is Won or Lost

Domain decomposition in explicit crash solvers splits the mesh into one subdomain per MPI rank. Every timestep, each rank exchanges boundary-node forces and coordinates with its neighbors. The communication volume scales with the surface area of the subdomain, while computation scales with its volume — so as rank count grows, communication dominates. This is why interconnect latency and bandwidth directly set the parallel scaling ceiling.

Three fabric generations appear in production clusters:

• InfiniBand HDR/NDR (200/400 Gb/s per port): the default for crash clusters above ~500 cores. NDR with adaptive routing keeps MPI point-to-point latency near 1.5 µs.
• RoCEv2 over 100/200 GbE: lower cost, but requires careful PFC/ECN tuning; misconfigured lossless Ethernet causes silent packet drops that appear as erratic solver slowdowns.
• Omni-Path or legacy FDR: still present in older installations; acceptable for jobs under ~256 ranks.

The practical rule: a well-tuned crash job on NDR InfiniBand typically achieves 75–85% parallel efficiency at 512 ranks; the same job on 100 GbE RoCE often drops to 55–65%. The gap is not academic — it is the difference between a 14-hour and a 22-hour overnight run.

Parallel Filesystems: Throughput for I/O-Heavy Phases

Crash simulation has a bimodal I/O profile. During the solve, each rank writes small restart and state files at intervals; at job end, the solver writes a large d3plot or animation database that can reach 500 GB–2 TB. A parallel filesystem such as Lustre, GPFS/Spectrum Scale, or BeeGFS aggregates many storage targets behind a single namespace, but its performance depends on stripe count and stripe size.

For a 20-million-element model producing a 1.2 TB result set, a stripe count of 8–16 across OSTs with a 4 MB stripe size is a reasonable starting point. Too few stripes bottleneck on a single OST; too many stripes fragment small metadata operations and slow directory listings. The lfs setstripe command (Lustre) or mmchattr (GPFS) sets this per file or per directory.

# Lustre: set stripe layout for a results directory
lfs setstripe -c 12 -S 4M /scratch/crash/results/veh_frontal_20M

# Verify the layout before launching the job
lfs getstripe /scratch/crash/results/veh_frontal_20M

Metadata servers (MDS) are the other half of the story. A job that opens 512 files simultaneously at startup can overwhelm a single MDS. Distributing ranks across multiple directories, or using a filesystem with distributed metadata (Spectrum Scale, BeeGFS), avoids the "all ranks stall at t=0" pattern.

Job Schedulers: Matching Policy to Solver Behavior

Slurm dominates modern HPC clusters; PBS Pro and LSF remain in some commercial CAE environments. The scheduler's job is not just to queue work — it must express the resource shape that crash solvers actually need.

A representative Slurm submission for a full-vehicle LS-DYNA run looks like this:

#!/bin/bash
#SBATCH --job-name=veh_frontal_20M
#SBATCH --partition=crash_fat
#SBATCH --nodes=8
#SBATCH --ntasks-per-node=64
#SBATCH --cpus-per-task=1
#SBATCH --mem=0
#SBATCH --gres=scratch:2000G
#SBATCH --time=24:00:00
#SBATCH --output=slurm-%j.out

module load ls-dyna/13.1
export LSTC_LICENSE=network
export OMP_NUM_THREADS=1

srun --mpi=pmi2 ls-dyna i=veh_frontal_20M.k ncpu=512 memory=2000m

Key parameters: --ntasks-per-node=64 matches the physical core count (no hyperthreading for MPI ranks); --mem=0 requests all node memory; --gres=scratch:2000G reserves local NVMe; ncpu=512 tells the solver the total rank count, which must equal nodes × ntasks-per-node. A mismatch here is the single most common cause of a job that starts but produces garbage or hangs at decomposition.

Backfill scheduling is essential for crash workloads. A 512-rank job that waits for 8 full fat nodes can idle the cluster for hours. Slurm's backfill scheduler fills gaps with shorter jobs, but only if users declare realistic --time limits. Declaring 72 hours for a job that finishes in 14 blocks backfill for everyone behind it.

How the Layers Interact

The four layers form a dependency chain: the scheduler allocates nodes, the nodes communicate over the fabric, and the filesystem absorbs I/O. A bottleneck in any layer propagates. The diagram below shows the data and control paths for a typical crash job, with the key parameters that govern each layer.

Practical Implications for Cluster Operations

Three operational habits follow from this architecture. First, always verify that the scheduler's rank count matches the solver's ncpu parameter and the actual node allocation — a mismatch produces either a hang or silent result corruption. Second, monitor fabric counters (perfquery on InfiniBand) during long runs; a rising symbol-error rate on one port will degrade a 512-rank job long before it triggers a link-down alarm. Third, treat local NVMe scratch as a finite resource: a 2 TB scratch disk shared by 64 ranks writing restart files every 5 minutes will fill in under an hour if the solver's restart interval is too aggressive.

When a crash job underperforms, the diagnostic order is: check rank count and node placement first, then fabric counters, then filesystem stripe layout, then solver decomposition settings. In most production incidents, the bottleneck is in the first two layers, not in the solver itself.