2. HPC cluster environment setup for PAM-CRASH

PAM-CRASH is a memory-bandwidth-hungry explicit finite element solver. A crashworthiness model with 5–20 million elements will typically consume 8–40 GB of RAM per MPI rank once contact, element, and material data are distributed, and each explicit time step exchanges nodal force and velocity data across partition boundaries. The cluster you build around it therefore has to satisfy three constraints simultaneously: enough memory bandwidth per core, an interconnect with low latency and high message rate, and a scheduler/MPI stack that does not add jitter to the tightly coupled communication pattern. This section walks through the practical decisions for each layer.

Hardware Selection: What Actually Matters

PAM-CRASH scales best when the ratio of memory bandwidth to core count stays high. A useful rule of thumb for explicit crash analysis is 2.5–4 GB of RAM per MPI rank and at least 4–6 GB/s of achievable STREAM Triad bandwidth per rank. On a dual-socket node with 64 cores and 512 GB of DDR5-4800 (12 channels per socket), the theoretical peak is roughly 920 GB/s; measured STREAM Triad is typically 65–75% of that, i.e. about 600–690 GB/s, or roughly 9–11 GB/s per core. That is comfortable. A node with 128 cores and the same memory subsystem would drop to 4.7–5.4 GB/s per core, which is where PAM-CRASH begins to stall on memory rather than on floating-point work.

Practical guidance for a PAM-CRASH-oriented node:

• CPU: Prefer fewer, higher-frequency cores over many low-frequency cores. Explicit solvers do not benefit much from SMT; disable hyper-threading and pin one rank per physical core.
• Memory: Populate all channels. A 12-channel socket with only 8 DIMMs installed loses roughly one third of its bandwidth — a common and expensive mistake.
• Local storage: NVMe scratch for the d3dump restart files and *.out result files. A 20-million-element model can write several GB per restart interval.
• Node count vs. node size: For a fixed core budget, more smaller nodes usually scale better than fewer large nodes, because the domain decomposition produces smaller per-rank subdomains and less communication per rank.

Interconnect: Latency Dominates at Scale

Explicit crash solvers communicate every time step, so the cost per message matters more than raw bandwidth. InfiniBand HDR (200 Gb/s) or NDR (400 Gb/s) with RDMA is the standard choice; 100 GbE/RoCEv2 is workable for smaller clusters but message-rate and latency are typically 1.5–2× worse. Key parameters to verify:

Always validate the fabric with ib_write_lat, ib_write_bw, and osu_latency before running any solver benchmark. A fabric that passes link-up but shows 8 µs latency usually has a PCIe generation mismatch or a misconfigured MTU.

Job Scheduler: Slurm as the Reference Stack

Slurm is the de facto scheduler for PAM-CRASH clusters. The configuration points that matter for solver benchmarking are core isolation, memory enforcement, and topology-aware placement. A minimal but effective slurm.conf fragment:

SelectType=select/cons_tres
SelectTypeParameters=CR_Core_Memory
TaskPlugin=task/affinity,task/cgroup
ProctrackType=proctrack/cgroup
SchedulerType=sched/backfill
NodeName=cn[01-16] CPUs=64 RealMemory=515000 Sockets=2 CoresPerSocket=32 ThreadsPerCore=1
PartitionName=crash Nodes=cn[01-16] Default=YES MaxTime=24:00:00 State=UP

Two settings deserve emphasis. TaskPlugin=task/affinity binds each rank to a physical core, which prevents the operating system from migrating ranks and destroying cache locality. CR_Core_Memory enforces the per-node memory request so that a 40 GB/rank job cannot be scheduled onto a node with insufficient free memory. For benchmarking, always request whole nodes (--exclusive) so that co-tenants do not perturb your timings.

MPI Configuration and Rank Placement

PAM-CRASH is normally launched through pamcrash_smp or the site wrapper pam -np N, which internally calls mpirun. The MPI implementation should be the vendor-tuned build shipped with the interconnect (HPE MPT, Intel MPI, or Open MPI with UCX). For Open MPI with UCX on InfiniBand, a working baseline is:

export UCX_TLS=rc,sm,self
export UCX_NET_DEVICES=mlx5_0:1
export OMPI_MCA_pml=ucx
export OMPI_MCA_btl=^openib
mpirun --bind-to core --map-by core:PE=1 \
       -np 64 --hostfile hosts.txt \
       pamcrash_smp -i model.key -o model.out

The critical flags are --bind-to core and --map-by core:PE=1. Without them, Open MPI may place two ranks on one core and leave another idle, which typically costs 30–50% of wall-clock time on a memory-bound crash model. Verify placement with mpirun --report-bindings before trusting any benchmark number.

Putting It Together: A Reproducible Baseline

Before running any PAM-CRASH benchmark, establish a node-level baseline. The following sequence takes about 15 minutes and catches most configuration errors:

1. Memory bandwidth: run STREAM Triad with 64 threads and confirm ≥ 4 GB/s per core. If not, check DIMM population and BIOS memory frequency settings.
2. Fabric latency: ib_write_lat -d mlx5_0 -a between two nodes. Expect < 2 µs for HDR. If higher, check PCIe link width with lspci -vv.
3. MPI placement: mpirun -np 64 --bind-to core --map-by core:PE=1 --report-bindings hostname. Confirm 64 distinct core bindings across the requested nodes.
4. Solver smoke test: run a small PAM-CRASH model (e.g. 500k elements) at 1, 2, 4, and 8 nodes. Plot wall-clock vs. node count. Near-linear scaling up to 8 nodes indicates the stack is healthy; sub-linear scaling points to a specific layer to investigate.

Once this baseline is stable, the cluster is ready for the parallel solver benchmarking work that follows. The key discipline is to change one layer at a time and re-validate — a fabric or placement regression will otherwise be misattributed to the solver.