2. HPC cluster prerequisites for PAM-CRASH

PAM-CRASH is a memory-bandwidth-hungry explicit finite element solver. Before any tuning flags are applied, the cluster must be sized so that the solver is not starved by the platform. This section covers the three prerequisite decisions that dominate early performance: CPU and memory selection, interconnect evaluation, and storage sizing. The goal is a platform where a crashworthiness model of 5–20 million elements can run at a predictable, reproducible throughput.

Hardware Selection: CPU, Memory, and Node Balance

PAM-CRASH's explicit time integration is dominated by element-level floating-point work and by the memory traffic needed to stream nodal coordinates, velocities, and element state variables. The practical consequence is that core count alone is a poor predictor of performance. A node with 64 fast cores but only 4 memory channels will often lose to a 32-core node with 8 channels, because the solver stalls on memory access rather than arithmetic.

For a modern deployment, the following per-node profile is a reasonable baseline for production crashworthiness work:

  • CPU: 32–64 physical cores per node, high all-core turbo frequency (≥2.8 GHz sustained), AVX-512 or equivalent wide SIMD support. PAM-CRASH's element kernels benefit measurably from wide vector units.
  • Memory: 256–512 GB DDR4/DDR5 per node, populated across all memory channels. A common mistake is filling only half the DIMM slots, which halves achievable bandwidth.
  • Memory bandwidth: target ≥200 GB/s per node for a 64-core configuration. This is the single most important hardware metric for explicit solvers.
  • Local scratch: NVMe SSD, 1–2 TB per node, for solver scratch files and restart dumps.

A useful sizing rule for memory is to budget roughly 1.5–2.5 GB per million solid elements, plus the model's contact and constraint overhead. A 10-million-element full-vehicle model with extensive self-contact typically needs 30–50 GB of resident memory, so a 256 GB node comfortably holds it while leaving headroom for the operating system and MPI buffers.

Practical check: Before committing to a node design, run the STREAM Triad benchmark on a single node. If measured bandwidth is below 60% of the theoretical peak, the memory population or BIOS configuration is wrong, and PAM-CRASH will underperform regardless of core count.

Interconnect Evaluation

Domain decomposition splits the model across MPI ranks, and every time step requires halo exchange of boundary node data between neighboring domains. The volume of this exchange scales with the surface area of the subdomains, while the useful computation scales with their volume. As rank count grows, communication becomes proportionally more expensive, so interconnect latency and bandwidth directly cap scalability.

Three interconnect classes are commonly encountered:

Interconnect Typical Latency Per-Port Bandwidth Suitability for PAM-CRASH
1 GbE ~60–100 µs ~0.12 GB/s Not viable beyond ~8 ranks
10/25 GbE ~10–20 µs ~1.2–3 GB/s Acceptable for small clusters, poor scaling past 64 ranks
InfiniBand HDR/NDR ~1–2 µs ~25–50 GB/s Recommended for production scaling to hundreds of ranks

When evaluating an interconnect, do not rely on the vendor's headline bandwidth. Measure two things on the actual fabric: point-to-point latency (via ib_send_lat or osu_latency) and collective bandwidth (via osu_allreduce). PAM-CRASH's halo exchange is nearest-neighbor, so point-to-point latency between adjacent ranks matters more than all-reduce performance. A fabric with 1.5 µs latency and 25 GB/s per port will scale a crash model to 256–512 ranks; a 10 GbE fabric will plateau around 64 ranks.

Also verify that the MPI implementation is tuned for the fabric. OpenMPI or Intel MPI must be built with the correct verbs or UCX transport, and the job scheduler must pin ranks to cores to avoid oversubscription. A misconfigured MPI layer can waste more time than a slower interconnect.

Storage Sizing

Storage requirements for PAM-CRASH fall into three distinct tiers, each with different performance characteristics:

  1. Home/shared filesystem: Stores input decks, material libraries, and results. Capacity is driven by model count and retention policy. A typical full-vehicle model with d3plot output can produce 50–200 GB per run, so a 100 TB parallel filesystem (Lustre, GPFS, or similar) is a reasonable starting point for a mid-sized team.
  2. Node-local scratch: PAM-CRASH writes scratch files and restart dumps here during the run. This must be NVMe SSD, not spinning disk. A 1–2 TB NVMe drive per node is sufficient for most crash models.
  3. Checkpoint/restart storage: Restart files are written periodically and can be large. If the job is long, restart I/O can become a bottleneck. Place restart files on the fastest available tier, ideally local NVMe or a dedicated high-speed parallel filesystem.

A common failure mode is running scratch on a shared network filesystem. When dozens of ranks simultaneously write scratch data over NFS or a saturated parallel filesystem, I/O wait time can exceed computation time. Always direct scratch and restart paths to node-local NVMe.

Warning: Do not underestimate output volume. A 10-million-element model writing d3plot every 1 ms of simulated time can generate hundreds of gigabytes. Size the shared filesystem for the aggregate output of all concurrent jobs, not just one.

Putting the Prerequisites Together

The three prerequisites are interdependent. A node with excellent memory bandwidth but a slow interconnect will scale poorly beyond a few nodes. A fast interconnect cannot compensate for a node that is memory-starved. Storage that cannot keep up with scratch I/O will stall the solver regardless of CPU or network quality.

The diagram below summarizes how these components interact during a PAM-CRASH run and where bottlenecks typically appear.

PAM-CRASH HPC Prerequisite Interaction Compute Node CPU: 32–64 cores, AVX-512 Memory: 256–512 GB, all channels Local NVMe scratch: 1–2 TB Interconnect InfiniBand HDR/NDR Latency: ~1–2 µs Bandwidth: 25–50 GB/s/port Storage Tiers Shared FS: 100 TB+ Node scratch: NVMe Restart: fastest tier halo exchange scratch I/O Common Bottleneck Symptoms and Root Causes • Low CPU utilization + high MPI wait → interconnect latency or MPI misconfiguration • High CPU utilization but slow time-step → insufficient memory bandwidth (check DIMM population) • Long I/O stalls during restart → scratch on shared FS instead of local NVMe Expected Scaling Behavior by Interconnect InfiniBand scales to 512+ ranks 10/25 GbE plateaus ~64 ranks

Use this interaction model as a checklist when commissioning a new cluster. Verify memory bandwidth first, then interconnect latency, then storage throughput. Each layer must be validated independently before PAM-CRASH tuning begins, because a deficiency in any one layer will mask improvements made elsewhere.

Commissioning Checklist

The following checklist captures the minimum validation steps before the cluster is accepted for production PAM-CRASH runs. Each item should be executed on the actual hardware, not inferred from specifications.

# Check Tool / Method Pass Criterion
1 Memory bandwidth STREAM Triad ≥60% of theoretical peak
2 Interconnect latency osu_latency ≤2 µs for InfiniBand
3 Interconnect bandwidth osu_bw ≥80% of link rate
4 Local scratch throughput fio sequential write ≥2 GB/s per node
5 MPI rank pinning Check job script 1 rank per physical core, no oversubscription
6 Scratch path isolation Inspect solver input scratch and restart on local NVMe

Once these prerequisites are satisfied, the platform is ready for the solver-level tuning covered in later sections. Skipping any of them will produce misleading benchmark results and make it impossible to distinguish solver inefficiency from platform deficiency.

← Previous 📖 Contents Next →