2. HPC I/O stack for crash simulation

Explicit crash solvers such as LS-DYNA, Radioss, and Abaqus/Explicit generate a distinctive I/O signature: long stretches of pure computation punctuated by short, violent bursts of checkpoint and state-file writes. A typical full-vehicle frontal impact model with 8–15 million elements may run for 20–40 hours of wall-clock time, yet write 50–200 GB of restart data in bursts that last only seconds. Understanding how that traffic traverses the HPC storage hierarchy — from the solver's MPI ranks down to spinning disks or NVMe — is the prerequisite for tuning it.

The Layered I/O Path

When an explicit solver writes a checkpoint, the data does not travel directly to the parallel file system. It passes through several layers, each with its own buffering, locking, and striping behavior:

• Solver-level I/O — the code decides what to write (d3plot, d3dump, binout), in what format (binary, ASCII), and from which ranks.
• MPI-IO / ROMIO — collective or independent writes are translated into POSIX or native file-system calls, with two-phase I/O and data sieving optimizations.
• Client layer — Lustre LNet, GPFS NSD clients, or BeeGFS client daemons on each compute node.
• Network fabric — InfiniBand HDR/NDR or Omni-Path carries the traffic to storage targets.
• Storage targets and OSTs — Lustre OSTs, GPFS NSDs, or BeeGFS storage targets, each backed by RAID arrays or NVMe.

The critical insight for crash simulation is that write patterns are not uniform. A d3plot write from 512 ranks may produce 512 small, non-contiguous writes per timestep, while a d3dump restart is a single large collective write. These two patterns stress the stack in completely different ways.

Parallel File System Characteristics

For a d3dump restart file of 80 GB written collectively by 1024 ranks, Lustre with a stripe count of 8 and stripe size of 4 MB will distribute the file across 8 OSTs. If the stripe count is too low, a single OST becomes a bottleneck; if too high, lock contention and network overhead grow. A practical starting point is lfs setstripe -c 8 -S 4M /scratch/run42/, then measure with lfs getstripe and adjust based on observed OST utilization.

Burst Buffers and Node-Local NVMe

Burst buffers sit between compute nodes and the parallel file system, absorbing checkpoint bursts so the solver does not stall waiting for spinning disks. Two deployment models dominate:

• Shared burst buffer — a dedicated tier of NVMe nodes (e.g., Cray DataWarp, DDN IME, or a Lustre-backed flash tier) accessible via the same fabric. Data is staged to the parallel file system asynchronously.
• Node-local NVMe — each compute node has one or more NVMe drives used as a local scratch or cache. The solver writes to local NVMe, and a background process migrates data to the parallel file system.

Node-local NVMe is attractive for crash simulation because the write pattern is naturally node-local: each rank writes its own element data. A 2 TB NVMe drive per node can hold several restart generations, and the solver sees microsecond-level write latency instead of millisecond-level. The trade-off is that data is not durable until migrated, and migration competes with the solver for network bandwidth.

Write Patterns from Explicit Solvers

Explicit solvers exhibit three distinct write patterns, each requiring different tuning:

1. Periodic state dumps (d3plot) — every N cycles, each rank writes its portion of the mesh state. These are frequent, moderate-sized, and often non-collective. They benefit from node-local NVMe or a burst buffer with small-block optimization.
2. Restart checkpoints (d3dump) — written at user-defined intervals or on termination. These are large, collective, and latency-sensitive. They benefit from high stripe counts on Lustre or GPFS.
3. Time-history and binout files — small, frequent, often written by a single rank. These can become metadata bottlenecks if not aggregated.

A common failure mode is treating all three patterns identically. Writing d3plot files directly to a Lustre file system with a low stripe count causes OST hotspots; writing d3dump files to node-local NVMe without migration risks data loss on node failure. The correct strategy is to route each pattern to the tier that matches its characteristics.

Practical Tuning and Verification

After routing each write pattern to the appropriate tier, verify the configuration with measurable checks. For a Lustre-backed d3dump write, run lfs df -h before and after a restart to confirm OST utilization is balanced. If one OST shows 90% while others show 20%, increase the stripe count or use lfs setstripe -c -1 to stripe across all available OSTs.

For node-local NVMe, monitor write latency with iostat -x 1 and watch the %util and await columns. If await exceeds 1 ms during a d3plot write, the NVMe is saturated and the solver may stall. In that case, reduce the d3plot frequency or increase the number of NVMe drives per node.

For burst buffers, the key metric is migration lag: the time between data landing in the burst buffer and being persisted to the parallel file system. If lag grows during a run, the migration process is not keeping up. Check the burst buffer's migration queue depth and consider increasing the migration concurrency or reducing the checkpoint size.

Summary

The HPC I/O stack for crash simulation is not a single path but a hierarchy of tiers, each suited to a different write pattern. Explicit solvers produce periodic d3plot dumps, large collective d3dump restarts, and small binout files. Mapping each pattern to the right tier — node-local NVMe for frequent small writes, burst buffers for staging large checkpoints, and parallel file systems for durable storage — is the core engineering decision. The tuning parameters that matter most are Lustre stripe count and size, NVMe migration watermarks, and burst buffer migration concurrency. Verify with lfs df, iostat, and migration lag metrics, and adjust until the solver's write stalls disappear from the wall-clock profile.