4. Checkpoint/restart mechanisms for explicit solvers

Explicit crash solvers such as LS-DYNA and Pam-Crash advance the solution with a stable time step governed by the smallest element in the mesh. A typical full-vehicle frontal impact model with 5–10 million elements runs at roughly 1–3 µs per step, so a 120 ms event requires on the order of 10⁵–10⁶ cycles. On a 64-core node this may consume 8–30 wall-clock hours. Any hardware fault, scheduler preemption, license timeout, or power event during that window destroys all accumulated work unless the solver can write a consistent snapshot of its state and later resume from it. Checkpoint/restart (C/R) is therefore not an optional convenience in crash CAE infrastructure — it is the primary mechanism that converts a long, fragile job into a recoverable one.

What a Restart File Actually Contains

Both LS-DYNA and Pam-Crash write a binary restart database, but the contents differ in scope and in how much of the model they capture. Understanding the payload is essential before tuning frequency, because the cost of a checkpoint is proportional to what is written.

An LS-DYNA d3dump file (or the single-precision d3dump01 family) stores the complete transient state: nodal coordinates and velocities, element stresses and strains, shell thickness and hourglass control variables, contact interface state (including stored penetration and friction history), rigid body and joint states, and the current time and cycle number. It does not store the original keyword input; on restart the solver re-reads the input deck and overlays the dump. This means the input deck must be byte-identical between the original run and the restart, otherwise the state overlay is inconsistent and the solver aborts or silently produces wrong results.

Pam-Crash uses a restart file (commonly .rf or the RESTART dataset) that similarly captures nodal and element state plus contact and material internal variables. Pam-Crash additionally supports a "safety" restart written at a user-defined interval and a "final" restart at termination. The key architectural difference is that Pam-Crash restart files are often written per-domain in a domain-decomposition run, so a restart on a different number of MPI ranks requires a repartitioning step.

State Capture and Frequency Tuning

The checkpoint interval is controlled by a time-based or cycle-based keyword. In LS-DYNA the relevant controls are *CONTROL_TERMINATION for the end time and *DATABASE_BINARY_D3DUMP for the dump interval:

*CONTROL_TERMINATION
         0.120
*DATABASE_BINARY_D3DUMP
      1.0e-3

Here DT = 1.0e-3 seconds means a dump every 1 ms of simulated time, i.e. 120 dumps for a 120 ms event. The CYCL field can be used instead to dump every N cycles. Pam-Crash exposes an equivalent RESTART frequency through the /RUN or /IO options, typically expressed as a simulated-time interval or a cycle count.

Choosing the interval is a trade-off between write overhead and lost work. If a dump takes t_w seconds and the job fails at a random point, the expected lost compute is half the interval. The total overhead fraction is approximately t_w / (t_w + t_interval). For a 64-core job where a dump of a 6-million-element model takes 40 s and the interval is 600 s of wall time, overhead is about 6%. Halving the interval to 300 s doubles overhead to 12% but halves expected lost work. On a stable cluster with a mean time between failures of several days, a 10–15 minute interval is a reasonable default; on preemptible cloud instances, 2–5 minutes is safer.

Two practical refinements matter. First, write dumps to a local NVMe scratch directory and copy asynchronously to shared storage, because a synchronous write to a parallel filesystem can stall all ranks. Second, avoid dumping at the same cycle as a contact or adaptive-remeshing event, since the state may be mid-update; LS-DYNA handles this internally but Pam-Crash users should verify the dump cycle is not inside an adaptive refinement pass.

Restart Fidelity in Practice

Restart fidelity is the degree to which a resumed run reproduces the trajectory of an uninterrupted run. In explicit dynamics, small state differences amplify because the solution is chaotic at the contact and failure level. A restart that loses even a few contact history variables can shift the timing of a spot-weld failure by several milliseconds, which changes the occupant load path.

To verify fidelity, run a short benchmark: execute a 20 ms model to completion, then execute the same model with a forced kill at 10 ms and a restart from the 10 ms dump. Compare the two runs at 20 ms. Acceptable agreement for crash metrics is typically within 1–2% on peak acceleration and within 1 mm on intrusion. If the deviation is larger, the usual causes are:

• Input deck drift. A parameter was edited between runs (e.g. a material constant or a contact thickness). The dump overlays state onto a different model.
• Domain count change. Restarting a Pam-Crash or MPP LS-DYNA job on a different number of ranks without repartitioning. Contact and element state are distributed differently, and the solver either aborts or produces a mismatched decomposition.
• Missing auxiliary files. LS-DYNA restart requires the original d3plot, d3thdt, and in some cases d3part files to be present or regenerated; a missing file causes the restart to fail at initialization.
• Adaptive or remeshing state. Adaptive refinement and element erosion change the mesh topology. If the dump was written mid-adaptation, the restart may not reconstruct the refined mesh correctly.

Operational Checklist for Production C/R

The following practices are drawn from production crash CAE workflows and are directly applicable to both solvers:

Finally, treat the restart file as a first-class artifact in the job's provenance record. Store the dump alongside the exact input deck hash, the solver version, and the MPI rank count. When a restart fails, this metadata is what allows you to determine whether the failure is a solver bug, a storage fault, or a configuration drift. In a fault-tolerant crash CAE infrastructure, the checkpoint is not just a file — it is the contract between the original run and its recovery.