4. Checkpointing and restart strategies for long-running automotive simulations

Automotive CAE jobs routinely run for hours or days: a full-vehicle frontal impact in LS-DYNA may need 20–60 wall-clock hours on 64–256 cores, a nonlinear Abaqus durability model with contact and plasticity can exceed a day, and a transient Fluent underhood thermal run may require thousands of time steps. In all three cases, a single node failure, a scheduler preemption, or a wall-clock limit will destroy the entire run unless the solver has been told to write restart data. Checkpointing is therefore not an optional convenience — it is the mechanism that converts an all-or-nothing job into a recoverable one.

The three solvers implement this idea differently, and the differences matter when you design a job script. LS-DYNA writes a binary restart file (d3dump) at intervals you control; Abaqus writes restart data into the .res, .mdl, .stt, and .prt family; Fluent writes case and data pairs (.cas.h5 / .dat.h5) plus an autosave mechanism. The common principle is the same: periodically flush the complete solver state — displacements, velocities, stresses, contact state, turbulence fields, and time-step history — to disk so that a new process can resume from that state rather than from the beginning.

What a Restart File Actually Contains

A restart file is a snapshot of the solution state at a specific time, not a copy of the input deck. For an explicit LS-DYNA run, d3dump holds nodal coordinates and velocities, element stresses and strains, hourglass energies, contact interface state, and the current cycle and time. For Abaqus/Standard, the restart file stores the converged increment state, including the Jacobian-relevant history variables needed to continue Newton iterations. For Fluent, the data file stores the full field solution at a time step, while the case file stores mesh, boundary conditions, and solver settings — both are required to restart.

This distinction drives a practical rule: if you change the mesh, material definition, or boundary conditions between the original run and the restart, the restart will either fail or silently produce physically inconsistent results. Restart is a continuation of the same model, not a way to apply a mid-run modification.

Solver-Specific Restart Mechanics

In LS-DYNA, restart output is controlled by *DATABASE_BINARY_D3DUMP and the *CONTROL_TERMINATION / *CONTROL_TIMESTEP cards. A typical automotive crash deck writes a dump every 5 ms of simulated time:

*DATABASE_BINARY_D3DUMP
         5.0
*CONTROL_TERMINATION
       120.0

To restart, you run the same executable with the r= option pointing at the dump file:

ls-dyna i=full_vehicle.k r=d3dump01 memory=2000m ncpu=128

LS-DYNA reads the dump, reconstructs the state, and continues to the termination time. The cycle counter and simulated time resume from the checkpoint, so the total simulated time is preserved.

Abaqus uses *RESTART, WRITE, FREQUENCY=n in the step definition to write restart data every n increments, and *RESTART, READ in a subsequent job to resume. The restart job must reference the original job name:

*RESTART, READ, STEP=2, INC=1500

Fluent's autosave is configured in the solution controls or via the TUI: /file/auto-save/data-frequency 500 writes a data file every 500 iterations or time steps, and /file/auto-save/case-frequency controls case saving. Restarting reads the case and the latest data file, then continues the transient from the saved time step.

Choosing Checkpoint Frequency

The interval is a trade-off between I/O cost and lost work. Writing a d3dump for a 5-million-element model can take 30–90 seconds and produce a file of several gigabytes. If you checkpoint every 1 ms of a 120 ms crash, you write 120 dumps and may spend more time on I/O than on solving. A practical starting point for crash is every 5–10 ms, adjusted so that the expected lost work (half the interval) is acceptable relative to the total run.

For Abaqus, FREQUENCY is in increments, not time. If the job takes 2000 increments and you set FREQUENCY=100, you get 20 restart points. For Fluent transient runs, autosave frequency is in time steps; a frequency of 500 on a 5000-step run gives 10 data files.

Operational Strategy on HPC Clusters

On a shared cluster, the dominant risk is not solver instability but scheduler preemption. A job with a 24-hour wall-clock limit that is killed at hour 23 with no checkpoint loses everything. The standard mitigation is to set the checkpoint interval so that at least one valid restart point exists within the last 10–15% of the wall-clock window, and to write a job script that automatically resubmits from the latest dump.

A minimal LS-DYNA resubmission pattern looks like this:

#!/bin/bash
#SBATCH --time=24:00:00
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=32

LATEST=$(ls -t d3dump* 2>/dev/null | head -1)
if [ -z "$LATEST" ]; then
    ls-dyna i=model.k memory=2000m ncpu=128
else
    ls-dyna i=model.k r=$LATEST memory=2000m ncpu=128
fi

The same pattern applies to Abaqus (abaqus job=restart_job oldjob=original_job recover) and Fluent (read the latest data file in a journal). The key operational discipline is to keep the checkpoint files on a filesystem that survives node failure — a parallel scratch or project directory, not local /tmp.

Failure Symptoms and Troubleshooting

Restart failures usually fall into three categories. First, missing or mismatched files: LS-DYNA reports "error reading d3dump" if the dump is truncated, which often happens when the job was killed mid-write. Second, model inconsistency: Abaqus aborts with "restart file does not match the current model" if the input deck changed. Third, state corruption: the job runs but produces non-physical results, typically because the checkpoint was written during a diverging step.

To reduce these risks, write checkpoints at stable points in the solution — for implicit Abaqus, avoid checkpointing during a step with active contact chattering; for Fluent, avoid saving during a transient ramp with large residuals. For LS-DYNA, the dump is written at the end of a cycle, so it is generally consistent, but a killed write still leaves a partial file. Keeping the previous two dumps and deleting older ones is a simple and effective safeguard.

Finally, document the restart chain. A run that has been restarted four times should have a log recording which dump was used, when, and why. This is essential for traceability in automotive validation, where the difference between a 119 ms and a 120 ms simulated result can matter for occupant injury criteria.