2. Fault tolerance fundamentals for HPC

Crash simulation campaigns are long-running, tightly coupled HPC workloads. A single explicit LS-DYNA or PAM-CRASH job may run for days across hundreds of cores, and a vehicle crashworthiness matrix can contain thousands of such jobs. At that scale, hardware and software failures are not exceptional events — they are statistical certainties. Fault tolerance is therefore not an optional reliability feature but a core design requirement of the computing infrastructure. This section establishes the failure models, quantitative metrics, and recovery mechanisms that underpin every later topic in this course.

Failure Models in Crash CAE Campaigns

Before designing recovery mechanisms, we must classify what actually fails. In HPC crash simulation, failures fall into four practical categories:

• Hardware failures — node crashes, ECC memory errors, GPU faults, InfiniBand link flaps, and storage controller failures. These are typically fail-stop: the affected process dies immediately.
• Software failures — solver segmentation faults, MPI rank aborts, license daemon timeouts, and scheduler misconfigurations. These may be deterministic (reproducible input bug) or transient (race conditions under specific rank counts).
• Silent data corruption (SDC) — bit flips that produce wrong results without crashing. This is the most dangerous class because a corrupted crash result may pass visual inspection yet violate physical plausibility.
• Environmental and operational failures — power loss, cooling failures, network partitions, and administrative actions such as node drains or firmware updates.

The distinction matters because each class demands a different mitigation. Fail-stop failures are handled by checkpoint/restart. SDC requires verification and, where available, redundant computation. Operational failures require graceful job suspension and rescheduling rather than blind restart.

MTBF and Failure Probability at Scale

Mean Time Between Failures (MTBF) is the standard reliability metric for a component or system. For a cluster of N nodes, each with node-level MTBF M_node, the system MTBF is approximately:

MTBF_system ≈ M_node / N

Example: M_node = 100,000 hours (≈ 11.4 years), N = 512 nodes
MTBF_system ≈ 100,000 / 512 ≈ 195 hours (≈ 8.1 days)

This linear scaling is the critical insight: reliability degrades proportionally with node count. A job that runs for 72 hours on a 512-node cluster has a substantial probability of encountering at least one failure. Using the exponential failure model, the probability of surviving a job of duration t is P(survive) = e^−t/MTBF. For t = 72 h and MTBF = 195 h, survival probability is only about 69%. In a campaign of 500 such jobs, roughly 155 will experience a failure. Without checkpointing, each failure means restarting from zero.

Checkpoint/Restart Mechanics

Checkpoint/restart (C/R) is the primary defense against fail-stop failures. The mechanism is conceptually simple: periodically dump the complete solver state to persistent storage, and on failure, reload the most recent checkpoint and continue.

For crash solvers, the checkpoint must capture:

• Nodal coordinates, velocities, and accelerations
• Element stresses, strains, and history variables (plastic strain, damage, hourglass control state)
• Contact interface states and penalty forces
• Time step counter, current simulation time, and adaptive time-step size
• MPI decomposition metadata (domain boundaries, ghost element lists)

LS-DYNA provides native C/R through the *DATABASE_BINARY_D3DUMP keyword, which writes restart files at a specified time interval. A typical configuration writes a dump every 30–60 minutes of wall-clock time:

*DATABASE_BINARY_D3DUMP
$#   dt      lcdt      beam      nplt       psid
   1800.0              0         0         0

The checkpoint interval involves a trade-off. Shorter intervals reduce lost work per failure but increase I/O overhead and storage consumption. The optimal interval T_opt can be estimated by minimizing total overhead:

T_opt = sqrt(2 × C_checkpoint × MTBF)

Where C_checkpoint = time to write one checkpoint
Example: C_checkpoint = 120 s, MTBF = 195 h = 702,000 s
T_opt = sqrt(2 × 120 × 702,000) ≈ sqrt(168,480,000) ≈ 12,980 s ≈ 3.6 hours

In practice, intervals of 30–60 minutes are common because they also bound the lost computation to an acceptable level and align with storage bandwidth constraints. The checkpoint write itself must not stall the solver for excessive periods; parallel I/O libraries such as MPI-IO or vendor-specific burst buffers are used to overlap checkpoint writes with computation where possible.

Redundancy Strategies

Checkpointing handles fail-stop failures but does not address SDC or correlated failures. Redundancy provides complementary protection:

• Spatial redundancy — running the same simulation on two independent partitions and comparing results. This detects SDC but doubles resource cost. It is typically reserved for certification-critical runs.
• Temporal redundancy — re-executing a time window and comparing against the previous result. Cheaper than spatial redundancy but only detects non-deterministic corruption.
• Algorithmic redundancy — using different numerical schemes or time-step sizes for cross-validation. Common in verification and validation (V&V) studies rather than production campaigns.
• Data redundancy — RAID, erasure coding, and replicated checkpoint storage. This protects against storage failures, not compute failures.

For most crash simulation campaigns, the practical combination is checkpoint/restart plus replicated checkpoint storage, with spatial redundancy reserved for final certification runs.

Availability Metrics

Availability quantifies the fraction of time a system or job is operational. The standard formula is:

Availability = MTBF / (MTBF + MTTR)

Where MTTR = Mean Time To Repair (or recover)
Example: MTBF = 195 h, MTTR = 2 h
Availability = 195 / (195 + 2) ≈ 0.9899 ≈ 98.99%

For crash simulation campaigns, two additional metrics are more directly useful:

• Effective utilization — the fraction of allocated core-hours spent on productive computation rather than checkpointing, restarting, or waiting for repairs. A campaign with 90% effective utilization loses 10% of its compute budget to fault tolerance overhead.
• Campaign completion rate — the fraction of submitted jobs that complete within their wall-clock limit without manual intervention. This is the operational metric that matters most to CAE managers.

Improving availability requires either increasing MTBF (better hardware, proactive monitoring) or decreasing MTTR (faster checkpoint reload, automated failure detection, pre-staged restart scripts). In practice, reducing MTTR through automation yields faster returns than hardware upgrades.

Putting It Together: A Campaign-Level Fault Tolerance Policy

The following policy synthesizes the concepts above into an actionable checklist for a crash simulation campaign. It is expressed as a decision table that maps job characteristics to fault tolerance measures.

Two verification steps should accompany any checkpoint/restart deployment. First, validate that a restarted job produces bit-identical or physically equivalent results to an uninterrupted run on a small test case. Second, measure the actual checkpoint write time and restart time under production I/O load, because these values directly determine the optimal interval and the effective utilization of the campaign.

Fault tolerance in crash simulation is ultimately an economic decision. Every checkpoint costs I/O bandwidth and storage; every restart costs compute time. The goal is not zero failures — that is unattainable at scale — but rather minimizing the total cost of failures while preserving result integrity. The metrics and mechanisms in this section provide the quantitative basis for that optimization.