2. Failure taxonomy in automotive simulation

Automotive simulation programs rarely fail for a single reason. A crash-solver run that aborts after six hours may have started with a distorted element, exhausted memory while rebuilding the contact tree, and finally lost its license lease when the job restarted. Treating such an event as "the solver crashed" hides the actual causal chain and makes recurrence almost certain. A failure taxonomy gives engineering teams a shared vocabulary so that symptoms, root causes, and recovery actions can be classified consistently across CAE, CFD, NVH, and multi-body workflows.

The taxonomy below groups failures by the layer at which they originate: numerical, geometric, resource, entitlement, physical infrastructure, and data integrity. The layers are not mutually exclusive — a hardware fault can corrupt a restart file, which then triggers solver divergence on the next run — but classifying the first observable failure helps teams route the incident to the right owner.

Numerical failures: solver divergence and non-convergence

Solver divergence is the most common failure class in implicit structural and CFD analysis. The residual norm grows instead of decaying, or the Newton–Raphson loop stalls above the convergence tolerance. Typical triggers include rigid body modes from insufficient constraints, contact chattering, highly nonlinear material models loaded beyond their calibration range, and time steps too large for the local dynamics.

Diagnosis starts with the residual and increment logs. In Abaqus/Standard, a diverging increment shows ***ERROR: TOO MANY ATTEMPTS MADE FOR THIS INCREMENT after five cutbacks. In ANSYS Mechanical, the Newton-Raphson residual plot typically shows the force residual rising across iterations rather than falling below the criterion. A practical first response is to reduce the initial increment, enable automatic stabilization, or switch the contact formulation from pure Lagrange to augmented Lagrange. For explicit codes such as LS-DYNA, the analogous symptom is a negative volume or an energy ratio (GLSTAT hourglass versus internal energy) exceeding roughly 5–10%, which usually points to a distorted element rather than a time-step problem.

Geometric failures: mesh failure

Mesh failure covers both mesh generation failures and mesh-induced solver failures. Generation failures include unmeshed faces, free edges, duplicate nodes, and collapsed elements with zero or negative Jacobian. Solver-induced mesh failures appear as negative volume elements, excessive aspect ratios, or warped shell elements that violate the code's quality thresholds.

A useful triage sequence is: check the mesh quality report first, then the element diagnostics in the solver output. For a typical crash model, warn on Jacobian below 0.3, aspect ratio above 5 for shells, and warpage above 15°. If a single element is responsible, local remeshing is faster than global refinement. If thousands of elements fail simultaneously, the CAD geometry or the meshing strategy is the real problem.

Resource failures: memory exhaustion

Memory exhaustion occurs when the solver's peak resident set exceeds the available RAM, forcing swapping or an out-of-memory kill. Symptoms include Out of memory in the solver log, sudden slowdowns, or the process being terminated by the OS OOM killer. Common causes are an oversized sparse matrix from a fine mesh, an excessive number of contact pairs, or a direct solver chosen where an iterative solver would suffice.

Mitigation options include switching from a direct sparse solver to an iterative one (e.g., PCG), reducing the number of retained load steps, or increasing the number of MPI ranks so that the matrix is distributed. On Linux clusters, checking /proc/<pid>/status for VmPeak and VmRSS during a run gives an early warning before the kill.

Entitlement failures: license loss

License loss is an infrastructure failure, not a numerical one. It appears as License server does not respond, Feature not available, or a job that starts and then aborts after the license heartbeat times out. Causes include license server downtime, exhausted license pool, firewall changes blocking TCP port 27000–27009, or a VPN drop during a long run.

For long jobs, enable license queuing and heartbeat retry where the vendor supports it. Record the license feature name and version in the job script so that a restart can request the same entitlement. If the pool is exhausted, the correct action is to queue rather than to retry aggressively, which can trigger a license-server ban.

Hardware faults

Hardware faults are the hardest class to diagnose because they are intermittent and often masquerade as software bugs. Typical manifestations include ECC memory errors reported in dmesg, GPU XID errors during CUDA-accelerated runs, thermal throttling that changes runtimes, and disk I/O errors that corrupt checkpoint files. A node that fails only on large jobs but passes small ones is a classic sign of a marginal DIMM or a failing PSU under load.

Standard practice is to correlate the failure timestamp with BMC/IPMI logs, run memtester or vendor diagnostics on suspect nodes, and quarantine the node until it passes. For GPU runs, check nvidia-smi -q for retired pages and XID codes; XID 48 and 63 typically indicate hardware faults rather than application bugs.

Data corruption

Data corruption can occur in input decks, restart files, result databases, or license tokens. Symptoms include checksum mismatches, unreadable .odb or .rst files, and results that are numerically plausible but physically wrong. Corruption often originates from a hardware fault, an interrupted write, or a filesystem that was not flushed before the job was killed.

Defenses include writing restart files atomically (write to a temporary file, then rename), verifying checksums after transfer, and keeping at least two generations of restart data. When corruption is suspected, re-run a short verification case on the same node to distinguish a bad file from a bad machine.

Using the taxonomy in practice

When a job fails, the first step is to capture the exact failure signature: the last 50 lines of the solver log, the exit code, the node name, and the wall-clock time. The second step is to map that signature to one of the six classes above. The third step is to apply the class-specific recovery action and record the outcome in the incident log.

A short incident record is often enough to prevent recurrence. The following template captures the minimum fields needed for trend analysis across a project:

With the taxonomy in place, the next chapter examines how each failure class maps to specific detection signals and automated monitoring hooks, so that recovery can begin before a human notices the job has stalled.