📚 Course ContentsChapter 2 Error taxonomy for crash solvers

2. Error taxonomy for crash solvers

A crash solver is a nonlinear finite element or multibody solver that has lost its equilibrium path and is being driven by an automatic recovery controller. Before any recovery policy can be selected, the failure must be classified. Misclassification is the single most common reason an automated recovery loop makes things worse: a memory fault "recovered" by bisecting the time step will simply fail again at the same location, while a convergence stall "recovered" by restarting the process throws away a perfectly valid tangent state. This chapter defines a four-family taxonomy — numerical failures, memory faults, convergence stalls, and hardware exceptions — and shows how to discriminate between them from solver telemetry.

Why a Taxonomy, Not a Severity List

It is tempting to rank errors by severity and escalate recovery actions accordingly. In practice, severity is a poor discriminator because the same symptom (a non-positive-definite stiffness matrix, for example) can originate in any of the four families. What the recovery controller actually needs is the causal family, because each family has a different set of admissible remedies and a different cost model. The taxonomy below is organized by root cause, not by symptom.

Family 1 — Numerical Failures

Numerical failures arise when the discretized problem is well-posed but the iterative solution process cannot resolve it at the current step size or tolerance. Typical members:

Divergence of the Newton–Raphson iteration — residual norm grows monotonically over iterations instead of decreasing.
Line-search failure — no step length satisfies the Armijo sufficient-decrease condition.
Ill-conditioning — condition number of the tangent stiffness exceeds a threshold (e.g. κ(K) > 1e12 in double precision).
Loss of positive definiteness — a negative eigenvalue appears in K, often at a limit or bifurcation point.
Singular Jacobian — pivot below eps during factorization.

These are the failures that respond to continuation remedies: load/time-step bisection, arc-length (Riks) continuation, line-search damping, or switching the nonlinear solver from full Newton to modified Newton with a relaxed tolerance. The diagnostic signature is a residual history that is finite and computable at every iteration — the solver is healthy, the step is simply too aggressive.

Family 2 — Memory Faults

Memory faults are defects in the solver's own data structures or in the interface between the solver and its host. They are not caused by the physics. Typical members:

Heap corruption — writing past the end of an element stiffness array; detected by guard pages or by a sanitizer.
Use-after-free — a material model holding a pointer to a deleted integration-point state.
Stack overflow — deep recursion in an adaptive mesh-refinement or contact-search routine.
Allocation failure — malloc/new returns null because the sparse factorization exceeded available RAM.
Out-of-bounds indexing — a DOF map referencing node n+1 in an n-node mesh.

The critical property of this family is non-determinism: the same input may succeed on one run and fail on the next, and the failure location may move. Remedies are structural — bounds checking, address sanitizers, memory pooling, or reducing the factorization footprint — never continuation. A recovery controller that bisects the time step in response to a heap-corruption signal is wasting cycles and may mask a real defect.

Family 3 — Convergence Stalls

A convergence stall is a non-failure in the strict sense: the solver is running, residuals are finite, but the iteration has stopped making progress. It is the most insidious family because it produces no exception and no error code — only a wall-clock budget that expires. Typical members:

Residual plateau — residual norm oscillates within a narrow band (e.g. 1e-4 to 3e-4) without reaching 1e-6.
Limit-cycle oscillation — the Newton correction alternates sign with nearly constant magnitude.
Stagnation at a contact or plasticity transition — the active set flips between two configurations each iteration.
Tolerance mismatch — the requested residual tolerance is below the achievable round-off floor for the current conditioning.

Stalls are detected by iteration-budget and progress-rate monitors rather than by exception handlers. The correct remedy is usually to change the iteration strategy (active-set stabilization, line search, or a looser tolerance with a subsequent tightening pass), not to restart from scratch.

Family 4 — Hardware Exceptions

Hardware exceptions originate below the solver: in the CPU, the memory subsystem, the accelerator, or the interconnect. Typical members:

Floating-point exceptions — invalid operation (0/0), divide-by-zero, or overflow raised as a trap.
GPU ECC errors — uncorrectable memory errors on a compute accelerator.
Thermal throttling — clock reduction that turns a 2-hour solve into a 6-hour solve and trips a watchdog.
PCIe/NVLink link errors — data corruption during host-to-device transfer.
Watchdog timeout — the OS or cluster scheduler kills the process.

These are the only failures for which restart on different hardware or checkpoint-and-resume is the primary remedy. They are also the only family where the failure is genuinely independent of the input deck.

Discriminating Between Families from Telemetry

The four families overlap in symptom space, so a single signal is never sufficient. The table below gives the minimum discriminating evidence for each family, using signals that a well-instrumented solver already emits.

Family	Primary signal	Discriminating evidence	Admissible remedy
Numerical failure	Residual norm grows or line search fails	Residual finite and monotone; `κ(K)` high; negative eigenvalue	Bisection, arc-length, damping
Memory fault	Signal (SIGSEGV/SIGABRT) or sanitizer report	Non-deterministic; address in report; allocation failure	Bounds check, sanitizer, reduce footprint
Convergence stall	Iteration budget exhausted, no exception	Residual plateau or limit cycle; active-set flip	Change iteration strategy, relax tolerance
Hardware exception	FP trap, ECC, watchdog	Reproducible on same hardware, not on another	Checkpoint/resume, migrate node

Worked Example: Classifying a Real Failure

Consider a transient nonlinear analysis of a rubber mount under a 10 mm displacement ramp, solved with an implicit Newmark scheme and full Newton. At step 47 the solver aborts. The telemetry shows:

step 47  t=0.470s  dt=1.0e-3
  iter 1  res=2.31e-2  kappa=4.7e11
  iter 2  res=8.94e-2  kappa=6.1e11
  iter 3  res=3.02e-1  kappa=9.8e11
  iter 4  res=1.11e+0  kappa=2.3e12
  line search: no step satisfies Armijo
  abort: NEWTON_DIVERGED

The residual is finite and grows monotonically, the condition number is high but not at the round-off floor, and the line search fails. This is a numerical failure, not a memory fault (no signal, deterministic) and not a stall (residual is not plateauing). The correct recovery is to bisect dt to 5.0e-4 and retry; if that fails, switch to arc-length continuation. Restarting the process would be wrong because the tangent state at step 46 is valid.

Now contrast with a second run where the solver aborts at step 12 with SIGSEGV and a sanitizer report pointing to element_stiffness.c:214. The residual history is clean up to the abort. This is a memory fault; bisecting dt will not help, and the correct action is to fix the out-of-bounds write and re-run.

Decision Flow for the Recovery Controller

The figure below summarizes how the four families are separated by the recovery controller before any remedy is chosen. It is a decision structure, not a linear process: the controller evaluates the exception channel first, then the residual history, then the iteration budget.

Practical Guidance for the Recovery Controller

Three rules follow directly from the taxonomy and are worth encoding as invariants in the recovery controller:

Classify before acting. The controller must evaluate the exception channel and the residual history before selecting any remedy. A remedy applied to the wrong family is worse than no remedy, because it consumes the step budget and may corrupt the tangent state.
Re-classify after every failed attempt. A numerical failure that survives bisection may have become a stall (the step is now so small that round-off dominates) or may reveal a memory fault that was previously masked. The family is not a permanent property of the step.
Keep the cost model explicit. Bisection is cheap and local; arc-length continuation is moderate; process restart is expensive; node migration is the most expensive. The controller should prefer the cheapest remedy admissible for the classified family, and escalate only when the cheaper remedy has demonstrably failed.

A useful instrumentation pattern is to log, at every Newton iteration, the tuple (res_norm, kappa, min_eigenvalue_sign, active_set_hash, wall_clock). The first three discriminate numerical failures from stalls; the fourth detects active-set flipping; the fifth catches thermal throttling and watchdog risk. This single log line is usually enough to classify a failure after the fact without re-running the job.

Do not treat a SIGSEGV as a numerical failure just because it occurred during a Newton iteration. The signal is raised by the memory subsystem, not by the residual computation. Bisecting the time step in response to a segmentation fault will produce a smaller step that faults at the same address, and the recovery loop will exhaust its budget without ever addressing the defect.

The taxonomy is the foundation for everything that follows in this course. Once a failure has been assigned to one of the four families, the recovery policy space collapses from "try anything" to a small, well-defined set of admissible actions, each with a known cost and a known success condition. The next chapters build the detection instrumentation and the recovery policies on top of this classification.