1. Crash solver fundamentals

A crash solver is not a single program but a coordinated pipeline that detects abnormal termination, captures the state of a failing process, classifies the failure, and drives a recovery action. Before you can automate recovery, you must understand what each stage produces and which signals are trustworthy. This chapter establishes the vocabulary and mental model used throughout the rest of the course.

Solver Architecture

A typical crash solver consists of five cooperating components. The supervisor owns the process lifecycle: it spawns the target application, monitors its exit status, and decides whether a restart is warranted. The signal handler (or exception filter on Windows) runs inside the crashing process and captures the fault before the OS tears the process down. The dump collector writes a core dump, minidump, or structured crash record to persistent storage. The classifier parses that record, extracts a signature, and maps it to a known failure category. The recovery controller applies a policy — restart, roll back, isolate, or escalate — and records the outcome.

These components communicate through a well-defined contract. The supervisor must know the exit code convention; the signal handler must be async-signal-safe; the dump collector must guarantee atomic writes so a second crash during collection does not corrupt the first record. A common mistake is to let the classifier run inside the crashing process — if the heap is corrupted, classification itself will fault. Keep classification and recovery in a separate, healthy process.

Crash Types

Crashes fall into four broad categories, each with distinct detection and recovery implications.

• Memory faults — segmentation violations (SIGSEGV), bus errors (SIGBUS), and access violations. These are the most common and usually indicate null-pointer dereference, buffer overflow, or use-after-free. The fault address and instruction pointer are the primary diagnostic signals.
• Logic faults — assertion failures, unhandled exceptions, and divide-by-zero. These are deterministic and often reproducible; the stack trace alone usually identifies the root cause.
• Resource exhaustion — out-of-memory (OOM killer), file-descriptor exhaustion, and stack overflow. These may not produce a traditional core dump; the supervisor must detect them from exit codes or system logs.
• External faults — hardware errors, driver bugs, and watchdog timeouts. These are the hardest to classify because the crash record may be incomplete or misleading.

The distinction matters because recovery policy differs. A memory fault from a known bug may warrant a simple restart with a backoff. A resource-exhaustion crash requires freeing resources or scaling before restart, otherwise the restarted process will crash again immediately.

Error Signals

Every crash leaves a trail of signals. The most valuable are:

Not all signals are equally reliable. A SIGKILL (9) cannot be caught, so no in-process handler runs; the supervisor must infer the cause from system logs. A SIGSEGV inside a signal handler produces a second fault that may overwrite the first dump. Always validate that the dump you are analyzing is the original fault, not a secondary one.

Recovery Objectives

Recovery is not simply "restart the process." A well-defined recovery objective specifies four things: what to restore (process, service, data), how fast (recovery time objective, RTO), how much data loss is acceptable (recovery point objective, RPO), and what to do if recovery fails (escalation path).

For a stateless worker, the objective might be: restart within 5 seconds, no data loss, escalate after 3 consecutive failures. For a stateful service, the objective must include checkpoint restoration and may require draining in-flight requests before restart. The recovery controller encodes these objectives as policy rules.

Putting It Together: A Minimal Example

Consider a worker process that crashes with SIGSEGV. The supervisor observes exit code 139. The signal handler has already written a minidump to /var/crash/worker-20250101-120000.dmp. The classifier parses the dump, extracts the fault address 0x00000000 and the top frame process_item+0x42, and maps this to the category NULL_DEREF. The recovery controller checks the policy: NULL_DEREF → restart with 1-second backoff, max 3 attempts. It restarts the worker, which succeeds on the second attempt. The controller logs the outcome and resets the failure counter after 60 seconds of clean uptime.

This example illustrates the full loop: detection, capture, classification, policy lookup, action, and verification. Every subsequent chapter in this course drills into one of these stages, showing how to implement it robustly and how to test it under realistic failure conditions.