4. Runtime error detection techniques

Static analysis can only reason about code paths it can see. Many of the most damaging crash classes — heap corruption, use-after-free, data races, stack overflow, and silent hangs — only manifest when a program actually executes with real inputs, real memory layouts, and real timing. Runtime error detection closes that gap by instrumenting the running process so that violations are caught at the moment they occur, ideally before they corrupt state beyond recovery. This chapter examines four complementary mechanisms: assertions, sanitizers, signal handlers, and watchdogs. Each operates at a different layer and catches a different failure family, so a mature crash-recovery pipeline typically uses all four together.

Assertions: Contract Enforcement at the Source Level

An assertion encodes an invariant that must hold at a specific program point. In C and C++, assert(expr) from <cassert> aborts the process when expr evaluates to false in debug builds, and compiles to nothing when NDEBUG is defined. This binary on/off behavior is the source of most misuse. A better pattern is a tiered assertion macro that distinguishes always-on contract checks from debug-only diagnostics:

#include <cstdio>
#include <cstdlib>

#define REQUIRE(cond, msg)                                            \
    do {                                                              \
        if (!(cond)) {                                                \
            std::fprintf(stderr, "REQUIRE failed: %s at %s:%d\n",     \
                         (msg), __FILE__, __LINE__);                  \
            std::abort();                                             \
        }                                                             \
    } while (0)

#ifdef DEBUG
#define CHECK(cond, msg) REQUIRE(cond, msg)
#else
#define CHECK(cond, msg) ((void)0)
#endif

REQUIRE stays active in release builds and guards preconditions whose violation would cause undefined behavior, such as a null pointer dereference or an out-of-range index. CHECK is compiled out in release and is used for expensive consistency checks. The key engineering discipline is to make assertions side-effect free: never write assert(consume(queue)), because the queue mutation disappears when NDEBUG is set. Assertions detect logic errors, not environmental errors — a failed assertion means the program's own assumptions are wrong, and the correct response is to abort with a diagnostic rather than attempt recovery.

Sanitizers: Compiler-Instrumented Memory and Concurrency Checking

Sanitizers are the highest-yield runtime detection tool for native code. They are enabled at compile time and inject checks around memory accesses, arithmetic, and synchronization operations. The four most relevant for crash detection are AddressSanitizer (ASan), UndefinedBehaviorSanitizer (UBSan), ThreadSanitizer (TSan), and MemorySanitizer (MSan).

A practical ASan invocation for a test binary looks like this:

clang++ -std=c++20 -g -O1 -fno-omit-frame-pointer \
        -fsanitize=address,undefined \
        -fno-sanitize-recover=all \
        -o app_asan app.cpp

ASAN_OPTIONS=abort_on_error=1:detect_leaks=1:log_path=asan.log \
UBSAN_OPTIONS=print_stacktrace=1:halt_on_error=1 \
./app_asan --input testcase_042.bin

When ASan detects a violation it prints a report containing the faulting address, the access type (read/write), the size, and a symbolized stack trace showing both the allocation site and the free site for use-after-free. Setting abort_on_error=1 converts the report into a SIGABRT, which lets a crash handler capture a core dump for offline analysis. The -fno-sanitize-recover=all flag is important: without it, UBSan merely logs and continues, which can mask the true first-error location.

Signal Handlers: Catching the Crash at the Boundary

When a process violates memory protection or executes an illegal instruction, the kernel delivers a signal: SIGSEGV, SIGBUS, SIGFPE, SIGILL, or SIGABRT. A signal handler can intercept these, record diagnostic state, and either terminate cleanly or hand off to a recovery supervisor. The critical constraint is that signal handlers run in an asynchronous context where only async-signal-safe functions may be called. malloc, printf, and most of the C++ standard library are not safe.

#include <csignal>
#include <cstring>
#include <unistd.h>

static int g_crash_fd = -1;

static void crash_handler(int sig, siginfo_t* info, void* uctx) {
    const char* name = (sig == SIGSEGV) ? "SIGSEGV" :
                       (sig == SIGABRT) ? "SIGABRT" : "SIGNAL";
    // write() is async-signal-safe; printf() is not.
    write(g_crash_fd, name, strlen(name));
    write(g_crash_fd, "\n", 1);
    // Re-raise with default disposition to produce a core dump.
    signal(sig, SIG_DFL);
    raise(sig);
}

void install_crash_handler(int fd) {
    g_crash_fd = fd;
    struct sigaction sa{};
    sa.sa_sigaction = crash_handler;
    sa.sa_flags = SA_SIGINFO | SA_RESETHAND;
    sigemptyset(&sa.sa_mask);
    sigaction(SIGSEGV, &sa, nullptr);
    sigaction(SIGABRT, &sa, nullptr);
    sigaction(SIGBUS,  &sa, nullptr);
}

The SA_RESETHAND flag restores the default handler before invoking ours, preventing infinite recursion if the handler itself faults. For richer diagnostics, the handler can capture the faulting address from info->si_addr and the instruction pointer from uctx, then write a compact binary record to a pre-opened file descriptor. Full stack unwinding should be deferred to a separate process that reads the core dump, because unwinding inside the handler is unsafe.

Watchdogs: Detecting Liveness Failures

Not all crashes are signals. A deadlock, an infinite loop, or a thread starved of CPU produces a process that is alive but not making progress. Watchdogs detect this class. The simplest form is a heartbeat counter that a worker thread increments and a monitor thread checks:

#include <atomic>
#include <chrono>
#include <thread>

std::atomic<uint64_t> g_heartbeat{0};

void worker() {
    while (running) {
        do_unit_of_work();
        g_heartbeat.fetch_add(1, std::memory_order_relaxed);
    }
}

void watchdog(std::chrono::seconds timeout) {
    uint64_t last = g_heartbeat.load();
    while (running) {
        std::this_thread::sleep_for(timeout);
        uint64_t now = g_heartbeat.load();
        if (now == last) {
            // No progress within the timeout window.
            log_stall_and_dump();
            std::abort();
        }
        last = now;
    }
}

For multi-threaded systems, a single global heartbeat is insufficient because one healthy thread can mask a stalled peer. Use a per-thread timestamp array and have the watchdog verify that every registered thread has advanced within its own deadline. Hardware watchdogs (the /dev/watchdog interface on Linux) extend the same idea to the whole system: the process must write to the device within the timeout or the kernel reboots the machine.

Combining the Layers in Practice

These four mechanisms are not alternatives; they are a layered defense. Assertions catch contract violations at the earliest possible point, before corruption spreads. Sanitizers catch memory and concurrency errors that assertions cannot express. Signal handlers catch the hardware faults that slip past both. Watchdogs catch the liveness failures that produce no signal at all. A production crash-recovery system should enable assertions and signal handlers in all builds, run sanitizers in CI and staging, and deploy watchdogs in every long-running service.

The practical integration point is a single crash-reporting subsystem. When any layer fires, it writes a structured record — fault type, timestamp, thread ID, faulting address, and a bounded stack trace — to a pre-allocated buffer or file descriptor. A separate supervisor process reads these records, classifies the fault, and decides whether to restart the worker, degrade functionality, or escalate. The feedback loop shown in the diagram is essential: every captured fault should produce a new assertion, a new sanitizer test case, or a tightened watchdog deadline, so the same failure cannot recur silently.