3. Crash data acquisition and preprocessing
Crash analytics is only as trustworthy as the pipeline that feeds it. Before any model can rank crash signatures, cluster stack traces, or predict regression risk, the raw telemetry emitted by applications, operating systems, and crash reporters must be collected, normalized, symbolized, and converted into stable numerical features. This chapter walks through that pipeline end to end: how crash events are captured, how stack traces are parsed into structured frames, how addresses are resolved to function names through symbolication, and how the resulting records are transformed into features suitable for downstream ML. The emphasis is on the concrete engineering decisions that determine whether the data reaching your models is faithful to what actually happened in production.
Log collection: from event to durable record
Crash data typically arrives through one of three channels: an in-process crash handler (for example, a signal handler on POSIX systems or a vectored exception handler on Windows) that writes a minidump or a structured crash report; an out-of-process watchdog that detects abnormal termination and captures context; or an application-level error reporter that serializes exceptions and breadcrumbs. Each channel produces a different payload shape, and the first design decision is to define a canonical envelope that all sources map into. A practical envelope includes a stable crash_id, app_version, build_id, platform, arch, timestamp, process_uptime_ms, signal_or_exception, thread_id, the raw stack memory or minidump reference, and a free-form attributes map for device and session metadata.
Collection must be resilient. Crash handlers run in degraded conditions—heap corruption, exhausted file descriptors, or a dying process—so the handler should write to a pre-opened file descriptor or a memory-mapped region rather than allocating. Reports are then uploaded asynchronously by a separate process on next launch. At the ingestion boundary, apply idempotency using crash_id so retries do not duplicate records, and enforce a schema version so that older clients remain parseable as the envelope evolves. A common failure symptom at this stage is truncated payloads: if minidumps are consistently smaller than expected or stack sections are empty, suspect the handler writing after the process has already lost address-space access, and move capture earlier in the teardown sequence.
Stack trace parsing
A raw stack trace is a sequence of program counters and, where available, frame pointers or unwind information. Parsing converts this into an ordered list of frames with module, offset, and—critically—a distinction between trusted and untrusted frames. On platforms with frame pointers, walking is straightforward but fragile under optimization; on platforms relying on unwind tables (DWARF CFI, .eh_frame, or Windows .pdata), the unwinder consults metadata emitted by the compiler. The parser should record, per frame, the module base address, the relative virtual address (RVA), and the frame index, and it should flag frames that could not be unwound rather than silently dropping them.
Two parsing hazards deserve explicit handling. First, inlined functions: a single physical frame may correspond to several logical frames, and modern symbolizers expand these using DWARF DW_TAG_inlined_subroutine or equivalent debug info. If your parser collapses inlined frames, crash grouping will merge distinct code paths. Second, tail calls and signal trampolines introduce frames that belong to runtime machinery rather than application logic; mark them with a frame_kind of runtime so feature extraction can exclude them. A useful validation check is to assert that the topmost application frame's module matches the crashing module reported by the OS; a mismatch usually indicates a corrupted unwind or a mismatched binary.
Symbolication
Symbolication maps (module, rva) pairs to human-meaningful symbols: function name, source file, line number, and inlining chain. The essential prerequisite is build-id matching. Every build must emit a unique identifier—GNU build-id on ELF, a UUID in the Mach-O LC_UUID load command, or a PDB GUID plus age on Windows—and that identifier must be stored alongside the crash record. Symbol servers are then keyed by this identifier, not by version string, because version strings are reused across rebuilds and will silently produce wrong symbols.
For a reproducible example, consider an ELF binary where the crash report contains module=librender.so, rva=0x1a3f0. The symbolication service looks up the build-id recorded in the report, fetches the matching .debug file, and resolves the RVA to render::Compositor::flush(Frame&) at compositor.cc:214. If the debug file is missing, the service should return a symbolication_status of missing_debug and preserve the raw RVA rather than fabricating a symbol. Downstream models must be able to distinguish symbolized from unsymbolized frames; treating an unresolved address as a stable feature will fragment clusters. Cache symbolication results keyed by (build_id, rva), since the same addresses recur across thousands of reports and symbolization is the dominant cost in most pipelines.
Feature extraction
Feature extraction turns a symbolized crash into a vector that downstream models can consume. The most informative features are structural rather than textual. A robust baseline includes: a hash of the top N application frames (with N typically 3–5, excluding runtime frames), the exception or signal type, the crashing module, the stack depth, the count of distinct modules in the trace, process uptime at crash, thread role (main vs. worker), and the symbolized ratio. Frame hashes should be computed over normalized symbols—strip template arguments, parameter lists, and addresses—so that render::Compositor::flush(Frame&) and render::Compositor::flush(Frame const&) collapse to the same token.
Two feature-engineering pitfalls recur in practice. First, version leakage: including app_version as a raw categorical feature lets a model memorize per-release crash patterns and fail on new builds. Prefer build-independent features and treat version as a grouping key for evaluation. Second, unstable ordering: if frames are hashed in raw order, a single extra runtime frame shifts every hash. Normalize by filtering runtime frames before hashing and by capping depth. For text-like signals such as exception messages, apply tokenization and hashing with a fixed vocabulary size (for example, 214 buckets) to keep dimensionality bounded and avoid unbounded cardinality from interpolated strings.
Validation and failure modes
Before features reach training, run a small set of pipeline assertions. Check that every record has a resolvable build_id; records without one should be quarantined rather than symbolized against a guessed binary. Verify that the symbolized ratio exceeds a threshold (for example, 0.9) for the release under analysis; a sudden drop usually means debug artifacts were not published for a build. Confirm that the top frame's module matches the OS-reported crashing module. Finally, recompute features from stored raw traces on a sample and compare against the production feature store—divergence indicates a normalization change that was not versioned.
Typical symptoms map to specific causes. Empty stack sections point to capture timing or unwind metadata problems. Symbols resolving to ?? or to functions in the wrong module indicate build-id collisions or stale symbol caches. Feature vectors that change for identical crashes across pipeline runs indicate non-deterministic normalization, often from unordered module iteration or locale-dependent string handling. Treating these as first-class observability signals—emitting counts of unsymbolized frames, missing build-ids, and feature-recompute mismatches—turns preprocessing from a silent stage into a monitored one, which is exactly what an MLOps pipeline for crash analytics requires.