1. Crash Report Ingestion and Normalization Pipeline

Why a Normalization Layer Comes First

Crash analytics only becomes useful when every record can be compared on the same terms. In practice, the raw inputs arrive from three very different producers: minidumps written by the OS crash handler, stack traces emitted by application-level exception handlers, and text logs produced by the runtime or a watchdog. Each source carries its own field names, its own time base, and its own notion of what identifies a crash. A minidump knows the process ID and the module list but not the human-readable exception message; a stack trace knows the exception type and frames but not the memory layout; a log knows the wall-clock moment the watchdog noticed the failure but not the faulting address.

The ingestion and normalization pipeline is the component that reconciles these differences. Its job is narrow and testable: read each source, map its fields onto a single canonical schema, align all timestamps to one reference, and emit one normalized record per crash event. Everything downstream — signature hashing, frequency aggregation, regression detection — depends on this record being complete and consistent.

The Canonical Crash Record

Define the target schema before writing any parser. A workable canonical record for this pipeline has the following fields. Keep the schema flat where possible; nested structures complicate downstream aggregation.

Two design decisions matter here. First, event_time_utc is the time the crash occurred, not the time the pipeline saw it; ingest_time_utc is kept separately so that late or replayed artifacts do not distort trend analysis. Second, frames is allowed to be empty. A log-only event is still a valid crash record; downstream signature hashing must handle the empty case rather than the ingestion layer silently dropping it.

Timestamp Alignment

Timestamp alignment is the most common source of silent corruption in crash pipelines. Three conventions must be fixed explicitly and applied consistently:

• Reference frame. All times are stored as UTC. Local times from logs are converted using the offset recorded in the log header, not the pipeline host's timezone.
• Precision. Minidump header timestamps are typically second-resolution; stack trace and log lines may carry milliseconds. Truncate to the coarsest precision present in the batch, or store full precision and document that sub-second ordering is only reliable within a single source type.
• Clock skew. Device clocks drift. If a log line and a minidump describe the same crash, prefer the minidump header time as authoritative and record the delta in a diagnostic field rather than overwriting either value.

A practical rule: never compare timestamps across sources at finer granularity than the coarsest source in the comparison. If you need sub-second ordering, restrict the comparison to records from the same source_type.

Field Mapping in Practice

The mapping layer is best implemented as a small set of per-source adapters that each return the canonical record. The following Python sketch shows the structure; it is self-contained and runnable against the sample inputs described below.

import json, uuid, re
from datetime import datetime, timezone

def parse_iso(s):
    return datetime.fromisoformat(s.replace("Z", "+00:00")).astimezone(timezone.utc)

def from_minidump(dump):
    # dump: dict already extracted from the minidump header/streams
    return {
        "crash_id": str(uuid.uuid4()),
        "event_time_utc": parse_iso(dump["header_time"]).isoformat(),
        "ingest_time_utc": datetime.now(timezone.utc).isoformat(),
        "source_type": "minidump",
        "app_version": dump.get("module_version", "unknown"),
        "os_version": dump.get("os_version", "unknown"),
        "device_id": dump.get("device_id", "unknown"),
        "exception_type": map_exception_code(dump.get("exception_code")),
        "faulting_module": dump.get("faulting_module", ""),
        "frames": dump.get("frames", []),
        "raw_ref": dump["path"],
    }

def from_stacktrace(trace):
    return {
        "crash_id": str(uuid.uuid4()),
        "event_time_utc": parse_iso(trace["timestamp"]).isoformat(),
        "ingest_time_utc": datetime.now(timezone.utc).isoformat(),
        "source_type": "stacktrace",
        "app_version": trace.get("app_version", "unknown"),
        "os_version": trace.get("os_version", "unknown"),
        "device_id": trace.get("device_id", "unknown"),
        "exception_type": trace.get("exception_class", "unknown"),
        "faulting_module": trace["frames"][0]["module"] if trace.get("frames") else "",
        "frames": trace.get("frames", []),
        "raw_ref": trace["path"],
    }

def from_log(line):
    # line: dict parsed from a structured log entry
    return {
        "crash_id": str(uuid.uuid4()),
        "event_time_utc": parse_iso(line["ts"]).isoformat(),
        "ingest_time_utc": datetime.now(timezone.utc).isoformat(),
        "source_type": "log",
        "app_version": line.get("app_version", "unknown"),
        "os_version": line.get("os_version", "unknown"),
        "device_id": line.get("device_id", "unknown"),
        "exception_type": line.get("keyword", "unknown"),
        "faulting_module": "",
        "frames": [],
        "raw_ref": line["path"],
    }

EXCEPTION_MAP = {0xC0000005: "ACCESS_VIOLATION", 0xC000001D: "ILLEGAL_INSTRUCTION"}
def map_exception_code(code):
    return EXCEPTION_MAP.get(code, f"OS_CODE_{code:08X}" if code else "unknown")

Each adapter is deliberately small. The value of the pattern is that adding a fourth source later means writing one more adapter, not touching the schema or the downstream consumers.

Pipeline Topology and Validation

The diagram below shows how the three source adapters converge on the canonical record, where timestamp alignment and field mapping sit relative to each other, and which validation gates a record must pass before it is emitted.

Reproducible Walkthrough

To verify the pipeline end to end, use three small sample artifacts that describe the same crash from different angles. The expected outcome is three canonical records that agree on event_time_utc (within the coarsest precision), app_version, and exception_type, while differing in frames and faulting_module as expected.

1. Prepare inputs. A minidump header with header_time = "2024-03-11T14:22:07Z", exception_code = 0xC0000005, module_version = "3.4.1", os_version = "10.0.19045". A stack trace with timestamp = "2024-03-11T14:22:07.412Z", exception_class = "ACCESS_VIOLATION", and two frames. A log line with ts = "2024-03-11T14:22:08Z" and keyword = "watchdog_timeout".
2. Run each adapter and inspect the emitted records. Confirm that all three carry source_type matching their origin and that event_time_utc is expressed in UTC with a trailing +00:00.
3. Check alignment. The minidump and stack trace should agree to the second; the log line is one second later because the watchdog fires after the fault. Record this delta rather than forcing equality.
4. Apply the validation gate. Deliberately remove raw_ref from one record and confirm it is rejected with a clear reason. Deliberately set frames = [] on the log record and confirm it is accepted.

Failure Symptoms and Troubleshooting

• All records show the same event_time_utc. The adapter is reading the pipeline clock instead of the source timestamp. Check that parse_iso receives the source field, not datetime.now().
• Timestamps drift by whole hours. A local-time log was parsed without applying its recorded offset. Confirm the offset is read from the log header and applied before conversion to UTC.
• exception_type is unknown for minidumps. The exception code was not mapped. Extend EXCEPTION_MAP or fall back to the OS_CODE_XXXXXXXX form so the value is at least stable and searchable.
• Log-only events disappear. A downstream filter is dropping records with empty frames. The ingestion layer is correct; fix the consumer, not the adapter.

Once the pipeline emits a stable canonical record for every source, the next analytical step — deriving comparable crash signatures from the frames array — becomes a well-defined problem on clean input rather than a parsing exercise tangled with source-specific quirks.