3. Crash Frequency and Trend Analysis

Why frequency alone is misleading

Crash frequency analysis answers a deceptively simple question: how many crashes happened, and is that number changing? In practice, the raw count is almost never the metric you want. A build that ships to 10,000 users and produces 500 crashes is healthier than a build that ships to 1,000 users and produces 200. A spike on a Tuesday may be a real regression or simply the day a large enterprise tenant upgraded. And a "trend" that looks like improvement may be an artifact of a release boundary that reset the denominator mid-window.

This case builds a small, self-contained time-series aggregation layer that produces three families of metrics — daily/weekly crash counts, unique users affected, and crash-free session ratio — and then applies spike and trend detection that is explicitly aware of release boundaries and sampling bias. The goal is a reproducible analysis you can run against a normalized crash table and trust enough to act on.

Input contract and the aggregation grain

Assume the ingestion pipeline from the previous stage has produced a normalized table with one row per deduplicated crash event. The columns we depend on are:

• crash_id — stable dedup key (string).
• occurred_at — UTC timestamp of the crash (ISO-8601).
• user_id — anonymized install/user identifier (string).
• app_version — semantic version string, e.g. 4.12.0.
• release_channel — stable, beta, canary.
• session_id — session in which the crash occurred (string).

Two denominators matter and must be joined in from a separate sessions_daily table (one row per day per version per channel with sessions and active_users): crash counts are meaningless without the exposure that produced them. If you only have crash data, you can still compute counts and unique users, but you cannot compute rates — and rates are what make cross-release comparison valid.

Computing the metrics

The aggregation is a single grouped pass. Daily grain is the base; weekly is a rollup of daily. Unique users must be counted with a set-cardinality operation, not by summing daily uniques — a user crashing on Monday and Wednesday is one affected user for the week, not two.

import pandas as pd

crashes = pd.read_parquet("normalized_crashes.parquet")
sessions = pd.read_parquet("sessions_daily.parquet")

crashes["day"] = pd.to_datetime(crashes["occurred_at"], utc=True).dt.floor("D")

# Daily counts + unique users, per version and channel
daily = (
    crashes.groupby(["day", "app_version", "release_channel"])
    .agg(
        crash_count=("crash_id", "nunique"),
        unique_users=("user_id", "nunique"),
        unique_sessions=("session_id", "nunique"),
    )
    .reset_index()
)

# Join exposure and compute rates
daily = daily.merge(
    sessions, on=["day", "app_version", "release_channel"], how="left"
)
daily["crashes_per_1k_sessions"] = 1000 * daily["crash_count"] / daily["sessions"]
daily["affected_user_ratio"] = daily["unique_users"] / daily["active_users"]

# Weekly rollup: sum counts, but re-count uniques from raw events
week = crashes.assign(
    week=pd.to_datetime(crashes["occurred_at"], utc=True).dt.to_period("W")
).groupby(["week", "app_version", "release_channel"]).agg(
    crash_count=("crash_id", "nunique"),
    unique_users=("user_id", "nunique"),
).reset_index()

Note the deliberate asymmetry: weekly crash_count is a sum of daily counts (safe, because crash_id is unique), but weekly unique_users is recomputed from raw events. Summing daily uniques would overcount returning users and inflate the impact metric — a classic aggregation bug that makes a stable week look like a spike.

Release boundaries as first-class structure

A version change is a discontinuity in the data-generating process: new code, new crash signatures, often a new sampling rate and a different user cohort. Treating a release boundary as just another day in a rolling average smears the effect of the new build across the old one. The correct approach is to segment the series at each release boundary and analyze within segments, then compare segment-level summaries across the boundary.

The figure shows the intended reading: each release segment gets its own baseline, the spike is flagged relative to the v4.12.0 baseline (not a global mean that would be dragged down by the older, lower-rate build), and the boundary is drawn explicitly so no rolling window straddles it.

Spike detection that respects segments

A robust, cheap detector is a per-segment robust z-score on the daily rate. Using median and MAD instead of mean and standard deviation prevents a single large spike from inflating the threshold and hiding itself.

import numpy as np

def robust_z(series):
    med = series.median()
    mad = (series - med).abs().median()
    scale = 1.4826 * mad if mad > 0 else series.std(ddof=0)
    return (series - med) / scale

# Segment by version so windows never cross a release boundary
daily = daily.sort_values("day")
daily["segment"] = daily["app_version"]
daily["z"] = daily.groupby("segment")["crashes_per_1k_sessions"].transform(robust_z)
daily["is_spike"] = daily["z"] > 3.5

spikes = daily[daily["is_spike"]][
    ["day", "app_version", "crashes_per_1k_sessions", "z"]
]
print(spikes)

Two guardrails keep this honest. First, require a minimum exposure (e.g. sessions >= 500) before a day is eligible for spike status — a low-traffic canary day can produce a huge rate from three crashes. Second, require the spike to persist for at least two consecutive days, or corroborate it with a matching jump in unique_users; single-day blips are frequently ingestion backfills rather than real regressions.

Trend estimation and the sampling-bias check

For trend, fit a simple least-squares slope on the daily rate within each segment and report it with a confidence interval. A slope near zero with a wide interval means "no detectable trend," which is a valid and useful conclusion. Before trusting any cross-release comparison, run the sampling-bias check: confirm sample_rate is constant within each segment, and if not, rescale counts by 1 / sample_rate and recompute. A release that halved its sampling rate will show an apparent 50% drop in crashes that is entirely an artifact.

Verification checklist

With these metrics and guardrails in place, the output of this stage is a per-segment time series with flagged spikes and a slope estimate — a clean, defensible input for the severity scoring and triage work that follows.