4. Labeling and ground-truth strategies for crash data

Crash analytics models — severity classifiers, near-miss detectors, injury-outcome predictors, and hotspot segmenters — are only as trustworthy as the labels they learn from. Unlike benchmark datasets with clean, curated annotations, crash data arrives as a noisy mixture of police reports, EMS narratives, telematics events, dashcam clips, and insurance claims. Each source has different reliability, latency, and granularity. This chapter focuses on how to build a labeling pipeline that is economically viable at scale while remaining defensible for safety-critical decisions: how to triage which records deserve scarce human attention, how to exploit weak supervision to bootstrap labels, and how to structure human-in-the-loop (HITL) annotation so that reviewer effort is concentrated where it changes model behavior.

Why crash labels are structurally hard

Three properties distinguish crash labeling from generic image annotation. First, class imbalance is extreme: fatal and serious-injury crashes may be under 2% of records, so a naive sampling strategy wastes reviewer time on easy negatives. Second, label ambiguity is inherent: a police report may code "injury" while EMS data codes "no transport," and the "correct" ground truth depends on the downstream task (property damage vs. injury severity vs. fault attribution). Third, temporal drift is real: reporting standards, vehicle fleets, and road infrastructure change, so a labeling guideline frozen in year one silently degrades. A robust strategy must therefore treat labels as versioned artifacts with provenance, not as immutable facts.

Triage workflows: routing records before humans see them

Triage is the discipline of deciding, per record, whether it needs human review, can be auto-labeled, or should be discarded. A practical triage layer combines deterministic rules with a lightweight model score. Consider a pipeline ingesting police crash records with structured fields (collision type, injury code, weather, road class) plus an optional narrative. A triage policy might look like this:

def triage(record, model_score, thresholds):
    # deterministic hard rules first
    if record.injury_code in ("FATAL", "SERIOUS"):
        return "PRIORITY_HUMAN"          # always reviewed
    if record.narrative is None and record.image_count == 0:
        return "AUTO_STRUCTURED"         # label from structured fields only
    # model-driven routing for the ambiguous middle
    if model_score >= thresholds.high:
        return "AUTO_POSITIVE"           # confident positive, spot-check later
    if model_score <= thresholds.low:
        return "AUTO_NEGATIVE"           # confident negative, audit sample
    return "HUMAN_REVIEW"                # uncertain band

The thresholds are not arbitrary. Set them by targeting a review budget: if reviewers can process 400 records per day and the daily inflow is 3,000, the uncertain band must be sized to roughly 400 records. Calibrate the model score (e.g., with isotonic regression) so that "high" and "low" correspond to empirically measured precision, not raw logits. A useful operating point is to choose thresholds.high such that the auto-positive bucket has ≥98% precision on a held-out audit sample, and thresholds.low such that auto-negatives have ≥99% recall of true positives excluded. These are example targets, not universal constants; they should be re-derived per task and re-audited quarterly.

Weak supervision: labeling functions as a scalable prior

Weak supervision replaces hand-labeling every record with a set of noisy labeling functions (LFs) — heuristics, regexes, external knowledge bases, or small models — whose votes are aggregated into probabilistic labels. For crash data, LFs are natural: a regex over narratives matching "rear-ended" implies a rear-end collision type; a rule that speed_limit >= 65 mph and weather == "rain" raises prior probability of a severe outcome; an LF derived from a pretrained dashcam detector flags "pedestrian in frame." Each LF has unknown accuracy and coverage, and their errors are correlated (two regexes over the same narrative field will agree on the same false positives).

Aggregate with a label model such as Snorkel's generative model, which estimates LF accuracies and correlations from unlabeled data alone, producing a probabilistic label per record. The output is not a final label but a prior that feeds triage: records with high-confidence weak labels go to the auto bucket; records where LFs disagree go to human review. This is where weak supervision earns its keep — it converts reviewer attention from "label everything" to "adjudicate disagreements."

A concrete example: suppose five LFs cover a collision-type task. LF1 (narrative regex "rear-end") fires on 40% of records; LF2 (structured field collision_type == "REAR_END") fires on 35%; LF3 (damage pattern heuristic) on 20%; LF4 (a small fine-tuned text classifier at 0.7 confidence) on 60%; LF5 (a rule flagging multi-vehicle + same direction) on 50%. The label model estimates per-LF accuracy and pairwise correlation, then emits per-record posteriors. Records where the posterior is between 0.4 and 0.6 — typically 10–15% of the corpus — are routed to humans. The remaining 85–90% receive weak labels, subject to stratified audit sampling.

Human-in-the-loop annotation: structure beats volume

HITL is not "send everything to a crowd." It is a designed interface where reviewers see the record, the model's current prediction, the weak-label posterior, and the specific disagreement that triggered review. Three design choices matter most:

• Adjudication over annotation. When LFs disagree, show the reviewer the conflicting signals side by side and ask for a decision plus a one-line rationale. Rationales become new LFs or guideline updates.
• Active selection. Prioritize records by expected information gain — e.g., high model uncertainty, rare class membership, or proximity to a decision boundary — rather than random sampling. A common heuristic is to rank by entropy of the model posterior combined with class-rarity weight.
• Inter-annotator agreement as a quality gate. Route a 5% overlap sample to two reviewers; if Cohen's κ on a task falls below a threshold (e.g., 0.65 for severity), the guideline is ambiguous and must be revised before more labels are collected.

Guideline versioning is non-negotiable. Every label carries guideline_version; when the guideline changes, affected labels are flagged for re-review rather than silently mixed. This is what makes labels auditable when a model's behavior is challenged after deployment.

Putting it together: a working labeling SOP

The following SOP is a compact, fillable template for a crash-analytics team standing up a labeling pipeline. It is deliberately short — the goal is a document reviewers and engineers can actually follow, not a compliance artifact.

Two failure modes recur in practice and are worth calling out explicitly. The first is silent weak-label leakage: a well-meaning engineer includes weak-labeled records in the evaluation set to "increase coverage," and the reported F1 jumps by several points without any real model improvement. The fix is structural — enforce split isolation in the data loader, not in a convention document. The second is guideline drift without versioning: reviewers gradually reinterpret an ambiguous class (e.g., "near-miss") until their labels no longer match the original definition, and the model learns the drift. The fix is to version the guideline, stamp every label, and re-review a stratified sample whenever the version changes.

Finally, treat the labeling pipeline as a model in its own right. It has inputs (records, LFs, guidelines), parameters (thresholds, LF weights, review budget), and outputs (labels with provenance). Monitor it with the same rigor: track auto-labeled share, audit disagreement rate, κ per class, and the fraction of labels whose guideline version is stale. When any of these degrades, the correct response is to revise the pipeline — not to retrain the downstream model on increasingly noisy labels.