3. Exploratory data analysis for crash data
Before a single tree is grown, the quality of a crash-injury model is largely determined by what the data actually look like. Crash datasets — typically assembled from police collision reports, hospital records, or state crash repositories such as FARS or NASS GES — carry three recurring pathologies that directly shape modeling decisions: skewed and heterogeneous feature distributions, structured missingness, and severe class imbalance in the injury severity target. Exploratory data analysis (EDA) is the stage where these pathologies are quantified rather than assumed, and where the modeling strategy for the rest of the pipeline is effectively chosen.
This section works through a concrete EDA pass on a representative crash table. Assume a dataset of roughly 120,000 vehicle-level crash records with a binary target serious_injury (1 = Killed or Suspected Serious Injury, KSI; 0 = otherwise), and predictors including speed_limit, vehicle_age, driver_age, impact_speed, seatbelt_used, road_type, weather, light_condition, and airbag_deployed. All numbers below are illustrative teaching values, not measured results.
Distribution analysis: what the predictors are telling you
Start with univariate summaries and histograms for every numeric field. Crash data is rarely Gaussian. speed_limit is multimodal because it clusters on posted values (25, 35, 45, 55, 65 mph), so a histogram will show discrete spikes rather than a smooth curve — treat it as ordinal-categorical for binning purposes. vehicle_age is right-skewed with a long tail of 20+ year-old vehicles; a log or square-root transform, or quantile binning, will stabilize it. impact_speed (often derived from delta-V or event data recorder estimates) is heavily right-skewed and frequently missing, which is itself a signal worth investigating.
For categorical fields, compute frequency tables and check for rare levels. weather may have a "fog/smoke" level with fewer than 200 records; road_type may include construction zones at under 1% prevalence. Rare levels cause unstable splits in tree ensembles and should be grouped (e.g., merge fog, smoke, and blowing snow into "reduced visibility") before modeling. Also inspect the target rate within each level — a cross-tab of seatbelt_used against serious_injury is one of the highest-value EDA outputs in crash work, because it exposes both the effect size and any coding inconsistencies (e.g., "unknown" coded as a separate level rather than missing).
Bivariate checks matter as much as univariate ones. Plot impact_speed against injury outcome using boxplots or binned rate curves; you should expect a monotonic rise in KSI probability with impact speed, and any non-monotonicity usually indicates a data problem (unit mixing, sentinel values like 999). Correlation among predictors — speed_limit with road_type, vehicle_age with airbag_deployed — is worth flagging early, since tree ensembles tolerate collinearity but feature importance becomes harder to interpret when predictors are redundant.
Missing value patterns: mechanism matters more than magnitude
Compute a missingness table per column and, critically, examine the pattern rather than just the count. Three mechanisms drive the treatment choice:
- MCAR (missing completely at random): missingness is unrelated to any variable. Rare in crash data but possible for administrative fields.
- MAR (missing at random): missingness depends on observed variables — e.g.,
impact_speedis missing more often for older vehicles that lack event data recorders. Conditional onvehicle_age, the missingness is ignorable. - MNAR (missing not at random): missingness depends on the unobserved value itself — e.g., severe crashes may have more missing
impact_speedbecause the vehicle was destroyed, or less because investigators prioritized them. This is the dangerous case.
A practical diagnostic is to create a binary indicator impact_speed_missing and cross-tabulate it against serious_injury. If the KSI rate differs materially between missing and non-missing rows (say 18% vs. 9%), the missingness is informative and the indicator itself is a legitimate feature. Tree-based models handle this gracefully: XGBoost learns a default direction for missing values during training, and scikit-learn's HistGradientBoostingClassifier and RandomForestClassifier (with imputation upstream) both accommodate it. The key EDA deliverable is a documented decision per column: drop, impute (and with what), or retain as-is with a missingness indicator.
Class imbalance assessment: quantify before you correct
KSI outcomes are typically a minority class. In a representative crash table, serious injuries might be 6–12% of vehicle records; fatalities alone are often under 1%. Compute the exact prevalence and then examine it by subgroup — imbalance is rarely uniform. Rural roads, higher speed limits, and older vehicle occupants may show KSI rates several times the overall base rate, which means a global resampling strategy can distort subgroup calibration.
Report the imbalance ratio and decide on a handling strategy before modeling. The options and their trade-offs:
| Strategy | Mechanism | When it helps | Watch out for |
|---|---|---|---|
| Class weights | Weight minority samples in the loss (e.g., scale_pos_weight in XGBoost) | Default first choice for tree ensembles; preserves all data | Over-weighting inflates false positives; tune via PR curve |
| Random undersampling | Drop majority-class rows | Very large datasets where majority is redundant | Discards information; unstable if minority is small |
| SMOTE / oversampling | Synthesize minority samples in feature space | Moderate imbalance with continuous features | Can create implausible crash records; risky with categorical fields |
| Threshold tuning | Keep data as-is; move the decision threshold | When ranking quality is good but default 0.5 cutoff is wrong | Requires a validation set and a cost rationale |
Because accuracy is meaningless under imbalance (a model predicting "no serious injury" for every row scores ~90% and is useless), EDA should establish the evaluation frame now: precision-recall curves, PR-AUC, and recall at a fixed precision are the relevant metrics. Record the base rate so that any later PR-AUC can be compared against the no-skill baseline, which equals the minority prevalence.
Turning EDA into a modeling contract
The output of this stage is not a set of plots but a short, written EDA report that the modeling stage must honor. It should record, per column, the observed distribution shape, the missingness percentage and mechanism judgment, and the chosen treatment. It should record the target base rate and the subgroup rates that matter for calibration. And it should fix the evaluation protocol: stratified train/validation/test split (stratify on serious_injury so each fold preserves the ~10% prevalence), PR-AUC as the primary metric, and the resampling or weighting strategy chosen before any model is fit.
A useful sanity check before moving on: fit a trivial baseline — logistic regression on a handful of clean predictors, or simply the base-rate predictor — and confirm that your chosen metric behaves as expected on the validation fold. If PR-AUC for the baseline lands near the minority prevalence, the metric and split are wired correctly. If it lands near 0.5 or near 1.0, something in the pipeline is leaking or mislabeled, and it is far cheaper to find that now than after tuning a random forest. EDA done well converts a vague "clean the data" step into a set of explicit, testable commitments that every subsequent model in this course will be measured against.