1. Load and Audit the Occupant Injury Dataset

Why the audit comes first

Every Random Forest and XGBoost experiment in this course inherits its assumptions from the raw occupant injury table. If the schema is misread, a missingness pattern is ignored, or the target distribution is misjudged, later metrics become misleading rather than merely noisy. This first case therefore does not train a model. It loads the raw dataset, profiles its structure, quantifies missingness, and characterizes the injury-severity target so that every downstream decision — label construction, leakage exclusion, stratified splitting, class weighting — rests on a documented baseline.

The workflow below assumes a tabular crash dataset in CSV form, one row per occupant involved in a crash, with a severity column such as INJ_SEVERITY and a mix of crash, vehicle, occupant, and restraint fields. Column names vary by source (for example NASS-CDS, CISS, or a synthetic teaching extract); the audit code is written to discover columns rather than hard-code them, so it stays valid when the schema differs.

Loading the raw table without silent coercion

The most common early defect is implicit type inference: pandas will happily read a coded severity field as int64, turn a sentinel like 99 or -1 into a plausible-looking number, and convert blank strings to NaN without telling you which columns were affected. Load with explicit low-memory settings and then inspect dtypes before trusting anything.

import pandas as pd
import numpy as np

RAW_PATH = "data/occupant_injury_raw.csv"

df = pd.read_csv(
    RAW_PATH,
    na_values=["", "NA", "N/A", "NULL", "Unknown", "UNK"],
    keep_default_na=True,
    low_memory=False,
)

print("shape:", df.shape)
print(df.dtypes.value_counts())
print(df.head(3).T)

Record the row and column counts immediately. A row count that differs from the source system's documented total usually means a quoting or delimiter problem, not a data change. If the file is large, pass usecols only after the first full pass — dropping columns before profiling hides missingness you may need to report.

Profiling schema, cardinality, and missingness

Build a single audit frame that captures, per column: dtype, non-null count, missing fraction, number of distinct values, and a few example values. This one table drives almost every later decision, including which fields are candidates for the target, which are identifiers to drop, and which are too sparse to model.

def audit_columns(frame):
    rows = []
    n = len(frame)
    for col in frame.columns:
        s = frame[col]
        rows.append({
            "column": col,
            "dtype": str(s.dtype),
            "non_null": int(s.notna().sum()),
            "missing_pct": round(100 * s.isna().mean(), 2),
            "n_unique": int(s.nunique(dropna=True)),
            "example": s.dropna().iloc[0] if s.notna().any() else None,
        })
    return pd.DataFrame(rows).sort_values("missing_pct", ascending=False)

audit = audit_columns(df)
print(audit.to_string(index=False))

Read the output in three passes. First, flag any column above roughly 40% missing — these are usually optional detail fields and should be excluded or imputed with an explicit indicator rather than silently dropped. Second, flag columns with n_unique == 1; constant columns carry no signal and often reveal a filtered extract. Third, flag identifier-like columns (high cardinality, string dtype, names containing ID, CASE, VEHICLE) so they are excluded from features later.

Characterizing the injury-severity target

The target distribution determines whether stratification, class weighting, or resampling will be needed. Compute the raw frequency and percentage of each severity code, and check for sentinel values that are not real severities.

TARGET = "INJ_SEVERITY"

target_counts = (
    df[TARGET]
    .value_counts(dropna=False)
    .rename_axis("severity_code")
    .reset_index(name="count")
)
target_counts["pct"] = (100 * target_counts["count"] / len(df)).round(2)
print(target_counts.to_string(index=False))

# Sentinel check: codes outside the documented severity range
documented = {0, 1, 2, 3, 4, 5}   # adjust to the codebook
unexpected = sorted(set(df[TARGET].dropna().unique()) - documented)
print("unexpected codes:", unexpected)

Two outcomes matter here. If the severe-injury class (typically code 3 and above, or a fatal flag) is a small minority — often under 5% in real crash data — the audit must state the exact proportion so the split in the next case can be verified against it. If unexpected codes appear, they are almost always missing-value sentinels (e.g., 9, 99) that must be mapped to NaN before any label is built, otherwise they will be treated as an extreme severity class.

Producing the audit report

The deliverable is a compact, reproducible report — not a narrative. Emit it as a structured object so later cases can assert against it. The example below writes a JSON summary plus a per-column CSV; both are small enough to commit alongside the notebook.

import json, pathlib

report = {
    "source": RAW_PATH,
    "n_rows": int(len(df)),
    "n_columns": int(df.shape[1]),
    "target_column": TARGET,
    "target_counts": target_counts.to_dict(orient="records"),
    "unexpected_target_codes": unexpected,
    "columns_over_40pct_missing": audit.loc[audit.missing_pct > 40, "column"].tolist(),
    "constant_columns": audit.loc[audit.n_unique <= 1, "column"].tolist(),
    "high_cardinality_columns": audit.loc[audit.n_unique > 0.5 * len(df), "column"].tolist(),
}

out = pathlib.Path("reports"); out.mkdir(exist_ok=True)
(out / "audit_summary.json").write_text(json.dumps(report, indent=2))
audit.to_csv(out / "column_audit.csv", index=False)
print(json.dumps(report, indent=2))

Verification and failure symptoms

Confirm the audit before moving on. The row count must match the source extract; the sum of target_counts["count"] must equal len(df) (including the NaN bucket); and the severe-class percentage must be recorded explicitly. Typical failure symptoms and their causes:

With the report committed, the modeling baseline is fixed: known row count, known schema, quantified missingness, and an explicit target distribution. The next case builds the severity label and the leakage-safe feature set on top of exactly these numbers.