4. Drift detection fundamentals

A deployed model does not fail the moment the world changes; it fails when the change becomes large enough to move inputs or labels outside the region where the model was validated. Drift detection is the discipline of turning that vague statement into a measurable signal. The goal is not to prove that data has changed — data always changes — but to decide, with controlled false-alarm behavior, whether the change is large enough to justify action: retraining, reweighting, alerting, or rolling back a model version.

This chapter builds the detection stack in three layers: the statistical test that scores a difference, the windowing strategy that decides which data the test sees, and the sequential procedure that converts a stream of scores into a change-point decision.

1. What Kind of Drift Are You Detecting?

Before choosing a test, classify the drift. The three canonical types behave differently under detection:

• Covariate shift (data drift): the input distribution P(X) changes while P(Y|X) stays fixed. Detectable without labels, using only features.
• Label shift (prior shift): P(Y) changes while P(X|Y) stays fixed. Requires labels or a calibrated estimator of label proportions.
• Concept drift: P(Y|X) itself changes. This is the most damaging and the hardest to detect, because inputs may look perfectly normal while the correct mapping has moved.

A practical detector therefore runs at least two monitors: an unsupervised feature monitor (catches covariate shift early, no labels needed) and a supervised performance monitor (catches concept drift, but only after labels arrive, often with delay).

2. Statistical Tests: Scoring a Difference

Each test answers the same question — "could this difference plausibly come from sampling noise?" — but makes different assumptions and has different power profiles.

For a univariate continuous feature, the two-sample KS statistic is the ratio of the maximum vertical gap between two empirical CDFs:

D = sup_x | F_ref(x) - F_cur(x) |

# Reference window: 10,000 samples from training period
# Current window:   2,000 samples from last 24 hours
# D = 0.041, p-value = 0.003  -> reject H0 at alpha = 0.01

PSI is often preferred in production because it produces a single interpretable number per feature and does not require the reference sample to be retained in full:

PSI = sum_i ( (p_cur_i - p_ref_i) * ln(p_cur_i / p_ref_i) )

# Bins: deciles of the reference distribution
# PSI < 0.10  -> no meaningful shift
# 0.10 - 0.25 -> moderate shift, investigate
# PSI > 0.25  -> significant shift, act

These thresholds are conventions, not laws. They must be calibrated against your own false-alarm tolerance and the cost of a missed drift.

3. Windowing Strategies: What the Test Sees

The same test applied to different windows gives different answers. Windowing is where most production detectors succeed or fail.

• Fixed reference window: a frozen sample from the training period. Simple, but becomes stale as the world moves; eventually every comparison flags drift.
• Sliding window: compare the last W samples against the previous W. Adapts to gradual change but has low power for small shifts and can miss slow drift entirely.
• Landmark window: compare everything since a fixed landmark (e.g., last retrain) against the reference. Good for detecting cumulative drift.
• ADWIN-style adaptive window: maintain a variable-length window and shrink it when a change is detected inside it. Balances sensitivity and stability automatically.

Window size is a bias–variance trade-off. Small windows react fast but produce noisy statistics and frequent false alarms; large windows are stable but slow. A common starting point is a reference window of 10,000–50,000 samples and a current window of 1,000–5,000, adjusted so that the expected number of positives per window is at least a few hundred for stable estimates.

4. Sequential Change-Point Detection

Running a test once per window and thresholding the p-value is a multiple-testing problem: with 100 features checked hourly, you will see false alarms constantly. Sequential change-point detection solves this by accumulating evidence over time and declaring a change only when the accumulated evidence crosses a boundary calibrated for a target average run length (ARL).

The CUSUM procedure for detecting a shift in the mean of a monitored statistic x_t (for example, a per-batch PSI or a per-batch error rate) is:

S_t = max(0, S_{t-1} + (x_t - mu_0 - k))
alarm when S_t > h

# mu_0 = in-control mean of the statistic
# k    = slack, typically half the shift you want to detect
# h    = threshold, set from the desired ARL

For a target shift of delta = 0.05 in a statistic with in-control mean mu_0 = 0.10, set k = delta/2 = 0.025. The threshold h is chosen so that under no drift the average number of samples between false alarms is acceptable — for example, h = 5 gives a long ARL for small k, while h = 2 reacts faster but alarms more often. The Page–Hinkley test is a two-sided variant that tracks both upward and downward shifts.

For multivariate streams, apply CUSUM to a scalar summary — the PSI of the most-drifted feature, the MMD between reference and current batches, or the model's rolling log-loss — rather than to each dimension independently. This keeps the false-alarm rate controllable.

5. Putting the Stack Together

The diagram below shows how the three layers connect in a streaming pipeline. Raw events are buffered into windows; each window is scored by a statistical test against the reference; the score sequence feeds a sequential detector that emits a change-point alarm with a timestamp and an attributed feature.

6. A Minimal Working Example

Suppose a fraud model monitors a single feature, transaction_amount, with a reference window of 10,000 training samples. Every hour, the last 2,000 transactions form the current window. The pipeline computes PSI per feature and feeds the maximum PSI into a CUSUM detector.

# Hourly loop (illustrative values)
# ref_window: 10,000 samples, decile bins fixed at training time
# cur_window: 2,000 samples

psi_amount   = 0.08   # below 0.10 -> no shift
psi_velocity = 0.31   # above 0.25 -> significant
score_t      = max(psi_amount, psi_velocity) = 0.31

# CUSUM with mu_0 = 0.10, k = 0.025, h = 2.0
S_t = max(0, S_{t-1} + (0.31 - 0.10 - 0.025)) = S_{t-1} + 0.185

# After ~11 consecutive hours at this level, S_t exceeds 2.0
# -> alarm: change point attributed to feature "velocity"

The alarm is not the end of the process. The detector should emit three things: the change-point timestamp, the feature or slice responsible, and the magnitude of the shift. That triple lets an operator decide whether to retrain, reweight, or simply investigate a data-pipeline bug — a surprisingly common cause of apparent drift.

7. Choosing and Calibrating

Start with PSI on binned features for interpretability, add KS for continuous features where shape matters, and use MMD when joint multivariate shifts are suspected. Wrap the chosen score in a CUSUM or Page–Hinkley detector calibrated to your target ARL. Re-calibrate the reference window on every retrain, and log every alarm with its attribution so that false positives can be traced back to pipeline changes rather than genuine drift.

The detector is only as good as its calibration. Before trusting any threshold, replay historical data with known drift events and measure detection delay and false-alarm rate. A detector that fires on every retrain is noise; one that never fires is decoration. The useful detector sits between those extremes, with a documented ARL and a clear action attached to each alarm.