2. Foundations of distribution shift

An out-of-distribution (OOD) detector is only as useful as the failure taxonomy it is built on. Before choosing a scoring function, threshold, or calibration strategy, you need to know what kind of distribution shift you are defending against, because different shifts break different assumptions and demand different mitigations. This chapter establishes a working taxonomy of distribution shift — covariate shift, label shift, concept shift, and semantic shift — grounded in real deployment scenarios, and shows how to reason about which one is present in a given failure.

We start from the standard supervised setup. A model is trained on samples drawn from a joint distribution P_train(X, Y) and deployed where samples arrive from P_test(X, Y). The joint factorizes as P(X, Y) = P(X) · P(Y | X), and every shift type below is defined by which factor changes and whether the label space itself is preserved. Keeping this factorization in mind is the single most useful diagnostic habit: when a model degrades in production, ask first whether the input distribution moved, the label prior moved, the input-to-label mapping moved, or the label set itself changed.

Covariate shift: the inputs move, the rule stays

Covariate shift occurs when P(X) changes but P(Y | X) is unchanged. The conditional relationship the model learned remains valid; it is simply being queried in regions of input space that were underrepresented during training. This is the most common and most benign form of shift, and it is the one classical importance-weighting methods target.

A concrete example: a defect-detection model trained on images captured under bright factory lighting is redeployed on a night shift where ambient light drops and sensor noise rises. The mapping from "scratch texture" to "defective" has not changed, but the pixel distribution has. Similarly, a fraud model trained on transactions from one geographic region sees a different spend-amount distribution when expanded to a new market, while the underlying fraud mechanics remain the same.

The practical signature of covariate shift is that input-space statistics drift — feature means, variances, and covariances move — while label-conditional behavior stays stable. Detection therefore leans on input-space monitors: feature-distribution distances (e.g., Maximum Mean Discrepancy, energy scores on logits), reconstruction error from an autoencoder, or density estimates under a generative model of P(X). The standard mitigation is importance weighting: estimate the density ratio w(x) = P_test(x) / P_train(x) and reweight training or calibration losses so the model is optimized for the deployment distribution.

Label shift: the class mix moves, the rule stays

Label shift (also called prior shift or target shift) occurs when P(Y) changes but P(X | Y) is unchanged. Each class still looks the same; the proportions in which classes appear have changed. This is common in any setting with a base-rate that varies over time or population.

Consider a rare-disease classifier trained on a screening population where prevalence is 1%. Deployed in a specialist referral clinic, prevalence may be 30%. The appearance of a diseased sample is identical, but the prior has shifted by more than an order of magnitude. A spam filter is another classic case: the fraction of spam in the inbox varies seasonally and by user, while the appearance of a given spam campaign is stable.

Label shift is dangerous because it silently corrupts calibrated probabilities and threshold-based decisions even when per-class accuracy looks fine. A model that outputs P(spam) = 0.6 under a 20% prior is badly miscalibrated under a 60% prior. Detection relies on label-space monitors: tracking the predicted label histogram over a window and comparing it to the training prior, or using black-box shift estimators such as BBSE (Black Box Shift Estimation) that solve for the new prior from the confusion matrix and observed predictions. Mitigation is a prior correction — reweighting predicted posteriors by the estimated ratio P_test(Y) / P_train(Y) — which restores calibration without retraining.

Concept shift: the rule itself changes

Concept shift (concept drift) occurs when P(Y | X) changes while P(X) may stay fixed. The same input now maps to a different label. This is the most insidious shift because input-space monitors see nothing wrong: the features look perfectly in-distribution, yet the model's decisions are wrong.

Real-world examples are abundant. A credit-risk model trained before an economic downturn has learned that a given income-to-debt ratio implies low default risk; after the downturn, the same ratio implies much higher risk. A recommendation model trained on pre-pandemic behavior sees the same user features but a fundamentally different relationship between "browses travel content" and "will book." Adversarial and label-poisoning attacks are a deliberate, fast-acting form of concept shift: the input is perturbed so that P(Y | X) is effectively rewritten for that region.

Because input statistics are unremarkable, concept shift must be caught with outcome signals: delayed ground-truth labels, online performance metrics, or drift detectors on the residual y − ŷ. When labels are unavailable, proxy signals such as prediction-confidence drift, disagreement across an ensemble, or changes in the decision-boundary margin serve as early warnings. Mitigation requires adaptation — periodic retraining, online learning, or explicit change-point detection that triggers a model refresh.

Semantic shift: the label space itself changes

Semantic shift is the case where test samples belong to classes that did not exist, or were not represented, in training — the label space Y itself is not preserved. This is the canonical "unknown unknown" and the primary target of open-set recognition and OOD detection. A model trained to classify cats and dogs encounters a fox: no amount of covariate or prior correction helps, because the correct answer is not in the label set.

Semantic shift is distinct from the three statistical shifts above in a crucial way: it is a support problem, not a density problem. The test sample may lie in a high-density region of input space (a fox looks a lot like a dog) yet have no valid label. This is why semantic OOD detection is hard and why methods based on input density alone often fail — the fox is not "far away," it is "off-manifold in label space." Detection relies on representation-space geometry: distance to class prototypes, Mahalanobis distance in feature space, energy or logit-margin scores, and contrastive or outlier-exposure training that shapes the feature space so unknowns fall outside class clusters. The correct response is to abstain or route to a human, not to force a label.

Why the distinction matters in practice

The four shift types are not academic categories — they determine which detector is even capable of firing. A covariate-shift detector built on input-space distance will be blind to concept shift, because the inputs look normal. A label-shift estimator will not flag a fox, because the fox is not a class whose prior moved. Conversely, a semantic OOD detector tuned to reject far-away inputs may over-reject legitimate covariate-shifted samples (the dimly lit but valid scratch), producing false alarms. Choosing the wrong detector for the shift you actually face is one of the most common root causes of "the OOD system didn't catch it" post-mortems.

A useful diagnostic sequence when a production model degrades:

1. Check input statistics first. Compare feature means, variances, and a distribution distance (e.g., MMD or a two-sample test) between a recent window and the training set. A large, broad drift points to covariate shift.
2. Check the predicted label histogram. If inputs look stable but the predicted class mix has moved, suspect label shift; confirm with a prior estimator such as BBSE using a held-out confusion matrix.
3. Check outcome signals. If inputs and predicted priors are stable but accuracy or residual drift has risen, concept shift is the leading hypothesis — the rule changed under a stable input distribution.
4. Check for unknowns. If errors concentrate on samples that are confidently misclassified or fall outside class clusters in representation space, semantic shift is present and the model needs an abstention path.

A worked diagnostic example

Suppose a churn-prediction model's weekly AUC drops from 0.82 to 0.74. The team runs the sequence above on a 7-day window of 50,000 scored accounts. Input monitoring shows the tenure feature's mean fell from 14.2 to 9.8 months and the MMD between window and training features is 0.11 (well above the 0.02 alert threshold) — a clear covariate shift, likely from a new onboarding campaign that brought in younger accounts. The predicted churn histogram, however, is stable at roughly 18%, and delayed 30-day labels show the residual rate unchanged within confidence intervals. The diagnosis is covariate shift with no concept change: the model is being queried in a younger-tenure region it saw less of during training. The correct response is importance weighting or targeted retraining on the new tenure band — not a full model rebuild, and not a semantic OOD alarm. Had the residual rate risen while inputs stayed stable, the diagnosis would have flipped to concept shift and demanded retraining on fresh labels.

Mastering this taxonomy turns OOD detection from a single score into a structured diagnosis: identify the changed factor, select the matching monitor, and apply the matching mitigation. The remaining chapters build detectors and calibration methods on top of this foundation.