4. Taxonomy of OOD detection methods

Out-of-distribution (OOD) detection is not a single algorithm but a family of strategies that differ in what signal they exploit, what they assume about the model, and what they cost at inference time. Practitioners routinely conflate these families, then wonder why a method that works on one benchmark collapses on another. The reason is usually a mismatch between the method's underlying assumption and the deployment setting. This chapter organizes the field into four canonical families — discriminative, generative, distance-based, and Bayesian — and explains what each one actually measures, when it is appropriate, and where it fails.

The organizing principle is simple: every OOD detector computes some score s(x) from an input x (and possibly the trained model), then thresholds it. What differs is where the score comes from. Discriminative methods read the classifier's own output distribution. Generative methods ask how well a density model explains the input. Distance-based methods measure proximity to representations of in-distribution (ID) data. Bayesian methods quantify uncertainty in the model's parameters or predictions. The figure below maps these four families onto their score sources and typical failure modes.

Discriminative Methods: Reading the Classifier's Own Confidence

Discriminative detectors require no auxiliary model. They reuse the trained classifier and derive a score from its output layer. The simplest is Maximum Softmax Probability (MSP): s(x) = max_c p(y=c|x). A well-trained network tends to assign high confidence to ID inputs and lower confidence to unfamiliar ones. In practice MSP is a weak baseline — modern networks are systematically overconfident on OOD inputs — but it is nearly free and worth reporting as a floor.

Two refinements dominate. ODIN applies temperature scaling and input preprocessing: it divides logits by a temperature T (commonly 1000) and adds a small gradient-based perturbation to the input before computing the softmax. Energy-based scoring replaces the softmax with the free energy E(x) = -T · logsumexp(logits / T), which is theoretically better behaved because it does not saturate the way softmax does. A practical energy implementation looks like this:

import torch

def energy_score(logits, T=1.0):
    # logits: [batch, num_classes]
    return -T * torch.logsumexp(logits / T, dim=1)

# At inference: lower energy = more ID-like
scores = energy_score(model(x))
is_ood = scores < threshold   # threshold from ID validation set

The key operational point: the threshold is not a universal constant. You calibrate it on a held-out ID validation split to achieve a target true-positive rate at a fixed false-positive rate (e.g., TPR@FPR=5%). Discriminative methods are cheap and integrate trivially into existing pipelines, but they inherit the classifier's calibration problems and degrade when the OOD data is semantically close to ID classes.

Generative Methods: Modeling the Input Density

Generative detectors train a density model p(x) on ID data and flag inputs with low likelihood. Variational autoencoders (VAEs), normalizing flows, and diffusion models all serve here. The appeal is conceptual cleanliness: an OOD input should be improbable under the ID distribution. The catch is the well-documented likelihood gap — deep generative models can assign higher likelihood to some OOD inputs than to ID data, because they capture low-level statistics (pixel smoothness, background texture) rather than semantic content.

Practical mitigations include using reconstruction error instead of raw likelihood, combining likelihood with a typicality term, or scoring in a learned latent space rather than input space. Generative methods are expensive at inference (a forward pass through a large decoder or iterative diffusion sampling) and are best reserved for settings where input-space anomalies are the actual threat model — for example, sensor corruption or adversarial perturbations — rather than semantic novelty.

Distance-Based Methods: Proximity in Representation Space

Distance-based detectors assume that ID data occupies a compact region in the model's penultimate feature space, and that OOD inputs land far from it. The canonical method is Mahalanobis distance: fit a class-conditional Gaussian to ID features, then score an input by its minimum Mahalanobis distance to any class centroid. A lighter alternative is k-nearest-neighbor distance to a stored bank of ID features, or cosine similarity to class prototypes.

# Mahalanobis scoring sketch
mu_c = features[id_labels == c].mean(0)          # per-class mean
Sigma = shared_covariance(features, id_labels)   # pooled covariance
Sigma_inv = torch.linalg.inv(Sigma)

def mahalanobis(f):
    d = [((f - mu_c) @ Sigma_inv * (f - mu_c)).sum(-1) for mu_c in class_means]
    return torch.stack(d, dim=-1).min(-1).values  # min distance = score

Distance-based methods are strong when features are well-separated and the ID manifold is compact, and they are relatively cheap once the feature bank is built. Their weakness is feature collapse: if the network maps OOD inputs into the same region as ID features (common with over-parameterized models), distances become uninformative. They also require storing ID statistics, which raises privacy and memory considerations in some deployments.

Bayesian Methods: Quantifying Predictive Uncertainty

Bayesian detectors treat uncertainty as the signal. Monte Carlo dropout approximates a posterior over weights by keeping dropout active at inference and running N stochastic forward passes; the variance across predictions becomes the OOD score. Deep ensembles do the same with independently trained models. The intuition is that ID inputs produce consistent predictions across samples from the posterior, while OOD inputs produce divergent ones.

model.train()  # keep dropout active
preds = torch.stack([model(x) for _ in range(30)])   # 30 MC samples
mean_pred = preds.mean(0)
uncertainty = preds.var(0).mean(-1)   # high variance → OOD
is_ood = uncertainty > threshold

Bayesian methods offer principled uncertainty and often the best AUROC on hard benchmarks, but they multiply inference cost by the number of samples (30× is typical) and their uncertainty estimates are only as good as the posterior approximation. MC dropout in particular is known to be miscalibrated; ensembles are more reliable but require training multiple models.

Choosing Among the Families

The right choice depends on three constraints: what OOD means in your deployment (semantic novelty vs. corruption vs. adversarial), what compute you can afford at inference, and whether you can retrain or only post-process. A useful decision heuristic:

The taxonomy is not a ranking. Each family encodes a different hypothesis about why OOD inputs look different, and each hypothesis can be wrong for a given deployment. The engineering skill is matching the hypothesis to the threat model, measuring with the right metrics on the right splits, and layering cheap and expensive detectors so that cost scales with ambiguity.