2. Prediction interval fundamentals

A point prediction is almost never the whole story in a pass–fail decision. When a model predicts a quality characteristic, a remaining useful life, or a risk score, the decision boundary is a threshold, and what matters is not the single predicted value but how far the true value may plausibly lie from it. A prediction interval (PI) quantifies exactly that: a range that, with a stated probability, is expected to contain the true target for a new, unseen observation. This chapter establishes the four ideas that every downstream margin calculation depends on: coverage probability, interval width, calibration, and the distinction between confidence intervals and prediction intervals.

Coverage probability: the promise a PI makes

For a nominal level 1 − α (for example, 0.90 or 0.95), a prediction interval [L(x), U(x)] is said to have coverage 1 − α if, over repeated draws of training data and new test points, the true target y falls inside the interval with that long-run frequency. Formally, for a new input x and target y:

P( L(x) ≤ y ≤ U(x) ) = 1 − α

Two subtleties matter in practice. First, coverage is a property of the procedure, not of any single interval — you cannot verify it on one prediction. Second, the guarantee is marginal unless stated otherwise: it holds on average across the input distribution. A model can achieve 90% marginal coverage while badly under-covering a specific subgroup (say, high-variance operating regimes) and over-covering another. For pass–fail decisions this is dangerous, because the region where coverage fails is often exactly the region near the decision threshold. When subgroup reliability matters, evaluate conditional coverage by binning the test set on predicted value, variance proxy, or domain-relevant strata and reporting empirical coverage per bin.

Interval width: the price of the guarantee

Coverage alone is trivially achievable — an interval of (−∞, +∞) covers everything. The useful question is how narrow the interval is at the required coverage. Width W(x) = U(x) − L(x) is the quantity that converts into decision margin: a narrow interval lets you accept or reject with confidence, while a wide interval forces abstention or escalation. Width is driven by three factors:

• Irreducible noise σ in the target — no model can shrink this.
• Model uncertainty — estimation error from limited training data, larger in sparse regions of input space.
• Nominal level — wider intervals for higher 1 − α; for a Gaussian residual model, width scales with z1−α/2 · σ.

Width should be adaptive: heteroscedastic problems (noise growing with load, temperature, or sensor age) demand input-dependent width. A constant-width interval that is valid on average will be too wide in quiet regions and too narrow in noisy ones — the worst possible trade for a threshold decision.

Calibration: does the stated level match reality?

A PI is calibrated when its empirical coverage on held-out data matches its nominal level. Calibration is checked, not assumed. The standard diagnostic is a coverage curve: for nominal levels α ∈ {0.01, 0.05, 0.10, …, 0.50}, compute the fraction of test targets inside the corresponding intervals and plot empirical versus nominal coverage. A well-calibrated procedure lies on the diagonal; systematic deviation above indicates conservative (over-wide) intervals, below indicates anti-conservative (over-confident) intervals.

Calibration can fail even when the underlying model is accurate, because PI construction depends on assumptions — Gaussian residuals, correct variance model, exchangeability of calibration and test data. Split conformal prediction is the practical remedy: hold out a calibration set, compute nonconformity scores si = |yi − ŷ(xi)|, take the ⌈(n+1)(1−α)⌉/n empirical quantile q̂, and form ŷ(x) ± q̂. This yields finite-sample marginal coverage under exchangeability, with no distributional assumption. For heteroscedastic data, normalize the score by a fitted scale σ̂(x) before taking the quantile, then rescale the interval — this restores adaptivity while preserving the coverage guarantee.

Confidence interval vs. prediction interval

The most common conceptual error is conflating these two. A confidence interval bounds an unknown parameter — for example, the conditional mean E[y | x]. A prediction interval bounds an unknown future observation y. Because a new observation carries both estimation uncertainty about the mean and irreducible noise around it, the PI is always wider, and the gap grows as the noise grows. For a simple linear model with residual standard deviation σ:

CI for mean:   ŷ ± t · σ · sqrt( 1/n + (x − x̄)² / Sxx )
PI for new y:  ŷ ± t · σ · sqrt( 1 + 1/n + (x − x̄)² / Sxx )

The extra 1 under the square root is the noise term. With large n, the CI collapses toward zero width while the PI converges to ± t·σ — it never vanishes. This is why a tight confidence interval around a predicted mean says nothing about whether a single future part will pass inspection. For pass–fail margins, the PI is the correct object; using a CI as if it were a PI produces systematically over-optimistic acceptance decisions.

Putting it to work: a reproducible calibration check

Suppose a regression model predicts a dimensional characteristic in millimetres, and the pass window is [9.80, 10.20]. Before trusting any margin, run a calibration audit on a held-out set of n = 500 points. The procedure is concrete:

1. Compute residuals ri = yi − ŷ(xi) and a scale estimate σ̂(x) (e.g., from a second model or local residual variance).
2. Form normalized scores si = |ri| / σ̂(xi).
3. For each nominal α, take q̂ = Quantile(s, 1−α) and build ŷ(x) ± q̂ · σ̂(x).
4. Report empirical coverage and mean width per α; compare to nominal.

Expected observations: coverage should track nominal within sampling error (roughly ±1.96·sqrt(α(1−α)/n), about ±2.6 percentage points at 90% with n=500). If empirical coverage at 90% comes out at 84%, the intervals are anti-conservative — typical causes are a misspecified scale model, exchangeability violation (calibration and test data drawn from different operating regimes), or too few calibration points. If coverage is 96% with very wide intervals, the procedure is conservative and the margin is being paid for in unnecessary abstentions. Both are actionable: refit the scale model, re-stratify the calibration set, or increase its size.

With coverage, width, and calibration understood, and with the CI/PI distinction fixed, the next step is to connect these interval properties to the decision rule itself: how a threshold, an interval, and an acceptable error rate combine into a defensible pass–fail margin.