2. Statistical foundations for comparison

Before any two models can be declared "different" or "equivalent," we must be precise about what a comparison statistic actually measures and how much of it is noise. A single evaluation run produces a point estimate — one accuracy, one latency percentile, one AUC. That number is a realization of a random variable, not a property of the model. This chapter builds the statistical vocabulary that the rest of the acceptance workflow depends on: sampling variability, the bias–variance decomposition of estimators, confidence intervals, and the logic of hypothesis testing. Every later acceptance criterion in this course is a special case of the ideas below.

Sampling variability: why one number is never enough

Suppose a classifier is evaluated on a held-out test set of n = 2,000 examples and achieves 0.842 accuracy. If we had drawn a different test set from the same distribution, the accuracy would differ. The standard error of a proportion quantifies this: SE = sqrt(p(1−p)/n). With p = 0.842 and n = 2,000, SE ≈ sqrt(0.842 × 0.158 / 2000) ≈ 0.0082. So the observed 0.842 is plausibly anywhere near 0.842 ± 0.016 at roughly 95% confidence. Two models whose accuracies differ by 0.01 on this test set are, statistically, indistinguishable — the gap is smaller than the noise floor.

Sampling variability has three distinct sources that must not be conflated:

• Test-set sampling noise — which examples happened to land in the evaluation split. Shrinks as 1/sqrt(n).
• Training-set sampling noise — retraining on a different data draw yields a different model. Often larger than test noise and invisible if you only train once.
• Algorithmic randomness — initialization, dropout, data shuffling, and non-deterministic GPU kernels. Reproducible only with fixed seeds, and even then not always bit-identical.

A comparison that ignores any of these will overstate confidence. The practical rule: estimate the variance of the comparison statistic itself, not just of each model's score.

Bias and variance of estimators

An estimator θ̂ of a true quantity θ has two error components. Bias is the systematic offset E[θ̂] − θ; variance is E[(θ̂ − E[θ̂])²]. The mean squared error decomposes as MSE = bias² + variance. This decomposition matters for model comparison because different evaluation protocols trade one for the other.

Consider estimating generalization accuracy. A single random train/test split gives an unbiased estimate of the accuracy of a model trained on that split's training portion, but it has high variance — a lucky or unlucky split can move the number by several points. k-fold cross-validation averages k estimates, sharply reducing variance, but each fold model is trained on only (k−1)/k of the data, so the estimate is slightly pessimistic (biased low) relative to a model trained on the full set. Neither protocol is "correct"; they answer slightly different questions. When comparing two models, use the same splits for both so that split-to-split variance cancels in the difference.

Confidence intervals

A 95% confidence interval is a procedure, not a probability statement about a fixed parameter. If we repeated the entire evaluation many times, 95% of the constructed intervals would contain the true value. For a proportion, the Wilson interval is preferred over the naive Wald interval because it behaves well near 0 and 1:

# Wilson score interval, 95%, for p_hat = 0.842, n = 2000
z = 1.96
p = 0.842; n = 2000
denom  = 1 + z**2 / n
center = (p + z**2 / (2*n)) / denom
half   = (z / denom) * sqrt(p*(1-p)/n + z**2/(4*n**2))
# center ≈ 0.8417, half ≈ 0.0159  →  CI ≈ [0.826, 0.858]

For a difference between two models evaluated on the same test set, the correct interval accounts for correlation. Paired methods — McNemar's test for classification correctness, or a paired bootstrap over examples — are far more sensitive than treating the two accuracies as independent. If model A is correct on 1,650 examples and model B on 1,640, with 1,600 correct for both, the discordant counts (A-only = 50, B-only = 40) drive the comparison, not the marginal accuracies.

Hypothesis testing and its limits

The null hypothesis H₀ is typically "the two models have equal expected performance." A test returns a p-value: the probability, assuming H₀, of observing a statistic at least as extreme as the one measured. A small p-value is evidence against H₀, but it is not the probability that H₀ is true, and it says nothing about effect size. With n = 100,000, a 0.001 accuracy difference can be "significant" while being operationally irrelevant.

Two failure modes dominate engineering practice. Multiple comparisons: testing ten model variants against a baseline at α = 0.05 yields roughly a 40% chance of at least one false positive. Apply Bonferroni (α/m) or Benjamini–Hochberg control when screening many candidates. Optional stopping: peeking at results and stopping when p < 0.05 inflates the false-positive rate dramatically; pre-register the sample size or use sequential tests designed for it.

The figure below traces how a single observed difference between two models flows through the decision logic — from raw scores, through paired variance estimation, to an interval and a test decision, with the two failure modes flagged at the points where they enter.

From statistics to an acceptance decision

The engineering discipline is to fix the decision rule before looking at results. A workable template for a two-model comparison:

1. State the estimand: e.g., "difference in top-1 accuracy on the production-distribution test set."
2. Fix the test set and the pairing scheme; both models see identical examples.
3. Pre-specify the minimum detectable effect (MDE) that matters operationally — say 0.01 accuracy.
4. Compute the required n from the paired variance estimate; if n is fixed, compute the achievable MDE and accept that smaller differences cannot be resolved.
5. Report the point estimate, its confidence interval, and the test statistic together. A bare p-value is not an acceptance artifact.
6. Apply multiplicity control if more than two candidates are compared.

These four ideas — variability, bias–variance, intervals, and tests — are the grammar of every acceptance criterion that follows. When a later chapter specifies "the candidate must not regress by more than 0.5% at 95% confidence," that sentence is a compressed version of everything above: a paired estimand, a variance estimate, an interval, and a pre-declared margin. Getting the statistics right at this layer is what makes the downstream acceptance criteria defensible rather than decorative.