2. Bias-Variance Decomposition of a Single Learner

Why a Single Learner Needs a Decomposition

A single regression learner trained once on one dataset gives you a point estimate and a residual error, but that error is a single draw from a distribution. If you retrained the same algorithm on a different sample of the same population, the fitted function would shift. The expected squared prediction error at a fixed input x can be decomposed into three terms:

E[(y − f̂(x))²] = Bias²(f̂(x)) + Var(f̂(x)) + σ²

where Bias(f̂(x)) = E[f̂(x)] − f(x) is the systematic offset of the average fitted value from the true regression function, Var(f̂(x)) = E[(f̂(x) − E[f̂(x)])²] is the spread of fitted values across training samples, and σ² is the irreducible noise. The first two terms are what a learner can trade against each other; the third is a floor no model can beat. This case estimates the first two empirically for a single learner, so that later ensemble work has a quantified target: variance reduction.

Estimating the Components by Repeated Subsampling

The decomposition is defined over the distribution of training sets, which we do not have. We approximate it by drawing B independent training subsets from the available data, fitting the learner on each, and evaluating all fits on a fixed held-out grid. For each grid point x:

• f̄(x) = (1/B) Σ_b f̂_b(x) — the average prediction across subsets, our estimate of E[f̂(x)].
• Var̂(f̂(x)) = (1/(B−1)) Σ_b (f̂_b(x) − f̄(x))² — the empirical variance.
• Bias²(f̂(x)) ≈ (f̄(x) − y(x))² — using the held-out target as a proxy for f(x); this is only valid when noise is small relative to bias, so report it as an approximation.

Two practical constraints matter. First, subsets must be drawn independently and must not overlap with the evaluation grid, otherwise the variance estimate is contaminated by memorization. Second, B must be large enough that the variance estimate itself is stable — with B = 30 the standard error of Var̂ is roughly Var·√(2/(B−1)) ≈ 0.26·Var, so use B ≥ 50 for a defensible number.

Learning-Curve Analysis as a Cross-Check

Subsampling at a fixed size gives one bias-variance split. Repeating the procedure at several training-set sizes n produces a learning curve that separates the two components by their scaling behavior. For a high-variance learner such as a deep decision tree, the variance term falls steeply with n while bias stays roughly flat; for a high-bias learner such as a shallow tree or a linear model, bias dominates and the curve flattens early. Plotting Bias² and Var against n on the same axes makes the diagnosis explicit and tells you whether more data or a different model class is the right lever.

Worked Example: A Deep Regression Tree on an Engineering Dataset

Consider a tabular dataset of 2000 rows describing a structural response variable (for example, peak displacement of a frame under a load case) with eight numeric predictors. The learner is a regression tree with max_depth=None, min_samples_leaf=1 — deliberately unregularized so that variance is the dominant failure mode. The evaluation grid is a held-out set of 400 rows never used in any subset.

The procedure, in plain Python with scikit-learn, is short enough to run end to end:

import numpy as np
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42)

B = 60
n_sub = 500
preds = np.empty((B, len(X_test)))

for b in range(B):
    idx = rng.choice(len(X_train), size=n_sub, replace=False)
    tree = DecisionTreeRegressor(random_state=b)
    tree.fit(X_train[idx], y_train[idx])
    preds[b] = tree.predict(X_test)

f_bar = preds.mean(axis=0)
var_hat = preds.var(axis=0, ddof=1)
bias_sq = (f_bar - y_test) ** 2

print("mean Var :", var_hat.mean())
print("mean Bias²:", bias_sq.mean())
print("mean σ² proxy (residual var of full fit):",
      np.var(y_test - DecisionTreeRegressor().fit(X_train, y_train).predict(X_test)))

With n_sub = 500 and B = 60, the expected output is a variance term that is large relative to the bias term — for an unregularized tree on this kind of data, Var̂ typically exceeds Bias² by a factor of several. That ratio is the quantitative justification for ensembling: bagging averages B such trees, and if the trees were independent the variance would fall by 1/B; correlated trees fall less, which is exactly what the next case measures.

Reading the Result and Common Failure Modes

The deliverable of this case is a two-number summary per learner — mean Bias² and mean Var̂ on the held-out grid — plus the learning-curve plot. Those numbers become the reference against which the variance reduction from bagging and random forests is measured in the following cases.