3. Bootstrap Aggregating for Regression Stability

Why a Single Tree Is Unstable

A regression tree is a high-variance estimator: a small perturbation of the training set — a few swapped rows, a slightly different bootstrap sample — can change the first split, and every split below it inherits that change. Bagging (bootstrap aggregating) attacks this directly. It trains B trees on B bootstrap resamples of the training data, then averages their predictions. Because the trees are identically distributed but not independent, the variance of the average is not σ²/B; it is

Var(f̄) = ρσ² + (1 − ρ)σ²/B

where σ² is the variance of a single tree's prediction at a given input and ρ is the pairwise correlation between trees. The second term vanishes as B grows; the first does not. This is the whole engineering story of bagging: more trees reduce the reducible part of the variance, but correlated trees leave a floor. The practical task in this case is to measure that reduction on a real regression problem rather than assume it.

Dataset and Experimental Contract

We use a synthetic but physically plausible engineering regression dataset so the experiment is fully reproducible: predicting the fatigue life (in 10^3 cycles) of a welded joint from four process/geometry features — weld_toe_radius_mm, leg_length_mm, heat_input_kJ_mm, and plate_thickness_mm. The response is generated with a smooth nonlinear law plus multiplicative noise, which is exactly the regime where trees are biased-but-flexible and variance dominates the error.

import numpy as np
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import BaggingRegressor
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(42)
n = 600
toe   = rng.uniform(0.5, 4.0, n)          # mm
leg   = rng.uniform(4.0, 12.0, n)         # mm
heat  = rng.uniform(0.5, 2.5, n)          # kJ/mm
thick = rng.uniform(6.0, 25.0, n)         # mm

# Smooth "true" law + heteroscedastic noise
base = 180.0 * (toe / 2.0) ** 0.6 * (leg / 8.0) ** 0.4
base /= (1.0 + 0.35 * heat) * (thick / 15.0) ** 0.25
noise = rng.normal(0.0, 0.08 * base)
y = base + noise

X = np.column_stack([toe, leg, heat, thick])
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, random_state=0)

The split is fixed with random_state=0 so every model below sees identical training data — otherwise the variance comparison would be contaminated by split-to-split differences.

Measuring Prediction Variance, Not Just Error

A single train/test split gives one error number per model, which says nothing about stability. To quantify variance we need repeated perturbation of the training set and repeated prediction at fixed test points. The cleanest protocol: draw R independent bootstrap resamples of the training set, fit the model on each, and record predictions on the same held-out test inputs. The per-test-point standard deviation across the R replicates is the prediction variance we care about.

def replicate_predictions(make_model, X_tr, y_tr, X_te, R=30, seed=7):
    r = np.random.default_rng(seed)
    preds = np.empty((R, X_te.shape[0]))
    for i in range(R):
        idx = r.integers(0, len(y_tr), len(y_tr))   # bootstrap resample
        model = make_model()
        model.fit(X_tr[idx], y_tr[idx])
        preds[i] = model.predict(X_te)
    return preds

single = replicate_predictions(
    lambda: DecisionTreeRegressor(max_depth=None, random_state=0),
    X_tr, y_tr, X_te)

bagged = replicate_predictions(
    lambda: BaggingRegressor(
        estimator=DecisionTreeRegressor(random_state=0),
        n_estimators=50, max_samples=1.0, bootstrap=True,
        random_state=0, n_jobs=-1),
    X_tr, y_tr, X_te)

var_single = single.var(axis=0, ddof=1)   # per test point
var_bagged = bagged.var(axis=0, ddof=1)
print("mean var single :", var_single.mean())
print("mean var bagged :", var_bagged.mean())
print("variance ratio  :", var_bagged.mean() / var_single.mean())

Two details matter for correctness. First, ddof=1 — we are estimating variance from a finite replicate set, not computing a population statistic. Second, the bagged model's internal random_state=0 is fixed so that the only randomness driving the outer replicates is the bootstrap resample itself; otherwise the inner tree randomness would inflate the measured variance and understate the bagging benefit.

Reading the Result

On this dataset the expected pattern is a variance ratio well below 1 — typically in the 0.1–0.3 range for 50 fully grown trees — while the mean absolute error on the test set also drops, because variance was the dominant error component. The ratio is the number to report: it is the empirical estimate of Var(f̄)/Var(f) and can be compared directly against the theoretical floor ρ once you estimate the tree-to-tree correlation from the same replicate matrix.

Interpreting the Variance Ratio Correctly

The ratio var_bagged.mean() / var_single.mean() is a mean over test points, and that averaging hides structure. Two checks are worth running before you trust it:

• Per-point distribution. Plot or summarize the ratio pointwise. If a handful of test points dominate the mean, the "reduction" may be an artifact of extrapolation regions where the single tree is wildly unstable and the bagged model is merely less wild — not a genuine improvement in the operating range.
• Bias check. Bagging reduces variance but does not reduce the bias of the base learner; averaging trees that all underfit the same region leaves that underfit intact. Compare mean prediction error, not only variance. A model can be more stable and still worse if the base tree is too shallow.

For the fatigue-life dataset, the expected outcome is that the bagged ensemble's variance ratio sits near the estimated tree-to-tree correlation ρ once B is large enough, and that its test MAE is lower than the single tree's. If the variance ratio is close to 1, the usual causes are: max_samples set well below 1.0 (each tree sees too little data and becomes noisier), n_estimators too small, or the base tree already heavily regularized so that σ² was small to begin with and there is little variance left to remove.

Reproducible Verification Checklist

The deliverable of this case is the pair of numbers — variance ratio and MAE change — plus the estimated ρ, reported together. That triple is what makes the bagging claim falsifiable: it states how much variance was removed, how close that is to the correlation floor, and whether the stability came at the cost of accuracy.