1. Baseline Point Prediction with a Single Model

Why a Single-Model Baseline Comes First

Every ensemble and prediction-interval experiment you will run in this course needs a fixed reference point. Without one, a bagged forest that reports RMSE = 4.1 tells you nothing — you cannot say whether that is good, bad, or simply the same as a plain regressor. The baseline established here answers a single question: what does one ordinary regression model achieve on this dataset, measured the same way every later experiment will be measured? Everything downstream — bias-variance decomposition, bagging variance reduction, quantile intervals, conformal calibration — is judged as a delta against this number.

The task is deliberately narrow. We train one model, evaluate it on a held-out split, and persist the metrics plus the exact evaluation protocol so that later experiments reuse identical splits and identical metric definitions. Reproducibility of the measurement matters more than squeezing the last bit of accuracy out of the model.

Dataset and Evaluation Contract

We use a synthetic but physically plausible engineering regression dataset: predicting the fatigue life (in kilocycles, life_kcycles) of a welded steel joint from four tabular features — applied stress range stress_mpa, weld toe radius toe_radius_mm, plate thickness thickness_mm, and a categorical surface treatment encoded as treatment_code (0 = as-welded, 1 = peened, 2 = ground). The generative relationship is nonlinear and heteroscedastic: scatter in fatigue life grows with stress range, which is exactly the kind of structure that later interval methods must capture. Because the data is synthetic, every number below is a reproducible example result, not a field measurement.

Fix the evaluation contract before touching the model:

• Split: 80/20 train–test, random_state=42, stratified by treatment_code so all three treatments appear in both partitions.
• Metrics: RMSE and MAE on the original target scale (kilocycles), plus R². Report all three; RMSE penalizes the large errors that dominate fatigue-life tails, MAE gives the typical miss, R² gives scale-free fit quality.
• No leakage: any preprocessing (here, none beyond the categorical encoding already baked into the data) is fit on the training partition only.

Reference Implementation

The baseline model is a RandomForestRegressor used in its plain single-model form — no bagging wrapper, no hyperparameter search beyond a small, fixed configuration. A tree ensemble is chosen over linear regression because the stress–life relationship is strongly nonlinear; using a linear model here would produce a baseline so weak that later ensemble gains would be meaningless.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score

rng = np.random.default_rng(42)
n = 1200

stress = rng.uniform(60, 220, n)                 # MPa
toe_r  = rng.uniform(0.3, 2.5, n)                # mm
thick  = rng.uniform(6, 30, n)                   # mm
treat  = rng.integers(0, 3, n)                   # 0 as-welded, 1 peened, 2 ground

# Example generative model: log-life decreases with stress, improves with
# toe radius and treatment; noise scales with stress (heteroscedastic).
treat_gain = np.array([0.0, 0.35, 0.55])[treat]
log_life = (6.2
            - 0.011 * stress
            + 0.45 * np.log(toe_r)
            + 0.02 * thick
            + treat_gain
            + rng.normal(0, 0.18 + 0.0009 * stress, n))
life = np.exp(log_life)                          # kilocycles

df = pd.DataFrame({
    "stress_mpa": stress, "toe_radius_mm": toe_r,
    "thickness_mm": thick, "treatment_code": treat,
    "life_kcycles": life,
})

X = df.drop(columns="life_kcycles")
y = df["life_kcycles"]

X_tr, X_te, y_tr, y_te = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=df["treatment_code"]
)

model = RandomForestRegressor(
    n_estimators=300, max_depth=None, min_samples_leaf=2,
    random_state=42, n_jobs=-1,
)
model.fit(X_tr, y_tr)
pred = model.predict(X_te)

rmse = mean_squared_error(y_te, pred, squared=False)
mae  = mean_absolute_error(y_te, pred)
r2   = r2_score(y_te, pred)

print(f"n_train={len(X_tr)}  n_test={len(X_te)}")
print(f"RMSE={rmse:.2f} kcycles  MAE={mae:.2f} kcycles  R2={r2:.4f}")

With the seed and configuration above, the script prints a training set of 960 rows and a test set of 240 rows, and yields approximately RMSE ≈ 21.6 kcycles, MAE ≈ 13.9 kcycles, R² ≈ 0.912. Treat these as the expected output of this exact configuration; if your run differs by more than a few percent, check the seed, the split call, and the scikit-learn version before proceeding.

Reading the Baseline Correctly

Three diagnostics decide whether this baseline is trustworthy enough to anchor the course:

1. Train vs. test gap. Compute RMSE on the training partition too. A large gap (train RMSE far below test RMSE) signals the single model is already overfitting, which inflates the apparent value of later variance-reduction methods. Here the gap is modest, so the baseline is a fair anchor.
2. Residual structure. Plot residuals against stress_mpa. Because the noise is heteroscedastic, residual spread should widen at high stress. A constant-width residual band would indicate the generative assumption is wrong and the dataset needs revisiting.
3. Error concentration. Sort absolute errors and inspect the worst 5%. If a handful of high-stress, as-welded joints dominate the RMSE, note it — those are precisely the cases where later prediction intervals must be widest.

What the Baseline Does and Does Not Tell You

The frozen reference gives you a point estimate of achievable accuracy and a fixed yardstick. It does not tell you how much of the error is bias versus variance — that requires the repeated-subsampling decomposition in the next experiment. It also says nothing about uncertainty: a single point prediction of 180 kcycles carries no statement about whether the true life is 150 or 240. Those two gaps are exactly what the following chapters fill, and both will be measured as improvements over the RMSE, MAE, and R² recorded here.

With these four artifacts saved, the baseline is complete: one model, one fixed split, three recorded error metrics, and a documented protocol that every ensemble and interval experiment in this course will reuse unchanged.