3. Model form error characterization

Before a calibration model can be trusted for prediction or for test-scatter separation, we must characterize how it fails to match the reference data. Model form error is not a single quantity; it is a family of structured residual patterns. Four signatures dominate engineering practice: bias (a systematic offset), lack of fit (residual structure the model cannot explain), curvature (a systematic bow in the residuals across the operating range), and heteroscedasticity (residual variance that changes with the predictor). Each has a distinct diagnostic footprint and a distinct remedy. Confusing them leads to wasted effort — for example, adding polynomial terms to cure a bias that is actually a sensor zero offset, or tightening tolerances when the real problem is variance that grows with load.

The working definition throughout this chapter is the residual convention e_i = y_i − ŷ_i, where y_i is the reference (calibration or validation) value and ŷ_i is the model prediction at the same input. Positive residuals mean the model under-predicts; negative residuals mean it over-predicts. All plots, signs, and thresholds below follow this convention consistently.

Bias: a constant or slowly varying offset

Bias is the mean of the residuals over a defined operating window. A model with bias b predicts ŷ = f(x) + b when the truth is f(x). In calibration data, bias is often absorbed by the intercept term, which is why it hides there and reappears in validation. The practical test is simple: compute the mean residual ē and its standard error s_e/√n. If |ē| exceeds roughly two standard errors, the offset is statistically real. But statistical significance is not the same as engineering significance — compare ē against the tolerance band that matters for the application. A 0.3 % offset may be irrelevant on a 10 kN load cell and fatal on a 50 N one.

Bias can be constant across the range or drift with a secondary variable (temperature, batch, fixture). The latter is conditional bias and is far more dangerous because it survives simple re-zeroing. Plot residuals against every candidate nuisance variable, not just the primary predictor; a flat residual-versus-x plot with a sloped residual-versus-temperature plot is a classic conditional-bias fingerprint.

Lack of fit: structure the model cannot explain

Lack of fit asks whether the chosen functional form is adequate. The standard tool is the decomposition of residual sum of squares into pure error (from replicate measurements at identical inputs) and lack-of-fit (the remainder). With n observations, p parameters, and r distinct input levels with replicates, the F-statistic is

SS_LOF = SS_res − SS_pe
df_LOF = (n − p) − (n − r)
df_pe  = n − r
F = (SS_LOF / df_LOF) / (SS_pe / df_pe)

A large F (p < 0.05) means the model's systematic departure from the data exceeds what measurement noise alone can produce. Crucially, lack of fit is only detectable when replicates exist. Without replicates, you cannot separate model error from noise, and the honest statement is "lack of fit untested," not "model adequate." In validation data, lack of fit appears as residuals that are small in magnitude but strongly autocorrelated or clustered by operating region — the model is precise but wrong in shape.

Curvature: the shape of the systematic departure

Curvature is a specific, directional form of lack of fit: residuals form a consistent arc — typically positive in the middle of the range and negative at both ends (concave-down truth fitted by a line), or the reverse. It is diagnosed by plotting residuals against the fitted value or the predictor and looking for a monotone trend in the running mean. A quantitative check is to fit a quadratic term to the residuals and test its coefficient; a significant quadratic coefficient with a non-significant cubic is the signature of simple curvature.

Curvature matters because it is range-dependent. A model that is excellent near the calibration midpoint can be badly wrong at the extremes, which is exactly where safety margins and acceptance limits usually sit. When curvature is present, extrapolation beyond the calibrated range is indefensible, and even interpolation near the edges carries elevated uncertainty. The remedy is rarely "add a quadratic term and move on" — first ask whether the curvature is physical (a real nonlinearity in the sensor or process) or an artifact (a mis-specified reference, a fixture compliance, a saturation effect). Physical curvature should be modeled; artifact curvature should be fixed at the source.

Heteroscedasticity: variance that changes with the predictor

Heteroscedasticity means the residual standard deviation is not constant. In calibration data it commonly grows with the magnitude of the predictor (multiplicative noise) or with the square of the fitted value. Ordinary least squares assumes constant variance; when that assumption fails, the parameter estimates remain unbiased but their standard errors are wrong, confidence intervals are misleading, and weighted least squares or a variance-stabilizing transform becomes necessary.

Diagnose it by plotting |e_i| or e_i² against ŷ_i and fitting a trend, or by computing the ratio of residual variance in the top and bottom thirds of the range. A ratio above roughly 3:1 is a practical red flag. The Breusch–Pagan or White test gives a formal p-value. In validation data, heteroscedasticity shows up as residuals that are tight at low load and wide at high load — a pattern that a single global RMSE will completely hide, because the global RMSE is dominated by the high-variance region.

A single aggregate metric such as RMSE or R² cannot distinguish these four error types. Two models with identical RMSE can have completely different residual structures — one biased, one curved, one heteroscedastic. Always inspect the residual plots before accepting any aggregate number.

Reading the four signatures together

The figure below maps each signature to its residual pattern, its primary diagnostic, and the engineering action it implies. The key insight is that the four are not mutually exclusive: a real calibration dataset often shows mild curvature plus growing variance, and the correct response is to address them in order — fix bias first (it is cheap and often a setup issue), then assess curvature (it determines whether the functional form is defensible), then handle heteroscedasticity (it determines how uncertainty is reported), and finally test lack of fit with replicates to confirm the model is adequate.

Residual Signatures and Their Engineering Response Bias flat offset, mean ≠ 0 test: |ē| vs 2·SE action: re-zero / intercept check nuisance variables Lack of Fit structured, clustered residuals test: F = (SS_LOF/df)/(SS_pe/df) needs replicates action: revise functional form Curvature consistent arc across range test: quadratic term on residuals range-dependent error action: model or fix source Heteroscedasticity variance grows with ŷ test: |e| vs ŷ trend, BP test global RMSE hides it action: WLS / transform Recommended diagnostic order when multiple signatures coexist 1. Bias cheap, often setup 2. Curvature is the form defensible? 3. Heteroscedasticity how to report uncertainty 4. Lack of fit confirm with replicates Residual convention: e = y − ŷ (positive = under-prediction). Each stage removes a confounder before the next is assessed. Aggregate metrics (RMSE, R²) are reported only after all four signatures have been inspected. Validation residuals must be examined separately from calibration residuals — intercept absorption hides bias in calibration.

Worked example: separating the signatures in a load-cell calibration

Consider a 10 kN load cell calibrated at six reference loads (0, 2, 4, 6, 8, 10 kN) with three replicates each, giving 18 calibration points. A linear model ŷ = a + b·x is fitted. The residual plot shows a mean residual of +0.04 kN, a mild concave-down arc peaking near 5 kN, and residual spread that roughly triples between the low and high ends. The following sequence is defensible:

  1. Bias check. Compute ē = +0.04 kN and s_e = 0.06 kN, so SE = 0.06/√18 ≈ 0.014 kN. The ratio 0.04/0.014 ≈ 2.8 exceeds 2, so the offset is real. Before adjusting the model, check the zero-load reading and the fixture preload — a 0.04 kN preload is a plausible physical cause and should be corrected at the source rather than absorbed into the intercept.
  2. Curvature check. Fit e = c₀ + c₁·x + c₂·x². Suppose the quadratic coefficient yields p = 0.01 while the cubic term is non-significant. The curvature is real and range-dependent. Decide whether it is physical (load-cell nonlinearity, typically specified as a fraction of full scale) or an artifact (fixture compliance that grows with load). If physical, a quadratic term or a piecewise-linear model is justified; if artifact, fix the fixture.
  3. Heteroscedasticity check. Compute residual variance in the 0–3 kN and 7–10 kN bands. A 3:1 ratio triggers weighted least squares with weights w_i = 1/ŷ_i² (for multiplicative noise) or a log transform if the noise is proportional. Report uncertainty per operating band rather than as a single global RMSE.
  4. Lack-of-fit test. With three replicates at each load, df_pe = 18 − 6 = 12. After the above corrections, compute F. If F is non-significant, the revised model is adequate; if still significant, the functional form needs further revision.
The order matters because each step changes the residuals seen by the next. Removing a bias first prevents it from inflating the apparent curvature; stabilizing variance before the lack-of-fit test prevents high-load points from dominating the F-statistic. Skipping the order produces false positives and wasted model complexity.

Validation data: where the signatures reappear

Calibration residuals are optimistically small because the model was fitted to them. Validation residuals are the honest test, and the four signatures behave differently there. Bias that was absorbed by the intercept in calibration reappears as a mean offset in validation. Curvature that was mild in-sample can be severe at the range edges. Heteroscedasticity is often more visible because validation points are not weighted by the fitting procedure. Lack of fit is hardest to detect in validation without replicates, so the practical rule is: characterize lack of fit on calibration data with replicates, and confirm the other three on independent validation data.

A compact characterization record for each model should capture, at minimum: mean residual and its standard error (bias), the quadratic-term p-value and the residual-versus-fitted plot (curvature), the variance ratio across the operating range and the chosen weighting (heteroscedasticity), and the lack-of-fit F and its degrees of freedom (adequacy). These four entries, taken together, tell a reviewer far more than any single goodness-of-fit number and directly determine whether the model is fit to support test-scatter separation.

← Previous 📖 Contents Next →