2. Statistical foundations

Before any scatter can be separated from model form error, we need a statistical vocabulary that lets us say precisely where the scatter comes from and how much of it each source contributes. This chapter builds that vocabulary: variance components, repeatability, reproducibility, and the crucial distinction between random scatter and systematic model error. The goal is not abstract statistics but a working framework you can apply to a real measurement or simulation dataset and get actionable numbers out of it.

Why variance components matter

Any measured or simulated response y can be decomposed into a deterministic part and a stochastic part. In the simplest additive model:

y_ij = μ + a_i + e_ij

Here μ is the grand mean, a_i is the effect of condition or unit i (a factor such as operator, fixture, batch, or model configuration), and e_ij is the residual error on replicate j within condition i. The variance of y is then the sum of the variances of these components, provided they are independent:

Var(y) = σ²_a + σ²_e

This is the core idea of variance components: total observed variance is partitioned into named, physically interpretable sources. When the components are estimated from data, they tell you how much of the scatter is due to changing conditions versus how much is irreducible noise at a fixed condition. That split is exactly what you need before you can decide whether a discrepancy between model and test is random or systematic.

Repeatability: scatter under fixed conditions

Repeatability is the variation observed when everything is held constant: same operator, same fixture, same instrument, same environmental conditions, same model configuration, repeated in quick succession. In the model above, repeatability is captured by σ²_e. It is the "shortest-range" scatter and represents the noise floor of your process.

In practice, repeatability is estimated from replicated runs. If you have n replicates at a single condition, the repeatability variance is the sample variance:

s²_r = Σ(y_j − ȳ)² / (n − 1)

For a quick sanity check with five replicates [10.2, 10.5, 9.9, 10.1, 10.3], the mean is 10.20 and s²_r ≈ 0.05, so the repeatability standard deviation is about 0.22 in the response's units. That number is your baseline: any scatter smaller than it is not meaningful.

Reproducibility: scatter across changed conditions

Reproducibility is the variation observed when a nominally "same" measurement is repeated under changed conditions — different operators, different fixtures, different labs, different model versions, or different days. It is always larger than or equal to repeatability because it includes repeatability plus the effect of the changed factor. In the two-level model, reproducibility variance is σ²_a + σ²_e when a_i represents the changed condition.

The practical consequence is important: if you only ever repeat under fixed conditions, you will underestimate the true scatter and may wrongly attribute a real effect to "noise." Conversely, if you pool data across conditions without separating the components, you will overstate the noise floor and may dismiss a genuine systematic effect as random.

Random scatter versus systematic model error

These two are fundamentally different in structure, and the distinction drives every downstream decision.

• Random scatter is zero-mean, uncorrelated, and shrinks as 1/√n when you average. It is described by a variance component and is irreducible at a given condition.
• Systematic model error is a bias: a repeatable offset or trend between model prediction and reference truth that does not average away. It is a function of the input, not a distribution around zero.

A useful diagnostic is the residual structure. Compute residuals r = y_test − y_model across the operating range. If the residuals look like white noise centered on zero with variance σ²_e, the model form is adequate and the scatter is random. If the residuals show a consistent sign, a slope, a curvature, or a condition-dependent offset, that pattern is systematic model form error — it will persist no matter how many replicates you average.

Estimating components from a nested design

A reproducible way to separate the sources is a simple nested (hierarchical) design: a conditions, n replicates each. The one-way random-effects ANOVA gives mean squares MS_between and MS_within, from which:

σ²_e  = MS_within
σ²_a  = (MS_between − MS_within) / n

If σ²_a comes out negative, it is conventionally set to zero — the between-condition effect is not distinguishable from noise at this sample size. Report the repeatability standard deviation √σ²_e and the reproducibility standard deviation √(σ²_a + σ²_e) alongside the estimate, because the components are what make the numbers interpretable.

Worked example (illustrative)

Suppose a fixture is measured at three nominal loads with four replicates each, and the within-condition mean square is MS_within = 0.04 while the between-condition mean square is MS_between = 0.28. Then σ²_e = 0.04 (repeatability SD 0.20) and σ²_a = (0.28 − 0.04)/4 = 0.06 (reproducibility SD √0.10 ≈ 0.32). The gap between 0.20 and 0.32 is the condition effect. If the model-vs-test residual mean at each load is, say, +0.15, +0.14, +0.16, that consistent positive offset is systematic and cannot be explained by either variance component — it is model form error. These figures are illustrative, not measured results.

What to report and how to use it

For every scatter-separation study, report the repeatability SD, the reproducibility SD, the number of conditions and replicates, and the residual mean and trend across the operating range. These four items let a reviewer reproduce your conclusion. The repeatability SD sets the threshold below which a discrepancy is not credible; the reproducibility SD sets the threshold for condition-driven variation; the residual structure decides whether the remaining gap is random or systematic. Only after that separation is it legitimate to say a model has form error rather than noise — and that is the foundation the next stages of the course build on.