4. Data collection strategies for separating scatter from model error

Once a measurement system has been characterized and a model form has been proposed, the next question is unavoidable: when the residual between prediction and observation is large, how much of that residual is random scatter, and how much is systematic model form error? You cannot answer this question from a single dataset collected in a single campaign. The answer depends on how the data were collected. Four design levers — replication, randomization, blocking, and formal design of experiments — determine whether the two error components are identifiable at all.

The core idea is variance decomposition. If we denote the observed response as y, the model prediction as f(x; θ), and the residual as r = y − f(x; θ), then the residual variance can be partitioned into a pure-error (scatter) component and a lack-of-fit (model form) component. Replication gives us a direct handle on pure error; the other three levers protect the lack-of-fit estimate from being contaminated by nuisance variation that masquerades as model error.

Replication: the only direct estimate of pure scatter

Replication means running the same nominal input conditions multiple times and observing the spread of the response. This is the only design feature that yields an unbiased, assumption-free estimate of pure error at a given operating point. If you set a chamber to 60 °C and 40 % RH three times and record 12.1, 12.4, and 11.9 units, the sample standard deviation of those three readings is a direct scatter estimate — no model required.

Two practical cautions matter. First, true replication requires re-setting the conditions, not merely re-reading the sensor. Re-reading the same stabilized state measures instrument noise only; re-setting the chamber, re-loading the fixture, and re-running the protocol captures the full scatter that the model will actually face. Second, replication must be spread across the design space, not concentrated at one convenient point. A pure-error estimate at a single center point tells you nothing about whether scatter grows at the edges of the operating envelope.

A useful rule of thumb: allocate at least 15–20 % of the total run budget to replicates, distributed across three to five distinct operating points that span the region of interest. With ni replicates at point i, the pooled pure-error mean square is

MS_PE = Σ (n_i − 1) · s_i²  /  Σ (n_i − 1)

and the lack-of-fit mean square is obtained by subtracting the pure-error contribution from the residual sum of squares. The ratio MSLOF / MSPE is then compared against an F-distribution — this is the classic formal lack-of-fit test. Without replication, the denominator does not exist and the test cannot be performed.

Randomization: protecting against drift and order effects

Randomization means deliberately shuffling the order in which runs are executed, so that any uncontrolled time-varying factor — ambient temperature drift, tool wear, operator fatigue, reagent aging — is spread across the factor levels rather than confounded with them. If you run all low-temperature trials on Monday and all high-temperature trials on Tuesday, a Monday-to-Tuesday shift in the lab is indistinguishable from a temperature effect, and the model will absorb that shift as form error.

In practice, randomization is applied at the run level, not the replicate level. Replicates of the same condition should also be separated in time where feasible, so that short-term drift inflates the pure-error estimate honestly rather than being hidden. When true randomization is impossible — for example, when a hard-to-change factor such as a furnace setpoint requires hours to transition — use a split-plot structure and randomize within whole plots, then account for the split-plot error structure during analysis.

Blocking: isolating known nuisance variation

Blocking is the complement of randomization. When a nuisance factor is known and measurable but not of interest — a batch of raw material, a day of testing, a specific test rig, an operator — group runs into blocks so that the nuisance effect can be estimated and removed. Randomize within blocks, and ensure every factor level appears in every block (or use an appropriate incomplete-block design).

Blocking directly improves the separation of scatter from model error: by removing block-to-block variation from the residual, the remaining lack-of-fit signal is cleaner and the pure-error estimate is not inflated by batch effects. A common failure mode is to ignore a known batch structure, then observe a large residual and attribute it to model form error when it is actually a block effect. If the block factor is recorded, it can be tested; if it is not recorded, it is lost.

Design of experiments: making the two components identifiable

Replication, randomization, and blocking are the hygiene factors. The design of experiments (DOE) determines whether the factor space is sampled in a way that lets you distinguish curvature (model form error) from noise. A screening design such as a fractional factorial can estimate main effects efficiently but confounds two-factor interactions with each other; a response-surface design (central composite or Box–Behnken) adds axial and center points specifically to detect curvature and lack of fit.

For scatter-versus-model-error separation, the critical design features are: (1) center-point replicates to estimate pure error and test curvature; (2) axial or edge points to expose model form error where it typically appears; (3) blocking structure aligned with known nuisance factors; and (4) randomized run order within blocks. A design that omits center points cannot test lack of fit; a design that samples only the interior cannot reveal edge curvature.

A practical check before committing to a full campaign: run a small pilot with three center-point replicates and two edge points. If the replicate spread is comparable to the edge-to-center deviation, scatter dominates and a more elaborate model is not justified. If the edge deviation is several times the replicate standard deviation, model form error is present and the full DOE is warranted.

Putting the levers together

The figure below shows how the four levers map onto the residual decomposition and what each one protects.

Observed residual r = y − f(x; θ) total variation to explain Pure scatter MS_PE from replicates random, repeatable spread Model form error MS_LOF after removing PE systematic, structure-dependent Design levers and what they protect Replication → direct estimate of pure scatter (MS_PE) Randomization → prevents drift from confounding factors Blocking → removes known nuisance variation Design of experiments → center + edge points expose curvature

Read the diagram as a budget allocation problem. Replication buys you the denominator of the lack-of-fit test. Randomization and blocking protect the numerator from contamination. DOE ensures the numerator actually contains the curvature signal you are trying to detect. Skip any one of them and the separation becomes ambiguous: without replication you cannot test; without randomization you may attribute drift to the model; without blocking you may attribute batch effects to the model; without a curvature-aware design you may never see the model error at all.

A reproducible collection plan

Consider a thermal response model T(t) fitted to a heater test stand, with factors heater power (40–100 W) and ambient temperature (15–35 °C). A defensible collection plan for separating scatter from model error looks like this:

  • Design: face-centered central composite in two factors, with three center-point replicates and two replicates at each of the four axial points — 13 runs total.
  • Blocking: two blocks by test-stand session (morning / afternoon), with the center points split across blocks.
  • Randomization: run order randomized within each block; the three center-point replicates are placed in different positions in the order.
  • Replication: each replicate requires a full power-down, cool-down, and re-set, not a re-read of the stabilized state.
  • Analysis: fit the model, compute MSPE from the center-point replicates, compute MSLOF, and compare the ratio to F0.05 with the appropriate degrees of freedom.

Expected observable outcomes: if the replicate standard deviation at the center is, say, 0.4 °C and the axial-point residuals are 1.5–2.0 °C, the lack-of-fit ratio is large and model form error is confirmed — the model likely misses a nonlinearity near the envelope edges. If the axial residuals are within roughly twice the replicate standard deviation, scatter dominates and the current model form is adequate for the intended use. Typical failure symptoms to watch for: replicate spread that is suspiciously small (re-reads rather than re-sets), a block effect that appears as a factor effect (randomization not applied), or a lack-of-fit test that cannot be computed because no true replicates exist.

Do not pool replicates collected under different block conditions into a single pure-error estimate without checking for a block effect first. If the block effect is real, pooling inflates MSPE and masks genuine model form error — the test becomes conservative in the wrong direction.

The collection strategy is therefore not a preliminary formality before modeling; it is the mechanism that makes the scatter-versus-model-error distinction measurable. Replication supplies the yardstick, randomization and blocking keep the yardstick honest, and the DOE ensures the thing being measured is actually present in the data.

← Previous 📖 Contents Next →