3. Data quality assessment for industrial time series

Industrial time series are rarely clean. A flow transmitter may drop its 4–20 mA signal during a lightning strike, a thermocouple reading may freeze at its last value when the input card fails, and a vibration sensor may slowly drift as its piezoelectric element ages. Before any model, alarm, or KPI is built on top of this data, the data itself must be assessed. This chapter covers the five defect classes that dominate industrial historian data — missing values, outliers, sensor drift, stuck signals, and noise — and shows how to detect, quantify, and interpret each one with engineering context.

Why Data Quality Is an Engineering Problem, Not Just a Cleaning Step

A data quality assessment is not a cosmetic preprocessing step. In an industrial setting, each defect class has a physical cause and a physical consequence. A missing value in a temperature trend may mean a failed transmitter, a communication dropout, or a planned maintenance window. An outlier may be a real process upset (a genuine pressure spike) or a sensor fault (an electrical transient). Treating a real upset as noise and deleting it can hide the very event the monitoring system exists to catch. Treating a sensor fault as a real event can trigger a false trip. The assessment must therefore separate data defects from process events, and that separation requires engineering context: tag metadata, sampling rate, operating envelope, and knowledge of the physical asset.

The Five Defect Classes

Missing values appear as gaps in the timestamp sequence. In historian data they are often encoded as NaN, empty strings, or a sentinel value such as -9999. The first task is to distinguish true gaps (no sample recorded) from sentinel-encoded gaps (a sample recorded but invalid). The gap length distribution matters: isolated single-sample gaps are usually communication noise, while long contiguous gaps suggest instrument failure or shutdown.

Outliers are samples that deviate from the expected behavior of the signal. They can be point outliers (a single spike), contextual outliers (a value that is normal globally but abnormal given the recent trend), or collective outliers (a short segment that is anomalous as a group). A robust first pass uses the median absolute deviation (MAD):

MAD = median(|x_i - median(x)|)
modified_z_i = 0.6745 * (x_i - median(x)) / MAD
# flag |modified_z_i| > 3.5 as candidate outlier

The 0.6745 factor scales MAD to be comparable to the standard deviation for normally distributed data. Unlike the z-score, this statistic is not inflated by the outliers themselves, which is critical when a spike is present.

Sensor drift is a slow, monotonic deviation of the reading from the true value. It is common in pH probes, thermocouples, and pressure transmitters. Drift is not visible in a single window; it requires a reference. In practice, drift is detected by comparing the sensor against a redundant measurement, a laboratory sample, or a long-window rolling mean that should be stationary during steady-state operation.

Stuck signals occur when a sensor reports the same value (or a value changing only within the resolution of the instrument) for an implausibly long period. A stuck signal is not the same as a genuinely constant process: a tank level during a hold, or a temperature during a soak, can be legitimately flat. The discriminator is the expected variance of the tag during that operating mode, which is why stuck-signal detection must be conditioned on the process state.

Noise is the high-frequency, zero-mean fluctuation superimposed on the signal. It is characterized by its amplitude (standard deviation of the residual after removing trend), its frequency content (power spectral density), and its autocorrelation. Noise is not always a defect — some noise is inherent to the measurement — but its level determines the smallest change the signal can reliably resolve.

Noise Characterization in Practice

Noise is the one defect class that should be characterized rather than removed. The standard approach is to estimate the noise floor from a steady-state window — a period where the process is known to be constant. For a temperature tag sampled at 1 Hz over a 10-minute steady window, compute the residual after subtracting a low-order polynomial or a moving average, then report the residual standard deviation. If the residual σ is 0.8 °C and the process control band is ±2 °C, the noise consumes 40% of the band, which is a meaningful engineering finding: the control loop is operating close to its measurement limit.

Frequency-domain characterization adds value when the noise is not white. A power spectral density (PSD) estimate using Welch's method reveals whether the noise is broadband (random) or concentrated at a specific frequency (e.g., 50/60 Hz mains interference, or a pump rotation frequency). A peak at 25 Hz on a flow signal, for example, points to a reciprocating pump rather than an electrical issue, and the fix is mechanical (pulsation dampener) rather than electrical (shielding).

A Reproducible Assessment Example

Consider a pressure transmitter tagged PT-101 on a separator vessel, sampled at 1 Hz, with a 4–20 mA signal scaled to 0–100 bar. The assessment proceeds as follows:

1. Ingest and align. Load the historian extract, confirm the timestamp index is monotonic and gap-free at 1 Hz. Convert the raw mA value to bar using the tag's range metadata. Check for sentinel values (-9999, NaN).
2. Gap analysis. Count gaps and their lengths. Suppose the extract shows 12 gaps, 10 of which are single-sample and 2 of which are 45 minutes long. The single-sample gaps are likely communication retries; the 45-minute gaps coincide with a documented maintenance window and should be marked as planned, not missing.
3. Outlier detection. Compute the rolling median over a 5-minute window and the MAD over the same window. Flag samples with modified z-score above 3.5. Suppose 8 samples are flagged, all within a 30-second burst during a known upstream valve slam. These are real process events, not sensor faults — they should be retained and annotated, not deleted.
4. Stuck-signal check. Compute the rolling standard deviation over a 15-minute window. If the standard deviation is below the transmitter resolution (0.05 bar) for more than 30 minutes while the process is in a flowing mode, flag as stuck. If the same flat period occurs during a documented hold, do not flag.
5. Drift check. Compare the tag against a redundant transmitter PT-102 on the same vessel. Compute the difference over a 7-day window. A slowly increasing difference of 0.3 bar/day indicates drift on one of the two devices; a third reference (a manual gauge reading) resolves which one.
6. Noise characterization. Select a 10-minute steady window, compute the residual σ after detrending. Report the noise floor and, if needed, the PSD to identify dominant frequencies.

The output is a per-tag quality report. A compact form is shown below; in practice this is generated programmatically and stored alongside the tag metadata.

From Assessment to Decision

The purpose of the assessment is not to produce a cleaned dataset for its own sake, but to support a decision: can this tag be used for its intended purpose? A tag with 2% missing values, a stable noise floor, and no drift is fit for KPI reporting. A tag with 15% missing values and a confirmed drift of 0.3 bar/day is not fit for custody transfer or safety interlocks until the instrument is repaired and recalibrated. The assessment report should therefore end with an explicit recommendation — use, repair, or reject — tied to the specific engineering application, not a generic quality score. This decision-oriented framing is what distinguishes industrial data quality assessment from generic data cleaning.