4. Data cleaning and imputation for industrial sensor streams

Industrial sensor streams rarely arrive as the tidy, evenly spaced tables that textbook examples assume. A vibration probe on a conveyor gearbox may report at 1 kHz for hours, then drop to 4 Hz when the edge gateway throttles under load. A thermocouple loop can freeze on its last value when the transmitter loses its bus connection, producing a long run of identical readings that look valid but carry no information. Pressure spikes from a pump start can exceed the physical range of the transducer, and a scheduled maintenance window can leave a six-hour gap in every tag on the line. Before any anomaly detection, KPI calculation, or predictive model can be trusted, these defects must be identified and handled deliberately.

This chapter focuses on three defect classes that dominate industrial time-series work: missing values, outliers, and irregular sampling intervals. The goal is not to produce a perfectly clean dataset — that is rarely possible — but to make every cleaning decision explicit, reproducible, and defensible to the engineers who own the equipment.

Characterizing the Defects Before Touching Them

The first mistake in cleaning is to treat all gaps as equivalent. In practice, missingness has distinct signatures, and each signature implies a different remedy.

• Point gaps — isolated single-sample dropouts, often caused by a dropped packet or a momentary sensor fault. These are usually safe to interpolate.
• Burst gaps — short runs of missing samples, typically from a brief network outage or a gateway restart. Interpolation is acceptable if the burst is short relative to the process time constant.
• Long gaps — hours or days of missing data from shutdowns, maintenance, or sensor replacement. Imputing these is dangerous; the correct action is usually to mark them as invalid and exclude them from downstream aggregation.
• Frozen values — repeated identical readings that are technically present but physically implausible. These are the most insidious because they pass a simple null check.

A useful first step is to compute, per tag, the fraction of missing samples, the maximum consecutive gap length, and the longest run of identical values. These three numbers alone tell you whether a tag is usable, needs light repair, or should be quarantined.

import pandas as pd
import numpy as np

def gap_profile(series, freq="1s"):
    s = series.asfreq(freq)
    missing = s.isna()
    # longest consecutive run of missing samples
    runs = missing.astype(int).groupby((~missing).cumsum()).cumsum()
    longest_gap = runs.max()
    # longest run of identical non-null values
    changed = s.ne(s.shift())
    frozen_runs = (~changed).astype(int).groupby(changed.cumsum()).cumsum()
    longest_frozen = frozen_runs.max()
    return {
        "missing_frac": missing.mean(),
        "longest_gap_samples": int(longest_gap),
        "longest_frozen_samples": int(longest_frozen),
    }

Running this on a real tag set immediately separates healthy channels from problematic ones. A tag with a missing fraction below 0.5% and a longest gap under 10 samples is a candidate for interpolation. A tag with a 40% missing fraction or a frozen run of 3,000 samples needs a conversation with the instrumentation team before any imputation.

Outliers: Physical Bounds and Statistical Bounds

Outlier handling in industrial data should always start with physical plausibility. A pressure transmitter rated 0–16 bar cannot legitimately report 240 bar; a temperature sensor on a bearing cannot jump from 60 °C to 400 °C in one second. Encoding these limits as hard bounds catches gross errors cheaply and without statistical assumptions.

Beyond hard bounds, statistical methods help with subtler excursions. For sensor streams, a robust rolling median with a median absolute deviation (MAD) band is usually preferable to a global z-score, because it adapts to slow drift and is insensitive to the very spikes you are trying to detect.

def flag_outliers(s, window="15min", k=6.0):
    med = s.rolling(window, center=True, min_periods=5).median()
    mad = (s - med).abs().rolling(window, center=True, min_periods=5).median()
    # 1.4826 scales MAD to a Gaussian-equivalent sigma
    robust_z = (s - med) / (1.4826 * mad.replace(0, np.nan))
    return robust_z.abs() > k

The choice of k is a trade-off. A value of 6 is conservative and flags only extreme excursions; a value of 3 will flag far more points and may remove legitimate process transitions such as a valve slam or a compressor surge. The right threshold depends on whether the downstream task is fault detection (where missing a real event is costly) or KPI reporting (where false alarms are costly).

Irregular Sampling Intervals

Many industrial historians store data as timestamped events rather than fixed-interval samples. A tag may report only when its value changes by more than a deadband, producing intervals that range from milliseconds to hours. Feeding such a series directly into a model that assumes uniform spacing — most classical time-series models, many FFT-based analyses, and virtually all fixed-window aggregations — produces misleading results.

The standard remedy is resampling to a regular grid. The choice of grid frequency should reflect the fastest dynamics you care about, not the fastest data you happen to have. For a bearing temperature that changes over minutes, a 1-minute grid is ample; for a vibration envelope, 1-second or finer may be required.

# Resample to a regular 1-minute grid
regular = (
    raw.set_index("timestamp")["value"]
       .resample("1min")
       .agg(["mean", "count", "max"])
)
# count reveals how many raw samples backed each grid point
sparse = regular["count"] < 3

The count column is essential. A grid point backed by zero raw samples is an imputed value; one backed by a single sample is weakly supported; one backed by many samples is reliable. Carrying this count forward lets downstream code weight or filter observations by evidence strength rather than treating all grid points as equally trustworthy.

Choosing an Imputation Strategy

Imputation is where cleaning decisions have the largest downstream impact, so the method should match the gap type and the process physics.

For short gaps, time-weighted interpolation is preferable to simple forward-fill because it respects the actual timestamps. Pandas provides this directly:

s = s.interpolate(method="time", limit=10, limit_area="inside")

The limit=10 argument caps interpolation at 10 consecutive samples, and limit_area="inside" prevents extrapolation beyond the first and last valid observations — a critical safeguard against fabricating data at the edges of a record.

Putting It Together: A Cleaning Pipeline

The diagram below shows how the three defect classes flow through a single cleaning pipeline, with the decision points that determine whether a sample is repaired, flagged, or excluded.

The key design principle is that cleaning produces two outputs, not one: the repaired value stream and a parallel metadata stream recording what was done. Downstream code can then filter on is_imputed or weight by sample_count without re-deriving the cleaning logic.

Verification and Common Failure Modes

After cleaning, verify three things. First, confirm that the missing fraction after imputation is zero only where you intended it to be — long gaps should still be NaN. Second, check that the distribution of imputed values is not artificially narrow; if every imputed point equals the rolling median, your interpolation is masking variance. Third, plot the cleaned series against the raw series for a sample window and inspect visually; automated checks miss systematic errors that the eye catches immediately.

Two failure modes recur in practice. The first is over-imputation: filling long gaps with interpolated values that create smooth, plausible-looking curves where no data exists. This is especially dangerous in KPI reporting, where it inflates uptime and hides equipment problems. The second is outlier removal without flagging: clipping spikes in place so that the original event is unrecoverable. Both are avoided by the same discipline — never overwrite the raw value, always add a flag column, and let the downstream consumer decide.

With a defensible cleaning pipeline in place, the next stage — feature engineering and windowed aggregation — can proceed on a foundation that reflects what the sensors actually measured, and what they did not.