2. Process data characteristics

Before any monitoring statistic is computed, the quality and structure of the underlying data determine whether that statistic will be trustworthy. Industrial process data are not the clean, independent, identically distributed samples assumed by textbook statistics. They arrive from hundreds or thousands of instruments wired into a distributed control system (DCS), historian, or edge gateway, each with its own sampling behavior, noise profile, and failure modes. This chapter characterizes the seven properties that most strongly shape multivariate monitoring design: sensor network topology, sampling rates, autocorrelation, cross-correlation, nonstationarity, missing values, and outliers.

Sensor Networks and Data Provenance

A typical refinery or chemical plant exposes data through a layered architecture. Field instruments (thermocouples, pressure transmitters, Coriolis flow meters, gas chromatographs) connect to I/O modules on controllers. Controllers publish to a supervisory network, and a historian archives the tags. Monitoring models rarely read the controller directly; they read the historian or an OPC UA endpoint. This matters because each layer introduces its own timestamping convention, deadband, and compression.

Two tags that appear adjacent in a data table may have completely different provenance. A temperature tag might be sampled every second at the controller but archived only on a 1% change deadband, while a laboratory analyzer result arrives once per shift. Treating them as a uniform matrix column is a common and costly mistake.

Sampling Rates and Resampling

Sampling rates in industrial datasets span several orders of magnitude. Fast loops (vibration, compressor surge) may run at 1–10 kHz. Regulatory control tags are typically 1 Hz. Slow quality variables (composition, density) may be 15-minute or 8-hour averages. Multivariate methods such as PCA and PLS require a rectangular matrix with a common time base, so resampling is unavoidable.

The safe direction is to downsample fast tags to the slowest relevant rate using an aggregation that preserves process meaning. For flow and production rates, use the mean over the interval. For temperatures subject to short excursions, consider the maximum or a percentile. Never upsample a slow analyzer by forward-filling and then claim the resulting model has fast dynamics — the interpolation fabricates information that the instrument never measured.

# Example: align three tags to a 1-minute grid
# temp_fast: 1 Hz, flow_fast: 1 Hz, lab_comp: 15 min
df = df.resample('1min').agg({
    'temp_fast': 'mean',
    'flow_fast': 'mean',
    'lab_comp':  'last'   # forward-hold until next lab result
})
df['lab_comp'] = df['lab_comp'].ffill(limit=15)

Autocorrelation

Process variables are almost never independent in time. A reactor temperature at time t is strongly correlated with its value at t−1, t−2, and beyond. This temporal dependence is autocorrelation, quantified by the autocorrelation function (ACF) and summarized by the lag-1 coefficient ρ₁. For a first-order process with time constant τ and sampling interval Δt, ρ₁ ≈ exp(−Δt/τ).

Autocorrelation has two consequences for monitoring. First, it inflates the effective sample size: 10,000 autocorrelated samples may carry the information of only a few hundred independent ones, so control limits computed from the nominal sample count will be far too tight. Second, it slows fault detection because a shift must persist long enough to overcome the inertia of the correlation structure. Dynamic PCA, dynamic PLS, and EWMA-based statistics exist precisely to address this.

Cross-Correlation

Variables are also correlated with each other, often with a time lag. A feed flow change propagates to column pressure after a few seconds and to bottom composition after many minutes. Cross-correlation functions reveal these lags and are the basis for selecting dynamic lag structures in monitoring models. Ignoring cross-correlation produces models that flag normal operating transitions as faults, because the variables move together in a way the static model has never seen.

Nonstationarity

Industrial processes drift. Catalysts deactivate, heat exchangers foul, ambient conditions change, and production rates shift between campaigns. A model trained on January data may show a persistent T² excursion in March even though nothing is wrong. Nonstationarity appears as slow changes in mean, variance, or correlation structure.

Practical remedies include recursive adaptation of the mean and covariance, moving-window PCA, and explicit modeling of operating modes. The choice depends on whether the drift is a genuine process change (adapt) or a fault (do not adapt). A useful diagnostic is to plot the model's Hotelling T² and SPE over months; a gradual upward creep suggests drift, while a step change suggests a fault.

Missing Values and Outliers

Missing values arise from communication dropouts, instrument maintenance, and analyzer cycle gaps. Simple listwise deletion destroys too much data when gaps are frequent. Mean imputation biases variance downward. Model-based imputation (PCA reconstruction, k-nearest neighbors, or expectation-maximization) preserves covariance structure and is preferred when the missing fraction is below roughly 10–20%.

Outliers are equally common and more dangerous. They come from sensor spikes, frozen values, and unit errors. A frozen transmitter reports the same value for hours; a spike reports a physically impossible jump. Both corrupt covariance estimates. Robust alternatives — median absolute deviation, Huber loss, or robust PCA — reduce their influence. A practical rule is to flag any sample whose SPE exceeds a high percentile for more than a few consecutive samples before declaring a fault.

Data Quality Assessment Workflow

A reproducible quality check before modeling should cover: tag inventory and provenance, timestamp alignment, missing-value fraction per tag, outlier fraction per tag, ACF at lag 1, and cross-correlation peak lag between key variable pairs. The table below summarizes typical acceptance thresholds used in practice.

The SVG below summarizes how these characteristics interact along the data path from field instrument to monitoring model, and where each quality issue typically enters.

Practical Implications for Monitoring Design

The characteristics above are not independent. Autocorrelation and cross-correlation interact: a dynamic model that captures both will have far fewer false alarms than a static one. Nonstationarity and missing values interact: adaptive models that update on incomplete data can drift in the wrong direction. Outliers and cross-correlation interact: a single-tag spike is easy to detect, but a correlated shift across several tags may look like a legitimate process transition.

A defensible workflow is to characterize each property quantitatively, document the chosen treatment, and validate the monitoring model on a held-out period that includes at least one known fault and one known operating-mode change. If the model cannot distinguish between them, the data characterization step was insufficient.