3. Data profiling and baseline characterization
Before any anomaly detector, rule engine, or remediation workflow can be trusted, the data itself must be understood. Data profiling and baseline characterization answer a deceptively simple question: what does normal look like for this signal, tag, or event stream? In industrial environments, "normal" is rarely a single number. A reactor temperature may legitimately swing 15 °C across a batch cycle, a vibration sensor may show a step change after a bearing replacement, and a flow transmitter may report a frozen value for hours when a line is idle. Profiling converts raw historian and event data into statistical summaries, distribution models, and temporal patterns that later stages use as reference baselines.
Statistical Summaries for Time-Series Tags
The first profiling pass computes descriptive statistics over a defined observation window — typically 30 to 90 days for continuous processes, or a fixed number of completed batches for batch processes. For each tag, capture:
- Central tendency and spread: mean, median, standard deviation, interquartile range (IQR), and coefficient of variation.
- Extremes and shape: min, max, skewness, kurtosis, and percentiles (P1, P5, P95, P99).
- Data quality counters: sample count, gap count, longest gap duration, out-of-range count, and stale-value count.
- Rate-of-change statistics: first-difference mean and standard deviation, useful for detecting spikes independent of absolute level.
A practical example: a historian tag TI-2041.PV (reactor jacket temperature) sampled at 1 Hz over 60 days yields 5,184,000 raw samples. After removing 12,400 samples flagged as BAD quality and 3 gaps longer than 10 minutes, the cleaned set has a mean of 142.3 °C, median 141.8 °C, standard deviation 6.7 °C, and P99 of 158.1 °C. The gap count and longest gap (47 minutes) immediately flag a data acquisition issue that must be resolved before the baseline is considered valid.
Distribution Fitting and Goodness-of-Fit
Industrial time-series data rarely follows a normal distribution. Sensor noise may be Gaussian, but process variables often exhibit skew, multimodality, or heavy tails. Distribution fitting identifies a parametric family that describes the data well enough to set statistically defensible thresholds.
Common candidates for continuous industrial signals include:
| Distribution | Typical Use Case | Key Parameters |
|---|---|---|
| Normal | Sensor noise, stable controlled variables | μ, σ |
| Log-normal | Concentrations, particle sizes, positive-only measurements | μ, σ of log(x) |
| Weibull | Time-to-failure, equipment degradation | shape k, scale λ |
| Gamma | Accumulated flow, residence times | shape α, rate β |
| Mixture models | Multimodal operation (e.g., day/night, batch phases) | component weights, per-component params |
Fit quality is assessed with the Kolmogorov–Smirnov (KS) statistic, Anderson–Darling test, or Akaike Information Criterion (AIC) for model comparison. A KS p-value above 0.05 suggests the fitted distribution is not rejected at the 5% significance level, but in large samples even trivial deviations become significant. Therefore, pair statistical tests with visual inspection (Q-Q plots, histogram overlays) and domain judgment.
For the jacket temperature example, a normal fit gives KS p = 0.12, but a two-component Gaussian mixture (one component for heating phase, one for cooling phase) reduces AIC by 340 points and visually captures the bimodal histogram. The mixture model is the better baseline.
Temporal Pattern Discovery
Time-series data carries structure that static statistics miss. Temporal pattern discovery extracts recurring shapes, cycles, and event relationships.
Autocorrelation and seasonality. The autocorrelation function (ACF) reveals periodicity. A strong peak at lag 86,400 samples (24 hours at 1 Hz) indicates daily cycling. Partial autocorrelation (PACF) helps identify the order of autoregressive behavior. For batch processes, align data to batch phase rather than wall-clock time.
Rolling window profiling. Compute statistics over sliding windows (e.g., 1-hour windows with 50% overlap) to see how mean and variance evolve. A gradual drift in rolling mean that stays within static control limits is invisible to simple thresholding but obvious in a rolling profile.
Event sequence mining. For discrete event data (alarms, operator actions, mode changes), profile inter-event intervals, co-occurrence frequencies, and sequence patterns. For example, if ALM_HIGH_PRESS is followed by ALM_HIGH_TEMP within 90 seconds in 78% of occurrences, that temporal coupling is a baseline characteristic worth encoding as a compound rule.
Change-point detection. Apply algorithms such as PELT (Pruned Exact Linear Time) or binary segmentation to identify structural breaks in mean or variance. Each detected change point defines a new baseline segment. A bearing vibration signal that shifts from 2.1 mm/s RMS to 3.8 mm/s RMS at a known maintenance timestamp should be segmented, not averaged into a single misleading baseline.
From Profile to Baseline Specification
The output of profiling is a baseline specification — a structured artifact that downstream monitoring consumes. A minimal baseline record for each tag or event stream includes:
- Tag identifier, engineering units, and sampling interval.
- Observation window start/end and operating mode filter.
- Cleaning rules applied (quality codes excluded, gap threshold, outlier treatment).
- Descriptive statistics table.
- Selected distribution family and fitted parameters, with goodness-of-fit score.
- Temporal patterns: dominant periods, rolling-window behavior, known change points.
- Recommended threshold strategy (static percentile, distribution-based, or dynamic rolling).
The following SVG illustrates the end-to-end profiling pipeline and how each stage feeds the baseline specification.
Practical Validation of a Baseline
A baseline is only useful if it correctly predicts what normal operation looks like. Validate it by replaying recent data through the baseline and checking:
- False positive rate: What percentage of known-good operating hours would trigger an alert? Target below 1% for most industrial monitoring.
- Coverage: Does the baseline account for all legitimate operating modes? If a mode is missing, the baseline will flag normal behavior as anomalous.
- Stability: Recompute the baseline on a held-out window. If parameters shift dramatically, the process may be non-stationary and require a dynamic baseline.
Once validated, the baseline specification becomes the reference for anomaly detection thresholds, data quality rules, and remediation triggers. It should be versioned and re-profiled whenever a significant process change, sensor replacement, or operating mode shift occurs — the dashed feedback path in the diagram above.