3. Exploratory data analysis for unlabeled industrial data

In supervised learning, labels tell you what to look for. In industrial anomaly detection, you rarely have that luxury: the data arrives as a stream of sensor readings, vibration spectra, PLC tags, or image frames, and nobody has annotated which segments are "normal" and which are "faulty." Exploratory Data Analysis (EDA) is therefore not a preliminary formality — it is the stage where you build a defensible mental model of what "normal" looks like, how it drifts, and where candidate anomalies might hide. This chapter covers three interlocking activities: distribution profiling, temporal pattern analysis, and anomaly candidate identification.

Why EDA Differs for Unlabeled Industrial Data

Industrial data has properties that break generic EDA habits. Sensors are sampled at fixed rates (e.g., 1 Hz for temperature, 25.6 kHz for vibration), producing strongly autocorrelated series where consecutive points are nearly redundant. Multiple channels are physically coupled — a bearing fault raises both vibration RMS and motor current. Operating regimes (idle, ramp-up, steady load) create multimodal distributions that are not anomalies but legitimate states. Finally, class imbalance is extreme: a fault may occupy 0.1% of a month-long log. EDA must therefore distinguish regime changes from genuine deviations, and must work without any ground-truth labels.

Distribution Profiling

Start by characterizing each channel's marginal distribution. For a vibration RMS channel sampled at 1 Hz over 30 days, compute summary statistics (mean, median, standard deviation, IQR, skewness, kurtosis) and plot a histogram with a kernel density estimate. Industrial signals frequently exhibit heavy tails and skewness, so mean ± 3σ thresholds are unreliable. Prefer robust statistics: median and MAD (median absolute deviation), where the robust z-score is

z_robust = 0.6745 * (x - median) / MAD

The constant 0.6745 scales MAD to be consistent with the standard deviation under normality. A common screening threshold is |z_robust| > 3.5, but this must be validated against the observed distribution rather than assumed.

Multivariate structure matters as much as marginals. Compute a correlation matrix across channels and inspect it for clusters. If two channels that are physically coupled (e.g., spindle speed command and actual speed) suddenly decorrelate, that is a stronger anomaly signal than either channel's marginal excursion. For higher-dimensional feature sets, a PCA projection showing the first two or three principal components colored by time can reveal regime clusters and outliers simultaneously.

When distributions are multimodal, do not force a single global threshold. Segment by operating regime first (using a known PLC state tag or an unsupervised clustering on contextual variables such as load and speed), then profile each regime separately.

Temporal Patterns

Industrial data is inherently temporal, and many "anomalies" are simply unmodeled periodicity. Decompose each series into trend, seasonal, and residual components using STL or a comparable method. For a 30-day log with daily production cycles, a seasonal period of 86,400 samples (at 1 Hz) captures the shift pattern; the residual is what you screen for anomalies.

Autocorrelation and partial autocorrelation functions reveal the memory structure. A slowly decaying ACF indicates a random-walk-like trend that should be differenced before thresholding. A sharp spike at lag 1 suggests strong short-term correlation, meaning point-wise thresholds will produce excessive false positives. In that case, apply a rolling window (e.g., 60-sample mean and standard deviation) and monitor the windowed statistics instead of raw values.

Change-point detection complements decomposition. Algorithms such as PELT or Bayesian online change-point detection flag locations where the statistical properties of the series shift. In an unlabeled setting, these change points are prime anomaly candidates — but they may also correspond to legitimate setpoint changes, so they must be cross-referenced with process context.

Anomaly Candidate Identification

With distributions and temporal structure characterized, you can generate a ranked list of anomaly candidates. A practical pipeline combines three complementary detectors:

  • Robust univariate screening: flag samples where |z_robust| exceeds a threshold calibrated on the residual component.
  • Multivariate distance: compute Mahalanobis distance using a robust covariance estimate (e.g., Minimum Covariance Determinant) and flag the top 0.5–1% of distances.
  • Reconstruction error: train a lightweight autoencoder or PCA reconstruction on the bulk of the data and flag samples with high reconstruction error.

Intersect or union these flags depending on your tolerance for false positives. A candidate that is flagged by all three detectors is a high-priority investigation target; one flagged by only one detector may be a detector-specific artifact.

The output of this stage is not a final anomaly label but a candidate set with supporting evidence: timestamp, channel(s) involved, detector(s) triggered, and the raw and residual values. This evidence package is what a process engineer reviews to confirm or dismiss each candidate.

EDA Pipeline for Unlabeled Industrial Data Raw Sensor Data Vibration, current, temp 1 Hz – 25.6 kHz, 30 days Distribution Profiling Robust stats: median, MAD Histogram + KDE Correlation matrix, PCA Regime segmentation Temporal Patterns STL decomposition ACF / PACF analysis Rolling window stats Change-point detection Anomaly Candidate Identification Robust z-score screening Mahalanobis distance (MCD) PCA / autoencoder reconstruction error Intersection / union of flags Evidence Package Timestamp · Channel · Detector · Raw/Residual → Process engineer review Iterate: refine thresholds & features Key thresholds (example) |z_robust| > 3.5 · Mahalanobis top 0.5–1% · Reconstruction error top 1% All thresholds calibrated on residual component, per operating regime

Practical Workflow and Verification

A reproducible EDA run for a 30-day, 1 Hz vibration RMS channel might proceed as follows. Load the series, resample to a uniform grid if timestamps are irregular, and segment by the PLC state tag into idle, ramp, and steady-load regimes. For each regime, compute median and MAD, then form the robust z-score. Apply STL with period 86,400 to the steady-load segment and retain the residual. Flag samples where |z_robust| of the residual exceeds 3.5. Separately, compute Mahalanobis distance on the multivariate feature vector (RMS, kurtosis, crest factor, current) using MCD covariance, and flag the top 1%. The intersection of both flag sets yields high-confidence candidates.

Verification is observational, not label-based. Plot each candidate's raw signal with a ±3.5 MAD band overlaid, and inspect whether the excursion coincides with a known event (maintenance, setpoint change) or is unexplained. Track the candidate rate over time: a sudden increase in flagged samples often indicates sensor drift or a genuine developing fault. If the candidate rate is implausibly high (e.g., >5% of samples), the threshold is too loose or the regime segmentation is missing a state.

Common failure modes: (1) applying global thresholds to multimodal data, producing false positives during legitimate regime transitions; (2) ignoring autocorrelation, which inflates the effective sample size and makes thresholds too tight; (3) treating change points as anomalies without checking process context; (4) using non-robust covariance estimates that are themselves corrupted by the outliers you are trying to find.

From Candidates to Actionable Insight

The deliverable of EDA on unlabeled industrial data is a prioritized, evidence-backed candidate list — not a classifier. Each candidate should carry enough context for a domain expert to make a fast judgment: when it occurred, which channels deviated, by how much relative to the regime-specific baseline, and which detectors agreed. This list feeds directly into the next stage of the pipeline, where unsupervised models (clustering, density estimation, autoencoders) are trained and tuned. By grounding those models in a well-understood distributional and temporal baseline, you avoid the classic trap of building a sophisticated detector on top of a misunderstood dataset.

← Previous 📖 Contents Next →