2. Data acquisition and sensor telemetry

Supervised learning for industrial prediction lives or dies on the quality of the data pipeline that feeds it. Before any model is trained, the raw material — sensor readings sampled from physical processes — must be acquired, timestamped, transported, stored, and shaped into a schema that a learning algorithm can consume. This chapter examines the three layers that make this possible: time-series logging at the edge, SCADA historians as the system of record, and the industrial data schemas that give structure and meaning to what would otherwise be an undifferentiated stream of numbers.

Time-Series Logging at the Edge

Industrial sensors rarely emit data in a form that is immediately useful for machine learning. A vibration accelerometer on a bearing housing may sample at 25.6 kHz; a thermocouple on a reactor jacket may update once per second; a flow meter on a feed line may report on change-of-state only. The first engineering decision is how to log these heterogeneous signals into a coherent time-series record.

Three logging strategies dominate practice. Periodic polling reads each sensor at a fixed interval, producing a uniform grid of timestamps. It is simple and predictable but wastes storage on slow-changing signals and can miss fast transients between polls. Change-of-state (COS) logging records a new sample only when the value deviates beyond a deadband, which is efficient for setpoints and discrete states but produces irregular timestamps. Event-triggered logging captures a burst of high-rate samples around a detected event — for example, 10 seconds of 25.6 kHz vibration data when the RMS exceeds a threshold — and is the standard approach for condition monitoring and fault diagnosis.

Regardless of strategy, every sample must carry a timestamp with a defined time base. In industrial environments, this means synchronizing edge devices to a common clock source, typically via NTP or PTP (IEEE 1588). A timestamp that drifts by even a few hundred milliseconds across devices will corrupt any cross-signal feature that assumes simultaneity. The practical rule: log the timestamp at the moment of acquisition, not at the moment of transmission, and record the clock source and synchronization status alongside the value.

When designing a logging schema, always include a quality field alongside the value. OPC UA and most industrial protocols define quality codes (Good, Bad, Uncertain) that indicate whether a reading is trustworthy. Discarding or flagging bad-quality samples during preprocessing is far cheaper than debugging a model that learned from frozen or stale values.

SCADA Historians as the System of Record

A SCADA historian is a specialized time-series database optimized for high write throughput, long retention, and efficient range queries over process data. Products such as OSIsoft PI, Wonderware Historian, and open-source alternatives like InfluxDB or TimescaleDB serve this role. The historian is the authoritative source from which training datasets are extracted, and understanding its storage model is essential for building reliable pipelines.

Historians typically store data in one of two forms. Raw archives hold every logged sample with its original timestamp and quality. Compressed archives apply a lossy algorithm — most commonly the swinging-door compression used in PI — that discards samples lying within a tolerance band around a linear interpolation of neighboring points. Compression ratios of 10:1 to 100:1 are common, which makes long-term retention affordable but means the historian is not a bit-exact record. For model training, you must decide whether to read from the compressed archive (fast, smaller, slightly lossy) or from a raw archive or edge buffer (slower, larger, exact).

Extracting training data from a historian involves a query that specifies a tag list, a time range, a sampling interval, and a retrieval mode. The retrieval mode matters: interpolated returns a value at each requested timestamp by linear interpolation between stored samples; raw returns the actual stored samples; plot returns a downsampled representation suitable for visualization. For feature engineering, interpolated retrieval at a fixed interval is usually the right choice because it produces a uniform time grid, but it can fabricate values during periods when the sensor was offline. Always cross-check interpolated output against the quality code and against raw retrieval for critical signals.

-- Example: extract 1-minute interpolated averages for a 30-day window
SELECT tag, timestamp, value, quality
FROM historian.interpolated(
    tags = ['REACTOR01.TEMP', 'REACTOR01.PRESS', 'FEED01.FLOW'],
    start = '2024-01-01T00:00:00Z',
    end   = '2024-01-31T00:00:00Z',
    interval = '1m',
    mode = 'average'
)
WHERE quality = 'Good';

The query above illustrates a common pattern: retrieve multiple tags over a shared time window, resample to a uniform interval, and filter to good-quality samples. The result is a tidy table with one row per timestamp and one column per tag — the canonical input shape for most supervised learning workflows.

Industrial Data Schemas

Raw historian extracts are rarely model-ready. Industrial data schemas provide the structure that maps physical assets, measurements, and context into a form that learning algorithms can use. Three schema families are relevant.

Tag-based schemas are the historian's native model: a flat namespace of tag names such as REACTOR01.TEMP or LINE3.MOTOR2.VIBRATION. They are simple and fast but carry no explicit relationships between tags, so the engineer must encode asset hierarchy and unit semantics in naming conventions or external metadata.

Asset-oriented schemas, exemplified by ISA-95 and OPC UA information models, represent the plant as a hierarchy of enterprises, sites, areas, work centers, and equipment, with measurements attached to assets. This makes it possible to query "all temperature sensors on reactors in Area B" without parsing tag names, and it supports transfer learning across similar assets.

Tabular feature schemas are the model-facing representation: a wide table with one row per time window and columns for engineered features (rolling means, standard deviations, rates of change, spectral bands) plus a label column. This is where the schema meets the learning algorithm, and it is where most pipeline failures occur — misaligned timestamps, unit mismatches, and label leakage are the usual culprits.

From Sensor to Training Table: Industrial Data Acquisition Pipeline Edge / Field Layer Vibration Sensor 25.6 kHz, event-triggered Thermocouple 1 Hz, periodic poll Flow Meter Change-of-state, deadband PLC / RTU Protocol gateway NTP / PTP Clock Sync Timestamp at acquisition OPC UA MQTT SCADA Historian Raw Archive Every sample, exact timestamp + quality Compressed Archive Swinging-door 10:1 – 100:1 ratio Retrieval Modes interpolated / raw / plot interval + aggregation Quality Filter Good / Bad / Uncertain Drop or flag bad samples SQL / API resample Industrial Data Schemas Tag-Based Schema REACTOR01.TEMP, LINE3.MOTOR2.VIB Flat namespace, naming conventions Asset-Oriented Schema ISA-95 / OPC UA hierarchy Enterprise → Site → Area → Equipment Tabular Feature Schema One row per time window Features + label column Common Failure: timestamp misalignment, unit mismatch, label leakage

Building a Reproducible Extraction Pipeline

The practical work of this chapter is constructing a pipeline that turns historian data into a training table without introducing silent errors. A reproducible pipeline has four stages, each with explicit inputs and verification checks.

Stage 1 — Tag selection and metadata capture. Enumerate the tags required for the prediction task, and for each tag record its engineering unit, sampling rate, expected range, and asset association. This metadata is not optional: without it, a model trained on temperature in degrees Celsius cannot be safely applied to a line instrumented in degrees Fahrenheit.

Stage 2 — Time-window extraction. Query the historian for the selected tags over the training period, using a fixed interval and an explicit aggregation function (mean, last, min, max). Record the query parameters — start, end, interval, mode — in a configuration file so the extraction can be replayed exactly.

Stage 3 — Alignment and resampling. Merge the per-tag series onto a common time grid. Decide how to handle gaps: forward-fill is acceptable for slow process variables but dangerous for fast ones; interpolation is acceptable for smooth signals but fabricates data during shutdowns. A defensible default is to mark gaps explicitly and let the model handle missingness, rather than silently filling.

Stage 4 — Feature and label assembly. Compute rolling statistics over causal windows (only past data), attach the label from the event or quality record, and write the result to a versioned table. The label timestamp must be strictly after the feature window to avoid leakage.

A frequent and costly mistake is to compute rolling features over a centered window, which uses future samples to predict the present. This produces excellent offline metrics and useless online predictions. Always verify that every feature at time t depends only on data at or before t.

Verification and Failure Symptoms

After extraction, run a small set of checks before training. Compare row counts against the expected number of intervals; a shortfall usually indicates historian gaps or a timezone mismatch. Plot each feature's distribution and compare against the tag's expected range; values outside the range point to unit errors or sensor faults. Check the label balance and the temporal distribution of positive labels; a label concentrated in one period suggests an event-logging artifact rather than a genuine process condition.

Typical failure symptoms and their causes: timestamps that are uniformly offset by a fixed number of hours indicate a timezone or daylight-saving error in the query; features that are constant over long stretches indicate a frozen sensor or a compression artifact; a model that performs well on a random train-test split but poorly on a temporal split indicates leakage or non-stationarity. Each of these is diagnosable at the pipeline stage, long before model architecture becomes the bottleneck.

The acquisition layer is where industrial machine learning projects most often succeed or fail. A historian that is queried with explicit parameters, a schema that preserves units and asset context, and a pipeline that respects temporal causality together provide the foundation on which every subsequent modeling decision rests.

← Previous 📖 Contents Next →