2. Industrial data sources and acquisition

Industrial data science rarely begins with a clean CSV file. It begins with a physical measurement — a thermocouple voltage, a flowmeter pulse, a vibration waveform — that must travel through several layers of industrial automation before it becomes a queryable record. Understanding this acquisition chain is essential, because every layer introduces its own sampling behavior, timestamping convention, and semantic contract. A model trained on data whose provenance is misunderstood will fail in ways that are difficult to diagnose.

The Automation Pyramid as a Data Path

The classical automation hierarchy still describes the data path accurately, even as IIoT gateways blur its boundaries. At the base, sensors and actuators convert physical phenomena into electrical signals: 4–20 mA current loops, 0–10 V analog voltages, RTD resistance, or digital pulses. These signals are wired to PLCs (Programmable Logic Controllers), which scan their I/O modules cyclically — typical scan times range from 1 ms for high-speed motion control to 50–100 ms for large process controllers.

PLCs expose data to SCADA (Supervisory Control and Data Acquisition) systems, which provide operator visualization and supervisory control. SCADA typically polls PLC registers at 100 ms to 1 s intervals. Above SCADA, historians archive time-series data with compression, while MES (Manufacturing Execution Systems) manage work orders, batch records, and quality events, and ERP systems hold orders, materials, and financial transactions. IIoT gateways now sit alongside this stack, publishing selected tags to cloud or edge platforms using protocols such as MQTT, OPC UA, or Sparkplug B.

Sampling Rates: Matching Measurement to Purpose

Sampling rate is not a single global setting; it is a per-signal decision driven by the physics being observed and the analytical goal. A temperature loop with a 30-second thermal time constant gains nothing from 10 Hz sampling. A bearing vibration signature at 3 kHz requires at least 6 kHz sampling to avoid aliasing, per the Nyquist criterion. Between these extremes, practical guidance is:

• Slow process variables (tank level, ambient temperature): 1–10 s intervals are sufficient for trend monitoring and mass balance.
• Fast control loops (pressure, flow, motor current): 100 ms–1 s captures dynamics relevant to control performance and fault detection.
• Vibration and acoustic monitoring: 5–50 kHz for spectral analysis; often captured in bursts rather than continuously.
• Discrete events (valve open/close, part count): event-driven, timestamped on change-of-state rather than polled.

A common pitfall is assuming that a historian's stored interval equals the original measurement rate. Historians frequently apply deadband compression — storing a new value only when it deviates from the last stored value by more than a configured threshold. This reduces storage but means the archive is not a uniform time series. Downstream analysis must either query with interpolation or work with the irregular timestamps as-is.

Timestamps: The Most Underestimated Metadata

Every data point carries at least one timestamp, but which one? Consider a pressure reading:

1. Sensor acquisition time — when the analog-to-digital conversion completed in the PLC input module.
2. PLC scan timestamp — when the controller copied the value into its process image.
3. SCADA poll timestamp — when the supervisory system read the register.
4. Historian storage timestamp — when the value was written to the archive.

These can differ by milliseconds to seconds. For control-loop tuning, the PLC scan timestamp matters. For correlating a quality defect with a process excursion, the historian timestamp is usually adequate. For root-cause analysis spanning PLC and SCADA logs, clock synchronization becomes critical. Practical deployments use NTP or PTP to keep PLCs, SCADA servers, and historians within 1–10 ms. Without this, event correlation across layers is unreliable.

Data Contracts: Defining What a Tag Actually Means

A tag name like FIC_101.PV is meaningless without a contract. A data contract specifies the tag's engineering unit, valid range, expected sampling interval, timestamp semantics, quality code behavior, and ownership. In practice, this is often captured in a tag dictionary or asset model.

Consider a flow transmitter:

Without such a contract, an analyst might treat a raw 4–20 mA count as engineering units, or average across a deadband-compressed archive as if it were uniformly sampled. Both errors silently corrupt results.

Acquisition in Practice: A Reproducible Example

Suppose you need to build a dataset for detecting feedwater pump degradation. The pump motor current is measured by a PLC analog input, polled by SCADA, archived in a historian, and also published via an IIoT gateway to an edge node.

Inputs and configuration:

• PLC: motor current tag EI_205.PV, range 0–400 A, scan 100 ms.
• SCADA: polls at 500 ms, writes to historian with deadband 1 A.
• IIoT gateway: subscribes via OPC UA at 200 ms, publishes MQTT with Sparkplug B, edge buffer 24 h.
• Time sync: all devices NTP-synced to plant master, offset < 5 ms.

Execution steps:

1. Query the historian for EI_205.PV over a 30-day window. Export with original timestamps and quality codes.
2. Query the edge node's MQTT archive for the same tag and window. Align timestamps to the PLC clock reference.
3. Compare the two series: the historian series will have fewer points due to deadband; the edge series will be denser but may have gaps during network outages.
4. Compute per-hour statistics (mean, standard deviation, min, max) from both sources. Differences in standard deviation indicate deadband effects.
5. Join with MES work-order data to label periods of known pump maintenance.

Expected observable results: The historian series is suitable for trend and threshold alarms but underestimates short current spikes. The edge series captures spikes but requires gap handling. A degradation feature such as increasing mean current at constant flow should appear in both, but the edge series will show higher variance. If the two series disagree on the timing of a step change by more than a few seconds, investigate clock synchronization or gateway buffering delays.

Typical failure symptoms and troubleshooting:

• Flat-lined historian trend: deadband too wide or tag not updating. Check PLC scan and SCADA poll status.
• Edge data gaps: network outage or broker disconnect. Verify store-and-forward buffer settings and reconnect logs.
• Timestamp misalignment: NTP failure or manual clock change. Compare device clocks against plant master.
• Unit mismatch: raw counts vs. engineering units. Verify data contract and scaling in the gateway or historian.

Acquisition is not a passive plumbing exercise. It is an engineering discipline that determines what questions your data can answer. By respecting the distinct roles of sensors, PLCs, SCADA, historians, MES, ERP, and IIoT gateways — and by explicitly documenting sampling rates, timestamps, and data contracts — you lay a foundation that supports reliable analytics from raw signal to operational decision.