3. Sensor data acquisition and ingestion

Predictive models are only as good as the data that feeds them. In industrial settings, that data originates from PLCs, drives, vibration probes, temperature transmitters, flow meters, and power monitors, and it must travel through several layers before it becomes a queryable time series. This chapter examines the acquisition and ingestion stack: field protocols (OPC UA, MQTT, Modbus), historians, edge gateways, and time-series databases (TSDBs). The goal is to understand how to move high-frequency sensor data reliably from the plant floor into a store that a modeling pipeline can consume.

Field Protocols: Choosing the Right Transport

Three protocols dominate industrial data acquisition, and each occupies a different niche.

Modbus (RTU over serial, or TCP over Ethernet) is the oldest and simplest. It exposes a flat register map: coils (1-bit read/write), discrete inputs (1-bit read-only), input registers (16-bit read-only), and holding registers (16-bit read/write). A master polls slaves by function code and address. Modbus is ubiquitous on legacy equipment, but it has no native timestamps, no type system, and no security. A 32-bit float typically spans two registers, and word order (big-endian vs. little-endian) must be confirmed per device — a common source of garbage values.

OPC UA is the modern, self-describing standard (IEC 62541). It provides a typed address space, browseable node hierarchies, subscriptions with configurable publishing intervals, and built-in security (X.509 certificates, signing, encryption). A client subscribes to a MonitoredItem and receives DataChangeNotification messages carrying value, source timestamp, and status code. OPC UA is ideal for PLCs and SCADA systems that support it natively, and its information model lets you discover tags rather than hard-code addresses.

MQTT is a lightweight publish/subscribe protocol built for constrained networks. A broker (e.g., Mosquitto, HiveMQ, EMQX) routes messages by topic (e.g., plant1/line2/motor3/vibration) with QoS levels 0, 1, or 2. MQTT carries an opaque payload — often JSON or Sparkplug B — so schema discipline is the publisher's responsibility. It excels at many-to-many distribution and crossing unreliable or segmented networks.

Edge Gateways: Where Protocols Meet

An edge gateway is the translation and buffering layer between field devices and the upstream store. It performs four jobs: protocol conversion (e.g., Modbus → MQTT or OPC UA), timestamping, local buffering during network outages, and optional edge processing (filtering, deadbanding, aggregation).

Consider a gateway polling a Modbus TCP drive at 100 ms. The gateway reads holding registers 40001–40004, interprets them as two 32-bit floats (motor current and speed), attaches a UTC timestamp from its own clock, and publishes to an MQTT topic. If the uplink drops, the gateway writes to a local store-and-forward queue (often SQLite or a ring buffer) and replays on reconnect. Without this buffer, a 30-second network blip silently deletes 300 samples per tag.

Historians and Time-Series Databases

A historian (e.g., OSIsoft/AVEVA PI, Honeywell PHD, Wonderware) is a purpose-built industrial archive optimized for high tag counts and long retention. Historians typically apply compression — PI uses a swinging-door algorithm that stores a point only when it deviates beyond a tolerance from the interpolated trend. This dramatically reduces storage but means the stored series is an approximation, not a raw record.

A general time-series database (InfluxDB, TimescaleDB, QuestDB, Prometheus) offers similar ingestion rates with SQL-like query languages and flexible retention policies. TimescaleDB, built on PostgreSQL, is attractive when you want to join time-series with relational metadata (asset registries, work orders) in one engine.

The critical design decision is raw vs. compressed. For predictive modeling, keep a raw tier (full-resolution, short retention, e.g., 90 days) and a downsampled tier (1-minute aggregates, multi-year retention). Train on raw; trend on aggregates.

Practical Ingestion: A Worked Example

Suppose a gateway polls a Modbus TCP drive every 100 ms for motor current (register 40001–40002, float32, big-endian) and speed (40003–40004, float32). The gateway publishes JSON to plant1/line2/motor3:

{
  "ts": "2025-03-14T08:22:01.300Z",
  "current_a": 12.47,
  "speed_rpm": 1480.2,
  "quality": "good"
}

An ingestion service subscribes to the topic and writes to TimescaleDB. The hypertable is partitioned by time:

CREATE TABLE motor_readings (
  ts        TIMESTAMPTZ NOT NULL,
  tag_id    TEXT NOT NULL,
  value     DOUBLE PRECISION,
  quality   SMALLINT
);
SELECT create_hypertable('motor_readings', 'ts');
CREATE INDEX ON motor_readings (tag_id, ts DESC);

To verify ingestion, query the sample rate over a one-minute window:

SELECT tag_id, count(*) / 60.0 AS hz
FROM motor_readings
WHERE ts > now() - interval '1 minute'
GROUP BY tag_id;

For a 100 ms poll, expect approximately 10 Hz per tag. If you see 5 Hz, the gateway is dropping polls or the broker is throttling. If you see 10 Hz but values are constant, the deadband filter may be too aggressive, or the register mapping is wrong.

Design Checklist for the Ingestion Layer

• Timestamp authority: Decide whether source or gateway timestamps are authoritative, and record both if possible.
• Raw retention: Keep full-resolution data long enough to cover your longest training window plus margin.
• Backpressure: Ensure the broker or gateway can buffer when the database slows down; otherwise samples are lost.
• Schema discipline: Use a consistent tag naming convention and a fixed payload schema (Sparkplug B or a documented JSON contract).
• Quality flags: Propagate device status codes so downstream code can exclude bad samples rather than imputing them blindly.

With a reliable ingestion path in place — correct protocol handling, synchronized timestamps, buffered transport, and a tiered time-series store — the next stage is cleaning and aligning that data before feature engineering. The ingestion layer determines the ceiling on model quality; no downstream technique can recover samples that were never captured or were captured with the wrong timestamp.