2. Industrial data pipelines

An industrial AI system is only as good as the data that reaches it. In a plant, that data originates from heterogeneous sources — PLCs, SCADA systems, MES applications, and historians — each with its own protocol, sampling behavior, and reliability characteristics. This chapter examines how to build ingestion pipelines that respect the latency and reliability requirements of production AI workloads.

Source Characteristics and Protocol Landscape

Before designing a pipeline, you must understand what each source actually provides. PLCs expose cyclic process data through fieldbus protocols such as PROFINET, EtherNet/IP, or Modbus TCP. A PLC scan cycle typically runs between 1 ms and 50 ms, but the data you can read is limited by the polling rate of your OPC UA or Modbus client. SCADA systems aggregate PLC tags and add alarm/event streams, often through OPC DA or OPC UA. MES provides work-order context, batch IDs, and quality results — usually via REST or SQL interfaces with latencies measured in seconds to minutes. Historians (OSIsoft PI, Wonderware, or open-source alternatives like TimescaleDB) store time-series data with timestamps that may be interpolated or compressed.

Image data comes from a different path entirely: industrial cameras, line-scan sensors, or thermal imagers connected via GigE Vision, USB3 Vision, or Camera Link. These produce frames at rates from 1 fps to several hundred fps, with payloads of 1–50 MB per frame. The ingestion challenge is not protocol diversity alone but the mismatch between deterministic control networks and best-effort IT networks.

Key insight: A pipeline that treats all sources identically will either over-engineer the slow path or under-engineer the fast path. Segment by latency class first, then by protocol.

Latency Classes and Their Implications

Production AI use cases fall into three latency tiers. Hard real-time (1–50 ms) covers closed-loop control and safety interlocks — AI rarely runs here, but data may need to be captured for later analysis. Soft real-time (50 ms–2 s) includes anomaly detection, visual inspection, and predictive alerts; this is where most industrial AI operates. Batch/near-real-time (minutes to hours) covers quality analytics, OEE calculation, and model retraining.

Each tier dictates different transport choices. For soft real-time, OPC UA subscriptions with a publishing interval of 100–500 ms are typical. For batch, you can poll a historian's REST API or run scheduled SQL extracts. The critical design decision is where to place buffering: at the edge (near the source) or in the cloud. Edge buffering with store-and-forward protects against network outages but adds operational complexity.

Industrial Data Pipeline: Source-to-Consumer Flow with Latency Tiers Sources PLC Modbus / OPC UA SCADA OPC DA / UA MES REST / SQL Historian PI / SQL / REST Camera GigE Vision Edge Gateway Protocol Adapter OPC UA / Modbus Timestamping NTP / PTP sync Local Buffer Store-and-forward Image Preprocess Resize / compress MQTT / Kafka Producer client Transport MQTT Broker QoS 1 / 2 Kafka Topics Partitioned Object Store S3 / MinIO Schema Registry Avro / Protobuf Consumers Real-time Inference < 2 s latency Feature Store Online / offline Data Lake Parquet / Delta Model Training Batch / scheduled Latency tiers: Hard RT < 50 ms Soft RT 50 ms–2 s Batch 1 min–hours Edge buffering protects against network outages; schema registry enforces contract compatibility across producers and consumers.

Reliability Patterns: Buffering, Retry, and Idempotency

Industrial networks are not reliable. A pipeline must assume that connectivity will drop, brokers will restart, and messages will be duplicated. The standard pattern is store-and-forward at the edge: the gateway writes incoming data to a local persistent queue (e.g., SQLite or a ring buffer on disk) before attempting transmission. If the broker is unreachable, the gateway continues buffering and retries with exponential backoff. Once connectivity returns, it drains the queue in order.

Idempotency is equally important. If a message is delivered twice, the consumer must not double-count. The common approach is to attach a deterministic message key — for example, {source_id}:{tag_name}:{timestamp_utc} — and have the consumer deduplicate on that key. For image data, the key might be {camera_id}:{frame_sequence}:{timestamp_utc}.

Consider a concrete example. A soft real-time anomaly detection pipeline reads 200 tags from a PLC via OPC UA at a 200 ms publishing interval. Each tag value is wrapped in a JSON payload with a UTC timestamp and a sequence number. The edge gateway buffers to a local SQLite database with a 2 GB cap. If the MQTT broker is down for 10 minutes, the gateway accumulates roughly 200 tags × 5 samples/s × 600 s = 600,000 messages. At an average payload of 200 bytes, that is approximately 120 MB — well within the buffer. When the broker recovers, the gateway drains at 5,000 messages/s, clearing the backlog in about 2 minutes.

Failure symptom: If the gateway drains too aggressively, it can saturate the network link and cause packet loss on other traffic. Rate-limit the drain to a configurable fraction of available bandwidth.

Timestamping and Time Synchronization

Every data point must carry a timestamp that reflects when the physical event occurred, not when the gateway processed it. PLCs and SCADA systems often have their own clocks, which drift. Use NTP or PTP (IEEE 1588) to synchronize edge devices to a common time source. For hard real-time applications, PTP provides sub-microsecond accuracy; for soft real-time, NTP within 10 ms is usually sufficient.

When ingesting from historians, be aware that some historians store interpolated values. If you query a 1-minute average, you are not getting raw data. For AI training, prefer raw or high-resolution data and perform your own resampling. Document the timestamp semantics — event time vs. ingestion time — in your schema registry.

Image Data: Bandwidth and Storage Considerations

Image data dominates pipeline bandwidth. A single 5 MP camera at 10 fps with 8-bit grayscale produces 50 MB/s. At 24-bit color, that becomes 150 MB/s. You cannot send this raw to a central broker. The standard approach is to preprocess at the edge: resize, crop to region of interest, convert to JPEG or PNG, and attach metadata (camera ID, timestamp, trigger signal). Only send full-resolution frames when an anomaly is detected or when the frame is selected for training.

For training data collection, use a separate path: write full-resolution frames to a local ring buffer and upload only when the network is idle or when a specific trigger occurs. This avoids competing with real-time inference traffic.

Schema Management and Data Contracts

As pipelines grow, schema drift becomes a major source of failures. A PLC firmware update might change a tag name; an MES upgrade might alter a REST response format. Use a schema registry (e.g., Confluent Schema Registry with Avro or Protobuf) to enforce compatibility. Producers register schemas; consumers validate against them. When a breaking change is needed, use a new topic or a new schema version with a migration path.

A practical data contract for a sensor stream might look like this:

{
  "source_id": "plc_line3",
  "tag_name": "temperature_zone2",
  "value": 187.4,
  "unit": "degC",
  "timestamp_utc": "2025-03-21T14:32:05.123Z",
  "quality": "good",
  "sequence": 1048576
}

The quality field is critical in industrial contexts: it indicates whether the value is valid, uncertain, or bad. Ignoring quality flags can poison your training data.

Putting It Together: A Reference Architecture

A production-grade pipeline for industrial AI typically includes: edge gateways with protocol adapters and local buffering; a message broker (MQTT for lightweight telemetry, Kafka for high-throughput streams); a schema registry; an object store for images and large payloads; and a feature store that serves both online inference and offline training. Latency requirements determine which path a data point takes: soft real-time data goes through the broker to the inference service; batch data goes to the data lake for retraining.

The key takeaway is that reliability and latency are not afterthoughts — they are design constraints that shape every component choice. Start by classifying your use cases into latency tiers, then select protocols and buffering strategies that meet those tiers without over-engineering the entire pipeline.

← Previous 📖 Contents Next →