2. Data quality monitoring architecture

Industrial data quality monitoring is not a single application that inspects a database. It is a layered architecture in which data is validated, contextualized, and progressively enriched as it moves from the physical process to enterprise analytics. Each layer has distinct responsibilities, latency budgets, and failure modes. Understanding where quality checks belong — and where they must not be placed — is the foundation for building a monitoring system that detects problems before they corrupt downstream decisions.

The Five-Layer Reference Architecture

A practical industrial data quality architecture can be organized into five functional layers. Data flows upward from the process, while quality metadata and control signals flow in both directions.

The key architectural principle is progressive validation: cheap, deterministic checks run close to the source, while expensive, context-dependent checks run where sufficient data and metadata are available. Placing a cross-tag correlation check at the sensor layer is impossible; placing a simple range check only at the data lake means bad data has already traversed the entire pipeline.

Edge Gateways: The First Meaningful Quality Boundary

The edge gateway is where raw protocol data becomes structured, time-stamped, and buffered. This is the most consequential layer for data quality because it determines what the rest of the system ever sees.

Consider a gateway collecting vibration data from an OPC UA server on a rotating equipment skid. A minimal quality-aware configuration might include:

# Edge gateway quality configuration (illustrative example)
source:
  protocol: opcua
  endpoint: "opc.tcp://plc-skid-04:4840"
  sampling_interval_ms: 100
  tags:
    - node_id: "ns=2;s=Vibration.Pump01.RMS"
      deadband: 0.05          # engineering units
      quality_on_bad: "HOLD"  # retain last good value, flag quality
      stale_timeout_ms: 2000

buffer:
  store_and_forward: true
  max_disk_mb: 512
  retention_hours: 72

publish:
  target: mqtt
  broker: "tls://edge-broker.plant.local:8883"
  topic: "plant/area3/skid04/vibration"
  qos: 1

Three quality mechanisms are visible here. Deadband filtering suppresses noise-driven updates but must be paired with a maximum silence interval, otherwise a slowly drifting signal can stop publishing entirely. Quality-on-bad handling determines whether a failed sensor produces a gap, a held value, or a null — each choice has different downstream consequences. Store-and-forward buffering ensures that a network outage does not create permanent data loss, but the buffer depth defines the maximum tolerable outage duration.

Historian and Streaming Layers

The historian provides long-term time-series storage with compression, while the streaming pipeline handles real-time transformation and routing. They serve different quality functions.

Historians excel at temporal integrity checks: detecting gaps, identifying out-of-order timestamps, and reconciling backfilled data. A typical historian quality query might flag tags where the actual sample count deviates from the expected count by more than a threshold:

-- Illustrative gap detection query (historian SQL dialect)
SELECT tag_name,
       expected_samples,
       actual_samples,
       (expected_samples - actual_samples) AS missing,
       ROUND(100.0 * actual_samples / expected_samples, 2) AS completeness_pct
FROM tag_statistics
WHERE period_start >= '2025-01-15 00:00:00'
  AND period_start <  '2025-01-16 00:00:00'
  AND completeness_pct < 99.0
ORDER BY completeness_pct ASC;

Streaming pipelines add schema and semantic validation in motion. A Flink or Kafka Streams job can enforce that every message conforms to an expected Avro or JSON schema, reject duplicates within a tumbling window, and compute rolling statistics that trigger alerts when a signal's variance collapses to zero — a strong indicator of a frozen sensor or a stuck communication channel.

Data Lake and Enterprise Layer

At the top of the architecture, the data lake aggregates data from multiple historians, gateways, and external sources. Quality monitoring here focuses on cross-source consistency and lineage. For example, if the same production count is reported by both the PLC and the MES, a reconciliation job can flag discrepancies exceeding a tolerance. Completeness scoring across dimensions — time coverage, tag coverage, and attribute coverage — provides the SLA metrics that operations and engineering teams use to prioritize remediation.

Where Checks Belong: A Decision Framework

Placing a quality check at the wrong layer creates either blind spots or bottlenecks. Use these criteria:

• Deterministic and stateless (range, rate-of-change, enum validity) → edge gateway. These are cheap and prevent bad data from propagating.
• Temporal and windowed (gap detection, staleness, variance collapse) → historian or streaming pipeline. These require history or windowing state.
• Cross-source and contextual (reconciliation, lineage, completeness scoring) → data lake. These require data from multiple upstream systems.

Practical Example: Detecting a Frozen Sensor

A frozen sensor produces a constant value that passes range checks. Detecting it requires a windowed check. In a streaming pipeline, a tumbling window of 60 seconds can compute the standard deviation of each tag. If the standard deviation is exactly zero for three consecutive windows, and the tag is not a known constant (such as a setpoint), the pipeline emits a quality alert.

At the edge, the same condition can be approximated with a simpler mechanism: if the value has not changed beyond the deadband for longer than the stale timeout, the gateway marks the quality as STALE. This is less precise but catches the failure earlier. The two checks are complementary — the edge provides fast detection, the pipeline provides confirmation and context.

Architectural Anti-Patterns

Three anti-patterns recur in industrial data quality projects:

1. Centralized-only validation. All checks run in the data lake. Bad data traverses the entire pipeline, and remediation is slow because the source has already moved on.
2. Edge-only validation. The gateway performs complex checks that require cross-tag context it does not have, producing false positives and alert fatigue.
3. Quality flags as an afterthought. Values are transmitted without quality metadata, forcing downstream systems to infer validity from value patterns — an unreliable and fragile approach.

The architecture described here addresses all three by distributing checks according to their data requirements and by making quality metadata a first-class part of every message from the sensor upward. The next chapter examines how to design the quality metrics and alerting rules that operate within this architecture.