3. Data acquisition and preprocessing pipelines

Raw sensor output is never directly usable by a fusion or analytics stage. Between the physical transducer and the estimator that consumes its values, a preprocessing pipeline must enforce a consistent sampling cadence, suppress noise, reject physically impossible samples, repair or flag gaps, and align every stream onto a common time base. This chapter walks through each stage of that pipeline, using a representative industrial setup: an inertial measurement unit (IMU) sampled at 200 Hz, a wheel odometry encoder at 50 Hz, and a GNSS receiver at 10 Hz, all feeding a fusion node that expects synchronized inputs at 100 Hz.

Sampling and the Nyquist Constraint

Sampling converts a continuous physical quantity into a discrete sequence. The governing rule is the Nyquist–Shannon theorem: to reconstruct a signal without aliasing, the sampling rate f_s must exceed twice the highest frequency component of interest, f_s > 2·f_max. For a vibration-monitoring accelerometer whose bearing fault harmonics reach 1.2 kHz, a 2.5 kHz sample rate is the absolute floor; 5 kHz gives comfortable margin. Undersampling does not merely lose detail — it folds high-frequency energy down into the passband as a false low-frequency component that no downstream filter can remove.

Two practical details matter. First, an analog anti-aliasing low-pass filter must sit before the ADC, with its cutoff below f_s/2. Second, when streams run at different rates, resampling to a common rate is itself a sampling operation and must respect the same constraint. Downsampling 200 Hz IMU data to 100 Hz is safe only after low-pass filtering at 50 Hz; decimating without that filter aliases high-frequency motion into the output.

Filtering: Choosing the Right Response

Filtering removes noise and out-of-band content while preserving the signal of interest. The choice of filter type follows from what must be preserved:

• Moving average / FIR low-pass — simple, linear-phase, and cheap. A 5-tap moving average on the 200 Hz IMU attenuates noise but introduces a group delay of 2 samples (10 ms), which must be accounted for during synchronization.
• Butterworth IIR — smooth passband, no ripple. A 4th-order Butterworth at 20 Hz cutoff is a common choice for smoothing velocity estimates.
• Complementary filter — fuses a high-pass-filtered gyroscope with a low-pass-filtered accelerometer to estimate orientation, exploiting the fact that gyro drift is low-frequency and accelerometer noise is high-frequency.
• Median filter — a nonlinear filter that replaces each sample with the median of a sliding window. It is the standard tool for impulsive (salt-and-pepper) noise because it removes spikes without smearing edges.

A critical implementation point: IIR filters carry state. When a stream restarts or a sensor reconnects, the filter state must be reset or re-initialized, otherwise the first output samples are contaminated by stale history. For offline processing, apply filtfilt-style zero-phase filtering (forward then backward) to eliminate phase distortion, but note this is not causal and cannot be used in real time.

Outlier Removal

Outliers are samples that are not merely noisy but physically inconsistent with the process. Three complementary detectors are commonly layered:

1. Range gating — reject any value outside the sensor's specified operating range. A temperature sensor rated −40 °C to 125 °C returning 300 °C is a fault, not a measurement.
2. Statistical gating — compute a robust z-score using the median and median absolute deviation (MAD): z = 0.6745·(x − median)/MAD. Flag samples with |z| > 3.5. MAD is preferred over standard deviation because a single large spike inflates the standard deviation and masks itself.
3. Model-based gating — compare against a predicted value from a motion or process model. If the IMU reports a 40 m/s² lateral acceleration while the vehicle model predicts under 5 m/s², the sample is suspect.

Rejected samples should be replaced by a flag or a NaN marker rather than silently deleted, so the missing-data stage can handle them explicitly and the analytics layer can see the gap.

Missing Data Handling

Gaps arise from dropped packets, rejected outliers, or sensor dropout. The handling strategy depends on gap length and signal dynamics:

Never fill a long gap with zeros or the last value — both inject a false step into the signal that corrupts derivative estimates and biases any filter that follows. For the fusion node, a gap longer than 50 ms on the IMU should trigger a covariance inflation on that measurement rather than a fabricated value.

Synchronization and Time Alignment

Fusing streams sampled at different rates requires a common time base. Two mechanisms are standard:

Hardware triggering uses a shared clock or trigger line so all sensors sample at the same instant. This is the most accurate but requires wiring and compatible hardware.

Timestamp-based alignment records each sample with a high-resolution timestamp (from a PTP-disciplined clock, ideally) and resamples onto a uniform grid. For the example system, the fusion node at 100 Hz needs IMU, odometry, and GNSS values at each 10 ms tick. The pipeline:

# Conceptual resampling onto a 100 Hz grid (t = 0, 0.01, 0.02, ...)
for t in grid:
    imu_val  = interpolate(imu_samples,  t)   # 200 Hz -> 100 Hz, low-pass first
    odo_val  = interpolate(odo_samples,  t)   # 50 Hz  -> 100 Hz, zero-order hold
    gnss_val = nearest(gnss_samples, t, tol=0.05)  # 10 Hz, accept within 50 ms
    if gnss_val is None:
        gnss_val = INVALID   # do not extrapolate a 10 Hz signal to 100 Hz

Note the asymmetry: upsampling a slow stream (GNSS at 10 Hz) to 100 Hz by interpolation invents 9 out of every 10 values. The correct approach is to hold the GNSS value with its original timestamp and let the fusion filter handle the asynchronous update, rather than pretending the position changed smoothly between fixes.

Putting the Pipeline Together

The complete pipeline for one stream runs: acquire → anti-alias filter → timestamp → range gate → statistical gate → gap flag → resample to common grid → output with validity mask. The order matters. Filtering before outlier removal lets a single spike contaminate the filter's internal state; outlier removal before filtering prevents that. Resampling must come after anti-alias filtering, never before.

The diagram shows the stage order and the effect on the signal: the raw trace carries a spike that the range/MAD gate removes, the low-pass filter smooths residual noise, and the resampler produces evenly spaced samples on the 100 Hz grid. The validity mask travels alongside the values so the fusion stage knows which samples are interpolated and which are measured.

The pipeline is not a one-time configuration. Sensor aging, firmware updates, and changing operating conditions all shift noise characteristics and dropout rates. Monitoring the rejection rate, gap fraction, and residual alignment error as running statistics turns the preprocessing stage into a diagnostic instrument for the entire sensing system.