2. Data acquisition and ingestion pipelines

Simulation and test-bench data rarely arrives in a form that is immediately ready for analytics. A CFD solver may emit millions of small files per run; a dynamometer may stream 200 kHz of vibration and torque samples over EtherCAT; a hardware-in-the-loop rig may produce a continuous binary log with no schema at all. The acquisition and ingestion pipeline is the layer that turns these heterogeneous, high-rate sources into durable, queryable, and replayable datasets. Getting this layer right determines whether downstream processing is a routine batch job or a permanent firefighting exercise.

Two Source Families, Two Ingestion Philosophies

It is useful to separate sources by their delivery semantics rather than by their physical interface.

• File-oriented simulation outputs — solvers such as OpenFOAM, Fluent, or LS-DYNA write checkpoint directories, VTK/HDF5 field dumps, and CSV time histories. These are bounded and pull-based: the pipeline discovers files, validates them, and moves them into a landing zone.
• Stream-oriented test-bench measurements — DAQ systems, CAN/LIN buses, and PXI chassis produce unbounded and push-based streams. The pipeline must absorb data continuously, tolerate bursts, and preserve ordering per channel.

Mixing these two philosophies in one code path is a common source of bugs. A file watcher that polls every 30 seconds will silently drop samples from a 10 kHz stream; a Kafka consumer that expects a schema registry will choke on a solver's ad-hoc CSV header. Keep the two ingestion paths distinct and converge only at the storage layer.

Reference Architecture

The diagram below shows a converged ingestion topology. File-based and stream-based sources enter through separate adapters, are normalized into a common envelope, and are written to a tiered store. The envelope is the key contract: every record carries a source identifier, a monotonic event timestamp, a channel or field name, and a payload.

Capturing High-Frequency Simulation Outputs

Solver output is typically bursty and file-heavy. A single transient run can produce a checkpoint every few seconds, each containing dozens of field files. The ingestion job should be idempotent and resumable, because a run may be restarted and overwrite earlier checkpoints.

A practical pattern is a manifest-driven file adapter. The adapter scans a run directory, computes a content hash for each candidate file, and compares it against a manifest table. Only new or changed files are ingested. For a 500 GB transient run with 20,000 files, this avoids re-reading unchanged checkpoints on every poll.

# Example: manifest-driven discovery for a solver run
run_dir=/data/sim/run_2024_11_07_caseA
manifest=/data/manifests/run_2024_11_07_caseA.parquet

find "$run_dir" -type f \( -name "*.vtk" -o -name "*.h5" -o -name "*.csv" \) \
  -newer "$manifest" -print0 \
  | xargs -0 -n 50 -P 4 sha256sum \
  | awk '{print $1","$2}' >> /data/staging/new_files.csv

Two details matter here. First, the -newer filter is a cheap pre-filter; the hash is the authoritative check. Second, parallelism is bounded (-P 4) so that hashing does not starve the solver's own I/O. In production, replace the shell pipeline with a scheduler-aware job, but keep the same two-phase structure: cheap discovery, then authoritative validation.

For field data, do not attempt to parse every cell value during ingestion. Ingest the file as an opaque object into the raw landing zone, and record metadata (variable names, mesh size, time step, solver version) in a catalog. Parsing is a downstream concern that can be re-run when the schema evolves.

Capturing Test-Bench Measurements

Streaming measurements require a different discipline. The three properties that most often break pipelines are event time, ordering, and backpressure.

Event time. The timestamp assigned by the DAQ hardware is the only trustworthy one. Wall-clock time at the consumer is useless for a 50 kHz signal, because network jitter alone can exceed the sample interval. Record the hardware timestamp in nanoseconds and treat it as the primary key for ordering.

Ordering. Multi-channel DAQ systems may deliver channels out of order across frames. If the analysis requires phase alignment, the pipeline must either buffer and sort within a bounded window, or attach a sequence number and let the consumer reassemble. A common choice is a 200 ms reorder window, which covers typical Ethernet and PCIe jitter without adding unacceptable latency.

Backpressure. When the consumer is slower than the producer, the pipeline must decide: drop, buffer, or block. For test-bench data, dropping is usually unacceptable because the run cannot be repeated. The correct response is to buffer to a durable local spool and apply backpressure to the producer, or to shed load at a well-defined boundary (for example, decimate a 200 kHz vibration channel to 20 kHz for online monitoring while preserving the full-rate stream to disk).

Normalization and the Common Envelope

Once data is captured, normalization converts source-specific formats into the common envelope. This is where unit conversion, channel naming, and quality flags are applied. Two rules keep this stage maintainable:

1. Never mutate the raw payload. Normalization produces a new record; the original bytes remain in the raw landing zone. This makes it possible to re-normalize when a unit mapping was wrong.
2. Make the mapping explicit and versioned. A channel map is a data artifact, not code. Store it as a table with columns such as source_id, raw_channel, canonical_name, scale, offset, unit, and map_version.

For example, a torque channel might arrive as a raw 16-bit integer with a scale of 0.05 N·m per count and an offset of −1000. The normalized value is raw * 0.05 - 1000, tagged with unit N·m and the map version that produced it. If the calibration changes, a new map version is published and the affected time range is re-normalized from raw.

Failure Modes and How to Detect Them

Verification Checklist

Before declaring an ingestion pipeline production-ready, verify the following on a representative run:

• Replay a known test-bench capture and confirm that the record count and per-channel sample counts match the source.
• Introduce a deliberate 5-second consumer pause and confirm that no samples are lost (durable spool absorbs the burst).
• Restart the solver mid-run and confirm that changed checkpoints are re-ingested while unchanged ones are skipped.
• Query the curated store for a specific channel and time range, and compare against the raw payload for at least three random samples.
• Confirm that the channel map version is recorded on every normalized record.

The ingestion layer is not glamorous, but it is the foundation on which every downstream analysis rests. Treat the envelope as a contract, keep raw data immutable, and design explicitly for event time, ordering, and backpressure. Pipelines built this way survive solver restarts, broker hiccups, and the inevitable calibration change six months after the test was run.