3. Storage architectures for simulation and test results
Simulation and test campaigns generate data with fundamentally different shapes, and no single storage system serves all of them well. A CFD run may emit a few multi-gigabyte restart files; a wind-tunnel test may stream millions of low-latency sensor samples per second; a vehicle fleet test may produce hundreds of thousands of small log files that must be retained for years. Choosing the right storage architecture — distributed file system, object store, or time-series database — is therefore an engineering decision driven by access pattern, file size distribution, retention policy, and the query semantics your analysis tools require.
Three Architectures, Three Access Patterns
Distributed file systems (HDFS, Lustre, GPFS, CephFS) present a POSIX-like namespace and are optimized for large sequential reads and writes. They excel when simulation solvers write checkpoint files and post-processing jobs read them back in bulk. Metadata operations are comparatively expensive, so millions of tiny files degrade performance.
Object stores (S3, MinIO, Azure Blob, Ceph RGW) expose a flat key-value namespace over HTTP. They scale to exabytes, tolerate node loss gracefully, and cost far less per terabyte than parallel file systems. The trade-off is higher per-request latency and no in-place append — objects are immutable once written. This makes them ideal for archival of raw test data, model artifacts, and long-term retention tiers.
Time-series databases (InfluxDB, TimescaleDB, Prometheus, TDengine) are purpose-built for timestamped numeric streams. They compress time-ordered data aggressively, support downsampling and retention policies natively, and answer range queries with sub-second latency. They are the natural home for sensor telemetry, CAN bus traces, and vibration measurements.
Decision Framework
| Dimension | Distributed FS | Object Store | Time-Series DB |
|---|---|---|---|
| Typical object size | MB – GB | KB – TB | Bytes – KB per point |
| Write pattern | Sequential append | Immutable PUT | High-frequency insert |
| Read pattern | Full-file scan | Key lookup / range | Time-range aggregation |
| Latency | Low (ms) | Moderate (10–100 ms) | Low (ms) |
| Retention | Days – months | Years – indefinite | Days – years (policy) |
| Cost per TB | High | Low | Medium |
Worked Example: A Wind-Tunnel Test Pipeline
Consider a wind-tunnel campaign with 64 pressure sensors sampled at 10 kHz for 30 minutes per run, plus a 2 GB CFD restart file per configuration. The raw sensor stream is 64 × 10,000 × 1,800 = 1.152 × 10⁹ points per run. At 8 bytes per point that is roughly 9.2 GB of raw telemetry — manageable for a time-series database but wasteful to store as flat files.
A practical pipeline writes sensor data directly into InfluxDB with a retention policy of 90 days and a continuous downsampling task that keeps 1-minute averages indefinitely. The CFD restart files land in an object store bucket partitioned by campaign and run ID, with lifecycle rules moving them to cold storage after 30 days. A small Lustre scratch tier holds the working set for the duration of active post-processing.
# InfluxDB line protocol sample (pressure sensor P03, run WT-2024-017)
pressure,run=WT-2024-017,sensor=P03 value=101.325 1718000000000000000
pressure,run=WT-2024-017,sensor=P03 value=101.331 1718000000000100000
# Object store layout for CFD artifacts
s3://wt-results/WT-2024-017/cfd/restart_0001.cgns
s3://wt-results/WT-2024-017/cfd/restart_0002.cgns
s3://wt-results/WT-2024-017/metadata/run_config.jsonTo verify the pipeline, query the time-series database for a known calibration signal and confirm the returned values match the expected amplitude within tolerance. For the object store, issue a HEAD request on a restart file and check that the ETag and content length match the upload manifest. A common failure symptom is timestamp drift: if the sensor clock and the database ingest clock disagree, range queries silently return partial windows. Always normalize timestamps to UTC at ingest and record the original timezone in metadata.
Operational Considerations
Three practical issues dominate real deployments. First, metadata management: object stores have no directories, so you must impose a key naming convention (for example campaign/run/artifact_type/filename) and maintain an external catalog — often a small PostgreSQL table — to make data discoverable. Second, consistency: distributed file systems offer strong consistency, object stores typically offer read-after-write consistency for new objects but eventual consistency for overwrites and listings, and time-series databases vary by implementation. Design your pipeline so that downstream jobs never depend on listing an object store immediately after a write. Third, cost modeling: a tiered strategy that keeps 7 days on Lustre, 90 days in a time-series database, and everything else in object storage typically reduces total cost by 60–80% compared with keeping all data on a parallel file system.
In summary, the three architectures are complementary rather than competing. A well-designed simulation and test data platform routes each data type to the tier that matches its access pattern, enforces a consistent naming and metadata convention across tiers, and validates the pipeline with concrete checks — timestamp alignment, object integrity, and query correctness — before trusting it for analysis.