4. Storage architecture for simulation data

CAE data science platforms rarely fail because of missing algorithms. They fail because the storage layer cannot answer three questions fast enough: Where is the raw result?, Which subset do I need?, and What does this field actually mean? A single crash simulation may produce 40–200 GB of transient field data across hundreds of timesteps, while a DOE campaign of 500 runs produces tens of thousands of files. Treating all of this as "files on a share" collapses under concurrency, partial reads, and schema drift. The practical answer is a polyglot storage architecture: each storage technology owns the access pattern it was designed for, and a metadata catalog stitches them together.

Matching Storage Technology to Access Pattern

The core design rule is that storage format follows access pattern, not file size. Four patterns dominate CAE workloads:

• Bulk immutable artifacts — solver decks, restarts, full-field results, CAD/STEP geometry. Written once, read whole or by byte range, rarely mutated. → Object store.
• Multi-dimensional field arrays — displacement, stress, temperature over mesh × timestep × component. Read as hyperslabs (a spatial region across selected timesteps). → HDF5.
• Tabular scalar results — one row per run or per probe point: max von Mises stress, mass, drag coefficient, solver wall-clock. Read as columns with filters and aggregation. → Parquet.
• High-frequency scalar streams — sensor channels, co-simulation coupling variables, convergence residuals sampled every iteration. Append-heavy, time-ordered, range-queried. → Time-series database (TSDB).

Metadata — mesh hash, solver version, unit system, boundary-condition set, material card, run status — belongs in none of the above as primary storage. It belongs in a catalog that indexes all four.

Object Stores: The Immutable Backbone

Object stores (S3, MinIO, Azure Blob, GCS) provide cheap, durable, horizontally scalable storage with HTTP range reads. They are the correct home for anything written once and read many times. The critical discipline is key naming, because object stores have no real directories — prefixes are just string conventions that determine listing performance and lifecycle policies.

s3://cae-platform/
  runs/2024/crash_frontal/A017/
    deck/A017.inp
    restart/A017.res
    results/A017_fields.h5
    scalars/A017_summary.parquet
    logs/solver.log
  runs/2024/crash_frontal/A017/_manifest.json

Partitioning by year/loadcase/run_id keeps listing operations bounded and lets lifecycle rules move cold campaigns to infrequent-access tiers. A _manifest.json per run records artifact checksums and the catalog record version, so a partially uploaded run is detectable rather than silently consumed.

HDF5: Multi-Dimensional Field Data

HDF5 remains the workhorse for mesh-based field results because it supports chunked, compressed, multi-dimensional datasets with hyperslab selection. A well-structured CAE HDF5 file separates mesh topology (static) from field data (time-varying):

/mesh/nodes          float32 [N_nodes, 3]
/mesh/elements       int32   [N_elem, 8]
/mesh/connectivity   int32   [N_elem, 8]
/fields/U            float32 [N_timesteps, N_nodes, 3]   chunk=(1, 65536, 3), gzip-4
/fields/S_vm         float32 [N_timesteps, N_nodes]      chunk=(1, 65536)
/time                float64 [N_timesteps]
/attrs/unit_system   "mm-tonne-s"
/attrs/mesh_hash     "sha256:9f3c..."

Chunk shape is the single most consequential tuning decision. Chunking along the timestep axis with a large node block means a request for "all timesteps at nodes 10000–20000" touches few chunks. Chunking the other way (large timestep block, small node block) optimizes the opposite query. Because CAE post-processing usually extracts a spatial region across many timesteps, the first layout is generally correct. Compression at level 4 typically yields 3–6× reduction on float32 field data with negligible read penalty.

Two operational cautions: HDF5 is not safe for concurrent writers, and a single corrupted metadata block can render the entire file unreadable. For parallel solvers, write per-rank files and merge, or use MPI-IO with a single writer rank. Always keep the raw solver output in the object store as the recovery source.

Parquet: Scalar Results and Analytics

Once results are reduced to scalars — one row per run, or one row per probe point — Parquet becomes far more efficient than HDF5 or CSV. Columnar layout with dictionary encoding and row-group statistics enables predicate pushdown: a query for max_stress > 450 skips entire row groups without decompressing them.

run_id        loadcase   max_stress  mass_kg  drag_coeff  wall_clock_s  status
A017          frontal    487.3       12.41    0.312       8420          converged
A018          frontal    502.1       12.44    0.318       9110          converged
A019          frontal    NULL        12.39    NULL        3600          diverged

Partition the dataset by loadcase and status. A DOE analysis then reads only converged frontal runs, touching a fraction of the bytes. Keep the schema stable: adding a column is cheap, renaming one breaks every downstream query. Record the schema version in the catalog and validate on ingest.

Time-Series Databases: Streams and Convergence

TSDBs (InfluxDB, TimescaleDB, Prometheus for metrics) handle append-heavy, timestamp-indexed scalar data. In CAE they serve two distinct roles. First, solver convergence monitoring: residuals, contact forces, and energy balance sampled every iteration, queried as "show me the last 200 iterations for run A017." Second, physical test data: strain-gauge and accelerometer channels at 10–100 kHz, aligned to simulation time for validation.

The schema pattern is a measurement with tags and fields:

measurement: solver_residual
  tags:   run_id=A017, equation=Ux, loadcase=frontal
  fields: value=1.2e-4, iterations=340
  time:   2024-06-11T09:14:22Z

Tags are indexed and used for filtering; fields are the measured values. Cardinality is the classic failure mode — putting a unique node_id in a tag creates millions of series and destroys query performance. High-cardinality dimensions belong in HDF5 or Parquet, not in TSDB tags.

The Metadata Catalog: The Glue

The catalog is a relational or graph database (PostgreSQL is sufficient for most platforms) that holds one authoritative record per run and points to every storage location. It answers the questions that no individual store can: Which runs used mesh hash 9f3c? Which are diverged? What is the lineage of the surrogate model trained on campaign 2024-Q2?

Putting It Together: A Query Walkthrough

Consider a request to train a surrogate for peak stress across a frontal-crash DOE. The platform first queries the catalog for loadcase='frontal' AND status='converged', returning 480 run records with their Parquet URIs. It reads the Parquet dataset with column pruning to build the feature matrix (mass, thickness, material grade) and target vector (max_stress). For runs flagged as needing full-field validation, it resolves the HDF5 URI and performs a hyperslab read of /fields/S_vm at the final timestep only. Convergence history for the ten worst-performing runs is pulled from the TSDB to explain outliers. Every artifact is fetched by byte range from the object store — no full-file downloads.

The design principle throughout is that the catalog is the entry point and the stores are the payload. Consumers never guess a path; they resolve one from metadata. This keeps the architecture evolvable: migrating HDF5 to a new chunking scheme, or swapping MinIO for S3, changes URIs in the catalog without rewriting analysis code.