4. CAE data storage architectures
CAE simulation data is not a single, uniform workload. A single vehicle crash model may contain hundreds of thousands of solver files, a CFD transient run may emit terabytes of time-step results, and a metadata record for one analysis may be only a few kilobytes. Treating all of this as "files on a share" is the most common architectural mistake in simulation data management. This chapter examines the four storage families that matter in practice — POSIX file systems, object stores, databases, and hybrid combinations — and shows how to match each data class to the storage tier that serves it best.
Why One Storage Tier Cannot Serve All CAE Data
Simulation data falls into four classes with sharply different access patterns. Input decks and solver models (e.g., .key, .inp, .cas, .dat) are small, frequently read, and rarely modified after release. Result files (.odb, .rst, .h3d, .ensight) are large, written once, and read sequentially during post-processing. Metadata — model version, solver version, boundary conditions, run status, owner, and lineage — is tiny but demands transactional consistency and fast queries. Derived artifacts such as images, animations, and reduced-order models are medium-sized and read-heavy.
A POSIX file system excels at the first and fourth classes: it offers byte-range locking, memory-mapped I/O, and the familiar directory semantics that solvers expect. It degrades badly when a directory accumulates 200,000 result files, because metadata operations become the bottleneck. An object store handles the second class elegantly — immutable, versioned, replicated blobs addressed by key — but cannot be mounted as a scratch directory for a solver that expects fseek() semantics. A relational database is the right home for the third class, but storing a 40 GB .odb as a BLOB is an anti-pattern that bloats backups and defeats streaming.
POSIX File Systems: The Working Tier
The working tier must deliver high sequential throughput and low metadata latency. For a crash solver writing 500 MB per time step across 64 MPI ranks, aggregate write bandwidth of 5–10 GB/s is typical on a parallel file system such as Lustre or GPFS. The critical configuration detail is striping: a single large result file should be spread across many OSTs (object storage targets) so that all ranks write in parallel. A common starting point is a stripe count of 4–8 and a stripe size of 1–4 MB, tuned by observing per-OST utilization during a representative run.
Directory layout matters as much as hardware. A flat directory with 100,000 files forces the file system to scan large directories on every ls or backup. A date- and job-partitioned hierarchy keeps directory sizes bounded:
/scratch/cae/2024/11/18/job_48213/
model/ # input decks, includes
run/ # solver scratch, restart files
results/ # .odb, .rst, .h3d
logs/ # solver stdout, .sta, .msgRetention on the working tier should be short — typically 7 to 30 days after run completion. The working tier is not an archive; it is a staging area whose contents are promoted to durable storage once validated.
Object Stores: The Durable Tier
Object stores (S3-compatible services, Ceph RGW, Azure Blob) provide virtually unlimited capacity, eleven-nines durability through erasure coding or replication, and immutable versioning. They are ideal for result archives, released model baselines, and derived artifacts. The key design decision is the key naming scheme, because object stores have flat namespaces and list operations are expensive at scale.
A workable convention encodes project, model, analysis, and revision in the key prefix so that lifecycle policies and access controls can be applied by prefix:
s3://cae-archive/proj_veh7/crash/frontal_40mph/rev03/
results/odb/frontal_40mph.odb
results/odb/frontal_40mph.odb.sha256
inputs/model.key
manifest.jsonTwo operational details are frequently overlooked. First, integrity verification: store a SHA-256 checksum alongside every uploaded object and verify it on download, because silent corruption during long-term retention is rare but catastrophic when it occurs. Second, lifecycle tiering: transition objects older than 90 days to infrequent-access or cold storage classes, and expire incomplete multipart uploads after 7 days to prevent orphaned parts from consuming capacity.
Databases: The Metadata and Query Tier
The metadata database is the system of record for what exists and what it means. A minimal but sufficient schema for CAE traceability includes an analysis table (project, model, analysis type, solver, solver version, status), a run table (analysis ID, start/end time, compute resource, exit code), and an artifact table (run ID, storage URI, checksum, size, format). The artifact table stores only the URI and checksum — never the payload.
Indexes should target the queries that actually occur: "all runs of model X in the last 30 days," "all failed runs on solver version 2023.2," "which analyses used boundary condition set BC-17." A composite index on (model_id, created_at) and a separate index on (solver_version, status) cover most of these. For lineage queries that traverse model-to-analysis-to-result chains, a graph database or a recursive CTE in PostgreSQL is preferable to ad-hoc joins.
.odb in a BLOB column inflates backup windows, breaks streaming post-processing, and forces the database to become a file server. Store the URI and checksum; keep the bytes in the object store.Hybrid Architecture: The Practical Default
In practice, mature CAE data management deployments combine all three tiers behind a single logical namespace. The working tier (parallel file system) hosts active runs; on completion, a promotion service computes checksums, uploads results to the object store, and writes the artifact record to the metadata database. Users and post-processing tools resolve a logical path through the database to the correct physical URI, so the migration from scratch to archive is invisible to them.
The diagram below shows this three-tier flow, including the promotion gate where validation decides whether data moves to durable storage or is quarantined.
Choosing and Tuning the Hybrid
The promotion gate is where data quality is enforced. A minimal promotion script should verify the solver exit code, compute a SHA-256 of each result file, compare it against the solver-reported checksum if available, upload to the object store with the checksum as object metadata, and only then insert the artifact record. If any step fails, the run is marked QUARANTINED in the database and the bytes remain on the working tier for inspection rather than being silently archived.
Two tuning parameters dominate hybrid performance. The promotion batch size controls how many files move per transaction; batching 50–200 files per upload reduces per-object overhead while keeping failure blast radius small. The cache retention window on the working tier determines how often post-processing must fetch from the object store; keeping results on scratch for 14 days after promotion covers the typical review cycle without exhausting capacity.
| Data class | Typical size | Access pattern | Recommended tier |
|---|---|---|---|
| Input decks, solver models | KB–MB | Read-heavy, versioned | Object store + DB record |
| Active solver scratch | GB–TB | Write-heavy, sequential | Parallel POSIX FS |
| Result files (.odb, .rst) | GB–TB | Write-once, sequential read | POSIX → object store |
| Metadata, lineage | KB | Transactional, query-heavy | Relational / graph DB |
| Derived artifacts | MB–GB | Read-heavy | Object store |
The practical takeaway is that storage architecture is a mapping problem, not a product choice. Identify each data class, assign it to the tier whose access semantics match, and make the transitions between tiers automatic and verified. When the promotion gate is rigorous and the logical namespace is stable, users experience a single coherent store while the underlying tiers do what each does best.