4. CAE data storage architectures

CAE simulation data is not a single homogeneous workload. A single vehicle crashworthiness program can generate hundreds of gigabytes of solver output per run, tens of thousands of small input decks and include files, a relational web of run metadata and approval states, and a long-lived archive of result files that must remain readable for a decade. No single storage technology serves all of these well. The practical question for a CAE data management architect is therefore not "which storage is best" but "which storage tier should hold which class of data, and how do the tiers connect so that metadata, files, and lifecycle policies stay consistent."

This chapter examines the four storage families that appear in CAE environments — POSIX file systems, object stores, databases, and hybrid combinations — and shows how to map simulation data classes onto them with concrete configuration and verification steps.

Classifying CAE Data Before Choosing Storage

Storage decisions become tractable once data is classified by access pattern and size distribution. In most CAE organizations, four classes dominate:

• Large binary result files (solver output, .odb, .rst, .op2, .h3d, EnSight case directories): written once, read sequentially by post-processors, often 1–50 GB each, retained for months to years.
• Small structured files (input decks, include files, material cards, scripts): thousands to millions of files, frequently read, versioned, and edited.
• Metadata and relationships (run parameters, solver version, boundary conditions, model-to-run lineage, approval status, tags): small records with high query frequency and strong consistency requirements.
• Derived artifacts (thumbnails, reports, reduced-order models, extracted curves): moderate size, generated automatically, regenerable from primary data.

The key insight is that classes 1 and 4 tolerate eventual consistency and high latency, while classes 2 and 3 demand low-latency random access and transactional integrity. This mismatch is what drives hybrid architectures.

POSIX File Systems: The Default and Its Limits

Network file systems (NFS, SMB) and parallel file systems (Lustre, GPFS/Spectrum Scale, BeeGFS, WekaFS) remain the backbone of CAE because solvers and pre/post-processors expect POSIX semantics: open, read, write, mmap, file locking, and directory traversal. A parallel file system is mandatory when dozens of compute nodes simultaneously write striped solver output; Lustre with a 4 MB stripe size across 8 OSTs is a common starting point for large transient result files.

However, POSIX file systems degrade under metadata pressure. A directory holding 500,000 small input files will slow ls and backup scans to a crawl. The standard mitigation is directory sharding by hash or date, for example storing runs under /runs/2024/07/15/<run_id>/ rather than a flat namespace. File systems also lack native query capability: finding "all runs where the mesh size exceeded 10 million elements and the solver was Abaqus 2023" requires external indexing, which is precisely the role of a database.

Object Stores: Scalable, Immutable, and Eventually Consistent

Object stores (S3-compatible systems such as MinIO, Ceph RGW, or cloud S3) address the scale and durability problem. Data is written as immutable objects addressed by key, with no directory hierarchy and no in-place modification. This suits CAE result archives extremely well: a completed run's output is never edited, only superseded by a new run. Object stores scale to petabytes, support erasure coding, and expose lifecycle rules that automatically transition cold data to cheaper tiers.

A typical key layout encodes the run identity so that objects are self-describing:

s3://cae-results/project-veh-2024/loadcase-frontal-40mph/
    run-20240715-0031/solver/odb/part-0001.odb
s3://cae-results/project-veh-2024/loadcase-frontal-40mph/
    run-20240715-0031/manifest.json

The manifest.json records checksums, file sizes, solver version, and input deck hash, making each run directory a self-contained, verifiable unit. The trade-off is latency and semantics: object stores have no partial writes, no file locking, and higher per-request latency, so they are unsuitable as a working directory for an active solver. They are also poor at holding millions of tiny files, since each object carries per-request overhead.

Databases: Metadata, Lineage, and Query

Relational databases (PostgreSQL is the common open-source choice) hold the metadata layer: run records, parameter sets, model versions, user ownership, approval workflow states, and lineage edges. A minimal schema for CAE run tracking looks like this:

CREATE TABLE run (
  run_id        UUID PRIMARY KEY,
  project_id    TEXT NOT NULL,
  loadcase      TEXT NOT NULL,
  solver        TEXT NOT NULL,
  solver_version TEXT,
  mesh_elements BIGINT,
  status        TEXT CHECK (status IN ('queued','running','completed','failed')),
  storage_uri   TEXT,          -- s3:// or file:// location of results
  created_at    TIMESTAMPTZ DEFAULT now()
);

CREATE TABLE run_input (
  run_id     UUID REFERENCES run(run_id),
  input_hash TEXT NOT NULL,    -- SHA-256 of the input deck
  file_path  TEXT NOT NULL,
  PRIMARY KEY (run_id, file_path)
);

With this schema, the earlier query becomes a single indexed statement:

SELECT run_id, storage_uri
FROM run
WHERE mesh_elements > 10000000
  AND solver = 'Abaqus'
  AND solver_version LIKE '2023%'
  AND status = 'completed';

The database never stores the bulk result files themselves — only their URIs and integrity hashes. This separation is the defining characteristic of a hybrid architecture: the database is the index and the source of truth for relationships, while the file system or object store is the payload.

Hybrid Architecture in Practice

A production CAE storage architecture typically layers these technologies by data lifecycle. The diagram below shows the flow from solver scratch space through the metadata index to long-term archive.

The flow is deliberate: solvers write to fast scratch storage, completed results are promoted to the project file system for immediate post-processing, then archived as immutable objects once the run is approved. The database is updated at each transition, so the URI in the run table always points to the current authoritative copy. Post-processors query the database for the URI and stream the object directly, avoiding a full copy back to local storage.

Verifying the Architecture End to End

After deploying such a layout, verify it with a concrete round-trip test rather than trusting configuration alone. A reproducible check proceeds as follows:

1. Submit a small solver run (for example, a 200,000-element static analysis) writing to /scratch/run-test-001.
2. On completion, compute sha256sum of each result file and compare against the manifest generated by the archiving script. A mismatch indicates a truncated write or a stripe-size problem on the parallel file system.
3. Upload to the object store with aws s3 cp --recursive (or mc cp for MinIO), then re-download one file and re-verify its hash. A mismatch here usually points to a multipart upload that was not finalized.
4. Insert the run record into PostgreSQL and confirm that storage_uri resolves to a readable object.
5. Run the parameter query from the previous section and confirm the test run appears with the correct URI.

Common failure symptoms map cleanly to tiers. Slow ls on the project file system indicates directory sharding is needed. Object uploads that succeed but produce hash mismatches suggest checksum-algorithm mismatch between client and server (for example, CRC32C versus SHA-256). Database rows that reference missing objects indicate the registration step ran before the upload completed — the fix is to make registration conditional on a successful HEAD request against the object key.

The architecture described here is not a fixed product stack but a pattern: fast POSIX storage for active work, object storage for durable scale, and a relational database binding them together through URIs and integrity hashes. Once this separation is in place, capacity planning, retention policy, and migration between vendors become independent decisions rather than coupled ones — which is exactly what large-scale CAE data management requires.