2. CAE data lake architecture

A cross-solver CAE data lake is not a single product but a layered system in which each layer has a narrow, well-defined responsibility. The five canonical layers — ingestion, storage, catalog, compute, and governance — must be designed together, because decisions in one layer constrain the others. For example, choosing an object-store layout in the storage layer determines how efficiently the compute layer can prune partitions, and the catalog schema determines whether governance can enforce per-solver access policies. This section walks through each layer, the interfaces between them, and the concrete design choices that matter when the data originates from multiple solvers such as Nastran, Abaqus, Fluent, and OpenFOAM.

Ingestion Layer: Normalizing Heterogeneous Solver Output

The ingestion layer accepts raw artifacts from solvers, HPC job schedulers, and test rigs, then converts them into a consistent, self-describing form. CAE data is unusually heterogeneous: a Nastran run produces .f06, .op2, and .pch files; Abaqus produces .odb and .dat; Fluent produces .cas.h5 and .dat.h5; OpenFOAM produces directory trees of ASCII/binary fields. Ingestion must therefore be solver-aware but schema-stable.

A practical ingestion pipeline has four stages:

1. Landing: raw files are copied to a raw/ prefix in object storage, keyed by project/study/run_id. No parsing occurs here; the goal is durability and immutability.
2. Extraction: solver-specific parsers read metadata (mesh size, element count, solver version, load cases, convergence history) and scalar results (max von Mises stress, drag coefficient, first natural frequency).
3. Normalization: extracted values are mapped to a common schema — units converted to SI, result names mapped to a controlled vocabulary (e.g., stress.von_mises.max), and provenance fields attached.
4. Registration: the normalized record is written to the catalog, and the raw artifact URI is stored alongside it.

For large field data (CFD volume meshes, transient displacement fields), ingestion should not attempt to load everything into memory. Instead, write columnar chunks (Parquet) partitioned by run_id and time_step, and store only references in the catalog. A typical partition size of 128–256 MB balances read amplification against file-count overhead.

Storage Layer: Tiered, Partitioned, and Solver-Neutral

The storage layer holds three distinct classes of data, each with different access patterns:

Object storage (S3-compatible) is the default substrate. The key design decision is the partition scheme. A scheme that works across solvers is:

s3://cae-lake/
  raw/{project}/{study}/{run_id}/...
  scalars/{project}/{study}/run_id={run_id}/part-*.parquet
  fields/{project}/{study}/run_id={run_id}/step={n}/part-*.parquet
  catalog/  (metadata DB, not object storage)

Partitioning by run_id first, then time_step, allows the compute layer to read a single run or a single time slice without scanning the whole study. Avoid partitioning by high-cardinality fields such as element ID; that produces millions of tiny files.

Catalog Layer: The Semantic Backbone

The catalog is what turns a pile of files into a queryable data lake. It stores technical metadata (schema, partitions, file locations), semantic metadata (what a result means), and provenance (which solver, version, mesh, and boundary conditions produced it).

A minimal catalog schema for cross-solver CAE includes:

• run: run_id, project, study, solver_name, solver_version, start_time, status
• model: mesh_id, node_count, element_count, unit_system, coordinate_frame
• load_case: case_id, type (static, modal, transient, steady), boundary_conditions_ref
• result: result_id, run_id, case_id, quantity (controlled vocabulary), unit, min, max, location_ref
• artifact: artifact_id, run_id, uri, format, size_bytes, checksum

The controlled vocabulary for quantity is the single most important governance artifact in the catalog. Without it, one analyst writes max_stress, another writes vonMisesMax, and cross-solver comparison becomes impossible. Define the vocabulary once, version it, and validate every ingestion against it.

Compute Layer: Query, Compare, and Reduce

The compute layer sits on top of storage and catalog and provides three capabilities: query (find runs matching criteria), compare (align results across solvers), and reduce (aggregate field data into scalars or derived metrics).

A typical query — "find all static runs in study S1 where max von Mises stress exceeds 250 MPa" — translates to a catalog lookup followed by a Parquet scan:

SELECT r.run_id, r.solver_name, res.max
FROM result res
JOIN run r ON res.run_id = r.run_id
WHERE r.study = 'S1'
  AND res.quantity = 'stress.von_mises.max'
  AND res.max > 250e6;

Cross-solver comparison requires a mapping step. Nastran reports stress at grid points, Abaqus at integration points, and Fluent at cell centers. To compare them, the compute layer must either resample to a common mesh or compare only invariant scalars (global max, total mass, integrated force). For most engineering decisions, invariant scalars are sufficient and far cheaper.

Governance Layer: Policy, Lineage, and Quality

Governance is not a separate system; it is a set of policies enforced at every layer boundary. Three concerns dominate:

• Access control: row-level and column-level policies in the catalog, plus prefix-level policies in object storage. A supplier working on project A must not see project B's runs.
• Lineage: every derived result must trace back to its raw artifacts, solver version, and input deck. This is what makes a result defensible in a design review.
• Quality gates: ingestion should reject runs with missing units, unknown quantities, or mesh/result mismatches. A simple rule — "every result must have a unit and a controlled quantity name" — catches the majority of bad data.

How the Layers Interact

The diagram below shows the five layers and the primary data and control flows between them. Note that governance is drawn as a cross-cutting concern rather than a sequential stage, because it applies at every interface.

Design Trade-offs to Watch

Three trade-offs recur in practice. First, schema-on-write vs. schema-on-read: enforcing a schema at ingestion (schema-on-write) catches errors early but slows onboarding of new solvers; schema-on-read is flexible but pushes complexity to every query. For CAE, a hybrid works best — enforce a minimal core schema (run, model, result quantity, unit) at write time, and allow solver-specific extensions as JSON columns.

Second, file size vs. file count: small files improve parallelism but overwhelm catalog and object-store metadata. Target 128–256 MB per Parquet file and compact regularly.

Third, centralized vs. federated catalog: a single catalog simplifies cross-solver queries but becomes a bottleneck; a federated catalog scales but complicates lineage. Most organizations start centralized and federate only when a single team's run volume exceeds roughly 10⁶ runs.

With the five layers defined and their interfaces understood, the next design question is how to model the data itself — which is the subject of the following section on schema and metadata modeling.