4. Data ingestion pipelines for CAE

A CAE knowledge graph is only as good as the metadata that flows into it. In practice, that metadata is scattered across three very different source families: solver decks and result files (Nastran, Abaqus, ANSYS), CAD assemblies (often STEP, JT, or native formats), and PLM systems (Teamcenter, Windchill, 3DEXPERIENCE). Each family speaks a different dialect, exposes different identifiers, and changes at a different cadence. An ingestion pipeline is the machinery that reconciles these dialects into a single, queryable graph without losing provenance or corrupting identity.

This chapter focuses on how to design and operate that pipeline. We treat ingestion not as a one-shot ETL job but as a repeatable, idempotent process with clear extraction rules, a canonical intermediate model, and a graph-loading stage that preserves lineage back to the source artifact.

Why a Canonical Intermediate Model Matters

Directly writing solver keywords into graph nodes produces a graph that is technically correct but practically unusable: the same physical part appears as GRID 1001 in Nastran, NODE-1 in Abaqus, and NS_42 in ANSYS. Without a normalization layer, cross-solver queries such as "find all analyses that loaded this bracket above 200 MPa" become impossible.

The standard remedy is a three-stage pipeline: extract → normalize → load. Extraction reads raw source artifacts and emits records in a source-specific schema. Normalization maps those records onto a canonical CAE ontology (Part, Assembly, Material, LoadCase, Mesh, Result, Analysis, Revision). Loading writes the canonical records into the graph store as nodes and typed edges, attaching a source_uri and extractor_version to every node for traceability.

Extracting Metadata from Solver Decks

Solver decks are the richest and most structured source. Nastran bulk data files are fixed-field or free-field text; Abaqus input files use keyword blocks; ANSYS MAPDL files use command syntax. A robust extractor parses these into a stream of typed records rather than a single monolithic object.

For Nastran, the extractor should capture: model ID and subcase definitions (SUBCASE), grid and element counts, property-to-material links (PSHELL → MAT1), boundary conditions (SPC, MPC), and load sets (FORCE, MOMENT, PLOAD). For Abaqus, the equivalent targets are *PART, *ASSEMBLY, *MATERIAL, *STEP, *BOUNDARY, and *OUTPUT. For ANSYS, the extractor tracks ET, MP, D, F, and SOLVE commands within each load step.

A practical extraction snippet for a Nastran deck might look like this:

# Pseudocode: Nastran bulk data extractor
for card in parse_bulk_data("wing_box.bdf"):
    if card.type == "GRID":
        emit("Node", id=card.id, x=card.x, y=card.y, z=card.z)
    elif card.type == "CQUAD4":
        emit("Element", id=card.id, nodes=card.nodes, pid=card.pid)
    elif card.type == "PSHELL":
        emit("Property", id=card.pid, mid1=card.mid1, t=card.t)
    elif card.type == "MAT1":
        emit("Material", id=card.mid, E=card.E, nu=card.nu, rho=card.rho)

Result files (OP2, .odb, .rst) are handled separately because they are binary and large. The extractor should read only the metadata envelope: analysis type, solver version, timestamp, subcase list, and a pointer to the result file location. Full field data stays in the file system; the graph stores a reference, not the array.

Extracting Metadata from CAD

CAD files contribute geometry-level identity: part numbers, revisions, material assignments, mass properties, and assembly structure. STEP AP242 is the preferred neutral format because it carries both geometry and semantic PMI. JT is common in automotive and aerospace for lightweight visualization but carries less semantic metadata.

The extractor should walk the assembly tree and emit one Part node per unique part number, with edges PART_OF linking child to parent assembly. Mass properties (mass, center_of_mass, inertia) are attached as properties. If the CAD file references a material name, that name is stored as a string and later resolved against the material master in PLM.

Extracting Metadata from PLM

PLM systems are the authoritative source for lifecycle metadata: part master, BOM structure, change orders, release status, and ownership. They are also the hardest to ingest because access is typically through vendor APIs (Teamcenter SOA, Windchill REST, 3DEXPERIENCE GraphQL) with authentication, pagination, and rate limits.

The pipeline should treat PLM as a periodic synchronization source rather than a streaming one. A typical cadence is hourly or daily. The extractor queries by last_modified_date to fetch only changed items, then emits Part, Revision, ChangeOrder, and BOMLink records. Each record carries the PLM object ID as source_id so that subsequent runs can upsert rather than duplicate.

Loading into the Graph Store

Loading is where identity resolution happens. The pipeline must decide when two records from different sources refer to the same real-world entity. A common strategy is a two-pass load: first insert all nodes with their source-local keys, then run a resolution pass that merges nodes sharing a canonical key (part number + revision, or a cross-reference table maintained by the PLM team).

Every node should carry at minimum: canonical_id, source_system, source_id, extracted_at, and extractor_version. Edges should carry confidence when the link is inferred rather than declared. This makes the graph auditable: when a query returns a suspicious result, you can trace it back to the exact source record and extractor version that produced it.

Idempotency, Incremental Loads, and Failure Handling

Production pipelines run repeatedly. If a nightly run re-inserts the same Nastran model, the graph doubles in size and queries return duplicates. The fix is idempotent upserts keyed on (source_system, source_id, extractor_version). When the extractor version changes, the pipeline should either version the nodes or mark old ones as superseded, never silently overwrite.

Incremental loads are essential for PLM and CAD, where full scans are expensive. Track a high-water mark per source (last successful modified_date or change sequence number) and store it in a small state table. On failure, the pipeline should not advance the high-water mark, so the next run retries the same window.

Validation Before Commit

Before writing to the graph, run a validation pass. Check that every Element references an existing Property, every Property references an existing Material, and every Part has a non-null canonical_id. Reject records that violate referential integrity and route them to a dead-letter store with the original source line for manual review. This keeps the graph consistent and makes ingestion failures debuggable.

A minimal validation checklist for a CAE ingestion run:

• Node count per type matches the extractor's reported count.
• No orphan edges (every edge endpoint exists as a node).
• All canonical_id values are unique within their type.
• Every node has source_system, source_id, and extracted_at.
• Material references resolve to a known material master entry.
• Revision strings match the PLM revision scheme (e.g., Rev-A, not A).

Putting It Together

An effective CAE ingestion pipeline separates concerns: source-specific extractors that understand Nastran, Abaqus, ANSYS, CAD, and PLM dialects; a canonical ontology that gives every entity a stable identity; and a loader that upserts idempotently and records provenance. The graph store then becomes a reliable substrate for cross-solver queries, impact analysis, and retrieval-augmented engineering workflows. The next chapter builds on this foundation by showing how to query the resulting graph for traceability and similarity search.