4. Artifact storage strategies

CAE experiments rarely produce a single tidy output file. A single transient structural run may emit a solver input deck, a restart file, hundreds of time-step result files, a modal database, and a set of post-processed contour images. A CFD campaign can generate terabytes of field data across dozens of cases. Treating all of these as opaque blobs attached to a run record is technically possible, but it quickly collapses under its own weight: storage costs explode, retrieval becomes slow, and reproducibility suffers because nobody can tell which files are authoritative. This chapter focuses on how to store these artifacts deliberately — using object stores for bulk capacity and content-addressable storage (CAS) for integrity and deduplication.

Why a Plain Filesystem Breaks Down

Local or network filesystems assume a mutable, hierarchical namespace. That model conflicts with experiment tracking in three ways. First, result files are immutable once a solver finishes; rewriting them in place destroys provenance. Second, the same mesh or the same baseline result is often reused across many runs, so naive storage multiplies identical bytes. Third, filesystem paths are fragile — a renamed directory or a moved mount point silently breaks every reference in the tracking database.

Object stores (S3, MinIO, Azure Blob, GCS) solve the capacity and durability problem: they scale horizontally, tolerate node loss, and expose a flat key-value interface. Content-addressable storage solves the identity problem: an artifact's key is derived from its content, so identical bytes always map to the same address. Combining the two gives a storage layer where large CAE outputs are cheap to keep, safe to reference, and naturally deduplicated.

Content Addressing in Practice

In a CAS layout, each artifact is hashed (typically SHA-256) and stored under a key derived from that digest. A common scheme splits the digest into a prefix directory to avoid hot spots:

artifacts/
  sha256/
    ab/cd/abcd1234...ef90        # full digest as filename
    ab/cd/abcd1234...ef90.meta   # size, mime, original name, producer run

The tracking database then stores only the digest, not the path. Two runs that produce byte-identical output — for example, a mesh file regenerated from the same parametrization — resolve to the same object. Deduplication is automatic and requires no special logic. Integrity verification is equally automatic: re-hashing a retrieved object and comparing against the key detects silent corruption.

Choosing a Storage Tier per Artifact Class

Not every artifact deserves the same treatment. A practical strategy classifies outputs by size, access frequency, and reproducibility, then routes each class to an appropriate tier.

The key insight is that field data dominates volume but is accessed in a narrow window, while metadata and derived plots are tiny but read constantly. Separating these tiers keeps hot storage small and cheap to query.

Handling Field Data Without Exploding File Counts

A transient run with 500 time steps and 20 fields per step can produce 10,000 individual files. Storing each as a separate object creates metadata overhead and slow listing. Two mitigations are common. The first is to write field data in a container format — CGNS, EnSight case, or HDF5 — so one object holds an entire time series. The second is to store per-step files but group them under a single logical artifact record with a manifest:

{
  "artifact": "field_data",
  "format": "hdf5",
  "digest": "sha256:9f2c...",
  "steps": 500,
  "fields": ["pressure", "velocity", "temperature"],
  "size_bytes": 412000000000
}

The manifest lets downstream tools resolve the whole series with one database lookup, while the object store holds the bytes. Derived plots are then generated from the manifest, not from directory listings.

Derived Plots and the Provenance Chain

Derived plots are artifacts too, and they must carry provenance. A contour plot is only meaningful if you know which run, which time step, and which field it came from. Store each plot with a metadata record linking it to its parent artifact digest and the post-processing parameters (colormap, range, view angle). This creates a directed acyclic graph: run → field data → derived plot. When a solver bug is found, you can traverse the graph and invalidate every downstream plot.

Retrieval, Caching, and Lifecycle

Reading a 400 GB field dataset from cold object storage is slow. A local cache layer — often an NVMe scratch volume — should hold recently accessed artifacts keyed by digest. Because digests are immutable, cache invalidation is trivial: a cache hit is always correct. Lifecycle policies then move objects between tiers automatically: field data transitions to infrequent-access after 30 days, restart files after 7 days, and derived plots stay hot indefinitely because they are small.

Practical Configuration Example

The following MinIO client commands illustrate how a CAS layout is populated and how lifecycle rules are applied. The bucket name and endpoint are illustrative.

# Upload an artifact under its content digest
DIGEST=$(sha256sum result.h5 | cut -d' ' -f1)
mc cp result.h5 local/cae-artifacts/sha256/${DIGEST:0:2}/${DIGEST:2:2}/${DIGEST}

# Apply lifecycle: move field data to infrequent tier after 30 days
mc ilm add local/cae-artifacts \
  --prefix "field/" \
  --transition-days 30 \
  --transition-tier "INFREQUENT"

# Verify integrity on retrieval
mc cat local/cae-artifacts/sha256/9f/2c/9f2c... | sha256sum

If the recomputed digest does not match the key, the object is corrupt and should be re-fetched from a replica or regenerated. This check is cheap relative to the cost of silently analyzing corrupted field data.

Common Failure Modes

• Hash collision on partial writes: the solver is still flushing when hashing begins. Symptom: digest mismatch on every retrieval. Fix: wait for file close, or hash a manifest that includes a completion marker.
• Unbounded cache growth: the NVMe scratch volume fills because eviction is not digest-aware. Symptom: write failures during post-processing. Fix: LRU eviction keyed by digest with a size cap.
• Orphaned objects: a run is deleted from the tracking DB but its artifacts remain. Symptom: storage grows without corresponding run records. Fix: periodic garbage collection that scans for digests with no referencing metadata.
• Provenance gaps: derived plots stored without parent digest. Symptom: cannot trace a plot back to its run. Fix: enforce a foreign key from plot metadata to the parent artifact digest.

A well-designed artifact storage layer turns CAE result management from a manual file-shuffling exercise into a queryable, deduplicated, and auditable system. The object store provides the capacity, content addressing provides the identity, and the tracking database provides the relationships that make the whole collection navigable.