4. Data versioning strategies

Versioning simulation data is not a single technique but a family of strategies, each trading storage cost against retrieval speed, implementation complexity, and reproducibility guarantees. This chapter examines four foundational approaches — snapshots, deltas, content-addressable storage (CAS), and immutable artifacts — and shows how they combine in practice to form a versioning layer suitable for AI studies.

Snapshots: Full-State Capture at a Point in Time

A snapshot records the complete state of a dataset or artifact at a specific moment. In simulation workflows, a snapshot might capture an entire HDF5 results file, a full parameter set, or a complete training dataset. The defining property is self-containment: a snapshot can be restored without reference to any other version.

Snapshots are the simplest strategy to reason about and the easiest to verify. If you can read a snapshot, you have everything needed to reproduce the associated study. The cost is storage: ten snapshots of a 50 GB simulation output consume 500 GB, even if only 2% of the data changed between runs.

Practical snapshot implementation typically uses filesystem-level or object-store-level mechanisms. On a POSIX filesystem, a hardlink-based snapshot creates a new directory tree where unchanged files share inodes with the previous snapshot:

# Create a snapshot using hardlinks (rsync with --link-dest)
rsync -a --link-dest=/data/sim/v003/ /data/sim/working/ /data/sim/v004/

# Verify that unchanged files share inodes
stat -c '%i %n' /data/sim/v003/params.json /data/sim/v004/params.json
# Both lines show the same inode number if the file was unchanged

This approach gives snapshot semantics — each version directory appears complete — while only new or modified files consume additional space. The tradeoff is that hardlinks are fragile under some backup tools and do not work across filesystems or in most object stores.

Deltas: Storing Only What Changed

Delta versioning stores the difference between consecutive versions rather than full copies. For simulation data, deltas are attractive because many runs share large invariant portions: mesh geometry, boundary condition files, or the first N timesteps of a transient solve.

Delta strategies fall into two categories. Binary deltas operate on byte streams and work for any file type. Tools like xdelta3 or bsdiff produce compact patches:

# Create a delta between two simulation result files
xdelta3 -e -s results_v003.h5 results_v004.h5 v003_to_v004.xdelta

# Apply the delta to reconstruct v004
xdelta3 -d -s results_v003.h5 v003_to_v004.xdelta results_v004_reconstructed.h5

# Verify reconstruction integrity
sha256sum results_v004.h5 results_v004_reconstructed.h5

Semantic deltas operate at the level of the data model. For a parameter sweep, a semantic delta might record only the changed parameter values and the resulting output arrays, leaving the mesh and solver configuration implicit. Semantic deltas are far more compact but require domain-specific tooling and careful validation that the reconstruction is faithful.

The critical risk with deltas is chain dependency. If version 5 depends on a delta from version 4, which depends on version 3, a single corrupted delta breaks the entire chain. Production systems mitigate this by periodically checkpointing — storing a full snapshot every K deltas — so that reconstruction never requires more than K patch applications.

Content-Addressable Storage: Identity by Hash

Content-addressable storage inverts the usual relationship between names and data. Instead of asking "what is at path X?", CAS asks "give me the object whose content hashes to H." The hash — typically SHA-256 — becomes the identity of the content.

This has profound consequences for versioning. Two files with identical content have the same address, so deduplication is automatic. A file cannot be silently modified without changing its address, so integrity verification is built in. And because addresses are deterministic functions of content, they are portable across machines and storage backends.

A minimal CAS layout on disk looks like this:

# Store an artifact by its content hash
HASH=$(sha256sum results.h5 | cut -d' ' -f1)
mkdir -p cas/${HASH:0:2}
cp results.h5 cas/${HASH:0:2}/${HASH}

# Retrieve by hash
cat cas/${HASH:0:2}/${HASH} > restored.h5

The two-character prefix directory (${HASH:0:2}) is a standard sharding technique that prevents any single directory from accumulating millions of entries. Git's object store, Docker's layer store, and Nix's store all use variants of this scheme.

For simulation data, CAS is particularly valuable because it decouples logical version labels from physical storage. A manifest file maps human-readable version tags to content hashes:

{
  "study": "thermal-sweep-2024",
  "versions": {
    "v003": {"results": "a3f9...c21", "params": "7b2e...d04"},
    "v004": {"results": "e81c...9af", "params": "7b2e...d04"}
  }
}

Note that params shares the same hash across v003 and v004 — the parameter file did not change, so it is stored once. This is deduplication in action, and it happens without any explicit dedup logic.

Immutable Artifacts: Write Once, Reference Forever

An immutable artifact is a versioned output that is never modified after creation. Any change produces a new artifact with a new identity. This discipline is the foundation of reproducible AI studies because it eliminates an entire class of failures: the "same version, different results" problem caused by in-place edits.

Immutability is enforced through a combination of conventions and technical controls. On object stores like S3, object lock or versioning policies prevent overwrites. In artifact repositories like MLflow or DVC, artifacts are addressed by content hash and the registry rejects duplicate writes with different content under the same version tag.

The practical workflow for immutable artifacts in a simulation study looks like this:

1. Run the simulation in a working directory.
2. Compute content hashes for all outputs.
3. Write artifacts to the CAS or artifact store under their hashes.
4. Record a manifest that maps the study version to the set of hashes.
5. Tag the manifest with a human-readable version label.

Once step 5 completes, the version is frozen. Re-running the study produces new hashes and a new version; it never mutates the existing one.

Choosing and Combining Strategies

These four strategies are not mutually exclusive. A robust versioning layer typically combines them: immutable artifacts provide the contract, CAS provides deduplication and integrity, snapshots provide fast restore points, and deltas provide storage efficiency for large binary outputs.

The decision of which to emphasize depends on access patterns. If you frequently restore arbitrary versions, snapshots or CAS with full objects are preferable. If storage is constrained and versions are accessed sequentially, delta chains with periodic checkpoints work well. If the same large files appear across many studies, CAS deduplication delivers the greatest benefit.

Verification and Failure Modes

Regardless of strategy, a versioning system is only as good as its verification. The minimum viable check is a reconstruction test: given a version label, can you retrieve the exact bytes that were stored? For CAS-backed systems, this reduces to hash verification. For delta chains, it requires applying the full chain and comparing the result to a known-good hash.

Typical failure symptoms and their causes:

The last row deserves emphasis. CAS deduplication only works if identical logical content produces identical bytes. Simulation outputs often embed timestamps, hostnames, or random seeds in file headers, which defeats deduplication. The fix is to canonicalize outputs before hashing — strip volatile metadata, sort keys in JSON/TOML, and use deterministic serialization formats.