1. Course overview
Modern AI studies rarely begin with a clean dataset. In scientific computing, autonomous driving, robotics, climate modeling, and chip design, the training corpus is produced by simulation pipelines: parameter sweeps, Monte Carlo runs, sensor models, and scenario generators. Each execution emits artifacts — logs, trajectories, meshes, checkpoints, metrics — whose meaning depends on the exact code, configuration, random seed, and upstream data that produced them. Simulation data versioning is the discipline of making those artifacts identifiable, addressable, and reconstructable. Reproducible AI studies is the broader practice of ensuring that a reported result — a trained model, a benchmark table, a figure in a paper — can be regenerated by another engineer, on another machine, at another time, and produce the same conclusion.
These two concerns are inseparable. A model checkpoint is only meaningful relative to the simulation dataset it consumed; that dataset is only meaningful relative to the generator version and its inputs. Versioning is therefore not a storage chore but the provenance backbone of the entire study.
Why Traditional Version Control Is Not Enough
Git handles source code well because code is small, text-based, and diff-friendly. Simulation outputs violate all three assumptions. A single parameter sweep can produce tens of thousands of files totaling hundreds of gigabytes; binary formats such as HDF5, Parquet, or ROS bags do not diff meaningfully; and the same logical dataset may be re-sharded, re-compressed, or partially regenerated. Committing such data into Git is impractical, while storing it on a shared drive with filenames like run_final_v3_USE_THIS.parquet destroys traceability.
The practical answer is a layered approach: keep code and configuration in Git, keep large artifacts in object storage addressed by content hash, and keep a metadata ledger that binds them together. The ledger records, for every artifact, which code commit, which config, which seed, and which parent artifacts produced it.
The Core Concepts
Five concepts recur throughout this course and deserve precise definitions up front.
- Artifact — any file or object produced or consumed by the pipeline: raw simulation output, a cleaned dataset, a trained checkpoint, an evaluation report. Each artifact has a content hash (e.g., SHA-256) and a size.
- Run — one execution of the pipeline, identified by a run ID. A run declares its inputs (artifact hashes), its code revision, its configuration, and its environment.
- Lineage (provenance graph) — the directed acyclic graph connecting runs to the artifacts they consumed and produced. Lineage answers "where did this number come from?"
- Environment lock — the pinned set of dependencies, container image digest, CUDA/driver versions, and hardware class that the run assumed.
- Determinism boundary — the explicit statement of which parts of the pipeline are bit-reproducible and which are only statistically reproducible (e.g., GPU non-determinism in some kernels).
The determinism boundary is the concept practitioners most often skip, and it causes the most confusion. If a training run uses non-deterministic GPU reductions, re-running it will not reproduce the checkpoint bit-for-bit — but it should reproduce the metric within a stated tolerance. A reproducible study declares that tolerance rather than pretending it does not exist.
How the Pieces Fit Together
The diagram below shows the flow from a simulation configuration to a reproducible result, with the versioning layer sitting alongside the compute pipeline rather than inside it.
A Minimal Working Example
To make the concepts concrete, consider a small parameter sweep. The configuration is a YAML file; the simulation is a Python script; the output is a set of Parquet files. The versioning discipline requires three concrete actions.
# config/sweep_042.yaml
sim:
scenario: "urban_intersection_v2"
duration_s: 120
seed: 20240517
params:
friction: 0.85
sensor_noise_sigma: 0.02
dataset:
shard_size: 5000
split: {train: 0.8, val: 0.1, test: 0.1}
First, the config is committed to Git, yielding a commit SHA. Second, the simulation writes its outputs to a staging directory, and a manifest is generated that records each file's SHA-256 and size. Third, the manifest, the Git SHA, the config hash, and the container image digest are written as a single run record into the metadata ledger. The run record is the atomic unit of reproducibility.
run_id: 20240517-urban-042
code_commit: 9f3c1a7
config_hash: sha256:4b8e...c21d
image_digest: sha256:7a1f...90be
inputs: []
outputs:
- path: raw/traj_0000.parquet
sha256: 1c9d...44af
bytes: 18432000
- path: raw/traj_0001.parquet
sha256: e07b...9d12
bytes: 18395104
determinism: statistical
metric_tolerance: {collision_rate: 0.002}
With this record, reproduction becomes a mechanical procedure: resolve the run ID, fetch each artifact by hash, verify integrity, re-execute in the locked environment, and compare metrics against the declared tolerance. If the collision rate differs by more than 0.002, the reproduction has failed and the discrepancy must be investigated — not silently accepted.
Common Failure Modes
Three failure modes appear repeatedly in real projects and are worth naming explicitly, because recognizing them early saves weeks of debugging.
| Failure mode | Symptom | Root cause |
|---|---|---|
| Silent drift | Metrics shift by a few percent between runs with "identical" configs | Unpinned dependency or unseeded RNG in a downstream stage |
| Orphaned artifact | A checkpoint exists but no run record references it | Artifacts written outside the pipeline's manifest step |
| Phantom lineage | Lineage graph claims an input that was never actually read | Inputs declared from config rather than observed at runtime |
The third is the most insidious. Declaring inputs from configuration is convenient but wrong: a stage may read a cached file, fall back to a default, or skip an input entirely. Lineage must be captured from observed reads, typically by instrumenting the data loader or by hashing files at open time.
What This Course Covers
The remaining chapters build from these foundations toward operational practice. Early chapters establish artifact identity, hashing strategies, and storage layout. Middle chapters address lineage capture, environment locking, and the treatment of non-determinism in GPU-accelerated training. Later chapters cover dataset splitting discipline, evaluation harness design, and the audit trail needed to defend a published result. Throughout, the emphasis is on mechanisms that work at scale — where "scale" means thousands of runs and terabytes of artifacts, not a single notebook.
The technical context is deliberately cross-domain. The same patterns apply whether the simulation is a physics engine, a traffic scenario generator, or a financial Monte Carlo model. What changes is the artifact format and the tolerance for non-determinism; what stays constant is the need for immutable identity, explicit lineage, and a declared reproduction procedure.