2. Simulation data characteristics

Simulation studies differ fundamentally from conventional software projects in the shape and volume of the data they produce. Before designing any versioning strategy, it is essential to understand three structural properties that dominate simulation workflows: large-scale time-series output, combinatorial parameter sweeps, and the metadata required to make either of them reproducible. Each property imposes distinct constraints on storage layout, identifier design, and provenance capture.

Large-Scale Time-Series Output

A single simulation run rarely produces one file. A transient CFD case, a molecular dynamics trajectory, or a power-grid stability study typically emits a sequence of snapshots indexed by simulation time or timestep. For a run with N timesteps and M state variables per cell or particle, the raw output volume scales as N × M × precision. A modest 3D case with 10 million cells, 500 saved timesteps, and 8 double-precision fields already exceeds 300 GB before compression.

This scale changes what "versioning" means. You cannot treat each timestep as an independent artifact, nor can you store full copies of every run. Practical strategies include:

  • Chunked, append-only storage (HDF5, Zarr, or NetCDF-4) so that new timesteps extend an existing dataset without rewriting prior chunks.
  • Content-defined chunking to deduplicate identical or near-identical timesteps across runs that share an initial transient.
  • Separating the time axis from the state axis in the manifest, so a version identifier can reference a time range rather than a monolithic file.

The key insight is that time-series data is monotonic and ordered. Version control systems designed for source code, which assume small text diffs, perform poorly here. Instead, simulation versioning relies on immutable run directories plus a lightweight index that records which timesteps belong to which logical version.

Parameter Sweeps and Combinatorial Explosion

Most simulation studies are not single runs but sweeps over a parameter space. A typical study might vary inlet velocity across 5 values, turbulence model across 3 options, and mesh resolution across 4 levels, yielding 60 distinct runs. If each run produces 5 GB of time-series output, the study generates 300 GB — and that is before considering restarts, sensitivity analyses, or ensemble members.

The versioning challenge here is not just volume but relationship tracking. Each run is a point in a multidimensional parameter grid, and the study as a whole is a versioned entity. A useful mental model is a two-level hierarchy:

  • Study level: the sweep definition (parameter names, ranges, sampling strategy) plus the analysis scripts that consume results.
  • Run level: one concrete parameter combination, its input deck, its output data, and its execution environment.

Version identifiers must therefore encode both levels. A run ID like study-042/vel=2.5/turb=kOmega/mesh=L3 is human-readable and machine-parseable. The study version, by contrast, is a hash or tag over the sweep specification and the set of run IDs it produced.

Example sweep manifest (YAML): 
study: thermal-optimization
version: v1.3.0
parameters:
  inlet_temp_K: [300, 350, 400, 450, 500]
  flow_rate_kgs: [0.5, 1.0, 1.5]
  turbulence_model: [kEpsilon, kOmegaSST]
runs:
  - id: run-001
    params: {inlet_temp_K: 300, flow_rate_kgs: 0.5, turbulence_model: kEpsilon}
    status: completed
    output_hash: sha256:9f2c...
  - id: run-002
    params: {inlet_temp_K: 300, flow_rate_kgs: 0.5, turbulence_model: kOmegaSST}
    status: completed
    output_hash: sha256:4a1b...

This manifest is the versioning anchor. It is small, diffable, and can be committed to Git while the bulk data lives in object storage or a parallel filesystem.

Metadata Requirements for Reproducibility

Data alone does not make a study reproducible. To re-execute or audit a simulation version, the following metadata categories are necessary:

CategoryRequired fieldsWhy it matters
Code & solver solver name, version, commit hash, compiler flags Numerical results change across solver versions and optimization levels.
Input deck full parameter set, boundary conditions, initial conditions Defines the physical problem; any omission makes the run unreproducible.
Environment OS, MPI library, GPU driver, container image digest Floating-point behavior and parallel decomposition depend on these.
Execution node count, wall-clock time, random seed Stochastic solvers and load-balanced decompositions require seeds and topology.
Output timestep range, field list, compression settings, checksums Allows verification that the stored data matches the declared version.

A common failure mode is capturing the input deck but not the solver commit hash. When a colleague reruns the study six months later with a patched solver, results diverge and no one can explain why. Another is omitting the random seed for Monte Carlo or LES inflow perturbations, which makes ensemble statistics irreproducible even with identical inputs.

Putting It Together: A Versioning Data Flow

The diagram below shows how a parameter sweep, its time-series outputs, and the associated metadata flow into a versioned study record. The study manifest references run-level manifests, each of which points to chunked time-series data and a metadata bundle. The version tag is computed over the manifest tree, not over the bulk data, keeping version control lightweight.

Study Manifest sweep spec + version tag study-thermal-opt v1.3.0 Run Manifest params + env + seed run-001, run-002, ... Time-Series Store chunked HDF5 / Zarr timesteps 0..N, fields Metadata Bundle solver hash, container digest, MPI, seed Checksum & Verify sha256 over chunks compare to manifest Version Tag = hash(study manifest + run manifests + metadata bundles) Bulk time-series data is referenced by checksum, not embedded in the tag Study level Run level Bulk data Metadata Verification

Practical Implications for Versioning Design

Three design rules follow from the characteristics above. First, never version bulk time-series data directly in Git; use Git for manifests, metadata, and analysis code, and use content-addressed storage for the data itself. Second, make run IDs deterministic from the parameter combination, so that re-running the same point in the sweep produces the same identifier and can be detected as a duplicate. Third, capture environment metadata at execution time, not retrospectively; a container digest or solver commit hash recorded after the fact is often wrong or missing.

When a sweep exceeds a few hundred runs, consider storing the run manifest as a partitioned Parquet table rather than a single YAML file. This keeps the manifest queryable (e.g., "find all runs where turbulence_model = kOmegaSST and status = failed") without loading the entire study definition into memory.
A frequent reproducibility failure is storing time-series output with lossy compression (e.g., JPEG2000 at high compression ratios) while the manifest records the original precision. Checksums will match the compressed file but not the intended data. Always record the compression codec and quality setting alongside the checksum, and verify that the decompressed data meets the study's numerical tolerance.

Understanding these characteristics — ordered large-scale time-series, combinatorial parameter sweeps, and the metadata triad of code, environment, and execution — provides the foundation for the versioning architectures introduced in later sections. The next step is to translate these requirements into concrete repository layouts and identifier schemes.

← Previous 📖 Contents Next →