3. Data versioning and lineage

Model artifacts are versioned almost by reflex; datasets usually are not. Yet a training run is only reproducible if you can reconstruct the exact bytes it consumed. Data versioning gives every dataset state an immutable identity, while data lineage records how that state was produced and which training runs consumed it. Together they answer the two questions that dominate production incidents: which data trained this model? and which models are affected if this partition is wrong?

Three tools dominate this space, and they solve the problem at different layers. DVC versions data as content-addressed pointers stored alongside Git. LakeFS exposes an object store as a Git-like repository with branches and commits. Delta Lake versions data inside the table format itself using a transaction log. Choosing among them is not a matter of preference but of where your data already lives and how it is mutated.

Content Addressing: The Common Foundation

All three rely on content addressing. A file's identity is the hash of its bytes, so identical content always maps to the same key and any change produces a new key. DVC computes an MD5 (or SHA-256) for each tracked file or directory, stores the actual bytes in a cache or remote, and commits only a small .dvc file to Git:

# dvc.yaml — a reproducible data pipeline stage
stages:
  prepare:
    cmd: python src/prepare.py --raw data/raw.csv --out data/processed.parquet
    deps:
      - data/raw.csv
      - src/prepare.py
    outs:
      - data/processed.parquet
    params:
      - prepare.min_rows

Running dvc repro hashes the dependencies, compares them against the lock file, and re-executes only stages whose inputs changed. The resulting dvc.lock records the exact hash of every input and output, so git checkout <commit> && dvc checkout restores the precise dataset that trained a given model. This is the lightest-weight option: it works with any storage backend, requires no server, and integrates naturally with existing Git workflows. Its limitation is granularity — DVC versions whole files or directories, so a single-row correction in a 200 GB table forces a new full artifact.

LakeFS: Git Semantics Over an Object Store

LakeFS sits in front of S3, GCS, or Azure Blob and presents the bucket as a versioned repository. Writes go through the LakeFS endpoint, which records them as commits on a branch. Because it operates at the object level, you can branch a multi-terabyte dataset instantly — a branch is metadata, not a copy — and merge changes back atomically.

lakectl branch create lakefs://retail/main-experiment \
  --source lakefs://retail/main@a1b2c3d4

# Train against the isolated branch, then merge if metrics pass
lakectl commit lakefs://retail/main-experiment \
  -m "backfill missing store_id for Q3"

lakectl merge lakefs://retail/main-experiment \
  lakefs://retail/main

The practical payoff is isolation without duplication. A data engineer can backfill a broken partition on a branch while training jobs continue reading the stable main commit. The merge is atomic, so consumers never observe a half-written state. LakeFS also records per-commit metadata and can tag releases, which makes it straightforward to pin a training run to lakefs://retail/main@v2024-11-01.

Delta Lake: Versioning Inside the Table

Delta Lake takes a different approach: versioning is a property of the table format. Every write creates a new JSON commit in the _delta_log/ directory describing added and removed Parquet files. Readers reconstruct a consistent snapshot by replaying the log, and time travel is a first-class query feature:

-- Inspect table history and read a prior version
DESCRIBE HISTORY retail.sales;

SELECT COUNT(*) FROM retail.sales VERSION AS OF 42;
SELECT * FROM retail.sales TIMESTAMP AS OF '2024-11-01T00:00:00Z';

Because the log tracks file-level additions and removals, Delta handles row-level updates and deletes efficiently — a MERGE rewrites only affected files. This makes Delta the natural choice when the dataset is a continuously updated table rather than a static snapshot. The trade-off is coupling: versioning only applies to data written through Delta, and the log must be maintained carefully to avoid small-file accumulation.

Lineage: Connecting Data to Models

Versioning alone tells you what changed; lineage tells you why and what depends on it. A practical lineage graph links three node types: data artifacts (identified by hash or commit), transformation code (identified by Git SHA), and model artifacts (identified by run ID). DVC's dvc.lock already encodes the first two edges. To close the loop, emit the dataset identity into your training run's metadata:

import mlflow, subprocess

data_version = subprocess.check_output(
    ["dvc", "get-url", "data/processed.parquet"]
).decode().strip()

with mlflow.start_run() as run:
    mlflow.log_param("data_version", data_version)
    mlflow.log_param("git_sha", subprocess.check_output(
        ["git", "rev-parse", "HEAD"]).decode().strip())
    mlflow.log_artifact("dvc.lock")

With data_version and git_sha recorded per run, you can answer impact questions directly: query the tracking server for all runs whose data_version matches the suspect artifact, then trigger retraining or rollback for exactly that set. This is the difference between a lineage system and a folder of CSV files.

Operational Practices That Prevent Drift

Versioning tools fail in predictable ways. The most common is pointer drift: a training job reads data/processed.parquet from a mutable path instead of resolving the pinned hash, so the recorded version no longer matches what was consumed. Always resolve the artifact through the versioning tool at read time, never through a bare path.

The second is uncommitted writes. In LakeFS, data written directly to the underlying bucket bypasses the commit log and becomes invisible to lineage. Route all writes through the LakeFS endpoint. In Delta, concurrent writers without proper isolation can produce conflicts; enable optimistic concurrency and retry on ConcurrentModificationException.

The third is retention. Delta's VACUUM deletes old Parquet files, and DVC cache garbage collection removes unreferenced objects. Both can silently destroy the ability to reproduce an old run. Set retention windows longer than your model's audit horizon, and tag the specific versions referenced by production models so they are exempt from cleanup.

The three tools are not mutually exclusive. A common production pattern uses Delta Lake for the feature table, LakeFS for the raw landing zone, and DVC to version the small curated training sets and their transformation code. What matters is that every training run carries a resolvable pointer to immutable data and that the pointer is queryable in both directions — from data to models, and from models back to data.