1. MLOps pipeline fundamentals

An MLOps pipeline is the automated, versioned, and observable path that moves a model from raw data to a monitored production service. Unlike a one-off training script, a pipeline treats every stage as a reproducible, auditable step with explicit inputs and outputs. The core value is not automation for its own sake, but the ability to answer three questions at any time: What data trained this model? Can I rebuild it exactly? Is it still performing as expected in production?

Lifecycle Stages

A production-grade pipeline typically spans six stages. Each stage produces an artifact that the next stage consumes, and each artifact is versioned and traceable.

• Data ingestion and validation: Pull raw data from source systems, enforce schema contracts, and detect drift or anomalies before training begins. Tools like Great Expectations or TFX Data Validation are common here.
• Feature engineering: Transform validated data into model-ready features. Feature stores (e.g., Feast, Tecton) ensure training and serving use identical transformations.
• Model training: Execute training with pinned dependencies, fixed random seeds, and logged hyperparameters. The output is a model artifact plus metrics.
• Evaluation and validation: Compare the candidate model against a baseline using held-out data and business-specific thresholds. This is a gate, not a formality.
• Deployment: Package the model (e.g., ONNX, TorchScript, SavedModel) and release it via canary, blue-green, or shadow deployment.
• Monitoring and feedback: Track prediction quality, data drift, latency, and resource usage. Trigger retraining or rollback when thresholds are breached.

These stages form a directed acyclic graph (DAG), not a linear sequence. For example, monitoring feedback can trigger data ingestion, and evaluation failures can loop back to feature engineering. Orchestrators like Kubeflow Pipelines, Apache Airflow, or Metaflow encode this DAG explicitly.

Reproducibility Requirements

Reproducibility means that given the same code, data, and configuration, the pipeline produces the same model artifact and metrics. In practice, this requires controlling five sources of nondeterminism:

1. Code versioning: Every pipeline run must reference a specific Git commit. Branch names are not sufficient because they move.
2. Data versioning: Raw and processed datasets must be content-addressed. DVC, LakeFS, or Delta Lake provide this. A path like s3://data/train.csv is not reproducible because the file can change.
3. Environment pinning: Use container images with locked dependency versions. A requirements.txt with unpinned versions is a reproducibility hazard.
4. Random seed control: Set seeds for Python, NumPy, and the ML framework. For distributed training, also control worker initialization order.
5. Deterministic operations: Avoid nondeterministic GPU kernels where possible, or document their expected variance.

A practical reproducibility check: re-run a pipeline from a previous run ID and compare the output model hash and evaluation metrics. If they differ beyond floating-point tolerance, the pipeline has a reproducibility defect.

run_id: train-2024-06-15-001
git_commit: a3f8c21e
data_version: dvc-hash:9b2e4f1a
container_image: registry.example.com/trainer:1.4.2
python_version: "3.10.12"
random_seed: 42
framework: pytorch==2.1.0
output_model_hash: sha256:7c3d...
eval_metrics:
  accuracy: 0.923
  f1: 0.918
  latency_p99_ms: 45

The Training-to-Production Gap

The most common failure mode in ML systems is not a bad model but a model that performs well in training and poorly in production. This gap arises from several structural differences between the two environments.

Training-serving skew is the most insidious form of this gap. It occurs when the feature transformation logic differs between the training pipeline and the serving path. For example, a training script might compute a rolling average over a full day, while the serving API computes it over the last 100 requests. The model sees different feature distributions and degrades silently.

To close the gap, three practices are essential:

• Shared feature logic: Use a feature store or a shared transformation library so that training and serving call the same code. Never reimplement transformations in two languages.
• Shadow deployment: Run the new model alongside the current one on live traffic, log its predictions, and compare without affecting users. This catches skew before it causes harm.
• Production monitoring: Track input feature distributions, prediction distributions, and business metrics. A drift alert on a key feature is often the first sign that the training-to-production gap has opened.

Putting It Together

An MLOps pipeline is not a single tool but a set of contracts between stages. Data validation contracts ensure that bad data never reaches training. Evaluation gates ensure that only models meeting business thresholds are deployed. Monitoring contracts ensure that degradation is detected and acted upon. The pipeline's job is to make these contracts explicit, automated, and auditable.

When designing your first pipeline, start with the artifact flow: what does each stage produce, where is it stored, and how is it versioned? Then add orchestration, then add monitoring. A pipeline that is reproducible but not monitored is fragile; a pipeline that is monitored but not reproducible is undebuggable. Both properties are required for production reliability.