2. AI model lifecycle and V&V integration

Verification and validation (V&V) is often treated as a gate that a model must pass before deployment. In practice, credibility is built incrementally: every lifecycle stage produces evidence, and that evidence either supports or undermines the eventual claim that the model is fit for its intended use. The central design question is therefore not "when do we test?" but "which V&V activities belong to each stage, what artifacts do they produce, and how do those artifacts flow into MLOps automation?"

This chapter maps the six canonical lifecycle stages — requirements analysis, design, data collection, training, deployment, and monitoring — against the V&V activities that should occur within each, and shows where those activities attach to MLOps pipelines, gates, and registries.

Stage-by-stage V&V placement

Requirements analysis. V&V begins before any data is touched. The verification activity here is requirements validation: confirming that the operational need, the intended use statement, and the acceptance criteria are complete, unambiguous, and testable. A requirement such as "the classifier should be accurate" cannot be verified; "top-1 accuracy ≥ 0.92 on the held-out 2024-Q4 evaluation set, with per-class recall ≥ 0.85 for classes A–F" can. The output is a requirements traceability matrix that later links each metric back to a test. In MLOps terms, this stage defines the evaluation contract that the training pipeline will consume as configuration, not as prose in a document.

Design. Design V&V reviews the model architecture, feature set, and training procedure against the requirements before compute is spent. Typical checks: does the chosen architecture support the latency budget; are input features available at inference time with the same distribution and freshness as at training time (training–serving skew); is the loss function aligned with the operational metric. A design review that catches a train–serve feature mismatch here costs hours; the same defect discovered in production costs a rollback.

Data collection. This is where most silent failures originate. V&V activities include schema and range validation, label-quality auditing (inter-annotator agreement, adjudication rates), coverage analysis against the operational distribution, and bias/fairness slicing. Concretely, a data validation step in the pipeline can assert, for example, that no feature has more than 2% nulls, that categorical vocabularies match the registered schema, and that each protected subgroup has at least 500 samples. Failures should halt the pipeline rather than warn, because a model trained on drifted data inherits the drift.

Training. Verification during training covers reproducibility (seeded runs, pinned container images, versioned data snapshots), hyperparameter and split integrity (no leakage between train and evaluation folds), and learning diagnostics (loss curves, per-slice metrics). Validation proper — performance against acceptance criteria — happens on data the model has never seen. The artifact of this stage is a model candidate plus an evaluation report, both registered together so that the report is inseparable from the weights it describes.

Deployment. Pre-deployment V&V includes shadow or canary evaluation on live traffic, latency and throughput testing under expected load, and robustness checks (perturbed inputs, missing fields, out-of-range values). The deployment gate should be automated: promote only if canary metrics are within tolerance of the offline evaluation and no error-budget burn is detected.

Monitoring. Post-deployment, V&V becomes continuous. Input drift detection, output distribution monitoring, ground-truth feedback loops, and periodic re-validation against fresh data all feed back into the lifecycle. Monitoring is not a separate concern from V&V; it is the validation activity that operates on the deployed system.

Requirements analysis Design Data collection Training Deployment Monitoring Requirements validation traceability matrix Design review skew analysis latency budget Schema & range label audit coverage / bias Reproducibility split integrity eval report Canary / shadow load & robustness promotion gate Drift detection feedback loop re-validation MLOps control plane experiment tracking · model registry · data versioning · CI/CD gates · feature store · monitoring dashboards each V&V artifact is registered and enforced as a pipeline gate monitoring evidence triggers re-validation and re-training

How V&V interacts with MLOps

The diagram above shows the two planes that must be wired together. The upper plane is the lifecycle; the middle band is the MLOps control plane. The interaction pattern is consistent: every V&V activity emits a machine-readable artifact, and every artifact is enforced as a gate in the pipeline.

Consider a concrete pipeline configuration. A data validation step runs as a containerized task and writes a JSON report:

{
  "dataset": "orders_2024q4",
  "checks": [
    {"name": "null_rate", "feature": "customer_age", "value": 0.004, "threshold": 0.02, "status": "pass"},
    {"name": "vocab_match", "feature": "region_code", "status": "pass"},
    {"name": "subgroup_min", "group": "region=APAC", "count": 812, "threshold": 500, "status": "pass"}
  ],
  "overall": "pass"
}

The training job declares this report as a dependency; if overall is not pass, the job does not start. After training, an evaluation task writes a second report containing the acceptance-criteria metrics, and the model registry records the candidate only if that report passes. The deployment stage reads the registry entry, runs canary evaluation, and promotes only when the canary report is within tolerance. Monitoring closes the loop by writing drift reports that can automatically open a re-validation ticket or trigger a retraining pipeline.

Treat V&V thresholds as versioned configuration, not hard-coded constants. When acceptance criteria change — for example, a tightened recall floor for a regulated subgroup — the change should be a reviewed commit that re-runs the affected gates, so the evidence trail stays auditable.

Common integration failures

Three patterns recur when teams first integrate V&V into MLOps. First, orphaned evidence: evaluation reports live in a wiki while the model lives in a registry, so nobody can prove which report belongs to which weights. Fix by co-registering the report hash with the model version. Second, warning-only gates: data checks log warnings but do not block, and drifted data silently reaches training. Fix by making the gate fail-closed for checks that affect validity. Third, monitoring without feedback: dashboards exist but no automated path connects a drift alert to a re-validation run, so the loop is open. Fix by wiring the alert to a pipeline trigger with a defined owner.

A model that passes every offline gate can still fail in production if the deployment-stage V&V is skipped. Canary evaluation on live traffic is the only stage that observes the actual serving distribution, feature freshness, and latency under real load — do not treat it as optional for high-consequence models.

What "good" looks like

A mature integration has four properties. Requirements are traceable: each acceptance metric maps to a test and a report field. Evidence is immutable and linked: model versions, data snapshots, and evaluation reports are registered together with hashes. Gates are automated and fail-closed where validity is at stake. And the loop is closed: monitoring produces actionable signals that re-enter the lifecycle as re-validation or retraining triggers. When these hold, V&V stops being a phase and becomes the connective tissue of the lifecycle — the mechanism by which each stage's output is checked before it becomes the next stage's input, and by which production evidence flows back to where it can improve the model.

← Previous 📖 Contents Next →