2. Build a Verification Test Suite for a Classification Pipeline

Why a Verification Suite Comes Before Accuracy Claims

A classification pipeline can report a respectable F1 score and still be untrustworthy. If the preprocessing silently reorders rows, if the label encoder accepts an out-of-range class index, or if two runs with the same seed produce different predictions, then the reported accuracy is measuring an unstable artifact rather than a model. Verification testing separates code correctness — does the pipeline behave as its specification says — from model accuracy — how well do those behaviors predict the target. This case builds a test suite that enforces the first, so that any later accuracy number rests on a pipeline that is known to be well-formed.

The suite is organized around specification-derived invariants: properties that must hold for every valid input, independent of the learned weights. Three families are covered here: input schema conformance, label-range integrity, and deterministic output under a fixed seed. Each invariant is expressed as an executable test with an explicit pass/fail criterion, so a failure is a verification finding about the code, not a prompt to retune hyperparameters.

Reference Pipeline Under Test

The pipeline is deliberately small so the invariants are unambiguous. It accepts a tabular feature frame, validates the schema, applies a scaler, and returns integer class predictions from a fitted estimator. The contract below is what the tests will assert against.

# pipeline.py
import numpy as np
import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

FEATURE_COLUMNS = ["sepal_length", "sepal_width", "petal_length", "petal_width"]
N_CLASSES = 3

class ClassificationPipeline:
    def __init__(self, seed: int = 0):
        self.seed = seed
        self.scaler = StandardScaler()
        self.model = LogisticRegression(random_state=seed, max_iter=500)

    def _validate_schema(self, X: pd.DataFrame) -> None:
        missing = [c for c in FEATURE_COLUMNS if c not in X.columns]
        if missing:
            raise ValueError(f"missing feature columns: {missing}")
        if X[FEATURE_COLUMNS].isna().any().any():
            raise ValueError("NaN values present in feature columns")
        if not all(np.issubdtype(X[c].dtype, np.number) for c in FEATURE_COLUMNS):
            raise ValueError("non-numeric feature column detected")

    def fit(self, X: pd.DataFrame, y: np.ndarray) -> "ClassificationPipeline":
        self._validate_schema(X)
        if y.min() < 0 or y.max() >= N_CLASSES:
            raise ValueError("label out of range")
        self.scaler.fit(X[FEATURE_COLUMNS])
        self.model.fit(self.scaler.transform(X[FEATURE_COLUMNS]), y)
        return self

    def predict(self, X: pd.DataFrame) -> np.ndarray:
        self._validate_schema(X)
        return self.model.predict(self.scaler.transform(X[FEATURE_COLUMNS]))

Two design choices matter for testability. First, schema validation is a single method called by both fit and predict, so a schema test exercises the same code path used in production. Second, the seed is threaded into the estimator constructor, which is what makes the determinism invariant checkable rather than aspirational.

Invariant Map

The figure below shows how each specification clause maps to a concrete test and to the failure it is designed to catch. Read it as the traceability record for the suite: every test exists because a clause in the pipeline contract demands it.

Implementing the Suite

The tests use pytest and a small synthetic fixture so they run without external data. The fixture is a fixed 60-row frame with three balanced classes; it is a test input, not a performance benchmark.

# test_pipeline.py
import numpy as np
import pandas as pd
import pytest
from pipeline import ClassificationPipeline, FEATURE_COLUMNS, N_CLASSES

@pytest.fixture
def frame():
    rng = np.random.default_rng(42)
    n = 60
    X = pd.DataFrame(rng.normal(size=(n, 4)), columns=FEATURE_COLUMNS)
    y = np.tile(np.arange(N_CLASSES), n // N_CLASSES)
    return X, y

def test_schema_conformance(frame):
    X, y = frame
    pipe = ClassificationPipeline(seed=0).fit(X, y)
    # missing column
    with pytest.raises(ValueError, match="missing feature columns"):
        pipe.predict(X.drop(columns=["petal_width"]))
    # NaN in a feature
    X_nan = X.copy()
    X_nan.loc[0, "sepal_length"] = np.nan
    with pytest.raises(ValueError, match="NaN values"):
        pipe.predict(X_nan)
    # non-numeric dtype
    X_str = X.copy()
    X_str["sepal_width"] = X_str["sepal_width"].astype(str)
    with pytest.raises(ValueError, match="non-numeric"):
        pipe.predict(X_str)

def test_label_range(frame):
    X, y = frame
    pipe = ClassificationPipeline(seed=0).fit(X, y)
    preds = pipe.predict(X)
    assert preds.min() >= 0 and preds.max() < N_CLASSES
    # out-of-range label at fit time must be rejected
    y_bad = y.copy()
    y_bad[0] = N_CLASSES
    with pytest.raises(ValueError, match="label out of range"):
        ClassificationPipeline(seed=0).fit(X, y_bad)

def test_determinism(frame):
    X, y = frame
    p1 = ClassificationPipeline(seed=7).fit(X, y).predict(X)
    p2 = ClassificationPipeline(seed=7).fit(X, y).predict(X)
    np.testing.assert_array_equal(p1, p2)

Run with pytest -q test_pipeline.py. A clean run reports three passing tests. The determinism test is the one most likely to expose a real defect: if the estimator's random_state were omitted, p1 and p2 would diverge on the solver's internal shuffling and the assertion would fail — a code finding, not a sign that the model is "bad".

Reading the Results

Each test maps to a distinct verdict. A schema failure means the pipeline's input contract is not enforced and downstream numbers are untrustworthy. A label-range failure usually points to an encoding mismatch between training and inference. A determinism failure means the reported metric cannot be reproduced, so any credibility claim built on it is void. When all three pass, the pipeline is code-verified: the remaining question — whether the model is accurate enough — is a separate, legitimate inquiry that the suite deliberately does not answer.

Verification Record

The following record captures the suite's outcome in a form suitable for a credibility dossier. The status column reflects the expected result of running the suite against the reference pipeline above; it is a template to be filled from an actual run, not a claim of measured performance.

With this record in hand, the distinction the case set out to draw is concrete: the suite certifies that the pipeline behaves as specified, and only then does an accuracy figure become a meaningful input to a credibility assessment.