3. Defining the comparison target

Model comparison is only meaningful relative to a precisely stated target. Two models that differ by 0.8 percentage points on a benchmark can reverse their ranking once the evaluation population, deployment conditions, or metric definition changes. Before any statistical test is run, the acceptance team must therefore fix four coupled decisions: who the model will serve (population), where and how it will run (deployment conditions), what data distribution it will actually encounter (operational distribution), and which metric defines success. These four elements form the comparison target, and every later step — sampling, confidence intervals, hypothesis tests, acceptance thresholds — inherits their assumptions.

Population: from "all users" to a sampling frame

A population statement such as "all production traffic" is not operational. It must be narrowed to a sampling frame that can actually be enumerated and sampled: for example, "all loan applications submitted through the mobile channel between 2025-01-01 and 2025-03-31, excluding duplicate submissions and test accounts." The frame determines the unit of analysis (application, session, device, or customer), and the unit of analysis in turn determines what counts as an independent observation. If one customer submits five applications, treating applications as independent inflates the effective sample size and understates variance.

Stratification is usually necessary because aggregate metrics hide segment behavior. A fraud model may improve overall precision while degrading recall on new accounts, which are the highest-risk segment. Define the strata before looking at results — channel, tenure band, region, device class, or traffic source — and commit to reporting per-stratum metrics alongside the pooled metric. The pooled number answers "did it improve on average"; the strata answer "for whom, and is any segment unacceptable."

Deployment conditions: the context that changes model behavior

Deployment conditions describe everything around the model that can alter its inputs or its outputs: serving latency budget, feature freshness, hardware and runtime version, preprocessing pipeline, threshold policy, and human-in-the-loop review. A model evaluated with batch-computed features at T+1 cannot be compared fairly against a candidate that must run with streaming features at 200 ms p99 latency. Record the conditions as a configuration contract, for example:

comparison_target:
  population: mobile_loan_applications_2025Q1
  unit: application_id
  strata: [channel, tenure_band, region]
  deployment:
    serving_mode: realtime
    latency_budget_p99_ms: 200
    feature_freshness: streaming_<=5s
    runtime: model_server_2.4 / python_3.11
    decision_threshold: 0.62
  operational_distribution:
    reference_window: 2025-01-01..2025-03-31
    expected_shift: holiday_seasonality
  metrics:
    primary: precision_at_threshold
    guardrails: [recall_new_accounts, calibration_ece]

Any condition that differs between the incumbent and the candidate must be either equalized or explicitly modeled as part of the comparison. Otherwise the measured difference conflates model quality with environment differences.

Operational data distributions: what the model will actually see

Offline evaluation sets are historical and therefore stale by construction. The operational distribution is the distribution the model will face after deployment, and it may differ from the reference window through covariate shift (input features drift), label shift (base rates change), or concept drift (the input–output relationship changes). The comparison target should state which of these is expected and how it will be handled.

A practical approach is to quantify the gap before trusting any offline comparison. Compute population stability index (PSI) or a two-sample test per feature between the reference window and the most recent window:

PSI = Σ (p_actual - p_expected) * ln(p_actual / p_expected)

With conventional interpretation: PSI < 0.1 indicates negligible shift, 0.1–0.25 moderate shift requiring review, and > 0.25 significant shift that invalidates a naive offline comparison. If a key feature such as credit_utilization_30d shows PSI = 0.31, the team should reweight the evaluation sample to the recent distribution or run a time-based backtest that mimics deployment cadence, rather than reporting the unadjusted offline metric.

Performance metrics: one primary, several guardrails

Metric selection is a modeling decision disguised as a reporting decision. Choose a single primary metric tied to the business objective, then a small set of guardrail metrics that must not regress beyond stated tolerances. For a threshold-based classifier, report the metric at the deployed threshold, not at the default 0.5, and include a calibration measure because thresholded metrics are sensitive to probability calibration.

State the aggregation rule in advance: macro-average across strata or micro-average over all observations. These produce different numbers when stratum sizes are unequal, and choosing after seeing results is a form of selection bias. Also fix the confidence level and the minimum detectable effect; a comparison designed to detect a 0.01 precision difference needs far more samples than one targeting 0.05.

Freezing the contract and checking it

Once the four decisions are made, write them into a single frozen evaluation contract and version it alongside the model artifacts. The contract is the reference against which both the incumbent and the candidate are scored, and it must be identical for both. A useful pre-flight check before running the comparison:

• Can every observation in the evaluation frame be traced to a population definition and a stratum label?
• Do the incumbent and candidate see the same feature values, thresholds, and preprocessing?
• Has PSI been computed for the top features, and is the reweighting or backtest decision documented?
• Is the primary metric defined at the deployed threshold, with the aggregation rule and confidence level stated?
• Are guardrail tolerances numeric and agreed by the accepting stakeholder, not left as "should not degrade"?

With the comparison target fixed, the remaining statistical work becomes well-posed: the sample size follows from the minimum detectable effect and variance, the confidence interval follows from the sampling design, and the acceptance decision follows from the pre-registered primary and guardrail rules. If the target is vague, no amount of statistical rigor downstream can recover a trustworthy conclusion.