4. Validation fundamentals

Verification asks whether a model was built correctly; validation asks whether the right model was built for a specific purpose. That distinction only becomes operational when three artifacts are defined before validation evidence is collected: an intended use specification, an operational domain definition, and a set of acceptance criteria. Together they convert "the model performs well" into a falsifiable claim that can be tested, audited, and defended.

Intended Use Specification: The Anchor of All Validation Claims

An intended use specification (IUS) states, in precise and bounded language, what the model does, for whom, on what inputs, under what human oversight, and with what consequence if it is wrong. It is not a marketing description. A well-formed IUS answers five questions:

• Decision or output: What exactly does the model produce — a probability, a ranking, a segmentation mask, a remaining-useful-life estimate?
• User and workflow: Who consumes the output, and at which point in the workflow? Is the output advisory or directly actuating?
• Input population: What data modality, sensor configuration, sampling rate, or document type is assumed?
• Consequence of error: What happens on a false negative versus a false positive, and is the cost asymmetric?
• Explicit exclusions: What the model must not be used for.

Consider a concrete example from industrial inspection. A vision model flags surface defects on machined aluminum housings on a production line. A weak IUS would say "detects defects in images." A defensible IUS reads: "The model provides a per-image defect probability for scratch, dent, and porosity classes on housings from Line 3, imaged under the fixed D65 ring-light rig at 0.15 mm/pixel, to prioritize images for human re-inspection. It is advisory only; no housing is scrapped without human confirmation. It is not validated for cast-iron parts, for handheld imaging, or for sub-0.2 mm defects." Every clause in that statement becomes a validation obligation.

Operational Domain Definition: Bounding the Input Space

The operational domain (OD) is the envelope of conditions under which the IUS claim is asserted to hold. It must be defined along dimensions that actually vary in deployment, and each dimension needs a stated range plus a rationale for the boundary. Typical dimensions include:

The OD is not a wish list of everything the model might encounter; it is a contract. Conditions outside the OD are out-of-distribution (OOD) by definition, and the validation report must state how OOD inputs are detected or rejected at runtime. A practical technique is to instrument the inference pipeline with input-quality gates — exposure histogram checks, blur metrics (variance of Laplacian), and a distance-to-training-set score — that route suspect inputs to human review rather than to the model's decision path.

Acceptance Criteria: Turning Purpose into Testable Thresholds

Acceptance criteria translate the IUS and OD into measurable pass/fail thresholds, each tied to a validation method, a dataset, and a statistical confidence statement. Criteria should be stratified by consequence: a criterion for the primary operating point, criteria for each OD boundary, and criteria for OOD rejection behavior.

For the inspection model, a defensible criteria set might be:

• Primary operating point: recall ≥ 0.95 for scratch and dent at the human-confirmed threshold, with the lower bound of the 95% Wilson interval ≥ 0.92 on a held-out set of ≥ 400 positive images.
• False-positive control: false-positive rate ≤ 0.08 per image on defect-free housings, because re-inspection labor is the dominant cost.
• Boundary performance: recall ≥ 0.90 at Ra = 3.2 µm and at ±2° tilt, tested separately.
• OOD behavior: ≥ 95% of handheld-camera images flagged by the input gate before inference.
• Subgroup parity: recall difference between the two housing suppliers ≤ 0.05.

Note what these criteria do: they encode the asymmetric cost structure from the IUS (high recall, moderate false positives), they test the OD edges rather than only the comfortable center, and they specify sample sizes and confidence bounds so that a passing result is not an artifact of a small test set.

Linking Validation to Purpose and Context of Use

The three artifacts form a traceable chain. Each acceptance criterion must trace upward to a clause in the IUS and a dimension in the OD; each validation test must trace to a criterion; each reported result must trace to a test. When a criterion fails, the finding is not "the model is bad" but "the IUS claim is unsupported at this OD boundary," which points directly at a remediation: retrain with boundary data, narrow the OD, or add a human-review gate.

The diagram above makes the chain explicit: the IUS and OD jointly generate acceptance criteria, criteria drive test design, and tests produce evidence that either supports or contradicts the claim. The dashed feedback path is the most important element in practice — validation is only useful if a failed criterion produces a concrete change to the model, the OD, or the workflow controls.

Common Failure Modes in Validation Scoping

Three patterns recur in audit findings. First, criteria without provenance: thresholds such as "accuracy ≥ 0.90" appear with no link to the IUS, so no one can say whether 0.90 is adequate for the consequence of error. Second, OD defined by training data: the domain is described as "images similar to the training set," which is circular and provides no runtime rejection rule. Third, aggregate-only reporting: a single overall metric hides boundary and subgroup failures, which is precisely where deployment risk concentrates.

Practical Scoping Checklist

Before any test data is touched, confirm the following are written down and reviewed by both the model owner and the process owner:

1. The IUS names the output, the consumer, the workflow position, and at least three explicit exclusions.
2. Every OD dimension has a numeric or categorical bound and a stated rationale.
3. Each acceptance criterion cites an IUS clause and an OD dimension, and specifies method, dataset, sample size, and confidence rule.
4. OOD detection behavior is specified as a criterion, not left to runtime improvisation.
5. A trace matrix exists linking IUS → OD → AC → test → result, and it is versioned alongside the model.

When these five items are in place, validation becomes a bounded engineering activity with a clear stopping rule: the model is acceptable when the evidence supports each criterion within its stated confidence, and the residual risk — the unsupported regions of the OD and the known failure modes — is documented and accepted by the accountable owner. That is the link between validation and context of use: the model is credible not in the abstract, but exactly to the extent that its demonstrated performance covers the decisions it is actually asked to support.