2. Defect taxonomy and data collection

A deep learning inspection model is only as good as the defect definitions and dataset behind it. Before any network architecture is chosen, the inspection team must answer three questions: what exactly counts as a defect, how will those defects be labeled consistently, and how will the resulting dataset be curated so that rare but critical failure modes are not drowned out by normal production. This chapter covers the practical construction of a defect taxonomy, labeling strategies for industrial imagery, class imbalance handling, and dataset curation workflows.

Building a Defect Taxonomy

A defect taxonomy is a controlled vocabulary that maps observable surface or structural anomalies to named classes with explicit acceptance criteria. In industrial inspection, taxonomy classes are usually derived from three sources: customer quality specifications (e.g., scratch length thresholds), process failure modes identified by manufacturing engineering (e.g., cold solder joints, edge chips, porosity), and historical escape data from field returns. Each class should specify the observable signature, the measurement basis, and the disposition rule.

For example, a metal stamping line might define the following classes:

Class	Observable signature	Measurement basis	Disposition
scratch_minor	Linear mark, low contrast	Length < 2 mm, width < 0.1 mm	Accept
scratch_major	Linear mark, visible under diffuse light	Length ≥ 2 mm or width ≥ 0.1 mm	Reject
dent	Localized depression with shadow gradient	Depth > 0.05 mm	Reject
porosity	Cluster of dark circular voids	Any cluster ≥ 3 voids within 1 mm²	Reject
contamination	Foreign material on surface	Any visible particle	Reject

Two design principles matter here. First, classes must be mutually exclusive at the labeling stage; if a scratch and a dent co-occur, the annotator needs a priority rule (e.g., label the more severe defect, or use a multi-label scheme). Second, the taxonomy should include a "reject_unknown" or "other" class to capture anomalies that do not fit existing categories. This prevents annotators from forcing ambiguous defects into wrong classes and provides a signal for taxonomy revision.

Labeling Strategies for Industrial Imagery

Industrial inspection labels are rarely simple image-level tags. Depending on the task, labels may be bounding boxes, polygons, pixel-wise masks, or keypoints. The choice of labeling strategy should match the defect geometry and the downstream model output.

• Bounding boxes are appropriate for compact, roughly rectangular defects such as dents or contamination particles. They are fast to annotate but include background pixels, which can confuse segmentation models.
• Polygons suit irregular shapes like scratches or cracks. They require more annotation time but provide tighter boundaries for instance segmentation.
• Pixel-wise masks are necessary when defect area or perimeter is a quality metric, or when defects are thin and elongated (e.g., hairline cracks).
• Image-level labels are sufficient for coarse pass/fail classification but cannot localize defects for root-cause analysis.

Annotation consistency is the dominant source of label noise. Practical countermeasures include: a written labeling guideline with edge-case examples, a calibration session where multiple annotators label the same 50 images and inter-annotator agreement (e.g., IoU for boxes, Dice for masks) is measured, and periodic re-audit of a random sample. A useful rule of thumb is to target IoU ≥ 0.75 between annotators for bounding boxes; below that, the taxonomy or guideline needs revision before scaling annotation.

Class Imbalance in Production Datasets

In a mature production line, defect rates are often below 1%, and some critical defect types may appear only a few times per million parts. A dataset collected by simply sampling production images will therefore be extremely imbalanced. Training on such data without mitigation causes the model to predict the majority class (normal) almost always, yielding high accuracy but near-zero recall on defects.

Imbalance can be addressed at three levels:

1. Data-level: oversample rare defect images, undersample normal images, or generate synthetic defects via augmentation (e.g., copy-paste of defect patches onto normal surfaces, or GAN-based synthesis). Oversampling must be done at the image level before splitting into train/validation/test to avoid leakage.
2. Algorithm-level: use class-weighted loss functions. For a binary cross-entropy loss, the weight for class c can be set as w_c = N_total / (N_classes * N_c), where N_c is the number of samples in class c. Focal loss is another option, down-weighting easy examples.
3. Evaluation-level: never rely on accuracy. Use per-class precision, recall, F1, and the confusion matrix. For inspection, recall on critical defect classes is usually the primary metric, with a secondary constraint on false-positive rate to avoid excessive scrap.

As a concrete example, suppose a dataset has 10,000 normal images and 100 defect images across 5 defect classes. A naive 80/20 split yields 80 defect images for training, with some classes having fewer than 10 examples. Applying class weights and augmenting rare classes to at least 200 instances each (via rotation, brightness variation, and patch-based synthesis) gives the model a better chance to learn decision boundaries. The validation set should remain at the natural prevalence if the goal is to estimate field performance, but a balanced validation subset is useful for debugging per-class behavior.

Dataset Curation Workflow

Curation is the ongoing process of selecting, cleaning, and versioning data. A practical workflow for an industrial inspection project includes:

1. Ingestion: collect images with metadata (timestamp, line ID, camera ID, lighting condition, part serial number).
2. Deduplication: remove near-duplicate frames from burst captures using perceptual hashing (e.g., pHash) or embedding similarity.
3. Quality filtering: discard images that are out of focus, overexposed, or contain no part. Blur can be detected via Laplacian variance; exposure via histogram clipping.
4. Labeling and review: annotate according to the taxonomy, then run a second-pass review on a stratified sample.
5. Splitting: split by part serial number or production batch, not randomly by image, to prevent the same physical defect from appearing in both training and test sets.
6. Versioning: store dataset snapshots with a manifest that records image IDs, labels, and split assignments. Tools like DVC or a simple CSV manifest with checksums work well.

Visual Overview of the Curation Pipeline

The following diagram summarizes how raw production images flow through taxonomy definition, labeling, imbalance handling, and curation into a versioned dataset ready for training.

Practical Curation Checklist

The following checklist can be used as a starting point for a new inspection dataset. It is intentionally short and focused on decisions that materially affect model performance.

Area	Check	Acceptance criterion
Taxonomy	All defect classes have written criteria	Each class has signature, measurement, disposition
Taxonomy	"Other" class exists and is monitored	Other-class rate < 5% of defect images
Labeling	Inter-annotator agreement measured	IoU ≥ 0.75 for boxes, Dice ≥ 0.70 for masks
Imbalance	Rare classes have ≥ 200 training instances	After augmentation, per-class count ≥ 200
Splitting	Split by serial number or batch	No serial appears in more than one split
Versioning	Manifest with checksums exists	Every image traceable to a dataset version
Evaluation	Per-class recall reported	Critical defect recall ≥ target (e.g., 0.98)

Dataset construction is not a one-time task. As the production process changes, new defect modes appear, and the taxonomy must evolve. The feedback loop from the "other" class and from field escapes back into taxonomy revision is what keeps an inspection system effective over time. The next chapter moves from data to model selection, building on the dataset structure established here.