3. Transfer learning fundamentals

Industrial inspection datasets are almost always small, imbalanced, and expensive to label. A line engineer may have only 200–800 annotated images of a specific defect class, while a generic vision backbone such as ResNet-50 or a ViT-B/16 has already been trained on ImageNet-1k or ImageNet-21k with over a million images. Transfer learning bridges this gap by reusing the feature representations learned on large-scale natural image corpora and adapting them to the narrow, high-precision requirements of industrial inspection. This chapter explains the four core mechanisms — pretraining, fine-tuning, feature extraction, and domain adaptation — and shows how to choose among them for real inspection tasks.

Pretraining: What the Backbone Already Knows

Pretraining produces a set of weights θ_pre that encode generic visual primitives: edges, oriented gradients, textures, and mid-level part detectors. For industrial imagery, the most useful pretrained sources are:

• ImageNet-1k / ImageNet-21k — broad natural-image statistics; strong for texture and shape priors.
• COCO — object-level localization and instance segmentation priors.
• Self-supervised backbones (DINOv2, MAE, SimCLR) — dense features that often transfer better to low-data regimes than supervised ImageNet weights.
• Domain-specific pretraining — e.g., MVTec AD or a company's own historical inspection archive, when available.

The key insight is that early convolutional layers learn near-universal filters (Gabor-like edge detectors), while later layers become increasingly task- and domain-specific. This layer-wise specialization is the foundation for deciding how much of the network to freeze or update.

Feature Extraction vs. Fine-Tuning

Two strategies dominate when adapting a pretrained backbone to a new inspection task:

Feature extraction is the safest starting point. You freeze the backbone, extract embeddings, and train a lightweight classifier (logistic regression, SVM, or a 2-layer MLP). It is fast, requires minimal GPU memory, and rarely overfits. However, when the industrial domain differs substantially from ImageNet — for example, grayscale X-ray, thermal IR, or high-magnification PCB imagery — frozen features may be suboptimal because the pretrained filters are tuned to RGB natural statistics.

Fine-tuning unlocks the backbone and updates weights with a small learning rate. A practical recipe for industrial inspection:

# Example: fine-tuning a ResNet-50 on a 4-class PCB defect dataset
# Assumptions: 1,200 training images, 300 validation images, 224x224 input

optimizer = SGD(params, lr=1e-3, momentum=0.9, weight_decay=1e-4)
scheduler = CosineAnnealingLR(optimizer, T_max=30)
loss_fn   = CrossEntropyLoss(label_smoothing=0.1)

# Stage 1: freeze backbone, train head for 5 epochs
freeze(backbone); train(head, epochs=5)

# Stage 2: unfreeze layer4 + head, lr=1e-4, 15 epochs
unfreeze(backbone.layer4); set_lr(1e-4); train(epochs=15)

# Stage 3: unfreeze all, lr=1e-5, 10 epochs
unfreeze_all(); set_lr(1e-5); train(epochs=10)

This three-stage schedule — head-only, then top block, then full network — is a reliable default. The learning rate drops by roughly 10× at each stage to avoid destroying pretrained features. BatchNorm statistics should generally be kept in eval mode during early stages if the batch size is small (<16), because industrial datasets often cannot fill large batches.

Domain Adaptation: Closing the Industrial Gap

Domain adaptation addresses the mismatch between the source domain (pretraining data) and the target domain (your inspection line). Three levels are common:

1. Input-level adaptation — normalize target images to resemble the source: histogram matching, CLAHE for low-contrast X-ray, or converting grayscale to 3-channel by replication.
2. Feature-level adaptation — align feature distributions using MMD, CORAL, or adversarial domain discriminators (DANN). Useful when you have unlabeled target images.
3. Parameter-level adaptation — fine-tune with a very low learning rate, or use adapters/LoRA modules that inject small trainable parameters while keeping the backbone frozen.

For most industrial teams, input-level normalization plus partial fine-tuning delivers 80–90% of the benefit at a fraction of the complexity. Feature-level methods are worth considering when labeled target data is scarce but unlabeled target images are abundant — a common situation when a new product line is ramping up.

Choosing a Strategy: A Decision Framework

The right choice depends on three factors: dataset size, domain similarity, and accuracy target. The diagram below maps these factors to a recommended strategy.

Practical Considerations and Failure Modes

Transfer learning is not a guarantee of success. Common failure modes in industrial settings include:

• Negative transfer — the pretrained features actively hurt performance because the source and target domains are too different. Symptom: fine-tuned model underperforms a model trained from scratch on the same data. Remedy: try a different pretraining source or reduce the number of unfrozen layers.
• Catastrophic forgetting — aggressive fine-tuning destroys useful pretrained features. Symptom: training loss drops fast but validation accuracy degrades. Remedy: lower the learning rate, use discriminative learning rates (lower for early layers), or freeze more layers.
• BatchNorm mismatch — running statistics from ImageNet do not match industrial image statistics. Symptom: unstable validation metrics, especially with small batches. Remedy: keep BN in eval mode during early stages, or use GroupNorm.
• Class imbalance amplification — rare defect classes get overwhelmed during fine-tuning. Remedy: use class-weighted loss, focal loss, or oversample rare defects.

What to Deliver

For a typical industrial inspection project, the transfer learning workflow should produce:

1. A documented baseline: backbone name, pretraining source, feature-extraction accuracy, and confusion matrix.
2. A fine-tuning configuration: which layers were unfrozen, learning rates per stage, batch size, augmentation policy, and number of epochs.
3. A validation protocol: held-out set composition, metric (macro-F1 is preferred over accuracy for imbalanced defects), and acceptance threshold.
4. A domain adaptation note: any input normalization applied, and whether feature-level alignment was used.

These artifacts make the result reproducible and allow a colleague to diagnose whether a future accuracy drop is due to data drift, a changed backbone, or a training configuration error.