3. Characterizing noise

Noise is not a single phenomenon. In robust machine learning, the term covers several distinct corruption mechanisms that enter a system at different points, obey different statistical laws, and demand different defenses. A model that survives 20% symmetric label noise may collapse under a 2% adversarial perturbation budget, and a system that filters annotation errors aggressively may still be defeated by feature noise injected at inference time. This chapter builds a working taxonomy of noise sources in both supervised and unsupervised settings, with the goal of letting you identify which mechanism is actually degrading your pipeline before you choose a mitigation.

Where Noise Enters the Pipeline

Consider a standard supervised pipeline: raw observations are collected, features are extracted, labels are assigned (by humans, heuristics, or upstream models), and a learner fits a mapping from features to labels. Noise can enter at each stage, and the stage determines its mathematical character.

• Feature noise corrupts the input vector x. It may be additive (x' = x + ε), multiplicative, missing-value masking, quantization, or sensor-specific distortion. It changes the geometry of the input space.
• Label noise corrupts the target y. It is typically modeled as a noise transition matrix T where T[i][j] = P(observed label = j | true label = i). Symmetric noise spreads probability uniformly; asymmetric noise concentrates it on a specific confusable class.
• Annotation errors are the human process that produces label noise. They include inter-annotator disagreement, guideline ambiguity, fatigue, and systematic bias. They are the cause; label noise is the effect.
• Adversarial noise is worst-case, input-dependent perturbation optimized to maximize loss. Unlike random noise, it is not drawn from a distribution — it is chosen by an adversary with knowledge of the model.

The distinction matters because the defenses are not interchangeable. Random feature noise is often handled by data augmentation and robust losses; adversarial noise requires adversarial training or certified defenses; label noise requires loss correction, sample reweighting, or noise-robust architectures.

Label Noise: Structure Determines Difficulty

Label noise is characterized by its transition structure. In symmetric (uniform) noise, a true label is flipped to any other class with equal probability. With C classes and noise rate η, the transition matrix is T = (1-η)I + η/C · (J - I), where J is the all-ones matrix. This is the easiest case to handle because the noise is class-independent and unbiased.

Asymmetric noise is harder. If "cat" is confused with "dog" 30% of the time but never with "car," the noise is structured. A classifier trained with standard cross-entropy will learn the confusion pattern and reproduce it at test time. The transition matrix is no longer uniform, and simple loss correction must estimate T per class pair.

Instance-dependent noise is the hardest: the flip probability depends on the feature vector itself. Ambiguous or low-quality inputs are more likely to be mislabeled. This breaks the assumption that noise is independent of x, which most theoretical guarantees rely on.

Feature Noise: Geometry and Missingness

Feature noise changes the input distribution. Additive Gaussian noise x' = x + N(0, σ²I) is the textbook case, but real systems encounter more varied forms: sensor dropout (random features set to zero), quantization (continuous values snapped to discrete bins), and covariate shift (the input distribution changes between training and deployment).

The key parameter is the signal-to-noise ratio (SNR). For a feature with variance σ²_signal and noise variance σ²_noise, SNR = σ²_signal / σ²_noise. When SNR drops below roughly 1, the feature carries more noise than signal, and the learner must rely on other features or regularization to compensate.

In unsupervised settings, feature noise directly corrupts the geometry that clustering and dimensionality reduction rely on. A Gaussian mixture model with additive noise will merge components; PCA will rotate its principal axes toward the noise directions. Denoising autoencoders address this by training the encoder to reconstruct clean inputs from corrupted ones, effectively learning a projection that discards noise directions.

Adversarial Noise: Worst-Case Perturbation

Adversarial noise is defined by an optimization problem, not a distribution. Given a model f_θ, input x, and true label y, the adversary solves:

δ* = argmax_{||δ||_p ≤ ε} L(f_θ(x + δ), y)

The perturbation budget ε is typically small — for images, ε = 8/255 in L∞ norm is a common benchmark. The resulting δ* is imperceptible to humans but causes misclassification. Unlike random noise, adversarial noise is input-dependent and model-dependent: the same image gets different perturbations against different models.

Adversarial noise is not a data quality problem — it is a security problem. It appears at inference time, not training time, and it is chosen by an actor with knowledge of the model. Defenses include adversarial training (augmenting the training set with adversarial examples), input preprocessing (JPEG compression, bit-depth reduction), and certified defenses (randomized smoothing, interval bound propagation).

Annotation Errors: The Human Layer

Annotation errors are the process-level cause of label noise. They arise from:

• Inter-annotator disagreement: two annotators assign different labels to the same sample. Measured by Cohen's kappa or Krippendorff's alpha.
• Guideline ambiguity: the labeling instructions do not cover edge cases, so annotators improvise.
• Systematic bias: an annotator consistently misreads a specific class due to fatigue, training gaps, or cultural assumptions.
• Adversarial annotation: crowdworkers deliberately mislabel to maximize pay per task.

The practical response is not to eliminate annotation errors — that is impossible — but to measure and model them. Gold-standard subsets, multiple annotations per sample, and agreement metrics give you an estimate of the noise rate. If kappa is below 0.6, the label noise is likely too high for the model to learn reliably, and the guideline or annotator pool needs revision before training.

Unsupervised Settings: No Labels, Same Noise

In unsupervised learning, label noise is absent by definition, but feature noise and adversarial noise remain. Clustering algorithms are particularly sensitive to feature noise because they rely on distance metrics. A single noisy feature with high variance can dominate the distance calculation and pull cluster assignments toward noise directions.

Adversarial noise in unsupervised settings targets the learned representation. An adversary can perturb inputs to shift cluster assignments, corrupt the latent space of an autoencoder, or cause a density estimator to assign high likelihood to out-of-distribution samples. Defenses include robust distance metrics (Mahalanobis instead of Euclidean), contrastive learning with noise augmentation, and consistency regularization across perturbed views.

Summary Table: Noise Types at a Glance

The next chapter moves from characterization to measurement: how to estimate noise rates, detect mislabeled samples, and quantify the gap between clean and noisy performance.