3. Quantization fundamentals

Quantization is the single most widely deployed optimization in edge AI. It reduces the numeric precision of weights and activations—typically from 32-bit floating point (FP32) to 8-bit integers (INT8)—cutting memory footprint by roughly 4×, shrinking model size, and enabling integer arithmetic that runs far faster on NPUs, DSPs, and MCUs. The trade-off is accuracy: every value must be mapped onto a coarse grid, and the engineering task is to choose that grid so the model's behavior barely changes.

This chapter covers the three pillars of practical quantization: the affine mapping that defines how floats become integers, the two dominant workflows (post-training quantization and quantization-aware training), and the calibration strategies that determine where the quantization grid is placed.

The Affine Quantization Mapping

Most edge runtimes use affine (asymmetric) quantization, which maps a real value r to an integer q via a scale s and a zero-point z:

q = round(r / s) + z
r ≈ s * (q - z)

For signed INT8, q is clamped to [-128, 127]. The scale s is derived from the observed range [r_min, r_max]:

s = (r_max - r_min) / (q_max - q_min)
z = round(q_min - r_min / s)

The zero-point z is the integer that exactly represents the real value 0.0. This matters because zero-padding in convolutions and ReLU outputs must map to an exact integer, otherwise padding introduces a systematic bias. Symmetric quantization forces z = 0 and is common for weights, where the distribution is roughly centered; asymmetric quantization is preferred for activations such as post-ReLU tensors, whose range is strictly non-negative.

Two granularity choices dominate. Per-tensor quantization uses one (s, z) pair for an entire tensor—simple and cheap. Per-channel quantization assigns a separate scale per output channel of a convolution or per row of a fully connected weight matrix. Per-channel weight quantization typically recovers most of the accuracy lost by per-tensor quantization at negligible runtime cost, and it is the default in TensorFlow Lite and ONNX Runtime for weights.

Post-Training Quantization (PTQ)

PTQ quantizes a model that has already been trained in FP32, without retraining. The workflow is:

1. Export the trained model to a supported format (e.g., ONNX or a SavedModel).
2. Run a representative dataset of a few hundred unlabeled samples through the model to observe activation ranges.
3. Compute scales and zero-points from those observations.
4. Convert weights and insert quantize/dequantize nodes at boundaries.

PTQ is fast—minutes to an hour—and requires no labels or training infrastructure. It works well for convolutional networks with ReLU activations, where activation distributions are bounded and well-behaved. It degrades sharply for transformers, depthwise convolutions, and models with wide dynamic-range activations (e.g., softmax outputs, LayerNorm inputs), because a single outlier can stretch the scale and crush the resolution of the remaining values.

Quantization-Aware Training (QAT)

QAT simulates quantization during training so the network learns to compensate. The key mechanism is the fake-quantization node: in the forward pass, weights and activations are quantized to INT8 and immediately dequantized back to FP32, so the loss sees the quantization error. In the backward pass, gradients flow through the quantizer using a straight-through estimator (STE), which treats the rounding operation as identity:

# Forward: simulate INT8
q = clamp(round(w / s) + z, -128, 127)
w_fake = s * (q - z)

# Backward: STE — pass gradient through unchanged
dL/dw = dL/dw_fake

A typical QAT schedule fine-tunes the FP32 model for a small number of epochs (often 5–20) with a reduced learning rate, then freezes the scales and exports the integer model. QAT consistently recovers most of the accuracy gap, especially for models below ~5M parameters and for transformer-based architectures. The cost is engineering complexity: you need the training pipeline, labeled data, and a framework that supports fake-quant nodes (PyTorch's torch.ao.quantization, TensorFlow's tf.quantization).

Calibration Strategies

Calibration determines the activation ranges used to compute scales. It is the highest-leverage decision in PTQ and a critical detail in QAT. Four strategies are common:

Percentile and KL divergence both trade a small amount of clipping for much finer resolution on the bulk of the distribution. In practice, start with KL divergence for CNNs and percentile (99.9–99.99%) for transformers, then validate on a held-out set.

The representative dataset itself matters as much as the algorithm. It should cover the deployment distribution—lighting conditions, sensor noise, class balance—but need not be labeled. A few hundred samples are usually sufficient; more than a thousand rarely helps. Critically, the calibration set must not be the training set, or the observed ranges will be optimistically narrow.

Choosing Between PTQ and QAT

The decision is driven by the accuracy budget and the model architecture. A practical rule of thumb:

• PTQ first. It is cheap and often sufficient for CNNs with per-channel weights and KL calibration.
• Escalate to QAT if PTQ loses more than ~1–2% of the target metric, or if the model uses transformers, depthwise separable convolutions, or LayerNorm.
• Mixed precision is a middle path: keep sensitive layers (first/last, attention projections) in FP16 or FP32 and quantize the rest. This is supported by most runtimes and often closes the gap without full QAT.

The diagram below summarizes how the three components—mapping, workflow, and calibration—interact across a typical deployment pipeline.

Practical Validation Checklist

After quantization, validate before deployment. The following checks catch the majority of failures:

• Numerical parity: compare FP32 and INT8 outputs on the same input. Max absolute difference should be small relative to the output range; large spikes indicate a bad scale on a specific layer.
• Per-layer error: instrument each quantized layer and compute cosine similarity between FP32 and INT8 outputs. Layers below ~0.99 are candidates for mixed precision.
• End-to-end metric: run the full validation set. Report the delta, not just the absolute number.
• Latency and memory: confirm the runtime actually uses integer kernels. A model that is "quantized" but falls back to FP32 kernels gains nothing.

Quantization is not a single switch but a design space: the mapping defines the grid, the workflow decides whether the model adapts to it, and calibration decides where the grid sits. Mastering these three levers is what separates a model that merely fits on an edge device from one that still performs.