4. Pruning and sparsity

Pruning removes parameters from a trained network to reduce memory footprint, compute, and energy. The critical engineering reality is that not all sparsity translates into real speedup. A model can be 90% sparse and still run at the same latency as its dense counterpart if the sparsity pattern does not match the hardware's execution model. This chapter builds the mental model needed to choose a pruning strategy that actually pays off on edge targets.

Two Fundamental Sparsity Patterns

Unstructured pruning (fine-grained) zeroes out individual weights anywhere in a tensor. It achieves the highest compression for a given accuracy budget because it has maximum freedom in choosing which weights to drop. The output is an irregular sparsity mask.

Structured pruning (coarse-grained) removes entire structural units — whole filters, channels, attention heads, or rows/columns of a weight matrix. The resulting tensor shapes are genuinely smaller, so downstream layers receive fewer input channels and the arithmetic itself shrinks.

The trade-off is fundamental. Unstructured sparsity gives better accuracy at equal sparsity but usually yields no wall-clock speedup on general-purpose CPUs, GPUs, or NPUs unless the hardware has dedicated sparse-tensor support. Structured pruning gives smaller, denser tensors that every runtime accelerates, but it is a harsher constraint, so accuracy degrades faster at the same nominal sparsity ratio.

Magnitude Pruning vs. Movement Pruning

Once the pattern is chosen, the next question is which weights to remove. Two scoring families dominate.

Magnitude pruning scores each weight by its absolute value |w| and removes the smallest. It is simple, training-free (works on a pretrained checkpoint), and effective when the pretrained model is already close to the target task. Its weakness appears during fine-tuning: a weight that is currently small may be on its way to becoming important, and magnitude pruning kills it prematurely.

Movement pruning scores weights by how much they move away from zero during fine-tuning. The score is essentially the product of the weight and its gradient signal accumulated over training. Weights that the optimizer is actively pushing away from zero are protected; weights drifting toward zero are removed. This is especially valuable in transfer learning, where the pretrained initialization is far from the target optimum and magnitude is a poor proxy for importance.

# Magnitude score (training-free)
score = abs(weight)

# Movement score (accumulated during fine-tuning)
# m_i tracks how far weight i has moved from its initial value
movement_score = weight * accumulated_gradient_signal
# Weights with negative movement are being pushed toward zero -> prune

In practice, magnitude pruning with iterative retraining is the default baseline. Movement pruning is worth the extra bookkeeping when fine-tuning a large pretrained backbone on a small target dataset, where the accuracy gap between the two methods can be several points at high sparsity.

Hardware-Aware Sparsity

The decisive factor on edge devices is whether the target accelerator can exploit the sparsity pattern. Three cases matter:

• Dense-only hardware (most mobile CPUs, many DSPs): only structured pruning produces speedup. Unstructured masks are stored but the multiply-accumulate units still process zeros.
• N:M structured sparsity (NVIDIA Ampere and later, some NPUs): the hardware requires exactly N non-zeros in every block of M consecutive weights. The canonical case is 2:4 sparsity — at most 2 non-zeros per 4 elements — which enables roughly 2× throughput on tensor cores. You must prune to this exact pattern, not arbitrary sparsity.
• Block sparsity (some edge NPUs): sparsity is defined over fixed-size blocks (e.g., 4×4 or 1×16). Whole blocks are zeroed, matching the hardware's tiling.

Choosing a Strategy: A Decision Path

The following diagram maps the decision from deployment constraint to pruning method. It is the practical core of this chapter: start from the hardware, not from the algorithm.

An Engineering Walkthrough: 2:4 Pruning a Convolutional Backbone

Consider a MobileNet-style backbone targeting an edge NPU with 2:4 sparse tensor core support. The goal is 2× MAC throughput with less than 1% top-1 accuracy drop on a 10-class industrial defect dataset.

Inputs and setup. A pretrained checkpoint, a calibration set of ~2,000 labeled images, and the target NPU's compiler toolchain. The constraint is 2:4 sparsity on all convolutional weight tensors; depthwise and fully-connected layers may be handled separately.

Procedure. First, fine-tune the dense model for a few epochs on the target dataset to establish a baseline accuracy (for example, 96.2% top-1). Then apply magnitude-based 2:4 pruning: for every group of 4 consecutive weights along the input-channel axis, keep the 2 with the largest |w| and zero the rest. This is a hard constraint, so expect an immediate accuracy drop — often 2–4 points. Recover it with 10–20 epochs of fine-tuning while keeping the 2:4 mask fixed. Optionally, allow the mask to be re-selected every few epochs (prune–regrow), which typically recovers another 0.5–1 point.

# Pseudocode: 2:4 magnitude mask on a weight tensor
# W shape: [out_channels, in_channels, kH, kW]
# Reshape so groups of 4 run along in_channels
Wg = W.reshape(out_channels, in_channels // 4, 4, kH, kW)
# Keep top-2 magnitude per group
_, idx = Wg.abs().topk(2, dim=2)
mask = zeros_like(Wg)
mask.scatter_(2, idx, 1)
W_pruned = Wg * mask

What to observe. After pruning and recovery, check three things: (1) accuracy on the held-out set — target within 1% of baseline; (2) the actual sparsity ratio reported by the compiler — it must be exactly 2:4, not "approximately"; (3) measured latency on the device. If latency does not improve, the compiler likely fell back to a dense kernel because the mask was not recognized — a common failure mode when the sparsity pattern is applied after export rather than baked into the graph.

Practical Guidance

• Start structured, not unstructured. Unless the target has documented sparse-tensor support, structured pruning is the only path to real speedup.
• Match the pattern to the hardware spec. Read the accelerator's sparsity documentation before choosing N:M or block granularity. A 2:4 mask on a block-sparse NPU is wasted effort.
• Use magnitude as the default scorer; switch to movement for transfer learning. The extra cost of tracking gradient movement is justified when the pretrained initialization is far from the target task.
• Always fine-tune after pruning. One-shot pruning without recovery almost always loses more accuracy than necessary.
• Measure on the target. Parameter count, FLOPs, and file size are proxies. Latency and energy on the actual device are the only metrics that matter.

Pruning is not a single operation but a co-design decision: the sparsity pattern, the scoring method, and the hardware's execution model must agree. When they do, sparsity delivers genuine edge efficiency; when they do not, it delivers only a smaller file and a false sense of optimization.