Structured vs. Unstructured Pruning: An Exponential Gap

This paper investigates the differences between structured (neuron) and unstructured (weight) pruning in neural networks, particularly in the context of the Strong Lottery Ticket Hypothesis (SLTH). It demonstrates that approximating a single bias-free ReLU neuron through neuron pruning necessitates a network size of $\Omega(1/\varepsilon)$, while weight pruning can achieve the same approximation with only $O(\log(1/\varepsilon))$ hidden units, highlighting an exponential gap in efficiency between the two methods. This distinction is crucial for practitioners aiming to optimize neural network architectures for performance and resource utilization in AI applications.