Structured Sparsity with Reparameterized Batch Normalization

Group sparsity is a practical approach to accelerating neural network inference, while reparameterization provides a promising mechanism for inducing sparsity. Existing reparameterization-based approaches to group sparsity generally apply reparameterization in filter space. However, this is not the only possible location for inducing sparsity. In this work, we find empirically that applying reparameterization in the Batch Normalization (BN) space achieves better accuracy–sparsity tradeoffs and identifies higher-quality sparse architectures. Experiments further show that BN-space reparameterization is substantially more robust to the BN stabilization parameter $ϵ$ than filter-space reparameterization. To understand these observations, we examine the optimization dynamics and find that BN suppresses the task sensitivity of filter-space gates, whereas BN-space gates remain coupled to the task loss, allowing task feedback to guide sparsification consistently.