acceptodds
Under review as a conference paper at ICLR 2027

Structured Sparsity with Reparameterized Batch Normalization

Abstract

Group sparsity is a practical approach to accelerating neural network inference, while reparameterization provides a promising mechanism for inducing sparsity. Existing reparameterization-based approaches to group sparsity generally apply reparameterization in filter space. However, this is not the only possible location for inducing sparsity. In this work, we find empirically that applying reparameterization in the Batch Normalization (BN) space achieves better accuracy–sparsity tradeoffs and identifies higher-quality sparse architectures. Experiments further show that BN-space reparameterization is substantially more robust to the BN stabilization parameter than filter-space reparameterization. To understand these observations, we examine the optimization dynamics and find that BN suppresses the task sensitivity of filter-space gates, whereas BN-space gates remain coupled to the task loss, allowing task feedback to guide sparsification consistently.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
sparsity optimization, group sparsity, reparameterization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FQQcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper focuses on applying sparsity-inducing reparametrization within the BN space instead of the traditional filter space. They introduce BN-gating, where the reparametrization is done w.r.t. the scale parameter of the respective BN layer instead of reparametrizing w.r.t. the respective filter, as in the established D-gating approach. Besides showing empirically that the proposed approach outperforms D-gating regarding the sparsity x accuracy trade-off, they also analyze the underlying optimization dynamics. Here they find that the key lies in the effect on the task loss, as BN in D-gating can harm task sensitivity, whereas the reparametrization of the scale parameter directly couples task loss and sparsification and thereby mitigates this.

Critical strengths
  • Research idea: Combining BN-based approaches with reparametrization seems like a natural extension of these existing ideas and is thus an interesting aspect to focus on.
  • Analysis depth: Connecting the position at which to apply gating to the effect on task loss sensitivity is a nice analysis and sheds further insight into the motivation behind this and existing BN-based approaches.
Critical weaknesses
  1. Novelty and Scope: 1.1) From what I understand, one strength of D-Gating is its versatility, s.t. can be applied to a vast range of model architectures (CNNs, transformer-style models, etc.). Explicitly focusing on BN layers takes away a crucial amount of this versatility and makes the proposed method feel rather constrained. 1.2) As already stated, I like the idea, yet the work right now feels pretty incremental (see question 1).

  2. Evaluation: 2.1) I understand that the primary objective is to present BN-gating as a direct competitor to D-Gating, and that the evaluation thus focuses (though not exclusively) on these two methods. However, I still expect a more thorough evaluation across more datasets and/or more sophisticated model architectures (see question 4). 2.2) I think the comparison in Table 1 between linearly interpolated values for D-Gating isn't fair, especially since this is one of the strongest claims in the paper. I expect an exact run and the corresponding sparsity strength reported across runs.

Questions for authors
  1. Regarding weakness 1.2): Couldn't one argue that BN-Gating is Network-slimming with a different optimization approach? In this light, how can one then explain the different behaviour in Figure 3b) for network slimming and BN-gating?

  2. Couldn't one argue that the analysis in Theorem 1 introduces ϵ\epsilon as a new sparsity-related hyperparameter for D-Gating? Based on this, couldn't one argue to simply tune D-Gating jointly over both λ\lambda and ϵ\epsilon, thereby keeping the versatility of D-Gating whilst mitigating the raised concerns?

  3. What are the FLOPs of BN-gating, D-Gating and the other baseline methods?

  4. Are there CNN architectures where you expect the gap between D-Gating and BN-Gating to widen? If so, why, and can you show it empirically?

Reviewer_m1zFOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a structured pruning algorithm called BN Gating. The algorithm is a combination of reparameterization and sparse constraints that operators on BN layers. The gating is done in a multiplicative way. The results show the proposed algorithm can have an improved performance in general compared to D-Gating.

Critical strengths
  • Reveal that D-gating's performance is sensitive to the value of ϵ\epsilon.
Critical weaknesses
  • The authors missed a large chunk of literature. There are a lot of works during 2020~2023 trying to develop similar ideas yet the authors did not discuss them. For example, Polarization Regularizer [1] also did pruning based on BN layers. GDP [2] and Gate Decorator [3] have very similar and relevant settings. There are definitely more papers out there yet the authors did not position their algorithm well. The comparison against baselines is therefore not sufficient as the authors only compare it against D-Gating.

  • Notations used inside this paper are not consistent. For example, there are superscripts on S in algorithms but never properly defined. The meaning of A is also ambiguous, which is different in D-Gating and Eq. (2). The definition of ``sparsity'' is ambiguous too as it can be in terms of percentages of 0 channels or 0 weights.

[1] Neuron-level Structured Pruning using Polarization Regularizer

[2] Gdp: Stabilized neural network pruning via gates with differentiable polarization

[3] Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks

Questions for authors

Please address above weaknesses.

Reviewer_jRJhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work presents BN-Gating which is a method that shifts multiplicative reparameterisation from the convolutional filters to the BN scaling. The idea stems from the observation that placing sparsity gates before BN causes the normalisation process to cancel out the gate scaling effect which in turn suppresses the task gradient until the gate magnitude approaches the BN stabilisation parameter ϵ\epsilon. To address this the authors propose to reparameterise the BN scale after normalisation, and their BN-gating ensures that the gating is directly associated with the task optimisation. The approach is empirically evaluated on the CIFAR-10/100, and imagine-100 on VGG and Resnet. Results show that BN-Gating achieves greater sparsity-accuracy tradeoffs compared to filter gating and traditional group-sparsity baselines.

Critical strengths
  1. The observation of the channel polarisation (Figure 4) adds valuable interpretability to the model. This demonstrates that multiplicative reparameterisation naturally forces a bimodal distribution therefore making the thresholding more robust and less reliant on arbitrary heuristics.
  2. Empirically, the work is good in places, although there are some limitations that are identified subsequently. Notably, The inclusion of the retraining from scratch evaluation is good to see to isolates the quality of the discovered sparse architecture proving that BN-Gating actually finds better sub-networks.
  3. The method itself is very simple but yields significant empirical improvements, which is in my opinion a major strength of the work! Beyond its immediate practical benefits for deployed models, this simplicity provides profound insights into the subtle impact normalisation has on optimisation geometry in pruning.
Critical weaknesses
  1. Arguably the presentation and the use of figures and space leaves some room for improvement, in places the work feels incomplete or under explored.
  2. FLOP reduction should be present in the main paper rather than appendix, you have space to move it, especially as it is heavily included in the concluding/trade-off discussion.
  3. Empirically, the work is lacking in terms of dataset, and task generalisation and breath of architectural approaches that include modern ones. It is expected that more modern architectures are explored, additional tasks, and more competitive benchmark datasets. Notably, you also omitted the CIFAR-100 and imagenet-100 results in places for example 5.3. The code is already there, why are you not at least comparing here?
  4. Section 3.4 claims that BN-Gating identifies higher-quality sparse architectures. However, this claim is only tested on a single arbitrary sparsity level per dataset. This needs to be extended to a wider range of sparsity levels to demonstrate completeness and avoid potential cherry picking.
  5. The primary results in Table 1, are marked with an asterisk to denote they were linearly interpolated. This should not estimated for baseline performance comparisons, you should have empirically tuned the baseline's regularisation strength to reach the target sparsity. This makes the primary empirical comparison feel incomplete and inconclusive.
  6. Comparisons to broader state-of-the-art structured pruning baselines are somewhat dated. Including more contemporary pruning techniques would better contextualise the performance.
Questions for authors
  1. Does the task-feedback suppression phenomenon you identified in BN also apply to other normalisation? Identifying this would be beneficial to architectures that do not use BN.
  2. How sensitive is the final sparsity level to small changes in the regularization strength λ\lambda and how is the tuning behaviour for this parameter to hit a pre-defined sparsity budget?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.