- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper focuses on applying sparsity-inducing reparametrization within the BN space instead of the traditional filter space. They introduce BN-gating, where the reparametrization is done w.r.t. the scale parameter of the respective BN layer instead of reparametrizing w.r.t. the respective filter, as in the established D-gating approach. Besides showing empirically that the proposed approach outperforms D-gating regarding the sparsity x accuracy trade-off, they also analyze the underlying optimization dynamics. Here they find that the key lies in the effect on the task loss, as BN in D-gating can harm task sensitivity, whereas the reparametrization of the scale parameter directly couples task loss and sparsification and thereby mitigates this.
Critical strengths
- Research idea: Combining BN-based approaches with reparametrization seems like a natural extension of these existing ideas and is thus an interesting aspect to focus on.
- Analysis depth: Connecting the position at which to apply gating to the effect on task loss sensitivity is a nice analysis and sheds further insight into the motivation behind this and existing BN-based approaches.
Critical weaknesses
-
Novelty and Scope: 1.1) From what I understand, one strength of D-Gating is its versatility, s.t. can be applied to a vast range of model architectures (CNNs, transformer-style models, etc.). Explicitly focusing on BN layers takes away a crucial amount of this versatility and makes the proposed method feel rather constrained. 1.2) As already stated, I like the idea, yet the work right now feels pretty incremental (see question 1).
-
Evaluation: 2.1) I understand that the primary objective is to present BN-gating as a direct competitor to D-Gating, and that the evaluation thus focuses (though not exclusively) on these two methods. However, I still expect a more thorough evaluation across more datasets and/or more sophisticated model architectures (see question 4). 2.2) I think the comparison in Table 1 between linearly interpolated values for D-Gating isn't fair, especially since this is one of the strongest claims in the paper. I expect an exact run and the corresponding sparsity strength reported across runs.
Questions for authors
-
Regarding weakness 1.2): Couldn't one argue that BN-Gating is Network-slimming with a different optimization approach? In this light, how can one then explain the different behaviour in Figure 3b) for network slimming and BN-gating?
-
Couldn't one argue that the analysis in Theorem 1 introduces as a new sparsity-related hyperparameter for D-Gating? Based on this, couldn't one argue to simply tune D-Gating jointly over both and , thereby keeping the versatility of D-Gating whilst mitigating the raised concerns?
-
What are the FLOPs of BN-gating, D-Gating and the other baseline methods?
-
Are there CNN architectures where you expect the gap between D-Gating and BN-Gating to widen? If so, why, and can you show it empirically?