- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper compares different allocation/correction pairings for single-shot post-training pruning with label-free data. Selection strategies for the allocation method are further studied.
Critical strengths
None
Critical weaknesses
Limited novelty : While a central claim of the paper is to shed new light on the allocation-repair pairing, this is hardly a new research direction. Lazerevich 2021 [1]--cited in the paper--already reports similar comparison across different sparsity selection criterion and correction methods. Moreover, their comparisons across allocation strategies already include LAMP and two magnitude variants. With respect to competitive allocation, the current paper only adds ERK (2020) to this existing benchmarking. Given that layerwise sparsity allocation is a very active topic, it is hard to justify not adding more recent proposed methods such as [2], [3] to list only two.
Flawed methodology : A fundamental practice in ML is to report results on an held-out test set after tuning on a train set. However, the results used to compare selection strategies in the main text are the ones LDMS- was tuned on, as evidenced by Table 4, Appendix C and Appendix I. Limited results on an held-out set is solely reported, without reference, in Appendix I and show severe deterioration of the criterion. On top of that, three different variables (sharing the same notation) are introduced in the definition of LDMS---without precising to which numerical values they were set nor explaining how they were selected.
Weak evaluations: Reported results are overall very limited and restricted to convolution architectures paired with small datasets; meaningful results on ImageNet-1k and transformer models are entirely missing. Moreover, base dense performance is often not reported and standard deviations are omitted from the main text--a verification in the Appendix shows those are often significant.
Other major issue: Lines 296-298 state that LDMS- selection is unchanged across a sweep. Reported numbers for that sweep in Table 4 of the Appendix appear to contradict directly this statement. Direct reference to this table is missing from the main text.
References:
[1]: Ivan Lazarevich, Alexander Kozlov, Nikita Malinin: Post-training deep neural network pruning via layer-wise calibration. ICCVW 2021
[2]: Kaixin Xu, Zhe Wang, Xue Geng, Min Wu, Xiaoli Li, Weisi Lin: Efficient Joint Optimization of Layer-Adaptive Weight Pruning in Deep Neural Networks. ICCV 2023
[3]: Ruihao Gong, Yang Yong, Zining Wang, Jinyang Guo, Xiuying Wei, Yuqing Ma, Xianglong Liu: Fast and Controllable Post-training Sparsity: Learning Optimal Sparsity Allocation with Global Constraint in Minutes. AAAI 2024
Questions for authors
Additional suggestions to the authors:
-
The reported results on ImageNet-1K in Table 19 are vacuous, with no method achieving relevant performance and by being limited to a single seed. Lazarevich (2021) reports significant results in the same setting with sparsity set at 50%, which could serve as a good starting point.
-
The authors should further justify the need for a pre-repair selection strategy such as LDMS- which requires an already trained and pruned network to be evaluated. The remaining steps, repair and evaluation, are already very cheap compared to training.
-
Emphasis should be put on presenting message-carrying figures rather than good looking figures--Figures 1 and 2 both carry very little information. In particular for Figure 1, the Unlabeled Calibration Set pictogram of (a) appears misplaced and missing axis definitions/graduations make (d) hard to interpret.