- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper applies a variation of saliency score averaged over time to dynamically prune convolutional neural nets, in an unstructured with fine-tuning setting. A specific metric, pruning impact, is further discussed and studied.
Critical strengths
None
Critical weaknesses
Weak Contributions: The listed main contributions of the paper are all hardly novel. The proposed score is simply a variant of the well-known saliency score, already studied under various formulations. Aggregating the score over multiple updates is not novel as well. While the authors correctly cite relevant works with similar accumulation strategies, they fail to establish why their specific formulation is superior beyond limited empirical observations. Finally, the stated last contribution--a metric--is simply the monitoring of the degradation at pruning observations. Moreover, there is an important flaw with the proposed metric. Pruning aims to sparsify a model while preserving its performance compared to its dense counterpart. As presented, a pruning intervention degrading learning could display advantageous pruning impact while never achieving decent performance throughout training.
Weak positioning: While the paper is positioned as an unstructured gradual pruning method with fine-tuning, it fails to position the work with respect to its pruning objective, which is necessary to establish the proper benchmark requirements. If the goal is to reduce training FLOPs, it becomes necessary to also report the computation budget. If the goal is simply to achieve the best final accuracy vs sparsity, then benchmarking must extend beyond gradual pruning methods to encompass more broadly other pruning approaches sharing the same objective.
Weak benchmarks: Reported benchmark is limited to three architectures and three datasets, omitting in particular the normally expected benchmarking on ImageNet. Transferability to transformers is also omitted. Chase [1] (for which its unstructured variant is a relevant baseline to the current work) is a good example of standard empirical expectation in pruning literature. Moreover:
- Dense performance is not reported for most experiments, making it impossible to compare degradation across methods, and to assess if the base training recipe is competitive.
- There are missing key comparisons to claim SOTA comparable performance. Table 2 (b-c) of SPFK [2] exemplifies what is missing for gradual unstructured pruning.
Flawed methodology: Gradual pruning comparisons are expected with matched underlying training recipes. Reporting as a main result (Table 3) comparison between methods with different learning rate schedules raises questions about the validity of all reported main results. If the proposed method performs differently under varied training regimes, matched comparisons across all those regimes should be provided. It shouldn't be relegated to an ablation study over a single setting. Similarly, not matching the introduces confounding factors invalidating the presented comparisons. When setups differ, matching experiments across setups should be provided.
Reproducibility issues: With the code unreleased and the training hyperparameters of reproduced methods (GraNet and FGGP) missing, the paper in its current version suffers from a major reproducibility issue. In particular, this makes it impossible to verify if the comparisons were performed on an equivalent basis.
References:
[1] Lu Yin, Gen Li, Meng Fang, Li Shen, Tianjin Huang, Zhangyang Wang, Vlado Menkovski, Xiaolong Ma, Mykola Pechenizkiy, Shiwei Liu: Dynamic Sparsity Is Channel-Level Sparsity Learner. NeurIPS 2023
[2]: Zhanfeng Mo, Haosen Shi, Sinno Jialin Pan: Probabilistic Neural Pruning via Sparsity Evolutionary Fokker-Planck-Kolmogorov Equation. ICLR 2025
Questions for authors
Please note that all remarks below are suggestions for future work to the authors:
- Please revise the excitation level raised by the observation that pruning can improve performance at some sparsity levels. This is amply discussed in pruning literature, in particular in the original Lottery Ticket Hypothesis paper, cited in this work.
- The overall manuscript is currently giving an impression of non-polished, uncompleted work. Background section spans three pages, conclusion spans an entire page while only restating previous content and implementation details occupy most of the results section. Much of those could be presented in the appendix, leaving ample place to develop further the proposed contributions.
- The background section, as currently organized, is alternating between presenting relevant works and the methods. Splitting the two sections, and getting rid of the unnecessary, would improve clarity.
Dear reviewer, Thank you for your thorough review and for providing valuable insights on our work allowing us to strengthen our work. Below are our responses to your comments and related modifications in the paper.
Weak Contributions: “The proposed score is simply a variant of the well-known saliency score, already studied under various formulations. Aggregating the score over multiple updates is not novel as well. While the authors correctly cite relevant works with similar accumulation strategies, they fail to establish why their specific formulation is superior beyond limited empirical observations.”
Our research studies continuous pruning while training from a random initialized network; thus, most assumptions about gradients in converged networks do not hold. To our knowledge, all works proposing similar saliency scores are iterative pruning methods applied to pretrained networks. Thus, the novelty is a one-step first order Taylor approximation as saliency score for gradual pruning while training. We have further clarified this in the introduction section also comparatively to the closest literature.
Further, in the appendix, we have provided a seed-wise analysis comparing aggregation before and after multiplication to further establish the advantage of aggregating before multiplication.
“Finally, the stated last contribution--a metric--is simply the monitoring of the degradation at pruning observations.” To the best of our knowledge, such metric was not used for evaluating pruning approaches. In our paper we show this metric to be a good complement and potentially even more indicative than accuracy.
“Pruning aims to sparsify a model while preserving its performance compared to its dense counterpart. As presented, a pruning intervention degrading learning could display advantageous pruning impact while never achieving decent performance throughout training.” This is partly true, and this is why PrIm should be used in combination with accuracy when evaluating pruning algorithms. This is also discussed in our paper: “Moreover, it does not capture subsequent recovery, nor whether the resulting pruned network is more or less serviceable by following training. ”
Nonetheless, the direct impact of pruning is an interesting metric to monitor since the aim of most saliency scores is to find the weights with the least effect on network performance. When comparing PrIm for TAPER and GraNet, it is clear that TAPER is better at finding these weights.
Weak positioning: “If the goal is simply to achieve the best final accuracy vs sparsity, then benchmarking must extend beyond gradual pruning methods to encompass more broadly other pruning approaches sharing the same objective” Our goal is the best final accuracy vs sparsity in high-sparsity regimes. We have added an additional table (Table 2), with results from EAST, a method especially developed for high sparsity regimes. Our benchmark comparisons now cover gradual, sparse to sparse and one-shot pruning methods.
Weak benchmarks: We have, following your suggestion, added experiments on the full ImageNet-1K where TAPER is still superior to GraNet. Further, dense accuracies are now reported for all dataset/network pairs, thank you for pointing this out.
“There are missing key comparisons to claim SOTA comparable performance. Table 2 (b-c) of SPFK [2] exemplifies what is missing for gradual unstructured pruning.” The methods listed in your referenced table, is iterative (10 iterations) unstructured pruning from a pretrained network which is a completely different pruning setup than ours with up to 110 pruning iterations, training and pruning simultaneously from a randomly initialized network. Thus, we do not see these methods as key comparisons.
Flawed methodology: “Gradual pruning comparisons are expected with matched underlying training recipes. Reporting as a main result (Table 3) comparison between methods with different learning rate schedules raises questions about the validity of all reported main results.” We see experiments 1-2 as an evaluation of complete methods, including training hyper parameters, pruning schedules and important scores. However, we have included new experiments in the ablation section with TAPER using the same exact setup as GraNet to provide a proper comparison of the importance scores while we keep our proposed TAPER settings for experiments 1-2. Furthermore, for Tiny-ImageNet and ImageNet, both methods use the same LR schedule (cosine) and pruning step.
Reproducibility issues: The code has now been made available here: https://anonymous.4open.science/r/TAPER-29E8/ and the training hyperparameters for GraNet and FGGP have been added to the Appendix.