- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a way to train sparse neural networks toward a user-specified sparsity level without repeatedly running training with different regularization parameters. It introduces two adaptive methods, TopK and SGap, which adjust the regularization parameter \lambda during training, and extends them with a gradual sparsity schedule. Experiments on CIFAR-100, TinyImageNet, and ImageNet-1K show that the proposed methods can reach target sparsity levels and often improve accuracy over fixed-\lambda training, especially when gradual sparsification is used.
Critical strengths
- The proposed TopK and SGap methods adapt the regularization parameter \lambda online, reducing the need for repeated training runs to find a suitable sparsity level, which is technically sound.
- Experiments across CIFAR-100, TinyImageNet, and ImageNet-1K show that the proposed methods achieve high sparsity while matching or outperforming fixed-\lambda baselines, particularly with gradual sparsity schedules.
- This paper combines practical methods with theoretical analysis and provides two complementary adaptation strategies, introduces scheduled sparsification, and analyzes convergence, while connecting TopK-based mirror descent to dynamic sparse training.
Critical weaknesses
- The novelty is limited. Setting \lambda to the (k*+1)-th largest |p| makes the soft-threshold step a top-k selection on the dual (accumulated-gradient) variable, with uniform shrinkage. This is very close to existing top-k and IHT-style methods on accumulated gradients.
- Some baselines are missing:
- The paper does not compare to methods that hit a target sparsity in a single run, such as constrained L0 (Gallego-Posada et al. 2022) and STR (Kusupati et al. 2020), which have been discussed in this paper.
- Following the novelty, Top-KAST is a similar method that the paper cites but does not compare against.
- The proposed method keeps a dense dual variable p and computes dense gradients at every step. Training is therefore not memory-sparse, which undercuts the abstract’s claim that sparse training reduces memory and compute.
- Experiments are limited to image-classification CNNs. Other architectures such as transformers and LLMs, where sparsity control currently matters most, are not tested.
Questions for authors
- Evaluate on other architectures.
- Compare with more baselines (see above).
- Discuss the dense training cost (see above).
4 comments
W4: "Experiments are limited to image-classification CNNs ... Other architectures such as transformers and LLMs ... are not tested."
We agree that extending the evaluation to transformers is of great interest. Following the reviewer's suggestion, we trained DeiT-S [4] on ImageNet-1k. Given the cost of training, a complete comparison could not be finished within the rebuttal period. Nevertheless, we report our preliminary results at 90% sparsity and (only) 100 training epochs.
| Optimizer | Method | Top-1 acc. [%] | Best epoch |
| AdamW | Dense | 70.88 | 69 (training ongoing) |
| AdaBreg + SGap + Ramp | 66.08 | 98 | |
| GraNet | 52.10 | 98 | |
| SGD | Dense | 57.75 | 100 |
| LinBreg + SGap + Ramp | 46.42 | 100 | |
| GraNet | 36.55 | 98 | |
Two observations from the table stand out: 1) our adaptive method outperforms GraNet by a large margin with both optimizers, and 2) all SGD runs are considerably worse than their AdamW counterparts. The latter is a known issue [5, 6], which our evaluation likely amplified by not tuning the hyperparameters for DeiT-S. The cause of GraNet’s large degradation is less clear. A possible contributing factor is the issue of cold-start for newly regrown parameters when using Adam-style optimizers in DST [7]. However, the authors in [7] linked this issue to loss spikes after topology updates, which we did not observe in our runs. Hence, we cannot explain GraNet's performance by this phenomenon.
In short, our preliminary results indicate that our adaptive schemes can train sparse transformers and compare favorably with the DST baseline (GraNet). Nevertheless, we opt not to include this analysis in the updated manuscript, since concrete conclusions require further investigation.
[1] Kusupati et al. “Soft threshold weight reparameterization for learnable sparsity”. ICML, 2020. https://proceedings.mlr.press/v119/kusupati20a/kusupati20a.pdf
[2] Gallego-Posada et al., “Controlled Sparsity via Constrained optimization or: how I learned to stop tuning penalties and love constraints”. NeurIPS, 2022, https://proceedings.neurips.cc/paper_files/paper/2022/file/089b592cccfafdca8e017
[3] Jayakumar et al. “Top-KAST: Top-K always sparse training”. NeurIPS, 2020, https://proceedings.neurips.cc/paper_files/paper/2020/file/ee76626ee11ada502d5d
[4] Touvron et al. “Training data-efficient image transformers & distillation through attention”. ICML, 2021. http://proceedings.mlr.press/v139/touvron21a/touvron21a.pdf
[5] Zhang et al. "Why are adaptive methods good for attention models?". NeurIPS, 2020. https://proceedings.neurips.cc/paper/2020/file/b05b57f6add810d3b7490866d74c0053-Paper.pdf
[6] Zhang et al. "Why transformers need Adam: a Hessian perspective". NeurIPS, 2024. https://proceedings.neurips.cc/paper_files/paper/2024/file/ee0e45ff4de76cbfdf07015a7839f339-Paper-Conference.pdf
[7] Xiao et al. "Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling". ICML, 2026. https://proceedings.mlr.press/v306/xiao26t.html
Thank you for the response and the additional discussions and experiments on ImageNet. The rebuttal has addressed my concerns, so I would like to increase my rating.
Thank you for raising your score and for the constructive feedback.
W1: “The novelty is limited … This is very close to existing top-k and IHT-style methods on accumulated gradients“
We agree that performing top-k selection on an accumulated gradient method is very similar to our TopK adaptation. Although this framing identifies the similarity between the methods, it does not recognize that our proposal is a bridge between sparse MD and these lines of work. We even formalized exactly this similarity with DST in Proposition 2 to show that they are related. More importantly, this framing reduces our contributions to the TopK adaptation only. Our listed contributions include the following: 1) proposing two λ adaptation schemes to control sparsity for MD, 2) an empirical study showing that adapting λ improves performance compared to tuning it for the same sparsity budget, 3) providing the convergence analysis for the iterates with adaptive λ, and 4) interpreting TopK adaptation as a DST method.
The essence of our work is hence to argue for adapting λ instead of tuning it in MD; TopK is only one possible way to perform the adaptation. In fact, we have shown that SGap adaptation leads to similar improvements. The performance gain following λ adaptation is highly non-trivial since the adaptations are designed to control sparsity, not to improve accuracy. We attempted to explain the reason based on the training dynamics as discussed in the “Topology exploration” paragraph and demonstrated in Figure 2.
We acknowledge that "contribution (III)" in the paper did not clearly connect these points. Hence, we updated its statement to be more explicit. We have also highlighted the last point in the results.
W2 & Q2: “The paper does not compare to methods that hit a target sparsity in a single run …”
The main relevant baseline is the (original) fixed-λ LinBreg as this comparison isolates the effect of our modification. Our paper already includes four additional sparse baselines, three of which are DST methods, since we interpret our method through the DST framework. Regarding the specific requested baselines:
W3 & Q3: “Discuss the dense training cost”
The computational costs are briefly discussed in Appendix D: “The training cost per iteration depends on whether a method computes the gradient for the active weights only or for all weights. ... LinBreg and our methods compute it for all weights at every iteration”. Furthermore, Table 2 reports the training FLOPs, which reflect the additional cost of the dense backward pass. So, using a dense dual is stated as a cost incurred by our method, and the higher training FLOPs reflect it.