- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes an unstructured pruning method, named Dynamic Activity-Dependent Pruning (DADP). It uses the gradient of pre-activation times pre-synaptic activation magnitude as a metric for pruning.
Critical strengths
None
Critical weaknesses
- The proposed method is overly simplistic. It just uses the gradient of pre-activation times pre-synaptic activation magnitude as a metric for unstructured pruning, which is equivalent to the magnitude of the weight gradient. Such a method lacks novelty.
- The compared baseline methods are outdated. SNIP and RigL still serve as classic baselines, but for a paper aimed at ICLR 2027, simply comparing it to them is indeed insufficient to demonstrate its current competitiveness. Please compare DADP with more up-to-date baselines, such as CHITA [1], STR [2], and NeurRev [3].
- The benchmarks used are too simple (MNIST, CIFAR-10, CoNLL-2003, and SST-2). Please validate DADP on more challenging benchmarks, such as ImageNet-1k for image classification.
- The proposed method can not directly control the final sparsity. Instead, hyperparameter tau indirectly controls the sparsity. However, the setting of tau depends on the model and task and relies on empirical tuning.
- The accumulation of the importance metric increases the training memory overhead. This paper has not reported additional training overhead of DADP.
[1] Benbaki, Riade, et al. Fast as CHITA: Neural network pruning with combinatorial optimization. International Conference on Machine Learning. 2023.
[2] Chen, Yanqi, et al. A unified framework for soft threshold pruning. International Conference on Learning Representations. 2023.
[3] Li, Gen, et al. NeurRev: Train better sparse neural network practically via neuron revitalization. International Conference on Learning Representations. 2024.
Questions for authors
- Please compare DADP with more up-to-date baselines.
- Please validate DADP on more challenging benchmarks.
- Please report the additional training overhead of DADP.