acceptodds
Under review as a conference paper at ICLR 2027

DADP: Dynamic Activity-Dependent Pruning, A Reverse Hebbian-Inspired Structural Plasticity Method

Abstract

Modern neural networks are heavily over-parameterized. This redundancy incurs substantial compute and memory overhead during training and inference. Existing pruning methods typically rely on post-hoc magnitude thresholds or static initialization heuristics. Consequently, they often require manual per-layer sparsity targets or expensive retraining cycles. We propose Dynamic Activity-Dependent Pruning (DADP), a biologically inspired structural plasticity mechanism. During training, DADP measures connection importance via the accumulated product of pre-synaptic activations and post-synaptic error gradients. Using a single global threshold instead of fixed layer budgets, DADP dynamically allocates sparsity across network depth while naturally inducing neuron- and channel-level pruning. Across MLP, VGG-16, ResNet-18, BiLSTM-CRF, and MiniBERT architectures, DADP matches or outperforms baselines like Magnitude, SNIP and RigL, retaining 73.67% accuracy (dense baseline: 76.06%) at 99% sparsity on ResNet-18. Finally, matrix-based Shannon entropy and effective rank measurements confirm that DADP preserves latent feature diversity at extreme sparsities without representation collapse.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Neural Network Pruning, Sparsity, Structural Plasticity, Biologically Inspired AI, Efficient Deep Learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_VnW6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes an unstructured pruning method, named Dynamic Activity-Dependent Pruning (DADP). It uses the gradient of pre-activation times pre-synaptic activation magnitude as a metric for pruning.

Critical strengths

None

Critical weaknesses
  • The proposed method is overly simplistic. It just uses the gradient of pre-activation times pre-synaptic activation magnitude as a metric for unstructured pruning, which is equivalent to the magnitude of the weight gradient. Such a method lacks novelty.
  • The compared baseline methods are outdated. SNIP and RigL still serve as classic baselines, but for a paper aimed at ICLR 2027, simply comparing it to them is indeed insufficient to demonstrate its current competitiveness. Please compare DADP with more up-to-date baselines, such as CHITA [1], STR [2], and NeurRev [3].
  • The benchmarks used are too simple (MNIST, CIFAR-10, CoNLL-2003, and SST-2). Please validate DADP on more challenging benchmarks, such as ImageNet-1k for image classification.
  • The proposed method can not directly control the final sparsity. Instead, hyperparameter tau indirectly controls the sparsity. However, the setting of tau depends on the model and task and relies on empirical tuning.
  • The accumulation of the importance metric increases the training memory overhead. This paper has not reported additional training overhead of DADP.

[1] Benbaki, Riade, et al. Fast as CHITA: Neural network pruning with combinatorial optimization. International Conference on Machine Learning. 2023.

[2] Chen, Yanqi, et al. A unified framework for soft threshold pruning. International Conference on Learning Representations. 2023.

[3] Li, Gen, et al. NeurRev: Train better sparse neural network practically via neuron revitalization. International Conference on Learning Representations. 2024.

Questions for authors
  1. Please compare DADP with more up-to-date baselines.
  2. Please validate DADP on more challenging benchmarks.
  3. Please report the additional training overhead of DADP.
Reviewer_dsnwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes DADP, which defines connection importance using accumulated statistics of the product of presynaptic activations and postsynaptic error gradients. The authors refer to this as a reverse Hebbian mechanism. The method uses a single global threshold to permanently prune connections with low importance during training. The authors evaluate DADP on MLP, VGG-16, ResNet-18, BiLSTM-CRF, and MiniBERT. According to Table 1, DADP achieves an accuracy of 72.28 ± 0.28% at 99.35 ± 0.03% sparsity on ResNet-18/CIFAR-10, compared with 76.06% for the dense baseline.

Critical strengths

1.Using both forward activation states and backward error signals for dynamic importance scoring offers an interesting perspective.

2.The authors attempt to characterize the latent feature manifold under high sparsity using matrix-based Shannon entropy and effective rank, rather than limiting their evaluation to reporting accuracy.

3.DADP achieves competitive results in some experimental settings, particularly at extremely high sparsity on ResNet-18/CIFAR-10.

Critical weaknesses

1.Although the authors describe the core importance metric as a “biologically inspired reverse Hebbian plasticity mechanism,” the chain rule gives ai ∂L/∂yj=∂L/∂wija_i\,\partial L/\partial y_j=\partial L/\partial w_{ij} for a fully connected layer evaluated on a single example. Therefore, this metric is closely related to existing gradient-based sensitivity criteria for pruning, and the “reverse Hebbian” terminology alone is insufficient to establish methodological novelty. The authors should more clearly explain how their method differs from existing approaches and identify the individual contributions of temporal accumulation, global thresholding, and permanent pruning.

2.DADP treats small gradient magnitudes as evidence of low connection importance. However, a small gradient does not guarantee that deleting the corresponding weight will cause only a small change in loss. When gradient magnitudes remain below the threshold over the accumulation window, the method may permanently prune connections that have converged but remain useful for prediction. Moreover, the permanent pruning mechanism in Equation (2) cannot restore deleted connections. The authors therefore need more rigorous theoretical conditions or targeted experiments to support the proposed self-correcting mechanism.

3.In Table 1, DADP is evaluated at approximately 95% sparsity on BiLSTM-CRF, whereas the other pruning baselines are evaluated at 80%, making the comparison mismatched. Table 7 already provides baseline results at 95% sparsity: Magnitude, SNIP, and RigL achieve F1 scores of 93.88%, 92.60%, and 93.64%, respectively, all substantially higher than the 84.28 ± 1.34% achieved by the DADP configuration selected for Table 1. Even the other DADP configuration at 95.09% sparsity achieves an F1 score of 87.99 ± 1.73%, which remains below these baselines.

4.Some references contain inconsistent information. The authors should systematically check the authors, years, venue names, and publication identifiers in their references.

Questions for authors

1.If the gradient magnitudes of certain weights remain below the threshold τ\tau over the accumulation window, even though those weights have converged and remain important for prediction, how does DADP avoid permanently pruning them? Under what conditions can the argument in Appendix A.2 guarantee that this will not occur?

2.I suggest that the authors add a bar chart or table comparing the training costs of DADP, RigL, SNIP, and Dense under the same hardware conditions.

Reviewer_Qi9cOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes to use an accumulated per-weight gradient score in disguise as a pruning criterion.

Critical strengths

None

Critical weaknesses

Major flaw: Despite all the discussion around the proposed criterion, it amounts to using accumulated per-weight gradients as a pruning score. If the authors simply plug back (8) into (10) (Appendix A.1), the criterion directly reduces to Iijl=∣δLδyijl∣I^l_{ij} = | \frac{\delta \mathcal{L}}{\delta y^l_{ij}}|. Under this lens, there is barely any novelty [1, 2]. And it highlights a major flaw : with the proposed algorithm, any model could end up entirely pruned by extending late training where gradients typically become increasingly small. This drawback with gradient-based saliency measures is known [1, section 3.2] and is among the reasons why a sparsity target (after which pruning stops) is generally set, or why gradient magnitudes are accumulated from initialization [2].

Clarity issues: The paper is disorganized and plagued by contradictory statements, including but not limited to:

  • The paper is not positioned with respect to pruning literature. Even after a full read, it is unclear if the method is framed as a dense-to-sparse partially-structured dynamic pruning method. This matters, because this body of pruning literature typically targets training acceleration, with FLOPs and/or speedup measurements required.
  • Important quantity such yjly^l_j are never fully defined and only described late into the methodology section. In particular, the proposed definition for yjly^l_j only holds for MLP, while presented results span CNN, LSTM and transformers.
  • Statements such as DADP consistently achieving superior classification accuracy (lines 2213-214) are contradicted right away by the next stated result (SST-2) and inspection of the tables Despite being put in bold, DADP often showcases worse performance than reported baselines.
  • Other statements obfuscate directly the bigger picture. Caption of Figure 3 is a clear example, where saying that DADP maintains normalized entropy above 0.85 is contradicted by the curves shown in Figure 3a.

Benchmarking is inappropriate:

  • None of the baseline are suited for the proposed gradual partially-structured pruning method, while there is plenty in literature [3,4,5,6].
  • The authors report training for 20 epochs (Appendix C2) which is unusual and does not fit the typical setting the proposed baselines were tuned and reported on. Resultingly, the reported performance for dense network is not on-par with expectations: a ResNet-18 trained on CIFAR-10 with a well-tuned recipe easily reaches higher than 92% top-1 accuracy.
  • The selected benchmarks all represent easy tasks for which selected models have excess capacity and are easy to prune on. Extended experiments on CIFAR-100 and ImageNet are the bare minimum to compare against, with meaningful baselines.
Questions for authors

On top of the critical weaknesses listed above, the authors also need to address the following important issues:

  • If the methods perform partially-structured pruning, the unstructured vs structured sparsity needs to be reported. See [5].
  • While presented as an advantage, not being able to set in advance the desired sparsity has quite a drawback. A practitioner now instead needs to perform a sweep over τ\tau to reach the desired sparsity, potentially counteracting any acceleration benefits from using the method.
  • The proposed methodology might not be suited for other activation functions than RELU, which sets multiple gradients to zero. An ablation over different choice of activation functions should be reported.
  • As currently described in section 3.2 and Algorithm 1, the proposed method does not prune the bias parameters, raising a question on how 99% sparsity can be achieved at all on a ResNet-18.
  • The sparsity levels in the different tables are often not matched, making comparison meaningless. Standard deviation is not reported on baselines.
  • How can there be active neurons when there is no active weights (Figure 2a, epochs 13/18 and 2b, epochs 3/5/12)?
  • How can an entirely pruned network (100% sparsity) performs than its dense counterpart in Table 7?
  • Not only Theorem 2 is never discussed in the main paper, it is plain wrong. First, if excessive pruning stops the signal, the network becomes frozen and logits become constant, which certainly do not imply the loss going to infinity. Second, even if the loss went to infinity, it does not imply that gradients blow to infinity as well (Eq. 11 of Appendix A.2). They rather remain at zero... How would the network remain frozen otherwise? (Indeterminate limits should be treated carefully, see L'Hôpital's Rule.)

References:

[1]: Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, Jan Kautz: Importance Estimation for Neural Network Pruning. CVPR 2019: 11264-11272

[2]: Maximilian Golub, Guy Lemieux, Mieszko Lis: Full Deep Neural Network Training On A Pruned Weight Budget. SysML 2019

[3]: John Rachwan, Daniel Zügner, Bertrand Charpentier, Simon Geisler, Morgane Ayle, Stephan Günnemann: Winning the Lottery Ahead of Time: Efficient Early Network Pruning. ICML 2022: 18293-18309

[4]: Simon Dufort-Labbé, Pierluca D'Oro, Evgenii Nikishin, Irina Rish, Pierre-Luc Bacon, Razvan Pascanu, Aristide Baratin: Maxwell's Demon at Work: Efficient Pruning by Leveraging Saturation of Neurons. Trans. Mach. Learn. Res. 2025 (2025)

[5]: Mike Lasby, Anna Golubeva, Utku Evci, Mihai Nica, Yani Ioannou: Dynamic Sparse Training with Structured Sparsity. ICLR 2024

[6]: Lu Yin, Gen Li, Meng Fang, Li Shen, Tianjin Huang, Zhangyang Wang, Vlado Menkovski, Xiaolong Ma, Mykola Pechenizkiy, Shiwei Liu: Dynamic Sparsity Is Channel-Level Sparsity Learner. NeurIPS 2023

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.