acceptodds
Under review as a conference paper at ICLR 2027

Tapering Neural Networks with Temporally Aggregated First-Order Importance

Abstract

In gradual pruning, network architecture is compressed simultaneously while learning the network parameters. Importance scoring of the parameters to prune allows to compare them such as for their ranking to cull a desired parameter set at a given pruning instance. In this paper we propose Temporal Aggregation of Parameter Effects for Reduction (TAPER), which uses an effective first-order estimation of weight importance on lost network capacity, combined by aggregation of importance scores for statistical robustness of gradual pruning decisions to batch variations. TAPER is evaluated favorably against the state of the art in gradual pruning in standard datasets using established metrics. It performs well both for pruning simpler CNNs such as VGG as well as more complex architectures such as ConvNeXt. Since network accuracy metrics compound major effects from training choices with pruning criteria, we also introduce Pruning Impact as a differential metric for assessing pruning criteria better isolated from cumulative training effects.

Primary area:
optimization
Keywords:
Gradual pruning, network compression, unstructured sparsity, Taylor approximation, pruning impact

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_M8K2OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper applies a variation of saliency score averaged over time to dynamically prune convolutional neural nets, in an unstructured with fine-tuning setting. A specific metric, pruning impact, is further discussed and studied.

Critical strengths

None

Critical weaknesses

Weak Contributions: The listed main contributions of the paper are all hardly novel. The proposed score is simply a variant of the well-known saliency score, already studied under various formulations. Aggregating the score over multiple updates is not novel as well. While the authors correctly cite relevant works with similar accumulation strategies, they fail to establish why their specific formulation is superior beyond limited empirical observations. Finally, the stated last contribution--a metric--is simply the monitoring of the degradation at pruning observations. Moreover, there is an important flaw with the proposed metric. Pruning aims to sparsify a model while preserving its performance compared to its dense counterpart. As presented, a pruning intervention degrading learning could display advantageous pruning impact while never achieving decent performance throughout training.

Weak positioning: While the paper is positioned as an unstructured gradual pruning method with fine-tuning, it fails to position the work with respect to its pruning objective, which is necessary to establish the proper benchmark requirements. If the goal is to reduce training FLOPs, it becomes necessary to also report the computation budget. If the goal is simply to achieve the best final accuracy vs sparsity, then benchmarking must extend beyond gradual pruning methods to encompass more broadly other pruning approaches sharing the same objective.

Weak benchmarks: Reported benchmark is limited to three architectures and three datasets, omitting in particular the normally expected benchmarking on ImageNet. Transferability to transformers is also omitted. Chase [1] (for which its unstructured variant is a relevant baseline to the current work) is a good example of standard empirical expectation in pruning literature. Moreover:

  • Dense performance is not reported for most experiments, making it impossible to compare degradation across methods, and to assess if the base training recipe is competitive.
  • There are missing key comparisons to claim SOTA comparable performance. Table 2 (b-c) of SPFK [2] exemplifies what is missing for gradual unstructured pruning.

Flawed methodology: Gradual pruning comparisons are expected with matched underlying training recipes. Reporting as a main result (Table 3) comparison between methods with different learning rate schedules raises questions about the validity of all reported main results. If the proposed method performs differently under varied training regimes, matched comparisons across all those regimes should be provided. It shouldn't be relegated to an ablation study over a single setting. Similarly, not matching the Δt\Delta t introduces confounding factors invalidating the presented comparisons. When setups differ, matching experiments across setups should be provided.

Reproducibility issues: With the code unreleased and the training hyperparameters of reproduced methods (GraNet and FGGP) missing, the paper in its current version suffers from a major reproducibility issue. In particular, this makes it impossible to verify if the comparisons were performed on an equivalent basis.


References:

[1] Lu Yin, Gen Li, Meng Fang, Li Shen, Tianjin Huang, Zhangyang Wang, Vlado Menkovski, Xiaolong Ma, Mykola Pechenizkiy, Shiwei Liu: Dynamic Sparsity Is Channel-Level Sparsity Learner. NeurIPS 2023

[2]: Zhanfeng Mo, Haosen Shi, Sinno Jialin Pan: Probabilistic Neural Pruning via Sparsity Evolutionary Fokker-Planck-Kolmogorov Equation. ICLR 2025

Questions for authors

Please note that all remarks below are suggestions for future work to the authors:

  • Please revise the excitation level raised by the observation that pruning can improve performance at some sparsity levels. This is amply discussed in pruning literature, in particular in the original Lottery Ticket Hypothesis paper, cited in this work.
  • The overall manuscript is currently giving an impression of non-polished, uncompleted work. Background section spans three pages, conclusion spans an entire page while only restating previous content and implementation details occupy most of the results section. Much of those could be presented in the appendix, leaving ample place to develop further the proposed contributions.
  • The background section, as currently organized, is alternating between presenting relevant works and the methods. Splitting the two sections, and getting rid of the unnecessary, would improve clarity.
1 comment

Dear reviewer, Thank you for your thorough review and for providing valuable insights on our work allowing us to strengthen our work. Below are our responses to your comments and related modifications in the paper.

Weak Contributions: “The proposed score is simply a variant of the well-known saliency score, already studied under various formulations. Aggregating the score over multiple updates is not novel as well. While the authors correctly cite relevant works with similar accumulation strategies, they fail to establish why their specific formulation is superior beyond limited empirical observations.”

Our research studies continuous pruning while training from a random initialized network; thus, most assumptions about gradients in converged networks do not hold. To our knowledge, all works proposing similar saliency scores are iterative pruning methods applied to pretrained networks. Thus, the novelty is a one-step first order Taylor approximation as saliency score for gradual pruning while training. We have further clarified this in the introduction section also comparatively to the closest literature.

Further, in the appendix, we have provided a seed-wise analysis comparing aggregation before and after multiplication to further establish the advantage of aggregating before multiplication.

“Finally, the stated last contribution--a metric--is simply the monitoring of the degradation at pruning observations.” To the best of our knowledge, such metric was not used for evaluating pruning approaches. In our paper we show this metric to be a good complement and potentially even more indicative than accuracy.

“Pruning aims to sparsify a model while preserving its performance compared to its dense counterpart. As presented, a pruning intervention degrading learning could display advantageous pruning impact while never achieving decent performance throughout training.” This is partly true, and this is why PrIm should be used in combination with accuracy when evaluating pruning algorithms. This is also discussed in our paper: “Moreover, it does not capture subsequent recovery, nor whether the resulting pruned network is more or less serviceable by following training. ”

Nonetheless, the direct impact of pruning is an interesting metric to monitor since the aim of most saliency scores is to find the weights with the least effect on network performance. When comparing PrIm for TAPER and GraNet, it is clear that TAPER is better at finding these weights.

Weak positioning: “If the goal is simply to achieve the best final accuracy vs sparsity, then benchmarking must extend beyond gradual pruning methods to encompass more broadly other pruning approaches sharing the same objective” Our goal is the best final accuracy vs sparsity in high-sparsity regimes. We have added an additional table (Table 2), with results from EAST, a method especially developed for high sparsity regimes. Our benchmark comparisons now cover gradual, sparse to sparse and one-shot pruning methods.

Weak benchmarks: We have, following your suggestion, added experiments on the full ImageNet-1K where TAPER is still superior to GraNet. Further, dense accuracies are now reported for all dataset/network pairs, thank you for pointing this out.

“There are missing key comparisons to claim SOTA comparable performance. Table 2 (b-c) of SPFK [2] exemplifies what is missing for gradual unstructured pruning.” The methods listed in your referenced table, is iterative (10 iterations) unstructured pruning from a pretrained network which is a completely different pruning setup than ours with up to 110 pruning iterations, training and pruning simultaneously from a randomly initialized network. Thus, we do not see these methods as key comparisons.

Flawed methodology: “Gradual pruning comparisons are expected with matched underlying training recipes. Reporting as a main result (Table 3) comparison between methods with different learning rate schedules raises questions about the validity of all reported main results.” We see experiments 1-2 as an evaluation of complete methods, including training hyper parameters, pruning schedules and important scores. However, we have included new experiments in the ablation section with TAPER using the same exact setup as GraNet to provide a proper comparison of the importance scores while we keep our proposed TAPER settings for experiments 1-2. Furthermore, for Tiny-ImageNet and ImageNet, both methods use the same LR schedule (cosine) and pruning step.

Reproducibility issues: The code has now been made available here: https://anonymous.4open.science/r/TAPER-29E8/ and the training hyperparameters for GraNet and FGGP have been added to the Appendix.

Reviewer_a8S9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TAPER gradually removes less important network weights during training. It identifies them by separately averaging weight and gradient magnitudes over several batches, then multiplying these averages. It also introduces Pruning Impact, which measures the immediate accuracy change after pruning. Experiments with VGG19, ResNet50, and ConvNeXt-Tiny on CIFAR10, CIFAR100, and Tiny-ImageNet, report substantial gains at very high sparsity. Ablations support temporal averaging, while the additional benefit of averaging the two factors separately is smaller.

Critical strengths
  1. Substantial reported gains at extreme sparsity. On ResNet50/CIFAR100 at 99.95% sparsity, TAPER improves over the reproduced GraNet [1] and FGGP baselines by 16.23 and 13.11 percentage points, respectively. On ConvNeXt-Tiny/Tiny-ImageNet at 99% sparsity, it improves over GraNet [1] by 22.47 points. TAPER’s reported standard deviations in these two settings are 0.75 and 0.60 percentage points, respectively. These results suggest that the proposed pipeline can preserve substantially more accuracy in demanding sparsity regimes.
  2. A simple importance rule with informative ablations. TAPER uses weight values and gradients already available during training, avoiding explicit Hessian estimation. On ResNet50/CIFAR100, temporal aggregation improves over instantaneous scoring by up to 7.77 percentage points. Stepwise TAPER also exceeds stepwise GraNet [1] by 12.67 points at 99.95% sparsity. This supports a benefit beyond the choice of cosine decay, although differences in pruning frequency still need to be controlled. The score’s simplicity is attractive; its actual runtime and memory overhead are not measured.
Critical weaknesses
  1. The main comparisons do not sufficiently isolate the importance score. On CIFAR10/100, TAPER uses cosine learning-rate decay and prunes every 391 batches, whereas GraNet [1]/FGGP use stepwise decay and prune every 1,000 batches. TAPER therefore prunes approximately 2.56 times as frequently, changing the size and timing of pruning interventions. The stepwise-TAPER ablation partially addresses the learning-rate difference, but does not report matching the pruning intervals. On ConvNeXt, the pruning interval is already shared between TAPER and GraNet [1]; the remaining schedule difference is not controlled by a corresponding ablation. Partial aggregation controls would further clarify which components are necessary. PrIm comparisons also depend on the network state and intervention size, and the measurement point relative to GraNet’s [1] regeneration needs clarification. The authors acknowledge PrIm’s dependence on network state, but the reported curves do not establish a comparison from common checkpoints with equal removal budgets.

  2. The specifically new aggregation choice and its robustness are insufficiently validated. The closest ablation, (⟨∣g∣⟩⟨∣w∣⟩\langle |g|\rangle\langle |w|\rangle) versus (⟨∣g∣∣w∣⟩\langle |g||w|\rangle), favors TAPER by 0.16–0.67 percentage points across the six sparsities, but is evaluated only on ResNet50/CIFAR100. The consistently higher means are encouraging; paired seed-level differences and uncertainty estimates would help establish the reliability of these smaller gains. Sensitivity to the aggregation horizon is unreported, despite different epoch-based pruning intervals across datasets. The EMA ablation tests only coefficients 0.5 and 0.9. If updated batchwise using the conventional EMA recurrence, these emphasize much shorter histories than a full-epoch mean, so the comparison may reflect history length as well as weighting shape. Finally, PLATON [2], which smooths first-order sensitivity estimates and incorporates temporal uncertainty, is neither discussed nor compared. Its original setting is pretrained Transformer fine-tuning, so an empirical comparison would require adaptation to a common training pipeline. Its conceptual relevance nevertheless warrants explicit positioning.

Questions for authors
  1. Controlled validation: Do the gains persist when the learning-rate schedule and pruning interval are both matched on ResNet50/CIFAR100? For ConvNeXt-Tiny/Tiny-ImageNet, please compare under a shared learning-rate schedule while retaining the already matched four-epoch pruning interval. A magnitude-only baseline and partial-aggregation controls, such as (⟨∣g∣⟩∣w∣\langle |g|\rangle|w|) and (∣g∣⟨∣w∣⟩|g|\langle |w|\rangle), within the same pipeline would help isolate the contribution of aggregating both factors.
  2. Aggregation clarification and robustness: How are the temporal averages initialized, updated, and reset? How were the one-epoch and four-epoch choices selected, and how sensitive are results when aggregation-window length is varied independently of pruning frequency? Please specify the EMA recurrence and update frequency, and compare averaging schemes with comparable temporal horizons under an explicitly stated matching criterion.
  3. Validation of the central distinction: Can you provide paired seed-level differences for separate aggregation versus product aggregation and extend this comparison to ConvNeXt or VGG19? Please also explain TAPER’s relationship to PLATON [2] and, if feasible, compare the scores in a common training pipeline.
  4. PrIm clarification: For GraNet [1], are the before/after measurements taken around the complete pruning-and-regeneration operation or only its pruning component? How is mean PrIm interpreted when pruning intervals and removal amounts differ on CIFAR? Were any common-checkpoint, equal-removal comparisons performed? Please also specify whether evaluation changes normalization statistics and whether test-set PrIm informed any design or hyperparameter choices.
  5. Reproducibility clarification: Please provide the ranking scope, eligible parameter types, permanent-mask/regrowth behavior, any safeguards against layer collapse, the exact ConvNeXt stem modification, and complete baseline configurations. Table 1 reports a literature GraNet [1] result of 76.10%, whereas Table 2 reports a reproduced result of 75.51% on ResNet50/CIFAR100 at 98% sparsity. This is not an internal contradiction, but identifying configuration differences would help assess reproduction fidelity.

References:

[1] Liu et al. Sparse Training via Boosting Pruning Plasticity with Neuroregeneration. NeurIPS 2021.

[2] Zhang et al. PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importance. ICML 2022.

1 comment

Dear reviewer, thank you for your thorough, constructive, and detailed review. Your input has helped us improve and strengthen our work.

Controlled validation: To address the concern regarding controlled validation of the proposed pruning metric, we have added several new experiments. We now include a TAPER version with the exact same training setup as GraNet, i.e. stepwise learning rate and 1000 batches between pruning steps (see the “GraNet var.” section in ablation Table 6). With this shared setup, TAPER still performs best, mainly through its impact scoring and aggregation (further ablated in that table). We see experiments 1-2 as a comparison of complete methods, including training hyperparameters, pruning schedules and importance scores. We therefore place the new experiments in the ablation section to compare importance scores directly, while keeping our proposed TAPER settings for experiments 1-2. We also include GraNet results with cosine learning rate for both tiny and full-size ImageNet (cosine being the originally proposed LR schedule for ConvNeXt). We have also performed your suggested partial aggregation controls (new rows in Table 6), which show that aggregating gradients is slightly more important, while separately aggregating both is best.

Aggregation clarification and robustness: TAPER’s averages are initialized as zero, updated after each minibatch and reset after each pruning step. EMAs are initialized with the first weight/gradient magnitude, updated after each batch as E(w_t) = alpha*E(w_(t-1)) + (1-alpha)w_t, and never reset. This is clarified in the paper and visible in our released code. We have expanded our ablations with two new EMA experiments, alpha=0.99 and alpha=0.995. The latter matches the mean age of aggregated samples to TAPER’s mean over one epoch: for a mean over N iterations this is (N-1)/2, for an EMA alpha/(1-alpha), giving alpha = (N-1)/(N+1) = 390/392 ≈ 0.995. A larger alpha improves performance, indicating the importance of a sufficiently long aggregation horizon, while TAPER’s plain mean performs equal or superior to the EMA. The pruning interval of one epoch for CIFAR100 gives equal input from all batches to the temporal score. For tiny-ImageNet it was increased to 4 epochs to let the network recover more between pruning steps, enabling informative pruning decisions on this harder task. We have not investigated shorter aggregation horizons for TAPER, but hypothesise it would follow the same pattern as the EMA.

“Can you provide paired seed-level differences for separate aggregation versus product aggregation?” This comparison is now in the Appendix: separate aggregation is superior in 15 of 18 seed/sparsity experiments, while product aggregation is clearly better for only one seed at one sparsity. A paired Wilcoxon t-test can then also substantiate the significant improvement from separate aggregation.

“Please also explain TAPER’s relationship to PLATON.” Like Sanh et al. (2020), PLATON aggregates the product (as opposed to our separate aggregation), and like Molchanov et al. (2019) it uses EMA (rather than our parameter-free, epoch-wise aggregation). Like both, PLATON targets fine-tuning of already converged networks, as opposed to our gradual pruning setting. We have added PLATON to our literature background with these points.

PrIm clarification: We now clarify that after-measurements are taken after the entire pruning step (including any parameter filtering/regrowth as in GraNet). Following your and other reviewers’ comments, we have also equalized the pruning interval for pruning impact evaluations with ResNet50 on CIFAR100 for a fair, direct comparison of reported PrIm values (TAPER with cosine LR and pruning step = 1000 batches). Test-set PrIm values were not used for tuning any hyperparameters or design choices.

Reproducibility clarification: Ranking is done globally, and we prune network weights but not bias terms. This, along with any ConvNeXt modifications, is now stated in the manuscript. No safeguards against layer collapse are used, and our GraNet implementation is, to our knowledge, the same algorithm as in the original paper. The standard deviation of our results (75.51 ± 0.74) includes the original GraNet result (76.10). Remaining differences are likely due to random initialization. Another possible explanation is that the GraNet paper reports the best checkpoint on the test set, while we report performance after the last epoch to avoid data leakage. Baseline configurations have been added to the appendix, and the code is available here: https://anonymous.4open.science/r/TAPER-29E8/

Reviewer_dj4DOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a pruning method inspired by Optimal Brain Damage (OBD).

In gradual pruning, where pruning is performed multiple times during training from initialization, gradients cannot be assumed to be zero. Therefore, the authors define weight importance using a first-order Taylor approximation.

In particular, to mitigate the instantaneous nature of importance estimates obtained from a single batch, the authors separately average the absolute values of weights and gradients across multiple batches (iterations) and then multiply these two averages.

To measure the impact of pruning while controlling for training-related factors, the authors evaluate the change in test accuracy immediately before and after pruning.

Critical strengths

The observation that gradients vary from batch to batch and may temporarily vanish is interesting. However, additional experimental evidence would further strengthen this observation.

Measuring the immediate impact of pruning is a precise and novel approach.

The pruning result at 99.95% sparsity for VGG19 on CIFAR-10 in Table 2 is impressive.

The analysis presented in Figure 2 is insightful.

Critical weaknesses

The experiments are limited to CNN architectures. I recommend exploring the application of the proposed algorithm to LLM quantization rather than restricting the evaluation to CNN pruning. Parameter-efficient fine-tuning could also be considered if necessary.

Regarding the measurement of pruning impact, I believe that directly controlling training-related factors would allow for a more meaningful comparison. Specifically, all algorithms could be evaluated under the same set of training configurations, and the experiments could then be repeated while systematically varying these configurations. This would allow the authors to control experimental variables while also evaluating the performance of the algorithms across different training conditions.

The result for VGG19 on CIFAR-100 at 99.95% sparsity in Table 2 is not particularly informative. Since the accuracy is below 30%, the result does not appear to be practically meaningful.

It would be helpful to report the unpruned baseline accuracies for the experiments in Table 2.

The subfigures in Figure 2 are missing labels such as (a), (b), and (c).

Questions for authors

See weaknesses section.

1 comment

Dear reviewer, thank you for your time and your acknowledgment of the paper’s strengths, ‘insightful’ findings, PrIm’s novelty, and TAPER’s ‘impressive’ pruning results at high sparsity. Further given the positive rankings on presentation, soundness, and informativeness, we were actually unsure if the overall rating was due to an error. Nonetheless, we address your concerns below and have made several corresponding improvements in the revised paper.

“I recommend exploring the application of the proposed algorithm to LLM quantization rather than restricting the evaluation to CNN pruning.”

Although we would love to test this in future work, and we speculate that TAPER will extend to that problem setting as well, LLMs were beyond the focus of this work. Indeed, there have been many prominent papers, including those in our bibliography, that use only computer vision tasks to introduce their contributions, some been studied and tested later in other domains as well, but not necessarily in the initial work. So, despite being an added benefit, it is not clear to us whether experimenting across such widely-differing tasks and domains is a necessity to show a method’s promise or a requirement for an ICLR contribution. Further our work focuses on gradual pruning during simultaneous training which requires huge computational resources for LLMs. We still would like to investigate LLM pruning, perhaps in limited settings, but this would require time and resources going beyond the focus of this work.

“Regarding the measurement of pruning impact, I believe that directly controlling training-related factors would allow for a more meaningful comparison.”

PrIm comparison of cosine vs stepwise learning rate in the ablations of our original submission already provided a controlled experiment setting for training related factors for comparison. In addition, in line with the other two viewers’ feedback, we now report PrIm for ResNet50 and CIFAR100 by comparing runs with the same pruning interval for fair comparison. These said, please note that pruning impact PrIm is proposed as a metric for comparing pruning algorithms such as importance scoring, in particular to disambiguate some training effects, hence is not meant to be a tool to investigate the impact of training hyperparameters.

“The result for VGG19 on CIFAR-100 at 99.95% sparsity in Table 2 is not particularly informative. Since the accuracy is below 30%, the result does not appear to be practically meaningful.”

Whether an accuracy is practical can only be assessed in the context of intended downstream task. Note that 30% is still far from random probability for a 100 item classification task. We included this setting, as our method aims particularly at high spasity. Furthermore, if we had omitted in Table 3 this particular combination only (e.g. with dashes, etc) it may have looked like we are avoiding reporting certain settings, and the same reviewers might have read why we fail to report these. For the sake of completeness and also to provide insights on different model backbones operation with pruning at extreme sparsities, we wish to keep these results in our paper.

Regarding your requested Figure 2 labels, they already existed in the top-left corners of each subfigure in the original submission. Unpruned accuracy for the baseline models are now reported in the table captions as requested.

Comments on the paper

1 comment

Dear reviewers,

Thank you for your time and your constructive feedback on our paper. An updated manuscript has been uploaded with the new and revised content marked in blue, and our code is now available at: https://anonymous.4open.science/r/TAPER-29E8/

Main changes in the revision include:

  • Matched-setup ablations: TAPER with GraNet’s training setup (stepwise learning rate, 1000 batches between pruning steps) and TAPER with cosine learning rate and 1000 batches between pruning steps, as well as GraNet with cosine learning rate on Tiny-ImageNet and ImageNet.
  • Experiments with ConvNeXt on full ImageNet-1K.
  • A new comparison against EAST (new Table 2).
  • Additional aggregation ablations: partial-aggregation controls, longer-horizon EMAs with α = 0.99 and α = 0.995, and a seed-level paired comparison of separate versus product aggregation (Appendix).
  • Pruning impact (PrIm) compared at matched pruning intervals
  • Dense (unpruned) accuracies for all dataset/network pairs, baseline hyperparameters (Appendix), and a discussion of PLATON.

Correction: The results for GraNet (stw) on Tiny-ImageNet have been updated after we discovered a configuration mismatch in the previous experiments: GraNet had pruned the first convolutional layer (conv1) and the classifier layer, whereas TAPER had not. Matching the GraNet experimental setting with the remaining resulted in a very minor change in GraNet performance, but did not change its relative standing, any of the earlier observations, or conclusions. Detailed responses to the comments and our corresponding revisions are provided as a response to each individual review below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.