acceptodds
Under review as a conference paper at ICLR 2027

SPRBM: Adaptive Parameter-Wise Pruning via Task–Complexity Gradient Alignment

Abstract

Parameter importance in neural-network pruning is commonly estimated from static weight statistics, predefined sparsity rules, or isolated gradient snapshots. Such criteria do not directly control how strongly each parameter is regularized during task optimization. We introduce SPRBM, a sparsification framework that uses task–complexity gradient agreement for parameter-wise adaptive regularization during training and post-training pressure calibration. For every trainable parameter, SPRBM measures the signed agreement between the task gradient and the gradient of a differentiable complexity objective. This signal determines an adaptive regularization coefficient during training. After training, coefficients recomputed on multiple validation mini-batches are averaged and converted into parameter-specific pruning thresholds. The resulting framework does not impose a fixed global sparsity pattern. Instead, pruning pressure emerges from the interaction between task optimization, parameter complexity, and post-training validation-batch calibration. We instantiate SPRBM with \(L_1\) and \(L_2\) complexity functions and evaluate it on autoencoders, restricted Boltzmann machines, vision transformers, and LoRA adapter sparsification for language adaptation. Across these settings, SPRBM produces architecture-dependent sparsification regimes, including approximately 85% sparsity for the RBM, 30–35% sparsity for vision transformers, and 38–53% sparsification of LoRA parameters. Matched-sparsity ablations further show that disrupting the correspondence between the learned alignment signal and parameter locations degrades performance, indicating that the observed behavior cannot be explained by sparsity level alone.

Primary area:
optimization
Keywords:
neural network pruning, unstructured pruning, adaptive regularization, task-complexity gradient alignment, parameter-wise pruning, model sparsification, LoRA pruning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_uejKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SPRBM, a pruning method based on proposed task and complexity gradient agreement. It is evaluated on the trainable LoRA parameter reduction of DistilGPT2.

Critical strengths
  • The intuition of using task and complexity gradient agreement as a parameter-wise signal may be interesting.
  • The paper is honest about the limitations.
Critical weaknesses
  • The motivation, i.e., gradient alignment is used to design a pruning criterion, can not be substantiated. First, gradient alignment alone is not strongly correlated with measuring the redundancy of the pruned parameters. If a parameter is important to the task, and its complexity gradient direction happens to be consistent with the task gradient, how should this be handled? Moreover, no theoretical basis is given to explain why the proposed criterion is more suitable than importance or magnitude criteria for determining pruning. Therefore, the claim stated, "Can parameter-wise pruning pressure be derived from task–complexity gradient agreement", is not substantiated.
  • Theoretically, in the First-Order Interpretation section, only a first-order Taylor expansion is used for the analysis. How are the perturbations caused by higher-order terms measured? In addition, the analysis is parameter-wise, ignoring the interactions between parameters. Moreover, it lacks convergence, optimality, or approximation guarantee for the overall procedure. At last, the designs of hyper-parameters and mask are heuristic, with no theory. Overall, it is only a local analysis that can provide some intuitive guidance, but the analysis is not rigorous enough.
  • Please explain the motivation for the threshold design. Please explain why the sigmoid function is used to transform the parameter-specific coefficient. In addition, numerous hyper-parameters such as the global scale, alignment sensitivity, and coefficient are introduced, which increases the difficulty of tuning. Moreover, there is no experiment on how these hyper-parameters affect the results.
  • For the experiments, are the baselines selected the state-of-the-art methods among magnitude, regularization-based, random, and SNIP-style criteria? Why not compare with Movement Pruning or PLATON? If the method proposed is a new criterion derived from task and complexity gradient agreement, then it needs to be compared with pruning methods of different styles, and with relatively SOTA specific methods in every criteria.
  • The model scale is extremely small, so the generalizability of the conclusions is limited. Whether it can be transferred to models that truly require pruning is completely unknown. In addition, the experiment is conducted on LoRA, which is itself a parameter-efficient fine-tuning method; therefore, the marginal benefit brought by sparsification is extremely small. Furthermore, the actual benefit is only a reduction in adapter storage, with no inference acceleration. In terms of results, the performance improvement is also not obvious.
  • The ablation claimed in the paper cannot prove that this information comes from the gradient alignment mechanism itself. The information provided by the ablation experiments is very limited and not comprehensive.
Questions for authors
  • If an important parameter happens to have aligned gradients, it still gets pruned. How to deal with it?
  • Your analysis is local and first-order. Why does that justify a discrete pruning decision?
  • What motivates the sigmoid mapping, hyper-parameter designs and mask designs?
  • You have many hyperparameters. How sensitive are results to them?
  • Why no comparison with strong baselines?
  • Your models are tiny. Will this work on real-scale models?
  • LoRA is already efficient. What is the actual gain?
  • Please provide more ablations.
Reviewer_7aEvOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes SPRBM, a pruning method that compares, for each parameter, the direction of the task gradient with the direction of an L1 or L2 penalty gradient. When the two agree, the parameter receives stronger regularization during training. After training, this agreement is measured again on 20 validation batches, averaged, and used to slightly adjust a per-parameter magnitude threshold. Results are reported on autoencoders, RBMs, vision transformers and LoRA adapters for DistilGPT2.

Critical strengths

The idea of letting pruning pressure differ across parameters during training is reasonable.

Critical weaknesses
  • The writing needs further polishing. Experiments and method are mixed in a single section, and the overall section arrangement is unreasonable, for example, the “Overall Procedure” subsection does not actually describe a procedure.
  • The method lacks novelty. The core signal essentially asks whether gradient descent pushes a weight toward zero, which is very similar to Movement Pruning. In addition, the final pruning rule is close to plain magnitude pruning.
Questions for authors
  • How does SPRBM differ from Movement Pruning, and can you add a direct comparison?
  • What happens if a single fixed threshold, or plain magnitude pruning, is applied to the same SPRBM-trained weights?
Reviewer_zwJ4OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes parameter-wise adaptive regularization based on task–complexity gradient agreement, followed by validation-batch calibration to obtain pruning thresholds. The idea is reasonable, but the evaluation is very limited in scale, comparison baseline that appears outdated and comparative evidence to establish its benefit. The computational overhead of the method does not justify the scale limit. I do not see a restriction on the dataset choice as well as the convergence of VITs on MNIST and CIFAR10 scale is not demonstrated. Hardware benefit is not present.

Critical strengths
  1. The mechanism is straightforward and interpretable: adjust regularization according to whether complexity reduction agrees with task descent.

  2. Averaging the signal across calibration batches is an interesting alternative to relying on a single gradient snapshot. It is still close to GraSP though.

  3. The authors acknowledge the limitations of their first-order interpretation and distinguish adapter sparsification from full-model compression.

Critical weaknesses
  • Limited evaluation. The vision experiments use MNIST and CIFAR-10 with small models, while the language experiment covers only DistilGPT2-LoRA on WikiText-2. There is no clear comparative table of absolute post-pruning performance for the vision experiments. Sparsity patterns and descriptions of training dynamics are not sufficient to establish effectiveness across architectures.

  • Missing relevant baselines. GraSP is mentioned but never evaluated. Movement Pruning and PLATON are also discussed without comparisons, and a direct weight–gradient-product baseline is missing. These omissions are not adequately justified. Although the reported LoRA results outperform SNIP and WarmStart-SNIP, these comparisons alone do not establish the value of the proposed procedure under comparable information access and computational budgets.

  • Unclear computational benefit. Section 3.10 states the additional O(N) memory requirement but does not measure training time, peak memory, or calibration overhead. The method introduces parameter-specific operations throughout training and an additional gradient-based calibration stage. It remains unclear whether the improvements justify this cost, especially when some gains over magnitude pruning are small and no deployment acceleration is demonstrated.

  • Insufficient separation of the components. The experiments should establish whether adaptive training is necessary, or whether ordinary training followed by the same terminal calibration would give similar results. Likewise, adaptive training followed by magnitude pruning would help isolate the contribution of the final thresholds.

  • Incomplete positioning. Parameter-wise regularization is already considered in Bayesian sparsification through parameter-wise priors and marginal-likelihood optimization, including “Shaving Weights with Occam’s Razor: Bayesian Sparsification for Neural Networks Using the Marginal Likelihood,” which is not cited. These approaches are not equivalent, but the paper should distinguish the contribution of its gradient-conditioned rule from the established idea of adapting regularization per parameter.

  • Limited mathematical justification. The first-order argument motivates local shrinkage, not finite parameter removal or the particular threshold mapping used here. For the proposed L1 and L2 penalties, the normalized agreement is also essentially a weight–gradient sign test away from the near-zero regime. A direct baseline would help establish what the formulation adds.

Questions for authors
  • Why is the vision evaluation limited to MNIST and CIFAR-10? The method is presented as architecture-independent and is already applied to several model types, so it is unclear what prevents evaluation on larger, more challenging benchmarks. Are there specific computational or methodological constraints behind this choice? If so, please explain them and quantify the additional cost. Otherwise, what evidence supports the method’s effectiveness beyond these small-scale settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.