acceptodds
Withdrawn from ICLR 2027

LoRScore: Low-Rank Refinement of Activation-Aware Scores for Learnable Large Language Model Pruning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Learnable LLM pruning adapts sparsity patterns to the language-modeling objective, helping preserve model quality under compression. However, existing approaches often attach trainable variables to individual weights or candidate mask patterns, introducing substantial optimization overhead. We propose LoRScore, a parameter-efficient framework that combines Wanda's activation-aware importance scores with a learnable low-rank residual while keeping pretrained weights frozen. The key idea is to constrain only the learned score correction, preserving fine-grained prior information while enabling coordinated, loss-driven adjustments to pruning decisions. By separating score learning from mask projection, LoRScore supports both unstructured and semi-structured sparsity within a shared parameterization and introduces no additional parameters at inference. Experiments across multiple LLM families and sparsity settings demonstrate competitive language-modeling and zero-shot performance with substantially fewer trainable parameters. Theoretical analysis connects the surrogate score gradients to first-order sensitivity and characterizes the induced low-rank updates, providing an interpretation of how the compact parameterization refines pruning decisions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Learnable Pruning, Low-Rank Score Refinement, Activation-Aware Importance

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_p2MnOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LoRScore, a learnable pruning method for LLMs. The core idea is to use Wanda’s score as a fixed prior and learn a low-rank correction to it, S=Sbase+AB/rS = S_{\text{base}} + AB/\sqrt{r}, so that pruning decisions can be optimized on the language-modeling loss with very few trainable parameters, while the pretrained weights stay frozen. The factors are trained with a straight-through estimator, alternating updates of AA and BB, and a progressive sparsity schedule, and the same parameterization supports both unstructured and N:M sparsity. The paper relates the STE score gradient to first-order Taylor saliency (G⊙WG \odot W), describes the low-rank updates as sketched saliency, and shows that a rank-1 residual can reproduce magnitude pruning, Wanda, and intermediate criteria.

Critical strengths
  1. The design is simple and easy to follow.
  2. Broad empirical coverage.
Critical weaknesses
  1. The gains over the strongest learnable baselines are small and have no measure of variance, since against MaskPro, average accuracy improves by +1.07 (LLaMA-2-7B), +0.12 (Vicuna) and +0.05 (DeepSeek) points. There's no results report standard deviations or multiple seeds.
  2. There are several numbers are inconsistent. eg. PPL in Table 3 vs. Table 8. numbers in Table 3 vs. Table 9. In Table 2, LoRScore's OpenBookQA accuracy on Gemma is 18.02%. OBQA has 500 test questions, so accuracy must be a multiple of 0.2%. Please explain this further. And the key ablation is missing: does the low-rank constraint matter?
Questions for authors
  1. Which learning rate produced each row of Table 3?
  2. Can you report mean ± std over at least 3 seeds for LoRScore and MaskPro on the 2:4 results in Table 2?
  3. How does a full-rank (per-weight) score residual perform when trained with the same prior, STE and schedule? And how do r=1r=1 and a column-only correction 1v⊤\mathbf{1}v^\top perform?
  4. What is the total training time to the final mask for LoRScore, MaskPro and MaskLLM? Did you check MaskPro's memory and time against its official implementation?
  5. Please fix the OBQA value for Gemma, the Table 6 vs. Appendix C.2 perplexity mismatch, and the learning rate for LLaMA-1-13B, and say whether these errors affect any other reported numbers.
Reviewer_6PvVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates LoRScore for LLM unstructured pruning and semi-structured N:M sparsity, which introduces a low-rank score residual framework that refines Wanda's fixed activation-aware importance scores through learnable thin-matrix factors, achieving competitive pruning quality with drastically fewer trainable parameter.

Critical strengths
  1. The proposed approach is intuitive and conceptually clean. The paper is also well-written and easy to follow.
  2. Under N:M sparsity, LoRScore substantially reduces memory footprint while maintaining competitive performance against existing baselines.
  3. Several design choices are both clever and effective, notably the alternating updates of the A and B factors and the progressive sparsity schedule.
Critical weaknesses
  1. Both main figures contain question mark, likely from MathType parsing errors. Converting equations to SVG before figure export is recommended.

  2. The motivation for expressing the Wanda criterion in log space is not clearly explained. How large can the magnitude gap become between the log-domain Wanda score and the correction? Without normalization or scaling analysis, it is unclear whether the low-rank residual can effectively compete with the prior on the same additive scale. Furthermore, regarding the PPL improvements reported in Table 2, it remains ambiguous whether the gains stem from flipping a small number of critical pruning decisions or from a large-scale reranking of weights.

  3. No empirical ablation on base-score exponent. Appendix B.4 shows that a rank-one residual can recover any exponent α\alpha in the importance criterion, but no experiment demonstrates how different α\alpha values affect final pruning quality.

  4. Missing wall-clock time comparison. Only per-step time and peak memory for 2:4 on one model are reported. No total wall-clock time or end-to-end cost comparison with other methods is provided.

  5. LoRScore consistently underperforms MaskLLM in accuracy. While the parameter efficiency is appealing, the accuracy cost may be unacceptable in practice. Experiments on larger models with more comprehensive evidence (wall-clock time, peak memory, accuracy–efficiency Pareto) are needed.

Questions for authors
  1. Why is the WikiText2 perplexity for Gemma-7B not reported in Table 2?
  2. For unstructured pruning, the method introduces overhead from score factor training. Have the authors compared with ALPS [1] and ELSA [2], which are recent efficient unstructured pruning methods? Intuitively, these approaches may achieve better accuracy and potentially faster end-to-end time under comparable sparsity settings.
  3. Could the authors report peak memory and total training wall-clock time for larger models (e.g., 13B, 30B)? Additionally, it would strengthen the paper to include experiments on more recent model families such as Qwen3-8B and Qwen3-14B. Aside from DeepSeek-7B, the models evaluated in the current manuscript are relatively dated, which limits the generalizability of the findings.

[1] ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Models [2] The Unseen Frontier: Pushing the Limits of LLM Sparsity with Surrogate-Free ADMM

Reviewer_pBJ5OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LoRScore, which learns low-rank corrections to fixed activation-aware pruning scores while keeping pretrained weights frozen. It uses hard mask selection with a straight-through estimator, alternating factor updates, and progressive sparsification to support unstructured and semi-structured pruning.

Critical strengths

The parameterization substantially reduces the number of trainable mask parameters. The paper includes downstream evaluations and ablations of several training choices.

Critical weaknesses
  • The baseline coverage is so narrow and outdated. The unstructured experiments are limited to Magnitude, SparseGPT, and Wanda, despite LoRScore using additional end-to-end training. For semi-structured pruning, HyperPrune is particularly relevant to the claimed reduction in mask-learning cost, yet it is only discussed in related work.

  • The main comparisons rely heavily on old checkpoints. Much of the evaluation centers on Llama-1 and Llama-2. The appendix includes Llama-3.1, but only compares the pruned model against its dense counterpart. I would expect the main comparative results to include newer models, especially for a paper motivated by practical LLM compression. Llama-3.1 is already a model that came out more than two years ago, not to mention 1s and 2s... At the same time, most references are from before 2024, which is kinda intolerable in an ICLR 2027 submission. Combining this and the baseline issues together, it's really hard for me to believe that this is actually a 2027 submission instead of 2025.

  • It is unclear when a user should choose this method. MaskLLM achieves better quality on most models in the main comparison, while Wanda avoids the additional training. The paper gives no complete runtime comparison against either. The efficiency table reports an OOM entry for MaskLLM and a per-step comparison with MaskPro. These measurements leave the actual cost of obtaining the reported quality unclear.

  • The methodological advance is modest. The main change is a low-rank score residual combined with established optimization techniques. The ablations examine the surrounding training recipe, but offer little direct evidence about the benefit of this particular parameterization over simpler score corrections.

Overall I recommend rejection. The parameter savings are useful, but the experimental comparisons leave too much uncertainty about the method's practical advantage and relevance to newer models.

Questions for authors
  • How does LoRScore compare with HyperPrune on a newer model checkpoint?

  • What does the quality-versus-total-training-cost comparison look like for Wanda, LoRScore, and MaskLLM, including their memory requirements?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.