- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces LoRScore, a learnable pruning method for LLMs. The core idea is to use Wanda’s score as a fixed prior and learn a low-rank correction to it, , so that pruning decisions can be optimized on the language-modeling loss with very few trainable parameters, while the pretrained weights stay frozen. The factors are trained with a straight-through estimator, alternating updates of and , and a progressive sparsity schedule, and the same parameterization supports both unstructured and N:M sparsity. The paper relates the STE score gradient to first-order Taylor saliency (), describes the low-rank updates as sketched saliency, and shows that a rank-1 residual can reproduce magnitude pruning, Wanda, and intermediate criteria.
Critical strengths
- The design is simple and easy to follow.
- Broad empirical coverage.
Critical weaknesses
- The gains over the strongest learnable baselines are small and have no measure of variance, since against MaskPro, average accuracy improves by +1.07 (LLaMA-2-7B), +0.12 (Vicuna) and +0.05 (DeepSeek) points. There's no results report standard deviations or multiple seeds.
- There are several numbers are inconsistent. eg. PPL in Table 3 vs. Table 8. numbers in Table 3 vs. Table 9. In Table 2, LoRScore's OpenBookQA accuracy on Gemma is 18.02%. OBQA has 500 test questions, so accuracy must be a multiple of 0.2%. Please explain this further. And the key ablation is missing: does the low-rank constraint matter?
Questions for authors
- Which learning rate produced each row of Table 3?
- Can you report mean ± std over at least 3 seeds for LoRScore and MaskPro on the 2:4 results in Table 2?
- How does a full-rank (per-weight) score residual perform when trained with the same prior, STE and schedule? And how do and a column-only correction perform?
- What is the total training time to the final mask for LoRScore, MaskPro and MaskLLM? Did you check MaskPro's memory and time against its official implementation?
- Please fix the OBQA value for Gemma, the Table 6 vs. Appendix C.2 perplexity mismatch, and the learning rate for LLaMA-1-13B, and say whether these errors affect any other reported numbers.