- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a structured pruning method called Putri that improves on prior work in three ways: updating unpruned FFN weights to compensate for pruning error, pruning FFN layers sequentially while accounting for updates to previous layers, and removing individual attention heads instead of full attention layers (extended to Grouped-Query Attention). Experiments show that Putri remains simple while achieving SOTA performance and can prune LLMs at extreme sparsity ratios.
Critical strengths
-
This paper is well-written and easy to follow.
-
The experiments are comprehensive and the improvements are significant.
Critical weaknesses
-
This paper is somewhat incremental. Regarding compensation for pruning error, there is a substantial body of literature [1,2,3]. In particular, [3] also uses the objective of minimizing L2-loss for error compensation. However, I do not see any discussion of the differences from these works in the main text of this manuscript.
-
The baselines are weak. This paper does not compare against state-of-the-art structured pruning methods, including Olica [3], SlimLLM [4], SlimGPT [5], and LoRAP [6].
-
Reproducibility. I do not see a section that provides implementation details, which substantially hurts the reproducibility of this paper.
References
[1] SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot. ICML 2023.
[2] Fluctuation-based Adaptive Structured Pruning for Large Language Models. AAAI 2024.
[3] Olica: Efficient Structured Pruning of Large Language Models without Retraining. ICML 2025.
[4] SlimLLM: Accurate Structured Pruning for Large Language Models. ICML 2025.
[5] SlimGPT: Layer-wise Structured Pruning for Large Language Models. NeurIPS 2024.
[6] LoRAP: Transformer Sub-Layers Deserve Differentiated Structured Compression for Large Language Models. ICML 2024.
Questions for authors
Please see the weaknesses.
2 comments
Many thanks for your efforts devoted to addressing my concerns. After reading your responses, I still have several concerns. First, if LS compensation is not your contribution, please indicate your positioning clearly in Section 3.1 of your paper. Second, why not compare these baselines under the same evaluation settings? In addition, public repository: LoRAP (https://github.com/lihuang258/LoRAP). Third, I cannot see the revised content of your latest manuscript. Therefore, I keep my rating as 2.
Reply to Reviewer 9VDj (1/1)
We thank the reviewer for the careful review and the constructive comments. We address each point below.
W1: Novelty and relation to compensation methods. We do not claim LS compensation as our invention. Our contributions are:
W2: Baselines. Our baselines were chosen as the strongest post-training structured methods that support GQA models out of the box: 2SSP (TMLR 2025), EvoPress (ICML 2025), BlockPruner, ShortGPT and UIDL. Olica, SlimLLM, SlimGPT and LoRAP report perplexity only on MHA models (LLaMA-1/2 and Vicuna, plus Baichuan-7B for SlimGPT). The only GQA result among them is SlimGPT on Mistral-7B, evaluated on LongBench after LoRA fine-tuning, and none of the four papers describes how it handles GQA. Olica and SlimLLM have public implementations, both for LLaMA models with MHA. We found no public repository for SlimGPT and LoRAP. This is why we did not include them in our experiments on Llama-3 and Qwen3.
Their reported results on Llama-2-7B without fine-tuning are shown below, next to 2SSP, our strongest baseline, on the same model. WikiText-2 perplexity:
All four methods follow the LLM-Pruner protocol: 128-token evaluation sequences and a sparsity defined over all parameters, including the embeddings. 2SSP and our paper use 2048-token sequences and a sparsity defined over the transformer blocks. As a result, the dense perplexities differ (12.19 vs. 5.47), and the absolute values cannot be compared directly. Under our protocol, Putri is behind 2SSP on this MHA model. The results of all our baselines on Llama-2-7B are in our answer to Reviewer c1oT, W1.3.
Part 2/2
W3: Reproducibility. The anonymous code (link in the abstract) includes scripts that reproduce all tables. We will add an "Implementation details" appendix with the following: