- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates "priority dilution" in one-shot LLM pruning, where uniformly weighted reconstruction can overemphasize low-sensitivity calibration token occurrences. It proposes Sensitivity-Weighted Reconstruction (SWR), which derives occurrence sensitivity from squared module-output loss gradients, then normalizes and clips these values to reweight existing pruning solvers. Experiments span five pruning methods and 11 language models, including fixed-mask controls and structured-sparsity evaluations. At 70% unstructured sparsity, SWR improves WikiText-2 perplexity and mean zero-shot accuracy in 51 of 55 configurations, with a median 3.0% relative perplexity reduction. In addition, the authosrs provide detailed analyses and ablation studies to investigate module-specific sensitivity patterns. This paper is well-organized and technically sound.
Critical strengths
-
Well-motivated and broadly applicable approach. The paper identifies priority dilution in uniform reconstruction and motivates occurrence sensitivity through an empirical-Fisher surrogate. SWR modifies only reconstruction weights, making it compatible with multiple existing pruning methods without changing their optimization procedures. This offers a simple and practical improvement.
-
Extensive experimental validation with consistent improvements. Experiments across five pruning methods and 11 LLMs demonstrate consistent perplexity reductions and zero-shot accuracy improvements. Controlled experiments, including fixed-mask comparisons, alternative sensitivity statistics, and calibration-size analyses, provide further evidence supporting the effectiveness of sensitivity-weighted reconstruction.
Critical weaknesses
-
Limited Baselines. This paper adopts five pruning methods to showcase the effectiveness of the proposed SWR. However, these five methods are fairly old and out-of-the-date. They are not the SOTA baselines. If the authors can provide the experimental results on more powerful method like DenoiseRotator[1]. It will strengthen this paper.
-
Modest practical gains relative to computational overhead. Although SWR consistently improves pruning performance, the improvements are relatively modest, with a median 3.0% perplexity reduction and only 0.38 percentage points of zero-shot accuracy improvement. Meanwhile, computing sensitivities requires additional backward passes, increasing pruning time and GPU memory consumption. This raises concerns about the practical significance and cost-effectiveness of the proposed method.
-
** Ignore related works ** For one-shot pruning method, the authors ignore a very relavant work [2].
[1] DenoiseRotator: Enhance Pruning Robustness for LLMs via Importance Concentration. NeurIPS 2025.
[2] One-Shot Sensitivity-Aware Mixed Sparsity Pruning for Large Language Models. ICASSP 2024.
Questions for authors
-
Comparison with stronger baselines. Can the authors provide comparisons with more competitive baselines such as DenoiseRotator (NeurIPS 2025) or demonstrate whether SWR can further improve their performance?
-
Ignore reference to related work. Can the authors discuss the "One-Shot Sensitivity-Aware Mixed Sparsity Pruning for Large Language Models." in the related work section?
4 comments
(Part II)
W2: Practical improvements and computational cost
1. Improvement relative to more calibration data. SWR can outperform Uniform that uses four to eight times as many sequences. With DenoiseRotator, SWR with 128 reconstruction sequences outperforms Uniform with 1,024 on both models (Table R4). In the paper's calibration-scaling experiment, SWR with 256 calibration sequences outperforms Uniform with 1,024 in 9 of 11 models. On Llama-2-7B, this fourfold reduction lowers pruning time, excluding data preparation and model loading, from 1,019.6 to 498.6 seconds (51.1%) while reducing PPL from 37.35 to 36.71.
2. Measured cost. We prune Llama-2-7B with SparseGPT at 70% sparsity on one RTX PRO 6000 Blackwell in FP16, using 128 calibration sequences of 2,048 tokens and batch size 1. Times exclude data preparation, model loading, and evaluation. Appendix E used batch size 2 for sensitivity collection and reports a peak memory of 37.3 GiB for SWR on Llama-2-7B; this measurement uses batch size 1.
Table R11. Sensitivity-collection time, total pruning time, peak GPU memory, and WikiText-2 PPL (↓) on Llama-2-7B with SparseGPT at 70% sparsity.
| Configuration | Sensitivity collection (s) | Total time (s) | Peak GPU memory (GiB) | WikiText-2 ↓ |
|---|---|---|---|---|
| Native SparseGPT | — | 190.82 | 19.46 | 28.35 |
| SWR, fresh sensitivities | 75.35 | 270.57 | 21.90 | 26.16 |
| SWR, cached sensitivities | — | 193.26 | 19.46 | 26.16 |
| Low-memory SWR, fresh sensitivities | 87.99 | 288.40 | 15.50 | 26.16 |
On first use, SWR adds 79.75 seconds (41.8%) and 2.44 GiB (12.5%) of peak memory over Native SparseGPT. For the same dense model state, calibration tokens, loss and gradient definition, and numerical settings, one sensitivity collection can be reused across compatible masks, sparsity levels, and solvers. With cached sensitivities, SWR takes 2.44 seconds (1.28%) longer than Native SparseGPT at the same peak memory and produces the same pruning results as fresh-sensitivity SWR. If one collection is amortized over five compatible runs, the average overhead is 9.38%. The low-memory implementation, which combines activation checkpointing with CPU calibration buffers, lowers peak memory to 15.50 GiB, below Native SparseGPT, at 6.59% more time than standard SWR and with the same pruning results.
We will replace the measurements in Appendix E with those in Table R11 and distinguish first-use and cached costs.
W3 and Q2: Relation to sensitivity-aware mixed-sparsity pruning
Thank you for bringing this relevant work [4] to our attention. We will include it in Related Work. It combines curvature-based weight selection with Hessian-based sensitivity for allocating sparsity across layers or weight matrices. SWR addresses a different decision: how to weight calibration-occurrence reconstruction errors within each module. Our fixed-mask control experiments isolate this effect while keeping the pruning mask and sparsity allocation unchanged. We will clarify this distinction in the revised paper.
[4] Hang Shao, Bei Liu, and Yanmin Qian. One-Shot Sensitivity-Aware Mixed Sparsity Pruning for Large Language Models. ICASSP, 2024.
Reply to Authors' Comments
Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns. The new results provide useful evidence and further strengthen the paper's empirical contributions.
1. Comparisons with Stronger Pruning Methods. I appreciate the additional experiments integrating SWR with DenoiseRotator and ROSE. The consistent improvements across different models and sparsity settings provide further evidence of the general applicability of SWR. These results substantially address my concern regarding the lack of comparisons with more recent and competitive pruning methods.
2. Practical Improvements and Computational Overhead. The additional analysis of sensitivity caching, calibration efficiency, and GPU memory usage provides a more comprehensive understanding of the computational costs. In particular, the ability to reuse sensitivity statistics across compatible pruning configurations is a useful practical advantage. However, the performance gains on stronger pruning baselines remain relatively modest, and the additional computational overhead for first-time sensitivity collection is still non-negligible. Therefore, I believe the practical cost-effectiveness of SWR remains an important consideration.
3. Related Work. Thank you for clarifying the differences between SWR and One-Shot Sensitivity-Aware Mixed Sparsity Pruning for Large Language Models (ICASSP 2024). I find the explanation reasonable and appreciate the authors' commitment to including this relevant work in the revised paper.
Overall Assessment
Overall, I find the rebuttal convincing and acknowledge that the additional experiments have substantially addressed my original concerns. Therefore, I will maintain my rating of 3 (Weak Acceptance).
Thank you for your follow-up and for recognizing that the additional experiments address your main concerns. We appreciate your point about practical cost-effectiveness and will make this trade-off explicit in the revised paper. Thank you again for your time and constructive feedback.
(Part I)
Dear Reviewer 9kiT,
We are grateful for your thoughtful and practical feedback. Your comments helped us assess the practical value of SWR more concretely. Please find our point-by-point responses below.
W1 and Q1: Integration with recent pruning methods
Thank you for suggesting DenoiseRotator and highlighting the practical cost. We integrated SWR with DenoiseRotator using SparseGPT and additionally with ROSE, evaluating both on Llama-3.1-8B and Qwen3-8B. The results are reported in the General Response; for your convenience, we reproduce them below.
For DenoiseRotator, we follow its setup and use the default SWR settings from our paper. We evaluate 50% and 70% unstructured sparsity and 2:4 sparsity.
Table R1. PPL (↓) and seven-task mean accuracy (%, ↑) results across sparsity settings. Each row reports DenoiseRotator → DenoiseRotator+SWR.
In all six configurations, SWR lowers PPL on all three datasets, with a median relative reduction of 1.15% (range 0.16–4.02%), and improves seven-task mean accuracy, with a median increase of 0.28 percentage points (range 0.11–0.50).
For ROSE, we apply SWR to its reconstruction Gram matrix while preserving and rerunning its ordering algorithm.
Table R2. PPL (↓) and seven-task mean accuracy (%, ↑) results with ROSE at 50% and 70% unstructured sparsity. Each row reports ROSE → ROSE+SWR.
SWR lowers PPL in 10 of 12 dataset comparisons, with a 1.14% median relative reduction, and improves seven-task mean accuracy in all four configurations, with a median increase of 0.25 percentage points.
In addition, DenoiseRotator exhibits priority dilution, the issue our paper identifies, and SWR mitigates it (General Response, Section 3).