- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the high latency of LLM decoding caused by memory-bound inference. The authors argue that common Hessian-based training-free pruning methods calibrate on fixed corpora, while real decoding feeds the model self-generated tokens, leading to a distribution shift that hurts pruned models. To mitigate this, the paper proposes SparseDecoding, which builds calibration matrices from activations collected during dense-model autoregressive decoding, making the pruning objective more aligned with decoding behavior. The paper also converts pruned weights into N:M semi-structured sparsity and designs a custom SpMV kernel to speed up decoding. Experiments on Llama and Qwen models cover both unstructured and semi-structured sparsity settings, showing some improvements over fixed-corpus calibration baselines and moderate decoding speedups.
Critical strengths
-
The paper is clearly written. The flow from motivation to method to experiments is easy to follow, and the figures and tables do a good job of conveying the main ideas.
-
The authors analyze the gap between calibration data and decoding data, and give an upper bound on the reconstruction error. This makes the idea of changing the calibration source well grounded rather than just a guess.
Critical weaknesses
-
The core pruning method builds on OBS + SparseGPT backend.
-
Lacks systematic comparison with recent SOTA pruning methods.
-
One problem I have is more about the pruning field in general than this paper alone. Too many pruning papers only compare against other pruning methods, and the whole area ends up looking fancy but not very useful in practice. Another common habit is pushing the sparsity ratio higher and higher, then claiming the method is better because it survives at 50% or 90% sparsity while others collapse. That comparison often means little, because a model that loses most of its ability at extreme sparsity is not something anyone would actually deploy. What is usually missing is a comparison with smaller dense models from the same family. If a pruned large model still cannot beat a smaller model, then pruning does not really save much, because people would just use the smaller one directly. Here it is especially easy to check, since both Qwen and LLaMA have smaller counterparts. The paper would be much more convincing if it showed that the pruned model is better than a smaller dense model in both quality and real cost, including calibration time and kernel implementation effort.
Questions for authors
I have one main question about all pruning methods: is pruning really more meaningful than just using a smaller model?
5 comments
Response to Reviewer CpsE (Part 2/3)
W2: Lacks systematic comparison with recent SOTA pruning methods.
W2-Ans: We thank the reviewer for raising this question.
We compare calibration methods while keeping the model, pruning backend, sparsity pattern, and calibration token budget fixed. SparseDecoding improves over C4 calibration in all reported settings with both SparseGPT and Wanda. We further compare with ROSE [1] and D²Prune [2], using their official implementations with C4 calibration and the same 1M-token calibration budget across methods. SparseDecoding outperforms both methods under 50% and 2:4 sparsity.
Table B: Results on the WritingBench benchmark under 50% and 2:4 sparsity with ROSE, D²Prune, and SparseDecoding on the Qwen3-14B.
| Model | Sparsity | Method | Overall |
|---|---|---|---|
| Qwen3-14B | 0% | Dense | 5.97 |
| 50% | D²Prune | 4.86 | |
| ROSE | 4.98 | ||
| SparseDecoding (SparseGPT) | 5.35 | ||
| 2:4 | D²Prune | 2.34 | |
| ROSE | 2.24 | ||
| SparseDecoding (SparseGPT) | 4.18 |
We hope these additional results and explanations address your concern.
References
[1] Su and Wang (2026). ROSE: Reordered SparseGPT for More Accurate One-Shot Large Language Models Pruning. CPAL 2026.
[2] Xiong et al. (2026). D²Prune: Sparsifying Large Language Models via Dual Taylor Expansion and Attention Distribution Awareness. AAAI 2026.
Response to Reviewer CpsE (Part 3/3)
W3 & Q1: Too many pruning papers only compare against other pruning methods, and the whole area ends up looking fancy but not very useful in practice.
W3 & Q1-Ans: We thank the reviewer for raising this point.
-
We agree that many pruning methods stop at reporting sparsity and accuracy without turning them into real efficiency gains, which leaves the impression that pruning is of limited practical use.
-
The concern that a smaller model can replace a pruned one is well supported for small models and structured pruning, where the smaller network can simply be trained from scratch [3]. For large models, pruning is a practical way to reduce inference cost, since training a smaller model from scratch is expensive. In practice, however, structured pruning often degrades quality, and unstructured or semi-structured sparsity requires a dedicated kernel that most prior work does not provide, so the sparsity rarely turns into real speedup.
-
This is exactly the gap our work addresses. SparseDecoding aligns calibration with decoding to preserve quality, and our specialized N:M SpMV kernel turns sparsity into measured end-to-end speedups of 1.34 to 1.48× over dense on A100 GPUs, whereas the existing 2:4 Tensor Core path runs at only 0.85 to 0.87× of dense during decoding.
-
We present the comparison between our pruned Qwen2.5-72B and dense Qwen2.5-32B with the downstream quality and throughput performance as follows:
Table C: Results on the WritingBench benchmark under dense and 16:32 sparsity with SparseDecoding on Qwen2.5-32B and Qwen2.5-72B.
| Model | Sparsity | Overall |
|---|---|---|
| Qwen2.5-32B | 0% (Dense) | 5.18 |
| Qwen2.5-72B | 16:32 | 5.26 |
Table D: End-to-end decoding throughput of Qwen2.5-72B with 16:32 sparsity and dense Qwen2.5-32B.
| Model | Sparsity | Tokens/s |
|---|---|---|
| Qwen2.5-32B | 0% (Dense) | 20.4 |
| Qwen2.5-72B | 16:32 | 29.7 |
As shown in Table C and Table D, the pruned larger model achieves higher quality and higher throughput than the smaller dense model.
-
Calibration cost. For Qwen2.5-72B with 1M calibration tokens, calibration takes 59.1 GPU-hours for generation, 29.3 GPU-hours for Hessian collection, and 5.6 GPU-hours for pruning, about 94 A100 GPU-hours in total. It is a one-time cost with no retraining, compared with millions of GPU-hours for pretraining.
-
Kernel implementation effort. Our kernel is implemented once and can be reused across many models. It is a single Triton implementation, inspired by nmSPARSE and instantiated for each sparsity pattern, with execution configurations autotuned for each matrix shape. Deploying a new model only requires replacing its linear layers and packing the sparse weights, with no new CUDA code.
In summary, pruned Qwen2.5-72B achieves higher quality and higher decoding throughput than dense Qwen2.5-32B, with a one-time calibration cost of about 94 GPU-hours and a reusable kernel. We hope this also answers Q1.
Thank you again for your suggestions. We hope this explanation addresses your concerns. The detailed revisions can be found in Sections 2.2, 3.2, 4.5, and Appendix D (Tables 11-14) in the revised manuscript.
References
[3] Liu et al. (2019). Rethinking the Value of Network Pruning. ICLR 2019.
Response to Reviewer CpsE (Part 1/3)
We thank the reviewer for the thoughtful review and for recognizing the clarity of our writing. We are glad that the reviewer finds that our analysis of the gap between calibration data and decoding data, with the reconstruction error bound, makes the idea "well grounded rather than just a guess". Below, we respond to each question in turn. Additional results are provided in the tables below.
W1-Ans: We thank the reviewer for this observation and agree that SparseDecoding uses an OBS-style solver (SparseGPT by default) as its backend. Our work addresses the mismatch between calibration activations and decode-time activations, a problem orthogonal to pruning solver design that has received limited attention in general non-thinking decoding. We replace activations collected from teacher-forced text with those collected during autoregressive decoding, while keeping the pruning solver unchanged. This defines the reconstruction objective over decode-time activations. Table A shows consistent gains with Wanda, demonstrating that the benefits extend beyond the SparseGPT backend.
Table A: Results on the WritingBench benchmark under 50% and 2:4 sparsity with Wanda on the Qwen3 model family.
We hope these clarifications address your concerns.