- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes the WUSPruner framework, which formulates basis selection as a local linear reconstruction problem that directly minimizes the reconstruction loss induced by pruning and constructs a WUS transform jointly from activation and weight statistics. Experiments on Llama and Qwen models ranging from 8B to 72B parameters demonstrate consistent improvements over structured-pruning baselines across various models and tasks, together with practical reductions in inference latency and memory usage. Overall, the results demonstrate that WUSPruner provides an effective and practical approach to structured LLM compression.
Critical strengths
(1) The motivation is clear, focusing on the basis dependence of coordinate deletion in structured pruning and the challenge of selecting a basis that minimizes reconstruction error.
(2) The optimization objective is well defined, as it directly minimizes the reconstruction loss of local linear layers.
(3) The method leverages both activation and weight statistics to construct the pruning basis, enabling more principled dimension selection and structured deletion.
(4) The method is easy to follow and understand.
(5) The experimental evaluation is comprehensive, covering multiple model families, model scales, and downstream tasks.
Critical weaknesses
(1) The compared baselines are limited and lack representative methods. It is suggested that the authors include more powerful baselines for comparison.
(2) The contribution of combining activation and weight second moments is unclear. The authors are encouraged to provide experiments evaluating the individual design components.
(3) The major concern is that there remains a significant performance gap between WUSPruner and the dense model, while the improvements in inference speed and reductions in GPU memory usage are modest.
(4) More computational metrics should be provided, including the number of trainable parameters, FLOPs, and compression time at each compression ratio, to enable a comprehensive comparison across baselines.
Questions for authors
(1) Why does the comparison lack representative structured-pruning methods such as LLM-Pruner, FLAP, and ZipLM?
(2) Could the authors provide ablations for different components of WUSPruner?
(3) Why is SVD-LLM evaluated only on Qwen3-8B at a 25% sparsity level? Could the authors provide additional comparisons across different sparsity levels and model scales? Although WUSPruner achieves better average performance, it does not consistently outperform SVD-LLM on every benchmark.
(4) Is the transformed model functionally equivalent to the dense model before coordinate deletion?
(5) The theory assumes positive-definite second moments, while the implementation uses diagonal damping for numerical stability. How does damping affect the optimality guarantee and pruning criterion?
5 comments
Response to Reviewer NzXx [2/4]
W4: More computational metrics should be provided, including the number of trainable parameters, FLOPs, and compression time at each compression ratio, to enable a comprehensive comparison across baselines.
We provide the parameter count and FLOPs comparison with baselines after compression. The additional results cover two settings:
Across models: comparisons with Wanda and SliceGPT on five models, from Llama3-8B to Qwen2.5-72B.
Across pruning ratios: comparisons with SliceGPT at seven ratios from 15% to 50% on Qwen3-8B and six ratios from 15% to 40% on Qwen3.5-9B.
| Model | Method | Model size | GFLOPs / token |
|---|---|---|---|
| Llama3-8B | Wanda | 6.0B | 10.9 |
| SliceGPT | 6.6B | 12.4 | |
| WUSPruner | 6.4B | 11.7 | |
| Qwen3-8B | Wanda | 6.0B | 10.7 |
| SliceGPT | 6.8B | 12.6 | |
| WUSPruner | 6.6B | 11.9 | |
| Qwen3.5-9B | Wanda | 6.6B | 11.2 |
| SliceGPT | 7.3B | 13.1 | |
| WUSPruner | 7.2B | 12.4 | |
| Qwen3.8-27B | Wanda | 20.3B | 38.3 |
| SliceGPT | 22.0B | 42.3 | |
| WUSPruner | 21.5B | 40.7 | |
| Qwen2.5-72B | Wanda | 54.8B | 107.1 |
| SliceGPT | 60.5B | 119.1 | |
| WUSPruner | 55.5B | 108.5 |
Qwen3-8B:
| Compression Ratio | WUSPruner Model size | WUSPruner GFLOPs | SliceGPT Model Size | SliceGPT GFLOPs |
|---|---|---|---|---|
| 15% | 7.4B | 13.7 | 7.8B | 14.6 |
| 20% | 7.1B | 12.9 | 7.3B | 13.6 |
| 25% | 6.6B | 11.9 | 6.8B | 12.6 |
| 30% | 6.3B | 11.4 | 6.3B | 11.7 |
| 35% | 5.9B | 10.6 | 5.8B | 10.8 |
| 40% | 5.5B | 9.8 | 5.3B | 9.9 |
| 50% | 4.7B | 8.3 | 4.3B | 8.1 |
Qwen3.5-9B:
| Compression Ratio | WUSPruner Model size | WUSPruner GFLOPs | SliceGPT Model Size | SliceGPT GFLOPs |
|---|---|---|---|---|
| 15% | 8.1B | 14.3 | 8.4B | 15.1 |
| 20% | 7.7B | 13.5 | 7.8B | 14.1 |
| 25% | 7.2B | 12.4 | 7.3B | 13.1 |
| 30% | 6.9B | 11.8 | 6.8B | 12.2 |
| 35% | 6.5B | 11.0 | 6.3B | 11.3 |
| 40% | 6.1B | 10.2 | 5.8B | 10.4 |
WUSPruner requires fewer FLOPs per token than SliceGPT throughout the 15%–40% range on both models. Parameter counts are lower for WUSPruner at lower pruning ratios, while SliceGPT becomes more compact at higher ratios. At 50% pruning on Qwen3-8B, SliceGPT also has slightly lower FLOPs. These results make the computational trade-offs across pruning ratios explicit.
For reference, compression time is reported in Table 10. We will add these results in our revision.
Q1: Why does the comparison lack representative structured-pruning methods such as LLM-Pruner, FLAP, and ZipLM?
We thank the reviewer for this suggestion. Our initial comparison prioritized SliceGPT because it is the most directly related transformation-based structured-pruning baseline: both methods transform the representation before removing dimensions. This makes SliceGPT particularly relevant for evaluating our proposed approach. We added the results in W1.
Q2: Could the authors provide ablations for different components of WUSPruner?
Thank you for raising this question. We provide the comparison in W2.
Response to Reviewer NzXx [3/4]
Q3: Why is SVD-LLM evaluated only on Qwen3-8B at a 25% sparsity level? Could the authors provide additional comparisons across different sparsity levels and model scales? Although WUSPruner achieves better average performance, it does not consistently outperform SVD-LLM on every benchmark.
We thank the reviewer for this suggestion. We have added comparisons at a higher compression ratio and a larger model scale:
Qwen3-8B at 35% compression ratio.
| Method | Wikitext ppl | C4 ppl | PTB ppl | Avg | PIQA | HellaSwag | ARC-C | MMLU | OBQA | CMMLU | Avg |
|---|---|---|---|---|---|---|---|---|---|---|---|
| SVD-LLM 35% | 21.4 | 82.5 | 204.5 | 102.8 | 61.4 | 41.9 | 27.6 | 25.0 | 30.1 | 25.9 | 35.3 |
| WUSPruner 35% | 21.4 | 64.3 | 59.1 | 48.3 | 60.7 | 42.3 | 28.2 | 37.7 | 30.6 | 29.9 | 38.2 |
Qwen3.8-27B 25% compression ratio:
| Method | Wikitext ppl | C4 ppl | PTB ppl | Avg | PIQA | HellaSwag | ARC-C | MMLU | OBQA | CMMLU | Avg |
|---|---|---|---|---|---|---|---|---|---|---|---|
| SVD-LLM 25% | 9.4 | 23.2 | 28.1 | 20.2 | 73.2 | 67.8 | 49.3 | 67.9 | 40.3 | 57.5 | 59.3 |
| WUSPruner 25% | 9.1 | 16.9 | 13.3 | 13.1 | 75.7 | 68.6 | 51.1 | 69.1 | 40.6 | 60.5 | 60.9 |
Qwen3-8B at 35% compression: WUSPruner achieves an average perplexity of 48.3, compared with 102.8 for SVD-LLM, and an average downstream accuracy of 38.2% versus 35.3%. It improves five of the six downstream benchmarks, including a 12.7-percentage-point gain on MMLU. WikiText-2 perplexity is identical, while SVD-LLM performs slightly better on PIQA.
Qwen3.8-27B at 25% compression: WUSPruner achieves lower perplexity on all three datasets and higher accuracy on all six downstream tasks. Average perplexity decreases from 20.2 to 13.1, and average accuracy increases from 59.3% to 60.9%.
We acknowledge that WUSPruner does not outperform SVD-LLM on every benchmark in every setting. Our claim concerns its overall performance across the evaluated tasks, rather than uniform superiority on individual benchmarks. These additional results extend the evidence beyond the original Qwen3-8B comparison at 25% compression, demonstrating aggregate improvements at both a higher compression ratio and a larger model scale.
Q4: Is the transformed model functionally equivalent to the dense model before coordinate deletion?
Yes. Before coordinate deletion, the transformed model is functionally equivalent to the dense model in exact arithmetic, provided each transformation is paired with its corresponding inverse.
For a linear map Y=XW+b, we define $ X'=XT^\top, W'=T^{-\top}W, $ where T is invertible and the bias remains unchanged. Consequently, $ X'W'+b =XT^\top T^{-\top}W+b =XW+b. $ This identity holds for any invertible T; orthogonality is not required. In WUSPruner, these paired transformations are applied locally at the linear maps. Their outputs are therefore preserved before deletion, while the surrounding nonlinearities, normalization operations, and residual connections remain unchanged.
Response to Reviewer NzXx [1/4]
Thank you very much for your detailed review and suggestions. We address your concerns via response below:
We thank the reviewer for this suggestion. We have expanded the comparison to include LLM-Pruner, FLAP, and ZipLM, alongside SliceGPT, on Qwen3-8B at the evaluated 25% pruning setting.
As shown in the table, WUSPruner achieves the lowest average perplexity (25.7) and the highest average downstream accuracy (49.5%) among all compared methods. FLAP is the strongest baseline on both aggregate metrics; WUSPruner improves upon it by 4.5 perplexity points and 3.2 percentage points in average accuracy, including gains of 7.4 points on MMLU and 9.8 points on CMMLU.
WUSPruner also achieves the lowest PTB perplexity and the highest accuracy on four of the six downstream benchmarks. Although individual baselines perform better on some datasets, the expanded comparison supports WUSPruner’s overall effectiveness against a broader set of representative structured pruning methods.
We thank the reviewer for this suggestion. We have added component ablations comparing activation-only, weight-only, and the full joint activation–weight construction on Qwen3-8B with 25% compression ratio. We apply the same settings for all experiments.
The joint construction achieves an average downstream accuracy of 49.5%, compared with 41.3% for activation-only and 30.5% for weight-only, improving by 8.2 and 19.0 percentage points, respectively. WUSPruner also outperforms both individual variants on all six downstream benchmarks. Relative to activation-only, the gains are particularly pronounced on MMLU and CMMLU, reaching 13.3 and 14.8 percentage points, respectively.
We acknowledge the reviewer’s concern. Our experiments focus on one-shot structured pruning without recovery fine-tuning. Removing model capacity in this setting can cause substantial degradation, since no subsequent training is performed to recover the lost capabilities. This limitation also affects the compared baselines. Our contribution is to improve performance preservation within this constrained setting. Specifically, at 25% pruning on Qwen3-8B, WUSPruner achieves the best average perplexity and downstream accuracy among LLM-Pruner, FLAP, SliceGPT, and WUSPruner.