acceptodds
Under review as a conference paper at ICLR 2027

WUSPruner: Second-Moment-Optimal Transforms for Structured Pruning

Abstract

The effectiveness of structured pruning depends on the basis in which model dimensions are removed. This raises a central question: how should a transformation be chosen to make coordinate deletion preserve the original computation? We introduce WUSPruner, a transform-based framework for structured pruning of large language models. We characterize the optimal invertible transformation for a local linear reconstruction objective and show that, under positive-definite second moments, the WUS transform attains the minimum deletion error. In this basis, the error decomposes into coordinate-wise contributions determined by a singular-value spectrum, providing a principled rule for truncation. Guided by this result, we construct a practical pruning method using blockwise polar factors of WUS, with coordinate importance recomputed in the resulting orthogonal bases. Experiments on Llama and Qwen models from 8B to 72B parameters show higher average accuracy than baselines across six knowledge and commonsense benchmarks and lower perplexity on PTB and C4. These findings demonstrate the value of designing the transformation around the coordinate-deletion objective for structured LLM pruning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Structured Pruning, Basis Transform

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_NzXxOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the WUSPruner framework, which formulates basis selection as a local linear reconstruction problem that directly minimizes the reconstruction loss induced by pruning and constructs a WUS transform jointly from activation and weight statistics. Experiments on Llama and Qwen models ranging from 8B to 72B parameters demonstrate consistent improvements over structured-pruning baselines across various models and tasks, together with practical reductions in inference latency and memory usage. Overall, the results demonstrate that WUSPruner provides an effective and practical approach to structured LLM compression.

Critical strengths

(1) The motivation is clear, focusing on the basis dependence of coordinate deletion in structured pruning and the challenge of selecting a basis that minimizes reconstruction error.

(2) The optimization objective is well defined, as it directly minimizes the reconstruction loss of local linear layers.

(3) The method leverages both activation and weight statistics to construct the pruning basis, enabling more principled dimension selection and structured deletion.

(4) The method is easy to follow and understand.

(5) The experimental evaluation is comprehensive, covering multiple model families, model scales, and downstream tasks.

Critical weaknesses

(1) The compared baselines are limited and lack representative methods. It is suggested that the authors include more powerful baselines for comparison.

(2) The contribution of combining activation and weight second moments is unclear. The authors are encouraged to provide experiments evaluating the individual design components.

(3) The major concern is that there remains a significant performance gap between WUSPruner and the dense model, while the improvements in inference speed and reductions in GPU memory usage are modest.

(4) More computational metrics should be provided, including the number of trainable parameters, FLOPs, and compression time at each compression ratio, to enable a comprehensive comparison across baselines.

Questions for authors

(1) Why does the comparison lack representative structured-pruning methods such as LLM-Pruner, FLAP, and ZipLM?

(2) Could the authors provide ablations for different components of WUSPruner?

(3) Why is SVD-LLM evaluated only on Qwen3-8B at a 25% sparsity level? Could the authors provide additional comparisons across different sparsity levels and model scales? Although WUSPruner achieves better average performance, it does not consistently outperform SVD-LLM on every benchmark.

(4) Is the transformed model functionally equivalent to the dense model before coordinate deletion?

(5) The theory assumes positive-definite second moments, while the implementation uses diagonal damping for numerical stability. How does damping affect the optimality guarantee and pruning criterion?

5 comments
AuthorsOpenReview ↗

Response to Reviewer NzXx [1/4]

Thank you very much for your detailed review and suggestions. We address your concerns via response below:

W1: The compared baselines are limited and lack representative methods. It is suggested that the authors include more powerful baselines for comparison.

We thank the reviewer for this suggestion. We have expanded the comparison to include LLM-Pruner, FLAP, and ZipLM, alongside SliceGPT, on Qwen3-8B at the evaluated 25% pruning setting.

MethodWikitext pplC4 pplPTB pplAvgPIQAHellaSwagARC-CMMLUOBQACMMLUAvg
LLM-Pruner24.751.455.343.865.142.228.031.830.226.437.3
FLAP17.533.140.130.269.751.337.846.934.537.746.3
ZipLM15.036.839.930.564.048.233.234.733.026.139.9
SliceGPT13.143.867.841.567.655.535.331.633.428.642.0
WUSPruner15.033.728.625.767.853.739.254.334.647.549.5

As shown in the table, WUSPruner achieves the lowest average perplexity (25.7) and the highest average downstream accuracy (49.5%) among all compared methods. FLAP is the strongest baseline on both aggregate metrics; WUSPruner improves upon it by 4.5 perplexity points and 3.2 percentage points in average accuracy, including gains of 7.4 points on MMLU and 9.8 points on CMMLU.

WUSPruner also achieves the lowest PTB perplexity and the highest accuracy on four of the six downstream benchmarks. Although individual baselines perform better on some datasets, the expanded comparison supports WUSPruner’s overall effectiveness against a broader set of representative structured pruning methods.

W2: The contribution of combining activation and weight second moments is unclear. The authors are encouraged to provide experiments evaluating the individual design components.

We thank the reviewer for this suggestion. We have added component ablations comparing activation-only, weight-only, and the full joint activation–weight construction on Qwen3-8B with 25% compression ratio. We apply the same settings for all experiments.

MethodWikitext pplC4 pplPTB pplAvgPIQAHellaSwagARC-CMMLUOBQACMMLUAvg
Activation only15.838.531.528.663.947.631.941.031.232.741.3
Weight only1.96E+052.33E+053.30E+052.53E+0553.626.825.624.128.025.130.5
WUSPruner15.033.728.625.767.853.739.254.334.647.549.5

The joint construction achieves an average downstream accuracy of 49.5%, compared with 41.3% for activation-only and 30.5% for weight-only, improving by 8.2 and 19.0 percentage points, respectively. WUSPruner also outperforms both individual variants on all six downstream benchmarks. Relative to activation-only, the gains are particularly pronounced on MMLU and CMMLU, reaching 13.3 and 14.8 percentage points, respectively.

W3: The major concern is that there remains a significant performance gap between WUSPruner and the dense model, while the improvements in inference speed and reductions in GPU memory usage are modest.

We acknowledge the reviewer’s concern. Our experiments focus on one-shot structured pruning without recovery fine-tuning. Removing model capacity in this setting can cause substantial degradation, since no subsequent training is performed to recover the lost capabilities. This limitation also affects the compared baselines. Our contribution is to improve performance preservation within this constrained setting. Specifically, at 25% pruning on Qwen3-8B, WUSPruner achieves the best average perplexity and downstream accuracy among LLM-Pruner, FLAP, SliceGPT, and WUSPruner.

AuthorsOpenReview ↗

Response to Reviewer NzXx [2/4]

W4: More computational metrics should be provided, including the number of trainable parameters, FLOPs, and compression time at each compression ratio, to enable a comprehensive comparison across baselines.

We provide the parameter count and FLOPs comparison with baselines after compression. The additional results cover two settings:

Across models: comparisons with Wanda and SliceGPT on five models, from Llama3-8B to Qwen2.5-72B.

Across pruning ratios: comparisons with SliceGPT at seven ratios from 15% to 50% on Qwen3-8B and six ratios from 15% to 40% on Qwen3.5-9B.

ModelMethodModel sizeGFLOPs / token
Llama3-8BWanda6.0B10.9
SliceGPT6.6B12.4
WUSPruner6.4B11.7
Qwen3-8BWanda6.0B10.7
SliceGPT6.8B12.6
WUSPruner6.6B11.9
Qwen3.5-9BWanda6.6B11.2
SliceGPT7.3B13.1
WUSPruner7.2B12.4
Qwen3.8-27BWanda20.3B38.3
SliceGPT22.0B42.3
WUSPruner21.5B40.7
Qwen2.5-72BWanda54.8B107.1
SliceGPT60.5B119.1
WUSPruner55.5B108.5

Qwen3-8B:

Compression RatioWUSPruner Model sizeWUSPruner GFLOPsSliceGPT Model SizeSliceGPT GFLOPs
15%7.4B13.77.8B14.6
20%7.1B12.97.3B13.6
25%6.6B11.96.8B12.6
30%6.3B11.46.3B11.7
35%5.9B10.65.8B10.8
40%5.5B9.85.3B9.9
50%4.7B8.34.3B8.1

Qwen3.5-9B:

Compression RatioWUSPruner Model sizeWUSPruner GFLOPsSliceGPT Model SizeSliceGPT GFLOPs
15%8.1B14.38.4B15.1
20%7.7B13.57.8B14.1
25%7.2B12.47.3B13.1
30%6.9B11.86.8B12.2
35%6.5B11.06.3B11.3
40%6.1B10.25.8B10.4

WUSPruner requires fewer FLOPs per token than SliceGPT throughout the 15%–40% range on both models. Parameter counts are lower for WUSPruner at lower pruning ratios, while SliceGPT becomes more compact at higher ratios. At 50% pruning on Qwen3-8B, SliceGPT also has slightly lower FLOPs. These results make the computational trade-offs across pruning ratios explicit.

For reference, compression time is reported in Table 10. We will add these results in our revision.

Q1: Why does the comparison lack representative structured-pruning methods such as LLM-Pruner, FLAP, and ZipLM?

We thank the reviewer for this suggestion. Our initial comparison prioritized SliceGPT because it is the most directly related transformation-based structured-pruning baseline: both methods transform the representation before removing dimensions. This makes SliceGPT particularly relevant for evaluating our proposed approach. We added the results in W1.

Q2: Could the authors provide ablations for different components of WUSPruner?

Thank you for raising this question. We provide the comparison in W2.

AuthorsOpenReview ↗

Response to Reviewer NzXx [3/4]

Q3: Why is SVD-LLM evaluated only on Qwen3-8B at a 25% sparsity level? Could the authors provide additional comparisons across different sparsity levels and model scales? Although WUSPruner achieves better average performance, it does not consistently outperform SVD-LLM on every benchmark.

We thank the reviewer for this suggestion. We have added comparisons at a higher compression ratio and a larger model scale:

Qwen3-8B at 35% compression ratio.

MethodWikitext pplC4 pplPTB pplAvgPIQAHellaSwagARC-CMMLUOBQACMMLUAvg
SVD-LLM 35%21.482.5204.5102.861.441.927.625.030.125.935.3
WUSPruner 35%21.464.359.148.360.742.328.237.730.629.938.2

Qwen3.8-27B 25% compression ratio:

MethodWikitext pplC4 pplPTB pplAvgPIQAHellaSwagARC-CMMLUOBQACMMLUAvg
SVD-LLM 25%9.423.228.120.273.267.849.367.940.357.559.3
WUSPruner 25%9.116.913.313.175.768.651.169.140.660.560.9

Qwen3-8B at 35% compression: WUSPruner achieves an average perplexity of 48.3, compared with 102.8 for SVD-LLM, and an average downstream accuracy of 38.2% versus 35.3%. It improves five of the six downstream benchmarks, including a 12.7-percentage-point gain on MMLU. WikiText-2 perplexity is identical, while SVD-LLM performs slightly better on PIQA.

Qwen3.8-27B at 25% compression: WUSPruner achieves lower perplexity on all three datasets and higher accuracy on all six downstream tasks. Average perplexity decreases from 20.2 to 13.1, and average accuracy increases from 59.3% to 60.9%.

We acknowledge that WUSPruner does not outperform SVD-LLM on every benchmark in every setting. Our claim concerns its overall performance across the evaluated tasks, rather than uniform superiority on individual benchmarks. These additional results extend the evidence beyond the original Qwen3-8B comparison at 25% compression, demonstrating aggregate improvements at both a higher compression ratio and a larger model scale.

Q4: Is the transformed model functionally equivalent to the dense model before coordinate deletion?

Yes. Before coordinate deletion, the transformed model is functionally equivalent to the dense model in exact arithmetic, provided each transformation is paired with its corresponding inverse.

For a linear map Y=XW+b, we define $ X'=XT^\top, W'=T^{-\top}W, $ where T is invertible and the bias remains unchanged. Consequently, $ X'W'+b =XT^\top T^{-\top}W+b =XW+b. $ This identity holds for any invertible T; orthogonality is not required. In WUSPruner, these paired transformations are applied locally at the linear maps. Their outputs are therefore preserved before deletion, while the surrounding nonlinearities, normalization operations, and residual connections remain unchanged.

Reviewer_ev54OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes WUSPruner, a transform-based method for structured LLM compression that selects coordinates using both activation and weight second moments. Under positive-definite second moments, it proves that the WUS transformation minimizes local output reconstruction error over invertible changes of basis, with deletion costs determined by a singular-value spectrum. The practical method uses blockwise transformations and compact input projections, including shared projections for gate/up mappings, while preserving the residual-stream width. Experiments on five models from 8B to 72B report improvements over the evaluated compression baselines, together with measured latency and memory reductions.

Critical strengths

S1: The local analysis is precise and appropriately scoped.

S2: The empirical evidence includes substantial gains at meaningful scales

Critical weaknesses

W1: The contribution beyond activation-aware SVD is not sufficiently isolated. Algebraically, under the theorem’s assumptions, WUS followed by coordinate deletion yields W^=LX−⊤[LX⊤W]r\widehat W=L_X^{-\top}[L_X^\top W]_r, where [⋅]r[\cdot]_r denotes truncated SVD—the same local approximation obtained by activation-whitened SVD, rather than a different reconstruction optimum. The SVD-LLM comparison is useful, but without a matched blockwise/shared-projection comparison or component ablations, it remains unclear whether the gains arise from block structure, projection sharing, retained ranks, or other implementation choices rather than the proposed basis itself.

W2: Important details needed to verify the accuracy–compression tradeoff are missing. Although Section 3.3 explicitly counts projection storage, the experimental setup does not specify the default block width, per-map retention budgets, final parameter counts, or the exact modules compressed by each baseline, making the nominal “25% compression” comparisons difficult to audit. Table 5 also leaves the model, precision, and precise latency definition unclear, while its largest speedup is reported at 50% compression, beyond the quality results provided in Table 9.

Questions for authors

Q1: Do you agree that, without damping, WUS coordinate deletion is equivalent to activation-whitened truncated SVD at the same retained rank? What distinguishes the proposed method beyond its blockwise representation and projection sharing, and do existing ablations isolate these contributions?

Q2: Does “25% compression” mean a 25% reduction in total model parameters, including projections and the language-modeling head? Please provide final parameter counts and compressed-module lists for WUSPruner and the baselines.

Q3: Which model and numerical precision were used in Table 5, and does latency include prefill and all 64 decoding steps, or represent a normalized per-token measurement? Are existing model-quality results available for the reported 50% setting?

4 comments
AuthorsOpenReview ↗

Response to Reviewer ev54 [1/3]

Thank you very much for your detailed review and feedback. The points you raised give us a chance to add more clarification to our method. We address your concerns via response below:

W1: The contribution beyond activation-aware SVD is not sufficiently isolated. Algebraically, under the theorem’s assumptions, WUS followed by coordinate deletion yields, where denotes truncated SVD—the same local approximation obtained by activation-whitened SVD, rather than a different reconstruction optimum. The SVD-LLM comparison is useful, but without a matched blockwise/shared-projection comparison or component ablations, it remains unclear whether the gains arise from block structure, projection sharing, retained ranks, or other implementation choices rather than the proposed basis itself.

We sincerely thank the reviewer for raising this important concern. We clarify that under the theorem’s assumptions, WUS coordinate deletion yields the same local approximation as activation-whitened truncated SVD at the same retained rank. We do not claim the better reconstruction error in our paper. However, compared with activation-whitened SVD, the main performance gain of WUSPruner comes from the higher retained rank under the same compression ratio.

For example, consider a 4096×40964096\times4096 weight matrix. A 25% reduction in total compression ratio leaves a budget of 0.75×409620.75\times4096^2 scalars. A standard SVD representation stores two factors with 2×4096×r2\times4096\times r scalars, giving a retained rank of 1536. For WUSPruner, the reduced weight matrix stores (4096r4096r) scalars. Including transform PP, the same budget requires $ 4096r+P=0.75\times4096^2, r=3072-\frac{P}{4096}. $ If we apply 512 as the block size of the transform and thus P=512×512×(4096/512)P= 512 \times 512 \times (4096 / 512). The retained rank r is 2560. Therefore, WUSPruner can retain a higher rank within the same compression ratio. The block structures and projection sharing are all practical designs to reduce the parameter cost of PP.

AuthorsOpenReview ↗

Response to Reviewer ev54 [2/3]

W2: Important details needed to verify the accuracy–compression tradeoff are missing. Although Section 3.3 explicitly counts projection storage, the experimental setup does not specify the default block width, per-map retention budgets, final parameter counts, or the exact modules compressed by each baseline, making the nominal “25% compression” comparisons difficult to audit. Table 5 also leaves the model, precision, and precise latency definition unclear, while its largest speedup is reported at 50% compression, beyond the quality results provided in Table 9.

Thank you for raising this concern. We will add these details in the revision. Specifically, we use 512 as the default block size in all experiments unless otherwise specified. We provide the final parameter count comparison with 25% compression ratio:

ModelMethodModel size
Llama3-8BWanda6.0B
SliceGPT6.6B
WUSPruner6.4B
Qwen3-8BWanda6.0B
SliceGPT6.8B
WUSPruner6.6B
Qwen3.5-9BWanda6.6B
SliceGPT7.3B
WUSPruner7.2B
Qwen3.8-27BWanda20.3B
SliceGPT22.0B
WUSPruner21.5B
Qwen2.5-72BWanda54.8B
SliceGPT60.5B
WUSPruner55.5B

For Table 5, the original model is Qwen3-8B and the precision is bfloat16. The latency includes the prefill and 64 decoding steps and is not a per-token measurement. We provide the quality results at 50% compression ratio:

MethodRatioWikitext pplC4 pplPTB pplAvgPIQAHellaSwagARC-CMMLUOBQACMMLUAvg
SliceGPT50%24.0254.2224.1167.453.429.121.321.225.424.129.0
WUSPruner50%49.9201.0173.2141.355.131.321.522.926.225.430.4

To accompany the largest reported speedup, we have added quality results for Qwen3-8B at 50% pruning. WUSPruner achieves an average downstream accuracy of 30.4%, compared with 29.0% for SliceGPT. WUSPruner also achieves lower C4 and PTB perplexity, while SliceGPT performs better on WikiText-2. Nevertheless, both methods exhibit substantial quality degradation at this aggressive pruning level. We therefore present the 50% speed result as a reference point illustrating the efficiency–quality trade-off, rather than as a quality-preserving operating point. We will place the corresponding quality results alongside the efficiency measurements to make this limitation clear.

Q1: Do you agree that, without damping, WUS coordinate deletion is equivalent to activation-whitened truncated SVD at the same retained rank? What distinguishes the proposed method beyond its blockwise representation and projection sharing, and do existing ablations isolate these contributions?

Yes. Under the theorem’s assumptions, unrestricted WUS coordinate deletion without damping yields an activation-whitened truncated-SVD optimum at the same retained rank. Our contribution is the structured coordinate-pruning framework, rather than a different fixed-rank reconstruction optimum.

Activation-whitened truncated SVD formulates compression as a rank-constrained matrix approximation, typically implemented through two dense factors. WUSPruner formulates compression as a change of basis followed by matched coordinate deletion. Its theoretical analysis characterizes the interactions between deleted coordinates and constructs a balanced basis in which their local reconstruction costs become additive and spectrally ordered. Its practical framework then implements the retained coordinates through compact projections and reduced linear maps. These viewpoints coincide in the unrestricted local setting, but their practical representations can have different storage costs. As illustrated in our response to W1, reducing the transformation overhead can allow WUSPruner to retain more dimensions within the same total compression budget. Blockwise construction and projection sharing are therefore integral components that make the coordinate-pruning framework economical, rather than evidence of a better fixed-rank optimum. We will clarify this contribution and distinguish the theoretical equivalence from the practical compression trade-offs in the revised discussion.

AuthorsOpenReview ↗

Response to Reviewer ev54 [3/3]

Q2: Does “25% compression” mean a 25% reduction in total model parameters, including projections and the language-modeling head? Please provide final parameter counts and compressed-module lists for WUSPruner and the baselines.

25% means the reduction ratio for each module, which includes projections and the language-modeling head. We provide the detailed parameter count after compression in W2. The compressed modules include q_proj, o_proj for attention module, gate_proj, up_proj, down_proj for MLP module and lm_head. All methods follow the same setting.

Q3: Which model and numerical precision were used in Table 5, and does latency include prefill and all 64 decoding steps, or represent a normalized per-token measurement? Are existing model-quality results available for the reported 50% setting?

For Table 5, the original model is Qwen3-8B and the precision is bfloat16. The latency includes the prefill and 64 decoding steps and is not a per-token measurement. We provided the model-quality results in the W2.

AuthorsOpenReview ↗

Rebuttal Follow-up for Reviewer ev54

Dear Reviewer ev54,

Thank you again for your thoughtful review and time spent on our work. With the discussion deadline approaching, we would appreciate a brief response, if time permits, on whether our response and additional experiments address your concerns. If any questions remain, we would be happy to clarify.

Best,

The Authors

Reviewer_SwQxOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes WUSPruner, a structured pruning method for large language models. The key idea is to find a better representation basis before pruning, rather than directly removing dimensions in the original space. The paper constructs a WUS transformation using both activation and weight statistics, which provides a principled way to identify less important dimensions based on their spectral contributions. The paper further introduces the blockwise implementation for scalability. Experiments on Llama and Qwen models show improvements over existing pruning baselines in downstream performance, with reduced inference latency and memory usage.

Critical strengths
  1. The paper presents a clear problem formulation with strong motivation. The authors identify the importance of the representation basis in structured pruning and provide a theoretical proof showing that the proposed WUS transformation achieves the optimal local reconstruction error under the stated assumptions. This gives the method a solid mathematical foundation.

  2. The experimental results are strong. The method is evaluated on multiple LLM architectures and model scales, with consistent improvements over existing pruning baselines across various downstream tasks. The paper also demonstrates practical improvements in inference latency and memory consumption.

Critical weaknesses

The paper mainly compares the proposed method with baseline methods at a single pruning ratio. Although the authors evaluate different pruning ratios, they do not compare against baselines under these settings. It remains unclear whether WUSPruner maintains its advantages over baseline methods at other pruning ratios.

Questions for authors
  1. Since PTB, C4, and WikiText-2 are all evaluated using perplexity, could the authors explain why WUSPruner achieves substantial improvements on PTB and C4, while the gains on WikiText-2 are relatively limited or even negative in Table3 and Table 4? Are these differences related to the calibration data type or the model architecture? Could the authors provide additional results using different types of calibration data?

  2. Could the authors provide comparisons between WUSPruner and the baseline methods across different pruning ratios?

4 comments
AuthorsOpenReview ↗

Response to Reviewer SwQx [1/2]

Thank you very much for your constructive feedback. We address your concerns via response below:

W1: The paper mainly compares the proposed method with baseline methods at a single pruning ratio. Although the authors evaluate different pruning ratios, they do not compare against baselines under these settings. It remains unclear whether WUSPruner maintains its advantages over baseline methods at other pruning ratios.

We thank the reviewer for highlighting the need for baseline comparisons across pruning ratios. To address this concern, we have expanded our comparison with SliceGPT to six pruning ratios—15%, 20%, 25%, 30%, 35%, and 40%—on both Qwen3-8B and Qwen3.5-9B.

Qwen3-8B:

MethodRatioWikitext pplC4 pplPTB pplAvgPIQAHellaSwagARC-CMMLUOBQACMMLUAvg
SliceGPT15%11.027.433.624.072.766.650.155.838.450.155.6
WUSPruner15%12.020.919.317.474.164.750.664.839.664.159.6
SliceGPT20%11.733.246.630.569.461.443.746.838.040.249.9
WUSPruner20%13.525.222.820.571.259.443.560.237.656.354.7
SliceGPT25%13.143.867.841.667.655.535.331.633.428.642.0
WUSPruner25%15.033.728.625.767.853.739.254.334.647.549.5
SliceGPT30%15.259.199.257.863.449.230.023.134.425.237.5
WUSPruner30%17.544.341.234.363.047.833.147.730.636.843.2
SliceGPT35%17.776.8133.475.960.843.326.822.930.425.134.9
WUSPruner35%21.464.359.148.260.742.328.237.730.629.938.2
SliceGPT40%21.794.5183.199.859.034.922.522.927.625.232.0
WUSPruner40%27.291.487.468.759.137.824.430.228.825.734.3
AuthorsOpenReview ↗

Response to Reviewer SwQx [2/2]

Qwen3.5-9B:

MethodRatioWikitextC4PTBAvgPIQAHellaSwagARC-CMMLUOBQACMMLUAvg
SliceGPT15%9.622.320.817.572.568.548.648.939.832.751.8
WUSPruner15%9.815.215.313.475.968.851.570.640.068.962.6
SliceGPT20%10.427.126.921.469.362.444.836.636.429.546.5
WUSPruner20%10.817.617.215.273.863.346.365.037.861.157.8
SliceGPT25%11.233.736.727.266.255.337.028.534.225.341.0
WUSPruner25%12.421.820.718.369.056.039.054.534.453.851.1
SliceGPT30%12.642.750.335.261.847.330.623.331.025.036.5
WUSPruner30%14.928.827.223.664.448.131.747.532.444.644.8
SliceGPT35%14.655.969.446.658.641.026.123.028.025.333.7
WUSPruner35%18.439.739.032.361.342.327.633.929.035.738.3
SliceGPT40%17.271.892.660.557.736.624.423.126.625.132.2
WUSPruner40%23.556.355.144.958.736.823.726.926.028.233.4

As shown in the results, WUSPruner achieves lower average perplexity across the three language modeling datasets and higher average accuracy across the six downstream tasks in all 12 model–ratio settings. These improvements hold across the entire evaluated range, including the more aggressive 35% and 40% pruning settings. At the individual benchmark level, WUSPruner consistently improves C4 and PTB perplexity, as well as MMLU and CMMLU accuracy, at every evaluated ratio on both models.

Q1: Since PTB, C4, and WikiText-2 are all evaluated using perplexity, could the authors explain why WUSPruner achieves substantial improvements on PTB and C4, while the gains on WikiText-2 are relatively limited or even negative in Table3 and Table 4? Are these differences related to the calibration data type or the model architecture? Could the authors provide additional results using different types of calibration data?

In our all experiments, we found that SliceGPT tends to overfit to the calibration data. To investigate this explanation, we additionally evaluated both methods on Qwen3-8B at 25% pruning using FineWeb-Edu for calibration:

MethodWikitext pplC4 pplPTB pplAvgPIQAHellaSwagARC-CMMLUOBQACMMLUAvg
Slicegpt 25% Fineweb-Edu35.327.939.134.169.755.941.443.136.637.647.4
WUSPruner 25% Fineweb-Edu26.926.932.528.768.656.140.356.537.250.951.6

With FineWeb-Edu calibration, WUSPruner achieves lower perplexity on all three datasets, including WikiText-2, and improves average accuracy across the six downstream tasks by 4.2 points.

Q2: Could the authors provide comparisons between WUSPruner and the baseline methods across different pruning ratios?

Thank you for your question. We provide the comparison with different pruning ratios in W1.

Thank you for the detailed responses and additional experiments. I am pleased that my concerns have been addressed.

Considering the methodological contribution, novelty, and experimental results, I will maintain my original rating.

Dear Reviewer SwQx,

Thank you for taking the time to review our responses and additional experiments. We are glad they addressed your concerns. Your questions helped us clarify our contribution and strengthen the experimental evaluation, and we appreciate your thoughtful feedback.

Best,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their careful reading and constructive feedback. Their comments have helped us clarify the scope of our contribution and strengthen the empirical evaluation. The revised manuscript includes the following additions and clarifications (the modified content is marked in Red):

  1. Broader baseline comparisons. We added comparisons with LLM-Pruner, FLAP, and ZipLM on Qwen3-8B at 25% compression (Appendix B.8, Table 14). WUSPruner achieves the lowest average perplexity and highest average downstream accuracy among the evaluated methods. We also expanded the SVD-LLM comparison to Qwen3-8B at 35% compression and Qwen3.8-27B at 25% compression, observing improvements in both aggregate metrics in these additional settings (Appendix B.2, Table 8).

  2. Comparisons across compression ratios. We added comparisons with SliceGPT at six nominal pruning ratios from 15% to 40% on Qwen3-8B and Qwen3.5-9B (Appendix B.3, Table 9). WUSPruner achieves lower average perplexity and higher average downstream accuracy in all 12 model–ratio settings.

  3. Component ablations and sensitivity analysis. We added activation-only and weight-only ablations to examine the contribution of jointly using activation and weight second moments (Appendix B.9, Table 15). We also evaluated damping coefficients from 0.001 to 0.05, finding relatively stable aggregate performance within the tested range. The default coefficient was not tuned for the reported experiments (Appendix B.7, Table 13).

  4. Alternative calibration data. We added FineWeb-Edu calibration experiments on Qwen3-8B at 25% compression (Appendix B.5, Table 11). WUSPruner retains its aggregate advantage over SliceGPT under this alternative calibration distribution, achieving lower perplexity on all three language-modeling datasets and higher average downstream accuracy.

  5. Additional implementation and computational details. We specified the default block width and the modules compressed by WUSPruner, and reported final model sizes and FLOPs per token across five models (Section 4.1 and Appendix B.6, Table 12). These results complement the existing latency, GPU memory, and compression-runtime measurements and clarify the computational costs associated with nominal pruning ratios.

We provide detailed responses to each reviewer’s concerns below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.