acceptodds
Under review as a conference paper at ICLR 2027

Constrained Refinement of Layerwise Sparsity Control for LLM Pruning

Abstract

Post-training pruning of large language models (LLMs) requires deciding which weights to remove within each layer and how to distribute a fixed sparsity budget across layers. At high sparsity, different layerwise allocations under the same global budget can yield substantially different perplexity (PPL) after pruning. Many existing methods derive layerwise sparsity ratios from importance or sensitivity scores or from predefined allocation rules, providing useful starting points. However, these initial allocations may not account for the layerwise reconstruction response of the selected pruning backend at the assigned sparsity levels. This motivates Plug-and-Prune, a constrained method that refines a uniform allocation or one produced by an existing allocator. At the input allocation, it measures layerwise reconstruction responses under the selected backend and uses them in a regularized update of the keep ratios. The update penalizes deviations from the input allocation and enforces the global budget and per-layer bounds, while leaving within-layer mask construction to the backend. We evaluate Plug-and-Prune with WANDA and SparseGPT on LLaMA, Qwen3, and Phi-4 models, primarily at 70% unstructured sparsity, supplemented by sparsity-sweep and structured-pruning experiments. The results show that our method generally lowers PPL relative to the input allocation, with gains tending to be larger at higher sparsity. Overall, response-guided budget redistribution is a practical, backend-agnostic complement to existing inner-layer criteria, especially at high sparsity.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Pruning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_fQj9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Plug-and-Prune, which allows for adjusting the sparsity of each layer to improve the performance of the pruned model when the total pruning ratio is fixed. Specifically, the proposed method, based on the allocation of existing methods, measures the layer-by-layer reconstruction response generated by the selected pruning backend and updates the weight retention ratio accordingly using regularization. The update process penalizes deviations from the initial allocation while satisfying global budget constraints and upper and lower bounds for each layer's ratio, while the intra-layer pruning mask is still constructed by the original pruning backend. Extensive experiments have demonstrated that the proposed method can further improve the performance of pruning methods such as OWL, DLP, and AlphaPruning.

Critical strengths
  • The proposed algorithm is a plugin that can be applied to other pruning algorithms.
  • Extensive experiments have demonstrated the effectiveness of the proposed method.
Critical weaknesses
  • Why are most experiments conducted at a 70% pruning rate? It is recommended to conduct experiments within a pruning rate range of 50%-90%.
  • The cost of the proposed algorithm needs to be reported further.
  • The existing experimental models are relatively old. Should we conduct experiments on some of the latest GLM, Qwen and other models?
  • Could you analyze why some layers can have high sparsity while others have relatively low sparsity?
Questions for authors

The performance gains in Table 2 are very small; could this be due to randomness?

1 comment

Dear Reviewer fQj9,

With the discussion period nearly over, we are writing to gently follow up on our initial response to your review. We would be very grateful to know whether our explanations and the additional results have addressed your concerns, and if there are any remaining issues that we can help clarify.

Thank you again for your time and consideration.

Best regards,

The Authors

Reviewer_XagYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

At high sparsity levels, the same global budget when distributed differently across layers, can lead to significant variations in PPL. Existing methods (OWL, DLP, and AlphaPruning) determine layer-wise allocation based on importance scores or statistical metrics; but these approaches fail to account for the actual reconstruction error incurred by the chosen pruning backend at a given sparsity rate. The authors propose "Plug-and-Prune" (PnP) to address this. It takes an existing allocation as an input, measures the layer-wise reconstruction response of the pruning backend under that configuration, and then fine-tunes the retention ratios for each layer via constrained optimization with regularization. Experiments on LLaMA-1/2 (7B–30B), LLaMA-3-8B/70B, Qwen3-0.6B/4B and Phi-4, mainly at 70% unstructured sparsity, show PPL reductions in all 40 main pairs.

Critical strengths
  1. Modular formulation is clean. The method is cheap and simple.
  2. Consistent empirical gains at high sparsity.
Critical weaknesses
  1. Methodological novelty is limited. The core method is a single-point, layer-separable quadratic surrogate with a proximal term, solved by standard water-filling/KKT. Using measured reconstruction error to drive layerwise allocation is close in spirit to LSA (Yang et al., 2026), which the paper cites but does not compare against. In Table 3, BESA is worse than uniform WANDA on LLaMA-1-7B (168.58 vs. 88.86) and LLaMA-2-7B (78.80 vs. 76.45), even though BESA learns both masks and sparsity. This suggests BESA may be misconfigured. The authors note they used a "target-matching layer-wise configuration rather than untouched default settings." There should be other baseline added to compare.
  2. All results appear to come from a single calibration draw and a single run. PPL is reported only on WikiText-2. C4 PPL and harder downstream tasks (e.g., MMLU, GSM8K) are missing, and per-task zero-shot results are not shown. λ\lambda varies by input and model and this should be reported. Evaluation scope is quite limited.
Questions for authors
  1. On what data were λ\lambda, cc, and the bounds selected?
  2. What are the per-layer bounds d‾l,dˉl\underline d_l, \bar d_l in all experiments? How sensitive are the results to them and to λ\lambda?
  3. Can you confirm that the BESA and DSA configurations reproduce their published numbers in settings they originally report, before using them at 70%? Why does BESA underperform uniform WANDA?
  4. Can you provide C4 PPL, per-task zero-shot accuracy, and at least one harder benchmark (e.g., MMLU)?
1 comment

Dear Reviewer XagY,

With the discussion period nearly over, we are writing to gently follow up on our initial response to your review. We would be very grateful to know whether our explanations and the additional results have addressed your concerns, and if there are any remaining issues that we can help clarify.

Thank you again for your time and consideration.

Best regards,

The Authors

Reviewer_pNnPOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a method for improving layerwise sparsity allocation in post-training LLM pruning. Starting from an allocation generated by an existing method, it measures the reconstruction error of each layer after pruning at its initial sparsity ratio. These errors are then used to redistribute sparsity across layers while maintaining the global sparsity budget and limiting deviations from the initial allocation. The refined allocation is subsequently applied using the original pruning backend.

Critical strengths
  1. The proposed method refines existing layerwise sparsity allocations without modifying within-layer pruning algorithms. Its compatibility with multiple allocation strategies and pruning backends makes it a practical extension to existing pruning pipelines.

  2. The method reduces perplexity across all 40 paired comparisons in Table 1, with particularly substantial gains from weaker initial allocations. The response-signal controls in Appendix F further support the effectiveness of using measured layerwise reconstruction errors.

Critical weaknesses
  1. The method measures reconstruction errors at the initial sparsity ratios and uses them to determine how much sparsity should be redistributed across layers. However, the paper does not examine how reconstruction errors actually change when the sparsity ratios are adjusted, leaving the proposed adjustment rule insufficiently justified. Comparisons with simpler redistribution strategies or measurements at multiple sparsity levels would help validate this design choice.

  2. Tables A5–A6 show substantial additional pruning time, while improvements over strong initial allocations such as DLP are often marginal. It remains unclear whether these gains justify the extra response-collection cost. A performance–cost comparison under matched computational budgets would better establish the method's practical value.

Questions for authors

My main concern is whether the quadratic formulation in Eq. (12), which uses reconstruction errors measured at the initial sparsity ratios as fixed coefficients, can accurately reflect how the actual reconstruction errors change as the sparsity ratios are adjusted (Weakness 1). I would expect the authors to measure reconstruction errors across different sparsity levels and compare the proposed formulation with simpler alternatives, such as linear or error-ranking-based redistribution rules. These experiments would provide direct evidence of whether the proposed formulation captures the actual sensitivity of each layer to sparsity changes and justifies the resulting redistribution decisions.

Reviewer_6a5JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Plug-and-Prune, a framework for refining layerwise sparsity allocations to improve pruning performance. It aims to address the limitation of existing methods that rely on predefined importance scores or statistical measures, which may not reflect the actual reconstruction response of a specific pruning backend. Specifically, Plug-and-Prune measures each layer's output reconstruction error under an initial sparsity allocation and uses the resulting response coefficients to formulate a constrained optimization problem. It then refines the layerwise keep ratios while preserving the global sparsity budget and per-layer bounds. Experiments are conducted with existing layerwise sparsity allocation methods across different LLM architectures and pruning settings.

Critical strengths
  1. Motivation is well justified. The paper identifies a practical limitation of existing layerwise sparsity allocation methods: predefined importance scores may not accurately reflect the reconstruction behavior of a specific pruning backend.
  2. The proposed method has good flexibility and compatibility with existing approaches. It can be integrated with different layerwise sparsity allocation methods to refine their allocations, as well as with different pruning methods without modifying their pruning procedures.
Critical weaknesses
  1. The theoretical analysis is insufficient. The proposed method assumes that reconstruction errors measured at the initial sparsity allocation can guide allocation at other sparsity levels, but this assumption is not sufficiently justified through theoretical analysis or validated experimentally. In addition, the authors use the activation RMS ratio as a proxy for the Jacobian gain without providing sufficient justification or empirical evidence.

  2. The experimental evaluation is limited. Most experiments focus on 70% sparsity, while the sparsity sweep from 50% to 85% is conducted primarily on LLaMA-2-7B. More systematic comparisons with existing layerwise sparsity allocation methods across different global sparsity levels and on more recent LLMs are needed. Furthermore, the paper does not report actual inference speedup results to demonstrate the practical benefits of the proposed sparsity allocation method.

Questions for authors
  1. Why can reconstruction errors measured at the initial sparsity allocation be used to guide allocation at other sparsity levels? Is there an approximately linear relationship between reconstruction error and sparsity? How large can the difference between the initial and final layerwise sparsity allocations be while the measured reconstruction errors remain reliable?

  2. Why is the activation RMS ratio a reasonable proxy for the Jacobian gain? Is there any specific theoretical or experimental evidence supporting this choice?

  3. Could the authors provide comparisons between the proposed method and existing layerwise sparsity allocation methods across different global sparsity levels on more recent LLMs, such as LLaMA 3?

  4. Given that unstructured sparsity is not readily exploitable for inference acceleration on commodity GPUs, I wonder what practical benefits further research into layerwise sparsity allocation can offer. The authors also include additional experiments on Mixed N:8 structured pruning. Are the resulting Mixed N:8 sparsity patterns supported by specific kernels? If so, could the authors specify which kernels support these patterns and provide actual inference speedup results?

4 comments

Dear Reviewer 6a5J,

With the discussion period nearly over, we are writing to gently follow up on our initial response to your review. We would be very grateful to know whether our explanations and the additional results have addressed your concerns, and if there are any remaining issues that we can help clarify.

Thank you again for your time and consideration.

Best regards,

The Authors

Dear Reviewer 6a5J,

Thank you again for taking the time to provide the helpful review.

As the discussion period draws to a close, we wanted to briefly follow up on our response to your four questions. In particular, we tested across five models whether initial response measurements can guide sparsity updates over the ranges those updates actually cover. We also compared the activation RMS proxy directly with JVP-based estimates of Jacobian gain.

We would appreciate hearing whether these results address your main concerns and would be happy to clarify any remaining questions.

Best regards,

The Authors

I appreciate the authors' efforts in conducting additional experiments. These results have addressed several of my technical concerns. However, my main concern regarding practical utility remains. Only a small fraction of projections can benefit from sparse kernels. The inference speedup is limited. Furthermore, can the packed sparse model be directly loaded for inference without reconstructing the dense FP16 weights?

I acknowledge the improvements in pruning performance, but I remain unconvinced of the broader contribution of this plug-and-play sparsity allocation framework. Beyond improving benchmark results, what meaningful value does this method bring to the model compression community, especially given its limited practical acceleration benefits?

Thank you for recognizing the improvements to pruning and for asking what they deliver in practice. Our response to your question is provided below.

Can the packed model be loaded directly for inference without reconstructing dense FP16 weights? The GPU path supports direct sparse loading for compatible projections after a one-time offline conversion. For the follow-up LLaMA-2-7B checkpoint at 62.5% sparsity, this converts the distribution file (5.77 GiB) into a 6.14 GiB inference checkpoint. At startup, 28 of 224 projections load directly in cuSPARSELt format. The loader reconstructs the other 196 in dense FP16. In four prefill checks and one cached decoding check, hybrid execution and dense FP16 execution predicted the same next token.

Beyond benchmark gains, what value does allocation refinement bring to model compression when inference speedups are limited? PnP lets a user keep their existing allocator and pruner. With the pruner and total weight budget fixed, PnP uses each layer’s measured reconstruction error to decide how many weights that layer should retain. For example, in Table 1, refining DLP’s allocation with WANDA lowered LLaMA-2-7B WikiText-2 perplexity from 24.36 to 22.14 at 70% unstructured sparsity.

Section 4.2 asks whether redistributing the same weight budget still lowers perplexity when each layer must satisfy an N:8 mask constraint. Following OWL (Yin et al., ICML 2024, Appendix B.1, Table 12), we hold the total number of retained weights fixed and compare perplexity under a uniform 3:8 allocation and a mixed N:8 allocation refined by PnP. To answer your question about practical utility, we also measured each allocation’s packed file size and GPU memory use after loading.

Model, 62.5% sparsityWikiText-2 PPL, fixed 3:8 → PnPPacked size, either allocationLoaded GPU memory, fixed 3:8 → PnP
LLaMA-1-7B36.33 → 19.505.77 GiB12.55 → 11.89 GiB (−5.26%)
LLaMA-2-7B38.28 → 20.415.77 GiB12.55 → 11.89 GiB (−5.26%)

In these WANDA experiments, sparse encoding made both allocations’ files 54.06% smaller than the original dense FP16 weights. At the same packed size, refinement lowered perplexity on both models. It also assigned 2:8 masks to four blocks per model, allowing their 28 projections to stay compressed on GPU. All fixed 3:8 projections required dense loading with this loader.

Measured inference speedups. In our earlier GPU test with 2,048-token prompts, hybrid prefill achieved speedups of 1.04× on LLaMA-1-7B and 1.10× on the LLaMA-2-7B checkpoint from Table 4, relative to dense FP16 execution of the same pruned weights. Our implementation uses cuSPARSELt for compatible projections. An open engineering question is how to support a wider range of Mixed N:8 patterns efficiently. In our follow-up DeepSparse test, the pruned models decoded 2.12× and 2.17× as fast as their respective unpruned FP32 models on CPU.

At the same weight budget and packed file size, PnP lowered perplexity on both models. The refined allocations also allowed our loader to keep more projections in sparse format, reducing GPU memory use. The measured GPU prefill speedups were modest with the available kernels. We will clarify this result in §4.2 and include the deployment measurements in the appendix. Thank you for prompting us to clarify how the allocation gains translate into practical benefits under the tested conditions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.