acceptodds
Under review as a conference paper at ICLR 2027

Prune, Update and Trim: Robust Structured Pruning for Large Language Models

Abstract

Large Language Models (LLMs) have experienced significant growth and development in recent years. However, performing inference on LLMs remains costly, especially for long-context inference or in resource-constrained devices. This motivates the development of new post-training pruning (PTP) methods. These methods reduce LLMs’ requirements by removing a substantial part of the model’s parameters. The discarded weights are selected depending on their impact on the models performance. Current PTP methods prune the models by removing the less informative hidden nodes from the FFN layers, and the least important attention layers. We propose Putri, a PTP method that introduces three changes to the State-of-the-art. First, we update the unpruned weights of the FFN to compensate for the introduced pruning error. Second, the FFN layers are pruned sequentially, taking into account the updates done to the previous layers. Third, instead of removing full attention layers, we remove individual attention-heads. We extend this method such that it can also address Grouped-Query Attention. In summary, Putri is a structure pruning method which remains simple while showing SOTA performance. Pruning experiments on multiple models with a wide variety of sparsity ranges and on different datasets, validate the generality of Putri.

Primary area:
optimization
Keywords:
Pruning, LLM.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8VDjOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a structured pruning method called Putri that improves on prior work in three ways: updating unpruned FFN weights to compensate for pruning error, pruning FFN layers sequentially while accounting for updates to previous layers, and removing individual attention heads instead of full attention layers (extended to Grouped-Query Attention). Experiments show that Putri remains simple while achieving SOTA performance and can prune LLMs at extreme sparsity ratios.

Critical strengths
  1. This paper is well-written and easy to follow.

  2. The experiments are comprehensive and the improvements are significant.

Critical weaknesses
  1. This paper is somewhat incremental. Regarding compensation for pruning error, there is a substantial body of literature [1,2,3]. In particular, [3] also uses the objective of minimizing L2-loss for error compensation. However, I do not see any discussion of the differences from these works in the main text of this manuscript.

  2. The baselines are weak. This paper does not compare against state-of-the-art structured pruning methods, including Olica [3], SlimLLM [4], SlimGPT [5], and LoRAP [6].

  3. Reproducibility. I do not see a section that provides implementation details, which substantially hurts the reproducibility of this paper.

References

[1] SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot. ICML 2023.

[2] Fluctuation-based Adaptive Structured Pruning for Large Language Models. AAAI 2024.

[3] Olica: Efficient Structured Pruning of Large Language Models without Retraining. ICML 2025.

[4] SlimLLM: Accurate Structured Pruning for Large Language Models. ICML 2025.

[5] SlimGPT: Layer-wise Structured Pruning for Large Language Models. NeurIPS 2024.

[6] LoRAP: Transformer Sub-Layers Deserve Differentiated Structured Compression for Large Language Models. ICML 2024.

Questions for authors

Please see the weaknesses.

2 comments
AuthorsOpenReview ↗

Reply to Reviewer 9VDj (1/1)

We thank the reviewer for the careful review and the constructive comments. We address each point below.

W1: Novelty and relation to compensation methods. We do not claim LS compensation as our invention. Our contributions are:

  1. Showing that LS is the missing piece of 2SSP-style structured pipelines. Removing it ("Putri (no FFN upd.)", Figs. 2–3) has the largest single effect, especially at high sparsity.
  2. GQA-compatible kv-group pruning, which also reduces the KV-cache.
  3. Sequential propagation in a structured pipeline. This is standard in SparseGPT-style pipelines, but absent from 2SSP and Olica.
  4. More graceful degradation at high sparsity. At 75% sparsity, Putri has the lowest perplexity on every model and dataset, 1.1–5× lower than 2SSP. At 90–95%, it is the best method on every model except Qwen3-0.6B at 95%, while several baselines fail (Table 2 and Appendix).

W2: Baselines. Our baselines were chosen as the strongest post-training structured methods that support GQA models out of the box: 2SSP (TMLR 2025), EvoPress (ICML 2025), BlockPruner, ShortGPT and UIDL. Olica, SlimLLM, SlimGPT and LoRAP report perplexity only on MHA models (LLaMA-1/2 and Vicuna, plus Baichuan-7B for SlimGPT). The only GQA result among them is SlimGPT on Mistral-7B, evaluated on LongBench after LoRA fine-tuning, and none of the four papers describes how it handles GQA. Olica and SlimLLM have public implementations, both for LLaMA models with MHA. We found no public repository for SlimGPT and LoRAP. This is why we did not include them in our experiments on Llama-3 and Qwen3.

Their reported results on Llama-2-7B without fine-tuning are shown below, next to 2SSP, our strongest baseline, on the same model. WikiText-2 perplexity:

MethodSource (eval. length)Dense20%25%30%40%50%
SlimLLMSlimLLM, Table 2 (128)12.1815.70–––38.64
LoRAPLoRAP, Table 11 (128)12.1915.02–18.5830.9460.89
SlimGPTSlimGPT, Table 9 (128)12.1916.49––––
OlicaOlica, Tables 3 and 7 (128)12.19––18.54–
2SSP2SSP, Table 1 (2048)5.47–9.25––31.40
Putriours (2048)5.50–11.77––65.28

All four methods follow the LLM-Pruner protocol: 128-token evaluation sequences and a sparsity defined over all parameters, including the embeddings. 2SSP and our paper use 2048-token sequences and a sparsity defined over the transformer blocks. As a result, the dense perplexities differ (12.19 vs. 5.47), and the absolute values cannot be compared directly. Under our protocol, Putri is behind 2SSP on this MHA model. The results of all our baselines on Llama-2-7B are in our answer to Reviewer c1oT, W1.3.

Part 2/2

W3: Reproducibility. The anonymous code (link in the abstract) includes scripts that reproduce all tables. We will add an "Implementation details" appendix with the following:

  • Calibration data. 256 sequences of 2048 tokens from the C4 train split (first shard), seed 0.
  • FFN stage. The score ∥zi∥2\|z_i\|^2 is accumulated over all calibration tokens. Z⊤ZZ^\top Z and Z⊤ZWZ^\top Z W are accumulated in fp32, and Eq. 2 is solved by Cholesky. If the Cholesky factorization fails, we add diagonal damping of 1% of the mean diagonal. Layers are processed from first to last, feeding the pruned outputs forward.
  • Attention stage. Perplexity on 1 calibration sequence of 2048 tokens, as in 2SSP. K kv-groups are removed per iteration (K = kv-heads per layer). The head budget follows Eq. 5 with α = 1.5.
  • Evaluation. Perplexity uses non-overlapping 2048-token windows on Wikitext-2 (test), C4 (validation, first 1,100 documents) and FineWeb-Edu (sample-10BT, 500 documents). Zero-shot uses LM-Eval-Harness with 0 shots and the accuracy (acc) metric.
  • Baselines. Implementations from the official 2SSP repository with default hyperparameters (2SSP: 32 calibration sequences).
  • Hardware and cost. A single GPU per experiment (RTX 3090, A40 or A100-80GB, depending on model size).

Many thanks for your efforts devoted to addressing my concerns. After reading your responses, I still have several concerns. First, if LS compensation is not your contribution, please indicate your positioning clearly in Section 3.1 of your paper. Second, why not compare these baselines under the same evaluation settings? In addition, public repository: LoRAP (https://github.com/lihuang258/LoRAP). Third, I cannot see the revised content of your latest manuscript. Therefore, I keep my rating as 2.

Reviewer_EdUFOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

Putri proposes a structured pruning pipeline combining activation-based FFN channel selection, least-squares weight reconstruction, sequential layer processing, and GQA group pruning. Experiments cover multiple Llama and Qwen3 models, evaluating perplexity, zero-shot accuracy, and memory requirements.

Critical strengths
  1. The method is easy to understand. The formulas and pruning steps are clearly explained, making the approach easy to follow.

  2. The experiments are extensive. The authors test multiple model families, sizes, datasets, and sparsity levels. This supports the reported perplexity gains across different settings.

Critical weaknesses
  1. MLP channel pruning with output reconstruction has already been explored in methods such as SlimLLM and ZipLM. The authors should clarify how their method differs from OBS-based structured pruning, particularly in channel selection and weight compensation.

  2. The authors already cite EvoPress in their discussion of perplexity-guided attention-layer evaluation. Applying this criterion to GQA groups appears to be an incremental extension of an existing importance criterion to an established pruning unit.

Questions for authors
  1. Why use activation-only FFN importance? Equation (1) ignores the corresponding down-projection weights. Weight-aware structured criteria have been explored in FLAP and Probe Pruning. Could the authors justify this choice and compare these scoring rules?

  2. Why separate channel selection from compensation? Equation (2) minimizes local reconstruction error, but channel selection does not account for the subsequent compensation. How does this approach compare with the compensation-aware, OBS-based structured criterion in ZipLM?

  3. Attention groups are selected using perplexity, whereas FFN channels use a local activation score. Why use different objectives for FFN and attention pruning?

[1] Jialong Guo, Xinghao Chen, Yehui Tang, and Yunhe Wang. SlimLLM: Accurate Structured Pruning for Large Language Models. ICML, 2025.

[2] Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Fluctuation-Based Adaptive Structured Pruning for Large Language Models. AAAI, 2024.

[3] Qi Le et al. Probe Pruning: Accelerating LLMs through Dynamic Pruning via Model-Probing. ICLR, 2025.

[4] Eldar Kurtić, Elias Frantar, and Dan Alistarh. ZipLM: Inference-Aware Structured Pruning of Language Models. NeurIPS, 2023.

2 comments
AuthorsOpenReview ↗

Reply to Reviewer EdUF (1/2)

We thank the reviewer for the careful review and the constructive comments. We address each point below.

W1: Relation to SlimLLM, ZipLM and OBS-based structured pruning. We agree that least-squares (LS) compensation of the output projection is not new. For the layer-wise quadratic loss, the LS solution of Eq. 2 is the same as the OBS update after removing the same channels. Putri differs from these methods in the following points:

  • Channel selection. Putri selects channels by their activation energy ∥zi∥2\|z_i\|^2 in one shot, and then solves Eq. 2 once per layer. ZipLM uses a compensation-aware OBS saliency, removes structures greedily one at a time, and updates the inverse Hessian after each removal.
  • Compensation. SlimLLM does not solve a full LS update of the output projection. It fits one scale and one offset per output dimension (a linear regression on each output feature). Putri re-fits all remaining weights of the down-projection (Eq. 2).
  • Attention. Putri removes whole GQA kv-groups, selected with a global perplexity criterion across layers. SlimLLM (Pearson correlation) and ZipLM (OBS) score heads within each layer, and were evaluated only on MHA models: LLaMA-1/2 and Vicuna for SlimLLM, BERT and GPT-2 for ZipLM.

We also compared their published results on WikiText-2 perplexity with our results:

Method (eval. length)Dense25%37.5%50%
--Llama-3.1-8B
ZipLM [1] (4096)5.8452.69125.98366.34
FLAP [1] (4096)5.8412.3021.54134.28
Putri [ours] (2048)6.4111.8117.5039.25
--Llama-3-8B
ZipLM [1] (4096)5.7628.7450.36102.76
FLAP [1] (4096)5.7612.6522.6182.12
Putri [ours] (2048)6.3112.3816.8832.00
  • Even with a longer evaluation context, which lowers perplexity (dense 5.84 vs. 6.41), ZipLM and FLAP reach higher perplexity than Putri at every sparsity.
  • The gap grows with sparsity: at 50%, ZipLM and FLAP are 2.6–9.3× worse.

[1] Li et al. (2025), "Týr-the-Pruner".

SlimLLM only reports LLaMA-1/2 and Vicuna results, evaluated with 128-token sequences and a sparsity over all parameters. WikiText-2 perplexity on Llama-2-7B, without fine-tuning:

Method (eval. length)Dense20%25%50%
SlimLLM [2] (128)12.1815.7038.64
2SSP [3] (2048)5.47–9.2531.40
Putri [ours] (2048)5.50–11.7765.28

[2] Guo et al. (2025), "SlimLLM".
[3] Sandri et al. (2025), "2SSP".

The two protocols give different dense perplexities (12.18 vs. 5.47), so values can only be compared within the same protocol.

Our runs on Llama-2-7B, with the same protocol as in the paper (C4 calibration, 2048-token evaluation). WikiText-2 perplexity:

Method25%50%
UIDL25.752800.00
ShortGPT12.38278.00
BlockPruner12.9484.50
EvoPress11.6356.25
2SSP9.6334.75
Putri [ours]11.7765.28

On this MHA model, Putri is behind 2SSP at both sparsities (see our answer to Reviewer c1oT, W1.3).

We do not consider Probe Pruning (Le et al., 2025) directly comparable. It prunes dynamically for each batch at inference time, using a probe, and keeps the full model in memory (as stated in their Table 1). On Llama-3-8B, it prunes only the MLP layers. This is a different setting from static post-training pruning, which reduces the stored model.

W2: Incremental extension of EvoPress. EvoPress uses a different approach: an evolutionary search with a KL-divergence fitness over whole attention and MLP sub-blocks. Putri uses perplexity instead. On the attention side, our contribution is not the criterion, but:

  • The pruning unit. Putri removes whole kv-groups. This keeps the GQA group size uniform within each layer and shrinks the KV-cache (see our answer to Reviewer c1oT, Q).
  • The budget. Eq. 5 converts the 2SSP attention-layer budget into a kv-head budget.

The ablation "Putri (full Attn.)" (Figs. 2–3) shows the gain from this finer granularity.

Q1: Activation-only FFN score. The down-projection weights are not ignored by the method: they define the target of the update, Y=ZWY = ZW. They are only left out of the selection score.

Q2: Separate selection and compensation. In early experiments, we changed the selection score and kept the LS update fixed. A compensation-aware OBS-style score with the same LS update was clearly worse than ∥zi∥2\|z_i\|^2 (Wiki perplexity; preliminary variant with full-attention-layer removal):

ModelScore25%37.5%50%
Qwen3-4B∥zi∥2\|z_i\|^219.030.082.3
Qwen3-4BOBS-style175.5219.4305.8
Llama-3.1-8B∥zi∥2\|z_i\|^212.418.629.3
Llama-3.1-8BOBS-style643K532K4928

For the comparison with the full ZipLM pipeline, see the table in W1.

AuthorsOpenReview ↗

Reply to Reviewer EdUF (2/2)

Q3: Different objectives for FFN and attention. Perplexity is a better criterion than the local layer-wise L2 error, because it measures the effect on the model output. We use it wherever it is affordable. The difference is the number of candidates. Qwen3-8B has 36 layers × 12,288 FFN channels/layer = 442k FFN channels, but only 36 layers × 8 kv-groups/layer = 288 kv-groups. Evaluating perplexity for each FFN channel is infeasible, while for each kv-group it is feasible.

Reviewer_c1oTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new structural pruning method for LLMs, named Putri. More specifically, this pruning method consists of two stages. The first stage focuses on the uniform pruning of FFN layers which removes the hidden nodes with smaller magnitudes of outputs in each layer progressively. In this stage, the weights of the pruned FFN layer will be updated via minimizing the reconstructed L2 loss of the layer’s outputs. So the pruning of the next FFN layer is conducted based on the pruned former FFN layers with updated weights. The second stage is non-uniform head pruning, which removes the heads that has less impact on the model’s perplexity on calibration dataset. This stage is compatible with both the architecture of multi-head attention (MHA) and grouped-query attention (GQA). Experimental results on diverse settings show that Putri can achieve better or comparable performance than the evaluated baselines.

Critical strengths

The critical strengths of this paper are listed as follows.

  1. The authors provided open-source codes for their methods, making this work easy to reproduce for the future readers.
  2. The design of the two-stage method is reasonable. Since it is a common practice to update the weights after pruning, updating the weights of previous layers first is sensible before deciding the pruning strategy for each layer. Moreover, the popularity of GQA in SOTA LLMs makes it necessary for the head pruning to be compatible with this architecture.
  3. The authors consider a wide range of pruning ratios from moderate value (25%) to large values (75%) when designing their experiments.
Critical weaknesses

The critical weaknesses of this paper are listed as follows.

  1. The design of experiments needs further improvement. Firstly, the authors didn’t compare their method with some representative structural pruning work such as LLM Pruner and SliceGPT. It would be better to have the baselines of semi-structured pruning methods such as the 2: 4 variants of SparseGPT and Wanda. Secondly, the authors should also evaluate model families with MHA to show that Putri also works well on this basic architecture.
  2. The experimental results are not strong enough. Although Putri is consistently better for Qwen3 models in terms of perplexity, its superiority on Llama models is not very stable. Besides, on the zero-shot settings, the performance improvement from Putri is marginal, where most of the improvement comes from MMLU.
  3. Compared with unstructured pruning, the benefit of structured pruning is its compatibility with the hardware, which means the improvement on the memory usage and inference speed. However, except for the peak VRAM of Qwen3-8B reported in the Appendix, the authors didn’t provide any metrics on inference speed to show the hardware benefits brought by Putri.
Questions for authors

You highlighted that the existing head pruning methods can only be applied to MHA instead of GQA. Could you elaborate more on this? What is the reason that existing head pruning methods cannot be applied to GQA? Is it just because when these pruning works were proposed, GQA hadn’t been widely used in the SOTA LLMs, and the authors just omit GQA? What are the challenges to extend the existing head pruning methods on MHA to GQA?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer c1oT (1/2)

We thank the reviewer for the careful review and the constructive comments. We address each point below.

W1.1: LLM-Pruner and SliceGPT. We agree these are reference structured baselines. Results for both methods on the Llama models we use have already been published, and we summarize them below. Each row also gives the dense perplexity from its own source, because the evaluation lengths differ.

WikiText-2 perplexity of the reported results, compared with our results:

Method (eval. length)Dense25%37.5%50%
--Llama-3.1-8B
LLM-Pruner [1] (4096)5.8419.3570.93288.3
SliceGPT [1] (4096)5.8455.43237.5353.2
2SSP [our run] (2048)6.4112.0020.3834.75
Putri [ours] (2048)6.4111.8117.5039.25
--Llama-3-8B
LLM-Pruner [1] (4096)5.7618.6587.93302.2
SliceGPT [1] (4096)5.7647.73176.8384.0
2SSP [our run] (2048)6.3111.8121.3834.75
Putri [ours] (2048)6.3112.3816.8832.00
--Llama-2-7B
SliceGPT [2] (2048)5.4714.8230.2857.66
2SSP [2] (2048)5.479.2514.6431.40
  • On Llama-3-8B and Llama-3.1-8B, even with a longer evaluation context, which lowers perplexity (dense 5.84 vs. 6.41), LLM-Pruner and SliceGPT reach 1.5× to more than 10× higher perplexity than Putri at the same sparsity.
  • On Llama-2-7B (MHA), under the same protocol as ours, Putri is better than SliceGPT at 25% but worse at 50%, and 2SSP is the best method. We discuss this model in W1.3.

[1] Baselines re-run by Li et al. (2025), "Týr-the-Pruner", without fine-tuning.
[2] Sandri et al. (2025), "2SSP".

W1.2: 2:4 semi-structured baselines. We see them as complementary rather than directly comparable, for three reasons:

  • Fixed sparsity. 2:4 is defined only at 50% sparsity. It cannot reach the 75–99% range, where Putri gives the largest gains.
  • Memory. 2:4 needs a compressed format with index metadata. In fp16, each kept value needs 16 + 2 bits, so the memory reduction is at most about 44%. Structured pruning gives smaller dense matrices, so the memory reduction follows the sparsity.
  • Hardware. The speed-up needs sparse tensor-core kernels (Ampere or newer). The smaller dense matrices from structured pruning run faster on any hardware.

W1.3: MHA models. Putri works on MHA without any change. MHA is the special case where each kv-group has a single query head: Eq. 3 removes one head, and Eq. 5 uses K = number of heads. We ran all methods on the MHA model Llama-2-7B, with the same protocol as in the paper (C4 calibration, 2048-token evaluation). WikiText-2 perplexity (dense: 5.50):

Method25%50%
UIDL25.752800.00
ShortGPT12.38278.00
BlockPruner12.9484.50
EvoPress11.6356.25
2SSP9.6334.75
Putri [ours]11.7765.28

On Llama-2-7B, Putri is better than UIDL, ShortGPT and BlockPruner. It is close to EvoPress at 25% (11.77 vs. 11.63), but behind EvoPress at 50% and behind 2SSP at both sparsities. Putri runs on MHA without changes, but its advantage on the GQA models does not carry over to Llama-2-7B at 25–50% sparsity. We will add these results to the appendix and discuss them as a limitation. For the comparison with SlimGPT, SlimLLM, LoRAP and Olica, see our answer to Reviewer 8VDj, W2.

W2: Strength of the results.

Perplexity. For 25–75% sparsity, Putri is the best method in 108 of 120 (model, sparsity, dataset) entries (Tables 1, 2 and Appendix). This covers all 72 Qwen3 entries and 36 of 48 Llama entries. All 12 losses are on Llama at ≤50% sparsity (Llama-3-8B 25%/50%, Llama-3.1-8B 50%, Llama-3.2-3B 25%/37.5%). At 75% sparsity, Putri is the best method on every model and dataset. At 90–99% sparsity, it is the best method on every entry where at least one method gives a finite perplexity, with one exception: Llama-3-8B at 99% on Wiki.

Zero-shot. We agree that the gains on the average are small. However, the average gives the same weight to tasks of very different sizes. MMLU (about 14k questions) and HellaSwag (about 10k) are much larger than WinoGrande, ARC-C, ARC-E and PiQA (1.2k–2.4k each). On these small tasks, one standard error is about 1–1.4 points, so differences of 1–2 points are within the noise. On the two large tasks, Putri is the best method at 37.5% and 50% sparsity, while 2SSP is slightly better at 25%. At 75% sparsity, all methods are close to random guessing.

W3: Inference speed. Setup: Qwen3-8B, bf16, HF transformers, batch size 1, 2048-token prompt, 32 generated tokens, single RTX 3090 GPU.

SparsityWeights (GB)Prefill tok/sDecode tok/s
0% (dense)16.4268214.9
25%12.9277815.1
37.5%11.2349616.2
50%9.4383816.8
75%6.0682619.2
90%3.91288022.6
AuthorsOpenReview ↗

Reply to Reviewer c1oT (2/2)

Q: Why existing head pruning does not apply to GQA. It is partly historical. Classic head pruning (Michel et al., 2019; Voita et al., 2019) and recent LLM methods (e.g., SlimLLM, SlimGPT, FLAP) were designed and evaluated on MHA models, and none of these papers describes how to prune heads in GQA. Some later code releases extend existing methods to GQA (e.g., the LLM-Pruner repository for Llama-3), but these extensions are not described or evaluated in a publication. However, it is not only an omission: GQA creates three concrete obstacles.

  1. Shared K/V. In GQA, one kv-head is shared by g query heads. Removing one query head only removes its columns of WqW_q and its rows of WoW_o. WkW_k, WvW_v and the KV-cache stay the same. For example, the Probe Pruning code keeps K and V unchanged on GQA models. This loses most of the memory benefit, which is the main reason to prune attention for long-context inference.
  2. Uniform group size. Standard GQA implementations (repeat_kv in HF transformers, FlashAttention) require every kv-head to serve the same number of query heads. Pruning query heads independently creates unequal groups. These need padding or custom kernels, which removes the speed-up.
  3. Scoring. Existing importance scores are defined per query head. To remove a kv-head, the whole group must be scored jointly. Summing per-head scores ignores the interactions inside the group, whose heads share the same keys and values.

Putri solves this by using the kv-group (one kv-head and its g query heads) as the pruning unit. It scores each group by its direct effect on perplexity (Eq. 3). This keeps the group size uniform within each layer and reduces the KV-cache. MHA is the special case g = 1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.