acceptodds
Withdrawn from ICLR 2027

Prune Better Before Recovering Harder: Trajectory-Aware Pruning and Recovery for Reasoning Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reasoning language models are expensive to serve because they generate long autoregressive reasoning traces. Structured layer pruning can reduce this cost by shortening model depth, but criteria based on hidden-state similarity may remove layers that are important for reasoning. We propose trajectory-aware pruning and recovery, a preservation-first framework for compressing large language models. For each candidate layer, we temporarily replace it with an identity mapping and measure how much the model's next-token predictions change on fixed reasoning trajectories generated by the original model. We prune layers with the smallest behavioral effect, then use on-policy distillation to correct the remaining distribution shift on contexts visited by the pruned model. This preserves reasoning behavior before compression and reserves distillation for the errors that remain. Experiments on mathematical reasoning, code generation, and open-ended generation show that our method consistently outperforms SOTA method demonstrating that better layer selection reduces the burden on recovery.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
on-policy distillation, model compression, pruning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NtAxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes trajectory-aware (TA) pruning and recovery, a framework for compressing LLMs under long reasoning scenarios. The idea is pretty straightforward: first score each layer based on how much removing it affects next-token predictions on fixed reasoning trajectories from the original model, then drop the lowest-scoring layers; after that, recover the pruned model via on-policy distillation from the teacher. Experimental results on Qwen3-4B and Qwen3-8B show that this layer selection beats Block Influence (BI) under the same recovery, especially at high pruning ratios.

Critical strengths
  1. The overall framing is clear and practical. And the main figure is clear and easy to follow.
  2. The proposed method is intuitive and effective. It beats BI on all benchmarks and pruning ratios under the same recovery setting.
Critical weaknesses
  1. The claimed benefit to recovery is insufficiently validated. The paper argues that better layer selection provides a stronger starting point and reduces the burden on recovery, but the score measures only the damage at pruning time. Section 3.1.1 explicitly distinguishes this damage from whether the removed computation can be recovered through training. Since the paper reports no results before recovery, it remains unclear whether the improvement comes from a better starting point or from easier recovery. This is consequential because the recovery benefit is one of the paper’s core claims, and the current experiments do not support it.

  2. Limited novelty and incomplete evaluation. Scoring layers by the change in loss or perplexity after removal has already been explored (e.g., Shortened LLaMA [1], SLEB [2]), but BI is the only directly evaluated pruning baseline. Without comparisons to existing loss-based methods, it is hard to determine the actual contribution of the trajectory-aware design. Some experimental results need further explanation. Several pruned-and-recovered models outperform the unpruned teacher on GSM8K, while the SFT baseline nearly collapses. Finally, although the paper emphasizes inference efficiency, it does not report actual inference speedups.

[1] Shortened LLaMA: A Simple Depth Pruning for Large Language Models, ICLR 2024 Workshop

[2] SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks, ICML 2024

Questions for authors
  1. How do TA and BI perform right after pruning, before recovery?
  2. How does TA compare with perplexity-based selection or SLEB under the same recovery?
  3. Why do the pruned models outperform the unpruned teacher on GSM8K, and what is the actual speedup?
Reviewer_6UeqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies which layers to prune for a reasoning LLM. The method skips one layer at a time by scoring each layer on how much the next-token predictions change: the KL to the full model and the NLL increase on the reasoning and answer tokens. It then recovers with on-policy distillation from the full model. With the same distillation, it beats Block Influence on Qwen3-4B-Instruct (68.35 vs. 56.85 overall at 25% pruning) and Qwen3-8B (58.60 vs. 41.00).

Critical strengths
  1. The paper separates two problems: which layers to remove, and how to recover afterwards. The selection criterion alone makes a large difference.
  2. The gain over Block Influence is large, especially at 50% pruning (30.63 vs. 7.19).
  3. The qualitative analysis is interesting and suggest a novel selection from BI: TA keeps most middle layers and removes some early layers, while BI removes a block of late layers.
Critical weaknesses
  1. Block Influence is the only baseline. The closest baseline is missing: Shortened LLaMA (Kim et al., 2024) also removes one block at a time and ranks blocks by perplexity on calibration data, and the paper cites it. Your method is very close to this, using KL/NLL on teacher-generated reasoning trajectories. Without this comparison under the same OPD, I cannot tell whether the gain comes from the criterion or just from using reasoning trajectories as calibration data.
  2. The reference numbers look wrong. For example, the pruned 4B model beats the unpruned model on GSM8K (87.33 vs. 84.31). This points to a problem with the answer parser or the decoding setup.
  3. The training set uses “12,500 examples from MATH”. 12,500 is the size of the full MATH dataset (7,500 train + 5,000 test), and MATH-500 is taken from the MATH test split.
  4. Contribution 3 says TA gives “a substantially stronger pruned initialization”, but the tables never report TA or BI before recovery. Please report re-recovery results. This would support the story better.
  5. BI + SFT is also missing, so we cannot tell whether SFT fails for BI as well.
  6. Please explain the SFT collapse: TA + SFT gets 1.66% on GSM8K and 3.80% on MATH-500, far below even the pruned model with ShortOPD.
Questions for authors

See weaknesses.

Reviewer_SoYsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors introduce a trajectory-aware method for selecting entire transformer blocks to be removed. The method derives a per-layer sensitivity score by removing layers (replacing them with identity) one at a time and computing the difference compared to the dense model’s trajectory. They follow this by a round of retraining using on-policy distillation and obtain good results compared to prior layer pruning procedures.

Critical strengths

1/ Structured pruning is a very interesting and impactful area of research to reduce inference costs. 2/ Efficient reasoning models is an important topic since they incur extra overhead due to their long decode sequences. 3/ The experimental setup is reasonable and the results seem quite strong.

Critical weaknesses

1/ The paper is hard to follow and the introduced metric for layer selection is not well motivated. Can the authors discuss more why they opted for each selection and why they needed two different terms. In particular, is there a universal way to select hyperparameters for best selection?

2/ It’s not clear whether there is a recovery round after each BI selection. There’s also no reported runtime for both methods, which is very important to highlight for a fair comparison.

3/ The ablations seem confusing. For example, Table 4 seems to suggest that the weighting is not very important and changes the results only marginally. Table 3 similarly suggests that the NLL scoring and KL scoring are similarly not crucial to the final method. Could the authors please clarify which components of the method are most important for outperformance of shortOPD?

Questions for authors

1/ The authors seem to ignore a lot of past related work on structured pruning for other types of models which should be cited and/or compared to e.g. OSSCAR [1], [2], and Sheared Llama [3]. There are also several comparisons which should be included. In particular Gromov et. al [2] also have a layer-pruning method for LLMs which should be compared to, under the same post-pruning recovery budget. 2/ See weaknesses for other questions.

[1] Xiang Meng, Shibal Ibrahim, Kayhan Behdin, Hussein Hazimeh, Natalia Ponomareva, and Rahul Mazumder. OSSCAR: One-Shot Structured Pruning in Vision and Language Models with Combinatorial Optimization. ICML, 2024. [2] Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning. ICLR, 2024. [3] Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A. Roberts. The Unreasonable Ineffectiveness of the Deeper Layers. ICLR, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.