- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes trajectory-aware (TA) pruning and recovery, a framework for compressing LLMs under long reasoning scenarios. The idea is pretty straightforward: first score each layer based on how much removing it affects next-token predictions on fixed reasoning trajectories from the original model, then drop the lowest-scoring layers; after that, recover the pruned model via on-policy distillation from the teacher. Experimental results on Qwen3-4B and Qwen3-8B show that this layer selection beats Block Influence (BI) under the same recovery, especially at high pruning ratios.
Critical strengths
- The overall framing is clear and practical. And the main figure is clear and easy to follow.
- The proposed method is intuitive and effective. It beats BI on all benchmarks and pruning ratios under the same recovery setting.
Critical weaknesses
-
The claimed benefit to recovery is insufficiently validated. The paper argues that better layer selection provides a stronger starting point and reduces the burden on recovery, but the score measures only the damage at pruning time. Section 3.1.1 explicitly distinguishes this damage from whether the removed computation can be recovered through training. Since the paper reports no results before recovery, it remains unclear whether the improvement comes from a better starting point or from easier recovery. This is consequential because the recovery benefit is one of the paper’s core claims, and the current experiments do not support it.
-
Limited novelty and incomplete evaluation. Scoring layers by the change in loss or perplexity after removal has already been explored (e.g., Shortened LLaMA [1], SLEB [2]), but BI is the only directly evaluated pruning baseline. Without comparisons to existing loss-based methods, it is hard to determine the actual contribution of the trajectory-aware design. Some experimental results need further explanation. Several pruned-and-recovered models outperform the unpruned teacher on GSM8K, while the SFT baseline nearly collapses. Finally, although the paper emphasizes inference efficiency, it does not report actual inference speedups.
[1] Shortened LLaMA: A Simple Depth Pruning for Large Language Models, ICLR 2024 Workshop
[2] SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks, ICML 2024
Questions for authors
- How do TA and BI perform right after pruning, before recovery?
- How does TA compare with perplexity-based selection or SLEB under the same recovery?
- Why do the pruned models outperform the unpruned teacher on GSM8K, and what is the actual speedup?