- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Residual stream pruning methods typically choose the dimensions by minimizing activation reconstruction error. They characterized the effect of a residual-stream perturbation through both its activation covariance and the local sensitivity of the model output and derived a tractable spectral upper bound that reduces subspace selection to an eigendecomposition of a sensitivity-weighted covariance matrix, retaining the efficiency and structural simplicity of rotation-based pruning methods.
Critical strengths
This article is well-written, with a clear description of the motivation (as shown in Figure 1), and the proposed method is simple and effective.
Critical weaknesses
- The author chose SliceGPT as the main baseline because it is the most similar method. However, did the author consider comparing with newer pruning methods?
Training-free pruning:
[1] Qianhao Yuan, Qingyu Zhang, Yanjiang Liu, Jiawei Chen, Yaojie Lu, Hongyu Lin, Jia Zheng, Xianpei Han, and Le Sun. ShortV: Efficient multimodal large language models by freezing visual tokens in ineffective layers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 329–339, 2025.
[2] Xinrui Chen, Hongxing Zhang, Fanyi Zeng, Yongxian Wei, Yizhi Wang, Xitong Ling, Guanghao Li, and Chun Yuan. Prune&Comp: Free lunch for layer-pruned LLMs via iterative pruning with magnitude compensation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 20316–20324, 2026.
Training-base pruning:
[3] Chen, Xiaodong, et al. "Streamlining redundant layers to compress large language models." International Conference on Learning Representations. Vol. 2025. 2025.
[4] Chen, Xinrui, et al. "A simple linear patch revives layer-pruned large language models." Advances in Neural Information Processing Systems 38 (2026): 154484-154509.
-
My main concern is that this work is an improvement for SliceGPT and it has relatively hard scalability. I hope the author can clarify whether the proposed method is applicable to quantitative methods such as YAQA, and whether it can improve the evaluation of other pruning methods?
-
The intermediate activation distribution of the Reasoning model is highly correlated with the input type. Did the authors consider the selection of the calibration set, and did they take into account the relevance between FineWeb/REDPAJAMA and the downstream tasks?
Questions for authors
see weakness
3 comments
The Table you submitted doesn't seem to include the comparison of Prune&Comp and the combination. Could you please check if you have provided the wrong table?
Thank you for your response, and I apologize for the confusion regarding the comparison and combination with Prune & Comp. The table I provided is correct, but the presentation could have been made clearer. The first row corresponds to Prune & Comp alone (8 layers removed) and the bottom two rows correspond to SliceGPT and our method without the combination. The intermediate rows correspond to Prune & Comp combined with either SliceGPT or our method, where fewer layers are removed and residual-stream pruning is applied to reach the same target parameter count of 6.28B. I hope this clarifies the table, and I am happy to answer any further questions or provide additional clarification.
We thank the reviewer for their time and helpful comments. Please find the answers to the raised concerns and questions below.
We thank the reviewer for bringing these layer pruning methods to our attention. We have updated our related works section to mention these. Below, we provide additional results both comparing residual-streaming pruning with one of the suggested layer pruning methods as well as an exploration into whether we may achieve greater performance at matched parameter counts through combining the two approaches.
For this experiment, we use the suggested Prune & Comp as the layer pruning method. Like residual-stream pruning, layer pruning incurs substantial performance degradation at larger pruning rates. However, as layer pruning leaves the size of the original layers intact, residual-stream pruning provides an alternative approach for reducing model size after layers have been removed. We conduct this experiment on Llama 3.1 8B Instruct with a fixed target parameter count of 6.28B parameters, equivalent to removing 8 transformer layers. For both layer pruning and residual-stream pruning, we use the same set of 1,024 FineWeb-edu samples as calibration. We vary both the number of layers removed as well as the residual-stream pruning method and present results below.
First, it can be seen that without combining the two approaches, our method achieves better WikiText performance but performs worse on common-sense reasoning. Combining our method with layer pruning consistently improves performance on both WikiText and common-sense reasoning relative to stand-alone residual-stream pruning. The best performance across all metrics is achieved when Prune & Comp removes 4 layers and residual-stream pruning is applied using our method. Finally, it can be seen that SliceGPT is not able to improve over layer pruning in this configuration. Even the best combination of SliceGPT with layer pruning results in both worse WikiText PPL and worse common-sense reasoning performance.
The above results provide an initial exploration into combining residual-stream pruning with other compression methods. These results show that layer pruning and residual-stream pruning are complementary and, when used together, can achieve greater results than when each is used in isolation.
We thank the reviewer for the opportunity to clarify the choice of calibration data. We agree with the reviewer that the choice of calibration data may significantly impact performance on downstream tasks. FineWeb-edu was chosen as the calibration dataset for our main experiments as it is a high quality dataset that has been heavily filtered and is composed entirely of text from web pages deemed to have high educational quality. We therefore anticipate that it will be relatively representative of both the language modeling and common-sense reasoning tasks we evaluate on.
Unlike FineWeb-edu, RedPajama is composed of general web text. We therefore choose RedPajama as an additional calibration set for evaluation due to its greater diversity. By additionally evaluating our method with RedPajama, we are able to evaluate whether our method continues to provide benefits as the calibration data distribution changes. Finally, by comparing the performance of models compressed with FineWeb-edu versus models compressed with RedPajama, we see that FineWeb-edu calibration results in greater retention of common-sense reasoning performance while achieving comparable language modeling performance.