- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
"Hard to remove, easy to replace" examines how damage to accuracy from layer removal can be repaired by linear or nonlinear surrogates and how such surrogates can recover accuracy transfer. The authors also consider removing pairs of layers and show that the surrogate is much more effective when it replaces the joint effect of the two layers rather than constructing two independent surrogates.
Critical strengths
This paper investigates an important issue about model editing; given the suppression of layer, can we construct efficient surrogates to mitigate the effects of the suppression? The authors answer this question largely in the affirmative. The basic set up is very clear replacing a linear surrogate for the action of a particular layer. on this matter. As far as I understood, the authors investigate removal of one layer at a time or two layers jointly.
Critical weaknesses
For me, this paper has three critical and two minor weaknesses. The first is that it concentrates only on recouping performance loss on particular layers the choice of which is not explained. There is no systematic investigation of what happens as we eliminate layers. Why didn't the authors look at systematic elimination of each layer at the very least? Already from De Vries etal, Dalvi et al and Sajjad et al, we know different layers have different effects . Single or pairwaise layer removal is only the tip of the iceberg and iterative layer removal really needs to be discussed. This is pursued in SLEB (general model optimization via eg perplexity) and TALE (task specific optimization) . In addition, the field has really moved on from performance loss only though.
The second weakness is that the authors leave out a new and important discovery. Layer removal sometimes INCREASES performance and greedy algorithms for iteratively deleting layers to optimize accuracy sometimes provides important gains. In particular, greedy iterative algorithms sometimes improve performance while eliminating up to 20% of the layers of some models. In the conclusion the authors say they will treat this in future work but it seems that the question is central to their efforts; in particular one doesn't want to construct surrogates for layers whose removal increases task performance. The authors should consult https://arxiv.org/pdf/2510.22767 where a systematic study is performed on this question, as well as https://arXiv:2605.14738v1, which investigates where and why increases in performance due to model editing happen. From what some of these papers say, the gains from model editing can be quite widespread at least on standard benchmarks, which the authors also use. In general there needs to be a more thorough literature review of pruning methods to motivate the paper's approach. I have provided some references below but there are more.
The third weakness is that the discussion in this pape bypasses the big fork in the road: do you think that model pruning should be task specific should it be somehow general? The path taken is kind of mushy. The authors train on one task but evaluate on 6. But why not train on a general measure like perplexity or some other general measure? And what about task specific model editing?
Fourth those who work on model editing know that the evaluation method can affect reported accuracy results on benchmarks. What evaluation methods were used on the multiple choice benchmarks? Appendix C is not at all explicit. Was accuracy measured on actual output or on logits as in LMeval? On generative benchmarks? Also the tasks are relatively old except for DART. And somewhat easy for mid sized models.
Finally I'm sort of missing the point of looking at surrogates in the first place. The results section is fragmentary. a lot of little results but not linked together in any kind of a coherent story. The authors say: "Our aim is to reduce the parameters used at selected layers while preserving model performance". If you're really interested in lowering model parametrers while keeping performance, distillation seems like a much better way to go. See the Nvidia Puzzle distillation paper for an example. Hasn't distillation made surrogates essentially moot?
Wietse De Vries, Andreas Van Cranenburgh, and Malv- ina Nissim. 2020. What’s so special about bert’s layers? a closer look at the nlp pipeline in mono- lingual and multilingual models. arXiv preprint arXiv:2004.06499.
Jiwon Song, Kyungseok Oh, Taesu Kim, Hyungjun Kim, Yulhwa Kim, and Jae-Joon Kim. 2024. Sleb: Streamlining llms through redundancy verification and elimination of transformer blocks. arXiv preprint arXiv:2402.09025.
Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. 2023. A simple and effective pruning ap- proach for large language models. arXiv preprint arXiv:2306.11695
Lu Yin, You Wu, Zhenyu Zhang, Cheng-Yu Hsieh, Yaqing Wang, Yiling Jia, Gen Li, Ajay Jaiswal, Mykola Pechenizkiy, Yi Liang, Michael Bendersky, Zhangyang Wang, and Shiwei Liu. 2024. Outlier weighed layerwise sparsity (owl) a missing secret sauce for pruning llms to high sparsity. In Proceed- ings of the 41st International Conference on Ma- chine Learning, pages 57101–57115 Fahim Dalvi, Hassan Sajjad, Nadir Durrani, and Yonatan Belinkov. 2020. Analyzing redundancy in pretrained transformer models. arXiv preprint arXiv:2004.04010.
Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. 2023. On the effect of dropping lay- ers of pre-trained transformer models. Computer Speech & Language, 77:101429.
Puzzle Distillation-based NAS for inference-optimized LLMs
Questions for authors
please see above
Some minor comments:
Learned replacement is not new, which replaces parts of the original large model ---> this is not English
use token F1 and the other tasks accuracy; ---> this isn't English either
from Dense across 13 selected Llama layers on eight datasets Why only 13 layers?
Greedy generation uses the same prompts for Dense, Skip and Replace please explain what greedy generation is
L31 line 195 should be critical since its job is to prepare for decoding no? L0 is known to be a layer that can't be removed without terrible results. See the TALE paper for discussion.
7 comments
Summary and Response to Reviewer's Comment (2/2)
Q4. Is the goal task-specific optimization or general model compression? Why train on one task and evaluate on other tasks?
A4. We thank the reviewer for the opportunity to clarify the purpose of cross-task evaluation. We use it to study task similarity through layer behavior. A module fitted on one task is evaluated on another to ask whether both tasks rely on a sufficiently similar transformation at the intervention layer. The resulting transfer patterns, together with deletion sensitivity, provide signals for investigating groups of tasks with shared layer requirements. Positive and negative transfers are both informative for this analysis. Our aim is to characterize these relationships and guide selective reuse, rather than to train a universally compressed model.
Q5. Are multiple-choice results obtained from generated answers or from candidate-answer logits? What does greedy evaluation mean?
A5. We thank the reviewer for requesting this clarification. In the submitted results, the model generates an answer by choosing the highest scored token at each decode step. A task specific parser then extracts the answer from that text; the multiple-choice tasks are not scored by ranking the likelihoods of candidate answers. The two QA tasks use token F1, and the other tasks use accuracy as metric.
Q6. What about the first/last layers and iterative layer removal?
A6. We thank the reviewer for these suggestions. The expanded full-layer sweep includes the first and last layers of both models, and Table R3 summarizes the resulting replacement coverage. The multi-layer experiment in Fig. R2 uses fixed sets intervened on simultaneously, allowing us to study how compact replacements restore computation after multi-layer skip.
Iterative removal addresses a complementary selection question by updating the choice after each intervention. Our current results concern simultaneous intervention; they provide a basis for studying how layer selection and replacement can be combined in an iterative procedure.
Q7. Can the terminology and unclear English descriptions of replacement and evaluation be clarified?
A7. We thank the reviewer for these suggestions.
We revised the Introduction to explain that learned replacements use surrogate modules to approximate parts of a pretrained model. We also clarified in the scoring section that ROPES and SQuAD use token F1 and all other tasks use accuracy, and updated the Figure 4 caption to state the same distinction for the tasks shown.
Greedy evaluation is described as token by token generation using the highest scored token, followed by task specific score of the generated answer.
We will also distinguish replacement to block parameter ratio, whole model parameter reduction and measured generation speedup. Fig. R1 reports runtime, Fig. R2 reports downstream task performance, and Tables R3–R4 summarize layer coverage, making the role of each measurement explicit.
reply to rebuttal
Many thanks for your rebuttal. Re question 1, you still didn't look at iterative effects of layer removal, and this has been shown to be important in the literature. I'm also still worried by the response to Q3 and the take home message from this that is new given what's in the literature.
Summary and Response to Reviewer's Comment (1/2)
Q1. Why were only 13 Llama layers scanned, and how were the sensitive layers selected?
A1. We thank the reviewer for asking us to clarify the selection procedure. For the initial Llama experiments, we grouped layers into low-, medium-, and high-impact groups based on the drop in task score after skipping each layer. We selected 13 candidates across these groups, with greater emphasis on layers that caused larger drops, because our initial question was whether a compact replacement could recover performance where skipping was most harmful. Within this candidate set, we selected sensitive layers by the size of the drop.
We have since evaluated both Skip and rank-64 Replace at every layer across all six tasks: 32 layers for Llama and 36 for Qwen. As shown in Table R3-R8, among settings where Skip lowered the task score by more than one point, Replace improved over Skip in 114 of 125 Llama settings and 101 of 117 Qwen settings. We refer the reviewer to Table R3 for the full comparison, including gains, ties, and declines beyond the original 13-layer subset.
Q2. If skipping a layer sometimes improves performance, why replace it at all?
A2. We agree with the reviewer: if skipping a layer preserves or improves performance on the target task, there is no need to replace it. Our initial experiments deliberately included layers with low, medium, and high sensitivity to skipping so that we could compare Skip and Replace across a range of conditions. The low-impact cases show where skipping alone can be sufficient. Our main goal, however, is to recover performance where skipping causes a substantial drop.
The expanded sweep supports this distinction: among the 242 settings where Skip reduced the task score by more than one point, Replace improved over Skip in 215 (88.8%; Table R3). The multi-layer results illustrate the same motivation: on Llama ARC Easy, skipping eight layers reduced accuracy from 93.0% to 12.0%, while Independent and Joint replacement recovered it to 55.5% and 67.0%, respectively (Fig. R2; Table R9).
Q3. What is novel relative to pruning approaches such as SLEB/TALE and distillation approaches such as Puzzle?
A3. We thank the reviewer for asking us to sharpen the distinction. SLEB iteratively removes blocks and updates calibration-based importance scores [1]; TALE selects task-adapted layer removals using validation performance [2]. Puzzle combines locally distilled alternative blocks, hardware-constrained architecture search and global distillation [3]. Our focus is the behavior of deletion and its recovery: can a block whose removal causes a large task-specific drop be approximated by a much smaller module, and when do these modules compose? For example, at Qwen layer 6 on ARC, a rank-64 linear replacement scores 4.5%, while an MLP with almost the same parameter count reaches 82.0% (Table 1). Our objective is to study and mitigate the disruption caused by deletion(skip), rather than optimize an entire student model or search for a deployment architecture. We have clarified this distinction in the revised Introduction (Section 1, red text). We fit compact surrogates and compare Skip and Replace at the same layer positions to measure how much of the loss from pruning can be recovered, rather than to train a smaller student model or search for an inference-optimized architecture. Particularly, this paper is not studying how to create a state-of-the-art model compression method. Instead, this paper systematically explores the essential mechanism of layer pruning, which is a big family in model compression methods, to examine their potentials and limitations. Based on this, we further explore the effectiveness of layer replacement to compensate the dropping from layer skip. Since the scope of this paper is layer pruning, we do not use or compare with model distillation methods
[1] SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks https://proceedings.mlr.press/v235/song24f.html.
[2] TALE. TELL-TALE: Task Efficient LLMs with Task Aware Layer Elimination. https://aclanthology.org/2026.findings-acl.1136/.
[3] Puzzle: Distillation-Based NAS for Inference-Optimized LLMs. https://proceedings.mlr.press/v267/bercovich25a.html.