acceptodds
Under review as a conference paper at ICLR 2027

Hard to Remove, Cheap to Replace: Compact, Transferable Surrogates for Sensitive LLM Layers

Abstract

Removing whole layers is a direct way to compress a large language model, but the damage it causes varies widely across layers and tasks, and removal alone does not show whether that damage can be recovered. We study layer sensitivity through recovery: how much of a removed layer’s contribution a compact learned replacement can restore, and whether that replacement transfers to tasks it was not fitted on. Using Llama 3.1 8B and Qwen3-8B, with the rest of each model frozen, we apply three interventions to complete layers during free text generation: Skip removes a layer, Replace substitutes a small surrogate fitted to the layer’s residual output, and Transfer reuses a surrogate fitted on a different dataset. We extend these interventions to pairs of layers, compare linear and nonlinear surrogates, and contrast trained surrogates with calibrated random maps. A linear surrogate of rank 64 with about 1/413 of a Llama layer’s parameters improves over Skip in all 12 sensitive Llama settings, recovering up to 93.2% of the lost performance. Surrogates also transfer across datasets and task categories: in Llama, surrogates fitted in one task can transfer to other tasks, still with significant improvement over Skip. Jointly fitting two surrogates for two layers improves the replacement over independent surrogate composition. And we prove the effectiveness of the surrogates training by showing it outperforms random maps in all tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Transformer Layers, Model Compression, Layer Replacement, Layer-wise Analysis, Task-dependent Computation, Model Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HupMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

"Hard to remove, easy to replace" examines how damage to accuracy from layer removal can be repaired by linear or nonlinear surrogates and how such surrogates can recover accuracy transfer. The authors also consider removing pairs of layers and show that the surrogate is much more effective when it replaces the joint effect of the two layers rather than constructing two independent surrogates.

Critical strengths

This paper investigates an important issue about model editing; given the suppression of layer, can we construct efficient surrogates to mitigate the effects of the suppression? The authors answer this question largely in the affirmative. The basic set up is very clear replacing a linear surrogate for the action of a particular layer. on this matter. As far as I understood, the authors investigate removal of one layer at a time or two layers jointly.

Critical weaknesses

For me, this paper has three critical and two minor weaknesses. The first is that it concentrates only on recouping performance loss on particular layers the choice of which is not explained. There is no systematic investigation of what happens as we eliminate layers. Why didn't the authors look at systematic elimination of each layer at the very least? Already from De Vries etal, Dalvi et al and Sajjad et al, we know different layers have different effects . Single or pairwaise layer removal is only the tip of the iceberg and iterative layer removal really needs to be discussed. This is pursued in SLEB (general model optimization via eg perplexity) and TALE (task specific optimization) . In addition, the field has really moved on from performance loss only though.

The second weakness is that the authors leave out a new and important discovery. Layer removal sometimes INCREASES performance and greedy algorithms for iteratively deleting layers to optimize accuracy sometimes provides important gains. In particular, greedy iterative algorithms sometimes improve performance while eliminating up to 20% of the layers of some models. In the conclusion the authors say they will treat this in future work but it seems that the question is central to their efforts; in particular one doesn't want to construct surrogates for layers whose removal increases task performance. The authors should consult https://arxiv.org/pdf/2510.22767 where a systematic study is performed on this question, as well as https://arXiv:2605.14738v1, which investigates where and why increases in performance due to model editing happen. From what some of these papers say, the gains from model editing can be quite widespread at least on standard benchmarks, which the authors also use. In general there needs to be a more thorough literature review of pruning methods to motivate the paper's approach. I have provided some references below but there are more.

The third weakness is that the discussion in this pape bypasses the big fork in the road: do you think that model pruning should be task specific should it be somehow general? The path taken is kind of mushy. The authors train on one task but evaluate on 6. But why not train on a general measure like perplexity or some other general measure? And what about task specific model editing?

Fourth those who work on model editing know that the evaluation method can affect reported accuracy results on benchmarks. What evaluation methods were used on the multiple choice benchmarks? Appendix C is not at all explicit. Was accuracy measured on actual output or on logits as in LMeval? On generative benchmarks? Also the tasks are relatively old except for DART. And somewhat easy for mid sized models.

Finally I'm sort of missing the point of looking at surrogates in the first place. The results section is fragmentary. a lot of little results but not linked together in any kind of a coherent story. The authors say: "Our aim is to reduce the parameters used at selected layers while preserving model performance". If you're really interested in lowering model parametrers while keeping performance, distillation seems like a much better way to go. See the Nvidia Puzzle distillation paper for an example. Hasn't distillation made surrogates essentially moot?

Wietse De Vries, Andreas Van Cranenburgh, and Malv- ina Nissim. 2020. What’s so special about bert’s layers? a closer look at the nlp pipeline in mono- lingual and multilingual models. arXiv preprint arXiv:2004.06499.

Jiwon Song, Kyungseok Oh, Taesu Kim, Hyungjun Kim, Yulhwa Kim, and Jae-Joon Kim. 2024. Sleb: Streamlining llms through redundancy verification and elimination of transformer blocks. arXiv preprint arXiv:2402.09025.

Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. 2023. A simple and effective pruning ap- proach for large language models. arXiv preprint arXiv:2306.11695

Lu Yin, You Wu, Zhenyu Zhang, Cheng-Yu Hsieh, Yaqing Wang, Yiling Jia, Gen Li, Ajay Jaiswal, Mykola Pechenizkiy, Yi Liang, Michael Bendersky, Zhangyang Wang, and Shiwei Liu. 2024. Outlier weighed layerwise sparsity (owl) a missing secret sauce for pruning llms to high sparsity. In Proceed- ings of the 41st International Conference on Ma- chine Learning, pages 57101–57115 Fahim Dalvi, Hassan Sajjad, Nadir Durrani, and Yonatan Belinkov. 2020. Analyzing redundancy in pretrained transformer models. arXiv preprint arXiv:2004.04010.

Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. 2023. On the effect of dropping lay- ers of pre-trained transformer models. Computer Speech & Language, 77:101429.

Puzzle Distillation-based NAS for inference-optimized LLMs

Questions for authors

please see above Some minor comments:
Learned replacement is not new, which replaces parts of the original large model ---> this is not English

use token F1 and the other tasks accuracy; ---> this isn't English either

from Dense across 13 selected Llama layers on eight datasets Why only 13 layers?

Greedy generation uses the same prompts for Dense, Skip and Replace please explain what greedy generation is

L31 line 195 should be critical since its job is to prepare for decoding no? L0 is known to be a layer that can't be removed without terrible results. See the TALE paper for discussion.

7 comments
AuthorsOpenReview ↗

Summary and Response to Reviewer's Comment (1/2)

Q1. Why were only 13 Llama layers scanned, and how were the sensitive layers selected?
A1. We thank the reviewer for asking us to clarify the selection procedure. For the initial Llama experiments, we grouped layers into low-, medium-, and high-impact groups based on the drop in task score after skipping each layer. We selected 13 candidates across these groups, with greater emphasis on layers that caused larger drops, because our initial question was whether a compact replacement could recover performance where skipping was most harmful. Within this candidate set, we selected sensitive layers by the size of the drop.
We have since evaluated both Skip and rank-64 Replace at every layer across all six tasks: 32 layers for Llama and 36 for Qwen. As shown in Table R3-R8, among settings where Skip lowered the task score by more than one point, Replace improved over Skip in 114 of 125 Llama settings and 101 of 117 Qwen settings. We refer the reviewer to Table R3 for the full comparison, including gains, ties, and declines beyond the original 13-layer subset.

Q2. If skipping a layer sometimes improves performance, why replace it at all?
A2. We agree with the reviewer: if skipping a layer preserves or improves performance on the target task, there is no need to replace it. Our initial experiments deliberately included layers with low, medium, and high sensitivity to skipping so that we could compare Skip and Replace across a range of conditions. The low-impact cases show where skipping alone can be sufficient. Our main goal, however, is to recover performance where skipping causes a substantial drop.
The expanded sweep supports this distinction: among the 242 settings where Skip reduced the task score by more than one point, Replace improved over Skip in 215 (88.8%; Table R3). The multi-layer results illustrate the same motivation: on Llama ARC Easy, skipping eight layers reduced accuracy from 93.0% to 12.0%, while Independent and Joint replacement recovered it to 55.5% and 67.0%, respectively (Fig. R2; Table R9).

Q3. What is novel relative to pruning approaches such as SLEB/TALE and distillation approaches such as Puzzle?
A3. We thank the reviewer for asking us to sharpen the distinction. SLEB iteratively removes blocks and updates calibration-based importance scores [1]; TALE selects task-adapted layer removals using validation performance [2]. Puzzle combines locally distilled alternative blocks, hardware-constrained architecture search and global distillation [3]. Our focus is the behavior of deletion and its recovery: can a block whose removal causes a large task-specific drop be approximated by a much smaller module, and when do these modules compose? For example, at Qwen layer 6 on ARC, a rank-64 linear replacement scores 4.5%, while an MLP with almost the same parameter count reaches 82.0% (Table 1). Our objective is to study and mitigate the disruption caused by deletion(skip), rather than optimize an entire student model or search for a deployment architecture. We have clarified this distinction in the revised Introduction (Section 1, red text). We fit compact surrogates and compare Skip and Replace at the same layer positions to measure how much of the loss from pruning can be recovered, rather than to train a smaller student model or search for an inference-optimized architecture. Particularly, this paper is not studying how to create a state-of-the-art model compression method. Instead, this paper systematically explores the essential mechanism of layer pruning, which is a big family in model compression methods, to examine their potentials and limitations. Based on this, we further explore the effectiveness of layer replacement to compensate the dropping from layer skip. Since the scope of this paper is layer pruning, we do not use or compare with model distillation methods

[1] SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks https://proceedings.mlr.press/v235/song24f.html.
[2] TALE. TELL-TALE: Task Efficient LLMs with Task Aware Layer Elimination. https://aclanthology.org/2026.findings-acl.1136/.
[3] Puzzle: Distillation-Based NAS for Inference-Optimized LLMs. https://proceedings.mlr.press/v267/bercovich25a.html.

AuthorsOpenReview ↗

Summary and Response to Reviewer's Comment (2/2)

Q4. Is the goal task-specific optimization or general model compression? Why train on one task and evaluate on other tasks?
A4. We thank the reviewer for the opportunity to clarify the purpose of cross-task evaluation. We use it to study task similarity through layer behavior. A module fitted on one task is evaluated on another to ask whether both tasks rely on a sufficiently similar transformation at the intervention layer. The resulting transfer patterns, together with deletion sensitivity, provide signals for investigating groups of tasks with shared layer requirements. Positive and negative transfers are both informative for this analysis. Our aim is to characterize these relationships and guide selective reuse, rather than to train a universally compressed model.

Q5. Are multiple-choice results obtained from generated answers or from candidate-answer logits? What does greedy evaluation mean?
A5. We thank the reviewer for requesting this clarification. In the submitted results, the model generates an answer by choosing the highest scored token at each decode step. A task specific parser then extracts the answer from that text; the multiple-choice tasks are not scored by ranking the likelihoods of candidate answers. The two QA tasks use token F1, and the other tasks use accuracy as metric.

Q6. What about the first/last layers and iterative layer removal?
A6. We thank the reviewer for these suggestions. The expanded full-layer sweep includes the first and last layers of both models, and Table R3 summarizes the resulting replacement coverage. The multi-layer experiment in Fig. R2 uses fixed sets intervened on simultaneously, allowing us to study how compact replacements restore computation after multi-layer skip. Iterative removal addresses a complementary selection question by updating the choice after each intervention. Our current results concern simultaneous intervention; they provide a basis for studying how layer selection and replacement can be combined in an iterative procedure.

Q7. Can the terminology and unclear English descriptions of replacement and evaluation be clarified?
A7. We thank the reviewer for these suggestions.
We revised the Introduction to explain that learned replacements use surrogate modules to approximate parts of a pretrained model. We also clarified in the scoring section that ROPES and SQuAD use token F1 and all other tasks use accuracy, and updated the Figure 4 caption to state the same distinction for the tasks shown. Greedy evaluation is described as token by token generation using the highest scored token, followed by task specific score of the generated answer.
We will also distinguish replacement to block parameter ratio, whole model parameter reduction and measured generation speedup. Fig. R1 reports runtime, Fig. R2 reports downstream task performance, and Tables R3–R4 summarize layer coverage, making the role of each measurement explicit.

Reviewer_HupMOpenReview ↗

reply to rebuttal

Many thanks for your rebuttal. Re question 1, you still didn't look at iterative effects of layer removal, and this has been shown to be important in the literature. I'm also still worried by the response to Q3 and the take home message from this that is new given what's in the literature.

Reviewer_XqkEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper explores replacing sensitive Transformer layers with small, learned surrogate modules instead of removing them entirely. Experiments show that these lightweight replacements can recover much of the lost performance and sometimes transfer well across different tasks

Critical strengths

1.The paper offers an interesting perspective on layer compression by studying how much performance can be recovered using replace method.

2.The cross-task transfer experiments are extensive and provide useful insights into how much layer behavior is shared across tasks.

3.The experiments cover different architectures.

Critical weaknesses

1.The experiments replace at most two layers.

2.The paper lacks direct comparisons with existing layer replacement methods such as ReplaceMe and SubFit.

3.Lower reconstruction error does not consistently improve task performance, yet the method still relies on this metric for checkpoint selection.

Questions for authors

1.How do accuracy, memory usage, and inference speed change as more layers are replaced?

2.How does the method compare with ReplaceMe and SubFit under similar compression ratios and training budgets?

3.Could output-level distillation help bridge the gap between reconstruction error and downstream performance?

3 comments
AuthorsOpenReview ↗

Summary and Response to Reviewer's Comment

Q1. Does replacement remain useful beyond the one- and two-layer experiments?
A1. We thank the reviewer for this suggestion. Fig. R2 extends the evaluation to simultaneous replacement of 3, 4, 6 and 8 layers. On Llama ARC at six layers, Skip, Independent and Joint achieve 28.0%, 79.5% and 88.0%, respectively, versus 93.0% for Dense. At eight layers, Independent and Joint reach 55.5% and 67.0% accuracy, respectively, versus 12.0% for Skip. Table R9 lists the layer set for each intervention size. These results support the use of compact modules to preserve useful computation as more blocks are removed. The Qwen and ROPES results also show that the preferred tuning strategy depends on the model and task.

Q2. How does the method compare directly with ReplaceMe and SubFit?
A2. We thank the reviewer for connecting our work to these approaches. ReplaceMe fits linear transformations for selected blocks and folds them into the remaining weights [1], while SubFit fits lightweight bypasses for selected attention and FFN submodules [2]. Our aim is to understand when and how performance lost by skipping layers can be recovered with compact replacements, rather than to propose a state-of-the-art compression method. So we are not directly comparable to the mentioned works. We therefore compare Skip and Replace at the same layer positions and measure how much of the lost performance is recovered across tasks, replacement architectures, and layer combinations. This design tests the effect of adding a replacement without conflating it with differences in which layers are selected.

Q3. Why select checkpoints by reconstruction error if that error may not track task performance?
A3. We thank the reviewer for the question. Reconstruction loss is aligned with our immediate objective: approximate the selected Dense block's transformation while leaving the rest of the model fixed. It provides a consistent validation criterion without selecting checkpoints by downstream answer correctness. The fitting target is the Dense representation, not the task's gold answer, so the procedure does not directly optimize evaluation accuracy or gold-answer labels. Downstream evaluation then tests whether this approximation is useful, as demonstrated by the recovery in Fig. R2. Reconstruction and task scores measure different properties, so a lower reconstruction error need not produce a higher task score. This distinction is itself informative for studying which local computations can be replaced. We select checkpoints by validation reconstruction error to measure how closely the surrogate matches the Dense representation. We use held-out task scores to evaluate whether that approximation preserves task performance.

Q4. Could an output-level training objective improve replacement quality?
A4. We thank the reviewer for suggesting an output-level objective. We see it as complementary to the goal of our current method: preserving the pretrained model’s behavior with fewer parameters. Skipping a block changes the hidden states passed to later layers, and these deviations can compound when several blocks are skipped. To reduce this error, we train each surrogate to learn to match the original block’s output activations. MSE directly measures that reconstruction error without using benchmark labels. Task accuracy is then used to evaluate the replacement, rather than to train it.
We also explored the reviewer’s suggestion in a generic-text pilot (Table R4). Joint next-token NLL training achieved a perplexity of 9.86, compared with 15.44 for local-MSE replacement and 17.46 for Skip, without task-specific supervision. This is a promising result for the joint NLL configuration.

[1] ReplaceMe. Network Simplification via Depth Pruning and Transformer Block Linearization. https://proceedings.neurips.cc/paper_files/paper/2025/file/1c10d0c087c14689628124bbc8fa69f6-Paper-Conference.pdf.
[2] SubFit. From Layers to Submodules: Rethinking Granularity in Replacement-Based LLM Compression. https://arxiv.org/abs/2606.02559v2.

Thank you for the experiments. However, the missing comparisons with closely related methods and the unresolved concerns about reconstruction-based checkpoint selection remain significant. The rebuttal does not address these issues. Therefore I will maintain my original rating

Dear Reviewer XqkE,

Thank you for reading the additional experiments and explaining your remaining concerns. Your comments on related methods and checkpoint selection helped us clarify what our experiments establish and where further comparisons would be useful. We appreciate the time you put into the review.

Reviewer_69Y7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work studies whether layers that are damaging to remove can be replaced by much smaller learned surrogates, and whether those surrogates transfer across tasks. Using Llama 3.1 8B and Qwen3-8B, each with hidden size 4,096, it compares layer skipping, local replacement, and reuse of replacements fitted on another dataset. The surrogates approximate a complete layer's residual output while the remaining model stays frozen. The experiments also compare linear and nonlinear surrogates, independent and joint fitting of layer pairs, and learned replacements against random maps.

Critical strengths
  1. Studying recovery alongside removal is a useful way to examine layer sensitivity. As the paper shows and gives examples for, a layer can be damaging to skip without requiring a similarly large replacement. The cross-task transfer grids also help show how much the replacements depend on the data used to fit them.

  2. I found the architecture comparison and layer-pair experiments to be quite informative. In Table 1, the nonlinear, cheap surrogate recovers performance at Qwen L6 where larger linear surrogates fail, suggesting that increasing linear rank alone can indeed miss useful replacements. I also really like the pair experimentsm, that show why separately fitting surrogates may be insufficient when the individual errors interact. The random map controls also provide a useful check.

Critical weaknesses
  1. Evaluation leaves me very uncertain about how well the findings extend to meaningful preservation of model capabilities. The main replacement and transfer results use only 100 test sequences per dataset and 2 training seeds, both questionable. Most benchmarks are rather simple classification or QA tasks, while the math results have very low original-model scores: Qwen reaches only 1% on DART (Table 1); Fig 1 shows Llama scores of 16% on ASDiv and 14% on DART. Recovering performance relative to such baselines provides limited evidence of preserving useful mathematical capability - there is hardly anything to preserve here; it is already poor in the first place. The claims therefore do not hold up in this context and would need further evaluation on better-chosen tasks or models with the corresponding capabilties. Appendix C.2 also notes that the first-number parser can misclassify answers containing multiple numbers, which makes me trust the results less (How often does that happen?). I would like to see a clearer explanation of these low baselines and a more reliable evaluation on tasks where the original models perform meaningfully well.

  2. The narrow scope of the experiments limits the broader conclusions, severely. Both tested models are dense 8B models with hidden width 4,096, and replacement covers only individual layers or selected pairs. This is a very narrow setting. The authors acknowledge that learned replacement already exists, so the contribution rests mainly on what these experiments reveal about recovery and transfer. I do find the architecture and interaction results useful, but stronger evidence across model sizes would make those findings more convincing. Joint fitting also receives additional optimization, so its gains do not isolate the effect of the joint objective from the training budget. Some analyses could be useful here. The very large parameter-reduction ratios apply to the replaced layer ONLY, while whole-model runtime and memory benefits remain unmeasured.

Questions for authors
  1. Why are the original-model math scores so low, esp. Qwen's 1% on DART? How much do answer formatting, the parser, and the generation settings contribute? Have you checked whether the recovery findings hold with more reliable answer extraction and larger test sets?

  2. What is the main empirical finding that goes beyond the fitted replacement methods discussed in the introduction? Do you have evidence that it holds at another model size, beyond the two 8B checkpoints?

  3. Does joint fitting still improve over independent fitting when the additional optimization budget is matched?

4 comments
AuthorsOpenReview ↗

Summary and Response to Reviewer's Comment (1/2)

Q1. The mathematical-task baselines are very low. How meaningful are the reported recovery ratios?
A1. We thank the reviewer for raising this point. Prior evaluations report low DART-5 accuracy for both unmodified models: 20.5% for Llama-3.1-8B-Instruct [1] and 11.7% for Qwen3-8B [2]. With such low Dense scores, the gap between Dense and Skip can be small, making recovery ratios difficult to interpret on DART-5.
On ASDiv, a math word problem benchmark, we found that the original answer extractor missed some correct answers. After correcting these errors, Dense accuracy rose from 16% to 49% for Llama and from 54% to 55% for Qwen. We define recovery as the fraction of the accuracy lost by skipping a layer that replacement restores. For Llama at layer 7 on ASDiv, Replace restores 37.5 of the 47.0 percentage points lost to Skip, a recovery rate of 79.8%. We report the gain in percentage points alongside the ratio so readers can see how much accuracy was actually recovered.

Q2. How often does first-number extraction mis-score answers?
A2. We thank the reviewer for highlighting the answer-extraction issue. We checked the saved outputs across all six tasks and found that correcting the identified extraction errors changed 2.1% of the submission’s evaluation records. After re-scoring, the overall pattern continues to support the effectiveness of our surrogate replacements, with stronger recovery in some settings and decreases in a few individual cases. We have updated the affected figures and tables accordingly.

Q3. Are 100 evaluation examples and two training seeds sufficient to support the conclusions?
A3. We thank the reviewer for this question. Given our limited computational resources and the large number of task, layer, architecture, and model combinations, we used 100 evaluation examples and two training seeds per setting to prioritize broader coverage. Based on the recorded evaluation times, for example, increasing to 500 examples per setting would require approximately five times the current evaluation compute for the expanded all-layer and multi-layer sweeps. We therefore prioritized coverage across configurations while reporting paired uncertainty intervals for representative results.

[1] Lina Berrayana, Ahmed Heakl, Abdullah Sohail, Thomas Hofmann, Salman Khan, and Wei Chen. Latent-DARM: Bridging Discrete Diffusion and Autoregressive Models for Reasoning. LIT Workshop at ICLR, 2026. arXiv:2603.09184.
[2] Ziyue Li, Yang Li, and Tianyi Zhou. Skip a Layer or Loop It? Learning Program-of-Layers in LLMs. Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026, pp. 68114–68135. arXiv:2606.06574v2.

AuthorsOpenReview ↗

Summary and Response to Reviewer's Comment (2/2)

Q4. Both models have about 8B parameters. Does the result generalize to other model scales?
A4. We thank the reviewer for this question. We chose the 8B scale as a practical balance between model capability and the cost of a broad experimental sweep. It allows us to examine complete layer ranges, multiple tasks, architectures and seeds across two distinct model families within a reproducible research budget. Increasing scale would reduce the breadth that can be tested under the same budget. Our present goal is to establish and characterize task-dependent replaceability at this scale, providing a basis for subsequent larger-model studies. The current comparison demonstrates cross-family behavior at approximately 8B parameters; extending the same settings to other scales is the next step in evaluating its scope.

Q5. Joint training starts from Independent weights and receives additional optimization. Could the gains be explained by the extra budget?
A5. We thank the reviewer for this observation. Independent fitting and joint refinement use different computational paths. Once the original model’s activations have been collected, Independent trains each surrogate on cached input–output pairs without running the original model during fitting. Joint instead optimizes the surrogates together through the intervening frozen layers. On an L40S, fitting the two Independent surrogates for Llama L2+L11 on ARC Easy took 14.35 seconds in total using cached activations. The Joint stage spent approximately 18.8 seconds on surrogate computation and updates. The measured surrogate-training costs were therefore on the same order of magnitude.

Q6. The approximately 1/413 parameter ratio is per replaced block. What is the actual whole-model efficiency benefit?
A6. We thank the reviewer for requesting the end-to-end measurement. The 1/413 ratio applies to the replaced block, not the whole model. Against Dense, replacing eight blocks sped up end-to-end generation by 1.308× on Llama and 1.268× on Qwen (Fig. R1). In the same L40S runs (2,048 input tokens, batch size 8, and 256 generated tokens), peak GPU memory allocated by PyTorch fell from 27.81 to 24.08 GiB on Llama (−13.4%) and from 28.36 to 25.00 GiB on Qwen (−11.8%). Final KV Cache memory decreased by 25.0% on Llama and 22.2% on Qwen in the same runs.

Thanks for the new results and analyses. I do like that the method actually is measurably faster and reduces KV cache usage. I am still not fully sold on its practical benefits, but I see some merit in it now and the work seems sound. I will increase my score to weak accept.

Dear Reviewer 69Y7,

Thank you for taking the time to consider our new results. We appreciate your recognition of the gains in generation speed and the reduction in KV cache usage. We are sorry about we did not have time to explore inference optimization in more depth in this study, but these results give us a starting point for testing how much further we can improve efficiency by combining layer removal and replacement while preserving task performance. Thank you for drawing attention to this direction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.