- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes using in-context learning (ICL) ability to estimate how well a model can recover after layer pruning, treating ICL as a rehearsal for repair. The method first uses block influence to select 2T candidate layers, then uses a JS divergence probe with demonstrations to choose T layers for removal, requiring only forward passes during selection. Across different models, the experiments show an average gain of about 2 percentage points over the strongest baseline at 25% layer removal.
Critical strengths
Defining the layer-selection goal in terms of recovery after repair is a relatively novel starting point.
Through many prune-and-repair experiments, partial correlation analysis, and ablations, the authors show that the probe provides a useful signal for layer selection. The experiments also cover multimodal models.
Critical weaknesses
-
The exact equivalence between ICL and gradient updates holds only under specific linear-attention settings. It does not directly show that in-context adaptation and LoRA repair rely on the same structures in pretrained Transformers. The correlation analysis and demonstration-count ablations support the probe’s predictive value, but do not confirm the proposed mechanism.
-
The candidate pool is fixed at twice the removal budget without a clear reason for this specific choice. Although the authors test different pool widths on multimodal models, the sensitivity analysis for language models is limited. Comparing performance and selection cost at widths such as 1.5T and 2.5T would strengthen the method.
-
The method assumes that preserving the ability to learn new rules from demonstrations makes a layer a good candidate for removal. However, the demonstrations cannot cover every domain, so a probe drawn from a limited set of domains can only directly test learning ability within those domains. Layer selection may therefore favor the domains covered by the probe, while its effect on learning in other domains remains unclear.
-
I suggest adding ICL probe data from more domains and reporting the removed layer indices and their overlap for each, to show whether the layer selection is affected by the probe domain.
-
MoE is an increasingly common model architecture, so testing the method on MoE models would help establish its broader use. In addition, at removal rates above 25%, the advantage over FinerCut appears limited.
-
The evaluation mainly covers knowledge questions, instruction following, mathematics, and multi-step reasoning, which are relatively close to the abilities covered by the repair data from Alpaca and the Tülu-3 subset. This is not enough to establish broad capability preservation. The authors should add code, long-context, and multi-turn tasks, as well as tool-use or agent tasks where the original models have these abilities, to test whether the method preserves more complex capabilities.
Questions for authors
The authors have conducted extensive experiments, but the method relies largely on intuition and the novelty is relatively incremental. I hope the authors can address my concerns, especially Weaknesses 3, 4, 5, and 6.
4 comments
Response to Reviewer YdfC (2/3): higher removal ratios, code, long-context and multi-turn tasks
W5-2: The advantage over FinerCut
As the reviewer notes, above 25% removal our advantage over FinerCut is small, although our method still has the best mean over the four axes of Table 2 at 37.5% and 50% removal (38.6 vs. 38.2 and 28.8 vs. 28.3, Tables 21–22). The probe itself, however, also improves on FinerCut when it ranks FinerCut's own candidates, as Appendix F (Table 19) shows for SFT. We took the 2T layers with the smallest d₀ (the output divergence that FinerCut measures without demonstrations), where T is the number of layers to remove, and let the probe choose T of them. At 25% removal, this choice beats FinerCut on all five backbones under both repairs (Avg, three seeds):
| Avg | Qwen3 | Qwen2 | Qwen1.5 | Llama | Mistral | Mean |
|---|---|---|---|---|---|---|
| FinerCut, SFT (Table 2) | 53.7 | 39.2 | 44.8 | 56.8 | 46.0 | 48.1 |
| d₀ candidates + probe, SFT | 60.3 | 41.8 | 45.8 | 59.4 | 49.3 | 51.3 |
| FinerCut, OPD (Table 2) | 56.5 | 39.3 | 39.5 | 58.8 | 42.7 | 47.4 |
| d₀ candidates + probe, OPD | 63.0 | 42.1 | 39.8 | 60.7 | 44.2 | 50.0 |
Table 19 reports this comparison as the share of the unpruned IFEval and BBH scores kept, and the table above gives the Avg and adds OPD. For simplicity, our method takes its candidates from block influence alone. When the probe instead ranks the union of the block-influence and FinerCut candidates, its mean rises to 41.4 at 37.5% and 31.0 at 50% removal, against 38.2 and 28.3 for FinerCut (Appendix G, Table 22). We report this as a diagnostic and keep the simpler default of the paper.
W6: Code, long-context and multi-turn tasks
Thank you for this suggestion. We evaluated the 180 repaired checkpoints of Table 2, 90 for each repair, on HumanEval [4] (code), BABILong [5] with 8k-token contexts (long context) and Multi-IF [6] (multi-turn), without any further training. After SFT, our method has the best five-backbone mean on all three. After OPD it is again the best on long contexts and multi-turn chat, while on code FinerCut and DarwinLM are ahead (23.6 and 21.4, against 19.9), a gap that comes mostly from Qwen3-8B. We added the table to the main text as Table 6 (§4.3), with each backbone in Table 37 (Appendix N):
| Five-backbone mean (%) | HumanEval, SFT | HumanEval, OPD | BABILong 8k, SFT | BABILong 8k, OPD | Multi-IF, SFT | Multi-IF, OPD |
|---|---|---|---|---|---|---|
| ShortGPT | 17.7 | 16.9 | 47.6 | 47.3 | 38.0 | 41.8 |
| Angular | 14.9 | 14.7 | 38.6 | 37.0 | 36.4 | 38.8 |
| Reverse order | 12.9 | 10.9 | 39.7 | 40.9 | 26.7 | 26.3 |
| FinerCut | 17.7 | 23.6 | 42.3 | 39.8 | 37.2 | 41.8 |
| DarwinLM | 14.1 | 21.4 | 32.4 | 32.5 | 34.5 | 35.8 |
| Ours | 20.2 | 19.9 | 48.8 | 49.0 | 39.8 | 45.3 |
Two of these benchmarks test abilities that the repair data does not contain. Repair examples are cut at 256 tokens (App. B.2), far shorter than BABILong's 8k-token contexts, and we repair on first turns only, while Multi-IF also scores the second and third turns. We could not run tool-use or agent benchmarks, which need a function-calling harness that our pipeline does not have.
Response to Reviewer YdfC (3/3): in-context learning and gradient descent, the size of the candidate pool
We now turn to W1 and W2.
W1: ICL equals gradient descent only for linear attention
As the reviewer notes, the equivalence is exact only for linear attention [7]. §3.1 now presents this view as the hypothesis that led to the probe, and the paper's claims rest on the experiments. The hypothesis makes two predictions that a present-damage view does not, and our experiments support both. The probe's signal is stronger with demonstrations than without them (Table 15), and the probe predicts recovery beyond present damage (Table 12), even among candidates whose present damage is matched, where it orders 73–89% of the pairs correctly.
W2: Why a pool of 2T candidates?
We ran the language models with pool widths from T to all layers, each repaired with three seeds (Table 17 in Appendix F). A width of T leaves the probe no choice and is exactly ShortGPT:
| Pool width | Qwen3 | Qwen2 | Qwen1.5 | Llama | Mistral | Mean |
|---|---|---|---|---|---|---|
| T (= ShortGPT, Table 2) | 55.4 | 36.3 | 42.3 | 59.2 | 49.5 | 48.5 |
| 1.5T | 56.0 | 37.0 | 44.5 | 59.4 | 48.9 | 49.2 |
| 2T (default, Table 2) | 61.7 | 39.4 | 44.5 | 59.4 | 48.9 | 50.8 |
| 2.5T | 55.8 | 42.5 | 45.4 | 59.4 | 48.9 | 50.4 |
| All layers (probe alone) | 55.9 | 42.6 | 45.4 | 59.4 | 48.9 | 50.4 |
Every width from 1.5T to all layers keeps the five-backbone mean between 49.2 and 50.8, above every baseline in Table 2 (48.5). On Llama-3.1-8B and Mistral-7B every width above T selects the same layers as 2T, so those cells repeat Table 2. On Qwen2-7B, where block influence predicts recovery poorly (Table 16), a wider pool helps.
Qwen3-8B is the only backbone where 2T clearly beats both narrower and wider pools, and its selections show why (they are the same in all three seeds). At 1.5T the pool does not contain layer 2, the probe's ninth choice at 2T, so the probe removes layer 25 instead. At 2.5T the pool also admits layer 34, the second-to-last layer. The probe scores it as cheap and removes it in place of layer 2, but removing it causes output-side damage that the probe does not detect and repair does not undo. Without a pool, the probe also removes this layer on Qwen3-8B (Appendix F). §3.2 now says that keeping such layers out is part of the pool's job.
Qwen3-8B shows that the pool can be too narrow or too wide. Over the five backbones, 2T gives the best mean. We fixed 2T before running these experiments and keep it as the default. Selection, including model loading, takes 5.5, 6.2 and 6.7 minutes on one GPU at 1.5T, 2T and 2.5T (five-backbone mean), and the probe passes themselves take 1–2.5 minutes (App. B).
We hope these answers address your concerns, and we would be glad to clarify anything further. Thank you again for your time and valuable feedback.
References
- [1] W. Chen et al. ICLEval: Evaluating In-Context Learning Ability of Large Language Models. COLING 2025.
- [2] N. Muennighoff et al. OLMoE: Open Mixture-of-Experts Language Models. ICLR 2025.
- [3] A. Yang et al. Qwen3 Technical Report. arXiv:2505.09388, 2025.
- [4] M. Chen et al. Evaluating Large Language Models Trained on Code. arXiv:2107.03374, 2021.
- [5] Y. Kuratov et al. BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack. NeurIPS 2024.
- [6] Y. He et al. Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Following. arXiv:2410.15553, 2024.
- [7] J. von Oswald et al. Transformers Learn In-Context by Gradient Descent. ICML 2023.
Kind reminder: happy to clarify any remaining points
Dear Reviewer YdfC,
Thank you again for the detailed review. Your request to focus on W3–W6 shaped our revision: it led us to add two MoE models (Table 5), the code, long-context and multi-turn evaluation (Table 6), the probe-domain comparison with layer overlap (Appendix D), and the pool-width sweep (Appendix F), all reported in our replies above. These additions made the paper noticeably stronger, and we are grateful for the direction.
If any point would benefit from further clarification, we would be glad to respond before the discussion period closes today.
Respectfully,
The Authors
Response to Reviewer YdfC (1/3): probe domain, MoE models
Dear Reviewer YdfC,
Thank you for the detailed review. Since you asked us to focus on W3–W6, we answer those first.
Yes, it can, mainly when the probe comes from a single narrow task family. Our submission included one such comparison, with the public ICLEval corpus [1] (now Table 20), and we have now added more. Our probe mixes seven families, and with this mix the removed layers change little. As the reviewer suggested, the table below lists the removed layers with our synthetic probe suite, with the ICLEval corpus, and with each of our seven families left out in turn (seed 53):
Replacing our suite with the ICLEval corpus changes 0–3 of the removed layers, and the repaired models stay close in quality on average (Table 20). Leaving out any one family changes at most one layer on four backbones, and two on Mistral-7B. A probe built from a single family changes about two of the 7–9 layers, and no single family works well everywhere. For example, "reorder with connective" gives a repaired Avg of 62.2 on Qwen3-8B but 51.4 on Llama-3.1-8B, the best family on the first backbone and the worst on the second (seed 53).
Each family is included only if the unpruned models cannot solve it without demonstrations (Table 9), so the probe mainly tests learning from demonstrations, not knowledge of a domain. On abilities the probe never covers, such as code, 8k-token contexts and multi-turn conversations, the selected models are also the best on average after SFT (Table 6, see W6). We added a short paragraph on these results to Appendix D.
We tested OLMoE-1B-7B [2] under both repairs of Table 2, and the larger Qwen3-30B-A3B [3] as a check at a larger scale. Our method is the best on OLMoE-1B-7B under both repairs and within 0.1 points of the best on Qwen3-30B-A3B. We removed 25% of the layers and repaired the models with rank-8 LoRA on attention and every expert. Qwen3-30B-A3B is repaired with SFT only, since OPD also runs the unpruned 30B teacher at every step and costs far more. The table gives the Avg of Table 2 as the mean ± standard deviation over three seeds, and we added it to the main text as Table 5 (§4.3):
On OLMoE, our method leads FinerCut by 3.3 points and every other criterion by more, and with OPD it is again the best (40.2 against 35.7 for FinerCut). Without the probe our method reduces to ShortGPT, so the gain over ShortGPT is the probe's own contribution, 5.6 points on OLMoE (8.3 with OPD) and 4.5 on Qwen3-30B-A3B. As with the dense models, the best baseline changes from model to model. FinerCut, the strongest baseline in Table 5, is 8.0 points behind the best on Qwen3-8B.