acceptodds
Under review as a conference paper at ICLR 2027

In-Context Learning as a Rehearsal of Recovery for Depth Pruning

Abstract

Depth pruning removes whole transformer blocks and then repairs the model. Existing criteria choose the blocks by the damage a removal does before repair. But they cannot tell which of two equally damaging removals repair will undo, because that depends on whether the remaining layers can still learn. Repairing and testing every candidate is out of reach (removing 9 of 36 layers alone leaves 94 million sets), so practice falls back on heuristics such as dropping a contiguous block of late layers. We instead find a better set with a few forward passes, drawing on the view that in-context learning is an implicit weight update. We therefore score a removal by how much it hurts the model's in-context learning ability. Over 1.6K prune-and-repair runs on five language models, we find that this probe helps predict post-repair quality better than present-damage scores alone, and more so with demonstrations than without, as our view expects. The same runs show why the choice matters. At a fixed budget, the set of removed layers changes post-repair quality far more than the choice among published criteria, and no single position is safe on every backbone. At 25% removal our method has the best five-model average under both repair objectives, even ahead of a baseline that briefly fine-tunes every candidate, for a few percent of pipeline cost. We also extend the study to vision-language models, where most prior depth pruning cuts only the language tower. We find that how the budget is split between the two towers decides which capabilities survive repair.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Multimodal Large Language Model, Depth Pruning, Model Compression

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_YdfCOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes using in-context learning (ICL) ability to estimate how well a model can recover after layer pruning, treating ICL as a rehearsal for repair. The method first uses block influence to select 2T candidate layers, then uses a JS divergence probe with demonstrations to choose T layers for removal, requiring only forward passes during selection. Across different models, the experiments show an average gain of about 2 percentage points over the strongest baseline at 25% layer removal.

Critical strengths

Defining the layer-selection goal in terms of recovery after repair is a relatively novel starting point.

Through many prune-and-repair experiments, partial correlation analysis, and ablations, the authors show that the probe provides a useful signal for layer selection. The experiments also cover multimodal models.

Critical weaknesses
  1. The exact equivalence between ICL and gradient updates holds only under specific linear-attention settings. It does not directly show that in-context adaptation and LoRA repair rely on the same structures in pretrained Transformers. The correlation analysis and demonstration-count ablations support the probe’s predictive value, but do not confirm the proposed mechanism.

  2. The candidate pool is fixed at twice the removal budget without a clear reason for this specific choice. Although the authors test different pool widths on multimodal models, the sensitivity analysis for language models is limited. Comparing performance and selection cost at widths such as 1.5T and 2.5T would strengthen the method.

  3. The method assumes that preserving the ability to learn new rules from demonstrations makes a layer a good candidate for removal. However, the demonstrations cannot cover every domain, so a probe drawn from a limited set of domains can only directly test learning ability within those domains. Layer selection may therefore favor the domains covered by the probe, while its effect on learning in other domains remains unclear.

  4. I suggest adding ICL probe data from more domains and reporting the removed layer indices and their overlap for each, to show whether the layer selection is affected by the probe domain.

  5. MoE is an increasingly common model architecture, so testing the method on MoE models would help establish its broader use. In addition, at removal rates above 25%, the advantage over FinerCut appears limited.

  6. The evaluation mainly covers knowledge questions, instruction following, mathematics, and multi-step reasoning, which are relatively close to the abilities covered by the repair data from Alpaca and the Tülu-3 subset. This is not enough to establish broad capability preservation. The authors should add code, long-context, and multi-turn tasks, as well as tool-use or agent tasks where the original models have these abilities, to test whether the method preserves more complex capabilities.

Questions for authors

The authors have conducted extensive experiments, but the method relies largely on intuition and the novelty is relatively incremental. I hope the authors can address my concerns, especially Weaknesses 3, 4, 5, and 6.

4 comments
AuthorsOpenReview ↗

Response to Reviewer YdfC (1/3): probe domain, MoE models

Dear Reviewer YdfC,

Thank you for the detailed review. Since you asked us to focus on W3–W6, we answer those first.


W3 and W4: Does the probe's domain affect which layers are removed?

Yes, it can, mainly when the probe comes from a single narrow task family. Our submission included one such comparison, with the public ICLEval corpus [1] (now Table 20), and we have now added more. Our probe mixes seven families, and with this mix the removed layers change little. As the reviewer suggested, the table below lists the removed layers with our synthetic probe suite, with the ICLEval corpus, and with each of our seven families left out in turn (seed 53):

Backbone (T)Our suite (Table 2)ICLEval corpus (Table 20)SharedOne family left out: min sharedOne family left out: mean shared
Qwen3-8B (9)2, 26–332, 26–339/98/98.7
Llama-3.1-8B (8)19, 21–23, 25–2819, 21–26, 287/87/87.3
Qwen1.5-7B (8)3, 23–293, 22–27, 297/87/87.7
Mistral-7B (8)21–23, 25, 27–3020–275/86/87.6
Qwen2-7B (7)5, 12, 15, 17, 18, 24, 255, 18–20, 23–254/76/76.7

Replacing our suite with the ICLEval corpus changes 0–3 of the removed layers, and the repaired models stay close in quality on average (Table 20). Leaving out any one family changes at most one layer on four backbones, and two on Mistral-7B. A probe built from a single family changes about two of the 7–9 layers, and no single family works well everywhere. For example, "reorder with connective" gives a repaired Avg of 62.2 on Qwen3-8B but 51.4 on Llama-3.1-8B, the best family on the first backbone and the worst on the second (seed 53).

Each family is included only if the unpruned models cannot solve it without demonstrations (Table 9), so the probe mainly tests learning from demonstrations, not knowledge of a domain. On abilities the probe never covers, such as code, 8k-token contexts and multi-turn conversations, the selected models are also the best on average after SFT (Table 6, see W6). We added a short paragraph on these results to Appendix D.


W5-1: MoE models

We tested OLMoE-1B-7B [2] under both repairs of Table 2, and the larger Qwen3-30B-A3B [3] as a check at a larger scale. Our method is the best on OLMoE-1B-7B under both repairs and within 0.1 points of the best on Qwen3-30B-A3B. We removed 25% of the layers and repaired the models with rank-8 LoRA on attention and every expert. Qwen3-30B-A3B is repaired with SFT only, since OPD also runs the unpruned 30B teacher at every step and costs far more. The table gives the Avg of Table 2 as the mean ± standard deviation over three seeds, and we added it to the main text as Table 5 (§4.3):

25% removedOLMoE-1B-7B (16 layers, 64 experts), SFTOLMoE-1B-7B, OPDQwen3-30B-A3B (48 layers, 128 experts), SFT
Unpruned59.859.876.3
ShortGPT31.1 ± 0.631.9 ± 0.161.2 ± 0.6
Angular29.1 ± 0.930.2 ± 0.364.7 ± 0.5
Reverse order28.0 ± 0.227.0 ± 0.537.6 ± 1.0
FinerCut33.4 ± 0.435.7 ± 0.365.8 ± 0.4
Ours36.7 ± 0.240.2 ± 0.365.7 ± 0.4

On OLMoE, our method leads FinerCut by 3.3 points and every other criterion by more, and with OPD it is again the best (40.2 against 35.7 for FinerCut). Without the probe our method reduces to ShortGPT, so the gain over ShortGPT is the probe's own contribution, 5.6 points on OLMoE (8.3 with OPD) and 4.5 on Qwen3-30B-A3B. As with the dense models, the best baseline changes from model to model. FinerCut, the strongest baseline in Table 5, is 8.0 points behind the best on Qwen3-8B.

AuthorsOpenReview ↗

Response to Reviewer YdfC (2/3): higher removal ratios, code, long-context and multi-turn tasks

W5-2: The advantage over FinerCut

As the reviewer notes, above 25% removal our advantage over FinerCut is small, although our method still has the best mean over the four axes of Table 2 at 37.5% and 50% removal (38.6 vs. 38.2 and 28.8 vs. 28.3, Tables 21–22). The probe itself, however, also improves on FinerCut when it ranks FinerCut's own candidates, as Appendix F (Table 19) shows for SFT. We took the 2T layers with the smallest d₀ (the output divergence that FinerCut measures without demonstrations), where T is the number of layers to remove, and let the probe choose T of them. At 25% removal, this choice beats FinerCut on all five backbones under both repairs (Avg, three seeds):

AvgQwen3Qwen2Qwen1.5LlamaMistralMean
FinerCut, SFT (Table 2)53.739.244.856.846.048.1
d₀ candidates + probe, SFT60.341.845.859.449.351.3
FinerCut, OPD (Table 2)56.539.339.558.842.747.4
d₀ candidates + probe, OPD63.042.139.860.744.250.0

Table 19 reports this comparison as the share of the unpruned IFEval and BBH scores kept, and the table above gives the Avg and adds OPD. For simplicity, our method takes its candidates from block influence alone. When the probe instead ranks the union of the block-influence and FinerCut candidates, its mean rises to 41.4 at 37.5% and 31.0 at 50% removal, against 38.2 and 28.3 for FinerCut (Appendix G, Table 22). We report this as a diagnostic and keep the simpler default of the paper.


W6: Code, long-context and multi-turn tasks

Thank you for this suggestion. We evaluated the 180 repaired checkpoints of Table 2, 90 for each repair, on HumanEval [4] (code), BABILong [5] with 8k-token contexts (long context) and Multi-IF [6] (multi-turn), without any further training. After SFT, our method has the best five-backbone mean on all three. After OPD it is again the best on long contexts and multi-turn chat, while on code FinerCut and DarwinLM are ahead (23.6 and 21.4, against 19.9), a gap that comes mostly from Qwen3-8B. We added the table to the main text as Table 6 (§4.3), with each backbone in Table 37 (Appendix N):

Five-backbone mean (%)HumanEval, SFTHumanEval, OPDBABILong 8k, SFTBABILong 8k, OPDMulti-IF, SFTMulti-IF, OPD
ShortGPT17.716.947.647.338.041.8
Angular14.914.738.637.036.438.8
Reverse order12.910.939.740.926.726.3
FinerCut17.723.642.339.837.241.8
DarwinLM14.121.432.432.534.535.8
Ours20.219.948.849.039.845.3

Two of these benchmarks test abilities that the repair data does not contain. Repair examples are cut at 256 tokens (App. B.2), far shorter than BABILong's 8k-token contexts, and we repair on first turns only, while Multi-IF also scores the second and third turns. We could not run tool-use or agent benchmarks, which need a function-calling harness that our pipeline does not have.

AuthorsOpenReview ↗

Response to Reviewer YdfC (3/3): in-context learning and gradient descent, the size of the candidate pool

We now turn to W1 and W2.

W1: ICL equals gradient descent only for linear attention

As the reviewer notes, the equivalence is exact only for linear attention [7]. §3.1 now presents this view as the hypothesis that led to the probe, and the paper's claims rest on the experiments. The hypothesis makes two predictions that a present-damage view does not, and our experiments support both. The probe's signal is stronger with demonstrations than without them (Table 15), and the probe predicts recovery beyond present damage (Table 12), even among candidates whose present damage is matched, where it orders 73–89% of the pairs correctly.


W2: Why a pool of 2T candidates?

We ran the language models with pool widths from T to all layers, each repaired with three seeds (Table 17 in Appendix F). A width of T leaves the probe no choice and is exactly ShortGPT:

Pool widthQwen3Qwen2Qwen1.5LlamaMistralMean
T (= ShortGPT, Table 2)55.436.342.359.249.548.5
1.5T56.037.044.559.448.949.2
2T (default, Table 2)61.739.444.559.448.950.8
2.5T55.842.545.459.448.950.4
All layers (probe alone)55.942.645.459.448.950.4

Every width from 1.5T to all layers keeps the five-backbone mean between 49.2 and 50.8, above every baseline in Table 2 (48.5). On Llama-3.1-8B and Mistral-7B every width above T selects the same layers as 2T, so those cells repeat Table 2. On Qwen2-7B, where block influence predicts recovery poorly (Table 16), a wider pool helps.

Qwen3-8B is the only backbone where 2T clearly beats both narrower and wider pools, and its selections show why (they are the same in all three seeds). At 1.5T the pool does not contain layer 2, the probe's ninth choice at 2T, so the probe removes layer 25 instead. At 2.5T the pool also admits layer 34, the second-to-last layer. The probe scores it as cheap and removes it in place of layer 2, but removing it causes output-side damage that the probe does not detect and repair does not undo. Without a pool, the probe also removes this layer on Qwen3-8B (Appendix F). §3.2 now says that keeping such layers out is part of the pool's job.

Qwen3-8B shows that the pool can be too narrow or too wide. Over the five backbones, 2T gives the best mean. We fixed 2T before running these experiments and keep it as the default. Selection, including model loading, takes 5.5, 6.2 and 6.7 minutes on one GPU at 1.5T, 2T and 2.5T (five-backbone mean), and the probe passes themselves take 1–2.5 minutes (App. B).


We hope these answers address your concerns, and we would be glad to clarify anything further. Thank you again for your time and valuable feedback.


References

  • [1] W. Chen et al. ICLEval: Evaluating In-Context Learning Ability of Large Language Models. COLING 2025.
  • [2] N. Muennighoff et al. OLMoE: Open Mixture-of-Experts Language Models. ICLR 2025.
  • [3] A. Yang et al. Qwen3 Technical Report. arXiv:2505.09388, 2025.
  • [4] M. Chen et al. Evaluating Large Language Models Trained on Code. arXiv:2107.03374, 2021.
  • [5] Y. Kuratov et al. BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack. NeurIPS 2024.
  • [6] Y. He et al. Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Following. arXiv:2410.15553, 2024.
  • [7] J. von Oswald et al. Transformers Learn In-Context by Gradient Descent. ICML 2023.
AuthorsOpenReview ↗

Kind reminder: happy to clarify any remaining points

Dear Reviewer YdfC,

Thank you again for the detailed review. Your request to focus on W3–W6 shaped our revision: it led us to add two MoE models (Table 5), the code, long-context and multi-turn evaluation (Table 6), the probe-domain comparison with layer overlap (Appendix D), and the pool-width sweep (Appendix F), all reported in our replies above. These additions made the paper noticeably stronger, and we are grateful for the direction.

If any point would benefit from further clarification, we would be glad to respond before the discussion period closes today.

Respectfully,

The Authors

Reviewer_rwKpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Existing depth-pruning criteria score the damage a layer removal does before repair, yet two equally damaging cuts can recover very differently. The authors treat in-context learning as a cheap rehearsal of that recovery process and score candidates by how much they impair the model’s ability to learn a new rule from demonstrations. A present-damage metric first shortlists layers, the ICL probe then ranks them with only forward passes. Across 1.6K prune-and-repair runs on five 7–8B models the method posts the strongest average recovery at 25 % removal, and the same probe also guides joint vision–language tower pruning.

Critical strengths

The strongest part of the paper is the finding that preserving in-context learning helps predict how well a pruned model will recover after fine-tuning. The authors support this with an extensive study of over 1,600 prune-and-repair runs, including controls showing that demonstrations add predictive information beyond standard damage scores. This gives the proposed criterion a convincing empirical basis. The method is also simple and inexpensive: it uses only forward passes, adds about 3% to the overall pipeline cost, and achieves the best average across five models under both repair objectives at 25% layer removal.

Critical weaknesses

My main concern is the gap between the interpretation of the probe and what the experiments establish. The results show that preserving predictions on demonstration-based tasks helps predict recovery, but they do not establish that the probe measures the ability of the remaining weights to learn during fine-tuning. Adding demonstrations supports the usefulness of the signal, but does not resolve this distinction, which matters because the connection between in-context learning and repair is the paper’s central explanation. The practical gains are also uneven: although the method achieves the best average, it falls behind existing criteria on several backbones

Questions for authors

Could you clarify how reliably the probe predicts recovery within each backbone, particularly when comparing candidates with similar immediate damage? The pooled correlations are encouraging, but the weaker results on Qwen2-7B make it important to understand when the signal can guide selection. Also, on Qwen2-VL, does the gap to the strongest baseline mainly come from choosing the split between vision and language layers, or from choosing layers within each tower? This distinction would help assess whether the limitation lies in the probe itself or in how it is used for selection.

4 comments
AuthorsOpenReview ↗

Response to Reviewer rwKp (1/2): candidates with similar damage, the Qwen2-VL split

Dear Reviewer rwKp,

Thank you for the precise summary and the careful questions. We answer the two questions first.


Q1: How reliably does the probe predict recovery within a backbone, for candidates with similar immediate damage?

Among such candidates, the probe still orders most pairs correctly on every backbone. Within each backbone, we took the pairs of removal sets from Table 12 whose present damage is close by two measures, block influence (BI, the score that ShortGPT uses) and d₀ (the output divergence that FinerCut measures). We kept the closest 20% of pairs and counted how often each signal orders the two sets correctly by their quality after repair, leaving out pairs whose repaired quality differs by less than 3% of the unpruned score.

BackbonePairsProbe (90% bootstrap CI)BId₀ρ(probe, outcome)
Qwen3-8B2940.81 (0.68–0.89)0.600.490.83
Qwen2-7B2370.82 (0.70–0.91)0.400.450.53
Qwen1.5-7B3890.84 (0.75–0.90)0.670.440.78
Llama-3.1-8B2640.89 (0.79–0.96)0.700.450.94
Mistral-7B1580.73 (0.59–0.86)0.680.530.90

The probe orders 73–89% of the pairs correctly on every backbone, while BI reaches 40–70% and d₀ is close to chance.

On Qwen2-7B, where block influence is the weakest predictor of recovery (Spearman ρ = 0.23 with repaired quality), the probe is still the best signal (ρ = 0.53, against 0.41 or below for every other signal) and orders 82% of the matched pairs correctly. In Table 2, our method is held back on Qwen2 by the candidate pool, which comes from block influence. Given all layers as candidates, the probe reaches 42.6 Avg on Qwen2 under SFT, above the best baseline (DarwinLM, 42.3) and our 39.4 with BI candidates. On the other backbones block influence tracks recovery well (ρ = 0.51–0.87), which is why it remains our default. We added this analysis and the table to Appendix E.


Q2: On Qwen2-VL, does the gap to the strongest baseline come from the split or from the choice within each tower?

On Qwen2-VL-2B, the gap comes from how the probe divides the removals between the two towers, and inside each tower the probe picks better layers than Angular. To separate the two, we paired each split of the removed layers between the vision and language towers with each way of choosing the layers inside the towers. The numbers are the Avg of Table 36 on one seed (Table 32):

Split between towersWithin towers: AngularWithin towers: probe
2B, Angular's split (ViT 6 / LM 2)42.5 (Angular)44.2
2B, probe's split (ViT 8 / LM 0)40.841.3 (Ours)
7B, Angular's split (ViT 8 / LM 0)49.5 (Angular)49.5 (same layers)
7B, probe's split (ViT 7 / LM 1)50.249.5 (Ours)

On Qwen2-VL-2B, the probe's choice within the towers beats Angular's under both splits, by 1.7 and 0.5 points. This suggests that the limitation lies in how the probe's scores are compared across the two towers, not in how the probe ranks the layers within a tower. On Qwen2-VL-7B, our method and Angular are tied (49.54 vs. 49.51), the four cells lie within 0.7 points on one seed, and with Angular's split the probe picks the same eight layers. §5 now says that the 2B gap comes from the split between the towers, and the table is in Appendix L.3 (Table 32).

AuthorsOpenReview ↗

Response to Reviewer rwKp (2/2): interpretation of the probe, gains across backbones

W1: The interpretation of the probe

Thank you for drawing this distinction. Our claims rest on experiments. If the probe tracks how well the pruned model can still learn, two things should hold that a present-damage view does not predict, and our experiments support both.

  1. The probe's extra signal should come from the demonstrations. Once block influence (BI) is known, the probe's correlation with IFEval after repair is 0.02 without demonstrations and 0.33 with two (Figure 3, right). On every backbone, the probe also predicts recovery better with two or more demonstrations than with none (Table 15), and FinerCut's d₀, the same divergence measured without demonstrations, predicts it less well.
  2. The probe should carry information about recovery that present damage does not. With BI held fixed, its partial correlation with repaired quality is 0.67 on IFEval and 0.45 on BBH (Table 12). Within each backbone, holding BI or d₀ fixed leaves partial correlations between 0.40 and 0.94, and among candidates with matched present damage the probe orders 73–89% of the pairs correctly (Q1 above).

The view behind the probe, that in-context learning acts like an implicit gradient step (Eq. 1) [1, 2], is exact for linear attention but not proven for large language models. §3.1 now presents it as the hypothesis that led to the probe.


W2: The gains are uneven across backbones

We answer this in the general response, under "Robustness across backbones". In short, the baselines that beat our method differ from backbone to backbone, none of them does so on more than two of the five, and our method is never more than 2.9 points behind the best (5.5 under OPD). We added this comparison to §4.3 as Table 3.


We hope these answers address your concerns, and we would be glad to clarify anything further. Thank you again for your time and valuable feedback.


References

  • [1] J. von Oswald et al. Transformers Learn In-Context by Gradient Descent. ICML 2023.
  • [2] D. Dai et al. Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers. Findings of ACL 2023.

I thank the authors for their thorough and thoughtful rebuttal. The responses have satisfactorily addressed my concerns, and I am therefore happy to maintain my original positive assessment and retain my score.

AuthorsOpenReview ↗

Thank you for the follow-up

Dear Reviewer rwKp,

Thank you for the follow-up and for confirming your assessment. We are glad the clarifications addressed your concerns.

Respectfully,

The Authors

Reviewer_J1SPOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a recovery oriented view to depth pruning of LLMs and VLMs via in-context learning (ICL). The authors define an ICL probe using Jensen-Shannon divergence between the full model and depth pruned model, coupeld with a ShortGPT style block-influence score. The method identifies which layers of a model cause the least damage during removal and are the easiest to recover from.

Critical strengths
  • Correlation of ICL proxy over 797 removal sets across 5 different base models at 25% pruning shows significant alignment with recovery potential. Spearman correlation shows that ICL predicts post repair IFEval and BBH better than BI scores.
  • Under both SFT and OPD, ICL has the highest mean of five average and demonstrates extension to VLMs for both vision tower and language model pruning.
Critical weaknesses
  • The individual gains on each model are modest, with 4/5 (SFT) and 3/5 (OPD) under performing compared to other baselines. This performance also holds when using more fine tuning steps as shown in Table 18/19.
  • The scope of the method is limited considering it only targets depth pruning and does not compare to width-pruning or hybrid depth-width pruning regimes and methods ([1], [2], [3], [4]). This would allow readers to see the impact of ICL at higher pruning ratios and different pruning granularity.

References:

[1] Cai, Ruisi, et al. "Flextron: Many-in-one flexible large language model." arXiv preprint arXiv:2406.10260 (2024).

[2] Sreenivas, Sharath Turuvekere, et al. "Llm pruning and distillation in practice: The minitron approach." arXiv preprint arXiv:2408.11796 (2024).

[3] Malettira, Prajna G., et al. "TraceNAS: Zero-shot LLM Pruning via Gradient Trace Correlation." arXiv preprint arXiv:2602.02891 (2026).

[4] Xia, Mengzhou, et al. "Sheared llama: Accelerating language model pre-training via structured pruning." International Conference on Learning Representations. Vol. 2024. 2024.

Questions for authors
  • Can the authors provide comparisons to existing training-free recovery proxies [3] which have been applied for LLM pruning and extend their analysis to width pruning and higher sparsity regimes?
  • It is unclear whether the gains seen on the depth pruned VLMs comes from the per-tower pool or from the ICL probe itself. Could the authors elaborate on this?
  • The correlation evidence is relevant, however it does not compare to a wider set of zero-cost proxies such as [3], [5], [6], [7], etc.

References:

[3] Malettira, Prajna G., et al. "TraceNAS: Zero-shot LLM Pruning via Gradient Trace Correlation." arXiv preprint arXiv:2602.02891 (2026).

[4] Xia, Mengzhou, et al. "Sheared llama: Accelerating language model pre-training via structured pruning." International Conference on Learning Representations. Vol. 2024. 2024.

[5] Mellor, Joe, et al. "Neural architecture search without training." International conference on machine learning. PMLR, 2021.

[6] Dong, Peijie, et al. "Lpzero: Language model zero-cost proxy search from zero." Findings of the Association for Computational Linguistics: EMNLP 2024. 2024.

[7] Dong, Peijie, et al. "Pruner-zero: Evolving symbolic pruning metric from scratch for large language models." arXiv preprint arXiv:2406.02924 (2024).

5 comments
AuthorsOpenReview ↗

Response to Reviewer J1SP (1/3): zero-cost proxies

Dear Reviewer J1SP,

Thank you for the detailed review and the pointers to related work, all of which the revised Related Work now discusses.


Q1 and Q3: Comparison with zero-cost proxies such as TraceNAS

We compared three of the four proxies the reviewer suggested (TraceNAS [3], NASWOT [5] and Pruner-Zero [7]), together with SNIP [8] and gradient norm, by how well each predicts the model's quality after repair. On the 550 pruned and repaired removal sets of Table 12, the table gives the Spearman correlation between each signal, computed before repair, and the repaired quality (the mean share of the IFEval and BBH scores kept). Block influence (BI, the score that ShortGPT uses) and d₀ (the divergence without demonstrations that FinerCut uses) are the two depth-pruning scores, and the rows below them are zero-cost proxies.

SignalQwen3Qwen2Qwen1.5LlamaMistralMeanWorstPooled
ICL probe (ours)0.830.530.780.940.900.800.530.68
Block influence0.510.230.680.870.870.630.230.53
d₀ (FinerCut)0.600.390.520.470.470.490.390.32
SNIP0.73−0.050.810.870.800.63−0.050.28
TraceNAS0.700.340.670.480.740.590.340.61
Pruner-Zero0.67−0.240.470.350.350.32−0.240.49
NASWOT0.490.41−0.33−0.24−0.59−0.05−0.590.00
Gradient norm−0.21−0.24−0.33−0.13−0.12−0.21−0.33−0.08

Our probe has the highest correlation on average (0.80, against at most 0.63 for any other signal), on its worst backbone (0.53) and when the removal sets of all five backbones are put into one correlation (0.68, the Pooled column). Among the zero-cost proxies, TraceNAS is the most robust in this comparison. It is second only to our probe in the pooled correlation, and it has the highest worst-backbone correlation of the proxies. The other proxies were designed for different questions, ranking architectures (NASWOT) or scoring single weights (SNIP, Pruner-Zero), so whole-layer removal is outside their intended use, and we aggregated their scores over the removed layers as described in Appendix E.8. We could not include LPZero [6], because its searched proxy is defined on the elastic-width search space of LoNAS [11] and we found no faithful way to apply it to whole-layer removal. We added this comparison to Appendix E.8 (Table 16), and Related Work now has a short paragraph on zero-cost proxies. We answer the questions on width pruning and higher sparsity under W2, in our third comment.

AuthorsOpenReview ↗

Response to Reviewer J1SP (2/3): where the VLM gains come from, results on each backbone

Q2: In the VLMs, does the gain come from the per-tower pool or from the probe?

The gain comes mainly from the probe, and the per-tower pool is what lets the probe decide how many layers each tower gives up. In our method, block influence (BI) only proposes candidates. It picks the 2T layers that change the hidden state least, where T is the number of layers to remove, and the probe decides which T of them to remove. Taking the top T by BI instead gives exactly ShortGPT, so the difference between our method and ShortGPT is the effect of the probe alone (now Table 4 in the main text).

On a vision-language model the probe decides two things, (1) how the removed layers are split between the vision and language towers and (2) which layers to remove inside each tower. To separate the two, each row of the table fixes the split, ShortGPT's or the probe's, and each column fixes how the layers inside each tower are chosen, by BI or by the probe. The cells are the Avg of Table 7 after repair (InternVL3.5, three seeds), and the cells marked ShortGPT and Ours are the two methods themselves:

Split between towersWithin towers: BIWithin towers: probe
2B, ShortGPT's split (ViT 0 / LM 8)37.0 (ShortGPT)37.0
2B, probe's split (ViT 8 / LM 0)46.247.0 (Ours)
8B, ShortGPT's split (ViT 1 / LM 7)48.4 (ShortGPT)52.2
8B, probe's split (ViT 7 / LM 1)56.156.5 (Ours)

Ranking the per-tower pool by BI alone gives exactly ShortGPT's selection, so the pool alone does not change which layers are removed. Most of the gain comes from the probe's split, which adds 4.3 to 10.0 points over ShortGPT's split on both models, whichever way the layers inside the towers are chosen. With ShortGPT's split held fixed, the probe's choice inside the towers adds another 3.8 points on 8B. The pool is still needed for the split, because a shared pool is dominated by the tower whose layers look cheaper to remove, and giving each tower the same number of candidates lets the probe compare the two. §5 now describes this breakdown in the paragraph "Where the removals are spent", and the table is in Appendix L.3 (Table 32). In this table HallusionBench is scored by one judge model for all rows, so the Avg can differ from Table 7 by up to 0.11.


W1: The gains on individual backbones are modest

As the reviewer notes, some baseline beats our method on four of the five backbones under SFT and on three under OPD. As we explain in the general response, these are different baselines, none of them beats ours on more than two backbones, and each falls far behind on another. Under SFT, for example, Angular is the best on Llama-3.1-8B but 16.0 points behind the best on Qwen1.5-7B, while our method is never more than 2.9 points behind the best (5.5 under OPD, Table 3). Since one usually cannot tell in advance which criterion suits a new model, and the repair is the expensive step, we think staying close to the best everywhere matters more than winning on a given backbone.

The gaps between criteria are much larger than seed noise, as the standard deviation of our Avg over three seeds is at most 0.7 points. Our method also has the best mean rank under both repairs (2.0, against 2.8 for the next criterion). We think this robustness comes from the probe tracking recovery itself. In the table of our first comment, it has the highest correlation with repaired quality on four of the five backbones, and even its worst backbone is above the worst of every other signal.

We see the same pattern at the other repair budgets the reviewer mentions (Tables 24–25, one seed and four baselines). At half, the original and double the budget, our method stays within 3 points of the best on every backbone, while every baseline falls at least 8 points behind on some backbone (mean of IFEval, GSM8K and BBH).

AuthorsOpenReview ↗

Response to Reviewer J1SP (3/3): width and hybrid pruning, higher removal ratios

W2 and Q1: Width pruning, hybrid methods and higher pruning ratios

Width and hybrid pruning are complementary to depth pruning. We chose to study depth because removing layers usually speeds up decoding more than narrowing them [2, 10], and we checked this ourselves. We pruned Llama-3.1-8B in width with the official Llama-3 recipe of LLM-Pruner [9] and measured its decoding speed against our depth-pruned model in the same engine. We used HF generate, because vLLM cannot load LLM-Pruner's layer-wise widths (Table 43):

Llama-3.1-8BParamsDecode speed, batch 1 / 8
Unpruned8.03B1.00× / 1.00×
Depth, 25% of layers (our Table 2 set)6.29B1.28× / 1.29×
Width, LLM-Pruner ratio 0.256.61B0.99× / 0.99×
Width, LLM-Pruner ratio 0.31 (matched params)6.34B1.00× / 0.93×

In our measurement, at a matched parameter count, removing 25% of the layers makes decoding about 1.28× faster, while width pruning gives no speedup. The Minitron study [2] also finds depth pruning faster on this model (2.7× versus 1.8× throughput), while width pruning keeps more accuracy. For depth pruning, a central question is which layers to remove, and this is the question our probe addresses.

Minitron [2], Flextron [1] and Sheared LLaMA [4] combine pruning with distillation or continued pre-training on billions of tokens (94B tokens for Llama-3.1-Minitron-4B), so a direct comparison would mainly reflect their much larger training budgets. Instead, we took the part of Minitron that chooses which layers to remove and ran it under our protocol. It removes the contiguous block of layers whose removal keeps 5-shot Winogrande accuracy highest. We repaired the pruned models as in Table 2, with three seeds (Table 44):

Avg, 25% removedQwen3LlamaQwen2Qwen1.5MistralMeanLargest gap
Minitron's depth rule56.060.922.242.449.346.220.1
Ours61.759.439.444.548.950.82.9

Minitron's rule beats ours on Llama-3.1-8B and Mistral-7B. Like the other rules that measure present damage, however, it falls far behind on another backbone. On Qwen2-7B it removes the last seven layers, and the repaired model reaches only 22.2 points, against 39.4 for ours. The rule also uses five demonstrations, but on Winogrande, a task the model already solves without them, so its score shows how much of the existing ability survives rather than how well the model learns a new rule. Related Work now discusses Minitron, Flextron and Sheared LLaMA and the speed advantage of depth pruning, and Appendix O adds both comparisons (Tables 43–44).

Higher pruning ratios

The probe's contribution grows as more layers are removed, most clearly on BBH, where it keeps 5.6, 7.0 and 15.0 more percentage points of the unpruned score than block influence alone at 25%, 37.5% and 50% removal (Table 18). At 37.5% and 50% removal, our method still has the best mean over the four axes of Table 2, slightly ahead of FinerCut (38.6 vs. 38.2 and 28.8 vs. 28.3, Tables 21–22). For simplicity, our method takes its candidates from block influence alone. When the probe instead ranks the union of the block-influence and FinerCut candidates, its mean rises to 41.4 at 37.5% and 31.0 at 50% removal, against 38.2 and 28.3 for FinerCut (Appendix G, Table 22). We report this as a diagnostic and keep the simpler default of the paper.


We hope these answers address your concerns, and we would be glad to clarify anything further. Thank you again for your time and valuable feedback.


References ([1]–[7] as numbered in the review, [8]–[11] added)

  • [1] R. Cai et al. Flextron: Many-in-One Flexible Large Language Model. ICML 2024.
  • [2] S. T. Sreenivas et al. LLM Pruning and Distillation in Practice: The Minitron Approach. arXiv:2408.11796, 2024.
  • [3] P. G. Malettira et al. TraceNAS: Zero-shot LLM Pruning via Gradient Trace Correlation. arXiv:2602.02891, 2026.
  • [4] M. Xia et al. Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning. ICLR 2024.
  • [5] J. Mellor et al. Neural Architecture Search without Training. ICML 2021.
  • [6] P. Dong et al. LPZero: Language Model Zero-cost Proxy Search from Zero. Findings of EMNLP 2024.
  • [7] P. Dong et al. Pruner-Zero: Evolving Symbolic Pruning Metric from Scratch for Large Language Models. ICML 2024.
  • [8] N. Lee et al. SNIP: Single-shot Network Pruning based on Connection Sensitivity. ICLR 2019.
  • [9] X. Ma et al. LLM-Pruner: On the Structural Pruning of Large Language Models. NeurIPS 2023.
  • [10] B.-K. Kim et al. Shortened LLaMA: A Simple Depth Pruning for Large Language Models. ICLR 2024 Workshop on Mathematical and Empirical Understanding of Foundation Models.
  • [11] J. P. Munoz et al. LoNAS: Elastic Low-Rank Adapters for Efficient Large Language Models. LREC-COLING 2024.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response (to all reviewers)

We sincerely thank all reviewers for their detailed reviews and constructive comments.

We are encouraged that the reviewers found the recovery-based view of layer selection a relatively novel starting point (YdfC), and the finding that preserving in-context learning helps predict recovery the strongest part of the paper, with a convincing empirical basis (rwKp). The reviewers also noted that the probe predicts post-repair quality better than block-influence scores (J1SP), that the method is simple and inexpensive (rwKp), and that the experiments are extensive (rwKp, YdfC) and extend to vision-language models (J1SP, YdfC).

We have revised the paper based on these reviews. Below, we first address the point raised by two reviewers, robustness across backbones, and then summarize the changes in the revised PDF. The replies to each reviewer follow in separate comments.


Robustness across backbones (J1SP, rwKp)

As J1SP and rwKp note, our method is not the best on every backbone. We would like to emphasize, however, that it stays close to the best criterion on every language model we tested, including the two mixture-of-experts models we added in Table 5. It is never more than 2.9 points behind the best criterion on any backbone (5.5 under OPD), while every baseline falls at least 6.3 points behind on some backbone. We consider this robustness important in practice, because practitioners usually cannot tell in advance which criterion suits their model, and a criterion that does well on every model is the safer choice.

No single baseline is consistently better, since each beats ours on at most two backbones and falls further behind on another. The table below gives the SFT Avg of Tables 2 and 5 on each model, the mean over the seven models, and, in the last column, each criterion's largest gap to the best criterion and where it occurs:

SFT Avg|Qwen3-8BQwen2Qwen1.5LlamaMistralOLMoEQwen3-30B-A3B|MeanLargest gap ↓
ShortGPT|55.436.342.359.249.531.161.2|47.96.3‾ (61.7 - 55.4 on Qwen3-8B)\underline{6.3}\ \text{(61.7 - 55.4 on Qwen3-8B)}
Angular|56.1‾\underline{56.1}36.028.860.849.4‾\underline{49.4}29.164.7|46.416.0 (44.8 - 28.8 on Qwen1.5)16.0\ \text{(44.8 - 28.8 on Qwen1.5)}
Reverse order|53.222.136.953.145.528.037.6|39.528.2 (65.8 - 37.6 on Qwen3-30B-A3B)28.2\ \text{(65.8 - 37.6 on Qwen3-30B-A3B)}
FinerCut|53.739.244.856.846.033.4‾\underline{33.4}65.8|48.5‾\underline{48.5}8.0 (61.7 - 53.7 on Qwen3-8B)8.0\ \text{(61.7 - 53.7 on Qwen3-8B)}
DarwinLM|47.742.341.048.040.6––|–14.0 (61.7 - 47.7 on Qwen3-8B)14.0\ \text{(61.7 - 47.7 on Qwen3-8B)}
Ours|61.739.4‾\underline{39.4}44.5‾\underline{44.5}59.4‾\underline{59.4}48.936.765.7‾\underline{65.7}|50.92.9 (42.3 - 39.4 on Qwen2)\mathbf{2.9}\ \text{(42.3 - 39.4 on Qwen2)}

Bold: best; underlined: second best. Mean is over the seven backbones.

Under OPD, the baselines' largest gaps are 7.6 to 25.4 points, while ours is 5.5. §4.3 now gives these gaps for the five backbones of Table 2 as Table 3, and also says that our method is not first on every backbone.


Changes in the revised PDF

In the revision, new or changed text is in blue, and table numbers in our replies refer to the revised version. All new experiments follow the original protocol, with three seeds unless stated otherwise. We revised the paper as follows:

Main text

  • §3.1: Eq. 1 stated as a hypothesis whose predictions we test [rwKp, YdfC]
  • §3.2: one sentence on what the candidate pool does [YdfC]
  • §4.3, Table 3: each criterion's largest gap to the best criterion [J1SP, rwKp]
  • §4.3, Table 4: the probe ablation, moved up from the appendix [J1SP, rwKp]
  • §4.3, Tables 5–6: two mixture-of-experts models, and code, long-context and multi-turn tasks [YdfC]
  • §5: where the gains on vision-language models come from [J1SP, rwKp]
  • Related Work: the suggested references and zero-cost proxies [J1SP]

Appendix

  • D: how much the probe's task domain matters [YdfC]
  • E: candidates with matched present damage [rwKp]
  • E.8, Table 16: zero-cost proxies, including TraceNAS [J1SP]
  • F, Table 17: the width of the candidate pool [YdfC]
  • G, Table 22: a diagnostic with a larger candidate pool at 37.5% and 50% removal [J1SP, YdfC]
  • L.3, Table 32: the split between the towers and the choice inside each tower [J1SP, rwKp]
  • N, Table 37: the results of Table 6 for each backbone, and how these benchmarks are scored [YdfC]
  • O, Tables 43–44: decoding speed of width- and depth-pruned models, and Minitron's depth rule [J1SP]

We would be glad to discuss any aspect of the paper that may need further clarification. Thank you again for your time and valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.