acceptodds
Under review as a conference paper at ICLR 2027

Budget-Aware Layer Pruning for Large Language Models

Abstract

Layer pruning is attractive for large language model deployment because it yields practical gains in latency and throughput. The key challenge is selecting which layers to remove under a pruning budget , where denotes the number of layers to remove. Many existing methods use a single, budget-independent ranking and select the top- layers, imposing a nested-prefix constraint across pruning budgets. Through exhaustive search on LLaMA3.1-8B, we show that optimal pruning sets can be non-nested across budgets, exposing a limitation of fixed global rankings. Motivated by this finding, we shift layer scoring from intrinsic importance to a budget-aware continuation value and formulate layer pruning as a budget-conditioned finite-horizon Markov decision process. We approximate this value using Double Dueling DQN, enabling a single scorer to construct pruning sets conditioned on both the target budget and the current pruning context. Experiments demonstrate better language modeling quality and downstream accuracy than strong depth-pruning baselines at the same pruning budget, and greater robustness under aggressive pruning. Code is available at https://anonymous.4open.science/r/Budget-Aware-Layer-Pruning-B2CD.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
pruning, large language models, layer pruning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qnQsOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper formulate layer pruning as a budget-conditioned finite-horizon Markov decision process and approximate this value using Double Dueling DQN, enabling a single scorer to construct pruning sets conditioned on both the target budget and the current pruning context. Extensive experiments have demonstrated the effectiveness of the proposed method.

Critical strengths
  • The experiments in the paper are very thorough.
  • It's quite interesting to formulate the layer pruning problem as a budget-constrained finite-term Markov decision process.
Critical weaknesses
  • Figure 2 appears to be a screenshot from a PowerPoint presentation; it's recommended to modify it as it contains small red lines.
  • Will the layers selected differ depending on the calibration set? An ablation experiment is missing.
  • There is a lack of theoretical proof for the effectiveness of the proposed method.
Questions for authors
  • How do the pruning costs and time required by this method compare to other baselines?
5 comments
AuthorsOpenReview ↗

Response 1/3

We thank the reviewer for the positive assessment, and for finding the experiments thorough and the budget-conditioned MDP formulation interesting. We address the calibration question, the theoretical justification, the cost comparison and Figure 2 below.

AuthorsOpenReview ↗

Response 2/3

C1. Do the selected layers differ with the calibration set? (W2)

Concern. Whether the selected layers depend on the calibration set; an ablation appears to be missing.

Response. The selected sets can differ, but the resulting quality is nearly unchanged. We show this in two ways: (a) the sampling ablation already in Appendix G of the submission (Figures 3–5, Qwen3-8B), which our main text did not point to clearly enough; and (b) a new experiment that replaces the calibration corpus itself (BookCorpus → PTB) on LLaMA3.1-8B, where log-PPL changes by at most 0.03 at every budget.

Evidence (a): sampling of the calibration data (Appendix G). We vary the calibration data along three dimensions around the default (pool of 64 BookCorpus sequences, 4 sequences sampled per episode, 512 tokens each). Each cell is log-PPL on WT2 / C4.

Calibration configurationK=4K=8K=12K=16
Default (pool 64, 4 samples, length 512)2.38 / 2.862.85 / 3.283.50 / 3.824.41 / 4.59
Pool size 162.37 / 2.872.76 / 3.203.56 / 3.844.45 / 4.67
Pool size 2562.34 / 2.862.78 / 3.173.57 / 3.794.39 / 4.61
1 sample per episode2.38 / 2.912.82 / 3.223.50 / 3.844.49 / 4.81
16 samples per episode2.34 / 2.862.79 / 3.173.57 / 3.794.40 / 4.65
Sequence length 1282.36 / 2.862.78 / 3.173.63 / 3.834.58 / 4.70
Sequence length 20482.34 / 2.862.78 / 3.173.41 / 3.674.31 / 4.58
  1. Sets can differ. PPL differs between configurations at every budget, so the sets are not always identical; some coincide (at K=4, pool 256, 16 samples and length 2048 all give 2.34 / 2.86).
  2. Small effect at mild budgets, larger at aggressive ones. All configurations are within 0.04 (WT2) / 0.05 (C4) log-PPL at K=4 and 0.09 / 0.11 at K=8. At K=12 and K=16 the WT2 spread grows to 0.22 and 0.27, mostly from sequence length (128 → 2048 tokens: 3.63 → 3.41 at K=12, 4.58 → 4.31 at K=16). Pool size, the dimension closest to "a different calibration set", changes log-PPL by at most 0.11 despite a 16× change.
  3. Interpretation. At mild ratios many sets are nearly equivalent (on LLaMA3.1-8B at K=4, the exhaustive optimum and the best nested set differ by 0.55 PPL, Table 6), so different calibration data can pick different layers with almost identical quality. At large budgets the continuation value is harder to estimate (seed variance also grows, Table 8) and longer sequences help.
  4. Robustness is built in. State features are computed on randomly sampled calibration slices (Sec. 3.4), so the scorer is trained over calibration draws rather than one fixed estimate; across seeds, which also change the calibration batch order, the sets largely coincide at K≤8 (Appendix B). The default configuration was not tuned in our favor: at K=8 it is the worst setting in each of the three figures.

Evidence (b): a different calibration corpus (new). The ablations above vary how data is drawn from BookCorpus. To test the source corpus itself, we retrained the scorer on LLaMA3.1-8B with PTB as the calibration corpus, keeping everything else unchanged, and compare it with the default BookCorpus calibration. Both settings use a calibration pool of 64 sequences of 512 tokens each, with 4 sequences sampled per episode. BookCorpus (fiction books) and PTB (Wall Street Journal news) differ in domain and style, and neither is an evaluation set. Each cell is log-PPL (lower is better); † marks the PTB-calibrated runs.

K (pruning ratio)BookCorpus: WT2 log-PPL ↓PTB: WT2 log-PPL ↓BookCorpus: C4 log-PPL ↓PTB: C4 log-PPL ↓
4 (12.50%)2.162.17†2.642.64†
8 (25.00%)2.662.65†3.073.08†
12 (37.50%)3.523.54†3.603.61†
16 (50.00%)5.145.17†4.894.92†
  1. Changing the corpus has almost no effect. Across all eight cells, the absolute difference is at most 0.03 log-PPL (about 3% in PPL), and the sign is mixed at small budgets (WT2 at K=8: PTB is 0.01 lower). This is smaller than the spread across the sampling configurations in (a) at K≥12 (0.22 / 0.27 on WT2).
  2. The ranking against baselines does not depend on the corpus. With PTB calibration, WT2 log-PPL at K=12 is 3.54 (PPL ≈ 34.5) and at K=16 is 5.17 (PPL ≈ 176), still well ahead of the best baselines in the main results (BlockPruner 56.96 at K=12, Prune&Comp 236.21 at K=16 on WT2). The largest change (+0.03 at K=16) matches the trend in (a): the aggressive regime is the most sensitive to calibration, yet still only marginally.
AuthorsOpenReview ↗

Response 3/3

C2. Theoretical justification (W3)

Concern. The paper lacks a theoretical argument for the effectiveness of the method.

Response. We agree that the consequences of our formal result (Eq. 4) should be stated explicitly. We give two propositions that follow from it, a provable limitation of fixed rankings, and an honest account of what cannot be proved.

Evidence.

  1. Proposition 1 (objective alignment). With the telescoping reward and γ=1, the return of any K-step trajectory equals −D(M_K, M_0) and depends only on the final set (Eq. 4). Since the MDP is deterministic with finite horizon, the greedy rollout w.r.t. Q* returns a set S*_K minimizing D over all feasible K-subsets, for every K. The sequential form introduces no order bias.
  2. Proposition 2 (error propagation). If |q(s,a) − Q*(s,a)| ≤ ε on states reachable under budget K, and Ŝ_K is the greedy rollout of q, then D(M_Ŝ, M_0) ≤ D(M_S*, M_0) + 2Kε. Sketch: at each step Q*(s,a*) ≤ q(s,a*) + ε ≤ q(s,â) + ε ≤ Q*(s,â) + 2ε, so V*(s_t) − Q*(s_t, â_t) ≤ 2ε; with deterministic transitions and V* = 0 at the terminal state, summing V*(s_t) − r_t − V*(s_{t+1}) over K steps gives exactly the return gap. As D is log-PPL, this is at most a factor exp(2Kε) in calibration PPL; it is the finite-horizon analogue of Singh and Yee (1994).
  3. Fixed rankings are provably suboptimal across budgets. Any budget-agnostic ranking yields nested sets; if optimal sets at two budgets are not nested, no ranking is optimal at both. This occurs in practice: on LLaMA3.1-8B the optimum is {10,25,26} at K=3 and {11,12,25,26} at K=4 (Table 6); on LLaMA3.2-1B, {3,4,5,6,7,8,12} at K=7 and {3,4,5,6,7,11,12,14} at K=8 (Table 12). A ranking optimal at the smaller budget loses at least 6.39% and 20.05% at the larger one, regardless of the importance metric.
  4. The policy class contains existing approaches. A fixed ranking ignores state and K; SLEB-style greedy removal maximizes only the immediate reward. Both are special cases, so the optimum of our formulation is never worse on the calibration objective, and strictly better when optimal sets are not nested.
  5. What cannot be proved. Cardinality-constrained minimization of a general set function is hard; this objective is not additive (otherwise optimal sets would be nested) and there is no reason to expect submodularity. Deep Q-learning with function approximation has no general convergence guarantee, so ε cannot be bounded a priori. We therefore certify empirically where the optimum is computable: within 0.55% of the exhaustive optimum at K=4 on LLaMA3.1-8B (Table 6), and better than beam search on the true objective at K≥8 (Table 13).

Scope: both propositions hold for the unclipped reward over the feasible action set (protected layers excluded); reward clipping and early termination are stability devices, and exact alignment holds on trajectories where they are not triggered.

C3. Pruning cost and time compared with baselines (Question)

Concern. How do the pruning cost and time compare with the baselines?

Response. Our offline cost is 2 A100 GPU-h: 8× SLEB and Prune&Comp, 2× BlockPruner, and about one eighth of Reassessing. It is paid once per base model for all budgets and adds nothing at inference.

Evidence. Table 15 (Appendix I), LLaMA3.1-8B, PPL averaged over WT2 and C4:

MethodK=4K=8K=12K=16Offline costMulti-budget
ShortGPT16.414145.3254885.812042.07secondsyes
Shortened LLaMA11.7022.0876.772114.957 minyes
SLEB11.8618.9395.15207.5515 minyes
Prune&Comp11.4318.1289.35206.8815 minyes
BlockPruner11.8218.4654.42234.571 hyes
Reassessing12.5423.05160.77299.1916.5 hno
Ours11.3617.9235.19152.072 hyes

About 95% of our cost is forward passes of the frozen LLM on small calibration probes; no gradient flows through the LLM, and the scorer is discarded once the sets are extracted. The extra cost pays off at larger budgets (35.19 vs. 54.42–95.15 at K=12; 152.07 vs. 206.88 at K=16) and buys little at K≤8. Using a quarter of the probe tokens gives close results up to K=12 (Appendix G), so the cost can be reduced further.

Minor: Figure 2 (W1)

We thank the reviewer for catching this. We will redraw Figure 2 and export it as a clean vector graphic without the stray red marks.

We thank the reviewer again for the constructive feedback.

Reviewer_fzJyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a budget-aware layer pruning method for LLMs. The authors show that optimal pruning sets may differ across pruning budgets and cannot always be represented by a fixed layer ranking. They formulate layer selection as a reinforcement learning problem and use Double Dueling DQN to learn a budget-conditioned pruning policy.

Critical strengths

The motivation is clear and interesting, with empirical evidence showing that optimal pruning sets are not necessarily nested across different budgets. The proposed framework considers both pruning budgets and interactions between removed layers, offering a reasonable improvement over fixed-ranking methods. The experiments cover multiple models, pruning budgets, and competitive baselines, with useful ablation studies and computational cost analysis.

Critical weaknesses

1 The evaluation relies heavily on ppl, which does not necessarily reflect the capabilities retained after layer pruning. Removing Transformer layers may affect complex generation capabilities such as multi-step reasoning and code generation, even when perplexity remains relatively low.

  1. Although the paper includes GSM8K and HumanEval, these evaluations are limited to a single pruning budget. More importantly, the authors do not report the performance of the original unpruned models, making it impossible to assess how much capability is actually retained.

3 The paper would benefit from reporting the original model performance and evaluating capability retention across different pruning ratios, particularly on generation-based reasoning benchmarks.

Questions for authors

see weakness

10 comments
AuthorsOpenReview ↗

Response 1/4

We thank the reviewer for recognizing the clear motivation, the evidence that optimal pruning sets are not nested across budgets, and the breadth of models, budgets, baselines and ablations. The reviewer's main concern is whether perplexity reflects retained capabilities, especially generation. We ran exactly the evaluation the reviewer suggested: generative benchmarks at four pruning ratios with the unpruned model as reference.

AuthorsOpenReview ↗

Response 2/4

C1. Perplexity does not necessarily reflect retained capabilities (W1)

Concern. The evaluation relies heavily on PPL; layer removal may hurt multi-step reasoning and code generation even when PPL stays low.

Response. We agree that perplexity alone is not sufficient, and our evaluation does not rely on it. Beyond PPL, the submission already evaluates nine zero-shot benchmarks, math reasoning, code generation and further instruction-model tasks, covering a broad range of capabilities; with the new multi-ratio generation results below, our evaluation is, to our knowledge, the most complete among the layer-pruning methods we compare against. PPL serves as the selection objective and one comparison metric among many (Sec. 4.4 notes that "perplexity and downstream accuracy need not improve in parallel").

Evidence.

  1. We evaluate well beyond PPL. PPL (WT2, C4) is reported for four models up to 70B and four budgets. Capabilities are measured separately: nine zero-shot benchmarks at all four budgets against six baselines (Table 3, per-task in Table 9); GSM8K and HumanEval (Tables 4 and 5); six further benchmarks on an instruction-tuned model (Appendix D, Table 11); and now GSM8K, HumanEval, MBPP and BBH at four ratios with an unpruned reference (C2 below). On the zero-shot suite our average is highest at K=8 (0.4826), on par with SLEB at K=12 (0.4022 vs. 0.4023), above SLEB at K=16 (0.3511 vs. 0.3320) and second at K=4 (0.5714); on GSM8K at 25% we reach 32.76%, the highest among compared methods (Table 4).
  2. What each benchmark measures. The suite covers knowledge, commonsense, language understanding, reasoning and generation:
BenchmarkCapability measuredFormatWhere
PIQAPhysical commonsense (choosing plausible ways to achieve a goal)Multiple choiceTable 3
HellaSwagCommonsense inference about how a situation continuesMultiple choiceTable 3, Table 11
ARC-Easy / ARC-ChallengeGrade-school science knowledge and reasoning (Challenge: questions that simple retrieval fails on)Multiple choiceTable 3
OpenBookQACombining a core science fact with commonsense knowledgeMultiple choiceTable 3
RACEReading comprehension of long passagesMultiple choiceTable 3
WSC273Pronoun coreference requiring commonsenseMultiple choiceTable 3
LAMBADALong-range discourse understanding (predicting the final word from broad context)Word predictionTable 3
MMLUKnowledge across 57 subjects (STEM, humanities, social sciences)Multiple choiceTable 3, Table 11
WinoGrandeCommonsense coreference with adversarially filtered pairsMultiple choiceTable 11
GSM8KMulti-step arithmetic reasoning in grade-school word problemsFree-form generationTable 4, Table 11, new (4 ratios)
HumanEvalFunction-level code synthesis from docstrings, checked by unit testsCode generationTable 5, new (4 ratios)
BBHChallenging multi-step reasoning (23 BIG-Bench tasks: logical, algorithmic, symbolic)Free-form generationTable 11, new (4 ratios)
XSumAbstractive single-sentence summarizationFree-form generationTable 11
  1. Our evaluation is the most complete among the compared layer-pruning methods. We checked the evaluation suites reported in each baseline paper:
MethodPPLZero-shot / multiple-choice tasksMath reasoningCode generationOther generationPruning ratios with generation results
SLEBWT2, C45 (PIQA, WinoGrande, HellaSwag, ARC-e/c)––––
Shortened LLaMAWT2, PTB7 (BoolQ, PIQA, HellaSwag, WinoGrande, ARC-e/c, OBQA)––––
ShortGPTnot in main results13+ incl. MMLU, CMMLU––XSumone (~25%)
BlockPrunerWT25 main + 4 in appendix (SWAG, TruthfulQA, OBQA, RACE)AQuA-RAT, zero-shot accuracy (appendix)––one per model (AQuA-RAT)
Prune&CompWT2, C4, PTB9 QA tasks + MMLU––––
ReassessingWT2, PTB8 (PIQA, HellaSwag, OBQA, ARC-e/c, MMLU, CMMLU, WinoGrande)GSM8KHumanEval (appendix)–one per task (GSM8K: 25%; HumanEval: 20%)
OursWT2, C4 (4 models up to 70B)9 + WinoGrande in Appendix DGSM8KHumanEvalBBH, XSum (Appendix D)one per task (GSM8K: Table 4; HumanEval: 21.88%, Table 5; Table 11 cross-study)

Already in the submission, our evaluation covers math reasoning (GSM8K), code generation (HumanEval), and complex reasoning and summarization (BBH, XSum) in addition to PPL and zero-shot tasks; among the compared methods, only Reassessing also reports both math reasoning and code generation, and ours additionally covers BBH and XSum. None of the compared methods reports any generation benchmark at more than one pruning ratio; the new results in this rebuttal add MBPP and extend GSM8K, HumanEval and BBH to four ratios against the unpruned model.

AuthorsOpenReview ↗

Response 3/4

C2. Generative benchmarks at multiple pruning ratios with an unpruned reference (W2, W3)

Concern. GSM8K and HumanEval are reported at a single budget and without the unpruned model, so capability retention cannot be assessed.

Response. On LLaMA3.1-8B at 12.5%–50% pruning ratios, our method is the best in 15 of 16 benchmark–budget cells and leads the best baseline in average retention by 12–18 points at every budget; the relative advantage grows with the pruning ratio. We also confirm the reviewer's point that generation degrades faster than perplexity suggests, for all methods including ours.

Evidence. Model: LLaMA3.1-8B. Budgets: K = 4, 8, 12, 16 of 32 layers. Benchmarks: GSM8K (multi-step reasoning), HumanEval and MBPP (code), BBH (reasoning). Baselines: ShortGPT, SLEB, Prune&Comp. No fine-tuning after pruning. Unpruned reference from Grattafiori et al. (2024), "The Llama 3 Herd of Models": GSM8K 57.2 (Table 12), HumanEval 37.2 and MBPP 47.6 (Table 10), BBH 64.2 (Table 13). Parentheses give retention relative to the unpruned model. The published unpruned reference uses 8-shot chain-of-thought prompting for GSM8K (maj@1; maximum generation length: 1,024 tokens), 0-shot prompting for HumanEval (Pass@1; 1,024 tokens), 3-shot prompting for the original MBPP benchmark (Pass@1; 256 tokens), and 3-shot chain-of-thought prompting for BBH (average exact-match accuracy across subtasks; 512 tokens). These settings are documented in Meta's official Llama 3.1 evaluation notes. The pruned models use the same.

MethodK (ratio)GSM8K Acc.HumanEval Pass@1MBPP Pass@1BBH Acc.
Unpruned0 (0.00%)57.20 (100.00%)37.20 (100.00%)47.60 (100.00%)64.20 (100.00%)
Ours4 (12.50%)48.16 (84.20%)32.42 (87.15%)44.31 (93.09%)62.80 (97.82%)
ShortGPT4 (12.50%)11.13 (19.46%)29.52 (79.35%)39.62 (83.24%)55.07 (85.78%)
SLEB4 (12.50%)49.18 (85.98%)23.26 (62.53%)37.12 (77.98%)56.45 (87.93%)
Prune&Comp4 (12.50%)43.88 (76.71%)27.50 (73.92%)37.89 (79.60%)54.01 (84.13%)
Ours8 (25.00%)32.76 (57.27%)21.75 (58.47%)33.99 (71.41%)53.27 (82.98%)
ShortGPT8 (25.00%)0.40 (0.70%)18.45 (49.60%)27.30 (57.35%)40.33 (62.82%)
SLEB8 (25.00%)27.98 (48.92%)8.96 (24.09%)22.40 (47.06%)43.47 (67.71%)
Prune&Comp8 (25.00%)27.18 (47.52%)15.80 (42.47%)25.31 (53.17%)40.37 (62.88%)
Ours12 (37.50%)23.23 (40.61%)12.66 (34.03%)23.74 (49.87%)42.67 (66.46%)
ShortGPT12 (37.50%)0.00 (0.00%)9.73 (26.16%)16.43 (34.52%)26.38 (41.09%)
SLEB12 (37.50%)14.56 (25.45%)2.44 (6.56%)11.25 (23.63%)30.45 (47.43%)
Prune&Comp12 (37.50%)14.43 (25.23%)7.58 (20.38%)14.88 (31.26%)27.68 (43.12%)
Ours16 (50.00%)13.20 (23.08%)6.49 (17.45%)15.21 (31.95%)32.33 (50.36%)
ShortGPT16 (50.00%)0.00 (0.00%)4.44 (11.94%)8.82 (18.53%)15.68 (24.42%)
SLEB16 (50.00%)6.55 (11.45%)0.50 (1.34%)4.84 (10.17%)19.69 (30.67%)
Prune&Comp16 (50.00%)6.63 (11.59%)3.07 (8.25%)7.75 (16.28%)17.39 (27.09%)

Bold: best at each budget.

Reviewer_rcheOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that most existing layer-pruning methods adopt a single, budget-independent ranking and take its prefix under each budget. The resulting nested-prefix constraint requires the pruning set for a smaller budget to be contained in that for a larger budget, which limits the performance of the pruned model. The authors therefore cast layer pruning as budget-aware subset optimization and formulate it as a budget-conditioned finite-horizon Markov decision process, scoring layers by a budget-aware continuation value rather than by intrinsic importance. They approximate this value with Double Dueling DQN, so that a single scorer can construct the pruning set from both the target budget and the current pruning context.

Critical strengths
  1. The paper has a clear and reasonable motivation. It formalizes the restriction of fixed global ranking into nested prefix constraints, and supports this with counterexamples obtained through exhaustive search on LLaMA3.1-8B;
  2. The research method is quite distinctive. It addresses the layer pruning issue from the perspective of reinforcement learning.
Critical weaknesses
  1. The author needs to provide a clearer explanation of the calculation costs. In Table 15, the offline cost of the proposed method is about 2 GPU-hours, compared with about 15 minutes for SLEB and Prune&Comp. At K=4K=4 and K=8K=8, the perplexity gains over these two methods are small, and the proposed method is worse in some settings;

  2. At K=4K=4 and K=8K=8, and several Qwen3 budgets, the WikiText-2/C4 perplexity gaps over Prune&Comp and SLEB are small;

  3. How to verify the validity of the learned scorer;

  4. At each step of pruning, the state is recalculated. Local features include representations of transformations, parameter sizes, and the KL divergence and entropy changes resulting from removing each layer; global features include progress, current perplexity, perplexity change amount, and budget ratio. Please report the acquisition costs of these contents;

  5. The table results show that when K is small, the advantages of the proposed method in the article are relatively small. Can the author provide a reasonable explanation for this phenomenon?

Questions for authors

The same as the weakness part.

6 comments
AuthorsOpenReview ↗

Response 1/4

We thank the reviewer for recognizing the clear motivation, the formalization of the nested-prefix constraint with exhaustive-search counterexamples, and the distinctive RL perspective. The main concerns are (1) the offline cost and the cost of acquiring the state features, (2) the small margins at small budgets, and (3) how the learned scorer can be validated. We address them below with existing evidence, a cost breakdown, and concrete revisions.

AuthorsOpenReview ↗

Response 2/4

C1. Offline cost and state-feature acquisition cost (W1 cost part, W4)

Concern. The method costs about 2 GPU-h versus about 15 min for SLEB and Prune&Comp, and the per-step state recomputation (local and global features) has unreported cost.

Response. About 95% of the 2 GPU-h is forward passes of the frozen LLM over small calibration probes. The state features add only a small overhead on top: at each pruning step, a single forward pass of the current model on the probe produces the intermediate values that the representation-transformation and predictive-influence features need, and the features are then computed from these cached values. This computation is not free, but it is small compared with the forward pass; together with the Q-network updates it makes up the remaining ~5% of training time.

Evidence.

Acquisition cost per state component (Sec. 3.2: 7 features per layer, 4 global features).

ComponentFeaturesIntermediate values neededSource of these valuesWhenAdditional cost beyond the forward pass
Parameter magnitudeL1, L2 norms of layer weightsLayer weightsDirectly from the weightsOnce before training (LLM weights are frozen)Negligible: one-time norm computation, then reused
Representation transformationCosine similarity, residual ratio, output–input norm ratioInput and output hidden states of every remaining layerCached during the single forward pass of the current model on the probeEvery pruning stepSmall: norms and cosine similarities on cached hidden states; no extra forward pass
Predictive influenceKL divergence, entropy shiftOutput logits / token distributions and the per-layer hidden statesCached during the same single forward passEvery pruning stepSmall: KL divergence and entropy on cached distributions; no extra forward pass
Global: current PPL, per-step PPL change2 scalarsOutput logitsSame single forward pass; this PPL also gives the rewardEvery pruning stepNegligible: read from the same logits
Global: progress t/K, budget ratio K/N2 scalarsStep counter and budgetKnown from the episodeEvery pruning stepNone

In short, one forward pass per pruning step supplies the intermediate values for both the representation-transformation and the predictive-influence features (and the global PPL features and reward); no per-layer or additional forward pass is run. By default a probe has 4 sequences × 512 tokens (2,048 tokens), drawn from a pool of 64 BookCorpus sequences (Appendix G). What remains after the forward pass is arithmetic on cached hidden states and logits (norms, cosine similarities, KL divergence, entropy); this adds some cost, but it is small relative to the forward pass itself.

Where the 2 GPU-h goes. Training runs 800 iterations with 4 workers, one episode per worker per iteration (Appendix A): 3,200 episodes, about 2.3 GPU-seconds per episode.

PartShare of training timeApprox. cost
Calibration forward passes of the frozen LLM (features and reward)~95%~1.9 GPU-h
Q-network updates, replay sampling, feature computation on cached values~5%~0.1 GPU-h

The cost is forward-only (no gradient through the LLM; only the small Q-network is trained), paid once per base model (one scorer yields the sets for all budgets via a K-step deterministic rollout), and absent at inference (the scorer is discarded; the pruned model is an ordinary dense model).

Cost versus quality (Table 15, LLaMA3.1-8B, PPL averaged over WT2 and C4).

MethodK=4K=8K=12K=16Offline costMulti-budget
SLEB11.8618.9395.15207.5515 minyes
Prune&Comp11.4318.1289.35206.8815 minyes
BlockPruner11.8218.4654.42234.571 hyes
Reassessing12.5423.05160.77299.1916.5 hno
Ours11.3617.9235.19152.072 hyes

We agree that the extra cost buys little at K=4 and K=8 (0.6% and 1.1% lower average PPL than Prune&Comp); if only such mild budgets are needed, a 15-minute method is a reasonable choice. The cost pays off at larger budgets: 35.19 vs. 54.42–95.15 at K=12, and 152.07 vs. 206.88 at K=16.

The cost can be reduced. Since forward passes dominate, cost scales with probe tokens. Appendix G (Figure 3, Qwen3-8B) shows that 1 sampled sequence per episode instead of 4 (a quarter of the tokens) gives close results up to K=12 (WT2 log-PPL 2.38 / 2.82 / 3.50 vs. default 2.38 / 2.85 / 3.50 at K=4/8/12) and is worse only at K=16 (4.49 vs. 4.41). This cuts the probe tokens, and hence the dominant forward-pass cost, by 4×.

AuthorsOpenReview ↗

Response 3/4

C2. Small gains at small budgets (W1 performance part, W2, W5)

Concern. At K=4, K=8 and several Qwen3 budgets, the WT2/C4 margins over Prune&Comp and SLEB are small, and in some settings our method is worse. Why?

Response. The margins at K=4 and K=8 are small because the problem itself leaves little room there, not because the scorer is weak. Our argument has five steps: (1) at small K the headroom over strong baselines is small, and we already capture most of it; (2) the headroom grows quickly with K, and our margin grows with it; (3) a small PPL margin can still hide a large capability margin; (4) the few cases where we are worse fall inside this small headroom, and we explain their likely cause; (5) the larger budgets, where our gains are largest, are also the ones that matter most for speed.

Evidence.

  1. At small K the headroom is small, and we already capture most of it. On LLaMA3.1-8B at K=4 (Table 6), even the exhaustive optimum (8.63 WT2 PPL) is only 3.1% better than the strongest baseline, Prune&Comp (8.91). Our set reaches 8.68, 0.55% above the optimum, so we capture about 83% of the headroom that any method could possibly gain over Prune&Comp. Beam search (width 16) that queries the true calibration PPL at every step returns exactly our result (8.68 / 14.03, Table 13). Different criteria also converge here: on Qwen3-8B at K=4, SLEB and Shortened LLaMA give identical PPL (10.40 / 17.50) and zero-shot accuracy (0.5595). A large PPL margin at K=4 is therefore not achievable by any selection method.
  2. The headroom grows with K, and so does our margin. The interactions between removed layers grow combinatorially: removing 4 of 32 layers involves 6 pairwise interactions among about 3.6×10^4 subsets, removing 16 involves 120 among about 6.0×10^8. Accordingly, the gap between the optimum and the best nested set grows from 0.55 PPL (6.4%) on LLaMA3.1-8B at K=4 (Table 6) to 172.66 PPL (20.05%) on LLaMA3.2-1B at K=8 (Table 12), and the spread among methods on LLaMA3.1-8B widens from 8.68–9.74 WT2 PPL at K=4 (excluding ShortGPT) to 33.87–184.94 at K=12. This is where our method separates from the baselines: 33.87 vs. 56.96 (BlockPruner) and 51.29 (beam search on the true objective) at K=12, and 170.71 vs. 236.21 (Prune&Comp) at K=16. The same pattern holds on LLaMA3.1-70B (Table 2): slightly behind Prune&Comp at K=10 (5.56 vs. 5.39 WT2), clearly ahead at K=20 (10.93 vs. 13.46) and K=40 (132.85 vs. 244.19).
  3. A small PPL margin does not imply a small capability margin. On LLaMA3.1-8B at K=4, our average PPL is only 0.6% lower than Prune&Comp's (11.36 vs. 11.43, Table 15), yet on the new generation benchmarks (our response to Reviewer 2) our average retention is 90.57% vs. 78.59% (Prune&Comp) and 78.60% (SLEB), a 12-point lead; at K=8 it is 67.53% vs. 51.51% and 46.95%. On Qwen3-8B, our K=4 PPL is worse than SLEB's, but our nine-task zero-shot average is higher (0.5714 vs. 0.5595; Prune&Comp 0.5681), and at K=8 it is the highest of all methods (0.4826, Table 3).
  4. Where we are worse, and why. The remaining cases all lie inside the small headroom of step 1: on Qwen3-8B at K=4, ours is 10.83 WT2 PPL vs. 10.40 (SLEB) and 10.19 (Prune&Comp); at K=8, ours is better than SLEB on WT2 (17.26 vs. 18.09) but worse on C4 (25.55 vs. 23.77); on LLaMA3.1-70B at K=10, see step 2; on Qwen3-14B, ours equals SLEB at K=4 (8.92 / 15.19) and is within about one standard deviation of SLEB and Prune&Comp at K≥8 (Appendix B). A plausible cause is that one scorer is trained jointly for all budgets, and the large-budget trajectories (larger return magnitudes) dominate the loss, which makes near-ties at K=4 harder to resolve. Prune&Comp additionally compensates the remaining weights, which is orthogonal to layer selection and could be combined with our sets.
  5. Larger budgets are where pruning pays off. Removing 4 of 32 layers yields at most 32/28 ≈ 1.14× ideal speedup, whereas K=12 gives a measured 1.53× (Appendix J). Our gains are concentrated exactly in this practically relevant regime.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.