- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper formulate layer pruning as a budget-conditioned finite-horizon Markov decision process and approximate this value using Double Dueling DQN, enabling a single scorer to construct pruning sets conditioned on both the target budget and the current pruning context. Extensive experiments have demonstrated the effectiveness of the proposed method.
Critical strengths
- The experiments in the paper are very thorough.
- It's quite interesting to formulate the layer pruning problem as a budget-constrained finite-term Markov decision process.
Critical weaknesses
- Figure 2 appears to be a screenshot from a PowerPoint presentation; it's recommended to modify it as it contains small red lines.
- Will the layers selected differ depending on the calibration set? An ablation experiment is missing.
- There is a lack of theoretical proof for the effectiveness of the proposed method.
Questions for authors
- How do the pruning costs and time required by this method compare to other baselines?
5 comments
Response 2/3
C1. Do the selected layers differ with the calibration set? (W2)
Concern. Whether the selected layers depend on the calibration set; an ablation appears to be missing.
Response. The selected sets can differ, but the resulting quality is nearly unchanged. We show this in two ways: (a) the sampling ablation already in Appendix G of the submission (Figures 3–5, Qwen3-8B), which our main text did not point to clearly enough; and (b) a new experiment that replaces the calibration corpus itself (BookCorpus → PTB) on LLaMA3.1-8B, where log-PPL changes by at most 0.03 at every budget.
Evidence (a): sampling of the calibration data (Appendix G). We vary the calibration data along three dimensions around the default (pool of 64 BookCorpus sequences, 4 sequences sampled per episode, 512 tokens each). Each cell is log-PPL on WT2 / C4.
| Calibration configuration | K=4 | K=8 | K=12 | K=16 |
|---|---|---|---|---|
| Default (pool 64, 4 samples, length 512) | 2.38 / 2.86 | 2.85 / 3.28 | 3.50 / 3.82 | 4.41 / 4.59 |
| Pool size 16 | 2.37 / 2.87 | 2.76 / 3.20 | 3.56 / 3.84 | 4.45 / 4.67 |
| Pool size 256 | 2.34 / 2.86 | 2.78 / 3.17 | 3.57 / 3.79 | 4.39 / 4.61 |
| 1 sample per episode | 2.38 / 2.91 | 2.82 / 3.22 | 3.50 / 3.84 | 4.49 / 4.81 |
| 16 samples per episode | 2.34 / 2.86 | 2.79 / 3.17 | 3.57 / 3.79 | 4.40 / 4.65 |
| Sequence length 128 | 2.36 / 2.86 | 2.78 / 3.17 | 3.63 / 3.83 | 4.58 / 4.70 |
| Sequence length 2048 | 2.34 / 2.86 | 2.78 / 3.17 | 3.41 / 3.67 | 4.31 / 4.58 |
- Sets can differ. PPL differs between configurations at every budget, so the sets are not always identical; some coincide (at K=4, pool 256, 16 samples and length 2048 all give 2.34 / 2.86).
- Small effect at mild budgets, larger at aggressive ones. All configurations are within 0.04 (WT2) / 0.05 (C4) log-PPL at K=4 and 0.09 / 0.11 at K=8. At K=12 and K=16 the WT2 spread grows to 0.22 and 0.27, mostly from sequence length (128 → 2048 tokens: 3.63 → 3.41 at K=12, 4.58 → 4.31 at K=16). Pool size, the dimension closest to "a different calibration set", changes log-PPL by at most 0.11 despite a 16× change.
- Interpretation. At mild ratios many sets are nearly equivalent (on LLaMA3.1-8B at K=4, the exhaustive optimum and the best nested set differ by 0.55 PPL, Table 6), so different calibration data can pick different layers with almost identical quality. At large budgets the continuation value is harder to estimate (seed variance also grows, Table 8) and longer sequences help.
- Robustness is built in. State features are computed on randomly sampled calibration slices (Sec. 3.4), so the scorer is trained over calibration draws rather than one fixed estimate; across seeds, which also change the calibration batch order, the sets largely coincide at K≤8 (Appendix B). The default configuration was not tuned in our favor: at K=8 it is the worst setting in each of the three figures.
Evidence (b): a different calibration corpus (new). The ablations above vary how data is drawn from BookCorpus. To test the source corpus itself, we retrained the scorer on LLaMA3.1-8B with PTB as the calibration corpus, keeping everything else unchanged, and compare it with the default BookCorpus calibration. Both settings use a calibration pool of 64 sequences of 512 tokens each, with 4 sequences sampled per episode. BookCorpus (fiction books) and PTB (Wall Street Journal news) differ in domain and style, and neither is an evaluation set. Each cell is log-PPL (lower is better); † marks the PTB-calibrated runs.
| K (pruning ratio) | BookCorpus: WT2 log-PPL ↓ | PTB: WT2 log-PPL ↓ | BookCorpus: C4 log-PPL ↓ | PTB: C4 log-PPL ↓ |
|---|---|---|---|---|
| 4 (12.50%) | 2.16 | 2.17† | 2.64 | 2.64† |
| 8 (25.00%) | 2.66 | 2.65† | 3.07 | 3.08† |
| 12 (37.50%) | 3.52 | 3.54† | 3.60 | 3.61† |
| 16 (50.00%) | 5.14 | 5.17† | 4.89 | 4.92† |
- Changing the corpus has almost no effect. Across all eight cells, the absolute difference is at most 0.03 log-PPL (about 3% in PPL), and the sign is mixed at small budgets (WT2 at K=8: PTB is 0.01 lower). This is smaller than the spread across the sampling configurations in (a) at K≥12 (0.22 / 0.27 on WT2).
- The ranking against baselines does not depend on the corpus. With PTB calibration, WT2 log-PPL at K=12 is 3.54 (PPL ≈ 34.5) and at K=16 is 5.17 (PPL ≈ 176), still well ahead of the best baselines in the main results (BlockPruner 56.96 at K=12, Prune&Comp 236.21 at K=16 on WT2). The largest change (+0.03 at K=16) matches the trend in (a): the aggressive regime is the most sensitive to calibration, yet still only marginally.
Response 3/3
C2. Theoretical justification (W3)
Concern. The paper lacks a theoretical argument for the effectiveness of the method.
Response. We agree that the consequences of our formal result (Eq. 4) should be stated explicitly. We give two propositions that follow from it, a provable limitation of fixed rankings, and an honest account of what cannot be proved.
Evidence.
- Proposition 1 (objective alignment). With the telescoping reward and γ=1, the return of any K-step trajectory equals −D(M_K, M_0) and depends only on the final set (Eq. 4). Since the MDP is deterministic with finite horizon, the greedy rollout w.r.t. Q* returns a set S*_K minimizing D over all feasible K-subsets, for every K. The sequential form introduces no order bias.
- Proposition 2 (error propagation). If |q(s,a) − Q*(s,a)| ≤ ε on states reachable under budget K, and Ŝ_K is the greedy rollout of q, then D(M_Ŝ, M_0) ≤ D(M_S*, M_0) + 2Kε. Sketch: at each step Q*(s,a*) ≤ q(s,a*) + ε ≤ q(s,â) + ε ≤ Q*(s,â) + 2ε, so V*(s_t) − Q*(s_t, â_t) ≤ 2ε; with deterministic transitions and V* = 0 at the terminal state, summing V*(s_t) − r_t − V*(s_{t+1}) over K steps gives exactly the return gap. As D is log-PPL, this is at most a factor exp(2Kε) in calibration PPL; it is the finite-horizon analogue of Singh and Yee (1994).
- Fixed rankings are provably suboptimal across budgets. Any budget-agnostic ranking yields nested sets; if optimal sets at two budgets are not nested, no ranking is optimal at both. This occurs in practice: on LLaMA3.1-8B the optimum is {10,25,26} at K=3 and {11,12,25,26} at K=4 (Table 6); on LLaMA3.2-1B, {3,4,5,6,7,8,12} at K=7 and {3,4,5,6,7,11,12,14} at K=8 (Table 12). A ranking optimal at the smaller budget loses at least 6.39% and 20.05% at the larger one, regardless of the importance metric.
- The policy class contains existing approaches. A fixed ranking ignores state and K; SLEB-style greedy removal maximizes only the immediate reward. Both are special cases, so the optimum of our formulation is never worse on the calibration objective, and strictly better when optimal sets are not nested.
- What cannot be proved. Cardinality-constrained minimization of a general set function is hard; this objective is not additive (otherwise optimal sets would be nested) and there is no reason to expect submodularity. Deep Q-learning with function approximation has no general convergence guarantee, so ε cannot be bounded a priori. We therefore certify empirically where the optimum is computable: within 0.55% of the exhaustive optimum at K=4 on LLaMA3.1-8B (Table 6), and better than beam search on the true objective at K≥8 (Table 13).
Scope: both propositions hold for the unclipped reward over the feasible action set (protected layers excluded); reward clipping and early termination are stability devices, and exact alignment holds on trajectories where they are not triggered.
C3. Pruning cost and time compared with baselines (Question)
Concern. How do the pruning cost and time compare with the baselines?
Response. Our offline cost is 2 A100 GPU-h: 8× SLEB and Prune&Comp, 2× BlockPruner, and about one eighth of Reassessing. It is paid once per base model for all budgets and adds nothing at inference.
Evidence. Table 15 (Appendix I), LLaMA3.1-8B, PPL averaged over WT2 and C4:
| Method | K=4 | K=8 | K=12 | K=16 | Offline cost | Multi-budget |
|---|---|---|---|---|---|---|
| ShortGPT | 16.41 | 4145.32 | 54885.81 | 2042.07 | seconds | yes |
| Shortened LLaMA | 11.70 | 22.08 | 76.77 | 2114.95 | 7 min | yes |
| SLEB | 11.86 | 18.93 | 95.15 | 207.55 | 15 min | yes |
| Prune&Comp | 11.43 | 18.12 | 89.35 | 206.88 | 15 min | yes |
| BlockPruner | 11.82 | 18.46 | 54.42 | 234.57 | 1 h | yes |
| Reassessing | 12.54 | 23.05 | 160.77 | 299.19 | 16.5 h | no |
| Ours | 11.36 | 17.92 | 35.19 | 152.07 | 2 h | yes |
About 95% of our cost is forward passes of the frozen LLM on small calibration probes; no gradient flows through the LLM, and the scorer is discarded once the sets are extracted. The extra cost pays off at larger budgets (35.19 vs. 54.42–95.15 at K=12; 152.07 vs. 206.88 at K=16) and buys little at K≤8. Using a quarter of the probe tokens gives close results up to K=12 (Appendix G), so the cost can be reduced further.
Minor: Figure 2 (W1)
We thank the reviewer for catching this. We will redraw Figure 2 and export it as a clean vector graphic without the stray red marks.
We thank the reviewer again for the constructive feedback.
Response 1/3
We thank the reviewer for the positive assessment, and for finding the experiments thorough and the budget-conditioned MDP formulation interesting. We address the calibration question, the theoretical justification, the cost comparison and Figure 2 below.