acceptodds
Under review as a conference paper at ICLR 2027

LOOP: Recovery-Aligned Iterative Layer Pruning via On-Policy Distillation

Abstract

Layer-level pruning is attractive for large language model (LLM) deployment as it reduces the model size while preserving a dense architecture that can be efficiently executed on standard hardware. Current methods typically follow a one-shot paradigm, which first removes selected layers to reach a target model size and then performs additional training to recover model capabilities. However, pruning directly to the target size can substantially degrade reasoning capabilities, making subsequent recovery more difficult under a limited training budget. Moreover, this paradigm separates layer selection from recovery, limiting the use of recovery information to guide selection and potentially hindering the recovery of reasoning capabilities. To address these limitations, we propose LOOP, a Layer-level Recovery-Oriented On-Policy Pruning framework that interleaves pruning and recovery across multiple shots. In each shot, LOOP removes a single layer and performs short on-policy distillation, mitigating the abrupt degradation in reasoning capabilities associated with one-shot pruning. To guide layer selection in subsequent shots, LOOP reuses on-policy trajectories and optimization signals from recovery to jointly assess each remaining layer's reasoning contribution and recovery sensitivity, thereby making layer selection recovery-aware. Evaluations on several benchmarks show that LOOP consistently outperforms one-shot baselines in average accuracy across different model scales and pruning ratios.

Primary area:
optimization
Keywords:
Pruning, On Policy Distillation, Model Compression

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_VaK3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LOOP, a recovery-aligned iterative layer-pruning framework for LLMs. To mitigate the performance degradation of one-shot removal, LOOP alternates single-layer pruning with short on-policy distillation (OPD) recovery. To guide this process, its importance criterion combines Block Influence with Recovery Sensitivity (RS), which is dynamically updated during the recovery phase.

Experiments on math and science reasoning tasks across various model scales and pruning ratios demonstrate the superiority of LOOP over one-shot baselines. For instance, on Qwen3-4B-Instruct with a 20% pruning ratio, the average math accuracy improves from 23.58% to 32.68% compared to one-shot OPD. Additionally, ablation studies are provided to support the contributions of both the iterative pruning strategy and the combined criterion.

Critical strengths
  1. Practical relevance and clear problem formulation. The paper try to address the practical challenge of maintaining reasoning performance under layer-level compression. The proposed alternating pruning and recovery scheme effectively mitigates the performance degradation inherent in one-shot layer removal. Empirical results demonstrate consistent improvements across various pruning ratios and model scales.

  2. Cohesive methodology with systematic component-level analysis. The framework cohesively integrates layer selection, on-policy recovery, and importance estimation into a unified procedure. Notably, reusing OPD-collected information for subsequent pruning decisions is a well-motivated design choice. Furthermore, the factorial ablation in Table 4 systematically isolates the individual contributions of iterative pruning and recovery-oriented importance estimation, and the extended analysis of the recovery phase further corroborates these empirical findings.

Critical weaknesses
  1. Limited Methodological Novelty. Iterative layer pruning and intermediate recovery have been explored in prior work, such as Iterative Layer-wise Distillation (Kovalev and Tikhomirov, 2025; arXiv:2511.05085) and Prune&Comp (Chen et al., AAAI 2026). LOOP's novelty therefore appears to lie primarily in integrating OPD with recovery sensitivity, rather than in iterative pruning itself. The paper should clarify its distinct contributions and provide stronger comparisons with existing iterative pruning methods to establish the added value of RS.

  2. Insufficient Comparisons with Relevant Baselines. The experiments mainly compare LOOP against one-shot pruning methods. Although Table 4 includes an iterative BI ablation, comparisons with existing iterative pruning-and-recovery methods are missing, making it difficult to establish LOOP's advantage over prior iterative frameworks. Moreover, the OPD baseline uses a fixed maximum rollout length rather than ShortOPD's original short-to-long schedule.

  3. Experimental Fairness and Reliability. Matching 200 optimizer steps does not ensure equal computational costs, as LOOP trains models at varying depths. Moreover, the evaluation protocol does not clearly specify the number of sampled responses per question. Given the stochastic nature of mathematical reasoning, reporting repeated-sampling accuracy (e.g., Avg@8 or pass@8) would strengthen the reliability of the results, particularly the modest improvement attributable to RS.

Questions for authors

Q1. Why is the squared gate gradient an appropriate measure of layer importance for actual pruning? Appendix A provides a local sensitivity interpretation, but layer removal corresponds to a finite change from (g_l=1) to (g_l=0). How does the squared-gradient score relate to the actual change in recovery loss following layer removal? Why is the gradient sign discarded?

Q2. How dependent is the method on ground-truth-conditioned teacher supervision? Equation (7) conditions the teacher on the ground-truth answer, while the student generates trajectories without this information. Could the authors clarify how this additional supervision affects recovery performance and whether the proposed importance criterion remains effective without ground-truth-conditioned teacher feedback?

Q3. What exactly does "Recovery-Aligned" mean in LOOP? Although the title emphasizes "Recovery-Aligned," the paper does not formally define or explicitly measure recovery alignment. The authors should clarify this concept and explain how the proposed criterion establishes such alignment.

6 comments

Methodological Novelty.

We thank the reviewer for the valuable feedback. In the revised Related Work section, We discuss prior work on iterative pruning. LOOP differs from prior work in the following key ways:

First, LOOP uses the recovery objective directly to score layers. Iterative Layer-wise Distillation ranks layers by the performance drop caused by candidate removals, and Prune&Comp reapplies a pruning criterion after magnitude compensation. LOOP augments Block Influence (BI) with recovery sensitivity (RS), the squared gradient of the OPD loss with respect to each layer’s gate. This lets subsequent pruning decisions account for how sensitive the current recovery loss is to perturbing each remaining layer. Under the same iterative pruning and OPD protocol, adding RS to BI improves mean math accuracy from 30.69% to 32.68% on Qwen3-4B-Instruct at 20% layer pruning.

Second, LOOP scores layers on the current student’s own reasoning trajectories. Iterative Layer-wise Distillation evaluates candidate removals on representative datasets, and Prune&Comp uses a calibration set for layer scoring and magnitude estimation. LOOP computes both BI and RS on the same on-policy trajectories used for recovery. As the student’s depth and parameters change, the trajectories used for layer selection reflect how the current student actually generates responses.

Finally, LOOP obtains both scores during OPD. Iterative Layer-wise Distillation runs separate evaluations of candidate removals, and Prune&Comp runs calibration passes to estimate the magnitude change. LOOP computes BI from activations collected during recovery and obtains RS from the OPD backward pass. Reusing these computations lets LOOP update the ranking without separate calibration passes or evaluations of candidate layer removals.

Comparisons with Relevant Baselines.

Thank you for highlighting these two gaps. We added a comparison with Prune&Comp and evaluated LOOP and one-shot OPD under a short-to-long rollout schedule.

  • Comparison with Prune&Comp.

    MethodRecoveryMetricDAPOMATH-500AMCAIMEMeanT
    One-shot PrunedNoneBI0.0040.0140.0000.0000.00450.833%
    Prune&CompNoneBI0.0070.0400.0000.0000.01182.174%
    One-shot OPDOPDBI0.1390.4680.3250.0110.235843.625%
    LOOPOPDBI + RS0.2380.5220.4250.1220.326860.474%

    We evaluated Prune&Comp with BI on Qwen3-4B-Instruct at 20% layer pruning, using trajectories on DAPO-17K-Math generated by Qwen3-235B as calibration data. Prune&Comp improves performance over direct pruning but still trails LOOP by a substantial margin.

  • Short-to-long rollout schedule.

    MethodRollout ScheduleMetricDAPOMATH-500AMCAIMEMeanT
    OPDFixedBI0.1390.4680.3250.0110.235843.625%
    OPDShort-to-LongBI0.1440.4280.3000.0220.223541.358%
    LOOPFixedBI + RS0.2380.5220.4250.1220.326860.474%
    LOOPShort-to-LongBI + RS0.2220.5040.3250.1110.290553.756%

    We reimplemented ShortOPD’s short-to-long schedule from the paper and applied the same schedule to both one-shot OPD and LOOP. Under fixed-length rollouts, LOOP achieved 32.68% mean math accuracy versus 23.58% for one-shot OPD (+9.10 percentage points). Under the short-to-long schedule, LOOP achieved 29.05% versus 22.35% (+6.70 points). LOOP’s accuracy advantage therefore persisted under the short-to-long schedule.

Experimental Fairness and Reliability.

Thank you for raising these points. We agree that matching 200 optimizer updates does not ensure equal compute, since LOOP trains the student at different depths. The table below reports the measured wall-clock time, GPU-hours, and peak per-GPU memory for LOOP and one-shot OPD at the same update budget.

ModelRatioMethodWall-Clock (h)GPU-HoursPeak Memory (GiB)
Qwen3-1.7B20%LOOP2.39519.16108.92
20%One-shot OPD2.30018.40108.86
Qwen3-4B15%LOOP2.80122.40114.85
15%One-shot OPD2.86422.92115.04
20%LOOP2.85922.87115.02
20%One-shot OPD3.10124.81115.04
25%LOOP2.90923.27115.14
25%One-shot OPD3.16125.28115.12
Qwen3-8B20%LOOP4.17533.40140.36
20%One-shot OPD4.17433.39140.30

LOOP used fewer GPU-hours at all three pruning ratios on Qwen3-4B, nearly the same GPU-hours on Qwen3-8B and Qwen3-1.7B. Peak memory was comparable across all five settings.

For the math benchmarks, we report Avg@4. We sample four responses per question, average their correctness scores, and then average across questions. We will clarify this protocol in the manuscript

Q1. Why is the squared gate gradient an appropriate measure of layer importance for actual pruning?

Thank you for asking this question. The squared gate gradient measures local sensitivity, and its connection to full layer removal comes from a first-order approximation to the immediate change in OPD loss. With model parameters and student trajectories held fixed, let Lℓ(g)\mathcal L_\ell(g) denote the OPD loss when the gate of layer ℓ\ell is set to gg, and let dℓ=Lℓ′(1)d_\ell=\mathcal L_\ell'(1). The immediate loss change from fully removing the layer is

=-\int_0^1\mathcal L_\ell'(g) \mathrm{d}g.$$ The first-order approximation at $g=1$ is $-d_\ell$, so $d_\ell^2$ is the squared magnitude of this approximation. We square the gradient because RS measures how strongly the current recovery objective responds to a gate perturbation. A large increase or decrease both indicate high local sensitivity. This follows the use of squared first-order Taylor terms in structured pruning (Molchanov et al., 2019). In LOOP, a low RS score therefore favors layers whose gate perturbations have a small first-order effect on the recovery objective.

Q2. How dependent is the method on ground-truth-conditioned teacher supervision?

Thank you for raising this question. In our comparison, the frozen teachers used by LOOP and one-shot OPD both receive the same ground-truth final answer during recovery. In both methods, the student generates trajectories from the question alone and receives only the question at evaluation. The observed advantage of LOOP is therefore measured under matched access to reference answers.

The final answer gives the teacher a target outcome when computing next-token distributions along the student’s own prefixes. The student samples trajectories from the question alone, and the teacher is given only the final answer as additional context. The answer gives the teacher a target outcome that can guide its token-level feedback. A teacher can also provide feedback based on the question and student-generated prefixes alone, as demonstrated by ShortOPD (Zhang et al., 2026) for post-pruning recovery.

The importance criterion remains applicable in that setting. Block Influence is computed on the student’s trajectories, and recovery sensitivity can be computed from the gradients of an OPD loss with an answer-free teacher.

Q3. What exactly does "Recovery-Aligned" mean in LOOP?

Thank you for asking us to clarify this concept. We have revised the title to use Recovery-Oriented, consistent with the terminology in the manuscript. The criterion links layer selection to recovery in two ways. First, we compute each layer’s Block Influence (BI) on the same on-policy student trajectories used for recovery. Second, we compute recovery sensitivity (RS) as the squared gradient of the recovery loss with respect to each layer’s gate, using the same OPD objective that updates the student. This lets LOOP select the next layer based on how it affects representations along the student’s current trajectories and how sensitive the recovery loss is to perturbing that layer.

References

  • Zhang, Qingyu, et al. "ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation." arXiv preprint arXiv:2607.13124 (2026).

  • Chen, Xinrui, et al. "Prune&comp: Free lunch for layer-pruned llms via iterative pruning with magnitude compensation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 40. No. 24. 2026.

  • Kovalev, Grigory, and Mikhail Tikhomirov. "Iterative Layerwise Distillation for Efficient Compression of Large Language Models." Pattern Recognition and Image Analysis 36.2 (2026): 461-472.

Reviewer_xHytOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes LOOP, an iterative layer-pruning framework for reasoning LLMs. Instead of removing all target layers at once and then recovering, LOOP removes one Transformer block per “shot.” After each removal it runs a short on-policy distillation (OPD) phase from the frozen original model, which serves as a teacher conditioned on the ground-truth answer. The next block to remove is chosen by an equal-weight combination of two scores: Block Influence (BI) and a recovery sensitivity score: the squared gradient of the OPD loss with respect to a per-block scalar gate. Under a fixed budget of 200 updates, LOOP outperforms one-shot BI pruning followed by SFT, KD, or OPD. The evaluation uses Qwen3-1.7B/4B/8B and Llama-3.1-8B at 15–25% depth pruning, on math reasoning tasks and SciKnowEval.

Critical strengths
  • The method reuses recovery rollouts and gradients for importance estimation, so selection adds almost no overhead and no separate calibration pass.
  • Table 4 separates the effect of iterative pruning from the effect of the combined importance score. It shows iteration provides most of the gain.
Critical weaknesses
  • Missing discussion of and comparison with training-aware or recovery-aware structure selection [1], [2], [4] and search- or mask-learning-based layer selection [3], [5]. Additionally, [6] directly challenges the core claim of LOOP and is an important reference to mention. They demonstrate that reverse-order pruning, removing final layers, is a simple yet highly effective layer selection metric.
  • How can you validate that representation change is a good metric for reasoning contribution? Furthermore, for recovery sensitivity, squaring the recovery gradient discards its sign, so a block whose removal. Scores are computed only at the block level. Separately gating attention and MLP sublayers would allow finer-grained pruning and test whether the criterion is robust at other granularity.

References:

[1] Tang, Shengkun, et al. "Darwinlm: Evolutionary structured pruning of large language models." arXiv preprint arXiv:2502.07780 (2025).

[2] Bercovich, Akhiad, et al. "Puzzle: Distillation-based nas for inference-optimized llms." arXiv preprint arXiv:2411.19146 (2024).

[3] Sieberling, Oliver, et al. "Evopress: Accurate dynamic model compression via evolutionary search." arXiv preprint arXiv:2410.14649 (2024).

[4] Malettira, Prajna G., et al. "TraceNAS: Zero-shot LLM Pruning via Gradient Trace Correlation." arXiv preprint arXiv:2602.02891 (2026).

[5] Yuan, Tao, et al. "E 3^ 3-Pruner: Towards Efficient, Economical, and Effective Layer Pruning for Large Language Models." arXiv preprint arXiv:2511.17205 (2025).

[6] Lu, Yao, et al. "Reassessing layer pruning in llms: New insights and methods." International Conference on Learning Representations. Vol. 2026. 2026.

Questions for authors

Clarifications

  • For iterative recovery training and final recovery, is LOOP performing doing full finetuning or PEFT? Are the embedding and lm_head kept frozen?
  • Table 6 provides a shallow picture of LOOPs computational cost. Specific hardware, GPU count, peak memory and total wall clock/GPU hours during the iterative pruning and recovery OPD are not provided.
  • Does λ\lambda have to be tuned for each pruning ratio?

Additional Experiments/ Suggestions:

  • BI + RS Pruned (before recovery) are not provided
  • How does LOOP compare with baselines [1]-[5] and reverse-order pruning [6]?
  • Evaluations are mostly restricted to Qwen models and at the 8B scale. Does LOOP scale to larger models, such as Qwen3-32B?
  • How does LOOP perform at higher pruning ratios? I agree that pruning >25% of layers shows drastic degradation. But this opens a good avenue to explore width pruning within the model, similar to [1], [2], [4].

Minor Points:

  • Define OPD and EMA before Fig. 1
  • The title says “Recovery-Aligned,” while the body says “Recovery-Oriented.”

References:

[1] Tang, Shengkun, et al. "Darwinlm: Evolutionary structured pruning of large language models." arXiv preprint arXiv:2502.07780 (2025).

[2] Bercovich, Akhiad, et al. "Puzzle: Distillation-based nas for inference-optimized llms." arXiv preprint arXiv:2411.19146 (2024).

[3] Sieberling, Oliver, et al. "Evopress: Accurate dynamic model compression via evolutionary search." arXiv preprint arXiv:2410.14649 (2024).

[4] Malettira, Prajna G., et al. "TraceNAS: Zero-shot LLM Pruning via Gradient Trace Correlation." arXiv preprint arXiv:2602.02891 (2026).

[5] Yuan, Tao, et al. "E 3^ 3-Pruner: Towards Efficient, Economical, and Effective Layer Pruning for Large Language Models." arXiv preprint arXiv:2511.17205 (2025).

[6] Lu, Yao, et al. "Reassessing layer pruning in llms: New insights and methods." International Conference on Learning Representations. Vol. 2026. 2026.

6 comments

Missing Discussion

Thank you for pointing this out. We have expanded the Related Work section to discuss training-informed structure selection, evolutionary search, and learned pruning masks. These works show how training signals and interactions among candidate layers can inform the choice of a pruned architecture. LOOP addresses a related question: after each pruning shot, can the current student’s on-policy trajectories and recovery-loss gradients improve the next layer selection? We also discuss reverse-order pruning, which shows that removing the final layers in a fixed order can be highly effective. This makes it an important baseline for testing when recovery-driven layer selection adds value over a simple fixed order.

Pruning Criterion Validity

Thank you for raising these important points. We address each of them in turn below.

  • BI for Reasoning. Prior work provides empirical motivation for using BI as a layer selection criterion. ShortGPT (Men et al., 2025) found that BI correlates with perplexity after single-layer removal, and ShortOPD (Zhang et al., 2026) uses BI to select layers before recovery. Bypassing a block replaces its output with its input at the next block. For nonzero input and output hidden states, the per-token BI is defined as:

    =1-\cos(h_{\mathrm{in}},h_{\mathrm{out}}) =\frac{1}{2}| \hat{h}_{i}-\hat{h}_o |_2^2,$$ where $\hat{h}_i$ and $\hat{h}_o$ denotes the unit-normalized in and out hidden states. Thus, a lower BI indicates that bypassing the layer results in a smaller change in the normalized representation passed to the downstream network. In LOOP, we compute BI over complete on-policy trajectories from the current pruned student, including tokens in its generated intermediate reasoning steps. We therefore use on-policy BI as a proxy for a block’s contribution to representation changes during reasoning. A low BI suggests that bypassing the block would induce a small immediate directional change in the representations passed downstream along these trajectories, making it a candidate for pruning.
  • Recovery Sensitivity. LOOP use RS to measure how sensitive the current OPD objective is to a small perturbation of a layer. With model parameters and student trajectories held fixed, reducing the gate from 1 to 1−ϵ1-\epsilon gives a first-order loss change of −ϵdℓ-\epsilon d_\ell, where dℓ=∂LOPD∂gℓd_\ell=\frac{\partial\mathcal L_{\mathrm{OPD}}}{\partial g_\ell} dℓ2d_\ell^2 ranks layers by the magnitude of this change, consistent with squared first-order Taylor criteria for structured pruning (Molchanov et al., 2019). Large increases and decreases both indicate high local sensitivity. LOOP therefore favors layers whose gate perturbations have a small first-order effect on the recovery objective.

  • Pruning Granularity. The scoring criterion can be extended to the sublayer level. We can introduce separate gates on the attention and MLP residual branches and compute recovery sensitivity from the squared OPD loss gradient with respect to each gate. BI can likewise be computed from the representations before and after each residual update. In future work, we will examine whether LOOP also improves upon baselines when pruning attention and MLP sublayers separately.

Clarifications

We thank the reviewer for these questions. For both iterative and final recovery, we use full fine-tuning for LOOP and all baselines that involve recovery training. The student’s input embeddings and language modeling head remain trainable, and the teacher’s weights are fixed. Each experiment used eight NVIDIA H20 GPUs. The table below reports the wall-clock time, GPU-hours, and peak per-GPU memory for LOOP and One-shot OPD at each model and pruning ratio listed.

ModelRatioMethodWall-Clock (h)GPU-HoursPeak Memory (GiB)
Qwen3-1.7B20%LOOP2.39519.16108.92
20%One-shot OPD2.30018.40108.86
Qwen3-4B15%LOOP2.80122.40114.85
15%One-shot OPD2.86422.92115.04
20%LOOP2.85922.87115.02
20%One-shot OPD3.10124.81115.04
25%LOOP2.90923.27115.14
25%One-shot OPD3.16125.28115.12
Qwen3-8B20%LOOP4.17533.40140.36
20%One-shot OPD4.17433.39140.30

We use the same value, λ=0.5\lambda=0.5, for all reported pruning ratios (15%15\%, 20%20\%, and 25%25\%). This gives equal weight to the two normalized scores. We did not tune λ\lambda separately for each pruning ratio and will make this explicit in the experimental setup.

Additional Experiments/ Suggestions

Thank you for these suggestions. We address each question in turn below:

  • BI + RS Pruned. At 20% layer pruning, we evaluated the one-shot BI + RS Pruned Qwen3-4B-Instruct model immediately after pruning without recovery. BI+RS achieves 0.83% mean accuracy, compared with 0.45% for BI alone. The gain indicates a modest advantage for BI+RS immediately after pruning, with substantial reasoning degradation still evident in both variants.

  • Comparison with Other Baseline. We include reverse-order pruning [6] to test whether a simple fixed removal order can match LOOP’s recovery-driven layer selection. On Qwen3-4B-Instruct at 20% layer pruning, we apply the same one-shot OPD recovery protocol to the reverse-order and BI baselines. The results are presented below:

    MethodSelectionDAPOMATH-500AMCAIMEMeanT (%)
    OPDReverse Order0.2030.4220.2750.0660.241544.689%
    OPDBI0.1390.4680.3250.0110.235843.625%
    LOOPBI + OPD0.2380.5220.4250.1220.326860.474%
    The results show that LOOP consistently outperforms both reverse-order pruning and BI-based one-shot OPD across all four mathematical reasoning benchmarks.
  • Scaling to Larger Models. We thank the reviewer for asking whether LOOP scales to larger models. At 20% layer pruning, LOOP consistently improves mean math accuracy over one-shot OPD across Qwen3-1.7B, Qwen3-4B-Instruct, and Qwen3-8B. LOOP also outperforms one-shot OPD at Qwen3-14B, consistent with the results at the three smaller scales. These results extend the empirical evidence for LOOP’s scalability.

    RatioMethodMetricDAPOMATH-500AMCAIMEMeanT (%)
    20%OPDBI0.1110.3040.2000.0110.156559.573%
    20%LOOPBI + RS0.2230.5340.3250.0440.2816107.178%
  • Higher Pruning Ratios. We evaluated LOOP and the best baseline, one-shot OPD, on Qwen3-4B-Instruct at layer pruning ratios of 30% and 40%. As shown in the table, LOOP improves mean accuracy across the four mathematical reasoning benchmarks from 1.38% to 4.45% at 30% pruning and from 0.00% to 2.15% at 40% pruning. These results show that LOOP retains an advantage over one-shot OPD at higher pruning ratios, although absolute reasoning accuracy remains low under our recovery setup.

    RatioMethodMetricDAPOMATH-500AMCAIMEMeanT (%)
    30%OPDBI0.0070.0480.0000.0000.01382.544%
    30%LOOPBI + RS0.0110.1420.0250.0000.04458.235%
    40%OPDBI0.0000.0000.0000.0000.00000.000%
    40%LOOPBI + RS0.0030.0580.0250.0000.02153.979%

Minor Points

Thank you for your suggestions. We have revised the manuscript to define on-policy distillation (OPD) in the main text before Figure 1 and exponential moving average (EMA) in the figure caption. We have also changed “Recovery-Aligned” to “Recovery-Oriented” in the title to match the terminology used in the main text.

References

  • Men, Xin, et al. "Shortgpt: Layers in large language models are more redundant than you expect." Findings of the Association for Computational Linguistics: ACL 2025. 2025.

  • Zhang, Qingyu, et al. "ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation." arXiv preprint arXiv:2607.13124 (2026).

  • Molchanov, Pavlo, et al. "Importance estimation for neural network pruning." 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE, 2019.

Comments on the paper

5 comments

Experiment with Short-to-Long Controller

We thank the reviewer for the valuable feedback. Since ShortOPD’s implementation was unavailable to us, we reimplemented the controller as described in the paper and evaluated both one-shot OPD and LOOP on Qwen3-4B-Instruct using the same rollout schedule in each comparison.

MethodRollout ScheduleMetricDAPOMATH-500AMCAIMEMeanT
OPDFixedBI0.1390.4680.3250.0110.235843.625%
OPDShort-to-LongBI0.1440.4280.3000.0220.223541.358%
LOOPFixedBI + RS0.2380.5220.4250.1220.326860.474%
LOOPShort-to-LongBI + RS0.2220.5040.3250.1110.290553.756%

As shown in the table, LOOP outperformed one-shot OPD under both schedules. With the reimplemented short-to-long controller, LOOP achieved 29.05% versus 22.35%, a gain of 6.70 points. This result is also consistent with the findings reported in the ShortOPD paper. Thus, LOOP’s accuracy advantage persisted when both methods used the short-to-long controller.

Fixed vs. Recovery-Updated Layer Rankings

We thank the reviewer for suggesting this control. We added an iterative baseline that computes a BI+RS ranking after warm-up and keeps it fixed for all subsequent pruning shots. The baseline and LOOP start from the same ranking and use the same pruning schedule, per-shot recovery budgets, and final recovery budget.

MethodDAPOMATH-500AMCAIMEMean
Iterative + Fixed BI+RS Ranking0.2520.5190.4000.0560.3068
LOOP (Updated BI+RS Ranking)0.2380.5220.4250.1220.3268

Updating the ranking during recovery improves mean accuracy from 30.68% to 32.68%, a gain of 2.00 percentage points. This comparison isolates the effect of updating layer importance during staged recovery.

Compute Budget and Extended KD Training

We thank the reviewer for raising the concern. In our setup, 200 OPD updates generate student rollouts for approximately one pass over the training set. Offline KD also requires the unpruned teacher to generate responses for approximately one pass over the same prompts. The two methods therefore incur generation workloads of comparable scale when response lengths are similar.

MethodStepsDAPOMATH-500AMCAIMEMeanTTime (h)
KD2000.0520.3380.0750.0000.116321.512%0.619
KD6000.0570.3040.0750.0000.109020.170%1.618
KD12000.0620.2960.0500.0000.102018.875%3.287
OPD2000.1390.4680.3250.0110.235843.625%3.101
LOOP2000.2380.5220.4250.1220.326860.474%2.859

To test whether more KD updates could close the accuracy gap, we extended one-shot KD from 200 to 600 and 1200 updates on Qwen3-4B-Instruct at 20% layer pruning. The KD times reported in the table exclude the time required to generate the offline distillation data. Even with more updates, KD remained below both one-shot OPD and LOOP at 200 updates. LOOP’s advantage over KD therefore persisted when KD has more recovery budget.

References

  • Zhang, Qingyu, et al. "ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation." arXiv preprint arXiv:2607.13124 (2026).

We thank all three reviewers for their careful reading and constructive feedback. We appreciate their recognition of the following strengths of LOOP.

  • Practical relevance. Reviewer VaK3 highlighted the importance of preserving reasoning performance under layer-level compression and the clear formulation of this problem.
  • Consistent empirical gains. Reviewers VaK3 and AwJz noted gains across pruning ratios and model scales, as well as gains on scientific reasoning tasks and under a longer recovery schedule.
  • Reuse of recovery signals and component analysis. Reviewers VaK3 and xHyt highlighted the reuse of OPD rollouts and gradients for layer selection. Both also noted that the original ablation examined the contributions of iterative pruning and the combined importance score.

In response to the reviewers’ feedback, we added the following results and clarifications.

  • Ranking updates. Under the same recovery schedule, updating the BI+RS ranking improved mean math accuracy from 30.68% to 32.68% over a fixed ranking.
  • Recovery sensitivity. RS rankings broadly agreed with the observed changes in OPD loss at both ends of the ranking after individual layers were fully removed.
  • Additional baselines. We expanded the related work and added comparisons with Iterative Layer-wise Distillation, Prune&Comp, and reverse-order pruning. LOOP also outperformed one-shot OPD under the same reimplemented short-to-long rollout schedule (29.05% vs. 22.35%).
  • Scale and compute. We extended evaluation to Qwen3-14B, reported wall-clock time, GPU-hours, and peak memory, and tested KD with up to 1,200 updates. The reported KD times exclude offline teacher-data generation.

Our individual responses provide further details. We thank the reviewers again for helping us strengthen the paper and would appreciate their taking the additional evidence into account when reconsidering their assessments.

Dear Reviewer,

Thank you again for taking the time to review our work.

We understand that this is a busy period. We would like to kindly follow up to let you know that we have addressed each of your questions and concerns and uploaded a revised manuscript.

When you have a chance, we would appreciate your feedback on whether our responses have addressed your concerns. We would also be happy to answer any remaining questions.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.