acceptodds
Under review as a conference paper at ICLR 2027

ShortOPD: Recovering Pruned LLMs with Short-to-Long On-Policy Distillation

Abstract

Structured pruning is a hardware-friendly way to compress LLMs, but preserving recognition scores does not ensure useful free-form generation. On-Policy Distillation (OPD) offers a recovery signal by having the frozen pre-compression model supervise the student's own response prefixes. In the pruning regime we study, however, long rollouts frequently develop repetitive suffixes. The teacher can also assign these suffixes high probability, so low distillation loss can coexist with poor generation. We propose ShortOPD, an OPD rollout schedule guided by terminal repetition and clean truncation. Repetition permits reducing the next budget toward an estimated prefix length, whereas non-repetitive responses that reach the cap permit growth. The controller reuses existing tokens and teacher scores, requires no additional model forward pass, and preserves the current batch's full-token distillation update. Its short-to-long phase follows an initial contraction from the maximum budget. Across four Qwen3 model settings with 25% of layers removed, ShortOPD attains 82.3–84.6% of the teacher's eight-task aggregate generation score, outperforming Vanilla OPD by 11.4–18.2 points. Under an 8192-token ceiling, it also uses 14.7–30.2% fewer rollout tokens while scoring 13.95–16.21 points higher. We hope this recipe helps move structured pruning beyond marginal gains on perplexity and multiple-choice benchmarks, a step closer to deployment-ready generation quality.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Model Compression

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XpkWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how to recover free-form generation quality after structured pruning of LLMs. The authors observe that pruned Qwen models retain much of their original ability, but pruning introduces periodic, repetitive suffixes that can be mistaken for mode collapse. To recover this latent ability, they propose Short On-Policy Distillation (ShortOPD). ShortOPD adds three components to standard OPD: a heuristic feedback controller over the rollout budget, a token-level terminal-periodic loop detector, and a clean-truncation statistic. Together, these components dynamically shrink and expand the budget, so that OPD avoids repetitive loops that waste compute and dilute the gradient signal. The paper reports strong results against vanilla OPD, RL baselines, and offline distillation baselines.

Critical strengths

This paper focuses on an interesting problem: mode collapse after structured pruning in LLMs. Their observation that structurally pruned LLMs can still exhibit former capabilities is surprising and worth investigating. The proposed method is cheap and can easily be plugged into existing OPD pipelines. The results, as presented, are compelling.

Critical weaknesses
  1. From Fig. 3b, it seems that the budget bounces back to the ceiling fairly early on, which is close to a hand-written two-phase scheduler. I believe we are missing: a) OPD trained exclusively at 1024, which is our floor; b) OPD trained with a fixed 1024-to-2048 step or linear schedule; and c) OPD that masks looped tokens from the loss. Without these three, it is hard for me to understand the true impact of ShortOPD.
  2. Again from Fig. 3b, once ShortOPD bounces back to 2048, it and OPD are the same algorithm. This happens quite early on (in fewer than 100 steps). Can the authors address where the performance uplift comes from? Does it all happen during the initial 100 steps?
  3. I believe the PPO and GRPO comparison is quite misleading. They both start from a student with GSM8k 1.14% and HumanEval %0 and a binary reward. Reward essentially never fires, so a near-zero results are expected. A fairer comparison would be to add a short SFT/KD/OPD warm-up before PPO or GRPO, to improve the odds of getting non-zero reward groups.
  4. Given that the controller and detector have a large number of hyperparameters, can the authors illustrate the selection criteria or protocol used to determine which values to use? The paper never discusses how they were chosen.
  5. Currently, all observations and results are derived from Qwen-based models. Can the authors discuss more on the generalizability of the observations and results?
Questions for authors

see weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer XpkW (1/2)

Thank you for your encouragement and the concrete suggestions for controls. We ran all three.

Weakness 1: Missing controls

We ran all three controls with the same setup as Table 3:

Control strategy (Qwen3-4B-Instruct)AvgDiff vs. ShortOPD
ShortOPD, repetition-gated feedback (Table 3)61.89–
Vanilla OPD, fixed H = 2048 (Table 3)49.39−12.50
(a) OPD trained exclusively at 1024 (the floor)44.19−17.70
(b) OPD with a fixed 1024-to-2048 step schedule (switch at step 58, from Fig. 3)54.17−7.72
(b) OPD with a fixed 1024-to-2048 linear schedule (over 58 steps, from Fig. 3)51.36−10.53
(c) OPD that masks looped tokens from the loss59.23−2.66

These three controls show that the main source of the gain is removing low-signal loop tokens, but a hand-written two-phase schedule cannot replace online feedback. We discuss each in turn:

  • (a) Training only at 1024. Avg is only 44.19, below Vanilla OPD. Responses longer than 1024 tokens are truncated throughout training, so positions beyond 1024 never receive supervision.
  • (b) 1024-to-2048 step and linear schedules. Both schedules are timed at the point where ShortOPD's budget starts to grow in Fig. 3 (around step 58). Yet they reach only 54.17 and 51.36, i.e., 7.72 and 10.53 points below ShortOPD.
  • (c) Masking loop tokens. Masking reaches 59.23, recovering 79% of ShortOPD's gain over Vanilla OPD (12.50 points). However, it is still 2.66 points below ShortOPD and still requires generating full 2048-token rollouts at every step.

In addition, directly replaying ShortOPD's full budget curve gives 58.68, still 3.21 points below ShortOPD (see the General Response). This curve is itself a product of the controller and cannot be known in advance. Online feedback therefore contributes beyond the short-to-long shape. Moreover, ShortOPD uses the same constants in all settings, so unlike a hand-written schedule, it does not require designing a switch time for each setting.

Weakness 2: Where does the gain come from?

Intermediate checkpoints of the two runs on Qwen3-4B-Instruct:

StepVanilla OPD AvgShortOPD AvgAvg diff
1008.6519.36+10.71
20022.8931.16+8.27
40028.9740.71+11.74
71049.3961.89+12.50

At step 100, the gap is already 10.71 points, about 86% of the final gap, so the gap does form mainly early on. However, once the budget returns to 2048, the two runs use exactly the same update rule, yet Vanilla OPD never catches up over the remaining 600+ steps. The gap at step 710 (12.50) is even larger than at step 100. This shows that differences formed early persist through the training data that each student generates itself. ShortOPD acts mainly early in training, but its benefit carries through the entire run.

AuthorsOpenReview ↗

Response to Reviewer XpkW (2/2)

Weakness 3: RL comparison

We agree: from a collapsed initialization, outcome rewards almost never fire, which is exactly the point of RQ2 (Section 3.3). To rule out the effect of initialization, we followed your suggestion and first ran a 462-step SFT cold start on the dataset of Appendix D.4, followed by 462 steps of PPO/GRPO:

Method (Qwen3-4B-Instruct, Math+Code)Avg
Pruned (reference)0.69
PPO, from the pruned model0.00
GRPO, from the pruned model0.43
SFT cold start, then PPO24.97
SFT cold start, then GRPO26.83
ShortOPD38.03

After the cold start, PPO and GRPO reach 24.97 and 26.83. Despite using twice as many training steps, they remain more than 11 points below ShortOPD.

Weakness 4: How were the hyperparameters chosen?

We set these constants directly on Qwen3-4B-Instruct according to their meaning, without dedicated tuning. All subsequent experiments reuse the same values: the other three Qwen3 settings, width/hybrid pruning, and Gemma. The main constants mean the following:

  • The budget shrinks when close to half of the rollouts in a batch loop (ρ_high = 0.45).
  • The budget grows when looping is rare (ρ_low = 0.20) and more than 10% of rollouts are clean truncations (τ = 0.10).
  • λ = 1.15 leaves a 15% margin beyond the usable length.

We varied the thresholds and smoothing parameters one at a time:

HyperparameterValueAvg
ShortOPD default (Table 9)–61.89
ρ_high (shrink threshold)0.3061.13
ρ_high (shrink threshold)0.6062.09
τ (growth threshold)0.0562.37
τ (growth threshold)0.2059.49
β = β_H (EMA smoothing)0.558.89
β = β_H (EMA smoothing)0.961.81
Vanilla OPD (reference)–49.39

All variants outperform Vanilla OPD by at least 9.50 points.

Weakness 5: Beyond Qwen

We added experiments on another model family, Gemma-4-E2B-it (9 of 35 layers removed):

Model (depth pruning)Pruned AvgVanilla OPD AvgShortOPD Avg
Gemma-4-E2B-it · 9/35 layers5.5939.8145.97

Pruning drops Gemma's Avg from 61.87 to 5.59. After recovery training, ShortOPD restores it to 45.97 (74% of the unpruned model), 6.16 points above Vanilla OPD. Neither the observation that capability remains recoverable after pruning nor ShortOPD's gains are therefore limited to Qwen3.

I thank the authors for running the experiments. The new results are informative. However, from the new ablation study, most of the gain from the current approach is simply from filtering or limiting looped tokens, which already reached 59.23. Compared to ShortOPD 61.89, this is already within noise variance. From the paper's own argument that loops dilute gradients, I believe masking is the more direct approach given your method for loop detection. I argue that the adaptive rollout budget here is not necessary.

As for my question 2, the table only shows that ShortOPD gives the model a head start. But subsequent steps improve at the same rate at around ~20%, which is not evidence for vanilla OPD not catching up. Longer training runs would perhaps have both runs saturate at the same accuracy.

Given that the gain is largely from masking looped tokens, I find insufficient evidence that ShortOPD, or its adaptive rollout budget, adds much benefit to overall training. As submitted, the claimed contribution does not match where the gains come from, so I am keeping my score.

Thank you for the follow-up. We respond to each point below.

1. Masking vs. ShortOPD. We agree that most of the gain over Vanilla OPD comes from removing looped tokens. The 2.66-point gap between ShortOPD and masking (61.89 vs. 59.23), however, is not within noise. Results across seeds:

Setting / MethodSeed 42 (Table 3)Seed 24Seed 64Mean ± std
4B-Instruct · Vanilla OPD49.3949.1149.1449.21 ± 0.15
4B-Instruct · ShortOPD61.8962.2161.7761.96 ± 0.23
8B no-thinking · Vanilla OPD52.0451.0951.7451.62 ± 0.49
8B no-thinking · ShortOPD63.4064.2163.5463.72 ± 0.43

2. Why not simply mask. Masking keeps a fixed budget, so a looping rollout still runs to the cap: its looped tokens are generated and only then discarded, after the generation compute has been spent. ShortOPD uses the same loop-detection signal to shrink the budget before generation, so these tokens are not generated. Under an 8192-token ceiling, ShortOPD uses 14.7–30.2% fewer rollout tokens and 15.3–24.8% less training time than Vanilla OPD (Table 5); under the 2048 ceiling, generation time over the first 100 steps falls from 35.4 to 25.1 seconds per step (Fig. 3). ShortOPD lowers generation cost while also scoring higher than masking.

3. Would longer training close the gap? In the intermediate-checkpoint table of our previous response, from step 400 to step 710 ShortOPD improves by 21.18 points (40.7→ 61.89) and Vanilla OPD by 20.42 points (28.97 → 49.39). The later improwith ShortOPD slightly faster, so the gap does not narrow (11.74 →12.50). Step 710 is the full training budget shared by all methods; under this budget, ShortOPD scores higher at a lower cost per step.

4. Contribution and the source of the gains. As stated in the first contribution of our introduction, the paper proposes a recovery recipe for generation quality after structured pruning, built on three ingredients: supervision on the studendense token-level teacher distributions, and a rollout budget matched towhat the damaged policy can currently generate. The first two adapt OPD to structured-pruning recovery; Vanilla OPD alone already exceeds SFT, SeqKD, and KD by 10.0–20.2 points across four settings (Table 3). The third comes from our finding tode of pruned models under OPD is repetitive suffixes, to which theteacher also assigns high probability. Both loop masking and ShortOPD build on this finding and on our loop detector, and ShortOPD is the cheaper and more effective of the two. As our abstract states, we hope this recipe helps move structured pruning beyond perplexity and multiple-choice benchmarks, toward deployment-ready generation quality.

Reviewer_bm5UOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ShortOPD, an adaptive rollout-length schedule for recovering structurally pruned LLMs through on-policy distillation. The method addresses repetitive suffixes that consume computation while receiving little corrective supervision, as the teacher may also assign them high probability. It shortens subsequent rollouts when repetition is frequent and increases the budget when clean truncation becomes common. Across four Qwen3 settings with 25% depth pruning, ShortOPD reaches 82.3–84.6% of the teacher’s aggregate generation score and improves both quality and training efficiency over fixed-length OPD.

Critical strengths
  • The paper provides a useful observation that low teacher–student divergence can coexist with repetitive and low-quality generation.

  • The reported improvements over fixed-length OPD are consistent across the tested Qwen3 model sizes and thinking modes.

Critical weaknesses
  • Insufficient baseline comparisons. The paper lacks comparisons with fixed short rollouts and preset length schedules. Repetition-tail masking and alternative loss normalization would also help establish whether the gains require adaptive control or mainly reflect reduced gradient dilution.

  • Limitations of batch-level control. Repetitive suffixes in the current batch still consume computation and enter loss normalization. Meanwhile, shortening a shared budget may truncate legitimate long responses because other samples are repetitive. These trade-offs are not sufficiently examined.

  • Limited generality. The main experiments cover only Qwen3 models at 4B/8B scales with 25% depth pruning, a setting selected for repetition-dominated failures. Applicability to other models, pruning levels, and failure modes remains unclear. The width and hybrid experiments also lack a Vanilla OPD baseline.

  • Insufficient robustness analysis. The controller uses heuristic thresholds and smoothing parameters without sensitivity analysis or reported variation across independent training seeds. The local gradient analysis motivates reducing repetitive tokens but does not establish the reliability of the specific control rules.

Questions for authors

same as weakness

2 comments
AuthorsOpenReview ↗

Response to Reviewer bm5U (1/2)

Thank you for your review. We address your four points below.

Weakness 1: Insufficient baselines

We added all the baseline types you mentioned, with the same setup as Table 3:

Control strategy (Qwen3-4B-Instruct)AvgDiff vs. ShortOPD
ShortOPD, repetition-gated feedback (Table 3)61.89–
Vanilla OPD, fixed H = 2048 (Table 3)49.39−12.50
Fixed short rollouts, H = 102444.19−17.70
Preset length schedule: shrink-then-grow (replays ShortOPD's budget curve)58.68−3.21
Preset length schedule: 1024-to-2048 step (switch at step 58, from Fig. 3)54.17−7.72
Preset length schedule: 1024-to-2048 linear (over 58 steps, from Fig. 3)51.36−10.53
Repetition-tail masking, H = 204859.23−2.66
Alternative loss normalization (token-level), H = 204843.27−18.62

These baselines show that mitigating gradient dilution is the main source of the gain, but not all of it: adaptive control makes an independent contribution on top of it. We discuss each in turn:

  • Fixed short rollouts. Avg is only 44.19, below Vanilla OPD. Responses longer than 1024 tokens are truncated throughout training, so positions beyond 1024 never receive supervision.
  • Preset length schedules. The best of these, directly replaying ShortOPD's own budget curve, reaches 58.68. This is still 3.21 points below ShortOPD, and the curve is itself a product of the controller, so it cannot be known in advance. Hand-designed step and linear schedules are lower still (54.17 and 51.36), which shows that results are sensitive to schedule timing.
  • Repetition-tail masking. Masking reaches 59.23, recovering 79% of ShortOPD's gain over Vanilla OPD. However, it is still 2.66 points below ShortOPD and still requires generating full 2048-token rollouts at every step.
  • Token-level normalization. Avg is 43.27, also below Vanilla OPD. Looping responses are longer, so token-level averaging actually increases the share of loop tokens in the update.

Weakness 2: Limitations of batch-level control

You point out two costs: loops in the current batch, and a shared budget that may truncate legitimate long responses. These are precisely the two ends of the trade-off that ShortOPD's controller balances, and we monitor both signals by design:

  • The repetition rate r̄ drives shrinking. The budget shrinks only when the batch repetition rate exceeds a threshold, so that loop tokens in subsequent rollouts are reduced quickly. Loops in the current batch still enter that step's update, but only briefly, because the budget moves smoothly toward the target within a few steps. In the diagnostic run of Fig. 3, ShortOPD's repeated-suffix tokens drop faster than Vanilla OPD's. Over the first 100 steps, generation takes 25.1 s per step for ShortOPD versus 35.4 s for Vanilla OPD.
  • The clean-truncation rate q̄ drives growth. q̄ is the fraction of rollouts that fill the budget without looping, i.e., legitimate responses that get truncated. When q̄ exceeds a threshold and repetition is low, the budget grows back. The budget also never drops below 1024, so truncated responses are still supervised on their prefixes. In Fig. 3, the budget falls to 1024 within roughly the first 15 steps, starts growing at around step 58, and returns to 2048 at around step 72.

Empirically, the benefit of this trade-off far outweighs its cost. ShortOPD outperforms Vanilla OPD by 11.36–18.20 points across all four settings (Table 3), with the largest gains in the Thinking settings that require long reasoning (+17.32 / +18.20).

AuthorsOpenReview ↗

Response to Reviewer bm5U (2/2)

Weakness 3: Generality

Appendix A sweeps all pruning depths. Whenever at most 12 of 36 layers are removed, repetition is the dominant failure mode, peaking at 84% with 9 layers removed. When the student does not loop, ShortOPD reduces exactly to Vanilla OPD (Eq. 3), so it does no harm outside this range. We added Vanilla OPD under width/hybrid pruning:

Student (parameter reduction)Pruned AvgMinitron KD AvgVanilla OPD AvgShortOPD Avg
4B-Instruct · width (25.1%)9.5641.2246.5950.44
4B-Instruct · hybrid (25.1%)15.2642.8448.9353.94
8B no-thinking · width (25.0%)12.4546.0847.1253.52
8B no-thinking · hybrid (25.1%)10.6745.7147.5553.59

ShortOPD outperforms Vanilla OPD by 3.85–6.40 points on all four students.

We also added experiments on another model family, Gemma-4-E2B-it (9 of 35 layers removed):

Model (depth pruning)Pruned AvgVanilla OPD AvgShortOPD Avg
Gemma-4-E2B-it · 9/35 layers5.5939.8145.97

On Gemma, ShortOPD outperforms Vanilla OPD by 6.16 points, which shows that its gains are not limited to Qwen3.

Weakness 4: Robustness

All experiments use the same thresholds and smoothing parameters, with no per-setting tuning. This covers the four Qwen3 settings, width/hybrid pruning, and the new Gemma runs, and the gains are consistent across all of them. We further varied these parameters one at a time on Qwen3-4B-Instruct:

HyperparameterValueAvg
ShortOPD default (Table 9)–61.89
ρ_high (shrink threshold)0.3061.13
ρ_high (shrink threshold)0.6062.09
τ (growth threshold)0.0562.37
τ (growth threshold)0.2059.49
β = β_H (EMA smoothing)0.558.89
β = β_H (EMA smoothing)0.961.81
Vanilla OPD (reference)–49.39

All variants outperform Vanilla OPD by at least 9.50 points, and 4 of them are within 0.8 points of the default. Results across seeds:

Setting / MethodSeed 42 (Table 3)Seed 24Seed 64Mean ± std
4B-Instruct · Vanilla OPD49.3949.1149.1449.21 ± 0.15
4B-Instruct · ShortOPD61.8962.2161.7761.96 ± 0.23
8B no-thinking · Vanilla OPD52.0451.0951.7451.62 ± 0.49
8B no-thinking · ShortOPD63.4064.2163.5463.72 ± 0.43

Across the three seeds, the standard deviation is 0.15–0.49, far smaller than the gap between methods. Together, the sensitivity and multi-seed results show that the specific control rules are stable and reliable.

Reviewer_dbA8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper is clearly written and I enjoyed reading the paper. The paper studies recovering structurally pruned LLMs for free-form generation. It argues that after 25% BI depth pruning (9/36 layers) of Qwen3 models, on-policy rollouts are dominated by repetitive suffixes that the frozen teacher also assigns high probability.

Critical strengths

Large, consistent gains on a problem that matters. Generation-level recovery after structured pruning is what blocks deployment. ShortOPD improves over Vanilla OPD by 11.4–18.2 Avg in all four Qwen3 settings and wins every task in every block. Their method also beats Minitron KD by 7.4–11.1 points under width and hybrid pruning reported in Table 7. The method is cheap: it reuses OPD's tokens and teacher scores and needs no extra forward pass.

Critical weaknesses
  • The controller's contribution is not isolated, and simpler alternatives are untested. By the paper's own explanation, the gain comes from removing low-signal loop tokens from the per-response normalization of Eq. (1). Several simpler interventions would test this directly: (i) a fixed H=1024, which is the controller's own lower bound; (ii) a preset shrink-then-grow schedule with no feedback, in the spirit of progressive-length schedules used in RL; (iii) masking the tokens after the loop onset that the detector already finds; (iv) token-level instead of per-response loss normalization. None of these is reported. Without them, the gap may reflect a weak fixed-2048 baseline rather than the repetition-gated feedback that the paper presents as its contribution. Only one seed is shown, so run-to-run variance is also unknown.

  • Several claims rest on narrow or loosely controlled evidence. The PPO/GRPO comparison in Table 4 starts from that collapsed initialization, so a failure of sparse rewards is expected; 4B PPO even falls to 0.00, below the pruned model, and its steps and tuning are not reported.

  • Table 7 compares ShortOPD only to Minitron KD and not to Vanilla OPD, so the claim that the controller generalizes to width/hybrid pruning is untested.

Questions for authors

-What are the evaluation decoding settings and maximum generation length, especially for the Thinking models, given that training rollouts are capped at 2048?

-How many seeds stand behind Table 3, and what is the variance?

  • How many steps and what KL coefficient and other tuning did PPO/GRPO use?
2 comments
AuthorsOpenReview ↗

Response to Reviewer dbA8 (1/2)

Thank you for the positive assessment and the very concrete list of controls. We ran all of them.

Weakness 1: The controller's contribution is not isolated

We ran all four controls you listed, with the same setup as Table 3:

Control strategy (Qwen3-4B-Instruct)AvgDiff vs. ShortOPD
ShortOPD, repetition-gated feedback (Table 3)61.89–
Vanilla OPD, fixed H = 2048 (Table 3)49.39−12.50
(i) Fixed H = 1024, the controller's lower bound44.19−17.70
(ii) Preset shrink-then-grow schedule, no feedback58.68−3.21
(iii) Masking the tokens after the detected loop onset59.23−2.66
(iv) Token-level instead of per-response loss normalization43.27−18.62

These four controls show that the main source of the gain is indeed removing low-signal loop tokens, and that online feedback makes an independent contribution on top of that. We discuss each in turn:

  • (i) Fixed H = 1024. Avg is only 44.19, below Vanilla OPD. Shortening the budget alone is not enough: responses longer than 1024 tokens are truncated throughout training, so positions beyond 1024 never receive supervision.
  • (ii) Preset shrink-then-grow schedule without feedback. Directly replaying ShortOPD's own budget curve gives 58.68. This recovers 74% of ShortOPD's gain over Vanilla OPD (12.50 points) but is still 3.21 points below ShortOPD. Moreover, this curve is itself a product of the controller and cannot be known in advance.
  • (iii) Masking tokens after the detected loop onset. Masking reaches 59.23, recovering 79% of this gain. This confirms your intuition: removing low-signal loop tokens is the main source of the gain. However, it is still 2.66 points below ShortOPD and still requires generating full 2048-token rollouts at every step.
  • (iv) Token-level normalization. Avg is 43.27, also below Vanilla OPD. Loop tokens remain in the gradient, and because looping responses are longer, token-level averaging actually increases their share of the update.

Results across seeds:

Setting / MethodSeed 42 (Table 3)Seed 24Seed 64Mean ± std
4B-Instruct · Vanilla OPD49.3949.1149.1449.21 ± 0.15
4B-Instruct · ShortOPD61.8962.2161.7761.96 ± 0.23
8B no-thinking · Vanilla OPD52.0451.0951.7451.62 ± 0.49
8B no-thinking · ShortOPD63.4064.2163.5463.72 ± 0.43

Across the three seeds, the standard deviation is 0.15–0.49, far smaller than the gap between methods. The controls above are single runs. At this noise level, the 2.66- and 3.21-point gaps are also clearly larger than seed-to-seed variation.

AuthorsOpenReview ↗

Response to Reviewer dbA8 (2/2)

Weakness 2: PPO/GRPO start from a collapsed initialization

You are right: from a collapsed initialization, sparse rewards rarely fire, which is exactly the point of RQ2 (Section 3.3). To rule out the effect of initialization, we first ran a 462-step SFT cold start on the dataset of Appendix D.4, followed by 462 steps of PPO/GRPO:

Method (Qwen3-4B-Instruct, Math+Code)Avg
Pruned (reference)0.69
PPO, from the pruned model0.00
GRPO, from the pruned model0.43
SFT cold start, then PPO24.97
SFT cold start, then GRPO26.83
ShortOPD38.03

After the cold start, PPO and GRPO reach 24.97 and 26.83. Despite using twice as many training steps, they remain more than 11 points below ShortOPD.

For the 4B model, PPO's 0.00 and the pruned model's 0.69 are both near zero. The same PPO setup reaches 24.97 after the cold start, which indicates that the 0.00 stems from the initialization rather than the training setup. All methods are trained for 462 steps. The remaining settings and tuning details are given in our response to Question 3.

Weakness 3: Table 7

You are right that Table 7 compares only against Minitron KD and does not isolate the controller. We additionally ran Vanilla OPD for these four students under the same setup:

Student (parameter reduction)Pruned AvgMinitron KD AvgVanilla OPD AvgShortOPD Avg
4B-Instruct · width (25.1%)9.5641.2246.5950.44
4B-Instruct · hybrid (25.1%)15.2642.8448.9353.94
8B no-thinking · width (25.0%)12.4546.0847.1253.52
8B no-thinking · hybrid (25.1%)10.6745.7147.5553.59

ShortOPD outperforms Vanilla OPD by 3.85–6.40 points on all four students. The controller is therefore also effective under width/hybrid pruning, although the gain is smaller than under depth pruning.

Question 1: Evaluation settings

All methods use greedy decoding. For the Instruct models, the maximum generation length at evaluation is 16384 tokens for MATH-500 and 2048 for the other tasks. For the Thinking models, the training rollout cap is 8192. At evaluation, the limit is 16384 for MATH-500 and 8192 for the other tasks. Evaluation limits are never lower than the corresponding training cap. Section 3.1 misstates the training budget for the Thinking models. We will correct the experimental setup in the main text and add the evaluation settings above.

Question 2: Seeds

Table 3 reports single runs (seed 42). For Vanilla OPD and ShortOPD on 4B-Instruct and 8B (no-thinking), we added seeds 24 and 64 (see the seed table under Weakness 1). The standard deviation across the three seeds is 0.15–0.49.

Question 3: RL settings

PPO and GRPO are both trained for 462 steps on the 29,585 Math+Code prompts of Appendix D.4. We use batch size 64, 8 rollouts per prompt, and a 2048-token limit on both prompts and responses. Other settings: actor learning rate 2×10⁻⁶, KL coefficient 0.0001, clip ratios ε_low = 0.20 and ε_high = 0.28 (i.e., the probability ratio is clipped to [0.80, 1.28]), and rollout temperature 0.7. PPO uses GAE with a critic initialized from the pruned student (learning rate 1×10⁻⁵). GRPO uses group-relative advantages.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response: Simpler alternatives, seeds, width/hybrid pruning, and RL after a cold start

We thank all reviewers for their constructive feedback. Reviewers found the problem important (dbA8, XpkW), the observation valuable and surprising (bm5U, XpkW), the improvements large and consistent (all), and the method cheap and easy to plug into existing OPD pipelines (dbA8, XpkW).

The shared question is whether the gains require ShortOPD's repetition-gated feedback, or whether a simpler intervention would suffice. We ran the controls suggested by the reviewers. The main new results are summarized below.

1. Simpler alternatives (dbA8 W1, bm5U W1, XpkW W1)

All runs use Qwen3-4B-Instruct with the same setup as Table 3.

Control strategy (Qwen3-4B-Instruct)AvgDiff vs. ShortOPD
ShortOPD, repetition-gated feedback (Table 3)61.89–
Vanilla OPD, fixed H = 2048 (Table 3)49.39−12.50
Fixed short budget, H = 102444.19−17.70
Preset shrink-then-grow, no feedback (replays ShortOPD's budget curve)58.68−3.21
Preset 1024-to-2048 step schedule (switch at step 58, from Fig. 3)54.17−7.72
Preset 1024-to-2048 linear schedule (over 58 steps, from Fig. 3)51.36−10.53
Loop-token masking, H = 204859.23−2.66
Token-level loss normalization, H = 204843.27−18.62

These controls support two conclusions: the main source of the gain is removing low-signal loop tokens, and online feedback makes an independent contribution on top of that.

  • Fixed short budget (H = 1024). Avg is only 44.19, below Vanilla OPD. Shortening the budget alone is not enough: responses longer than 1024 tokens are truncated throughout training, so positions beyond 1024 never receive supervision.
  • Preset schedules. Directly replaying ShortOPD's own budget curve gives 58.68, still 3.21 points below ShortOPD. Moreover, this curve is itself a product of the controller and cannot be known in advance. Hand-designed step and linear schedules, timed at the point where the budget starts to grow in Fig. 3 (around step 58), are lower still (54.17 and 51.36). This shows that results are sensitive to schedule timing, whereas online feedback requires no per-setting timing design.
  • Loop-token masking. Masking reaches 59.23, confirming that removing low-signal loop tokens is the main source of the gain. However, it is still 2.66 points below ShortOPD and still requires generating full 2048-token rollouts at every step.
  • Token-level normalization. Avg is 43.27, also below Vanilla OPD. Loop tokens remain in the gradient, and because looping responses are longer, token-level averaging actually increases their share of the update.

2. Multiple seeds (dbA8 Q2, bm5U W4)

The paper reports a single run with seed 42. For each method, we added two independent runs with seeds 24 and 64:

Setting / MethodSeed 42 (Table 3)Seed 24Seed 64Mean ± std
4B-Instruct · Vanilla OPD49.3949.1149.1449.21 ± 0.15
4B-Instruct · ShortOPD61.8962.2161.7761.96 ± 0.23
8B no-thinking · Vanilla OPD52.0451.0951.7451.62 ± 0.49
8B no-thinking · ShortOPD63.4064.2163.5463.72 ± 0.43

Across the three seeds, the standard deviation is 0.15–0.49, far smaller than the gap between the two methods (12.74 and 12.09 points, computed from the means). The worst ShortOPD run is still clearly above the best Vanilla OPD run. The controls in the table above are single runs. At this noise level, the 2.66- and 3.21-point gaps are also clearly larger than seed-to-seed variation.

3. Vanilla OPD under width/hybrid pruning (dbA8 W3, bm5U W3)

Student (parameter reduction)Pruned AvgMinitron KD AvgVanilla OPD AvgShortOPD Avg
4B-Instruct · width (25.1%)9.5641.2246.5950.44
4B-Instruct · hybrid (25.1%)15.2642.8448.9353.94
8B no-thinking · width (25.0%)12.4546.0847.1253.52
8B no-thinking · hybrid (25.1%)10.6745.7147.5553.59

ShortOPD outperforms Vanilla OPD by 3.85–6.40 points on all four students, so the controller is also effective under width and hybrid pruning.

4. RL after a cold start (dbA8 W2, XpkW W3)

From the collapsed initialization, outcome rewards almost never fire, which is exactly the point of RQ2 (Section 3.3). To rule out the effect of initialization, we first ran a 462-step SFT cold start on Qwen3-4B-Instruct with the Math+Code corpus, followed by 462 steps of PPO/GRPO:

Method (Qwen3-4B-Instruct, Math+Code)Avg
Pruned (reference)0.69
PPO, from the pruned model0.00
GRPO, from the pruned model0.43
SFT cold start, then PPO24.97
SFT cold start, then GRPO26.83
ShortOPD38.03

After the cold start, PPO and GRPO reach 24.97 and 26.83, still more than 11 points below ShortOPD.

AuthorsOpenReview ↗

Summary of Contributions and Rebuttal Results

We thank all reviewers for their time and detailed feedback during the discussion. As the discussion period comes to a close, we briefly summarize our contributions and the results added during the rebuttal, for the reviewers and the AC.

Contributions. Most structured-pruning work reports only perplexity and multiple-choice benchmarks, while the free-form generation of pruned models can collapse severely. This paper addresses this overlooked problem with a recovery recipe aimed at generation quality:

  • Observation. Pruning demotes generation ability rather than erasing it: correct trajectories remain in the student's own sampling distribution (pass@64 on GSM8K reaches 91%, above the unpruned teacher's greedy 88%), so recovery calls for token-level supervision on the student's own trajectories.
  • Method. We therefore adapt OPD to structured-pruning recovery; Vanilla OPD alone already exceeds SFT, SeqKD, and KD by 10.0–20.2 points across four settings. We further find that the dominant failure mode of pruned models under OPD is repetitive suffixes, to which the teacher also assigns high probability. ShortOPD uses a loop-detection signal to control the rollout budget. Across four Qwen3 settings, it reaches 82.3–84.6% of the teacher's generation score and outperforms Vanilla OPD by 11.4–18.2 points; under an 8192-token ceiling, it uses up to 30% fewer rollout tokens and 25% less training time.

Results added during the rebuttal. Following the reviewers' suggestions, we added:

  • Simpler alternatives: a fixed short budget, preset schedules, a replay of ShortOPD's budget curve, loop-token masking, and token-level normalization all score below ShortOPD.
  • Multiple seeds: standard deviations across three seeds are 0.15–0.49, far smaller than the gaps between methods.
  • Broader scope: ShortOPD outperforms Vanilla OPD by 3.85–6.40 points under width and hybrid pruning, and by 6.16 points on Gemma.
  • RL after a cold start: PPO and GRPO after an SFT cold start remain more than 11 points below ShortOPD.
  • Hyperparameter sensitivity: every variant outperforms Vanilla OPD by at least 9.50 points.

We also state plainly that these controls show most of the gain over Vanilla OPD comes from removing low-signal looped tokens; building on this, ShortOPD shortens the budget before generation and achieves a higher score at a lower cost.

We hope this work helps move structured pruning beyond perplexity and multiple-choice benchmarks toward deployment-ready generation quality. We kindly ask the reviewers to take these additional results into account in their final assessment. Thank you again for your valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.