- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies how to recover free-form generation quality after structured pruning of LLMs. The authors observe that pruned Qwen models retain much of their original ability, but pruning introduces periodic, repetitive suffixes that can be mistaken for mode collapse. To recover this latent ability, they propose Short On-Policy Distillation (ShortOPD). ShortOPD adds three components to standard OPD: a heuristic feedback controller over the rollout budget, a token-level terminal-periodic loop detector, and a clean-truncation statistic. Together, these components dynamically shrink and expand the budget, so that OPD avoids repetitive loops that waste compute and dilute the gradient signal. The paper reports strong results against vanilla OPD, RL baselines, and offline distillation baselines.
Critical strengths
This paper focuses on an interesting problem: mode collapse after structured pruning in LLMs. Their observation that structurally pruned LLMs can still exhibit former capabilities is surprising and worth investigating. The proposed method is cheap and can easily be plugged into existing OPD pipelines. The results, as presented, are compelling.
Critical weaknesses
- From Fig. 3b, it seems that the budget bounces back to the ceiling fairly early on, which is close to a hand-written two-phase scheduler. I believe we are missing: a) OPD trained exclusively at 1024, which is our floor; b) OPD trained with a fixed 1024-to-2048 step or linear schedule; and c) OPD that masks looped tokens from the loss. Without these three, it is hard for me to understand the true impact of ShortOPD.
- Again from Fig. 3b, once ShortOPD bounces back to 2048, it and OPD are the same algorithm. This happens quite early on (in fewer than 100 steps). Can the authors address where the performance uplift comes from? Does it all happen during the initial 100 steps?
- I believe the PPO and GRPO comparison is quite misleading. They both start from a student with GSM8k 1.14% and HumanEval %0 and a binary reward. Reward essentially never fires, so a near-zero results are expected. A fairer comparison would be to add a short SFT/KD/OPD warm-up before PPO or GRPO, to improve the odds of getting non-zero reward groups.
- Given that the controller and detector have a large number of hyperparameters, can the authors illustrate the selection criteria or protocol used to determine which values to use? The paper never discusses how they were chosen.
- Currently, all observations and results are derived from Qwen-based models. Can the authors discuss more on the generalizability of the observations and results?
Questions for authors
see weaknesses.
4 comments
Response to Reviewer XpkW (2/2)
Weakness 3: RL comparison
We agree: from a collapsed initialization, outcome rewards almost never fire, which is exactly the point of RQ2 (Section 3.3). To rule out the effect of initialization, we followed your suggestion and first ran a 462-step SFT cold start on the dataset of Appendix D.4, followed by 462 steps of PPO/GRPO:
| Method (Qwen3-4B-Instruct, Math+Code) | Avg |
|---|---|
| Pruned (reference) | 0.69 |
| PPO, from the pruned model | 0.00 |
| GRPO, from the pruned model | 0.43 |
| SFT cold start, then PPO | 24.97 |
| SFT cold start, then GRPO | 26.83 |
| ShortOPD | 38.03 |
After the cold start, PPO and GRPO reach 24.97 and 26.83. Despite using twice as many training steps, they remain more than 11 points below ShortOPD.
Weakness 4: How were the hyperparameters chosen?
We set these constants directly on Qwen3-4B-Instruct according to their meaning, without dedicated tuning. All subsequent experiments reuse the same values: the other three Qwen3 settings, width/hybrid pruning, and Gemma. The main constants mean the following:
- The budget shrinks when close to half of the rollouts in a batch loop (ρ_high = 0.45).
- The budget grows when looping is rare (ρ_low = 0.20) and more than 10% of rollouts are clean truncations (τ = 0.10).
- λ = 1.15 leaves a 15% margin beyond the usable length.
We varied the thresholds and smoothing parameters one at a time:
| Hyperparameter | Value | Avg |
|---|---|---|
| ShortOPD default (Table 9) | – | 61.89 |
| ρ_high (shrink threshold) | 0.30 | 61.13 |
| ρ_high (shrink threshold) | 0.60 | 62.09 |
| τ (growth threshold) | 0.05 | 62.37 |
| τ (growth threshold) | 0.20 | 59.49 |
| β = β_H (EMA smoothing) | 0.5 | 58.89 |
| β = β_H (EMA smoothing) | 0.9 | 61.81 |
| Vanilla OPD (reference) | – | 49.39 |
All variants outperform Vanilla OPD by at least 9.50 points.
Weakness 5: Beyond Qwen
We added experiments on another model family, Gemma-4-E2B-it (9 of 35 layers removed):
| Model (depth pruning) | Pruned Avg | Vanilla OPD Avg | ShortOPD Avg |
|---|---|---|---|
| Gemma-4-E2B-it · 9/35 layers | 5.59 | 39.81 | 45.97 |
Pruning drops Gemma's Avg from 61.87 to 5.59. After recovery training, ShortOPD restores it to 45.97 (74% of the unpruned model), 6.16 points above Vanilla OPD. Neither the observation that capability remains recoverable after pruning nor ShortOPD's gains are therefore limited to Qwen3.
I thank the authors for running the experiments. The new results are informative. However, from the new ablation study, most of the gain from the current approach is simply from filtering or limiting looped tokens, which already reached 59.23. Compared to ShortOPD 61.89, this is already within noise variance. From the paper's own argument that loops dilute gradients, I believe masking is the more direct approach given your method for loop detection. I argue that the adaptive rollout budget here is not necessary.
As for my question 2, the table only shows that ShortOPD gives the model a head start. But subsequent steps improve at the same rate at around ~20%, which is not evidence for vanilla OPD not catching up. Longer training runs would perhaps have both runs saturate at the same accuracy.
Given that the gain is largely from masking looped tokens, I find insufficient evidence that ShortOPD, or its adaptive rollout budget, adds much benefit to overall training. As submitted, the claimed contribution does not match where the gains come from, so I am keeping my score.
Thank you for the follow-up. We respond to each point below.
1. Masking vs. ShortOPD. We agree that most of the gain over Vanilla OPD comes from removing looped tokens. The 2.66-point gap between ShortOPD and masking (61.89 vs. 59.23), however, is not within noise. Results across seeds:
| Setting / Method | Seed 42 (Table 3) | Seed 24 | Seed 64 | Mean ± std |
|---|---|---|---|---|
| 4B-Instruct · Vanilla OPD | 49.39 | 49.11 | 49.14 | 49.21 ± 0.15 |
| 4B-Instruct · ShortOPD | 61.89 | 62.21 | 61.77 | 61.96 ± 0.23 |
| 8B no-thinking · Vanilla OPD | 52.04 | 51.09 | 51.74 | 51.62 ± 0.49 |
| 8B no-thinking · ShortOPD | 63.40 | 64.21 | 63.54 | 63.72 ± 0.43 |
2. Why not simply mask. Masking keeps a fixed budget, so a looping rollout still runs to the cap: its looped tokens are generated and only then discarded, after the generation compute has been spent. ShortOPD uses the same loop-detection signal to shrink the budget before generation, so these tokens are not generated. Under an 8192-token ceiling, ShortOPD uses 14.7–30.2% fewer rollout tokens and 15.3–24.8% less training time than Vanilla OPD (Table 5); under the 2048 ceiling, generation time over the first 100 steps falls from 35.4 to 25.1 seconds per step (Fig. 3). ShortOPD lowers generation cost while also scoring higher than masking.
3. Would longer training close the gap? In the intermediate-checkpoint table of our previous response, from step 400 to step 710 ShortOPD improves by 21.18 points (40.7→ 61.89) and Vanilla OPD by 20.42 points (28.97 → 49.39). The later improwith ShortOPD slightly faster, so the gap does not narrow (11.74 →12.50). Step 710 is the full training budget shared by all methods; under this budget, ShortOPD scores higher at a lower cost per step.
4. Contribution and the source of the gains. As stated in the first contribution of our introduction, the paper proposes a recovery recipe for generation quality after structured pruning, built on three ingredients: supervision on the studendense token-level teacher distributions, and a rollout budget matched towhat the damaged policy can currently generate. The first two adapt OPD to structured-pruning recovery; Vanilla OPD alone already exceeds SFT, SeqKD, and KD by 10.0–20.2 points across four settings (Table 3). The third comes from our finding tode of pruned models under OPD is repetitive suffixes, to which theteacher also assigns high probability. Both loop masking and ShortOPD build on this finding and on our loop detector, and ShortOPD is the cheaper and more effective of the two. As our abstract states, we hope this recipe helps move structured pruning beyond perplexity and multiple-choice benchmarks, toward deployment-ready generation quality.
Response to Reviewer XpkW (1/2)
Thank you for your encouragement and the concrete suggestions for controls. We ran all three.
Weakness 1: Missing controls
We ran all three controls with the same setup as Table 3:
These three controls show that the main source of the gain is removing low-signal loop tokens, but a hand-written two-phase schedule cannot replace online feedback. We discuss each in turn:
In addition, directly replaying ShortOPD's full budget curve gives 58.68, still 3.21 points below ShortOPD (see the General Response). This curve is itself a product of the controller and cannot be known in advance. Online feedback therefore contributes beyond the short-to-long shape. Moreover, ShortOPD uses the same constants in all settings, so unlike a hand-written schedule, it does not require designing a switch time for each setting.
Weakness 2: Where does the gain come from?
Intermediate checkpoints of the two runs on Qwen3-4B-Instruct:
At step 100, the gap is already 10.71 points, about 86% of the final gap, so the gap does form mainly early on. However, once the budget returns to 2048, the two runs use exactly the same update rule, yet Vanilla OPD never catches up over the remaining 600+ steps. The gap at step 710 (12.50) is even larger than at step 100. This shows that differences formed early persist through the training data that each student generates itself. ShortOPD acts mainly early in training, but its benefit carries through the entire run.