acceptodds
Under review as a conference paper at ICLR 2027

Compress to Improve: Reclaiming Output Budget from On-Policy Distillation for Stronger Reasoning

Abstract

On-policy distillation (OPD) teaches a student model to reason through teacher feedback on its own generated responses. We find that this process also produces substantial length inflation: across six OPD-family methods evaluated from a common base model on competition mathematics, responses are consistently longer than those produced by supervised fine-tuning (SFT) and reinforcement learning (RL). Under a finite output budget, longer initial solutions leave less room for checking, revision, and further exploration. We introduce concise post-training as an intermediate stage that favors shorter correct solutions and frees output space for further reasoning. Across OPD, ExOPD, and OPSD, it reduces mean output length by 29.6–36.6% while improving both sample accuracy and problem coverage. We then reuse the released budget through prompted continuation, which asks the compressed model to check and revise its answers with fixed parameters, or renewed OPD, which resumes teacher-guided learning. Continuation improves sample accuracy in every route, while renewed distillation increases problem coverage. The strongest final endpoint, ExOPD followed by compression and continuation, raises average@8 from 25.2% to 27.3% and pass@8 from 36.7% to 40.8% relative to the original ExOPD checkpoint. Both complete paths outperform their original distilled models without exceeding their mean output lengths, supporting compression as an intermediate stage for further reasoning and learning gains.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, concise post-training, mathematical reasoning, test-time scaling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZovFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies output-length inflation after on-policy distillation and proposes an intermediate compression stage. Its reward favors shorter correct responses, with stronger length feedback at higher sampled success rates. The compressed model then undergoes either prompted self-checking or renewed distillation. Experiments report substantial compression with modest accuracy gains, and show that renewed distillation expands responses again while improving solved-problem coverage.

Critical strengths
  1. The paper addresses a relevant efficiency problem. Its experiments show consistent compression across three distillation routes using both DPO and the proposed RL objective, while preserving or slightly improving accuracy.

  2. The renewed-distillation experiment provides an informative observation: even after the student has learned shorter responses, restoring the original distillation procedure expands outputs toward their previous lengths while preserving or improving accuracy. This suggests that the original distillation setup continues to drive length growth despite a shorter-response initialization.

Critical weaknesses
  1. The advance over closely related work remains limited. Relative-length rewards for correct responses [1], success-conditioned length penalties [2], and compression–expansion training [3] already exist. The OPD-specific observations are useful, but the proposed pipeline largely adapts these ideas to post-distillation training. Its additional practical benefit over existing compression objectives is also unclear: Table 2 compares against DPO and correctness-only GRPO, without a closely related correctness-aware length-control baseline.

  2. Dependence on the model configuration remains unclear, and essential experimental details are missing. All student models belong to the Qwen3 family. The complete accuracy–length evidence is concentrated on Qwen3-1.7B, while Qwen3-4B results report only average@8 (Appendix D, Table 7). Section 4.1 also mentions differing teacher setups without identifying the teacher checkpoints or OPSD teacher conditioning. Consequently, it is difficult to assess whether the observed length inflation, rebound, and compression benefits depend on particular teacher–student configurations. Evaluation sampling parameters, answer grading, and termination handling also require clarification for reproducibility.

[1] Daman Arora and Andrea Zanette. Training Language Models to Reason Efficiently. NeurIPS 2025.

[2] Violet Xiang et al. Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learning. arXiv:2506.05256, 2025.

[3] Haoming Wen, Yushi Bai, Juanzi Li, and Jie Tang. SIRI: Scaling Iterative Reinforcement Learning with Interleaved Compression. arXiv:2509.25176, 2025.

Questions for authors
  1. Which teacher checkpoints, teacher-conditioning setups, and termination settings are used in each route? Which settings remain unchanged when distillation resumes?
  2. How does the proposed reward compare with existing correctness-aware compression objectives when applied to the same distilled checkpoints with comparable training budgets?
Reviewer_rnkdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies output-length inflation in on-policy distillation and proposes a compression stage before further reasoning or learning. Its main method uses GRPO with a correctness-aware length reward that favors shorter correct responses, with stronger length feedback on problems solved more reliably. The compressed model then either checks its answers through prompted continuation or undergoes renewed distillation. On competition mathematics, compression reduces mean output length by 29.6–36.6% across three distillation routes while improving accuracy. The final pipelines outperform the original distilled checkpoints at slightly lower mean output lengths.

Critical strengths
  1. Substantial compression without an observed accuracy loss. Across OPD, ExOPD, and OPSD, the proposed stage reduces mean output length by roughly one third while improving both average@8 and pass@8. This consistent improvement is the paper’s strongest result and has clear practical value.
  2. Useful comparisons of compression objectives and training order. The DPO, correctness-only GRPO, and length-aware GRPO comparisons help distinguish different accuracy–length trade-offs. The ordering experiments also show that applying distillation after compression largely reverses the length reduction, providing useful guidance for composing post-training stages.
Critical weaknesses
  1. The main controls do not isolate the benefit of compression for subsequent reasoning or learning. The continuation experiments lack a direct comparison with continuation from the original distilled checkpoints under matched output budgets. Similarly, renewed distillation is not compared with continuing the original checkpoint for a matched additional training budget. Saturation in training accuracy does not establish that further training cannot improve held-out performance. These controls are central to determining whether compression enables the reported gains or whether similar gains follow from additional inference or training alone.

  2. The accuracy gains are small relative to the evaluation size, and their reliability is unclear. The test suite contains only 120 problems. For compressed ExOPD, continuation improves pass@8 by two problems, while renewed distillation improves it by one. No uncertainty estimates or repeated-run results are reported in the provided text. Likewise, RL2’s pass@8 advantage over DPO corresponds to only one or two problems per route. The large length reductions are convincing, but the evidence for improved coverage and the superiority of the proposed reward is less secure.

Questions for authors
  1. What happens when the original distilled checkpoints receive the same continuation intervention under matched total output budgets, or undergo further distillation with matched additional training compute? These are requests for controls supporting the central claim, rather than suggestions for future work.
  2. Are the accuracy improvements consistent across training and evaluation seeds? Please provide paired, problem-level uncertainty estimates, particularly for the incremental gains after compression and the comparison between RL2 and DPO.
  3. Is the 16K generation limit shared between the initial response and continuation, or applied separately? How frequently do the original and compressed models reach the limit? This would clarify whether the gains reflect reduced truncation or other changes in reasoning behavior.
Reviewer_84vwOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper addresses the problem of length inflation related to on-policy distillation (OPD). In OPD, a student generates its own responses, and the teacher simply gives feedback on these responses (rather than directly generating reference solutions). OPD can cause the student to generate longer and longer answers (which is called length inflation). To improve OPD, the paper proposes a compression step after OPD that results in a policy that tends to generate shorter outputs. This frees more space to do more reasoning, so after compression, the proposed pipeline includes another step of prompted continuation (similar to the popular s1 paper, where the goal is to output more useful tokens) or continual learning with another OPD stage.

Critical strengths
  1. It seems promising to think of reasoning as a budget that needs to be allocated properly, so I quite like the post-compression steps the paper proposes.

  2. The compression step reduces output length while slightly improving the accuracy (or at least not hurting it, see point 4 in the weaknesses section). If this trend is consistent, then that is a nice result.

Critical weaknesses

The paper proposes a pipeline consisting of 3 stages (Fig. 1): (1) OPD, (2) compression (to correct for OPD’s length inflation), and (3) reuse (prompted continuation or renewed OPD). However, it is not clear how or if any of the paper’s contributions really extends the current work on OPD.

  1. The abstract claims to find a length inflation issue associated with OPD ("We find that this process also produces substantial length inflation"). But this length inflation was already observed in [1], which the paper itself cites in Section 3 for this exact finding. So I would suggest rephrasing that line of the abstract. The larger issue, however, is that [1] comprehensively alleviates the issue through several training stabilization techniques, which this submission does not discuss. So it is not clear how the proposed solution is better without empirical comparisons.

  2. Even if we don’t consider [1], the compression step is not fully justified. This is a bigger gap in the paper's comparisons in my opinion. Controlling the reasoning budget has been studied extensively, for example in [2] and [3], which are cited in the paper. My understanding is that the proposed RL2 reward mainly modifies the length term in [2]. But the paper does not compare its compression step to either [2] or [3]. It only compares to correctness-only GRPO and DPO, neither of which is an established length-control method. So it is not clear what the benefit of this modification is for compression performance.

  3. The final step largely depends on strategies proposed in s1 [4]. That is fine as the paper properly cites s1. But, similar to the previous point, it is not clear to me what that part of the method adds to the existing literature. Also, as far as I could tell, the paper never applies continuation to the original distilled checkpoints, so it is unclear whether the compression step is even needed for the continuation gains.

  4. On the empirical side, nearly all experiments use Qwen3-1.7B only. The test set is small with 120 problems from AIME and HMMT (so a method outperforming another by 2 points simply means it answered two more questions correctly). Evaluation on benchmarks this small is really noisy, and [5] showed that scores can move by several points just from decoding settings like temperature or changing the seed. Most gains here are between 0.5 and 2 points on average@8, so I don’t think they can establish a clear improvement. I would follow [5]'s suggestion of using many evaluation seeds on small reasoning benchmarks and reporting mean and std.

[1]: Luo et al., Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models, arXiv April 2026

[2]: Arora & Zanette, Training Language Models to Reason Efficiently, NeurIPS 2025

[3]: Aggarwal & Welleck, L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning, COLM 2025

[4]: Muennighoff et al., s1: Simple test-time scaling, EMNLP 2025

[5]: Hochlehnert et al., A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility, COLM 2025

Questions for authors
  1. How does the compression step compare against the stabilization strategies in [1] or the length control methods in [2] and [3]? Are they comparable to compression in this empirical setup? I am open to reconsidering the first two weaknesses based on the authors' clarification.

  2. How sensitive to random sampling are the reported results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.