- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies output-length inflation after on-policy distillation and proposes an intermediate compression stage. Its reward favors shorter correct responses, with stronger length feedback at higher sampled success rates. The compressed model then undergoes either prompted self-checking or renewed distillation. Experiments report substantial compression with modest accuracy gains, and show that renewed distillation expands responses again while improving solved-problem coverage.
Critical strengths
-
The paper addresses a relevant efficiency problem. Its experiments show consistent compression across three distillation routes using both DPO and the proposed RL objective, while preserving or slightly improving accuracy.
-
The renewed-distillation experiment provides an informative observation: even after the student has learned shorter responses, restoring the original distillation procedure expands outputs toward their previous lengths while preserving or improving accuracy. This suggests that the original distillation setup continues to drive length growth despite a shorter-response initialization.
Critical weaknesses
-
The advance over closely related work remains limited. Relative-length rewards for correct responses [1], success-conditioned length penalties [2], and compression–expansion training [3] already exist. The OPD-specific observations are useful, but the proposed pipeline largely adapts these ideas to post-distillation training. Its additional practical benefit over existing compression objectives is also unclear: Table 2 compares against DPO and correctness-only GRPO, without a closely related correctness-aware length-control baseline.
-
Dependence on the model configuration remains unclear, and essential experimental details are missing. All student models belong to the Qwen3 family. The complete accuracy–length evidence is concentrated on Qwen3-1.7B, while Qwen3-4B results report only average@8 (Appendix D, Table 7). Section 4.1 also mentions differing teacher setups without identifying the teacher checkpoints or OPSD teacher conditioning. Consequently, it is difficult to assess whether the observed length inflation, rebound, and compression benefits depend on particular teacher–student configurations. Evaluation sampling parameters, answer grading, and termination handling also require clarification for reproducibility.
[1] Daman Arora and Andrea Zanette. Training Language Models to Reason Efficiently. NeurIPS 2025.
[2] Violet Xiang et al. Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learning. arXiv:2506.05256, 2025.
[3] Haoming Wen, Yushi Bai, Juanzi Li, and Jie Tang. SIRI: Scaling Iterative Reinforcement Learning with Interleaved Compression. arXiv:2509.25176, 2025.
Questions for authors
- Which teacher checkpoints, teacher-conditioning setups, and termination settings are used in each route? Which settings remain unchanged when distillation resumes?
- How does the proposed reward compare with existing correctness-aware compression objectives when applied to the same distilled checkpoints with comparable training budgets?