- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies on-policy distillation (OPD) as initialization for outcome-based RL. Using Qwen3 and Qwen3-VL student–teacher pairs, it compares direct RL, SFT followed by RL, and OPD followed by RL. OPD initialization achieves the highest final performance, even when its immediate accuracy improvement is small. The paper further shows that initial Pass@k does not fully explain downstream performance, and that forward-KL OPD can outperform reverse-KL OPD after RL despite weaker standalone performance. Behavioral analyses suggest teacher distributional alignment and retained uncertainty as possible explanations.
Critical strengths
The distinction between immediate performance and suitability for subsequent RL is interesting and practically relevant. In the text experiments, OPD improves the initial average by only 0.54 points over the base student, but its advantage grows to 7.28 points after RL. The nearly overlapping initial Pass@k curves make this observation particularly compelling. This provides useful evidence that initialization should be evaluated through the learning it enables, rather than only its current benchmark accuracy.
The comparison of distillation objectives and trajectory sources provides actionable insight. With student-generated trajectories, reverse KL performs better before RL, whereas forward KL finishes 1.85 points higher afterward. The comparison with teacher-generated trajectories further helps distinguish the effects of the supervision source and objective. These findings make the paper informative beyond simply demonstrating that an additional distillation stage can help.
Critical weaknesses
The connection to large-scale post-training pipelines remains underdeveloped. The experiments use 8B students and 32B teachers within the Qwen family. It is unclear how the proposed preparation stage should fit into MOPD-style pipelines that first train domain-specific RL teachers and then consolidate their capabilities through multi-teacher distillation. The paper also explicitly excludes preparation cost from its comparison. Consequently, the results establish a benefit under the evaluated downstream RL settings, but provide limited guidance on when adding this stage is preferable to spending the same resources on further RL or distillation. This limits the practical scope of the conclusions.
The downstream training budget and checkpoint selection need clearer reporting. Appendix A.3 states that RL stops when instability is detected, but does not clearly report the actual training lengths and checkpoint-selection rule for all comparisons. Shared hyperparameters do not necessarily imply matched training budgets. Learning curves at comparable update or token counts would help distinguish differences in learning progress from differences in training duration or stability. This is particularly relevant to the smaller VL improvements.
Questions for authors
In a pipeline with domain-specific RL teachers followed by MOPD, where would you place the proposed OPD preparation stage? Would it initialize the domain experts before RL, or prepare the consolidated student for additional RL? Please clarify which conclusions from the current experiments support that use case.
What are the actual RL update counts and checkpoint-selection criteria for each compared pipeline? Do the main rankings persist when comparing checkpoints at matched update or generated-token counts?
What is the additional cost of OPD relative to downstream RL, including teacher inference? Do you have measurements or existing results that help assess whether the gains remain worthwhile under a fixed total training budget?