- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates which type of teacher training is more effective for on-policy distillation (OPD). Through experiments across three domains, agentic, reasoning, and perception, the authors demonstrate that distillation from RL-trained teachers consistently outperforms distillation from SFT-trained teachers. To explain this phenomenon, the authors analyze parameter displacement and show that RL-trained teachers remain closer to the initial model, potentially making them easier for students to learn from. Furthermore, the authors report higher top-K alignment for RL-trained teachers, providing additional evidence of better compatibility between the teacher and student.
Critical strengths
-
Interesting Research Question. Although OPD has been introduced to address limitations associated with off-policy KD, the question of which teacher is more suitable for OPD has received relatively limited attention. Investigating teacher selection from the perspective of training methodology is an interesting and meaningful research direction.
-
Clarity and Readability. The paper is well-organized and easy to follow. The motivation, research questions, experimental findings, and main conclusions are clearly presented, making the overall message straightforward to understand.
Critical weaknesses
-
Limited Analysis of the Underlying Mechanism. The paper primarily compares RL-trained and SFT-trained teachers and attributes the superior effectiveness of RL-trained teachers to their smaller parameter displacement from the initialization. However, it remains unclear whether this advantage stems from the RL optimization algorithm itself or from the regularization used during RL training. For instance, GRPO can incorporate a KL penalty that discourages deviation from the reference policy, potentially contributing to the smaller parameter displacement. A more detailed analysis distinguishing these factors would strengthen the paper's central claim. In particular, it would be informative to investigate whether RL-trained teachers still outperform SFT-trained teachers when the latter are trained with a comparable KL regularization term. Such a comparison could clarify whether the observed advantage is specific to RL training or is primarily associated with maintaining proximity to the initial model.
-
Limited Baselines and Generalizability. Given the paper's hypothesis that smaller parameter displacement facilitates OPD, an additional baseline using the original reference (base) model as the teacher would provide a useful comparison. Comparing OPD with the base model, SFT-trained teacher, and RL-trained teacher could help clarify the relationship between parameter displacement and distillation performance. Furthermore, the experiments are limited in terms of model sizes, teacher configurations, and architectures. Evaluations across a broader range of settings would strengthen the generalizability of the findings.
Questions for authors
Note: I recognize that most of the concerns raised from weaknesses part would require additional training experiments, which may be difficult to complete within the rebuttal period. Therefore, I would appreciate any existing experimental results, small-scale or toy experiments, or logically grounded explanations based on the current findings that could help address these concerns.
-
Practical Implications. What are the practical implications of the findings? As already discussed in the Qwen3 technical report, RL-trained models are used as teachers for OPD in existing training pipelines. Given that this practice has already been adopted, it would be helpful to clarify what additional insights or practical guidance the comparison between SFT-trained and RL-trained teachers provides beyond validating the existing practice.
-
Student Parameter Displacement and Performance. In Figure 4(b), the student's parameter displacement continues to increase as training progresses. Would the student's performance eventually approach that of the SFT-trained teacher if its parameter displacement became comparable to the teacher's? Although the paper indicates that increasing training steps does not necessarily lead to linear performance improvements, I am curious whether the student's performance follows a trend similar to that of the SFT-trained teacher across training stages. Any existing evidence connecting parameter displacement, training progress, and student performance would help clarify this relationship.
Minor Comments
-
Line 100: Consider replacing "Team et al." with "Xiaomi LLM-Core Team" for a more appropriate author attribution.
-
Line 115: Consider using lowercase for the domain names ("agentic," "reasoning," and "perception") unless they are intended as formal category names.
-
Figure 4: The reference to Figure 4 appears to be formatted as a citation rather than a standard figure cross-reference. Please check the formatting.
8 comments
Teacher displacement and OPD performance within the same training algorithm
2. Teacher displacement and OPD performance within the same training algorithm
To further examine the relationship between teacher parameter displacement and subsequent OPD performance, we trained two reasoning SFT teachers on the same dataset using different batch sizes. This yields teachers with similar task accuracy but different parameter displacements. We then trained their students using the same OPD configuration.
| SFT teacher | Teacher accuracy (%) | Teacher parameter displacement (L2) | Best student accuracy (%) | Teacher gain recovery (%) |
|---|---|---|---|---|
| Batch size 32 | 88.35 | 3.619 | 88.30 | 99.36 |
| Batch size 128 | 87.91 | 2.590 | 89.24 | 116.22 |
Although the teacher with smaller displacement has slightly lower task accuracy, it produces a student with both higher accuracy and higher teacher gain recovery. This comparison holds the teacher-training algorithm and training data fixed, providing further evidence for the relationship between smaller teacher displacement and better OPD outcomes. These results have been added to Section 6.1.
Additional teacher-training paths and the base-model teacher
3. Additional teacher-training paths and the base-model teacher
We appreciate the suggestion to examine additional teacher configurations. We added the commonly used SFT→RL training path: starting from Qwen3.5-9B, we first apply SFT to reach a MobileWorld success rate of 21.65%, then continue with RL to reach 23.36%, closely matching the task performance of the original teachers. All OPD students start from the same original checkpoint.
| Teacher-training path | Teacher success rate (%) | Teacher parameter displacement (L2) | Best student success rate (%) | Teacher gain recovery (%) |
|---|---|---|---|---|
| SFT | 22.79 | 2.271 | 19.94 | 44.44 |
| SFT→RL | 23.36 | 1.765 | 21.37 | 65.00 |
| RL | 23.36 | 0.528 | 24.22 | 115.00 |
The three training paths—SFT, SFT followed by RL, and direct RL—produce teachers with similar task performance but different parameter displacements. Their OPD students also differ in accuracy and teacher gain recovery, with the direct-RL group performing best, followed by SFT→RL and then SFT. These results support our finding that RL teachers are more effective for subsequent OPD. We have added this comparison to Section 6.2.
We believe that teacher performance also affects OPD outcomes. Although the base model has zero parameter displacement, it offers no performance gain over the initial student, making it difficult for OPD to improve the student. Prior work likewise emphasizes that, for OPD to yield gains, the teacher needs to provide new capabilities beyond those already available to the student [1].
[1] Yaxuan Li et al. Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. arXiv:2604.13016, 2026.
The role of KL regularization in the RL teacher advantage
We thank the reviewer for the thoughtful and constructive comments, and for welcoming additional experiments and explanations based on our current findings. We have further examined KL regularization, teacher parameter displacement, teacher-training paths, and another model backbone. We have also extended the agentic SFT-guided student's training to investigate the relationship between training progress, parameter displacement, and performance. We address each concern below.
1. The role of KL regularization in the RL teacher advantage
Thank you for highlighting the possible influence of KL regularization on teacher displacement. We investigate this factor by removing the KL penalty during RL teacher training.
First, in our original agentic experiments, the RL teacher was trained without a KL penalty, yet its parameter displacement remained substantially smaller than that of the SFT teacher. Its OPD student also achieved a higher success rate, at 24.22% versus 19.94%, and higher teacher gain recovery, at 115.00% versus 44.44%.
We additionally trained a reasoning RL teacher without the KL penalty, keeping the remaining teacher-training configuration unchanged. The results are shown below; reasoning accuracies are averaged over three evaluation runs.
After removing the KL penalty, the RL-guided student still outperforms the SFT-guided student. Together with the agentic results, this supports our interpretation that explicit KL regularization is not the primary source of the RL teacher advantage.