acceptodds
Under review as a conference paper at ICLR 2027

Train4Merge: A Controlled Single-Teacher Study of RL vs. SFT Teachers for OPD-Based Model Merging

Abstract

Domain experts trained from a shared checkpoint can be merged into one model through on-policy distillation (OPD), where they act as teachers supervising a student on its own trajectories. One upstream choice is rarely examined: whether to build each expert with supervised fine-tuning (SFT) or reinforcement learning (RL). Yet equally strong teachers need not be equally good teachers. We probe this choice through controlled single-teacher OPD, a building block of multi-teacher OPD: in Agentic, Reasoning, and Perception, comparably performing SFT and RL teachers are trained from Qwen3.5-9B, each guiding a student initialized from it. At their best checkpoints, RL-guided students outperform SFT-guided students by 4.27, 1.50, and 0.86 percentage points in Agentic, Reasoning, and Perception, respectively, and recover more of their teachers' performance gains over the base model. The contrast is clearest in Agentic, where the best SFT-guided student recovers only 44.44% of its teacher's gain, whereas the best RL-guided student recovers 115.00%, surpassing its teacher. Our analysis points to an explanation: RL teachers stay much closer to the shared initialization in parameter space than SFT teachers and are therefore easier for their students to follow.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On Policy Distillation, Vision Language Model, Model Merging

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_UiR6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates which type of teacher training is more effective for on-policy distillation (OPD). Through experiments across three domains, agentic, reasoning, and perception, the authors demonstrate that distillation from RL-trained teachers consistently outperforms distillation from SFT-trained teachers. To explain this phenomenon, the authors analyze parameter displacement and show that RL-trained teachers remain closer to the initial model, potentially making them easier for students to learn from. Furthermore, the authors report higher top-K alignment for RL-trained teachers, providing additional evidence of better compatibility between the teacher and student.

Critical strengths
  • Interesting Research Question. Although OPD has been introduced to address limitations associated with off-policy KD, the question of which teacher is more suitable for OPD has received relatively limited attention. Investigating teacher selection from the perspective of training methodology is an interesting and meaningful research direction.

  • Clarity and Readability. The paper is well-organized and easy to follow. The motivation, research questions, experimental findings, and main conclusions are clearly presented, making the overall message straightforward to understand.

Critical weaknesses
  • Limited Analysis of the Underlying Mechanism. The paper primarily compares RL-trained and SFT-trained teachers and attributes the superior effectiveness of RL-trained teachers to their smaller parameter displacement from the initialization. However, it remains unclear whether this advantage stems from the RL optimization algorithm itself or from the regularization used during RL training. For instance, GRPO can incorporate a KL penalty that discourages deviation from the reference policy, potentially contributing to the smaller parameter displacement. A more detailed analysis distinguishing these factors would strengthen the paper's central claim. In particular, it would be informative to investigate whether RL-trained teachers still outperform SFT-trained teachers when the latter are trained with a comparable KL regularization term. Such a comparison could clarify whether the observed advantage is specific to RL training or is primarily associated with maintaining proximity to the initial model.

  • Limited Baselines and Generalizability. Given the paper's hypothesis that smaller parameter displacement facilitates OPD, an additional baseline using the original reference (base) model as the teacher would provide a useful comparison. Comparing OPD with the base model, SFT-trained teacher, and RL-trained teacher could help clarify the relationship between parameter displacement and distillation performance. Furthermore, the experiments are limited in terms of model sizes, teacher configurations, and architectures. Evaluations across a broader range of settings would strengthen the generalizability of the findings.

Questions for authors

Note: I recognize that most of the concerns raised from weaknesses part would require additional training experiments, which may be difficult to complete within the rebuttal period. Therefore, I would appreciate any existing experimental results, small-scale or toy experiments, or logically grounded explanations based on the current findings that could help address these concerns.

  • Practical Implications. What are the practical implications of the findings? As already discussed in the Qwen3 technical report, RL-trained models are used as teachers for OPD in existing training pipelines. Given that this practice has already been adopted, it would be helpful to clarify what additional insights or practical guidance the comparison between SFT-trained and RL-trained teachers provides beyond validating the existing practice.

  • Student Parameter Displacement and Performance. In Figure 4(b), the student's parameter displacement continues to increase as training progresses. Would the student's performance eventually approach that of the SFT-trained teacher if its parameter displacement became comparable to the teacher's? Although the paper indicates that increasing training steps does not necessarily lead to linear performance improvements, I am curious whether the student's performance follows a trend similar to that of the SFT-trained teacher across training stages. Any existing evidence connecting parameter displacement, training progress, and student performance would help clarify this relationship.

Minor Comments

  • Line 100: Consider replacing "Team et al." with "Xiaomi LLM-Core Team" for a more appropriate author attribution.

  • Line 115: Consider using lowercase for the domain names ("agentic," "reasoning," and "perception") unless they are intended as formal category names.

  • Figure 4: The reference to Figure 4 appears to be formatted as a citation rather than a standard figure cross-reference. Please check the formatting.

8 comments
AuthorsOpenReview ↗

The role of KL regularization in the RL teacher advantage

We thank the reviewer for the thoughtful and constructive comments, and for welcoming additional experiments and explanations based on our current findings. We have further examined KL regularization, teacher parameter displacement, teacher-training paths, and another model backbone. We have also extended the agentic SFT-guided student's training to investigate the relationship between training progress, parameter displacement, and performance. We address each concern below.

1. The role of KL regularization in the RL teacher advantage

Thank you for highlighting the possible influence of KL regularization on teacher displacement. We investigate this factor by removing the KL penalty during RL teacher training.

First, in our original agentic experiments, the RL teacher was trained without a KL penalty, yet its parameter displacement remained substantially smaller than that of the SFT teacher. Its OPD student also achieved a higher success rate, at 24.22% versus 19.94%, and higher teacher gain recovery, at 115.00% versus 44.44%.

We additionally trained a reasoning RL teacher without the KL penalty, keeping the remaining teacher-training configuration unchanged. The results are shown below; reasoning accuracies are averaged over three evaluation runs.

Teacher typeTeacher accuracy (%)Best student accuracy (%)
SFT88.3588.30
RL, with KL88.0791.29
RL, without KL88.5791.68

After removing the KL penalty, the RL-guided student still outperforms the SFT-guided student. Together with the agentic results, this supports our interpretation that explicit KL regularization is not the primary source of the RL teacher advantage.

AuthorsOpenReview ↗

Teacher displacement and OPD performance within the same training algorithm

2. Teacher displacement and OPD performance within the same training algorithm

To further examine the relationship between teacher parameter displacement and subsequent OPD performance, we trained two reasoning SFT teachers on the same dataset using different batch sizes. This yields teachers with similar task accuracy but different parameter displacements. We then trained their students using the same OPD configuration.

SFT teacherTeacher accuracy (%)Teacher parameter displacement (L2)Best student accuracy (%)Teacher gain recovery (%)
Batch size 3288.353.61988.3099.36
Batch size 12887.912.59089.24116.22

Although the teacher with smaller displacement has slightly lower task accuracy, it produces a student with both higher accuracy and higher teacher gain recovery. This comparison holds the teacher-training algorithm and training data fixed, providing further evidence for the relationship between smaller teacher displacement and better OPD outcomes. These results have been added to Section 6.1.

AuthorsOpenReview ↗

Additional teacher-training paths and the base-model teacher

3. Additional teacher-training paths and the base-model teacher

We appreciate the suggestion to examine additional teacher configurations. We added the commonly used SFT→RL training path: starting from Qwen3.5-9B, we first apply SFT to reach a MobileWorld success rate of 21.65%, then continue with RL to reach 23.36%, closely matching the task performance of the original teachers. All OPD students start from the same original checkpoint.

Teacher-training pathTeacher success rate (%)Teacher parameter displacement (L2)Best student success rate (%)Teacher gain recovery (%)
SFT22.792.27119.9444.44
SFT→RL23.361.76521.3765.00
RL23.360.52824.22115.00

The three training paths—SFT, SFT followed by RL, and direct RL—produce teachers with similar task performance but different parameter displacements. Their OPD students also differ in accuracy and teacher gain recovery, with the direct-RL group performing best, followed by SFT→RL and then SFT. These results support our finding that RL teachers are more effective for subsequent OPD. We have added this comparison to Section 6.2.

We believe that teacher performance also affects OPD outcomes. Although the base model has zero parameter displacement, it offers no performance gain over the initial student, making it difficult for OPD to improve the student. Prior work likewise emphasizes that, for OPD to yield gains, the teacher needs to provide new capabilities beyond those already available to the student [1].

[1] Yaxuan Li et al. Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. arXiv:2604.13016, 2026.

Reviewer_DxCfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares RL- and SFT-trained teachers for OPD when teachers and students share a base checkpoint. It selects approximately performance-matched teacher pairs in Agentic, Reasoning, and Perception, then distills each pair under a shared domain-specific OPD setup. The best RL-guided student performs better in all three domains. Parameter displacement and Top-20 overlap are used to motivate the hypothesis that teachers closer to the initialization are easier to distill.

Critical strengths
  1. The paper studies a concrete upstream decision in OPD. A teacher's standalone score is different from the capability it can transfer to a student. Examining teacher construction complements work focused on the distillation objective.

  2. The comparison controls model and task origins. Teachers and students start from Qwen3.5-9B, with shared within-domain task pools and OPD configurations. This is easier to interpret than comparing unrelated off-the-shelf teachers.

Critical weaknesses
  1. Single-teacher learnability does not directly determine multi-teacher merging choices. The introduction motivates constructing specialists for later merging, while the experiments isolate each teacher's supervision. A teacher that is easy to follow alone may produce updates that conflict with other teachers. Even nearby teachers can be mutually incompatible in shared parameters.

  2. Matching questions and task scores does not isolate the algorithm from supervision style. In Reasoning, the SFT teacher learns complete NVIDIA chain-of-thought targets, while RL learns from model-generated responses with answer-based rewards. This changes the objective together with the supervision source; effects of target style, length, and compatibility with the base model are not isolated. Easier transfer from the RL teacher could therefore partly reflect on-policy style compatibility rather than the RL algorithm category alone.

  3. Top-20 overlap mixes prediction alignment with changes in visited states. Section 5 measures overlap on each student's own rollouts. These are the relevant supervision positions for OPD, but the resulting prefixes differ between students as training proceeds. A student can improve overlap by visiting states more familiar to its teacher, without absorbing supervision more effectively on the same difficult states. Faster overlap growth therefore does not independently establish easier teacher learning.

  4. Parameter distance is descriptive, but its use as an optimization-difficulty proxy requires stronger justification. A nearby teacher parameter vector does not establish that useful OPD update directions are easier to obtain. OPD optimizes output distributions on student-visited prefixes and need not reproduce all teacher parameters. Different parameter directions can also have very different behavioral effects. The analysis mainly compares displacement magnitudes, without separating them from functional distance or update alignment. A control that changes displacement while preserving teacher capability as much as possible would provide stronger mechanistic evidence.

Questions for authors

Are actual advantage scales and effective updates comparable under the two teachers? Does the SFT-guided gap remain after modest changes to transfer budget or optimization settings?

14 comments
AuthorsOpenReview ↗

Updated reasoning teachers with consistent freezing settings and more closely matched performance

We thank the reviewer for the thoughtful and constructive feedback. We have strengthened the controlled comparisons, added MOPD experiments and controls for demonstration style and teacher displacement, and repeated OPD training using the same teachers and training data. We address the concerns below.

1. Updated reasoning teachers with consistent freezing settings and more closely matched performance

To strengthen the comparison, we retrained the reasoning SFT teacher with the visual encoder and merger frozen, aligning its freezing settings with the other teachers. We also replaced the reasoning RL teacher with a stronger model whose Geo3K accuracy more closely matches that of the SFT teacher. The corresponding teacher and student results have been updated throughout the revised manuscript, including the MOPD experiments.

Teacher and student accuracies below are averaged over three evaluation runs.

Teacher typeTeacher accuracy (%)Teacher parameter displacement (ΔL2)Best student accuracy (%)Teacher gain recovery (%)
SFT88.353.61988.3099.36
RL88.070.27491.29138.50

The teachers now differ by only 0.28 percentage points, while the RL-guided student achieves both higher accuracy and higher teacher gain recovery. The revised comparison therefore preserves the original finding under more closely matched teacher performance and consistent freezing settings.

AuthorsOpenReview ↗

Extending the comparison to MOPD

2. Extending the comparison to MOPD

We have added MOPD experiments using the same domain teachers as the single-teacher experiments. The three SFT teachers jointly supervise one student, while the three RL teachers jointly supervise another. Training examples are routed to the teacher corresponding to their domain, and both student groups use the same MOPD configuration.

The following table reports the domain teachers and each group’s student checkpoint with the highest three-domain mean score.

ModelAgentic (%)Reasoning (%)Perception (%)Mean (%)Overall teacher gain recovery (%)
Qwen3.5-9B17.6679.7065.1954.19N/A
SFT teachers22.7988.3568.2459.80N/A
RL teachers23.3688.0767.7159.71N/A
Best SFT-guided student19.9489.5767.3058.9484.68
Best RL-guided student23.6590.6369.6261.30128.61

Despite having a slightly lower mean teacher score, the RL teacher group produces a stronger student in all three domains, with higher overall accuracy and teacher gain recovery. Although interference between teachers may arise during MOPD, these results show that the RL teacher advantage persists when their capabilities are merged into a single student.

These experiments are included in Section 5.2, Table 3.

AuthorsOpenReview ↗

Controlling for the source and style of SFT demonstrations

3. Controlling for the source and style of SFT demonstrations

We appreciate the reviewer’s suggestion to distinguish the effects of teacher training from those of externally generated demonstrations. We therefore added a rejection fine-tuning (RFT) experiment using responses generated by the shared base model.

Specifically, we use the Geo3K training questions as prompts for Qwen3.5-9B, collect correct responses through rejection sampling, and retain at most one correct response per question. We train a teacher on these responses using the same SFT configuration as the main experiment, select its best-performing checkpoint, and train its student using the same reasoning OPD configuration.

Teacher typeTeacher accuracy (%)Best student accuracy (%)Teacher gain recovery (%)
Main SFT88.3588.3099.36
RFT88.9690.45116.03
RL88.0791.29138.50

RFT improves student accuracy and teacher gain recovery relative to the original SFT teacher. However, its student still underperforms the RL-guided student on both metrics, despite the RFT teacher itself having higher accuracy. Thus, the use of external demonstrations alone does not fully explain the observed RL teacher advantage.

We have added this experiment to Appendix C, Table 10. The improvement from RFT also identifies demonstration style as a promising direction for future work: demonstrations closer to the student’s own output style may facilitate subsequent OPD. We thank the reviewer for highlighting this direction.

Reviewer_1dRdOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether SFT or RL-trained (GRPO) domain teachers are better teachers for on-policy distillation (OPD), the step used to merge domain experts into one model. It studies single-teacher OPD: from a shared Qwen3.5-9B checkpoint, performance-matched SFT and RL teachers are trained in three domains (Agentic/MobileWorld, Reasoning/Geo3K, Perception/GUI grounding), and students re-initialized from the same checkpoint are distilled under identical settings within each domain. RL-guided students outperform SFT-guided students by 4.27, 1.50, and 0.86 pp at their best checkpoints and recover more of their teachers’ gains (115% vs. 44% in Agentic). The authors attribute this to RL teachers moving much less from the initialization (0.19–0.37x the SFT teachers’ displacement), supported by student displacement and Top-20 overlap dynamics.

Critical strengths
  1. A practically relevant question with a well-controlled design. Existing OPD-merging pipelines disagree on SFT vs. RL experts, and the paper shows that a teacher’s own score does not determine how well it teaches. Shared initialization, matched teachers, and fixed OPD settings isolate this choice better than prior pipeline papers, and the reporting (dataset revisions, per-checkpoint tables) is unusually complete.
  2. The Agentic result is not a best-checkpoint artifact: with teachers 0.57 pp apart, the RL-guided student leads at all seven checkpoints from step 16 to 64 and exceeds its teacher, while the SFT-guided student never reaches its teacher.
Critical weaknesses
  1. The compared variable is confounded, so the causal claim is not established. (a) SFT teachers imitate external targets (NVIDIA CoT, demonstration rationales), whereas RL teachers learn from their own samples. The effect may therefore stem from on-policy vs. off-policy data rather than RL itself, and an RFT teacher trained on its own correct samples is the needed control. (b) Displacement is also shaped by differing hyperparameters: in Reasoning, SFT uses a 5x higher learning rate than RL, and the Reasoning/Perception RL teachers have a KL penalty toward the initialization. The non-visual analysis in Appendix A addresses the frozen vision encoder but not these factors, and teacher training steps are not reported. Since smaller displacement under RL is already known (as the paper notes in Section 5; see also RL’s Razor, Shenfeld et al., 2025), the contribution rests on linking it to teachability. That link remains correlational over three domains without performance- and displacement-matched SFT teachers, and the paper does not argue why parameter-space proximity makes the teacher’s function easier to match under reverse KL.
  2. Statistical and metric support is weak relative to the recommendation. Results are single-seed and compared at the best of 8–11 checkpoints. The Reasoning gap (≈9 of 601 Geo3K problems, one sample at temperature 1) and the Perception gap are within plausible noise. Gain recovery favors the weaker teacher when students reach similar levels: in Reasoning, the RL teacher’s gain is 4.49 pp vs. 8.65 pp, which largely drives 263% vs. 119%. Matching is also loose. The Reasoning teachers differ by 4 pp on Geo3K, and on the four-benchmark average the SFT teacher (72.75) falls below the base (72.78) while the RL teacher rises (74.61), suggesting forgetting in the SFT teacher. Finally, the title targets OPD-based merging, but no multi-teacher experiment is run.
Questions for authors

Q1. Does a performance-matched RFT (self-generated SFT) teacher behave like the RL or the SFT teacher under OPD?

Q2. Do the gaps hold across multiple seeds? Please report paired per-task tests and checkpoint-averaged (or AUC) comparisons.

Q3. Do SFT teachers with comparable performance but smaller displacement (lower LR, earlier checkpoint, KL-regularized SFT) yield students that match RL-guided students?

11 comments
AuthorsOpenReview ↗

Freezing the vision encoder, matching teacher performance, and interpreting teacher gain recovery

We thank the reviewer for the thoughtful and constructive feedback. These comments have helped us strengthen the controlled comparisons and provide additional evidence for our findings. In the revised manuscript, we have aligned the visual-encoder freezing settings, matched teacher performance more closely, added controls for demonstration style and parameter displacement, and extended the comparison to MOPD. We have also repeated OPD training with different random seeds and conducted paired per-task statistical tests. We address each concern below.

1. Freezing the vision encoder, matching teacher performance, and interpreting teacher gain recovery

Thank you for highlighting the importance of consistent training settings and teacher-performance matching. We retrained the reasoning SFT teacher with the visual encoder and merger frozen and replaced the corresponding teacher and student results throughout the revised manuscript. We also replaced the reasoning RL teacher with one whose Geo3K accuracy more closely matches that of the SFT teacher, updating the corresponding single-teacher OPD and MOPD results.

The revised comparison is shown below. Teacher and student accuracies are averaged over three evaluation runs, and parameter displacement is measured from the shared initialization. Relavent top-k overlap information has also been updated in the revisied paper. RL teacher's student's top-k overlap is still always higher than SFT teacher's student while training.

Teacher typeTeacher accuracy (%)Teacher parameter displacement (ΔL2)Best student accuracy (%)Teacher gain recovery (%)
SFT88.353.61988.3099.36
RL88.070.27491.29138.50

The teachers now differ by only 0.28 percentage points in accuracy. The RL teacher exhibits a substantially smaller parameter displacement and produces a student with both higher accuracy and higher teacher gain recovery.

We also clarify that our conclusion draws on both absolute student accuracy and teacher gain recovery. A lower teacher score can increase the recovery ratio for a given student score, which is why we report the student's absolute performance alongside recovery. In the original comparison, the RL teacher already produced a stronger student despite its lower task accuracy. The revised comparison preserves this advantage with more closely matched teachers, strengthening the evidence through both metrics.

AuthorsOpenReview ↗

Controlling for the output style of SFT demonstrations

2. Controlling for the output style of SFT demonstrations

We appreciate the reviewer's suggestion to examine the influence of demonstration style. To further isolate this factor, we added a rejection fine-tuning (RFT) experiment using responses generated by the shared base model.

Specifically, we use the Geo3K training questions as prompts for Qwen3.5-9B, collect correct responses through rejection sampling, and retain at most one correct response per question. We train a teacher on these responses using the same SFT configuration as the main experiment, select its best-performing checkpoint, and then train its student using the same reasoning OPD configuration.

Teacher typeTeacher accuracy (%)Best student accuracy (%)Teacher gain recovery (%)
Main SFT88.3588.3099.36
RFT88.9690.45116.03
RL88.0791.29138.50

RFT improves student accuracy and teacher gain recovery relative to the original SFT teacher. However, the RFT-guided student still achieves lower best-checkpoint accuracy and teacher gain recovery than the RL-guided student.

We have added this experiment to Appendix C, Table 10. The improvement from RFT also highlights demonstration style as a promising direction for future work: demonstrations closer to the student's own output style may facilitate subsequent OPD. We thank the reviewer for bringing attention to this useful research direction.

AuthorsOpenReview ↗

Connecting teacher displacement to OPD performance

3. Connecting teacher displacement to OPD performance

To further examine the relationship between teacher displacement and OPD performance, we added a comparison between two SFT teachers trained on the same dataset with different batch sizes. This produces teachers with similar Geo3K accuracy but different parameter displacements while keeping the teacher-training algorithm fixed.

SFT teacherTeacher accuracy (%)Parameter displacement ΔL2Relative displacement (%)Best student accuracy (%)Teacher gain recovery (%)
Batch size 3288.353.6190.201488.3099.36
Batch size 12887.912.5900.144289.24116.22

Although the smaller-displacement teacher has slightly lower accuracy, it produces a stronger student and higher teacher gain recovery. Its parameter displacement is 28.42% smaller, while its best student's accuracy increases by 0.94 percentage points and gain recovery increases from 99.36% to 116.22%. To answer Q3 directly, this student improves to 89.24% but remains below the RL-guided student's 91.29%.

This comparison provides additional evidence within the same training algorithm and dataset: with closely matched teacher performance, smaller teacher displacement is associated with better subsequent OPD outcomes. These results strengthen the empirical support for our hypothesis and are included in Section 6.1, Table 4.

Section 6.2 provides a complementary comparison across training paths. The agentic SFT→RL and direct-RL teachers both achieve 23.36% accuracy, but their parameter displacements are 1.765 and 0.528, respectively. Their best students achieve 21.37% and 24.22%, with gain recovery of 65.00% and 115.00%. This comparison at identical teacher accuracy further connects smaller displacement with better subsequent OPD outcomes.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Statement on Manuscript Revisions

We thank the reviewers for their constructive feedback. We have updated the manuscript and PDF with the following revisions and additional experiments:

  1. More closely matched reasoning teachers. We retrained the SFT teacher with the visual encoder frozen and replaced the RL teacher with one whose accuracy more closely matches that of the SFT teacher. We updated the corresponding student results to provide a more consistent comparison.

  2. A controlled comparison of SFT teachers with different parameter displacements. Using the same training algorithm and dataset, we trained SFT teachers with similar accuracy but different parameter displacements. The smaller-displacement teacher produces a stronger OPD student and higher teacher gain recovery, supporting our hypothesis connecting teacher displacement with subsequent merging performance.

  3. An additional SFT→RL teacher-training path. We added an agentic experiment in which the teacher undergoes SFT followed by RL. Its student's accuracy and teacher gain recovery fall between those obtained with the standalone SFT and direct-RL teachers, providing further evidence that teacher-training choices affect subsequent OPD outcomes.

  4. Multi-teacher OPD experiments. We added MOPD experiments using the same domain teachers as the single-teacher comparisons. The RL teacher group produces a student with higher accuracy in all three domains, as well as higher overall accuracy and teacher gain recovery. These results extend our findings to joint capability merging.

  5. An RFT control for demonstration style. We added an experiment in which the teacher is trained on correct responses generated by the shared base model through rejection sampling in Appendix C. This control improves student performance over the original SFT teacher, while the RL teacher still produces a stronger student with higher teacher gain recovery.

  6. Evaluation on an additional model backbone. We added experiments using Qwen3-VL-4B-Instruct in Appendix D. Despite slightly lower teacher accuracy, the RL teacher again produces a stronger OPD student than the SFT teacher, showing that the observed advantage also holds with a different model backbone.

  7. A discussion of additional factors and future directions. We added a Discussion section clarifying that parameter displacement is one of several teacher properties that training algorithms may influence. Other effects of teacher training may also shape subsequent OPD outcomes, and we identify their investigation as a direction for future work.

AuthorsOpenReview ↗

Further Revision

Statement on the Latest Manuscript Revision

We thank the reviewers for their continued feedback. Building on our previous revision, we have further updated the manuscript as follows:

  1. An additional RL comparison in Section 6.1. Keeping the training algorithm unchanged, we used RL teachers from different training rounds for separate OPD experiments. The results further support our hypothesis linking teacher parameter displacement to student performance.

  2. A more concise Conclusion. We streamlined the Conclusion to improve readability while preserving its original meaning and main findings.

  3. Direct measurements of advantage scale and effective update magnitude in Appendix E. We added measurements of signed advantage means, advantage RMS, and actual parameter-update norms during OPD under SFT and RL teachers across all three domains, providing additional evidence on supervision signals and student updates.

These additions and revisions are highlighted in blue in the updated PDF.

AuthorsOpenReview ↗

Summary of Revisions Since the Original Submission

We thank the reviewers for their constructive feedback. Compared with our original submission, the revised manuscript includes the following changes and additional experiments.

Main Text

  1. More closely matched reasoning teachers (Sections 4 and 5.1; Appendices A.2 and B). We retrained the SFT teacher with the visual encoder frozen and replaced the RL teacher with one whose accuracy more closely matches that of the SFT teacher. We updated the corresponding student results and statistical analyses. The relative relationships and main conclusions remain unchanged.

  2. Multi-teacher OPD experiments (Sections 3.2 and 5.2). We added MOPD experiments using the same domain teachers as the single-teacher comparisons. The RL teacher group produces a student with higher accuracy in all three domains, as well as higher overall accuracy and teacher gain recovery. These results extend our findings to joint capability merging.

  3. Additional experimental evidence (Sections 6.1–6.2). The within-SFT and within-RL comparisons further support our hypothesis linking teacher parameter displacement to student performance. The SFT→RL comparison provides additional evidence that teacher-training paths affect subsequent OPD student performance.

  4. A more concise Conclusion (Section 8). We streamlined the Conclusion to improve readability while preserving its original meaning and main findings.

  5. A discussion of additional factors (Section 7). We added a Discussion section clarifying that parameter displacement is one of several teacher properties that training algorithms may influence. Other effects of teacher training may also shape subsequent OPD outcomes.

Appendix

  1. An RFT control for demonstration style (Appendix C). We added an experiment in which the teacher is trained on correct responses generated by the shared base model through rejection sampling. This control improves student performance over the original SFT teacher, while the RL teacher still produces a stronger student with higher teacher gain recovery.

  2. Evaluation on an additional model backbone (Appendix D). We added experiments using Qwen3-VL-4B-Instruct. Despite slightly lower teacher accuracy, the RL teacher again produces a stronger OPD student than the SFT teacher, showing that the observed advantage also holds with a different model backbone.

  3. Direct measurements of advantage scale and effective update magnitude (Appendix E). We added measurements of signed advantage means, advantage RMS, and actual parameter-update norms during OPD under SFT and RL teachers across all three domains, providing additional evidence on supervision signals and student updates.

  4. Further discussion and future directions (Appendix F). We discuss additional factors that may influence OPD student performance and identify their investigation as a direction for future research.

In the revised PDF, we highlight in blue the updated reasoning teacher and student results in the main single-teacher OPD and MOPD experiments (Sections 4 and 5), the additional RL comparison in Section 6.1, the direct measurements in Appendix E, and the revised Conclusion (Section 8), making these updates easier to locate.

Our main claim has remained unchanged throughout these revisions. The additional experiments strengthen the evidence that teacher-training algorithms affect subsequent OPD outcomes and further support our hypothesis that smaller teacher parameter displacements may help students achieve better performance through OPD. We sincerely thank the reviewers for their time, effort, and insightful feedback, which have helped us improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.