- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses the problem of Multi-teacher on policy distillation (MOPD) by proposing MOPD-Router, a dense token-level multi-teacher distillation method where student token level supervision is routed by a weighting metric to all available teacher models based on each teacher's alignment between specialization direction and the teacher's correction direction. Rather than the hard prompt-level domain teacher routing of existing MOPD methods, this paper's proposed method is supervised by a mixture of teacher experts at the token level, with teacher weighting for each student token calculated based on the teacher expert direction alignment to the correction direction. The paper's claimed contributions are its MOPD-Router plug-and-play method for multi-teacher expert selection during policy distillation, it's proposed ExpertAlign teacher specialization aware routing metric, and experimental results showing that the proposed method's cross-domain teacher signals provide the superior distillation performance compared to competing hard domain label MOPD methods.
Critical strengths
For strengths, the paper itself is well written, with the problem statement and proposed method clear and easy to understand. The authors highlight an important issue with existing hard-domain expert routing, where many tasks can require expertise in multiple domain categories that makes a single expert insufficient. Additionally, from the experimental results shown, the proposed MOPD-Router outperforms existing hard MOPD methods like standard MOPD and open-MOPD when averaged across all the selected evaluation datasets over the three different task categories of Math, Coding, and Instruction Following.
Critical weaknesses
For weaknesses, the paper suffers from two main weaknesses in my opinion.
The first weakness is the actual paper contributions. While authors claim the proposed MOPD-Router and ExpertAlign metric as main contributions of the paper, most of the method components and ideas are taken directly from existing literature with little innovations on top. For example, as mentioned in the paper introduction, dense token level student supervision is a standard operation in OPD. ExpertAlign itself uses the log probability delta between teacher and student/base model introduced in prior work. The main difference comes from using the delta as a routing signal instead of a distillation signal. This seems like an incremental work rather than novel method.
The second, more important, weakness is that the evaluation protocol and comparison fairness seems questionable to me. This paper's training and evaluation set differ from those in MOPD and open-MOPD, meaning all comparisons to these two methods are based on the authors' own implementation of these methods on their chosen training/evaluation set. A fair comparison would also involve comparing MOPD-Router on the training and evaluation set used the MOPD or open-MOPD papers (choose the one that's the strongest competitor). If this is not practical, then at least explain in the paper what steps you took to make sure your training and implementation of these methods are fair representation of these methods' capabilities.
Additionally, the paper emphasizes that MOPD-Router is a label-free method, however it relies on a shared base model for expert selection. That same base model could just be prompted to assign hard domain labels, so in practice the requirement is not that different from label-based routing. This also raises a question about using a separate llm for hard domain labels in the unlabeled setting when in theory the domain routing should actually align with the teacher's training distribution. This difference is reflected in the experimental results, where the gap between MOPD-Router and open-MOPD is much smaller when training domain labels are available. I'm also curious how much of the performance gain can be attributed to ExpertAlign being a direct estimate of the task alignment to each expert domain, vs gain from the soft multi-expert routing. An additional experiment training both open-MOPD and MOPD using hard prompt-level domain routing calculated from the ExpertAlign metric would answer this question.
Questions for authors
See the weaknesses section.
6 comments
Response to Reviewer JpQd (W3)
W3. Prompted labels vs. label-free routing, and the source of the gain
Why not prompt a model for domain labels? Thank the reviewer for such a valuable suggestion, and we do evaluate this option, with a much stronger labeler than the 4B base. On the unlabeled set, DeepSeek-V4-Flash labels each example from its prompt and reference answer, agreeing with human labels on 87.3% of a validation sample in Appendix C. Standard MOPD and Open-MOPD trained with these labels still trail MOPD-Router with ExpertAlign by 2.25–5.00 Overall, according to Table 8. Moreover, any prompted label assigns one teacher to the entire rollout, which is exactly the limitation MOPD-Router removes.
Domain estimate vs. token-level multi-teacher routing. We understand reviewer's question as whether ExpertAlign helps mainly by being an accurate estimate of each prompt's domain, or by softly mixing several teachers. We have conducted the corresponding ablation experiments on the labeled set, and the results are shown in Table 4:
| Variant | What it keeps | Overall-1.7B | Overall-4B |
|---|---|---|---|
| MOPD-Router w/ ExpertAlign | Token-level alignment over the full pool | 40.19 | 54.58 |
| w/o Token-Level Routing | ExpertAlign as a per-response domain estimate | 36.57 | 52.61 |
| w/o Cross-Domain Teachers | Token-level alignment, labeled-domain teacher only | 37.79 | 51.83 |
| Mean Aggregation | All teachers by average | 35.78 | 48.16 |
| Standard MOPD | True hard domain labels | 37.52 | 50.63 |
| Open-MOPD | True hard domain labels | 39.08 | 52.37 |
According to these results, neither factor alone explains the gain. Used only as a domain estimate, ExpertAlign performs at the level of label-based routing. Mixing all teachers softly without the metric (Mean) is worse still. Removing either token-level allocation or the off-domain teachers costs 1.97–3.62 Overall. The gain therefore comes from applying the alignment estimate at every token over the full teacher pool. The experiment of hard prompt-level labels derived from ExpertAlign, is an offline version of the second row; at best it recovers the true domain labels used by Standard MOPD and Open-MOPD, which still perform worse than the full MOPD-Router with ExpertAlign.
In summary, these controlled comparisons show that the improvement of MOPD-Router does not come from better domain labels or from soft mixing alone. It comes from routing every token over the full teacher pool with metric ExpertAlign. If we have misunderstood your question, please let us know; we would be glad to discuss it further and address any remaining concerns.
We sincerely appreciate the reviewer's constructive suggestions and believe that the additional explanations significantly improve the quality of our submission. We hope that this provides sufficient reasons to raise the score.
References
[1] Heo, Byeongho, Jaehui Hwang, Sangdoo Yun, Dongyoon Han. On-Policy Delta Distillation. arXiv:2607.15161, arXiv, 2026.
[2] Feng, Shiyuan, Huan-ang Gao, Haohan Chi, Hanlin Wu, Zhilong Zhang, Zheng Jiang, Bingxiang He, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou. Weak-to-Strong Generalization via Direct On-Policy Distillation. arXiv:2607.05394, arXiv, 2026.
[3] Zhang, Kongcheng, Qi Yao, Shunyu Liu, Wenjian Zhang, Min Cen, Yang Zhou, Wenkai Fang, Yiru Zhao, Baisheng Lai, Mingli Song. Replay Failures as Successes: Sample-Efficient Reinforcement Learning for Instruction Following. arXiv:2512.23457, arXiv, 2025.
Thank you for the detailed response to my review. Your response and additional experiments, has partially addressed my questions regarding the proposed method. However, the main two weaknesses identified in my review are still mostly unaddressed or deflected. Regarding novelty, the paper "H-OPD: Confidence Aware Heterogeneous Multi-Teacher Multimodal On-policy Distillation" already proposes token level arbitration for MOPD. While I agree that the ExpertAlign metric itself builds on the teacher-base delta idea, this contribution alone is incremental rather than novel. The most important weakness I identified, comparison fairness, has not been sufficiently addressed in the response. My concern is not whether the evaluation benchmarks are reasonable for the paper's task, but rather whether the existing competitor methods have been adequately represented in the evaluation. A method’s published training configuration is tuned for its own setup, including the student, teacher pool, data and objective. Changing any of these requires retuning. The paper changes all of them for the competing methods and then applies a single shared configuration, while the only reported method-specific tuning (top-k for ExpertAlign, selected on the evaluation benchmarks; Appendix F.4) favors the proposed method. A fair comparison requires either evaluating on a baseline’s published setup, where its configuration and reported numbers apply, or tuning every method with an equal budget on a held-out validation set in the new setup. Where this is impractical, a more accurate framing would present the experiments as controlled comparisons against routing strategies inspired by prior work, e.g., “prompt-level hard routing (MOPD-style)” and “domain budget balancing (Open-MOPD-style, re-implemented)”, rather than attributing the results to the original methods. Note that such reframing would only make the claims accurate; it would not address the underlying weakness. Without a comparison that fairly represents the prior methods, either on their published setups or with equal tuning effort, the paper cannot establish improvement over existing work, and my assessment would remain the same.
Response to Reviewer JpQd (W1, W2)
Thank you for the constructive comments. The experiment you propose at the end is already in the paper (Table 4); we explain below.
W1. Contributions
First of all, we respectfully clarify that MOPD-Router is not a standard operation. Dense token-level supervision is indeed standard within OPD, but how that supervision is allocated among multiple teachers is not: standard MOPD pipelines assign one teacher per prompt from a domain label and keep it for the whole rollout.
To our knowledge, MOPD-Router is the first systematic study of token-level teacher routing in MOPD. It replaces prompt-level assignment with label-free, token-level routing over the full teacher pool, without training a separate router, and compares three metrics from three information sources under one interface, objective and budget, across labeled and unlabeled data and two student scales. Beyond the method, the study also shows that off-domain teachers supply complementary supervision, which lets a same-size student surpass its RL teachers on math and code.
ExpertAlign builds on, rather than reuses, the teacher–base delta. We refer to the idea from prior work [1, 2] that the teacher–base log-policy difference$e_i$ captures the specialization a teacher acquired in post-training. However, prior work uses it only as a distillation signal for a single teacher. We propose a student-dependent teaching vector di and use its alignment with the expertise vector as a token-level routing criterion across teachers. To show that the teaching vector is necessary, we have added an new ablation that routes by the expertise vector ei alone. On the labeled set, ‖e_i‖-only routing reaches 37.73 / 52.67 Overall for the 1.7B / 4B students, against 40.19 / 54.58 for ExpertAlign, and is lower in all three domains (math/code/if) at both scales.
We have revised the contribution statement in Section 1 and related work in Section 2.2 to draw these lines explicitly.
W2. Evaluation protocol and fairness
In addition, we take the following measures to ensure a fair comparison across methods:
All methods share the objective, teacher pool, rollout configuration, optimization budget and hyperparameters, and differ only in routing (Appendix B, Table 5).
For Open-MOPD, we use its officially released training framework with our training data.
To give the label-based baselines their strongest form, we evaluate them in two training-data settings: a domain-labeled set, where they use the true domain labels, and an unlabeled set, where every example is labeled by a strong LLM, DeepSeek-V4-Flash, with labels validated against human annotation.