- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes B-MOPD, a two-stage method for MOPD. It balances input-token exposure across domains, then adjusts teacher weights using estimated gradient strength and the teacher-student reverse KL gap, giving greater emphasis to teachers with smaller gaps.
Critical strengths
- The idea of balancing input exposure and gradient strength is intuitive, and the reported improvements across domains partly support the method's motivation.
Critical weaknesses
- For B-MOPD, in stage 2, it includes several hyperparameters, such as the weight-ratio cap, estimation window, and EMA coefficient, but it does no provide any sensitivity analysis of these settings. The cap and window are described as "fixed empirical choices." This leaves unclear whether the method works consistently on different MOPD scenarios, such as including more domains, highly imbalanced data mixtures, and teachers from different model architectures.
- The experiments are limited to two small students from the Qwen2.5 family and three domains. Comparisons with existing MOPD methods, such as [1], are missing. The related-work discussion also gives limited coverage of MOPD, making it difficult to assess the method's contribution relative to prior approaches.
[1] Gao, Huan-ang, et al. "Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillation." arXiv preprint arXiv:2608.19098 (2026).
Questions for authors
- Do the same hyperparameter settings remain effective across different MOPD configurations?
- How does B-MOPD compare with existing MOPD methods such as [1]?