- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes "Swift-MOPD" which was based on the observation that uniform data allocation in multi-teacher on-policy distillation (MOPD) leads to uneven performance gain. To alleviate this problem the authors propose two components - 1. merged checkpoint for student initialization, 2. a data allocation scheme. The model merging step involves treating the teacher model on the hardest domain as subject, and treating other teachers as auxiliary. They use an appropriate model merging framework to ensure that the student initialization checkpoint preserves the performance on the hardest-fit domain knowledge. The data allocation step involves allocating lower weightage to the subject domain and allocating higher weightage to auxiliary domains, to compensate the model initialization scheme. Experiements were conducted with Qwen3-4B model on various domains such as mathematics, science (biology, physics, chemistry, materials), long codebench and sudoku. Prominent baselines include vanilla MOPD, EXOPD, Linear merging (for model merging) and several RL-based baselines.
Critical strengths
-
Well motivated problem - multi-teacher distillation require careful consideration for data allocation and optimization function - the paper did a good job in tackling the problem from two different orthogonal aspects -- model merging (to keep the hardest domain as dominating dimension) and difficulty-aware data allocation.
-
Easy to follow paper - the presentation of the paper is good, its easy to follow and necessary implementation-level details are provided.
Critical weaknesses
-
Inadequate experiments - ablations are not sufficient to conclude the empirical strengths of Swift MOPD. There are two main components - difficulty-aware merging (part-1) and diffculty-aware data allocation (part-2). Only Part-1 and Part-2 in isolation are considered in experiments. The authors must show part-1+MOPD, part-2+MOPD as baselines. Similarly linear merge+part-2 should be considered as baseline.
-
Part 2.2 conclusions are not well-supported - one of the main contribution of the paper is part 2.2. However, majority of the conclusions are not sufficiently supported by the provided results (Fig 2). Several critical issues - the authors only consider Maths and Biology for demonstration, with Biology highlighted as most difficult domain. For supporting evidence the authors should have demonstrated all sub-domains of Science task. Takeaway 1 says "more concise" - this is true only for Maths, not for Biology. Takeaway 2 can not be generalized with just 2 tasks. None of these insights are validated statistically. The word "convergence" is used very loosely, and in many cases inappropriately.
Questions for authors
Few more comments/suggestions.
- Experiments on larger models. A single model is insufficient to establish stronger claims.
- Although some of these baselines came after ICLR submission, try considering some of these for benchmarking - Open-MOPD [Gao et al., 2026], -MOPD [Sang et al., 2026], MOPD-router [Xu et al., 2026]. Several of these methods rely on very similar philosophy.