acceptodds
Under review as a conference paper at ICLR 2027

Swift MOPD: A Better Balanced Method for Parameter and Data Allocation in Multi Teacher On Policy Distillation

Abstract

Multi-Teacher On-Policy Distillation (MOPD) aims to transfer complementary capabilities from multiple domain-specialized teachers into a single student policy, but directly applying OPD from a generic base model can lead to unstable optimization and uneven absorption of teacher knowledge across domains. In this work, we first analyze the difficulty of transferring multiple domain experts into a single student and show that different domains exhibit different fitting difficulty, which implies that multi-teacher distillation should carefully balance both data allocation and parameter allocation across teachers. Building on this analysis, we propose Swift-MOPD, a framework that assigns the initial merging coefficients according to per-task fitting difficulty, so that the student starts from a policy that preserves the capabilities of the harder-to-fit domains, and dynamically allocates more training data to under-fitted tasks during distillation. This enables a better initialization and substantially faster fitting of the student. Through comprehensive experiments on MOPD across four distinct domains, we derive two main findings: (1) initializing the merged weights according to task difficulty coefficients effectively preserves the performance on harder-to-distill domains, approaching the level of the corresponding teachers; (2) under the same training budget, while vanilla MOPD suffers from uneven training across domains, Swift-MOPD consistently recovers the performance of all domain experts. We hope our work offers new insights for future research on MOPD.

Primary area:
reinforcement learning
Keywords:
On policy distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hHUSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes "Swift-MOPD" which was based on the observation that uniform data allocation in multi-teacher on-policy distillation (MOPD) leads to uneven performance gain. To alleviate this problem the authors propose two components - 1. merged checkpoint for student initialization, 2. a data allocation scheme. The model merging step involves treating the teacher model on the hardest domain as subject, and treating other teachers as auxiliary. They use an appropriate model merging framework to ensure that the student initialization checkpoint preserves the performance on the hardest-fit domain knowledge. The data allocation step involves allocating lower weightage to the subject domain and allocating higher weightage to auxiliary domains, to compensate the model initialization scheme. Experiements were conducted with Qwen3-4B model on various domains such as mathematics, science (biology, physics, chemistry, materials), long codebench and sudoku. Prominent baselines include vanilla MOPD, EXOPD, Linear merging (for model merging) and several RL-based baselines.

Critical strengths
  • Well motivated problem - multi-teacher distillation require careful consideration for data allocation and optimization function - the paper did a good job in tackling the problem from two different orthogonal aspects -- model merging (to keep the hardest domain as dominating dimension) and difficulty-aware data allocation.

  • Easy to follow paper - the presentation of the paper is good, its easy to follow and necessary implementation-level details are provided.

Critical weaknesses
  • Inadequate experiments - ablations are not sufficient to conclude the empirical strengths of Swift MOPD. There are two main components - difficulty-aware merging (part-1) and diffculty-aware data allocation (part-2). Only Part-1 and Part-2 in isolation are considered in experiments. The authors must show part-1+MOPD, part-2+MOPD as baselines. Similarly linear merge+part-2 should be considered as baseline.

  • Part 2.2 conclusions are not well-supported - one of the main contribution of the paper is part 2.2. However, majority of the conclusions are not sufficiently supported by the provided results (Fig 2). Several critical issues - the authors only consider Maths and Biology for demonstration, with Biology highlighted as most difficult domain. For supporting evidence the authors should have demonstrated all sub-domains of Science task. Takeaway 1 says "more concise" - this is true only for Maths, not for Biology. Takeaway 2 can not be generalized with just 2 tasks. None of these insights are validated statistically. The word "convergence" is used very loosely, and in many cases inappropriately.

Questions for authors

Few more comments/suggestions.

  • Experiments on larger models. A single model is insufficient to establish stronger claims.
  • Although some of these baselines came after ICLR submission, try considering some of these for benchmarking - Open-MOPD [Gao et al., 2026], Δ\Delta-MOPD [Sang et al., 2026], MOPD-router [Xu et al., 2026]. Several of these methods rely on very similar philosophy.
Reviewer_HeHnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to address the multi-teacher on-policy distillation problem. While existing methods improve MOPD with different supervision signals and objectives, the initialization of the student policy remains overlooked. Also, the paper shows that the initialization is important in preserving the distillation performance in hard domains, and proposes Swinft-MOPD to improve MOPD problem. The effectiveness of Swift-MOPD is grounded with the empirical experiments.

Critical strengths
  1. The MOPD problem is interesting and relevant.
  2. The paper is well written and easy to follow.
  3. The empirical performance of Swift-MOPD is attractive, with a big performance gain over baselines.
Critical weaknesses
  1. The paper (line 142) asks: "Is there a correlation between the fitting difficulty required to train a teacher model and the difficulty for a student model to distill toward that teacher’s performance level?" However, there is no answer to this question in the paper. The difficulty to train the teacher and the difficulty for distillation still remain unclear. I highly recommend answering this question with empirical experiments.

  2. The paper mention that the proposed method is "faster" (line 027 and line 078). However, there is no analysis ground this claim. It is recommended to include compuation/memory analysis and compare the overhead with baseline methods.

Questions for authors
  1. What is the correlation between the fitting difficulty of the teacher v.s. distillation difficulty of students? Are there any empirical / theoritical analysis that could ground the possible answer?
  2. What is the computation/memory overhead of Swift-MOPD compared with other baseline methods?
  3. In the paper (line 215) "hardness is measured by the number of optimization steps each RL expert requires to reach its peak performance". There is no information about the difficulty of the domains. What are the optimization steps for each domain used in paper's experiment section, and how are they measured (with what experiment, what model, what metric, etc.)?
  4. How many seeds are used in the experiments? What is the standard deviation of the numbers in the table?
Reviewer_uyfEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposed Swift-MOPD to address imbalanced capability integration issue. It introduces a difficulty-aware initialization step using parameter-space merging to protect hard-to-fit domains, along with a data allocation scheme that biases sampling toward under-fitted domains during distillation.

Critical strengths
  • The paper highlights student initialization as a crucial, under-explored design lever in MOPD, offering a logical premise that starting with a protected parameter space for hard domains can mitigate uneven distillation.

  • The proposed framework operates on initialization and data batching, making it complementary to objective-level distillation improvements (like EXOPD), which is an attractive modular design.

Critical weaknesses
  • The distillation experiments are limited to an extremely short 10 gradient steps (see Appendix A). This is insufficient to establish convergence, evaluate long-term trade-offs, or prove that the method does not suffer from catastrophic interference or overfitting under a standard training budget.

  • The fundamental "difficulty estimator" and the precise mathematical derivation of the per-teacher weights (waw_a) are missing. The method relies on a qualitative notion of optimization steps to peak performance, but lacks the quantitative, reproducible formulas necessary for peer validation.

Questions for authors
  • How exactly are the per-teacher weights (waw_a) computed from the difficulty estimator? Please provide the precise formula and scale normalization used.

  • How do the student's performance and stability evolve beyond the initial 10 gradient steps? Please provide longer training curves to validate convergence.

  • Can you explicitly report and match the compute/budget parity (wall-clock time, total tokens, or FLOPs) between your method and the RL baselines (Mix-RL, Cascade-RL) to ensure a fair comparison?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.