acceptodds
Under review as a conference paper at ICLR 2027

Open-MOPD: Diagnosing and fixing capability im-balance in multi-teacher on-policy distillation

Abstract

Multi-teacher on-policy distillation (M-OPD) has emerged as a promising paradigm for consolidating domain-specialized reinforcement learning (RL) experts into a single generalist student via dense, token-level reward supervision. Despite its practical success, the optimization dynamics governing multi-teacher capability integration remain poorly understood, and open and rigorously reproducible recipes remain scarce. In this work, we construct an evaluation framework for multi-domain capability integration using SmolLM3-3B-Base. Our investigation reveals a pronounced capability integration gap: standard M-OPD recovers only of the improvement from mixed-domain SFT to RouteRL. Our analysis shows that teacher disagreement is not the bottleneck; the main failure is a severe misallocation of the token-level optimization budget. We identify three separable contributors to this imbalance: structural sequence-length disparities across domains, different convergence rates across domains, and reward staleness caused by repeated minibatch updates on a shared rollout. To resolve these imbalances, we introduce Open-MOPD, a novel framework incorporating three mechanisms: token-share balancing, gap-aware dynamic budget allocation, and student reward refresh. Together, these mechanisms systematically restore cross-domain balance, raising the recovery rate from to of the improvement from mixed-domain SFT to RouteRL for a single student model. We fully open-source our end-to-end post-training recipe, training trajectories, and evaluation suites on an 8A100-80GB academic setup.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, multi-teacher distillation, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_aEYEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Open-MOPD investigates capability integration from three domain-specific RL teachers into a SmolLM3-3B student. It combines domain token-share balancing, reward-magnitude-based allocation, and student reward refresh. Across math, code, and instruction following, the reported macro score rises from 28.05 to 31.24, recovering 83.4% of the improvement from mixed-domain SFT to RouteRL and approaching RouteOPD’s 31.55.

Critical strengths
  1. The paper clearly develops the capability-integration problem through response-length differences, domain reward dynamics, and within-batch reward staleness, then motivates corresponding balancing, allocation, and refresh mechanisms.
  2. The experiments support the practical value of Open-MOPD for consolidating domain specialists into one student, with useful evidence on component benefits and runtime overhead. For example, Open-MOPD improves all three domain averages over Naive M-OPD and reaches 31.24 versus RouteOPD’s 31.55. Incremental comparisons at fixed K show component gains, while paired timing reports no measurable additional overhead from reward refresh.
Critical weaknesses
  1. Missing controls prevent the experiments from sufficiently separating the recipe’s gains from training-setting effects. Naive uses K=1 and the full method K=4; changing K after share+gap adds 0.49 points, but Table 2 lacks a plain K=4 control. The paper also uses a batch-wide token average (§4.1), without a same-K comparison to per-response loss averaging, as used in MOPD [1]. These controls matter for distinguishing benefits of the proposed combination from those of the inner-update schedule and length normalization.
  2. The diagnostics and ablations do not yet establish optimization-budget allocation as the principal bottleneck. Reward magnitude serves as a proxy for the remaining teacher–student capability gap, but its relationship to actual per-domain performance gaps is insufficiently validated. Moreover, the inverse-allocation comparison changes both exponent magnitude and clamping: the forward rule uses α=1 with a gap-factor clamp, whereas the inverse rule uses α=-0.5 without it. The evidence supports the chosen recipe’s effectiveness, but cannot isolate allocation direction as the cause of inverse-rule failure or fully substantiate the proposed mechanism explanation.

References

[1] Wenhan Ma et al. (2026). MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training. arXiv:2606.30406v1.

Questions for authors
  1. Could the authors provide batch sizes, rollout counts, optimizer updates, processed tokens, and checkpoint-selection rules for Table 2, together with a plain K=4 control at comparable exposure? Please also explain Table 7’s 600 steps versus the plotted 300-step trajectories.
  2. Could the authors compare per-response loss averaging (as in MOPD [1]) with the current recipe at the same K and comparable training budget, to determine how much improvement length normalization explains and what allocation and refresh add?
  3. Could the authors use existing checkpoints to assess whether reward magnitude tracks the remaining per-domain teacher–student performance gaps, and compare forward/inverse allocation with matched clamps and exponent magnitudes? Would these checks support the proposed mechanism explanation?

References

[1] Wenhan Ma et al. (2026). MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training. arXiv:2606.30406v1.

3 comments
AuthorsOpenReview ↗

Follow-up on our response to Reviewer aEYE

Dear Reviewer aEYE,

We appreciate your constructive feedback on our submission. We have uploaded our responses and additional analyses addressing each of your points. As the discussion phase is drawing to a close, we would welcome your thoughts on whether our reply resolves your concerns, and we remain happy to provide any further clarification.

Thank you again for your time,

The authors

Thanks for the detailed rebuttal and the new experiments. The matched K=4K=4 comparison—showing consistent gains across three seeds—addresses my concern about the original training setup. Adding the per-response averaging baseline and the forward–inverse comparison (with matched exponent magnitudes and clamping) also builds a much stronger case that Open-MOPD improves capability integration here.That said, my main concern remains the claim that budget misallocation is the primary bottleneck. While the new correlations indicate that reward magnitude tracks the teacher–student gap over time within each individual domain, this still doesn't demonstrate that the signal can identify which domain would yield the largest overall gain from extra weight at any given point in training. In short, the results confirm that budget allocation matters, but they leave open how much of the integration challenge actually stems from cross-domain gradient interference. Since this distinction is central to the paper's core contribution, I am keeping my rating at 2.

Thank you for the follow-up, and we are glad that the new experiments addressed your earlier concerns. We answer your remaining question below and will include these points in the revised version.

  1. About the bottleneck claim and gradient interference

Thank you for raising this important question. In §3 we compared budget allocation with teacher disagreement. Intervening on disagreement lowered the score, while only correcting the budget at K=1 already brought back about half of the gap to RouteOPD and nearly all of the lost IF score.

For gradient interference, we tried to look at it from two sides during the discussion. First, RouteOPD trains each domain in its own student, so it has no cross-domain interference. Open-MOPD has no gradient-level component, and with the full recipe the joint student comes to 0.31 points behind RouteOPD, from the original 3.50, with Code and IF at almost the same level. Whatever interference contributes has to fit within this remaining gap. Second, in the branch experiment shared with Reviewer JUAj, we measured the cost of joint training directly. Code-only training was 1.22 pp ahead in the first 15 steps, and this became smaller and not significant in the next 15 steps. So interference does exist, mainly in early training, but its effect is limited compared with what the budget correction recovers. Even if part of this gain comes through reducing interference, it is the allocation that needs to be corrected, and this alone closes most of the gap. We believe this supports budget misallocation as the main bottleneck in our setting.

  1. About the reward signal

We agree with the distinction you made. Reward magnitude tells us how far a domain still is from its teacher. It is not designed to pick the best domain for extra weight at every step, and what we show is that allocating by this signal works better than the alternatives in the matched forward–inverse comparison.

In the revision, we will make clear in the abstract and §3 that the comparison is between budget allocation and teacher disagreement, add the RouteOPD comparison and branch experiments as an analysis on interference, and update Lines 77 and 83 to say that extra budget goes to domains with larger remaining gap.

Over the discussion period we have tried our best to address each of your questions with additional experiments, including the matched multi-seed comparison, the per-response averaging baseline, the matched forward–inverse allocation, and the analysis above. We hope these together give a clearer picture of our contribution, and we would be grateful if you could take them into account in your final assessment.

Reviewer_N1G1OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates why multi-teacher on-policy distillation (M-OPD) struggles to integrate domain-specialized capabilities. The authors argue that the primary bottleneck is cross-domain budget misallocation, not teacher disagreement. To address this, they propose "Open-MOPD," which introduces three mechanisms: token-share balancing, gap-following allocation based on reward magnitudes, and student-side reward refresh. The approach is validated on a SmolLM3-3B base model across three domains (math, code, instruction following).

Critical strengths

The paper reframes the common "teacher disagreement" narrative by identifying and measuring concrete imbalances, such as length-induced token share and intra-rollout reward staleness.

Critical weaknesses
  • The proposed mechanisms, particularly token-share balancing and domain reweighting, are highly incremental. They conceptually overlap with existing ideas—most notably RM-OPD’s Domain Contribution Reweighting [1]. Although [1] should be considered as a concurrent work, I still want the authors to clarify the difference and the novelty of the proposed work.

  • The experimental validation is restricted to a single small base model (SmolLM3-3B) and only three domains. The lack of comparisons against strong, standard baselines (e.g., GradNorm-style balancing, RA-OPD/RG-OPD) leaves the generalizability and true effectiveness of the method in question.

[1] RM-OPD: On-Policy Distillation for Generative Reward Models. arXiv:2610.20174.

Questions for authors
  • How does Open-MOPD conceptually differ from RM-OPD’s domain contribution reweighting (DCR), and what are the empirical results when they are directly compared?

  • Why were standard optimization balancing baselines, such as GradNorm, omitted from the evaluation?

  • Have you evaluated the robustness of these heuristics on larger backbones, varied routing granularities, or a wider array of domains?

Reviewer_VuUnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies capability imbalance in multi-teacher on-policy distillation and introduces Open-MOPD, a recipe for integrating domain-specialized teachers into one student. It identifies three potential bottlenecks: unequal response lengths across domains, differences in distillation progress, and stale student-dependent rewards during inner updates. The proposed remedies are token-share balancing, reward-magnitude-based dynamic domain weighting, and reward refresh. Experiments use SmolLM3-3B with mathematics, coding, and instruction-following teachers, evaluated on AIME 2024/2025, LiveCodeBench v5/v6, IFEval, and IFBench test. Open-MOPD improves the three-domain macro-average from 28.05 to 31.24, increasing recovery of the improvement from mixed-domain SFT to routed RL teachers from 35.6% to 83.4%. The largest domain gain is in instruction following, from 43.64 to 49.58. The final score approaches separately distilled, domain-routed students (31.55) and exceeds task-arithmetic parameter merging (30.46). Incremental ablations support the recipe’s practical effectiveness, although changes in update configuration and limited validation leave some mechanistic conclusions unresolved.

Critical strengths
  1. A concrete diagnosis with an actionable correction. The paper shows that instruction-following examples constitute approximately 20.3% of prompts but only 0.99% of response tokens. This clearly explains how token-averaged optimization can underweight an entire domain. Token-share balancing directly corrects this mismatch and improves the overall score by 1.17 points. The connection between a measurable training property and a simple intervention is the paper’s strongest contribution.
  2. A useful integration result supported by component ablations. The final student improves all three domain averages over naive M-OPD and approaches the routed single-domain OPD reference. Table 2 separately demonstrates benefits from gap-based weighting and reward refresh within their respective fixed configurations. The paired timing measurements also support the claim that refresh adds no measurable overhead in this implementation. These results make the recipe practically useful beyond the descriptive diagnosis.
Critical weaknesses

The headline comparison does not fully isolate the proposed mechanisms, and robustness remains uncertain. Table 2 compares naive M-OPD at K=1K=1 with the full recipe at K=4K=4, but does not include an otherwise unchanged naive K=4K=4 baseline. Although the incremental ablations are informative, the headline gain cannot be attributed entirely to the three mechanisms without clarifying matched training budgets and update schedules. Moreover, the full method is evaluated on only one backbone and teacher configuration, without reported independent training-seed variability. This particularly limits confidence in the 0.78-point advantage over task-arithmetic merging.

The evidence supports the interventions more strongly than the proposed causal explanation. Sampled-token disagreement between teachers is not equivalent to interference between domain gradients, so unsuccessful masking and consensus interventions do not establish that cross-domain interference is unimportant. Likewise, the reward-magnitude statistic is not validated against downstream capability gaps or marginal learning gains. Its computation sums signed candidate rewards before taking the absolute value, allowing cancellation. The inverse-weighting comparison also changes both the exponent and clipping safeguards (Appendix B.5), weakening the claim that the weighting direction itself explains collapse. These limitations matter because mechanistic diagnosis is central to the contribution.

Questions for authors

Budget and attribution clarification: What are the rollout sizes, total generated tokens, optimizer-update counts, and GPU-hours for each Table 2 configuration? Does the improvement persist against naive M-OPD at K=4K=4 under a matched budget?

Mechanism clarification: What evidence connects the reward-magnitude proxy to actual capability gaps or subsequent learning gains? Does the inverse rule still collapse when compared using equal exponent magnitudes and identical clipping safeguards?

Reliability clarification: Are independent training-seed results available for Open-MOPD, its main ablations, and task-arithmetic merging? How much uncertainty surrounds their score differences?

1 comment
AuthorsOpenReview ↗

Follow-up on our response to Reviewer VuUn

Dear Reviewer VuUn,

Thank you for your time and thoughtful review. We have posted our detailed responses addressing your comments and questions. With the discussion period wrapping up soon, we would be grateful if you could take a look and let us know if anything remains unclear or if further details would help.

Best regards,

The authors

Reviewer_JUAjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper studies capability integration in multi-teacher on-policy distillation. The authors identify three potential sources of training imbalance: unequal token contributions across domains, differences in distillation reward magnitudes, and stale rewards during repeated updates. Open-MOPD addresses these issues through token-share balancing, gap-following allocation, and reward refresh. Experiments on math, coding, and instruction-following tasks show that the proposed method substantially improves capability recovery over naive multi-teacher distillation.

Critical strengths
  • The paper investigates a meaningful problem in multi-teacher distillation. The observation that similar prompt sampling ratios can produce highly unequal token contributions across domains is interesting and provides a concrete explanation for some of the difficulties in capability integration.

  • Open-MOPD raises the overall score from 28.05 to 31.24, approaching the performance of the domain-routed reference models. The diagnostic experiments and incremental ablations also help explain how the proposed mechanisms affect training. Providing an open training pipeline is another useful contribution.

Critical weaknesses
  • Open-MOPD first trains three domain-specific RL teachers and then performs an additional distillation stage to obtain a unified student. Although the paper compares against mixed-domain RL and parameter merging, it does not include sequential or cascaded RL under the same setting. This makes it difficult to assess whether the multi-teacher pipeline is preferable to simpler alternatives, especially considering its additional training costs. The reported cost analysis focuses mainly on reward refresh, without evaluating the end-to-end performance-cost trade-off.

  • Token-share balancing essentially rescales the loss contributions of different domains, while gap-following allocation adjusts these weights using reward magnitude. Reward refresh similarly relies on recomputing student-dependent signals during optimization. These are reasonable design choices, but their methodological novelty relative to existing domain reweighting and on-policy distillation techniques appears limited. The paper also lacks theoretical analysis explaining why these particular allocation rules should lead to better capability integration.

  • The gap-following mechanism treats average reward magnitude as a proxy for the remaining teacher-student capability gap. However, larger reward magnitudes do not necessarily indicate larger gradient contributions or greater potential for further improvement. Meanwhile, the teacher-conflict analysis relies on token-level probability disagreement rather than directly examining gradient interference. As a result, the evidence does not fully establish that optimization-budget imbalance, rather than interference between teachers, is the primary bottleneck.

  • All three teachers are derived from the same SmolLM3-3B initialization, which may naturally limit their disagreement. It remains unclear whether the proposed diagnosis and mechanisms generalize to heterogeneous or cross-family teachers, where capability differences and conflicting supervision could be more pronounced. The evaluation is also limited to one student scale, with no reported uncertainty across independent training runs. Additionally, the main comparison uses K=1 for naive M-OPD and K=4 for the full method, making matched-configuration results important for attributing the improvements.

Questions for authors
  1. How does Open-MOPD compare with sequential or cascaded RL in terms of both performance and total training cost? 
  2. What is the main methodological novelty beyond existing domain reweighting and reward recalculation techniques? Is there any theoretical justification for the proposed allocation rules? 
  3. How well does reward magnitude reflect the remaining capability gap or actual gradient contribution? Have the authors examined gradient interference between teachers? 
  4. How would the method behave with heterogeneous or cross-family teachers? Could the authors also clarify the matched-K comparison?

Comments on the paper

5 comments
Area_Chair_4DKoOpenReview ↗

Discussion

Dear Reviewers,

We are reaching the end of the discussion period! If the authors have responded to your review, please read their response, leave a comment, and update your rating if your assessment has changed.

Thank you,

AC.

Thank you for the detailed response and additional experiments. The sequential RL comparison, matched training settings, and multi-seed results address several of my concerns and make the empirical findings more convincing. I have raised my soundness score to 3, but will maintain my overall rating, as I still have some concerns about the methodological novelty and the strength of the proposed explanations for the observed training imbalance.

AuthorsOpenReview ↗

Response on novelty and imbalance diagnostics

Thank you for continuing the discussion and for noting that our added experiments make the empirical findings more convincing. Below, we clarify how our work relates to prior M-OPD studies and report new branch-ablation and diagnostic results, all of which we will add to the revised paper.

  1. Relation to prior M-OPD work

In our setup, single-domain OPD recovers 88.0% of the SFT-to-RL gain, whereas joint distillation with the same teachers and routing recovers 35.6%. Rather than stopping at prompt sampling or routing, we focus on what happens inside the loss once prompts and teachers are fixed: response-length skew (IF has 20.3% of prompts vs. 0.99% of tokens), uneven domain convergence speeds, and stale cached rewards across inner updates.

Among prior studies, MOPD (Ma et al., 2026) normalizes per-response loss to handle length; D$^3$-MOPD (Sun et al., 2026) tracks learning speeds via KL gap and decay velocity to adjust which prompts to sample next; and CaMOPD (Chen et al., 2026) alternates between general recovery and domain preservation while prioritizing large-gap samples. We take one step downstream: keeping sampling and routing unchanged, we fix how the current batch updates the student—reweighting domain loss shares and refreshing student rewards within a single loss.

WorkMain question studiedWhat is directly modified
MOPD (Ma et al., 2026)Can domain RL experts be consolidated into one student?Routes by domain and averages token loss per response
D$^3$-MOPD (Sun et al., 2026)When domains learn at different speeds, what should be sampled next?Adjusts prompt sampling ratios using KL gap and decay speed
CaMOPD (Chen et al., 2026)How to recover general skills while preserving domain skills?Alternates updates and selects samples with larger teacher–student gaps
Every Coin (Wei et al., 2026)How does one teacher's influence spill over to other domains?Varies prompt ratios between two teachers and tracks capability trade-offs
When Top-K (Shen et al., 2026)Which tool-call decision tokens are missed by top-K?Modifies the candidate token set and its supervision strength
H-OPD (Yin et al., 2026)Where are VL vs. text teachers most useful in one trajectory?Combines supervision from both teachers at the token level
UI-MOPD (Lian et al., 2026)How to unify desktop and mobile GUI teachers in one student?Routes each trajectory to its platform-specific teacher
MAD-OPD (Wang et al., 2026)Can multiple teachers give better supervision on the same prompt?Has teachers debate a task and merges guidance by confidence
VG-OPD (Xu et al., 2026)On which tokens does an expert provide verifiable help?Uses a verifier to gate where each expert supervises
DF-OPD (Li et al., 2026)Where to get distillation prompts when teacher data is unavailable?Generates prompts directly from the teachers themselves
Open-MOPDOnce teachers and prompts are fixed, how much weight does each domain get?Corrects token share, allocates loss weight by gap, and refreshes rewards
  1. New branch ablations on teacher conflict vs. budget imbalance

To test whether gradient cosine predicts held-out capability transfer—as ForkMerge (Jiang et al., 2023; §3.1) questioned in auxiliary-task learning—we branched Math-only and Code-only runs from a 60-step joint training trajectory at steps 0 and 15. Each branch ran 15 steps of on-policy distillation with the exact same domain prompts and loss coefficients as the joint run, simply dropping the other domains' updates. Below we compare single-domain vs. joint performance (95% CIs via paired bootstrap) alongside the mean full-parameter Math–Code cosine measured every 5 steps:

Training stepsMean Math–Code cosineMath-only − Joint, pp [95% CI]Code-only − Joint, pp [95% CI]
0–15+0.060+0.060+1.46+1.46 [−1.67,+4.58][-1.67, +4.58]+1.22+1.22 [+0.13,+2.36][+0.13, +2.36]
15–30−0.169-0.169−0.21-0.21 [−4.38,+4.17][-4.38, +4.17]+0.70+0.70 [−0.39,+1.81][-0.39, +1.81]

When the mean cosine is positive, joint training still trails Code-only; when it turns negative, the Code gap does not widen. We therefore do not use gradient cosine to guide allocation (aside from its overhead: 11–15 s for per-domain backward passes vs. ~0.03 ms to reduce cached rewards to magnitude).

We also checked whether teacher log-probability spread (max minus min teacher log-probability on the same prefix and token) captures genuine supervision disagreement. Restricted to the student's top-16 candidates, higher spread corresponds to much higher distributional disagreement:

Teacher log-probability spreadTokensMean pairwise JS (nats)Teachers differ on top-1
$ \le 0.1 $ nat98,2750.001470.22%
$ > 1 $ nat4,5750.0418131.15%

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.