- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Open-MOPD investigates capability integration from three domain-specific RL teachers into a SmolLM3-3B student. It combines domain token-share balancing, reward-magnitude-based allocation, and student reward refresh. Across math, code, and instruction following, the reported macro score rises from 28.05 to 31.24, recovering 83.4% of the improvement from mixed-domain SFT to RouteRL and approaching RouteOPD’s 31.55.
Critical strengths
- The paper clearly develops the capability-integration problem through response-length differences, domain reward dynamics, and within-batch reward staleness, then motivates corresponding balancing, allocation, and refresh mechanisms.
- The experiments support the practical value of Open-MOPD for consolidating domain specialists into one student, with useful evidence on component benefits and runtime overhead. For example, Open-MOPD improves all three domain averages over Naive M-OPD and reaches 31.24 versus RouteOPD’s 31.55. Incremental comparisons at fixed K show component gains, while paired timing reports no measurable additional overhead from reward refresh.
Critical weaknesses
- Missing controls prevent the experiments from sufficiently separating the recipe’s gains from training-setting effects. Naive uses K=1 and the full method K=4; changing K after share+gap adds 0.49 points, but Table 2 lacks a plain K=4 control. The paper also uses a batch-wide token average (§4.1), without a same-K comparison to per-response loss averaging, as used in MOPD [1]. These controls matter for distinguishing benefits of the proposed combination from those of the inner-update schedule and length normalization.
- The diagnostics and ablations do not yet establish optimization-budget allocation as the principal bottleneck. Reward magnitude serves as a proxy for the remaining teacher–student capability gap, but its relationship to actual per-domain performance gaps is insufficiently validated. Moreover, the inverse-allocation comparison changes both exponent magnitude and clamping: the forward rule uses α=1 with a gap-factor clamp, whereas the inverse rule uses α=-0.5 without it. The evidence supports the chosen recipe’s effectiveness, but cannot isolate allocation direction as the cause of inverse-rule failure or fully substantiate the proposed mechanism explanation.
References
[1] Wenhan Ma et al. (2026). MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training. arXiv:2606.30406v1.
Questions for authors
- Could the authors provide batch sizes, rollout counts, optimizer updates, processed tokens, and checkpoint-selection rules for Table 2, together with a plain K=4 control at comparable exposure? Please also explain Table 7’s 600 steps versus the plotted 300-step trajectories.
- Could the authors compare per-response loss averaging (as in MOPD [1]) with the current recipe at the same K and comparable training budget, to determine how much improvement length normalization explains and what allocation and refresh add?
- Could the authors use existing checkpoints to assess whether reward magnitude tracks the remaining per-domain teacher–student performance gaps, and compare forward/inverse allocation with matched clamps and exponent magnitudes? Would these checks support the proposed mechanism explanation?
References
[1] Wenhan Ma et al. (2026). MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training. arXiv:2606.30406v1.
3 comments
Thanks for the detailed rebuttal and the new experiments. The matched comparison—showing consistent gains across three seeds—addresses my concern about the original training setup. Adding the per-response averaging baseline and the forward–inverse comparison (with matched exponent magnitudes and clamping) also builds a much stronger case that Open-MOPD improves capability integration here.That said, my main concern remains the claim that budget misallocation is the primary bottleneck. While the new correlations indicate that reward magnitude tracks the teacher–student gap over time within each individual domain, this still doesn't demonstrate that the signal can identify which domain would yield the largest overall gain from extra weight at any given point in training. In short, the results confirm that budget allocation matters, but they leave open how much of the integration challenge actually stems from cross-domain gradient interference. Since this distinction is central to the paper's core contribution, I am keeping my rating at 2.
Thank you for the follow-up, and we are glad that the new experiments addressed your earlier concerns. We answer your remaining question below and will include these points in the revised version.
- About the bottleneck claim and gradient interference
Thank you for raising this important question. In §3 we compared budget allocation with teacher disagreement. Intervening on disagreement lowered the score, while only correcting the budget at K=1 already brought back about half of the gap to RouteOPD and nearly all of the lost IF score.
For gradient interference, we tried to look at it from two sides during the discussion. First, RouteOPD trains each domain in its own student, so it has no cross-domain interference. Open-MOPD has no gradient-level component, and with the full recipe the joint student comes to 0.31 points behind RouteOPD, from the original 3.50, with Code and IF at almost the same level. Whatever interference contributes has to fit within this remaining gap. Second, in the branch experiment shared with Reviewer JUAj, we measured the cost of joint training directly. Code-only training was 1.22 pp ahead in the first 15 steps, and this became smaller and not significant in the next 15 steps. So interference does exist, mainly in early training, but its effect is limited compared with what the budget correction recovers. Even if part of this gain comes through reducing interference, it is the allocation that needs to be corrected, and this alone closes most of the gap. We believe this supports budget misallocation as the main bottleneck in our setting.
- About the reward signal
We agree with the distinction you made. Reward magnitude tells us how far a domain still is from its teacher. It is not designed to pick the best domain for extra weight at every step, and what we show is that allocating by this signal works better than the alternatives in the matched forward–inverse comparison.
In the revision, we will make clear in the abstract and §3 that the comparison is between budget allocation and teacher disagreement, add the RouteOPD comparison and branch experiments as an analysis on interference, and update Lines 77 and 83 to say that extra budget goes to domains with larger remaining gap.
Over the discussion period we have tried our best to address each of your questions with additional experiments, including the matched multi-seed comparison, the per-response averaging baseline, the matched forward–inverse allocation, and the analysis above. We hope these together give a clearer picture of our contribution, and we would be grateful if you could take them into account in your final assessment.
Follow-up on our response to Reviewer aEYE
Dear Reviewer aEYE,
We appreciate your constructive feedback on our submission. We have uploaded our responses and additional analyses addressing each of your points. As the discussion phase is drawing to a close, we would welcome your thoughts on whether our reply resolves your concerns, and we remain happy to provide any further clarification.
Thank you again for your time,
The authors