acceptodds
Under review as a conference paper at ICLR 2027

-MOPD: ynamic omain Scheuling for Efficient Multi-Teacher Distillation

Abstract

Multi-teacher on-policy distillation (MOPD) distills several domain-expert teachers into a single student by minimizing per-domain reverse-KL divergence on the student's own rollouts. Existing approaches typically fix the per-domain data mixture before training, overlooking the fact that different domains converge at substantially different rates: some plateau early while others continue to improve throughout the training budget. A fixed mixture therefore wastes compute on fast-converging domains and undertrains slower-converging ones. To address this, we propose D-MOPD (ynamic omain Scheuling for MOPD), a zero-overhead scheduler that repurposes the per-domain reverse-KL signal already produced during training to adapt the domain mixture online. Running asynchronously outside the training process, an off-process watcher periodically tracks each domain's KL trajectory, estimates remaining headroom and current improvement rate, and accordingly adjusts the domain sampling ratios without altering the core training loop. Our D-MOPD scales naturally to arbitrary numbers of domains, and the expected benefit grows as more domains introduce more diverse convergence patterns for the scheduler to exploit. On a Qwen3.6-35B-A3B student distilled from four domain-expert teachers, D-MOPD closes 97% of the average student-to-teacher performance gap, compared with 63% for vanilla MOPD, reaches the same peak performance with an approximately reduction in rollout steps, and surpasses the specialist teachers on three of seven benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Teacher On-Policy Distillation, Dynamic Domain Scheduling, Large Language Model Post-Training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hXpzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes D³-MOPD, a dynamic domain scheduling method for multi-teacher on-policy distillation. The key observation is that different domains converge at different rates during joint training, making a fixed domain mixture potentially inefficient. To address this, the authors adaptively adjust the sampling ratio of each domain based on its remaining reverse-KL gap and descent velocity. The proposed method improves both final performance and training efficiency compared to vanilla MOPD.

Critical strengths
  1. Well-motivated and intuitive approach. The paper identifies an important problem in multi-domain training: different domains converge at different rates, so maintaining a fixed sampling ratio may waste computation on already-converged domains. The proposed approach is simple, intuitive, and practically relevant.

  2. Practical and lightweight implementation. The scheduler reuses the per-domain reverse-KL signals already available during training without modifying the core optimization procedure or introducing substantial computational overhead.

  3. Promising empirical results. The method consistently improves over vanilla MOPD across multiple benchmarks, and the ablation studies demonstrate the contributions of the remaining gap, descent velocity, and smoothing mechanisms.

Critical weaknesses

1. Repeated oscillations in domain sampling ratios (Figure 8).

Figure 8 shows that some domains experience repeated increases and decreases in their sampling ratios, rather than a simple transition from high allocation to low allocation as they converge. Appendix F partially explains this behavior through KL rebound, but the underlying cause remains unclear.

Are these oscillations primarily caused by interference or forgetting between domains, or by instability in the estimated descent velocity? It would be useful to analyze whether these repeated reallocations are necessary for maintaining performance or whether a more stable scheduling strategy could achieve comparable results.

2. Missing comparisons with sequential or curriculum-based domain training.

The primary baseline is vanilla MOPD with a fixed domain mixture. However, the central motivation of this paper is that different domains converge at different rates.

A natural alternative would be to train domains sequentially until convergence before switching to another domain. Another possibility is alternating domain training or sequential training with rehearsal to mitigate catastrophic forgetting.

Since the paper already discusses alternating domain training in the related work, comparing against such strategies would help clarify whether the improvements come specifically from the proposed adaptive scheduling mechanism or more generally from reallocating training budgets across domains.

3. Robustness to non-monotonic convergence and KL rebound.

The proposed descent velocity is defined as

vk(t)=max⁡(0,−1R∑i=0R−1δk(i)(t)).v_k(t)=\max\left(0,-\frac{1}{R}\sum_{i=0}^{R-1}\delta_k^{(i)}(t)\right).

When a domain's reverse-KL increases, its estimated descent velocity becomes zero, causing its composite score to vanish regardless of the remaining gap. However, training dynamics are not necessarily monotonic, and temporary increases in KL do not always indicate that further training would be unproductive.

I appreciate that Appendix H.1 acknowledges this failure mode and introduces an optional velocity floor. Nevertheless, this raises a question about the robustness of the default scheduling rule. In particular, the claim that the scheduler is self-correcting appears to require further clarification: an increase in the remaining KL gap alone cannot restore the sampling priority when the descent velocity remains zero.

Could the authors clarify under what conditions a domain recovers from such a state, and whether the velocity-floor extension provides more reliable performance across different training settings?

4. Limited comparisons against simple adaptive scheduling baselines.

Although the ablation studies are useful, the experimental comparisons are largely limited to fixed-mixture MOPD and variants of the proposed method.

Since the main contribution is an adaptive domain allocation strategy, it would be valuable to compare against simpler alternatives, such as a plateau-detection scheduler that reduces a domain's sampling ratio once its KL stops improving, or other heuristic curriculum-based scheduling strategies.

These comparisons would help establish whether the specific gap-velocity formulation provides benefits beyond simpler adaptive allocation rules.

Questions for authors

see weakness

6 comments
AuthorsOpenReview ↗

Response to Reviewer hXpz (Part 1/3, W1)

We thank Reviewer hXpz for the positive feedback on our motivation, design, and empirical results. We address your four points below with multi-checkpoint trajectory analysis, recovery conditions under KL rebound, and experimental comparisons against plateau-detection and sequential-rehearsal baselines.


W1. Repeated oscillations in domain sampling ratios (Figure 8)... Are these oscillations primarily caused by interference or forgetting between domains, or by instability in the estimated descent velocity? ...

Answer W1:

Comparing the trajectories in Figures 6–9 with the multi-checkpoint ablation results in Table R1 (Appendix G) and Table R2 (Appendix H.1, Table 8) clarifies the sources of these oscillations and shows that a smoother schedule achieves comparable accuracy:

  1. Within the convergence window, domain ratios change smoothly without repeated oscillations (Figure 6): Because 1 rollout step equals 4 gradient updates over ∼4,000\sim 4,000 prompts per domain, most domains converge within the first 100 rollout steps (∼1\sim 1 epoch), where Code decreases monotonically, Math rises smoothly, and IF transitions cleanly (Figure 6). The oscillations in steps 100–256 of Figures 6 and 8 appear because training continues well past saturation (>2>2 epochs), where small residual KL fluctuations across converged domains take turns shifting the ratios.
  2. High-frequency spikes come from single-window velocity noise (R=1R=1) and are not necessary (Table R1): In Figure 8(d), estimating velocity from a single W=10W=10 window (R=1R=1) causes frequent short-term spikes in Math and Code. Across the 5 shared checkpoints in Table R1, unsmoothed R=1R=1 exhibits a 4.7×4.7\times larger standard deviation (0.85 vs. 0.18), whereas averaging over R=3R=3 windows (Figure 7e) filters out these spikes and raises the 5-checkpoint mean by +1.36 (62.13 vs. 60.77).
  3. Low-frequency swings come from cross-domain KL rebound under zero-clipping (vk=0v_k=0), and smoothing them via the velocity floor (ϕ=0.2\phi=0.2) achieves comparable accuracy (Table R2): In Figure 7(c, e), cross-domain updates during steps 60–150 cause IF's normalized KL to rebound, clipping vIF(t)v_{\text{IF}}(t) to 0 and holding IF near the floor until s180. Enabling the velocity floor (ϕ=0.2\phi=0.2, Figure 9c and Table R2 below) eliminates this sharp swing and achieves comparable average accuracy (62.2 vs. 62.3) while improving IFBench (+0.8) and IFEval (+0.3).

Table R1: Multi-checkpoint trajectory comparison at the 5 shared ablation checkpoints (Appendix G).

MethodStep 63Step 95Step 127Step 191Step 2555-Ckpt Mean ± SDΔ\Delta MeanPeak (Table 1)
Vanilla MOPD60.4560.1160.6861.1560.6060.60 ± 0.38—61.36 (s143)
w/o smoothing (R=1R=1)60.0962.1760.9160.4960.1860.77 ± 0.85+0.1762.17 (s95)
w/o gap (Velocity-only)60.6260.5360.9360.9361.4660.89 ± 0.36+0.2961.46 (s255)
w/o velocity (Gap-only)60.8860.8661.4160.3861.4060.98 ± 0.43+0.3861.41 (s127)
w/o jitter (η=0,R=3\eta=0, R=3)61.3161.6361.4661.3961.4461.45 ± 0.12+0.8561.63 (s95)
D³-MOPD (η=0.30,R=3\eta=0.30, R=3)62.2462.3462.0962.1261.8662.13 ± 0.18+1.5362.34 (s95)

Thus, repeated oscillations are not required, and the performance gain comes from shifting budget away from saturated domains.

AuthorsOpenReview ↗

Response to Reviewer hXpz (Part 2/3, W3)

W3. Robustness to non-monotonic convergence and KL rebound... an increase in the remaining KL gap alone cannot restore the sampling priority when... whether the velocity-floor extension provides more reliable performance across different training settings?

Answer W3:

Thank you for this careful observation. While an increase in the remaining gap KL~k(t)\tilde{\text{KL}}_k(t) alone does not make sk(t)>0s_k(t) > 0 when vk(t)=0v_k(t) = 0, our scheduler still recovers priority through a two-stage mechanism under the default rule (ϕ=0\phi=0), which we have clarified in Appendix F:

1) Two-stage recovery condition under the default rule (ϕ=0\phi=0):

  • Stage 1 — Baseline exposure via Softmax + Floor (Eq. 6): Because s~k(t)=0\tilde{s}_k(t)=0 still contributes exp⁡(0/T)=1\exp(0/T)=1 to the softmax numerator on top of the floor ϵ=0.10\epsilon=0.10, a zero-score domain always receives at least pk(t)≥0.10+0.601+3e2≈12.6%p_k(t) \ge 0.10 + \frac{0.60}{1 + 3e^2} \approx 12.6\% (K=4,T=0.5K=4, T=0.5).
  • Stage 2 — Priority restoration once vk(t)>0v_k(t) > 0: As dominant domains saturate, this continual ≥12.6%\ge 12.6\% exposure pulls KLk\text{KL}_k downward again. Once the RR-window average KL change turns negative (vk(t)>0v_k(t) > 0), it multiplies the accumulated gap KL~k(t)\tilde{\text{KL}}_k(t) to restore priority (Figure 7c, s180).

2) Reliability of the velocity-floor extension (ϕ>0\phi > 0, Appendix H.1, Table 8): If cross-domain interference is strong enough that ∼12.6%\sim 12.6\% exposure cannot turn the KL slope negative, recovery under ϕ=0\phi=0 is delayed until other domains plateau. With the velocity floor v~k(t)=ϕ+(1−ϕ)min⁡(vk(t),1)\tilde{v}_k(t) = \phi + (1-\phi)\min(v_k(t), 1) (Eq. 9), a rebounding domain retains a positive fallback score sk(t)=ϕ⋅KL~k(t)>0s_k(t) = \phi \cdot \tilde{\text{KL}}_k(t) > 0 even when vk(t)=0v_k(t)=0. In Table R2, setting ϕ=0.2\phi=0.2 keeps IF's allocation within [0.30,0.40][0.30, 0.40] and cuts the IF KL rebound in half (Figure 9b) while improving both IF benchmarks.

Table R2: Velocity-floor robustness comparison on Qwen3.6-35B-A3B (Appendix H.1, Table 8).

Method (Checkpoint)AIME25HMMT(N)IFBenchIFEvalLCB-v6OJB-C++BFCL(B)Average
Vanilla MOPD (s143 peak)73.271.045.791.061.027.660.061.4
D³-MOPD Default (ϕ=0\phi=0, s95)74.071.445.191.262.529.762.562.3
D³-MOPD w/ Velocity Floor (ϕ=0.2\phi=0.2, s95)74.071.345.991.562.729.361.062.2

Thus, the default rule recovers whenever baseline exposure reverses the KL rebound, while ϕ=0.2\phi=0.2 provides immediate gap-proportional recovery under strong cross-domain interference.

AuthorsOpenReview ↗

Response to Reviewer hXpz (Part 3/3, W2 & W4)

W2 & W4. Missing comparisons with sequential or curriculum-based domain training (e.g., sequential training with rehearsal)... / Limited comparisons against simple adaptive scheduling baselines, such as a plateau-detection scheduler...

Answer W2 & W4:

Following your suggestion, we compare D³-MOPD with a Plateau-Detection Scheduler, Sequential Training with Rehearsal, and single-signal adaptive baselines on our 4-domain Qwen3.6-35B-A3B setup in Table R3:

Table R3: Comparison with Plateau-Detection Scheduler, Sequential Training with Rehearsal, and single-signal adaptive baselines on Qwen3.6-35B-A3B.

Method (Checkpoint)AIME25HMMT(N)IFBenchIFEvalLCB-v6OJB-C++BFCL(B)Average
Vanilla MOPD (s143)73.1771.0445.7491.0460.9527.5960.0061.36
w/o velocity (Gap-only, s127)72.6671.1545.1790.9159.5228.4562.0061.41 (+0.05)
w/o gap (Velocity-only, s255)71.7271.4649.0190.8360.1024.5762.5061.46 (+0.10)
Plateau-Detection Scheduler (s127)72.4069.9047.7891.6459.8127.1662.0061.53 (+0.17)
Sequential Training with Rehearsal (s127)72.6670.6248.9892.1459.4326.7262.5061.86 (+0.50)
D³-MOPD (Ours, s95)74.0171.3545.0791.1662.5229.7462.5062.34 (+0.98)

Analysis of Table R3:

  1. Plateau-Detection Scheduler (61.53) cuts early-converging domains too abruptly: Once a domain's KL stops improving (vk(t)=0v_k(t)=0), its sampling ratio drops to the floor (ϵ=0.10\epsilon=0.10) and the remaining budget is split equally among unconverged domains. Because Code and Math plateau earliest, this step-down shifts almost all subsequent training to IF and Tool-use, improving IFBench and IFEval but degrading earlier code and math benchmarks (LCB-v6, OJB-C++, and AIME25).
  2. Sequential Training with Rehearsal (61.86) exhibits the same early-domain trade-off and requires prior knowledge of convergence order: Using prior knowledge of convergence speed, this baseline trains domains sequentially (Code →\to Math →\to Tool-use →\to IF, 32 steps per stage over 128 steps) with an ϵ=0.10\epsilon=0.10 rehearsal floor for earlier domains, resulting in the same trade-off between late-stage IF gains and degradation on earlier code and math domains.
  3. Single-signal adaptive rules (61.41 and 61.46) fail to separate stalled high-gap domains from actively improving ones: Gap-only scheduling (w/o velocity) continues to allocate budget to high-gap plateaus, while velocity-only scheduling (w/o gap) oscillates across low-gap converged domains.

Therefore, by weighting domains continuously via KL~k⋅vk\tilde{\text{KL}}_k \cdot v_k without prior knowledge of convergence order, D³-MOPD balances active learning against retention and outperforms both single-signal and sequential/plateau schedules.


Overall, our trajectory analysis and velocity-floor results confirm that D³-MOPD is stable under non-monotonic KL dynamics, and our comparisons against Vanilla MOPD (+2.23 at matched s95), Plateau-Detection (+0.81), and Sequential Training with Rehearsal (+0.48) verify that the composite gap ×\times velocity formulation is effective. We have refined §3.2, §4.4, Appendix F, and Appendix G to incorporate these clarifications. We hope these results address your concerns, and we would be very grateful if you would consider raising your score.

Reviewer_SyZkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies using static domain mixture as a weakness in existing MOPD approaches. It claims that some domains can plateau quickly while others converge slowly, due to different learning speeds of the student on different domains. To this end, the authors propose a lightweight scheduler that uses the per-domain reverse-KL metric as a signal to dynamically update the data mixture online. The proposed method aims to achieve the best performance within the limited training time and budget. It combines each domain's current reverse-KL relative to its initial value with a smoothed recent fractional KL descent rate, then uses a softmax and allocation floor to update the training mixture. An asynchronous watcher and a stratified loader with batch-level jitter implement the scheduler without changing the core student rollout, teacher-prefill, or student-update operations. Experiments use one Qwen3.6-35B-A3B student with four specialist teachers, evaluated on seven benchmarks. The best-average single checkpoint improves from 61.36 to 62.34, and the proposed method exceeds the baseline's best average.

Critical strengths
  1. The paper writing is clear and easy-to-follow, with good presentation, clear figures and plots.
  2. The motivation is clear and the observation on the different convergence speed in different domains is unique and interesting.
Critical weaknesses
  1. The experiments are done on a small training budget and cannot be served as evidence that the proposed method will work at scale. The main experiments are done with only 256 rollout steps. In this region, the student may be very under-trained, and there seems to be a lot of variance in the benchmark results (with the accuracy percentage fluctuating up and down). If one extends the training runs to thousands of rollout steps, the accuracy of vanilla MOPD vs D3-MOPD may converge.
  2. The shown accuracy gains (62.34 vs 61.36) are marginal, and their reliability is unclear. The accuracy is fluctuating rapidly, so the accuracy gains might come from variance.
Questions for authors
  1. Could you show results on training substantially beyond 256 rollout steps under matched data access, optimizer settings, and compute budget?
  2. Why is there so much variance in the benchmark accuracy across rollout steps?
7 comments
AuthorsOpenReview ↗

Response to Reviewer SyZk (Part 1/2, W1 & Q1)

We thank Reviewer SyZk for the positive feedback and for finding our observation on asynchronous domain convergence unique and interesting. Under a fair comparison at the same training step (s95), D³-MOPD outperforms Vanilla MOPD by +2.23 on average, and without any checkpoint selection it wins all 17/17 evaluated checkpoints across 320 rollout steps (1,280 gradient steps), which we detail below.


W1 & Q1. The experiments are done on a small training budget... with only 256 rollout steps... / Could you show results on training substantially beyond 256 rollout steps...

Answer W1 & Q1:

1) 256 rollout steps equal 1,024 gradient steps (>2>2 epochs), which is 5×5\times longer than standard MOPD budgets: Unlike SFT, MOPD provides dense token-level teacher log-probabilities on on-policy rollouts and is a sample-efficient stage. In our setup (Table 4), each rollout step generates 128×4=512128 \times 4 = 512 trajectories and performs 4 gradient updates over ∼4,000\sim 4,000 prompts per domain (32,768 prompts in total). In comparison, prior MOPD work with the same batch size ([1], Table 3) converges within 40–52 steps, and MOPD ([2], §4.2) plateaus within 25K–30K samples per domain.

2) Models saturate around steps 127–143, and extending training to 320 rollout steps (1,280 gradient steps) confirms that D³-MOPD still outperforms Vanilla MOPD (Table R1): As shown in Figures 3 and 5, per-domain KL and accuracy already saturate around steps 127–143. Following your suggestion, we extended both runs to 320 rollout steps under matched settings in Table R1:

Table R1: Extended training comparison at rollout s319 vs. peak checkpoints on Qwen3.6-35B-A3B.

Method (Checkpoint)Gradient StepsAIME25HMMT(N)IFBenchIFEvalLCB-v6OJB-C++BFCL(B)Average
Vanilla MOPD (s95)38072.6671.0443.5389.9060.1925.4358.0060.11
Vanilla MOPD (s143 peak)57273.1771.0445.7491.0460.9527.5960.0061.36
D³-MOPD (s95 peak)38074.0171.3545.0791.1662.5229.7462.5062.34 (+0.98 vs. s143)
Vanilla MOPD (s319 extended)1,28073.2871.0445.4090.6960.6726.2960.5061.12
D³-MOPD (s319 extended)1,28072.6672.4047.6591.5961.4327.5961.5062.12 (+1.00)

Table R1 shows two clear results:

  • Extending the training budget from s143 to s319 does not improve Vanilla MOPD (61.12 vs. 61.36): by s319, the prompt pool has been traversed ~2.6 epochs (163,840 rollouts), and repeated passes cause converged code benchmarks to degrade (OJB-C++ and LCB-v6), so training for thousands of steps on the fixed pool would only worsen over-optimization.
  • At s319, D³-MOPD still outperforms Vanilla MOPD by +1.00 on average, winning 6 out of 7 benchmarks and exceeding Vanilla's peak (61.36).

In summary, 256 rollout steps already exceed standard MOPD budgets, and extending training to 320 rollout steps (1,280 gradient steps) confirms that D³-MOPD maintains a clear advantage.

AuthorsOpenReview ↗

Response to Reviewer SyZk (Part 2/2, W2 & Q2)

W2 & Q2. The shown accuracy gains (62.34 vs 61.36) are marginal, and their reliability is unclear... / Why is there so much variance in the benchmark accuracy across rollout steps?

Answer W2 & Q2:

1) Fair matched-step comparison and multi-sample evaluation across 16-step checkpoints:

  • Fair comparison at the same training step (Table R1): In Table 1, 62.34 vs. 61.36 (+0.98) compares D³-MOPD at s95 against Vanilla MOPD at s143, which uses 50% more training steps. At the same training step (s95), D³-MOPD outperforms Vanilla MOPD by +2.23 on average (62.34 vs. 60.11), winning all 7 benchmarks.
  • Multi-sample evaluation and dense 16-step checkpoints: Adjacent checkpoints fluctuate slightly because MOPD uses a constant learning rate (10−610^{-6}) with on-policy sampling at temperature 1.01.0 (Table 4). We reduce single-point evaluation noise via multi-sample evaluation in Appendix B (AIME25 avg@64, HMMT(N) avg@32, LCB-v6 avg@6) and dense evaluation every 16 rollout steps (17 checkpoints in total).

2) Full-trajectory statistics (Table R2) and Qwen3.5-4B results (Table R3) confirm that the gain is systematic:

Table R2: Exact 7-benchmark average across all evaluated rollout checkpoints (Figure 5 Average + extended s319).

Rollout Step153147637995111127143159175191207223239255319Mean ± SD (s47–s255)
Vanilla MOPD59.9260.6961.1460.4561.3560.1160.2160.6861.3660.9861.1261.1561.2860.9161.0160.6061.1260.88 ± 0.41
D³-MOPD60.1061.0562.0962.2462.0062.3461.6262.0961.8761.7062.0662.1262.1561.5462.2561.8662.1261.99 ± 0.24
Gain (Δ\Delta)+0.18+0.36+0.95+1.79+0.65+2.23+1.41+1.41+0.51+0.73+0.93+0.97+0.87+0.62+1.24+1.26+1.00+1.11 ± 0.48

Table R3: Replication on Qwen3.5-4B student (Appendix D, Table 7).

Method (Checkpoint)AIME25HMMT(N)IFBenchIFEvalLCB-v6OJB-C++BFCL(B)Average
Domain-Expert Teachers (4B)63.466.455.985.353.318.163.057.91
Initial Student (Qwen3.5-4B)47.851.735.985.339.413.851.046.41
Vanilla MOPD (s159 peak)59.363.445.482.753.718.563.055.14
D³-MOPD (s119 peak)61.864.449.584.554.619.864.557.01
Gain (Δ\Delta vs. Vanilla)+2.5+1.0+4.1+1.8+0.9+1.3+1.5+1.87

Tables R2 and R3 highlight three findings:

  1. D³-MOPD wins all 17 out of 17 evaluated checkpoints: Averaging over all 14 post-warmup steps (s47–s255) requires no checkpoint selection and widens the peak gain from +0.98 to +1.11.
  2. D³-MOPD's lowest post-warmup checkpoint (61.54 at s223) exceeds Vanilla MOPD's highest peak (61.36): Across s47–s319, the accuracy ranges of D³-MOPD [61.54, 62.34] and Vanilla MOPD [60.11, 61.36] do not overlap.
  3. Every benchmark improves in trajectory mean on Qwen3.6-35B-A3B and on Qwen3.5-4B (Table R3): Averaged over s47–s255 on Qwen3.6-35B-A3B, all 7 benchmarks improve by +0.78 to +1.54, and Qwen3.5-4B likewise improves across all 7 benchmarks by +1.87 on average.

Therefore, the accuracy gain of D³-MOPD is consistent across the full training trajectory and across model scales, rather than coming from step-to-step variance.


Overall, our extended 320-step experiments confirm that D³-MOPD maintains a +1.00 lead at longer training budgets, and our 17-checkpoint trajectory statistics and Qwen3.5-4B results verify that the performance gains are consistent across steps and model scales. We have refined §4.1, §4.3, and Appendix C to incorporate these clarifications. We hope these results address your concerns, and we would be very grateful if you would consider raising your score.

AuthorsOpenReview ↗

References

[1] Yang Z, Liu Z, Chen Y, et al. Nemotron-cascade 2: Post-training llms with cascade rl and multi-domain on-policy distillation[J]. arXiv preprint arXiv:2603.19220, 2026.

[2] Ma W, Wei J, Zhao L, et al. Mopd: Multi-teacher on-policy distillation for capability integration in llm post-training[J]. arXiv preprint arXiv:2606.30406, 2026.

Reviewer_HfcyOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

When you distill one student from several domain-expert teachers, the mix of training data across domains is usually fixed ahead of time, even though some domains improve much faster than others. This paper proposes a scheduler for multi-teacher on-policy distillation that tracks each domain's KL to its teacher during training and samples more from domains where the KL is still high and still dropping (remaining gap times recent rate of decrease, passed through a softmax with a minimum share per domain). With a Qwen3.6-35B-A3B student and four teachers (math, code, instruction following, tool use), they report closing 97% of the student-teacher gap vs. 63% for a uniform mix, and matching the uniform run's best score about 3x sooner. The pitch is that the KL signal distillation already computes is enough to set the data mix online at basically no extra cost.

Critical strengths
  • Simple and cheap: it only changes which prompts get sampled
  • Like ODM (Albalak et al., 2023) and unlike DoReMi (Xie et al., 2023), it needs no proxy or reference model because it reuses the per-domain KL that's already being computed. I think that's really desirable
Critical weaknesses
  • The gains are small and I can't tell them apart from noise. Everything is one run, checkpoints are picked on the test benchmarks, and the average gain at the best checkpoint is about 1 point, even on benchmarks like AIME and HMMT that have ~30 problems. The headline 97% vs. 63% also takes each benchmark's best score across 16 checkpoints. At a single checkpoint it's 73% vs. 48%, and dropping HMMT and OJBench (which have tiny teacher-student gaps), gives 76% vs. 59%. So the results seem a bit brittle.
  • It's not clear the scheduling is what helps. Without the "per-batch jitter" (random noise added to each batch's domain mixture) the gain drops a lot. The uniform baseline also uses a different data loader, so a lot of the improvement could just come from how batches are built.
  • Uniform is the only baseline, but I think there are relevant prior works to consider. This is basically online data mixing applied to multi-teacher distillation, and there's a lot of prior work using the same kind of signal: ODM (Albalak et al., 2023), DWA (Liu et al., 2019), and DUMP (Wang et al., 2025). Open-MOPD (Gao et al., 2026) also reweights domains by the remaining teacher-student gap in this exact setting, which is basically their "w/o velocity" ablation. So the claim that existing multi-teacher methods all use a fixed mix isn't right.
Questions for authors
  • Why does the score (remaining KL times how fast it's dropping) tell you which domain to give more data? The justification assumes every domain gets equal data and domains don't affect each other, but neither holds once the scheduler runs. Training on one domain can also hurt another as your own experiments show. In Appendix F, instruction-following KL increases dramatically when that domain is downweighted by the scheduler. A domain that gets more data will look like it's dropping faster just because it's getting more data, so the score partly reinforces the scheduler's own earlier choices.
  • What is specific to multi-teacher distillation here, compared to online data mixing in general? Reallocating data toward domains whose loss is still dropping is the core idea behind ODM and similar methods, and the gap-weighted version is basically Open-MOPD. If the main difference is just using reverse KL to each teacher as the loss, then what are the key insights you want people to take away from this?
7 comments
AuthorsOpenReview ↗

Response to Reviewer Hfcy (Part 1/4, W1)

We thank Reviewer Hfcy for the constructive feedback and for noting that D³-MOPD is simple, cheap, and requires no proxy or reference model. Without any checkpoint selection, D³-MOPD wins all 17/17 evaluated checkpoints and outperforms Vanilla MOPD by +2.23 on average at matched s95, and below we address your comments point by point with matched-step, full-trajectory, controlled ablation, and baseline comparisons.


W1. The gains are small and I can't tell them apart from noise. Everything is one run, checkpoints are picked on the test benchmarks... The headline 97% vs. 63% ... At a single checkpoint it's 73% vs. 48%...

Answer W1:

1) Fair matched-step comparison and single-checkpoint headroom (Table R1):

  • Fair comparison at the same training step: In Table 1, 62.34 vs. 61.36 (+0.98) compares D³-MOPD at s95 against Vanilla MOPD at s143, which uses 50% more training steps. When compared fairly at the same training budget (s95), D³-MOPD improves average accuracy from 60.11 to 62.34 (+2.23, winning all 7/7 benchmarks). We have updated the Abstract, Introduction, and §4.3–§4.5 to report matched-step, single-checkpoint (Appendix C, Table 6), and full-trajectory results alongside Table 2.
  • Teacher-student headroom and excluding small-gap tasks: Because MOPD trains teachers from the student initialization to avoid distribution-shift collapse ([1], §4.4), the average gap between the initial student (57.23) and the 4 teachers (64.68) is 7.45. Within this headroom, a single D³-MOPD checkpoint reaches 96.4% of the teachers' average accuracy and closes 73.3% vs. 48.5% of the normalized gap. Even after dropping both HMMT(N) and OJB-C++, D³-MOPD still improves gap closure from 59% to 76% (+17.1%) across per-benchmark peaks (Table 2) and from 50.6% to 64.9% (+14.3%, +0.87 raw accuracy) at the single checkpoint (Table 6).

Table R1: Summary of comparisons across evaluation settings.

Evaluation SettingVanilla MOPDD³-MOPD (Ours)Gain (Δ\Delta)
Matched Training Step (s95)60.1162.34+2.23
Best-Average Checkpoint — Accuracy (Appendix C, Table 6)61.36 (s143)62.34 (s95)+0.98
Best-Average Checkpoint — Gap Closure (Table 6)48.5%73.3%+24.8%
Best-Average Checkpoint excl. HMMT(N) & OJB-C++50.6% / 66.1864.9% / 67.05+14.3% / +0.87
Post-Warmup Trajectory Mean ± SD (s47–s255, 14 ckpts)60.88 ± 0.4161.99 ± 0.24+1.11
All-Checkpoints Trajectory Mean ± SD (s15–s255, 16 ckpts)60.81 ± 0.4561.82 ± 0.56+1.01
Extended Training Endpoint (s319)61.1262.12+1.00
Held-Out Benchmarks (AIME26 / HMMT-Feb / LCB-Exec)76.88 / 60.21 / 50.7379.48 / 61.56 / 51.98+2.60 / +1.35 / +1.25
Qwen3.5-4B Student (Appendix D, Table 7)55.14 (s159)57.01 (s119)+1.87

2) Table R1 confirms that our gains are reliable and not evaluation noise:

  • Multiple evaluation runs per benchmark reduce randomness: Although AIME25 has 30 problems, we evaluate it 64 times per checkpoint (1,920 responses), along with 32 runs on HMMT(N) and 6 runs on LCB-v6 (Appendix B), and the largest single-checkpoint gains come from the larger tool-use and coding benchmarks (BFCL(B) +2.5, OJB-C++ +2.1, LCB-v6 +1.5).
  • Dense evaluation every 16 steps (17 checkpoints in total) ensures trajectory robustness: D³-MOPD wins all 17 out of 17 checkpoints, and averaging all 14 post-warmup checkpoints requires no checkpoint selection while widening the gain to +1.11. Even D³-MOPD's lowest post-warmup score (61.54 at s223) exceeds Vanilla's highest peak (61.36 at s143), and on three held-out benchmarks not used for selection, comparing best-average checkpoints (s95 vs. s143) also yields +2.60 to +1.25.
  • Consistent gains across model scales: On Qwen3.5-4B (Appendix D, Table 7), D³-MOPD improves all 7 benchmarks by +1.87 on average, reaching 98.5% of the 4B teachers' average.

In summary, D³-MOPD achieves consistent improvements across matched steps, single-checkpoint evaluations, full 17-step trajectories, held-out benchmarks, and both 35B-A3B and 4B model scales.

AuthorsOpenReview ↗

Response to Reviewer Hfcy (Part 2/4, W2)

W2. It's not clear the scheduling is what helps. Without the "per-batch jitter"... the gain drops a lot. The uniform baseline also uses a different data loader...

Answer W2:

1) Single-variable comparison in Table R2 isolates the scheduling rule from the loader and jitter: All five scheduled variants in Table R2 use the same Stratified Loader, and w/o velocity, w/o gap, and D³-MOPD also share the same η=0.30\eta=0.30 jitter:

Table R2: Multi-checkpoint comparison across all ablation variants (Appendix G).

MethodData LoaderJitter η\etaBatch SD σ\sigmaStep 63Step 95Step 127Step 191Step 2555-Ckpt Mean ± SDPeak (Table 1)
Vanilla MOPD (Uniform)Pool-ShuffleNatural3.83%60.4560.1160.6861.1560.6060.60 ± 0.3861.36 (s143)
w/o smoothing (R=1R=1)Stratified0.303.75%60.0962.1760.9160.4960.1860.77 ± 0.8562.17 (s95)
w/o gap (Velocity-only)Stratified0.303.75%60.6260.5360.9360.9361.4660.89 ± 0.3661.46 (s255)
w/o velocity (Gap-only)Stratified0.303.75%60.8860.8661.4160.3861.4060.98 ± 0.4361.41 (s127)
w/o jitter (η=0,R=3\eta=0, R=3)Stratified0.000.00%61.3161.6361.4661.3961.4461.45 ± 0.1261.63 (s95)
D³-MOPD (η=0.30,R=3\eta=0.30, R=3)Stratified0.303.75%62.2462.3462.0962.1261.8662.13 ± 0.1862.34 (s95)
  • Holding the Stratified Loader and η=0.30\eta=0.30 jitter fixed, our composite scheduling rule contributes +1.15 to +1.24 in 5-checkpoint mean (+0.88 to +0.93 at peak) over the single-signal variants (Figure 8a, b), confirming that the loader and jitter alone do not drive the gain.
  • Moreover, during warmup (steps 0–20, §3.2), D³-MOPD runs the exact Stratified Loader and η=0.30\eta=0.30 jitter under uniform pk=0.25p_k=0.25, and its score at s15 is nearly identical to Vanilla MOPD's pool-and-shuffle (60.10 vs. 59.92) before dynamic scheduling activates.
  • Even without jitter (η=0\eta=0), D³-MOPD beats Vanilla MOPD at all 5 out of 5 checkpoints and improves the 5-checkpoint mean by +0.85 (Table R2).

2) Setting η=0.30\eta=0.30 jitter restores (rather than inflates) Vanilla MOPD's natural batch variance (§3.1): Under Vanilla MOPD's pool-and-shuffle loader, each domain's batch share fluctuates around 0.250.25 with binomial standard deviation σshuffle=p(1−p)/B=3.83%\sigma_{\text{shuffle}} = \sqrt{p(1-p)/B} = 3.83\%. Deterministic rounding (η=0\eta=0) reduces this batch variance to 0%0\%, whereas η=0.30\eta=0.30 yields σjitter≈p1−pη/3=3.75%\sigma_{\text{jitter}} \approx p\sqrt{1-p}\eta/\sqrt{3} = 3.75\%, restoring the natural ∼3.8%\sim 3.8\% batch variation of Vanilla MOPD.

Therefore, the performance gain is driven by our composite scheduling rule, while η=0.30\eta=0.30 jitter simply preserves the natural batch-level variation of standard training.

AuthorsOpenReview ↗

Response to Reviewer Hfcy (Part 3/4, W3 & Q2)

W3 & Q2. Uniform is the only baseline, but I think there are relevant prior works to consider... What is specific to multi-teacher distillation here... and what are the key insights you want people to take away from this?

Answer W3 & Q2:

1) Comparisons with online mixing baselines, Open-MOPD, and two additional scheduling baselines (Table R3): In Table R3, we compare D³-MOPD against ODM, DWA, DUMP, Open-MOPD [2], a Plateau-Detection Scheduler, and Sequential Training with Rehearsal on our 4-domain Qwen3.6-35B-A3B setup:

Table R3: Comparison of online allocation methods on Qwen3.6-35B-A3B.

MethodAllocation SignalWhere Applied35B Result (Mean / Peak)
Vanilla MOPD [1]Fixed uniformPrompt sampling60.60 / 61.36
DWA (Liu et al., 2019)Loss ratio Lk(t−1)/Lk(t−2)\mathcal{L}_k(t-1)/\mathcal{L}_k(t-2)Loss weightFavors slower-dropping tasks
DUMP (Wang et al., 2025)Advantage magnitude + UCBPrompt samplingRequires verifiable rewards
ODM (Albalak et al., 2023) / w/o gapDescent velocity vk(t)v_k(t)Prompt sampling60.89 / 61.46 (+0.10)
Open-MOPD [2] / w/o velocityRemaining gapLoss / Prompt60.98 / 61.41 (+0.05)
Plateau-Detection SchedulerStep to floor ϵ\epsilon on KL plateauPrompt sampling— / 61.53 (+0.17)
Sequential Training w/ RehearsalFixed stage order + floor ϵ\epsilonPrompt sampling— / 61.86 (+0.50)
D³-MOPD (Ours)Normalized gap ×\times velocity (KL~k⋅vk\tilde{\text{KL}}_k \cdot v_k)Prompt sampling62.13 / 62.34 (+0.98)

2) Two key insights specific to MOPD:

  • Insight 1 — Combining remaining gap with descent velocity overcomes the failure modes of both rate-only (ODM / w/o gap) and gap-only (Open-MOPD / w/o velocity) allocation: Specialist teachers have up to 65×65\times differences in baseline reverse-KL (Figure 2) and plateau at different non-zero levels. Allocating purely by descent rate (w/o gap) oscillates on low-gap converged domains (+0.10 in Table R3). Conversely, without initial-anchor normalization KLk(0)\text{KL}_k^{(0)} or velocity vk(t)v_k(t), a domain that plateaus at a high residual gap continues to dominate the budget: in Open-MOPD ([2], Figure 7a & Appendix B.5), gap-only weighting assigns up to 87.6% peak budget (55.4% mean) to Code while shrinking IF from 34.5% to 17.1%, and in our pipeline (Table R3) gap-only scheduling (w/o velocity) gains only +0.05. Multiplying initial-anchored gap by smoothed velocity vk(t)v_k(t) isolates domains with both large headroom and active progress.
  • Insight 2 — Prompt-level sampling (D³-MOPD) saves rollout and teacher-prefill compute that post-hoc loss reweighting (Open-MOPD / DWA) wastes: In OPD, over 75% of step time is spent on student rollout generation and teacher prefill ([2], Figure 8). Loss-reweighting methods keep prompt sampling static, generating rollouts and querying teachers on converged domains before down-weighting their gradients. Scheduling at the prompt loader redirects rollout generation to active domains, reaching Vanilla MOPD's peak in 3×3\times fewer rollout steps.

In summary, while prior published MOPD works [1] use a fixed domain mix and concurrent work Open-MOPD [2] reweights losses under a static prompt mix (now discussed in §1 and §5.2), D³-MOPD schedules prompt sampling directly via the composite KL~k⋅vk\tilde{\text{KL}}_k \cdot v_k signal. As recognized by Reviewers SyZk and hXpz, our observation on asynchronous domain convergence in MOPD is "unique and interesting" and the method is "simple, intuitive, and practically relevant".

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Author Response

Dear Area Chairs and Reviewers,

Thank you for your time and constructive feedback on our paper. As the discussion period is coming to a close, we briefly summarize the strengths highlighted in the reviews and how our response and new experiments address all reviewer questions:

1. Strengths Highlighted by the Reviewers

  • Motivation: Reviewers found our observation on asynchronous domain convergence in MOPD "unique and interesting" (Reviewer SyZk) and our scheduling approach "simple, intuitive, and practically relevant" (Reviewer hXpz).
  • Lightweight and Practical Design: All three reviewers valued that D³-MOPD reuses native reverse-KL signals without modifying the training loop (Reviewer hXpz, Reviewer SyZk), noting that it is "simple and cheap" and "needs no proxy or reference model... which is really desirable" (Reviewer Hfcy).
  • Presentation and Results: Reviewers noted our "clear figures and plots" (Reviewer SyZk) and "promising empirical results" (Reviewer hXpz).

2. Summary of Responses

  • Reliability and Scale Generalization (SyZk W2/Q2, Hfcy W1): In Table 1, 62.34 vs. 61.36 (+0.98) compared D³-MOPD at s95 against Vanilla MOPD at s143, which uses 50% more training steps. At the same training step (s95), D³-MOPD improves average accuracy by +2.23 from 60.11 to 62.34 across all 7 benchmarks and closes 73.3% vs. 48.5% of the student-teacher gap at a single checkpoint (Appendix C, Table 6). Without checkpoint selection, D³-MOPD wins all 17/17 evaluated checkpoints. Moreover, D³-MOPD generalizes effectively across model scales, improving all 7 benchmarks by +1.87 on average on a Qwen3.5-4B student (Appendix D, Table 7).
  • Extended Training to 320 Rollout Steps (SyZk W1/Q1): We clarified that 256 rollout steps correspond to 1,024 gradient updates (>2>2 epochs) and extended both methods to 320 rollout steps (1,280 gradient steps), where Vanilla MOPD degrades to 61.12 on over-exposed code data while D³-MOPD maintains a +1.00 lead at 62.12 and wins 6 out of 7 benchmarks.
  • Controlled Attribution of Loader and Jitter (Hfcy W2): Under the exact same Stratified Loader and η=0.30\eta=0.30 jitter across 5 shared checkpoints (Appendix G), our composite signal gains +1.15 to +1.24 over single-signal variants, and matches pool-and-shuffle during uniform warmup at s15 (60.10 vs. 59.92). Even without jitter (η=0\eta=0), D³-MOPD wins all 5/5 checkpoints with a +0.85 mean gain, while η=0.30\eta=0.30 analytically restores the ∼3.8%\sim 3.8\% natural binomial batch variation of pool-and-shuffle (§3.1).
  • New Baselines and Theoretical Analysis (hXpz W1–W4, Hfcy W3/Q1/Q2): We implemented both a Plateau-Detection Scheduler (61.53) and Sequential Training with Rehearsal (61.86), showing that D³-MOPD (62.34) outperforms both without prior knowledge of convergence order, and situated our work against ODM, DWA, DUMP, and concurrent work Open-MOPD (§1, §5.2). We also showed that R=3R=3 smoothing removes high-frequency spikes with a +1.36 mean gain over R=1R=1, clarified the two-stage recovery condition under KL rebound and the ϕ=0.2\phi=0.2 velocity floor (Appendix F & H.1), and explained why sk=KL~k⋅vks_k = \tilde{\text{KL}}_k \cdot v_k avoids permanent lock-in and 1/pk=10×1/p_k = 10\times mini-batch noise amplification (Appendix E).

We hope this summary is helpful, and we welcome any further feedback or questions from the reviewers before the discussion period ends.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.