- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes D³-MOPD, a dynamic domain scheduling method for multi-teacher on-policy distillation. The key observation is that different domains converge at different rates during joint training, making a fixed domain mixture potentially inefficient. To address this, the authors adaptively adjust the sampling ratio of each domain based on its remaining reverse-KL gap and descent velocity. The proposed method improves both final performance and training efficiency compared to vanilla MOPD.
Critical strengths
-
Well-motivated and intuitive approach. The paper identifies an important problem in multi-domain training: different domains converge at different rates, so maintaining a fixed sampling ratio may waste computation on already-converged domains. The proposed approach is simple, intuitive, and practically relevant.
-
Practical and lightweight implementation. The scheduler reuses the per-domain reverse-KL signals already available during training without modifying the core optimization procedure or introducing substantial computational overhead.
-
Promising empirical results. The method consistently improves over vanilla MOPD across multiple benchmarks, and the ablation studies demonstrate the contributions of the remaining gap, descent velocity, and smoothing mechanisms.
Critical weaknesses
1. Repeated oscillations in domain sampling ratios (Figure 8).
Figure 8 shows that some domains experience repeated increases and decreases in their sampling ratios, rather than a simple transition from high allocation to low allocation as they converge. Appendix F partially explains this behavior through KL rebound, but the underlying cause remains unclear.
Are these oscillations primarily caused by interference or forgetting between domains, or by instability in the estimated descent velocity? It would be useful to analyze whether these repeated reallocations are necessary for maintaining performance or whether a more stable scheduling strategy could achieve comparable results.
2. Missing comparisons with sequential or curriculum-based domain training.
The primary baseline is vanilla MOPD with a fixed domain mixture. However, the central motivation of this paper is that different domains converge at different rates.
A natural alternative would be to train domains sequentially until convergence before switching to another domain. Another possibility is alternating domain training or sequential training with rehearsal to mitigate catastrophic forgetting.
Since the paper already discusses alternating domain training in the related work, comparing against such strategies would help clarify whether the improvements come specifically from the proposed adaptive scheduling mechanism or more generally from reallocating training budgets across domains.
3. Robustness to non-monotonic convergence and KL rebound.
The proposed descent velocity is defined as
When a domain's reverse-KL increases, its estimated descent velocity becomes zero, causing its composite score to vanish regardless of the remaining gap. However, training dynamics are not necessarily monotonic, and temporary increases in KL do not always indicate that further training would be unproductive.
I appreciate that Appendix H.1 acknowledges this failure mode and introduces an optional velocity floor. Nevertheless, this raises a question about the robustness of the default scheduling rule. In particular, the claim that the scheduler is self-correcting appears to require further clarification: an increase in the remaining KL gap alone cannot restore the sampling priority when the descent velocity remains zero.
Could the authors clarify under what conditions a domain recovers from such a state, and whether the velocity-floor extension provides more reliable performance across different training settings?
4. Limited comparisons against simple adaptive scheduling baselines.
Although the ablation studies are useful, the experimental comparisons are largely limited to fixed-mixture MOPD and variants of the proposed method.
Since the main contribution is an adaptive domain allocation strategy, it would be valuable to compare against simpler alternatives, such as a plateau-detection scheduler that reduces a domain's sampling ratio once its KL stops improving, or other heuristic curriculum-based scheduling strategies.
These comparisons would help establish whether the specific gap-velocity formulation provides benefits beyond simpler adaptive allocation rules.
Questions for authors
see weakness
6 comments
Response to Reviewer hXpz (Part 2/3, W3)
W3. Robustness to non-monotonic convergence and KL rebound... an increase in the remaining KL gap alone cannot restore the sampling priority when... whether the velocity-floor extension provides more reliable performance across different training settings?
Answer W3:
Thank you for this careful observation. While an increase in the remaining gap alone does not make when , our scheduler still recovers priority through a two-stage mechanism under the default rule (), which we have clarified in Appendix F:
1) Two-stage recovery condition under the default rule ():
- Stage 1 — Baseline exposure via Softmax + Floor (
Eq. 6): Because still contributes to the softmax numerator on top of the floor , a zero-score domain always receives at least (). - Stage 2 — Priority restoration once : As dominant domains saturate, this continual exposure pulls downward again. Once the -window average KL change turns negative (), it multiplies the accumulated gap to restore priority (
Figure 7c, s180).
2) Reliability of the velocity-floor extension (, Appendix H.1, Table 8):
If cross-domain interference is strong enough that exposure cannot turn the KL slope negative, recovery under is delayed until other domains plateau. With the velocity floor (Eq. 9), a rebounding domain retains a positive fallback score even when . In Table R2, setting keeps IF's allocation within and cuts the IF KL rebound in half (Figure 9b) while improving both IF benchmarks.
Table R2: Velocity-floor robustness comparison on Qwen3.6-35B-A3B (Appendix H.1, Table 8).
| Method (Checkpoint) | AIME25 | HMMT(N) | IFBench | IFEval | LCB-v6 | OJB-C++ | BFCL(B) | Average |
|---|---|---|---|---|---|---|---|---|
| Vanilla MOPD (s143 peak) | 73.2 | 71.0 | 45.7 | 91.0 | 61.0 | 27.6 | 60.0 | 61.4 |
| D³-MOPD Default (, s95) | 74.0 | 71.4 | 45.1 | 91.2 | 62.5 | 29.7 | 62.5 | 62.3 |
| D³-MOPD w/ Velocity Floor (, s95) | 74.0 | 71.3 | 45.9 | 91.5 | 62.7 | 29.3 | 61.0 | 62.2 |
Thus, the default rule recovers whenever baseline exposure reverses the KL rebound, while provides immediate gap-proportional recovery under strong cross-domain interference.
Response to Reviewer hXpz (Part 3/3, W2 & W4)
W2 & W4. Missing comparisons with sequential or curriculum-based domain training (e.g., sequential training with rehearsal)... / Limited comparisons against simple adaptive scheduling baselines, such as a plateau-detection scheduler...
Answer W2 & W4:
Following your suggestion, we compare D³-MOPD with a Plateau-Detection Scheduler, Sequential Training with Rehearsal, and single-signal adaptive baselines on our 4-domain Qwen3.6-35B-A3B setup in Table R3:
Table R3: Comparison with Plateau-Detection Scheduler, Sequential Training with Rehearsal, and single-signal adaptive baselines on Qwen3.6-35B-A3B.
| Method (Checkpoint) | AIME25 | HMMT(N) | IFBench | IFEval | LCB-v6 | OJB-C++ | BFCL(B) | Average |
|---|---|---|---|---|---|---|---|---|
| Vanilla MOPD (s143) | 73.17 | 71.04 | 45.74 | 91.04 | 60.95 | 27.59 | 60.00 | 61.36 |
| w/o velocity (Gap-only, s127) | 72.66 | 71.15 | 45.17 | 90.91 | 59.52 | 28.45 | 62.00 | 61.41 (+0.05) |
| w/o gap (Velocity-only, s255) | 71.72 | 71.46 | 49.01 | 90.83 | 60.10 | 24.57 | 62.50 | 61.46 (+0.10) |
| Plateau-Detection Scheduler (s127) | 72.40 | 69.90 | 47.78 | 91.64 | 59.81 | 27.16 | 62.00 | 61.53 (+0.17) |
| Sequential Training with Rehearsal (s127) | 72.66 | 70.62 | 48.98 | 92.14 | 59.43 | 26.72 | 62.50 | 61.86 (+0.50) |
| D³-MOPD (Ours, s95) | 74.01 | 71.35 | 45.07 | 91.16 | 62.52 | 29.74 | 62.50 | 62.34 (+0.98) |
Analysis of Table R3:
- Plateau-Detection Scheduler (61.53) cuts early-converging domains too abruptly: Once a domain's KL stops improving (), its sampling ratio drops to the floor () and the remaining budget is split equally among unconverged domains. Because Code and Math plateau earliest, this step-down shifts almost all subsequent training to IF and Tool-use, improving IFBench and IFEval but degrading earlier code and math benchmarks (LCB-v6, OJB-C++, and AIME25).
- Sequential Training with Rehearsal (61.86) exhibits the same early-domain trade-off and requires prior knowledge of convergence order: Using prior knowledge of convergence speed, this baseline trains domains sequentially (Code Math Tool-use IF, 32 steps per stage over 128 steps) with an rehearsal floor for earlier domains, resulting in the same trade-off between late-stage IF gains and degradation on earlier code and math domains.
- Single-signal adaptive rules (61.41 and 61.46) fail to separate stalled high-gap domains from actively improving ones: Gap-only scheduling (w/o velocity) continues to allocate budget to high-gap plateaus, while velocity-only scheduling (w/o gap) oscillates across low-gap converged domains.
Therefore, by weighting domains continuously via without prior knowledge of convergence order, D³-MOPD balances active learning against retention and outperforms both single-signal and sequential/plateau schedules.
Overall, our trajectory analysis and velocity-floor results confirm that D³-MOPD is stable under non-monotonic KL dynamics, and our comparisons against Vanilla MOPD (+2.23 at matched s95), Plateau-Detection (+0.81), and Sequential Training with Rehearsal (+0.48) verify that the composite gap velocity formulation is effective. We have refined §3.2, §4.4, Appendix F, and Appendix G to incorporate these clarifications. We hope these results address your concerns, and we would be very grateful if you would consider raising your score.
Response to Reviewer hXpz (Part 1/3, W1)
We thank Reviewer hXpz for the positive feedback on our motivation, design, and empirical results. We address your four points below with multi-checkpoint trajectory analysis, recovery conditions under KL rebound, and experimental comparisons against plateau-detection and sequential-rehearsal baselines.
Answer W1:
Comparing the trajectories in
Figures 6–9with the multi-checkpoint ablation results inTable R1(Appendix G) andTable R2(Appendix H.1, Table 8) clarifies the sources of these oscillations and shows that a smoother schedule achieves comparable accuracy:Figure 6): Because 1 rollout step equals 4 gradient updates over ∼4,000 prompts per domain, most domains converge within the first 100 rollout steps (∼1 epoch), where Code decreases monotonically, Math rises smoothly, and IF transitions cleanly (Figure 6). The oscillations in steps 100–256 ofFigures 6 and 8appear because training continues well past saturation (>2 epochs), where small residual KL fluctuations across converged domains take turns shifting the ratios.Table R1): InFigure 8(d), estimating velocity from a single W=10 window (R=1) causes frequent short-term spikes in Math and Code. Across the 5 shared checkpoints inTable R1, unsmoothed R=1 exhibits a 4.7× larger standard deviation (0.85 vs. 0.18), whereas averaging over R=3 windows (Figure 7e) filters out these spikes and raises the 5-checkpoint mean by +1.36 (62.13 vs. 60.77).Table R2): InFigure 7(c, e), cross-domain updates during steps 60–150 cause IF's normalized KL to rebound, clipping vIF(t) to 0 and holding IF near the floor until s180. Enabling the velocity floor (ϕ=0.2,Figure 9candTable R2below) eliminates this sharp swing and achieves comparable average accuracy (62.2 vs. 62.3) while improving IFBench (+0.8) and IFEval (+0.3).Table R1: Multi-checkpoint trajectory comparison at the 5 shared ablation checkpoints (
Appendix G).Table 1)Thus, repeated oscillations are not required, and the performance gain comes from shifting budget away from saturated domains.