- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work studies how multiple teacher models that share a base reference model but undergo domain-specific SFT or RL can be consolidated into one single student model via Multi-teacher On-Policy Distillation (MOPD). The authors find that student initialization heavily impacts the subsequent recovery of capabilities and benchmark performance at initialization cannot reliably predict the final performance after MOPD. To address this issue the paper proposes Iterative Merging for MOPD. It first uniformly merges teacher task vectors to initialize the student. During MOPD it then regularly evaluates normalized recovery across all domains and keeps adding teacher task vectors at a fixed scale for domains with insufficient recovery. The authors run experiments on Qwen3-1.7B and Qwen3-4B across five domains. It outperforms vanilla MOPD, SFT warm-up and uniform merge initialization.
Critical strengths
- The paper identifies a key distinction between initialization performance and downstream learnability under MOPD. Initial benchmark scores may not adequately capture whether a model can keep benefiting from teacher supervision. This observation carries practical implications for understanding and designing on-policy distillation pipelines.
- The method is simple and delivers substantial empirical gains across two model sizes. Table 2 further shows that iterative application works better than adding all vectors at initialization or only at the end of training. This suggests the performance gain does not stem purely from the final parameter shift.
Critical weaknesses
- The core contribution of adaptive domain selection lacks thorough validation. Table 2 only demonstrates that staged addition of task vectors beats adding all vectors at initialization or at the end. It does not isolate the effect of dynamically selecting domains based on validation set recovery. Current results cannot tell whether improvements come from adaptive decisions or mainly from repeatedly injecting specific task vectors and continuing training after each adjustment.
- The experimental baselines are not sufficiently fair. The static merge baseline in the main table only uses uniform merge with . The appendix of this paper already shows that non-uniform fixed merging with larger scales yields much stronger results.
- Tool Use serves as a major source of overall gains. Appendix A.5 acknowledges that 20 out of 114 test task IDs carry upstream training overlap labels. Tool validation samples from the full dataset. It remains unclear whether training, validation and test sets are strictly disjoint.
- The paper also does not evaluate whether the strong parameter extrapolation at the final hurts general capabilities.
Questions for authors
- How are , , , and selected?
- Are the 128 validation tasks for Tool Use strictly disjoint by task ID from the final 114 test tasks? What are the results for Table 1 and Table 2 after removing the 20 upstream overlap tasks mentioned in Appendix A.5?
- Would non-adaptive strategies such as always selecting Tool, random domain selection and round-robin domain selection still produce worse results than IM-MOPD? This would verify the effectiveness of recovery-driven adaptive selection.
- The final cumulative merge scale reaches . Does this model show obvious degradation in general capabilities and non-target domains?
4 comments
Dear Reviewer pwf4,
Thank you again for your thoughtful review. As the discussion period closes on Nov 18, we would like to check whether our response has addressed your concerns.
In brief, we clarified that the Tool Use training, validation, and test tasks are strictly disjoint by task ID and that γ and δ were selected on validation data (App. A.2 and A.4; revisions are highlighted in orange). We also showed that IM-MOPD remains comparable to the baselines on general benchmarks despite its larger cumulative merge scale (Table R6). Moreover, it outperforms MOPD initialized from stronger fixed merges, including TIES, DARE-TIES (Table R5), and its own final coefficients applied at initialization (Table 2). Regarding adaptive selection, the domains to correct are identified by the recovery rule during training rather than predefined, and they include Medical, Law, and IF in addition to Tool Use (Table 7), corrections that an always-Tool schedule would miss.
If anything remains unclear, we would be happy to clarify it further in the remaining days. If our response has addressed your concerns, we would sincerely appreciate it if you could reflect this in your evaluation.
Best regards,
The Authors
reply to author
The authors have addressed most of my concerns, so I have decided to increase my rating.
Thank you for your positive assessment.
This review process has helped us further improve the quality of our manuscript.
We thank the reviewer for thoughtful questions. In particular, Q4 helped us further validate the robustness of our method in general domains. Before reading our responses below, we kindly ask to refer to our general response.
[W1, W3, Q2] Validation-based selection, Tool Use gains, and train-set overlap.
The wording in Appendices A.4 and A.5 was unclear; the training, validation, and test sets for Tool Use are strictly disjoint by task ID. Training vs. test. The "upstream training overlap" in Appendix A.5 refers to the overlap between the public AReaL-tau2-data and the test tasks. We removed this overlap before training, as stated in Appendix A.1, and the MOPD prompt pool was built in the same way. None of the 114 test tasks therefore appears in our training data. All experiments, including those added for this rebuttal, use the same data. Validation vs. test. The 128 validation tasks are strictly disjoint by task ID from the 114 test tasks. They were sampled from the τ2-Telecom task pool after excluding all 114 test tasks and every task that appears in our training data, so they are also disjoint from the training data. They are used only for IM-MOPD merge decisions, never for reported scores. In conclusion, we removed all overlap with the test tasks and constructed the training, validation, and test sets so that they do not overlap.
[W2] Baseline fairness and non-uniform merging.
We believe there may be some misunderstanding in our problem settings and contributions. We kindly request for checking the general comment before considering the remaining answer.
As the reviewer notes, our paper already reports fixed merges stronger than the uniform one, with different ratios and scales (Figure 3-(c), Table 8) and with IM-MOPD's final coefficients applied at initialization (Table 2). We note that the non-uniform coefficients in Table 2 were themselves discovered by IM-MOPD within a single run; they were not known before running IM-MOPD. IM-MOPD still outperforms all of these fixed merges, as well as MOPD initialized from the stronger merging methods TIES and DARE-TIES that we added in this rebuttal (Table R5). A natural follow-up question is whether a further search could find an even better fixed ratio. Such a search, however, is hard to carry out. Above all, the search space grows exponentially with the number of domains, as the ratio of every domain and the overall scale must be chosen together. Moreover, a candidate cannot be judged by its initial performance (Figure 1); its quality becomes visible only after MOPD. Finding the best ratios would therefore require running MOPD to completion for every candidate in this exponentially large space, which is infeasible in practice.
[W4, Q4] Impact of parameter extrapolation on general and non-target capabilities.
We evaluated general capabilities beyond the target domains on GPQA-Diamond and MMLU-Pro (Table R.6). IM-MOPD achieves comparable scores on both benchmarks: 49.0 on GPQA-Diamond and 66.2 on MMLU-Pro, compared with 50.9 and 67.8 for base MOPD and 48.1 and 67.2 for SFT warm-up + MOPD. These results suggest limited additional degradation beyond the target domains despite the cumulative merge scale. We will add these results to the appendix.
Table R.6 General benchmark evaluation results on Qwen3-4B-OT3.
[Q1] Regarding the hyperparameter selection.
We clarify that both γ and δ were selected based on validation-set performance, not test benchmarks. Tables 3, 9, and 10 report test-set results for sensitivity analysis only and were not used for hyperparameter selection. We will clarify this selection protocol in the revised manuscript (updated version: line 736).
[Q3] Effectiveness of IM-MOPD vs. non-adaptive strategies.
Thank you for this suggestion. The repeated selection of Tool Use was discovered by IM-MOPD within a single run, not predefined. Thus, an always-Tool schedule assumes prior knowledge that IM-MOPD itself discovers. Moreover, the benefit does not come from Tool Use alone. MOPD alone leaves Medical and Law under-recovered (Table 1), and IM-MOPD corrected them together with IF at its first decision (Table 7), reaching 76.6 on MedQA and 70.9 on CaseHOLD versus 72.7 and 67.6 for Uniform Merge + MOPD; an always-Tool schedule would skip these corrections. The required corrections also depend on the setting: starting from TIES or DARE-TIES merges, the same rule corrected only Tool Use at updates 25 and 50 and nothing at update 75 (Table R5), which no fixed schedule such as always-Tool or round-robin could anticipate. We are currently running the random-selection control and will add the results as soon as it completes.