acceptodds
Under review as a conference paper at ICLR 2027

No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation

Abstract

Multi-teacher on-policy distillation (MOPD) combines independently developed domain teachers into a single student by distilling their predictions on student-generated samples. We study a setting where teachers share a reference model but undergo different post-training procedures, and find that MOPD can struggle to recover some teacher capabilities. Because distillation occurs on student-generated prefixes, the student initialization can strongly affect subsequent recovery. However, initial benchmark performance is not a reliable predictor of a good MOPD initialization. For example, merge initialization can start below SFT warm-up yet finish higher after MOPD. We further find that effective merging depends on both the relative teacher contributions and the overall merge scale, with some strong configurations lying outside the simplex of convex parameter averaging. Thus, selecting a good merge initialization requires evaluating not only its immediate performance but also the learning it enables under MOPD, making one-shot coefficient search difficult. We propose Iterative Merging for MOPD (IM-MOPD), which starts from a uniform merge and progressively adds task-vector increments for under-recovered domains during distillation. In a 5-domain setting, IM-MOPD achieves higher average normalized recovery than MOPD with either uniform merge initialization or SFT warm-up, showing that effective teacher contributions can be determined progressively during training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-teacher on-policy distillation, large language models, alignment

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_pwf4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work studies how multiple teacher models that share a base reference model but undergo domain-specific SFT or RL can be consolidated into one single student model via Multi-teacher On-Policy Distillation (MOPD). The authors find that student initialization heavily impacts the subsequent recovery of capabilities and benchmark performance at initialization cannot reliably predict the final performance after MOPD. To address this issue the paper proposes Iterative Merging for MOPD. It first uniformly merges teacher task vectors to initialize the student. During MOPD it then regularly evaluates normalized recovery across all domains and keeps adding teacher task vectors at a fixed scale for domains with insufficient recovery. The authors run experiments on Qwen3-1.7B and Qwen3-4B across five domains. It outperforms vanilla MOPD, SFT warm-up and uniform merge initialization.

Critical strengths
  1. The paper identifies a key distinction between initialization performance and downstream learnability under MOPD. Initial benchmark scores may not adequately capture whether a model can keep benefiting from teacher supervision. This observation carries practical implications for understanding and designing on-policy distillation pipelines.
  2. The method is simple and delivers substantial empirical gains across two model sizes. Table 2 further shows that iterative application works better than adding all vectors at initialization or only at the end of training. This suggests the performance gain does not stem purely from the final parameter shift.
Critical weaknesses
  1. The core contribution of adaptive domain selection lacks thorough validation. Table 2 only demonstrates that staged addition of task vectors beats adding all vectors at initialization or at the end. It does not isolate the effect of dynamically selecting domains based on validation set recovery. Current results cannot tell whether improvements come from adaptive decisions or mainly from repeatedly injecting specific task vectors and continuing training after each adjustment.
  2. The experimental baselines are not sufficiently fair. The static merge baseline in the main table only uses uniform merge with λ=1\lambda=1. The appendix of this paper already shows that non-uniform fixed merging with larger scales yields much stronger results.
  3. Tool Use serves as a major source of overall gains. Appendix A.5 acknowledges that 20 out of 114 test task IDs carry upstream training overlap labels. Tool validation samples from the full dataset. It remains unclear whether training, validation and test sets are strictly disjoint.
  4. The paper also does not evaluate whether the strong parameter extrapolation at the final λ=2.8\lambda=2.8 hurts general capabilities.
Questions for authors
  1. How are γ\gamma, δ\delta, HH, and λ\lambda selected?
  2. Are the 128 validation tasks for Tool Use strictly disjoint by task ID from the final 114 test tasks? What are the results for Table 1 and Table 2 after removing the 20 upstream overlap tasks mentioned in Appendix A.5?
  3. Would non-adaptive strategies such as always selecting Tool, random domain selection and round-robin domain selection still produce worse results than IM-MOPD? This would verify the effectiveness of recovery-driven adaptive selection.
  4. The final cumulative merge scale reaches λ=2.8\lambda=2.8. Does this model show obvious degradation in general capabilities and non-target domains?
4 comments

We thank the reviewer for thoughtful questions. In particular, Q4 helped us further validate the robustness of our method in general domains. Before reading our responses below, we kindly ask to refer to our general response.

[W1, W3, Q2] Validation-based selection, Tool Use gains, and train-set overlap.

The wording in Appendices A.4 and A.5 was unclear; the training, validation, and test sets for Tool Use are strictly disjoint by task ID. Training vs. test. The "upstream training overlap" in Appendix A.5 refers to the overlap between the public AReaL-tau2-data and the test tasks. We removed this overlap before training, as stated in Appendix A.1, and the MOPD prompt pool was built in the same way. None of the 114 test tasks therefore appears in our training data. All experiments, including those added for this rebuttal, use the same data. Validation vs. test. The 128 validation tasks are strictly disjoint by task ID from the 114 test tasks. They were sampled from the τ2\tau^2-Telecom task pool after excluding all 114 test tasks and every task that appears in our training data, so they are also disjoint from the training data. They are used only for IM-MOPD merge decisions, never for reported scores. In conclusion, we removed all overlap with the test tasks and constructed the training, validation, and test sets so that they do not overlap.

[W2] Baseline fairness and non-uniform merging.

We believe there may be some misunderstanding in our problem settings and contributions. We kindly request for checking the general comment before considering the remaining answer.

As the reviewer notes, our paper already reports fixed merges stronger than the uniform one, with different ratios and scales (Figure 3-(c), Table 8) and with IM-MOPD's final coefficients applied at initialization (Table 2). We note that the non-uniform coefficients in Table 2 were themselves discovered by IM-MOPD within a single run; they were not known before running IM-MOPD. IM-MOPD still outperforms all of these fixed merges, as well as MOPD initialized from the stronger merging methods TIES and DARE-TIES that we added in this rebuttal (Table R5). A natural follow-up question is whether a further search could find an even better fixed ratio. Such a search, however, is hard to carry out. Above all, the search space grows exponentially with the number of domains, as the ratio of every domain and the overall scale must be chosen together. Moreover, a candidate cannot be judged by its initial performance (Figure 1); its quality becomes visible only after MOPD. Finding the best ratios would therefore require running MOPD to completion for every candidate in this exponentially large space, which is infeasible in practice.

[W4, Q4] Impact of parameter extrapolation on general and non-target capabilities.

We evaluated general capabilities beyond the target domains on GPQA-Diamond and MMLU-Pro (Table R.6). IM-MOPD achieves comparable scores on both benchmarks: 49.0 on GPQA-Diamond and 66.2 on MMLU-Pro, compared with 50.9 and 67.8 for base MOPD and 48.1 and 67.2 for SFT warm-up + MOPD. These results suggest limited additional degradation beyond the target domains despite the cumulative merge scale. We will add these results to the appendix.

Table R.6 General benchmark evaluation results on Qwen3-4B-OT3.

ModelGPQA-DiamondMMLU-Pro
base MOPD50.967.8
SFT warm-up + MOPD48.167.2
IM-MOPD (ours)49.066.2

[Q1] Regarding the hyperparameter selection.

We clarify that both γ\gamma and δ\delta were selected based on validation-set performance, not test benchmarks. Tables 3, 9, and 10 report test-set results for sensitivity analysis only and were not used for hyperparameter selection. We will clarify this selection protocol in the revised manuscript (updated version: line 736).

[Q3] Effectiveness of IM-MOPD vs. non-adaptive strategies.

Thank you for this suggestion. The repeated selection of Tool Use was discovered by IM-MOPD within a single run, not predefined. Thus, an always-Tool schedule assumes prior knowledge that IM-MOPD itself discovers. Moreover, the benefit does not come from Tool Use alone. MOPD alone leaves Medical and Law under-recovered (Table 1), and IM-MOPD corrected them together with IF at its first decision (Table 7), reaching 76.6 on MedQA and 70.9 on CaseHOLD versus 72.7 and 67.6 for Uniform Merge + MOPD; an always-Tool schedule would skip these corrections. The required corrections also depend on the setting: starting from TIES or DARE-TIES merges, the same rule corrected only Tool Use at updates 25 and 50 and nothing at update 75 (Table R5), which no fixed schedule such as always-Tool or round-robin could anticipate. We are currently running the random-selection control and will add the results as soon as it completes.

Dear Reviewer pwf4,

Thank you again for your thoughtful review. As the discussion period closes on Nov 18, we would like to check whether our response has addressed your concerns.

In brief, we clarified that the Tool Use training, validation, and test tasks are strictly disjoint by task ID and that γ and δ were selected on validation data (App. A.2 and A.4; revisions are highlighted in orange). We also showed that IM-MOPD remains comparable to the baselines on general benchmarks despite its larger cumulative merge scale (Table R6). Moreover, it outperforms MOPD initialized from stronger fixed merges, including TIES, DARE-TIES (Table R5), and its own final coefficients applied at initialization (Table 2). Regarding adaptive selection, the domains to correct are identified by the recovery rule during training rather than predefined, and they include Medical, Law, and IF in addition to Tool Use (Table 7), corrections that an always-Tool schedule would miss.

If anything remains unclear, we would be happy to clarify it further in the remaining days. If our response has addressed your concerns, we would sincerely appreciate it if you could reflect this in your evaluation.

Best regards,

The Authors

Reviewer_pwf4OpenReview ↗

reply to author

The authors have addressed most of my concerns, so I have decided to increase my rating.

Thank you for your positive assessment.

This review process has helped us further improve the quality of our manuscript.

Reviewer_rBESOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the "Multi-teacher On-policy Distillation" (MOPD) method. In this setting, domain-specific teacher models share a common reference checkpoint but undergo different subsequent training processes—Supervised Fine-Tuning (SFT) for the medical, legal, and tool-use domains, and Reinforcement Learning from Verifiable Rewards (RLVR) for the finance and instruction-following domains. The study finds that while MOPD effectively distills knowledge from teacher models trained via reinforcement learning (RL), it performs less effectively when applied to teacher models trained via SFT. Furthermore, initializing the student model using "task-vector merging" helps improve distillation outcomes. This approach is grounded in two key observations: first, the initial performance on benchmarks is not a reliable predictor of the final performance after MOPD (for instance, initialization via SFT "warm-up" starts from a higher baseline but ultimately yields poorer results); second, effective model merging depends not only on the relative weights assigned to each domain but also on a global scaling factor, λ=∑iαi\lambda = \sum_i \alpha_i (where λ\lambda may occasionally exceed 1).

Critical strengths
  1. The paper offers a valuable insight into achieving an effective initialization for MOPD (Multi-task Mixture-of-Experts Model Merging). It explicitly notes that a model’s performance on standalone benchmarks is a poor predictor of its learning outcomes within the MOPD framework. As shown in Table 8, an SFT (Supervised Fine-Tuning) warm-up strategy yields an initial τ2\tau^2 metric of 30.7, whereas a merging strategy with λ=2.1\lambda=2.1 starts at 13.2; yet, the final results are 31.6 and 50.9, respectively. Figure 3a demonstrates that this same pattern holds when merging occurs midway through training. Furthermore, Table 11 (a dual-domain grid experiment) indicates that an initialization strategy deemed "best" based on initial scores does not necessarily yield the optimal performance after MOPD is applied. A diagnostic analysis of "teacher-continuation" (Figure 5) provides a plausible explanation: prefixes generated by the merged student model enable the teacher model to perform subsequent generation more effectively—even when the student model’s own accuracy is comparable to or lower than that of the SFT warm-up baseline. This is the most thought-provoking aspect of the paper and serves as a highly useful reference for any researcher attempting to integrate independently trained expert models via OPD (Mixture-of-Experts Model Merging).

  2. The paper introduces a simple yet highly effective method that incorporates well-conceived temporal control. IM-MOPD eliminates the need for a mixed-domain SFT phase or access to the teacher model’s training data, requiring only a few basic "task-vector" addition operations. At the 4B parameter scale, this approach delivers significant performance gains—gains that are not merely attributable to enhanced "Tool Use" capabilities. According to Table 1, when considering only the average performance across four non-tool-related domains, IM-MOPD improves results from 65.0 (using a uniform merging strategy) to 82.7. Table 2 provides an excellent controlled experiment: applying the final cumulative factor directly to the initialization phase yields a result of 79.0, whereas employing a dynamic schedule achieves 86.9. This demonstrates that the sequencing of merge operations is crucial; the performance gain is not solely attributable to the final merge factor.

Critical weaknesses
  1. The core metrics presented in the paper are based on single-run results, and hyperparameters were selected using the test set. Furthermore, the benchmark datasets employed are small and highly noisy, yet they exert a disproportionate influence on the calculated average scores. According to the normalization method described in the paper, "Tool Use" accounts for approximately 57% (for the 4B model) and 79% (for the 1.7B model) of the performance gain achieved by IM-MOPD over "Uniform Merge + MOPD." The evaluation of "Tool Use" capabilities is based on 114 τ2\tau^2-Telecom tasks, using sampling with a temperature of 1 and the pass@1 metric; this implies a standard error of approximately 4.3 points for an individual score. For the 1.7B model, the normalization denominator is only 20.2 points (28.1 points for the teacher model versus 7.9 points for the baseline). Consequently, the "Tool Use" component alone introduces approximately ±4\pm 4 points of noise into the average score across the five domains—with the 28.1-point score for IM-MOPD derived from only 32 of the 114 tasks. If the average is calculated excluding the "Tool Use" domain, the performance gain of the 1.7B model over "Uniform Merge" drops from +29.7 to +7.9 (increasing from 62.3 to 70.2). Notably, the score for FinQA decreases from 58.6 to 55.9, yet the paper characterizes this as "roughly comparable." Moreover, the paper does not report the variance resulting from different random seeds. Additionally, the γ\gamma and δ\delta parameters appear to have been selected directly on the test benchmarks; Table 3 presents these test results, and the paper’s primary conclusions are based on the parameter combinations that yielded the best performance for each model size (specifically, γ=0.6\gamma=0.6 for the 4B model and γ=0.4\gamma=0.4 for the 1.7B model). In the parameter sweep for the 1.7B model, the results exhibit non-monotonic behavior as a function of δ\delta (62.2 / 57.2 / 76.0), which is consistent with the presence of stochastic noise across multiple runs. Furthermore, it is worth noting that 20 out of the 114 "telecom" tasks were identified as having an overlap with the upstream training data (see Appendix A.5). I do not believe this "overlap" can account for the observed performance gaps, as it affects all methods—including the "teacher anchor"—equally; moreover, verifying this by constructing a "clean subset" would be computationally inexpensive. Taken together, these issues render the experimental results at the 4B scale quite compelling, while those at the 1.7B scale are less convincing; this also limits the generalizability of the study’s core conclusions.

  2. The contribution described as "validation-based adaptive selection" does not function independently; in terms of the actual scheduling scheme employed, it effectively reduces to a set of fixed rules. Table 2 demonstrates that the use of "interleaving merges" in conjunction with MOPD is indeed effective, yet it fails to establish that "validation-based selection" plays a critical role in this process. Table 7 reveals that across all nine configurations, the "Tool Use" domain is consistently selected at every merge point, whereas other domains show less stability; in the primary 4B configuration, these other domains are selected only at the 25th update. In practice, the behavior of IM-MOPD is more akin to: "performing a uniform merge—adding each task vector once—followed by the continuous addition of the 'tool-use' vector." In fact, employing a static merging strategy with a higher coefficient assigned to the "tool-use" vector yields comparable results (see Table 8: with λ=2.1\lambda=2.1 and αtool=0.8\alpha_{\text{tool}}=0.8, the τ2\tau^2 metric reaches 50.9). The study lacks essential control experiments—such as non-adaptive scheduling (e.g., adding δτi\delta\tau_i to all domains at identical intervention steps and budget constraints, or applying a pre-defined, fixed schedule of additions, such as at the 25th, 50th, and 75th update iterations)—as well as a well-tuned, static, non-uniform merging strategy operating under the same budget. Without these controls, the paper’s claim that "teacher contributions are determined incrementally via recovery feedback" remains insufficiently supported, beyond merely demonstrating the significance of "timing." Furthermore, the paper fails to quantify the validation costs incurred by each decision (which entails 128–256 prompt tests per domain, including multiple rounds of simulated tool invocations).

Questions for authors

SEE WEAKNESS

9 comments

We appreciate the reviewer’s feedback and kindly encourage to review our general comment before proceeding to the detailed responses below.

[W1] Regarding experimental reliability and evaluation protocol.

Hyperparameter selection. We clarify that γ\gamma and δ\delta were selected using validation data, not test benchmarks. Tables 3, 9, and 10 report test-set sensitivity analyses and were not used for hyperparameter selection. We will make this distinction explicit in the revised manuscript.

Evaluation variance. Our original evaluation already uses avg@3 for all four non-Tool benchmarks (Lines 317–319). Due to the high API cost of τ2\tau^2-Telecom evaluation, we initially reported pass@1 results. To address the reviewer's concern, we have additionally conducted multiple independent τ2\tau^2 evaluations (updated version: Table 1).

Contribution of Tool Use. We agree that Tool Use contributes substantially to the normalized improvement. However, the gains are not exclusive to this domain. At 4B, average normalized recovery across the four non-Tool domains improves from 65.0 to 82.7 with IM-MOPD. We will make the domain-wise contributions more explicit.

Tool-use dataset validation selection. The 128 Tool Use validation tasks are strictly disjoint by task ID from the 114 test tasks and from our training data: we sampled them from the τ²-Telecom task pool after excluding all test tasks and every task used for training. They are used only for IM-MOPD's merge decisions, never for reported scores. We also removed the overlap between the public AReaL-tau2-data and the test tasks before training (Appendix A.1). The revised paper clarifies Appendices A.4 and A.5 accordingly (see also our response to Reviewer pwf4).

[W2] Regarding adaptive selection and fixed merging schedules.

We discuss it in our General Comment and summarize the main point here. We respectfully clarify that the observed correction schedule is an outcome of IM-MOPD, not a predefined rule: our method does not assume that Tool Use requires repeated corrections. At each decision, IM-MOPD measures how much of each teacher's capability the student has recovered on held-out validation data, and corrects only the domains that remain below the threshold. Tool Use was corrected repeatedly in our main experiment because it remained under-recovered after each correction, whereas Medical, Law, and IF were corrected once and no longer needed it afterwards.

Reviewer_rBESOpenReview ↗

Replying to authors

Question on Practical Applicability.

The proposed method requires all teacher models to share the same architecture and originate from a common reference checkpoint. However, many widely used models, such as GPT and Claude, are black-box systems with inaccessible parameters, while even open-weight models like Qwen and DeepSeek may have substantially different architectures. Given these constraints, could the authors clarify the practical application scenarios in which IM-MOPD would be most suitable, and discuss how its insights might extend to more heterogeneous or black-box teacher settings?

Thank you for your further question. We believe this question stems from a misunderstanding of the problem setting explicitly defined in our paper and how MOPD is used in practice.

1. Same-origin teachers are an explicit and practically relevant setting.

Our work does not aim to combine arbitrary foundation models. Rather, we study the integration of multiple domain teachers that originate from a shared reference model and are independently specialized through domain-specific post-training into a single student.

This is an established and practically relevant scenario, as demonstrated by the original MOPD paper [1], MiMo-V2-Flash [2], Nemotron 3 Ultra [3], Open-MOPD [4], and others [5,6,7].

These assumptions are also explicitly stated in Section 2, Lines 145–150: "the student policy … and all domain teachers share the same architecture and originate from a common reference checkpoint."

Thus, sharing the same architecture and reference checkpoint is an explicitly defined scope of our research and a prerequisite of the proposed method, rather than an overlooked practical consideration.

Extending the method to heterogeneous teachers is an interesting but separate research direction, as already discussed in our response to uB9n's [Q7].

2. Black-box teacher access is a limitation of OPD itself.

Regarding black-box teachers such as GPT and Claude, on-policy distillation requires the teacher's token-level log-probabilities on the student's own responses, which these models do not expose. They therefore cannot serve as teachers for OPD in general, including MOPD, so this is a limitation of the OPD setting itself rather than of our IM-MOPD.

References

[1] https://arxiv.org/abs/2606.30406

[2] https://arxiv.org/abs/2601.02780

[3] https://arxiv.org/abs/2606.15007

[4] https://arxiv.org/abs/2608.19098

[5] https://arxiv.org/abs/2610.02324

[6] https://arxiv.org/abs/2608.24987

[7] https://arxiv.org/abs/2609.23697

Reviewer_uB9nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies multi-teacher on-policy distillation (MOPD) where all teachers share a reference checkpoint (Qwen3-Base fine-tuned on OpenThoughts3) but are post-trained differently. Medical, Law and Tool Use teachers use SFT, Finance and IF teachers use RLVR. It observes that MOPD from the reference model fails to recover the SFT teachers, especially Tool Use. It reports that task-vector merge initialization helps, and that initial benchmark performance does not predict post-MOPD performance. It reports that effective merges can have total scale λ > 1. Proposed method (IM-MOPD): start from a uniform merge, every H = 25 MOPD updates, add δ·τ_i for each domain whose validation recovery is at or below a threshold γ.

Critical strengths
  1. The observation that initial performance does not predict post-MOPD quality is useful and well demonstrated. Fig. 1 and the branching experiment in Fig. 3a (continuing MOPD from the same checkpoint, with and without a merge intervention) show that a temporarily worse student can end up better after distillation.

  2. The setup is realistic and the method is simple, with large headline gains.

  3. Five heterogeneous domains with mixed SFT and RLVR teachers reflect real modular post-training workflows. The method is a four-line loop that needs no mixed-domain SFT stage, and the merge-timing ablation (Table 2) is the right kind of experiment for testing its core claim.

Critical weaknesses
  1. In my opinion, the novelty is limited relative to existing work. λ > 1 scaling, model extrapolation (e.g., ExPO), and online or iterative merging during training (e.g., online merging optimizers, WARP) already exist. Raw task vectors anchored at θ_ref are added to a student that MOPD has already moved, and the paper does not analyze the resulting interference.

  2. I noticed that most of the gain comes from better final coefficients, not from iterating. Applying IM-MOPD’s final cumulative coefficients (about Tool 1.1, Medical/Law/IF 0.5, Finance 0.2; λ ≈ 2.8) once at initialization already reaches 79.0, compared with 53.8 for uniform merge. So roughly 25 of the 33 points come from a larger, Tool-heavy merge, and only about 8 points come from the iterative schedule itself.

  3. Also it is shown in the paper that the headline result depends mainly on one small benchmark with possible contamination. τ²-Telecom accounts for about 19 of the 33 normalized points. It has only 114 tasks, is scored pass@1 from a single run with a GPT-4.1 user simulator, and includes 20 task IDs flagged for upstream training overlap.

  4. Also I would like to request the author to clear that why the validation data may overlap with the test set? Tool Use validation draws 128 tasks from the full dataset. If this is the τ²-Telecom pool, validation and test may overlap or be closely related. Tool Use is selected for correction at every intervention point, so it drives the merge decisions.

  5. Hyperparameters appear to have been chosen on test results and are unstable. γ differs by model size. The reported configurations are the best rows of sweeps that use test benchmarks. At 1.7B, δ = 0.1 / 0.2 / 0.3 gives 62.2 / 57.2 / 76.0, a non-monotonic 19-point spread that suggests high variance.

  6. Every result is from a single seed. I think it is better to report the average of at least three runs (if possible). The key iterative-vs-one-shot gap (79.0 vs 86.9) is a few points and comes mostly from about 14 τ² tasks plus IFBench, so its reliability cannot be judged without variance estimates.

  7. The baselines are weak or handicapped. SFT warm-up is deliberately stopped at about 30% normalized recovery. There are no validation-tuned task arithmetic, TIES, DARE or AdaMerging initializations, even though tuning λ above 1 is already standard task-arithmetic practice.

  8. The merging and MOPD reinforce each other claim rests on a weak control, and key controls are missing. Adding λ ≈ 2.8 of raw task vectors after MOPD predictably overshoots. There is no merge-only control with the final coefficients and zero MOPD steps, and no longer MOPD budget for any method (only 100 updates at lr 10⁻⁶).

Questions for authors
  1. I think it is better to report Tables 1 and 2 over at least 3 seeds? In particular, does the 79.0 vs 86.9 gap between one-shot and iterative merging (with identical final coefficients) hold?

  2. It is important to show: are the 128 Tool Use validation tasks drawn from the τ²-Telecom pool used for testing, and do they overlap with the 114 base-split test tasks? How do results change when the 20 overlap-flagged tasks are excluded?

  3. Were γ and δ chosen using the test benchmarks reported in Tables 3, 9 and 10? If so, please re-select them on validation data only.

  4. Also I would like to suggest to show how does IM-MOPD compare with one-shot task arithmetic whose ratios and λ are tuned on the same validation sets under an equal evaluation budget, and with TIES/DARE initializations? And also please show, how does the final IM-MOPD merge perform without any MOPD?

  5. How does SFT warm-up perform when trained to convergence rather than stopped at about 30% recovery? Do the rankings hold with a longer MOPD budget for all methods? Also please re-anchor task vectors to the current student (φ_i − θ_t) and compare with raw τ_i.

  6. Is it suggested to ablate the γ threshold (correcting all domains) and the correction interval H. And also better to evaluate IF consistently (IFBench vs IFEval) and report the compute cost of validation every 25 steps.

  7. Discuss model extrapolation and online merging literature, and why RL task vectors appear easy for MOPD to recover while SFT task vectors do not. I think this will also help increase the paper's expressiveness.

6 comments

We sincerely appreciate the reviewers' insightful feedback. In particular, the experiments suggested in Q5 further demonstrate that our method outperforms SFT warm-up, even when substantially more compute is devoted to the warm-up stage. However, we believe there may be some misunderstandings regarding our problem setting and contributions. We would appreciate it if you read our general response going through our detailed responses.

[W1] Novelty relative to extrapolation and iterative merging.

We agree that parameter extrapolation and iterative merging have been studied individually. However, our contribution lies not in these methods in isolation, but in identifying a distinct challenge in MOPD and proposing a method to improve the effectiveness of MOPD by addressing this challenge.

Unlike ExPO, which aims to improve after-merging performance through extrapolation, we show that better initial performance does not necessarily lead to better post-MOPD recovery (Figures 1 and 3-(a), Lines 222–230). Moreover, effective initialization depends on both merge ratios and global scale, making one-shot coefficient selection difficult. This motivates IM-MOPD, which progressively adapts teacher contributions during distillation. We have also included this paper in our revised manuscript (updated version: line 190).

While WARP also employs iterative merging, it targets the reward-KL trade-off in RLHF rather than uneven capability recovery from independently specialized domain teachers under MOPD.

Thus, the novelty of IM-MOPD lies in the problem-specific insights and the resulting recovery-guided integration of merging with MOPD, rather than in claiming either operation as new.

[W1, Q5] Regarding potential interference.

Re-anchoring task vectors to the current student would partially undo the updates accumulated during MOPD. Specifically, letting θt=θ0+ΔMOPD\theta_t = \theta_0 + \Delta_{\mathrm{MOPD}}, a re-anchored correction becomes

θt+δ(ϕi−θt)=θ0+(1−δ)ΔMOPD+δ(ϕi−θ0).\theta_t + \delta(\phi_i-\theta_t) = \theta_0 + (1-\delta)\Delta_{\mathrm{MOPD}} + \delta(\phi_i-\theta_0).

Thus, with δ=0.3\delta=0.3, re-anchoring explicitly attenuates 30% of the accumulated MOPD updates, potentially undermining previously acquired capabilities.

Our objective is not simply to interpolate between the student and teacher parameters, but to use teacher-specific task vectors to facilitate subsequent MOPD while preserving accumulated updates. As shown in Figure 3-(a), even when merging temporarily reduces performance, subsequent MOPD can achieve stronger recovery. Furthermore, Figure 4 shows that iterative corrections improve under-recovered domains while largely preserving already-recovered capabilities.

These results support our use of reference-anchored task vectors, which introduce teacher-specific corrections without explicitly attenuating accumulated MOPD updates.

[W2, W6] Regarding the contribution of final merge coefficients versus iterative merging.

Crucially, the effective final merge coefficients were themselves discovered by IM-MOPD within a single MOPD training run, rather than through a separate coefficient search. As discussed in Lines 233-241, the key challenge is not whether effective extrapolated coefficients exist, but how to identify them efficiently. Since initial merge performance does not reliably predict post-MOPD recovery (Figures 1 and 3-(a)), candidate coefficients must be assessed after MOPD. With KK teachers, this involves a continuous KK-dimensional search space which grows exponentially as number of domain increases.

IM-MOPD instead discovers effective teacher contributions within a single MOPD run. Applying the coefficients discovered by IM-MOPD entirely at initialization already achieves 79.0, compared with 53.8 for uniform merging followed by MOPD. Furthermore, interleaving the same cumulative coefficients with MOPD improves recovery to 86.9, demonstrating an additional 7.9-point benefit. Thus, both efficient coefficient discovery (+25.2 points) and iterative merging (+7.9 points) contribute to IM-MOPD's improvement.

[W3-W4, Q2] Tool-Use evaluation reliability and data disjointness.

We acknowledge the sentence in Appendices A.4 and A.5 was unclear. In short, the Tool Use validation tasks do not overlap with the test tasks, and no test task was used for training. Please see the details below.

Regarding Figure 1. The remaining domains were omitted from the figure for clarity, not due to selective reporting. Complete results across all five domains are provided in Table 1 and Appendix B.1’s Table 8.

Validation vs. test. As stated in Appendix A.4, the 128 validation tasks are drawn from the full τ2\tau^2-Telecom task pool. However, before sampling, we excluded all tasks used for training and all 114 test tasks, so the validation set is cleanly separated from both. Mentioning only the full pool in Appendix A.4 may have caused this confusion. The merge decisions, including those for Tool Use, therefore never use test tasks.

Data overlap concern. The public AReaL-tau2-data contains some overlap with the test tasks, but as stated in Appendix A.1, we removed all of this overlap before training, and the MOPD prompt pool was built in the same way.

Again, we removed all overlap with the test tasks and constructed the training, validation, and test sets so that they do not overlap.

In the revised paper, we remove the misleading sentence and clarify how the validation tasks are drawn and how the overlap was removed from the MOPD data (see updated version: line 710-712).

[W5, Q6] Regarding hyperparameter sensitivity and selection.

We emphasize that IM-MOPD consistently outperforms the baselines across all reported hyperparameter configurations (Tables 3, 9, and 10), demonstrating that its effectiveness does not depend on a narrowly tuned setting. More importantly, IM-MOPD replaces expensive search over a continuous, KK-dimensional merging coefficient space with a simple threshold-based decision rule. This requires only two global scalars, γ\gamma and δ\delta, shared across all domains. The correction interval H=25H=25 was fixed across all experiments by design, enabling small, frequent corrections rather than large, infrequent interventions. Unlike exhaustive coefficient search, the number of tunable parameters does not grow with the number of teachers. Finally, γ\gamma and δ\delta were selected using validation data, not test benchmarks. Tables 3, 9, and 10 report test-set results solely for sensitivity analysis. We clarified this in the revised manuscript (updated version: line 736).

[W6, Q1] Regarding statistical reliability and multiple seeds.

We clarify that our original evaluation already uses avg@3 for all four non-Tool benchmarks (Lines 317-319), rather than relying on a single generation. Due to the high API costs associated with τ2\tau^2-Telecom evaluation, we initially reported single-run results. To address the reviewer’s concern, we have conducted additional evaluations with multiple random seeds and provide the avg@3 result. (updated version: Table 1). Furthermore, we are also conducting multiple-seed MOPD training runs to assess training stability. We will include these results and standard deviations in the revised manuscript.

[W7, Q5] Regarding SFT warm-up convergence and computational cost.

To address this concern, we substantially extended SFT warm-up training, followed by the same MOPD procedure. In Table R.2, while extending SFT warm-up improves subsequent MOPD performance, even the 3× SFT warm-up underperforms IM-MOPD, despite requiring approximately 7.6× more compute. More importantly, we emphasize that our work focuses on improving MOPD given independently trained domain teachers. As already discussed in our paper (Lines 73-82), SFT warm-up requires generating additional training data from these already-trained teachers, followed by mixed-domain SFT. This stage reintroduces the challenges of data-mixture design, training-budget allocation, and checkpoint selection that modular teacher training aims to avoid. IM-MOPD instead directly leverages existing teacher checkpoints without additional data generation or mixed-domain SFT, achieving stronger post-MOPD recovery at substantially lower cost. Table R.4 Effect of SFT warm-up training budget on post-MOPD performance.

MethodCompute Cost (GPU-hours)Post-MOPD Norm. Score (%)
SFT Warm-up (1×) + MOPD2564
SFT Warm-up (2×) + MOPD4781
SFT Warm-up (3×) + MOPD6884
IM-MOPD (Ours)988

[W8, Q4] Regarding stronger merging baselines and merge-only controls.

We thank the reviewer for these suggestions. First, we have additionally evaluated merge-only controls using the final IM-MOPD coefficients, TIES, and DARE-TIES, without any MOPD updates. As shown in Table R5, merging alone recovers only a small part of the teachers' capabilities: before any MOPD update, the normalized score is 32.9 for the uniform merge, 46.6 for TIES, and 44.1 for DARE-TIES, far below the 85.0 of IM-MOPD. So the gains of IM-MOPD do not come from the merged parameters alone. We also clarify that strong non-uniform merging followed by MOPD is already investigated in our paper (Table 2 and Appendix B.1). In particular, applying IM-MOPD’s final coefficients entirely at initialization outperformed the uniform merge initialization baseline by 25.2 points. This demonstrates that IM-MOPD can discover effective non-convex coefficients within a single training run, without an expensive search across merge ratios and extrapolation scales. Importantly, IM-MOPD further improves recovery from 79.0 to 86.9 despite using the same cumulative coefficients. This additional gain highlights the benefit of interleaving merging with MOPD, beyond simply identifying better coefficients. Thus, our method provides two complementary benefits: efficiently identifying effective teacher contributions and exploiting their interaction with MOPD through iterative corrections. Finally, we also ran MOPD initialized with the TIES and DARE-TIES merges (Table R5). Although these merges start well above the uniform merge (46.6 and 44.1 vs. 32.9), MOPD from them reaches only 72.3 and 71.0, well below IM-MOPD started from the simple uniform merge (85.0). Also note that IM-MOPD is complementary to these merges: starting IM-MOPD from TIES or DARE-TIES gives 85.0 and 86.5.

Table R.5 Comparison of merging strategies with MOPD.

InitializationStageMedQACaseHOLDFinQAIFBenchτ²Norm.
Reference model–69.561.859.124.94.70.0
Domain teachers–80.373.375.059.471.9100.0
UniformMerge only72.366.569.832.89.932.9
Uniform+ MOPD73.667.674.957.313.258.8
Uniform+ IM-MOPD76.071.773.158.967.085.0
TIESMerge only75.769.268.741.17.346.6
TIES+ MOPD78.971.073.455.316.172.3
TIES+ IM-MOPD77.671.373.758.657.685.0
DARE-TIESMerge only73.669.869.041.37.344.1
DARE-TIES+ MOPD76.870.774.158.816.471.0
DARE-TIES+ IM-MOPD76.370.773.257.378.186.5

[Q3] Hyperparameter selection using validation data.

We clarify that both γ\gamma and δ\delta were selected based on validation-set performance, not test benchmarks. Tables 3, 9, and 10 report test-set results for sensitivity analysis only and were not used for hyperparameter selection. We will clarify this selection protocol in the revised manuscript.

[Q7] Regarding model extrapolation, online merging, and differences between RL- and SFT-trained teachers.

We have discussed the distinction between our method and prior extrapolation/online merging approaches in our response to [W1] and will expand the related-work discussion accordingly. Regarding the different recovery patterns, we hypothesize that this difference stems from teacher–student distribution mismatch. RL-trained teachers optimize rewards on their own generated trajectories, potentially preserving greater compatibility with the base student model. In contrast, SFT-trained teachers learn from externally generated or curated data, which may introduce substantial shifts in generation style, response structure, and knowledge distribution, making their capabilities harder to recover through MOPD. This interpretation is consistent with our teacher-continuation analysis (Figure 5). Indeed, Figure 5 shows that this teacher–student gap appears in the SFT-teacher domains, and that merging the corresponding task vectors reduces it; this is exactly the role of merging in IM-MOPD.

Reviewer_T6XCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes IM-MOPD, which improves multi-teacher on-policy distillation (MOPD) by initializing the student from a uniform merge of the teachers' task vectors and periodically adding more weight from any teacher whose domain performance is below a threshold. It observes that plain MOPD struggles when teachers were post-trained differently, and that a student's initial score doesn't necessarily predict its final performance after MOPD. The paper experiments on five domains with Qwen3-4B and Qwen3-1.7B and shows superior overall performance compared to plain MOPD.

Critical strengths
  • Writing is overall clear, and the proposed method is well justified by their findings.
  • The proposed method is simple and seems to show strong performance.
Critical weaknesses
  • Limited model coverage: only two small models are used for experiments, and they are from the same model family. I would like to see results using at least one model from a different family (Llama, Olmo, etc.), which would make the paper stronger.
  • Training budget and convergence: it seems each training run used 100 steps, but there is no explanation of how this number was chosen. Looking at Figure 3 and 4, models do not seem to have converged after 100 steps. It would be great to see whether IM-MOPD results in a higher final score compared to the baselines, or whether it simply accelerates training without converging to a higher score.
  • Applicability: as the authors noted, the merginig initialization is only possible if the teachers and the student share the same exact architecture. I think this limits the applicability of the proposed method, unlike MOPD which can use larger teachers.
Questions for authors
  • IM MOPD only adds task vectors. Did the authors experiment with subtracting the task vector of teachers that seem to hurt other domain performance?
  • The motivation for the method seems somewhat related to Speculative KD (Xu et. al, ICLR 2025) [1], since adding teacher weights where the domain performance falls short makes the student's generated tokens similar to those of that teacher. What is the authors' opinion on the relationship between the proposed method and the paper?

[1] Xu et al. Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling. ICLR, 2025.

3 comments

We thank the reviewers for their thoughtful questions. In particular, the experiments suggested by reviewer W1 helped us further strengthen our paper.

[W1] Model coverage.

Following the reviewer's suggestion, we conducted additional experiments using DeepSeek-Distill-Llama-8B and OLMo-3-7B-Think to further validate our method. Tables R1 and R2 show the results. IM-MOPD clearly outperforms MOPD on both models, raising the normalized score from 64.1 to 95.8 on DeepSeek-R1-Distill-Llama-8B and from 32.0 to 82.7 on OLMo-3-7B-Think. Together with the Qwen-3 family results, these results show that IM-MOPD applies consistently across different model families. (Due to the high API cost, τ²-Bench evaluations use a locally served Qwen3.8-27B instead of GPT-4.1 as the user simulator)

Table R1. Evaluation results on DeepSeek-Distill-Llama-8B

MethodMedQACaseHOLDFinQAIFBenchτ²Norm.
Reference model58.663.953.318.811.40.0
Domain teachers79.976.462.553.185.4100.0
MOPD60.068.667.949.332.764.1
IM-MOPD (Ours)76.174.564.752.078.995.8

Table R2. Evaluation results on OLMo-3-7B-Think

MethodMedQACaseHOLDFinQAIFBenchτ²Norm.
Reference model62.764.261.844.312.90.0
Domain teachers71.071.573.967.271.6100.0
MOPD59.265.672.362.123.132.0
IM-MOPD (Ours)69.370.071.862.168.182.7

[W2] Training budget and convergence.

First of all, we clarify that the 100-step checkpoint was not selected based on convergence or to favor IM-MOPD. Rather, we just fixed the same budget of 100 MOPD updates for all methods. This ensures a controlled comparison. To further address the reviewer's concern about convergence, we trained MOPD, Uniform Merge + MOPD, and IM-MOPD for 200 MOPD updates under the same settings and evaluated each at 100 and 200 updates (Table R3). All methods show only slight improvements with an additional 100 training steps, while the ranking remains unchanged. Thus, the 100-step budget does not favor IM-MOPD; its advantage holds with twice the training budget.

Table R3. 200 steps evaluation results on Qwen3-4B-OT3

MethodUpdatesMedQACaseHOLDFinQAIFBenchτ²Norm.
Reference model–69.561.859.124.94.70.0
Domain teachers–80.373.375.059.471.9100.0
MOPD10070.263.974.357.69.444.2
MOPD20069.867.474.257.914.051.0
Uniform Merge + MOPD10073.667.674.957.313.258.8
Uniform Merge + MOPD20073.869.174.557.217.562.5
IM-MOPD (Ours)10076.071.773.158.967.085.0
IM-MOPD (Ours)20078.471.373.059.476.691.7

[W3] Applicability to larger or heterogeneous teachers.

Our primary contribution is to identify the challenge of uneven capability recovery in MOPD and address it through iterative task-vector corrections, enabling effective teacher integration within a single distillation run. As acknowledged in our paper, this approach assumes same-origin teachers. Extending IM-MOPD to heterogeneous teachers, potentially through logit-space aggregation, is a promising direction of future work.

[Q1] Why only positive task-vector corrections?

Our method uses corrections to recover under-recovered teacher capabilities. Adding a task vector moves the model toward the direction learned by the teacher, whereas subtracting it moves in the opposite direction, potentially degrading performance. Such subtraction is more relevant to forgetting undesirable knowledge than to our capability recovery setting.

[Q2] Relationship to Speculative KD (Xu et al., ICLR 2025; SKD).

We clarify that SKD is already cited and discussed in our related work (Section 5.1, Lines 445-446) in the context of teacher–student distribution mismatch in on-policy distillation. While SKD addresses this mismatch by modifying sampling trajectories through teacher intervention, IM-MOPD focuses on effective student initialization and progressive parameter-space corrections to improve multi-teacher capability recovery, without modifying the sampling procedure. The two approaches are therefore complementary, and SKD could be incorporated into the distillation stage of IM-MOPD.

Dear authors,

thank you for your comprehensive and thoughtful response. The additional results largely resolve my concerns about somewhat limited experimental score.

In my opinion, the main strength of the paper is its observation that good initialization for MOPD does not necessarily mean good final performance, and that a simple modification to the MOPD algorithm, in which the expert weights are periodically mixed into the student to receive better learning signal, is effective. On the other hand, the main weakness of the paper is that the method is largely a combination of existing methods rather than a fundamentally new algorithm, and that the theoretical implications of the observation is not really discussed.

My opinion of the paper is somewhat mixed, but I think the paper's strength outweighs its weakness, and I think it could provide a useful foundation for future work in this direction. Therefore, I increase my score to 3.

Thank you for raising your score.

We are glad that we were able to address your concerns through this rebuttal.

Your questions have genuinely helped us strengthen our paper.

Comments on the paper

2 comments

We thank all reviewers for recognizing the simplicity and strong empirical performance of our method. We would like to clarify several common misunderstandings.


Problem setup and contributions (for uB9n, rBES, pwf4). Given independently specialized domain teachers, our goal is to improve MOPD capability recovery without costly coefficient search, additional mixed-domain SFT warm-up requiring extra data generation.

Some of our key observations are that (1) initial merge performance does not reliably predict post-MOPD recovery [Figures 1 and 3-(a)], and (2) effective coefficients can lie beyond convex merging [Figure 3-(c)]. Thus, finding effective coefficients requires exploring a continuous, high-dimensional search space, with full MOPD runs and evaluations needed to assess each candidate.

IM-MOPD addresses this challenge within a single MOPD run, replacing coefficient search with simple recovery-based thresholding and only two globally shared scalars, γ\gamma and δ\delta. Importantly, the effective coefficients in Table 2 were themselves discovered by IM-MOPD, while interleaving merging with MOPD provides additional gains.

Validation setup (for uB9n, rBES, pwf4). Both γ\gamma and δ\delta were selected using validation data, and all merge decisions use validation feedback. Tool Use training, validation, and test sets are disjoint by task ID. We further clarified this in the revised manuscript (see the orange line).

Additional experiments. In response to the reviewers, we conducted the following additional experiments:

  • Broader model coverage: IM-MOPD improves normalized recovery over MOPD on DeepSeek-R1-Distill-Llama-8B and OLMo-3-7B-Think [T6XC-Table R.1 and R.2].
  • Extended training budgets: IM-MOPD's advantage persists with 200 MOPD updates and against baselines using up to 3× SFT warm-up, with substantially lower compute than the extended warm-up baselines [T6XC-Table R.3].
  • Stronger merging baselines: We evaluate TIES and DARE-TIES alone and followed by MOPD. IM-MOPD outperforms these baselines and further improves recovery when initialized from either merge [uB9n-Table R.5].
  • Repeated Tool Use evaluations: We report avg@3 results from additional independent τ2\tau^2-Telecom evaluations [Updated manuscript: Table 1].
  • General capabilities: We additionally evaluate GPQA-Diamond and MMLU-Pro to assess the effect on capabilities beyond the target domains [pwf4-Table R.6].

We believe that our additional experiments have addressed many of the reviewers’ concerns. More detailed responses and additional results are provided in our individual responses.

Dear reviewers,

It seems that the authors have carefully responded to your reviews.

If you haven't already, please take your time to go over them and let authors know what you think, to your earliest convenience.

Best,
AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.