acceptodds
Under review as a conference paper at ICLR 2027

FusionEval: Benchmarking Model Fusion for Large Language Models

Abstract

Model fusion aims to integrate the capabilities of multi-expert models into a single model. Existing evaluations offer limited coverage of reasoning with long chains of thought and tool use, while strong average performance can mask substantial losses in individual expert capabilities. We introduce FusionEval, a benchmark that provides controlled expert pools, integrates fusion methods with and without additional training, and offers a unified evaluation suite. The pools contain domain experts independently trained from Qwen3 and Llama base models across math, code, science, instruction following, and tool use. The suite evaluates capability retention and computational cost on eight public benchmarks under comparable data access rules, reporting normalized average and worst performance, runtime, and peak GPU memory usage. Extensive experiments show that training-free methods preserve reasoning performance but substantially degrade tool use at both model scales. In contrast, MOPD preserves tool use and achieves a more balanced capability profile, but requires additional data preparation, inference, and training. Jointly retaining reasoning, instruction following, and tool use at low computational cost remains a challenge. We hope FusionEval supports future research on efficient model fusion that jointly preserves expert capabilities.

Primary area:
datasets and benchmarks
Keywords:
model fusion, large language models, knowledge transfer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gLm6OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper poses the problem of evaluation of model fusion approaches and proposes a new benchmark that provides controlled expert pools, integrate fusion methods with and without additional training and provide a unified evaluation suite.

Critical strengths
  • The topic is relevant because combining models can help leverage complementary capabilities across tasks.
Critical weaknesses
  • The definition of model fusion conflates model merging and knowledge distillation, which are generally distinct techniques.
  • It is unclear how the method operates, which models are combined, and whether the methods merge parameters or distill knowledge. For distillation, the teacher and student models are not explicitly identified
  • The paper does not adequately specify which models are combined, their architectures, the composition of each expert pool, or how the resulting models are evaluated on the datasets
  • Technical terms and abbreviations (MOPD, M2RL, GKD, SFT) are not defined before their first use
  • Incomplete related work: relevant references appear to be missing (https://llm-merging.github.io/prizes.html- particularly, the consideration of knowledge graph in model merging "Towards an approach combining Knowledge Graphs and Prompt Engineering for Merging Large Language Models")
  • In the paper, the term "open model" is not complete. Is it "open-source" model?
Questions for authors
  • What precisely does model fusion mean in the paper?
  • How do the paper distinguish model merging from knowledge distillation?
  • Is multi-teacher on-policy distillation (MOPD) considered a fusion method? If yes, why?
  • For each evaluated method in the experimental section, which models are combined? Are their parameters merged, or is a student trained using one or more teacher? (in this case, the teacher(s) and student should be clearly specified)
  • What constitutes an expert pool? How many models does each pool contain, and how are the models selected?
  • Could the paper provides a table specifying the input models, their architectures, the fusion or distillation method, the resulting model, and the evaluation datasets?
5 comments
AuthorsOpenReview ↗

Response to gLm6 (1/2)

Thank you for the helpful review. We respond to each of your points below.

W-1/Q-1/Q-2/Q-3: Definition of model fusion and distinction between merging and distillation

The definition of model fusion conflates model merging and knowledge distillation […] What precisely does model fusion mean in the paper? How do the paper distinguish model merging from knowledge distillation? Is MOPD considered a fusion method?

Rather than conflating the two, we use model fusion as the umbrella term for combining multiple specialized experts into a single standalone model without inference routing [1]. Under this taxonomy, model merging (such as TA and TIES) combines weights directly in parameter space, while multi teacher distillation (such as Seq-KD and MOPD) transfers expert capabilities in behavior space. MOPD is a behavior level fusion method because it unifies specialized teachers through on policy rollout supervision. Evaluating both paradigms side by side in FusionEval is intentional, as they represent the main competing ways practitioners build a single unified model from domain experts. We will sharpen this taxonomy early in the paper to avoid any confusion.


W2/Q4: Method Mechanisms and Teacher–Student Roles

It is unclear how the method operates, which models are combined, and whether the methods merge parameters or distill knowledge. For distillation, the teacher and student models are not explicitly identified […] For each evaluated method, which models are combined? Are their parameters merged, or is a student trained using one or more teachers?

Every method within a model pool (Qwen3-4B or Qwen3-8B) builds on the same base model and the exact same five domain experts (Math, Code, Science, Instruction Following, and Agent).

The ten merging methods (Soup, TA, TIES, DARE, DELLA, Fisher, L&S, Ada, RegMean, and LOT) combine weights or task vectors directly in parameter space without a student model (Table 3). For distillation, Seq-KD and MOPD use all five domain experts as teachers to train one shared student initialized from the base model via mixed domain SFT. Seq-KD trains offline on teacher generated responses, while MOPD trains online on student rollouts using token level supervision from the relevant domain teacher. Both output a single standalone checkpoint for inference. We will provide a more detailed explanation in the appendix of the next version.


W-3/Q-5/Q-6: Expert pools, model selection, and experimental setup

The paper does not adequately specify which models are combined, their architectures, the composition of each expert pool, or how the resulting models are evaluated […] What constitutes an expert pool? How many models does each pool contain, and how are the models selected? Could the paper provide a table specifying the input models, architectures, methods, resulting models, and evaluation datasets?

We construct three expert pools based on Qwen3-4B-Base, Qwen3-8B-Base, and Llama-3.2-3B. Starting from base rather than instruct models avoids prior alignment confounders and keeps capability attribution clean, while freezing the five shared architecture experts across methods guarantees controlled comparisons. Qwen3-4B-Base and Qwen3-8B-Base test size scaling across both merging and distillation, whereas Llama-3.2-3B tests cross family generalization for merging and calibration.

Table. Overview of expert pools, architectures, methods, and evaluation benchmarks.

Base ModelArchitectureInput ModelsMethodsOutputEvaluation
Qwen3-4B-BaseDecoder-only5 domain experts9 merging methods, Seq-KD, MOPDSingle 4B model8 benchmarks
Qwen3-8B-BaseDecoder-only5 domain experts9 merging methods, Seq-KD, MOPDSingle 8B model8 benchmarks
Llama-3.2-3BDecoder-only5 domain expertsParameter merging and calibrationSingle 3B model8 benchmarks

The nine merging methods are Soup, TA, TIES, DARE, DELLA, Fisher, L&S, Ada, and RegMean. All resulting standalone models are evaluated on the eight benchmarks above using identical prompts, decoding parameters, and scoring scripts within each pool (Table 4 and Appendix B.4). We will include this summary table in Section 3.


W-4: Undefined technical abbreviations

Technical terms and abbreviations (MOPD, M2RL, GKD, SFT) are not defined before their first use.

We will define missed technical terms and abbreviations at their first occurrence and ensure consistent terminology throughout the paper.

AuthorsOpenReview ↗

Response to gLm6 (2/2)

W-5: Related work

Incomplete related work, particularly "Towards an Approach Combining Knowledge Graphs and Prompt Engineering for Merging Large Language Models."

We thank the reviewer for highlighting this work [2]. It uses knowledge graphs to guide model selection based on task performance, followed by parameter merging and prompt engineering. FusionEval instead uses fixed expert pools for controlled comparisons, so knowledge graph based model selection is outside our scope. We already evaluate TA and TIES, the merging methods considered in this paper. We will add this citation and include a further discussion of this graph-based model fusion approach in the appendix.

W-6: Open model terminology

In the paper, the term "open model" is not complete. Is it "open-source" model?

We will use "open-source model" to refer to models with publicly available weights.

References

[1] Cai, Shuo, et al. "From Parameters to Behaviors: A Survey of Model Fusion for Large Language Models." Findings of EMNLP 2026.

[2] Teguimene Yendji, Furel De Consol, et al. "Towards an Approach Combining Knowledge Graphs and Prompt Engineering for Merging Large Language Models." NeurIPS 2024 LLM Merging Competition.

Thanks again for reviewing our work. We have posted our responses and additional results under your review. With the discussion period ending soon, please let us know if our replies address your concerns or if anything else needs clarification.

Reviewer_pcBAOpenReview ↗
Rating
2
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
2
Read review
Paper summary

FusionEval constructs independently SFT-trained expert pools and compares model fusion across reasoning, instruction-following and tool-use benchmarks, reporting expert-relative retention and algorithm costs. The results expose large tool-use losses hidden by strong average retention, while MOPD achieves more balanced Qwen3 profiles with additional training.

Critical strengths
  1. The shared expert pools make it easier to compare the methods, and the full score breakdown shows tool-use losses that an average score can hide.
  2. The Student baseline helps show what MOPD actually adds. The Llama results and the checks on weight updates also make the limits of the findings easier to see.
Critical weaknesses
  1. Some of the same questions are used to choose models and report their final scores, including all AIME questions. Without confidence intervals, it is also hard to tell how much to trust the small gaps between methods.
  2. Some reported runtimes come from settings different from those used for the final scores. The methods also use different GPU counts and numbers of updates, so the cost-performance plots are hard to compare directly.
Questions for authors
  1. Could you check whether the rankings hold up on a fresh set of questions that wasn't used to choose the models? It would also help to see confidence intervals for the main results.

  2. Could you report the runtime, number of GPUs, and training steps for the same models used in the score tables?

  3. Based on the update rules, continual TA and joint TA seem to give the same final model, so what explains the different scores? Also, how exactly did you run continual TIES?

7 comments
AuthorsOpenReview ↗

Response to pcBA (1/3)

Thank you for carefully reviewing our work and for your constructive suggestions. We respond to your comments and questions point by point below.

W-1/Q-1: Selection bias, confidence intervals, and held-out evaluation

Some of the same questions are used to choose models and report their final scores, including all AIME questions. Without confidence intervals, it is also hard to tell how much to trust the small gaps between methods. […] Could you check whether the rankings hold up on a fresh set of questions that wasn't used to choose the models?

We acknowledge that overlap between model selection and final evaluation may introduce selection bias. To quantify evaluation uncertainty, we compute 95% confidence intervals using 10,000 paired question-level bootstrap resamples, applying identical resampled question indices across all methods. Scores are reported as accuracy (%) with [2.5th, 97.5th] percentile intervals.

Qwen3-4B evaluation results (%, 95% CI):

MethodAIME24AIME25LCBv5LCBv6GPQA-DIFEvalIFBenchBFCLv3
TIES69.0 [60.6, 76.5]64.4 [56.8, 71.9]41.6 [36.2, 46.8]35.9 [30.6, 40.1]36.9 [32.1, 41.7]65.3 [61.8, 68.9]36.2 [31.4, 41.1]30.0 [29.9, 30.1]
RegMean43.5 [35.0, 49.5]40.2 [33.7, 49.0]31.5 [26.7, 36.4]29.4 [24.9, 34.3]47.1 [41.8, 52.4]68.9 [65.4, 72.2]28.7 [24.2, 33.4]30.4 [30.0, 30.7]
DARE66.9 [58.3, 74.8]62.5 [54.4, 69.8]47.5 [42.1, 52.7]42.6 [37.3, 48.0]49.0 [43.7, 54.3]70.9 [67.6, 74.3]37.0 [32.0, 42.0]30.6 [30.2, 31.0]
Ada67.5 [60.6, 76.8]58.8 [50.6, 67.7]42.3 [36.9, 47.5]38.7 [32.7, 44.4]49.6 [44.2, 54.9]66.6 [63.0, 70.1]35.0 [30.2, 39.8]31.0 [30.7, 31.4]
DELLA68.5 [60.6, 77.0]64.8 [57.9, 71.9]47.0 [41.8, 52.0]40.8 [35.6, 45.1]49.5 [44.3, 54.8]64.4 [60.7, 68.1]36.0 [31.7, 40.2]31.1 [30.8, 31.4]
TA67.5 [60.4, 73.0]67.1 [60.8, 71.6]48.2 [42.3, 52.4]42.2 [37.1, 47.2]51.4 [46.1, 56.7]69.4 [65.9, 72.9]37.8 [32.1, 41.1]31.1 [30.7, 31.5]
L&S67.1 [58.3, 74.4]62.5 [53.5, 70.4]49.3 [44.7, 52.8]39.5 [34.2, 45.0]44.6 [39.5, 49.9]70.3 [66.2, 73.3]42.3 [38.9, 45.2]31.5 [31.1, 32.0]
Soup45.4 [37.7, 53.8]35.6 [28.5, 42.2]27.8 [20.3, 34.8]27.5 [21.2, 34.2]51.3 [45.7, 56.7]62.9 [59.3, 66.4]24.8 [21.9, 27.0]33.8 [33.2, 34.4]
Fisher39.8 [32.1, 47.1]38.1 [31.2, 45.9]38.2 [33.2, 43.2]31.9 [28.2, 35.7]47.1 [41.9, 52.3]63.9 [60.4, 67.3]29.6 [24.8, 34.4]45.1 [44.0, 46.3]
Student49.1 [42.0, 56.5]47.9 [40.2, 55.5]33.5 [29.0, 38.0]31.5 [27.0, 35.5]30.9 [26.7, 35.3]64.8 [61.3, 68.3]27.8 [23.0, 32.8]63.7 [62.2, 65.3]
Seq-KD47.9 [41.8, 55.0]43.8 [36.2, 50.3]36.4 [31.4, 41.6]31.9 [27.2, 36.9]34.5 [29.7, 39.4]67.7 [64.1, 71.3]37.5 [32.7, 42.3]63.7 [62.1, 65.4]
MOPD59.6 [52.9, 65.9]53.3 [44.8, 60.9]42.8 [37.6, 48.0]38.9 [33.7, 43.4]36.7 [32.1, 41.5]77.5 [74.4, 80.7]41.4 [36.5, 46.5]66.1 [64.5, 67.7]
AuthorsOpenReview ↗

Response to pcBA (2/3)

W-1/Q-1: Selection bias, confidence intervals, and held-out evaluation (continued)

Qwen3-8B evaluation results (%, 95% CI):

MethodAIME24AIME25LCBv5LCBv6GPQA-DIFEvalIFBenchBFCLv3
Fisher60.4 [51.7, 68.8]41.0 [33.9, 49.0]30.6 [25.8, 35.7]28.8 [24.1, 33.7]45.2 [40.4, 50.1]64.0 [60.4, 67.5]26.8 [22.4, 31.2]29.8 [29.6, 30.0]
Ada69.0 [60.4, 76.7]65.8 [57.9, 72.7]55.9 [50.7, 61.1]48.5 [43.0, 53.9]48.2 [42.8, 53.8]73.9 [70.6, 77.1]41.4 [36.6, 46.3]31.3 [30.9, 31.6]
TA69.8 [62.3, 76.7]68.1 [60.2, 74.8]56.6 [51.3, 61.8]52.7 [47.2, 58.3]55.4 [50.0, 60.9]74.5 [71.2, 77.7]39.8 [34.9, 44.7]31.5 [30.9, 32.0]
DARE71.9 [63.8, 78.2]67.1 [59.8, 73.4]57.5 [52.2, 62.7]48.7 [43.0, 54.3]55.6 [50.1, 61.1]74.7 [71.4, 77.9]35.2 [30.6, 39.9]31.8 [31.2, 32.3]
RegMean52.5 [45.0, 60.2]42.1 [33.3, 51.3]37.1 [32.1, 42.3]35.7 [30.6, 41.1]54.0 [48.6, 59.6]72.3 [69.0, 75.6]29.0 [24.4, 33.8]31.8 [31.3, 32.3]
Soup53.1 [44.6, 61.7]39.8 [31.7, 47.1]41.2 [36.2, 46.4]36.1 [31.0, 41.3]57.2 [51.9, 62.5]75.0 [71.7, 78.1]31.8 [27.1, 36.4]34.1 [33.4, 34.8]
TIES69.0 [61.6, 75.7]63.5 [55.2, 71.0]58.4 [53.4, 63.6]44.5 [39.2, 50.1]52.3 [46.7, 57.8]73.7 [70.3, 76.8]43.2 [38.3, 48.2]34.8 [34.1, 35.4]
DELLA75.6 [67.7, 82.3]69.4 [61.5, 76.3]59.7 [54.3, 65.1]51.3 [46.1, 56.6]58.2 [52.7, 63.8]71.9 [68.5, 75.1]39.2 [34.2, 44.2]37.8 [37.0, 38.6]
L&S72.7 [64.6, 79.8]72.1 [63.8, 79.0]55.4 [50.0, 60.8]50.8 [45.3, 56.4]51.6 [46.0, 57.6]75.1 [71.7, 78.3]42.0 [37.1, 46.9]39.1 [38.3, 40.0]
Student52.8 [45.6, 60.1]51.5 [44.1, 59.1]39.2 [34.5, 44.0]36.2 [31.6, 41.0]31.2 [27.1, 36.2]67.4 [63.8, 70.9]30.8 [25.8, 35.8]63.1 [61.5, 64.6]
Seq-KD45.8 [39.0, 52.5]55.8 [47.3, 64.4]37.1 [31.9, 42.3]34.9 [30.1, 40.2]44.4 [39.3, 49.8]68.5 [65.0, 72.0]40.7 [35.7, 45.8]65.0 [63.4, 66.7]
MOPD67.9 [59.8, 75.4]59.8 [50.6, 67.3]45.9 [40.5, 51.3]42.6 [37.5, 48.0]45.5 [40.4, 50.5]80.3 [77.3, 83.3]46.1 [41.1, 51.0]66.9 [65.3, 68.6]

The added 95% CIs leave the point-estimate rankings in Tables 5–6 unchanged and support the main finding. MOPD retains the highest observed Min. Norm. scores (0.708 and 0.859), consistent with its more balanced capability profile. However, some close comparisons, including MOPD versus Seq-KD on BFCLv3, have overlapping intervals. We therefore distinguish point-estimate rankings from statistically established differences. The broad performance patterns remain supported, but the marginal CIs alone do not establish every pairwise ordering.

To verify generalization without selection bias, we further evaluated the fixed Qwen3-4B checkpoints on unseen MATH-500 and HumanEval.

Held-out evaluation results on MATH-500 and HumanEval for Qwen3-4B (%, 95% CI)

MethodMATH-500HumanEval
Soup85.20 [82.00, 88.20]78.66 [72.55, 84.76]
TA92.60 [90.40, 94.80]90.85 [85.98, 95.12]
TIES89.80 [87.20, 92.40]55.49 [47.56, 63.41]
DARE92.00 [89.60, 94.20]87.80 [82.93, 92.68]
DELLA92.40 [90.00, 94.60]89.63 [84.76, 93.90]
Fisher80.80 [77.40, 84.00]87.80 [82.93, 92.68]
L&S64.00 [59.80, 68.20]20.12 [14.63, 26.83]
Ada93.00 [90.80, 95.20]92.07 [87.80, 95.73]
RegMean83.80 [80.40, 87.20]78.66 [71.95, 84.76]
Seq-KD86.00 [83.00, 89.00]81.71 [76.20, 87.20]
MOPD92.40 [89.40, 94.20]86.37 [82.05, 91.68]

Top rankings remain stable, led by Ada (93.00 and 92.07), TA (92.60 and 90.85), DELLA (92.40 and 89.63), and DARE (92.00 and 87.80), while L&S drops sharply (64.00 and 20.12), confirming that held out tests catch overfitting to selection benchmarks. We will add these intervals and held-out results to the revised paper.

AuthorsOpenReview ↗

Response to pcBA (3/3)

W-2/Q-2: Consistent runtime, GPU counts, and training steps

Some reported runtimes come from settings different from those used for the final scores. The methods also use different GPU counts and numbers of updates […] Could you report the runtime, number of GPUs, and training steps for the same models used in the score tables?

We thank the reviewer for noting these reporting inconsistencies. Below, we report standardized H800 GPU hours, GPU counts, and update steps matching the exact checkpoints evaluated in Tables 5 and 6.

Runtime costs of parameter fusion methods on Qwen3-4B and Qwen3-8B (1 H800 GPU):

Method4B GPU-hours8B GPU-hours
Soup0.00570.0125
TA0.01040.0219
TIES0.39000.7931
DARE0.15330.2588
DELLA0.02470.0540
Fisher0.05470.3994
L&S0.49961.0564
Ada2.02494.6569
RegMean0.25500.9198

Distillation runtime costs on Qwen3-4B and Qwen3-8B (16 H800 GPUs):

Method4B GPU-hours8B GPU-hours
Seq-KD205.12295.32
MOPD162.92205.14

The original figures report wall-clock runtime in minutes, which does not account for merging using 1 GPU versus distillation using 16 GPUs. We will instead report computational cost in H800 GPU-hours, so the comparison reflects both elapsed time and the number of GPUs used.

Q-3: Continual TA numerical precision and Continual TIES protocol

Based on the update rules, continual TA and joint TA seem to give the same final model, so what explains the different scores? Also, how exactly did you run continual TIES?

Continual TA and joint TA are mathematically equivalent. The score differences arise from accumulated rounding errors. Joint TA accumulates task vectors in FP32 and converts to BF16 only once, whereas continual TA converts the merged weights to BF16 after each stage. We verified that both methods produce identical weights when using the same precision and summation order. We will clarify these implementation details in the revision. Continual TA and joint TA are mathematically equivalent under the same task-vector coefficients. The discrepancy arises from numerical precision. Joint TA accumulates task vectors in FP32 and converts the final weights to BF16 once, whereas continual TA converts the merged weights to BF16 after each stage, introducing accumulated rounding errors. We compared their final weights under the original and matched precision settings:

Final-weight differences between continual and joint TA.

Precision settingRelative L2 difference ↓Maximum absolute difference ↓Bitwise parameter mismatch rate ↓
Joint FP32 accumulation vs. Continual stage-wise BF16 rounding0.348626%0.062543.9210%
BF16 accumulation and the same summation order in both0%00%

With matched BF16 accumulation and summation order, the final weights are bitwise identical. This confirms that the weight discrepancy is numerical rather than an inherent difference between the two TA update rules. We will clarify these implementation details in the revision.

For continual TIES, we follow a strict checkpoint free setting without revisiting historical experts. Starting from the IF expert, we sequentially merge the Math, Science, Code, and Agent experts. At each stage, we compute task vectors for both the running merged model and the incoming expert relative to the original base model, then apply standard TIES trimming at density 0.3, sign election, and disjoint mean merging with scaling factor 1.0. Because the accumulated update is reprocessed online at every step, earlier task vectors undergo repeated trimming and 2 way sign elections.

Reviewer_n46BOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper establishes FusionEval, a benchmark for LLM merging. Building upon previous benchmarks such as MergeBench, FusionEval focuses on more recent LLM capabilities such as long reasoning and tool use, which are lacking in previous evaluations. The authors construct controlled experiments by individual SFT on shared base models on five domains, and evaluate various model merging and distillation algorithms. The experiments demonstrate that conventional model merging methods preserve reasoning, but do not preserve tool use capabilities well.

Critical strengths
  • The work is a meaningful extension from MergeBench. It introduces more up-to-date base models and benchmarks for evaluation, with focus on emerging agentic capabilities such as tool use.
  • It is encouraging to see a comparison between MOPD and model merging. Those are complimentary ways to combine the strength of different models into a unified model, and systematic comparisons between them are valuable for understanding their advantages and limitations.
Critical weaknesses
  • It is questionable whether the model merging and MOPD comparison is fair. MOPD introduces an additional SFT stage that trains the base model on a training data mixture pooled from all domains. Although I appreciate the authors comparing MOPD and model merging, the additional SFT stage is a clear confounder for MOPD’s performance. For example, on Qwen3-8B, the student already achieves 63.1 on BFCLv3 before distillation, compared to 66.9 after MOPD. This suggests that much of the tool-use capability is acquired during the initial SFT stage rather than through MOPD. Consequently, the current results do not establish an apples-to-apples comparison between the two approaches.
  • The conclusion of tool-use degradation is also questionable. Tool use is naturally very prone to small output twists because the tool name and function calling requires specific and precise format. Parameter merging can introduce small changes in model outputs, which may disproportionately affect strict tool-use evaluation metrics such as BFCLv3. However, this does not mean that the abilities to use tools degrade. In practice, some simple SFT on the correct tool formats after merging could probably solve those problems easily. The benefits of MOPD could largely due to the fact that the SFT mixture and teachers never show wrong tool formats, so there is no correction needed. Without disentangling formatting errors from semantic tool-use failures, it is difficult to determine whether the reported degradation represents a fundamental challenge for model merging or primarily an evaluation sensitivity issue.
Questions for authors
  • Would FKLD be a better setting for distillation? I understand that MOPD naturally requires a SFT stage to make the students not that far from the teacher, otherwise RKLD would not work, but this introduces the SFT confounder. What about doing FKLD which does not have the problem, and remains a valid on-policy distillation method? Or, could the authors provide an ablation comparing Base → MOPD with Base → Mixed SFT → MOPD? This would help disentangle the contributions of mixed-domain SFT and on-policy distillation.
  • Why is 70% of the domain SFT data selected for student initialization? I originally thought that the authors will use 70% of the domain SFT data, then the remaining 30% for distillation, to ensure that the models train on the same size of data for MOPD and model merging, but that does not seem to be the case.
4 comments
AuthorsOpenReview ↗

Response to n46B (1/2)

Thank you for your detailed suggestions. We address the remaining concerns below.

W-1/Q-1: Fairness of comparison between model merging and MOPD

MOPD introduces an additional SFT stage that trains the base model on a training data mixture pooled from all domains […] the additional SFT stage is a clear confounder for MOPD’s performance […] Could the authors provide an ablation comparing Base → MOPD with Base → Mixed SFT → MOPD, or evaluate FKLD without the SFT stage?

Thank you for raising this concern. We agree that the contributions of SFT warmup and subsequent distillation should be distinguished. In practice, SFT warmup is a standard procedure in on policy distillation [1-3] to align the student's initial generation behavior with the teachers before rollout optimization.

Assuming FKLD refers to forward KL distillation DKL(pT∥pS)D_{\mathrm{KL}}(p_T \| p_S) (please let us know if you meant otherwise), flipping the KL direction only alters the token level loss while training prefixes are still sampled from the student. Because the rollout distribution remains unaligned, changing the KL direction alone cannot stabilize training from a raw base model.

To assess the role of initialization, we ran MOPD, FKLD, and Seq-KD directly from Qwen3-4B-Base without the mixed domain SFT warmup under the same budget (50 steps, batch size 512).

InitializationDistillationAIME24AIME25LCBv5LCBv6GPQA-DIFEvalIFBenchBFCLv3Avg. Norm.Min. Norm.
Qwen3-4B BaseNone12.089.5812.0110.1132.4545.8415.5025.240.3450.142
Mixed SFTNone (Student)49.147.933.531.530.964.827.863.70.7410.596
Mixed SFTSeq-KD47.943.836.431.934.567.737.563.70.7840.649
Mixed SFTMOPD59.653.342.838.936.777.541.466.10.8990.708
BaseSeq-KD29.328.222.722.929.864.230.736.40.5680.399
BaseMOPD w/RKLD18.317.76.510.725.626.412.329.60.3100.146
BaseMOPD w/FKLD2.33.114.713.726.228.314.729.70.3000.031

Without SFT warmup, both reverse-KL and forward-KL MOPD exhibited unstable training. By contrast, Base → Seq-KD did not exhibit generation collapse but underperformed Mixed SFT → Seq-KD (0.568 vs. 0.784). Extending Base → Seq-KD training to 200 steps improved performance but did not close the gap to its SFT-initialized counterpart. To examine the role of initialization mismatch in OPD instability, we measured the student–expert log-probability gap on each student’s on-policy rollouts before distillation.

Initial student–expert mismatch, measured on each student’s own on-policy rollouts.

Student initializationInitial log-probability gap to expert ↓
Qwen3-4B-Base0.868
Mixed SFT0.325

Mixed SFT reduces the initial gap by 62.6%, indicating substantially better student–expert alignment before OPD. Although OPD aims to reduce this discrepancy, a large initial mismatch can make expert feedback on student-generated trajectories difficult to exploit, hindering stable optimization. This supports initialization mismatch as a contributor to the observed instability. MOPD also improves Avg. Norm. from 0.741 to 0.899 over SFT alone, demonstrating gains beyond initialization, although SFT remains an additional training cost outside the matched distillation budget. Overall, these results support the practical necessity of SFT cold-start for stable MOPD in our setting. Switching to forward KL did not stabilize training from the base model or remove the need for warmup.

Overall, applying OPD directly to a raw base model is unstable, especially when learning from long CoT experts, and FKLD cannot bridge this gap. Consistent with prior findings on OPD pitfalls and warmup recipes [1-3], an SFT cold start is essential for stable and effective distillation, which makes our experimental setup for MOPD well justified.

References

[1] Agarwal et al. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. ICLR, 2024.
[2] Zhu et al. The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes. NeurIPS 2026.
[3] Li et al. Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe. TMLR, 2026.

AuthorsOpenReview ↗

Response to n46B (2/2)

W-2: Disentangling formatting sensitivity from tool-use capability degradation

Tool use is naturally very prone to small output twists […] Parameter merging can introduce small changes in model outputs, which may disproportionately affect strict tool-use evaluation metrics such as BFCLv3 […] Without disentangling formatting errors from semantic tool-use failures, it is difficult to determine whether the reported degradation represents a fundamental challenge for model merging or primarily an evaluation sensitivity issue.

Thank you for highlighting this distinction. We agree that aggregate BFCLv3 scores alone cannot distinguish formatting failures from semantic tool-use errors. To provide a more informative comparison, we report single-turn and multi-turn results separately and inspect prediction logs for both Qwen3-4B and Qwen3-8B.

BFCLv3 single-turn and multi-turn accuracy (%):

Method4B Single-turn4B Multi-turn8B Single-turn8B Multi-turn
Agent Expert84.3730.0085.3733.88
Soup11.520.0012.590.38
TA5.120.005.320.13
TIES0.920.0013.230.25
DARE3.120.005.040.63
DELLA4.320.0021.110.88
Fisher60.181.250.280.13
L&S7.720.0025.950.50
Ada3.880.004.200.00
RegMean2.120.136.040.00
LOT5.000.000.080.00
Student (Mixed SFT)81.7725.1383.6923.75
Seq-KD81.9328.2584.6927.00
MOPD82.6530.8885.4531.75

The breakdown reveals poor multi-turn reliability after merging. Fisher on Qwen3-4B achieves 60.18% single-turn accuracy but only 1.25% multi-turn accuracy. Log inspection identifies both malformed calls and valid-format calls with incorrect arguments or variable bindings. BFCLv3’s multi-turn evaluation also checks execution states and required call sequences. These findings support degraded end-to-end tool-use reliability, rather than proving irreversible knowledge loss.

Format preservation is itself a merging challenge. TIES documents parameter interference [4], while RAIN-Merging reports disruptions to structured reasoning formats [5]. Correct-format supervision may contribute to MOPD’s advantage, but MOPD also improves multi-turn accuracy over the SFT-only student at both scales. Whether a small format-focused update restores performance without harming other capabilities remains untested. We will include this as a future direction in the next version of the paper.


Q-2: Allocation of 70% domain SFT data for student initialization

Why is 70% of the domain SFT data selected for student initialization? I originally thought that the authors will use 70% of the domain SFT data, then the remaining 30% for distillation […]

Thank you for pointing out this ambiguity. SFT uses fixed demonstrations, whereas OPD learns from teacher feedback on student-generated responses and does not require pre-collected answers. Related RL studies find that targeting data compositions outside SFT distribution can improve generalization in controlled tasks [6]. We therefore selected OPD prompts separately rather than restricting them to the SFT pool. We also ran MOPD using the remaining 30% SFT prompts to compare the performance:

Effect of OPD prompt source on AIME25 and BFCLv3 performance (%).

MethodDistillation prompt sourceAIME25BFCLv3
Student (70% SFT warmup)—47.963.7
MOPDRemaining 30% SFT prompts49.663.9
MOPDSeparate OPD prompt pool53.366.1

Using the remaining SFT prompts improves over the initialized student, but our default prompt pool yields higher scores by 3.7 points on AIME25 and 2.2 points on BFCLv3. These results support our distillation-data choice. We will clarify the allocation and report the SFT and distillation data budgets separately. Our OPD prompt pool achieves 53.3 on AIME25 and 66.1 on BFCLv3, compared with 49.6 and 63.9 using the remaining SFT prompts. This comparison supports our OPD data choice in the tested setting.

References

[4] Yadav, P., et al. TIES-Merging: Resolving Interference When Merging Models. NeurIPS, 2023.

[5] Huang, Z., et al. RAIN-Merging: A Gradient-Free Method to Enhance Instruction Following in Large Reasoning Models with Preserved Thinking Format. arXiv:2602.22538, 2026.

[6] Kong, L., et al. From Reasoning Traces to Reusable Modules: Understanding Compositional Generalization in Language Model Reasoning. arXiv:2606.18089, 2026.

Thanks again for reviewing our work. We have posted our responses and additional results under your review. With the discussion period ending soon, please let us know if our replies address your concerns or if anything else needs clarification.

Dear Reviewer,

As the discussion period is nearing its end. I wanted to ensure we have addressed all your concerns satisfactorily. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us.

Thanks for your time.

Comments on the paper

1 comment

We thank the reviewers and the Area Chair for their careful assessment and suggestions. Our point-by-point responses provide additional experiments and analyses. We summarize the key findings below.


  1. Our main findings hold after removing selection overlap and adding uncertainty estimates. Excluding the 101 model-selection questions on Qwen3-4B leaves the trends unchanged, with MOPD still leading on IFEval and BFCLv3. Adding 95% bootstrap confidence intervals across both Qwen3 scales preserves all point-estimate rankings, and held-out evaluations on MATH-500 and HumanEval confirm generalization for top merging methods and MOPD.
  2. MOPD improves substantially over the shared SFT initialization and retains multi-turn tool use. Mixed-domain SFT warmup reduces the initial student–expert log-probability gap by 62.6%, whereas neither RKLD nor FKLD stabilizes MOPD directly from the base model. Beyond SFT alone, MOPD raises Qwen3-4B Avg. Norm. from 0.741 to 0.899 and outperforms distilling on the remaining 30% SFT prompts (+3.7 on AIME25, +2.2 on BFCLv3). On BFCLv3, all evaluated merging methods drop below 2% on multi-turn tasks, while MOPD improves over the SFT-only Student by 5.8 and 8.0 points on 4B and 8B, respectively.
  3. Standardized H800 GPU-hours clarify the compute costs of the evaluated checkpoints. Parameter merging uses 0.0057–2.02 GPU-hours (4B) and 0.0125–4.66 GPU-hours (8B) on 1 GPU. Using 16 GPUs, MOPD requires 162.92 / 205.14 GPU-hours on 4B / 8B compared with 205.12 / 295.32 for Seq-KD, including rollout generation and student training while excluding the shared SFT stage.

Complete experimental settings, results, and analyses are provided in our responses under each review. The revised paper clarifies the fusion taxonomy, selection protocol, teacher–student roles, and the numerical equivalence of continual and joint TA in the main text, and adds the new results and expert-pool overview in Appendices B.5, C, and D. We will release the data used for training as well as the expert pool, with the hope that FusionEval will help drive further progress in model fusion. We welcome any remaining questions or requests for clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.