- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper poses the problem of evaluation of model fusion approaches and proposes a new benchmark that provides controlled expert pools, integrate fusion methods with and without additional training and provide a unified evaluation suite.
Critical strengths
- The topic is relevant because combining models can help leverage complementary capabilities across tasks.
Critical weaknesses
- The definition of model fusion conflates model merging and knowledge distillation, which are generally distinct techniques.
- It is unclear how the method operates, which models are combined, and whether the methods merge parameters or distill knowledge. For distillation, the teacher and student models are not explicitly identified
- The paper does not adequately specify which models are combined, their architectures, the composition of each expert pool, or how the resulting models are evaluated on the datasets
- Technical terms and abbreviations (MOPD, M2RL, GKD, SFT) are not defined before their first use
- Incomplete related work: relevant references appear to be missing (https://llm-merging.github.io/prizes.html- particularly, the consideration of knowledge graph in model merging "Towards an approach combining Knowledge Graphs and Prompt Engineering for Merging Large Language Models")
- In the paper, the term "open model" is not complete. Is it "open-source" model?
Questions for authors
- What precisely does model fusion mean in the paper?
- How do the paper distinguish model merging from knowledge distillation?
- Is multi-teacher on-policy distillation (MOPD) considered a fusion method? If yes, why?
- For each evaluated method in the experimental section, which models are combined? Are their parameters merged, or is a student trained using one or more teacher? (in this case, the teacher(s) and student should be clearly specified)
- What constitutes an expert pool? How many models does each pool contain, and how are the models selected?
- Could the paper provides a table specifying the input models, their architectures, the fusion or distillation method, the resulting model, and the evaluation datasets?
5 comments
Response to gLm6 (2/2)
W-5: Related work
Incomplete related work, particularly "Towards an Approach Combining Knowledge Graphs and Prompt Engineering for Merging Large Language Models."
We thank the reviewer for highlighting this work [2]. It uses knowledge graphs to guide model selection based on task performance, followed by parameter merging and prompt engineering. FusionEval instead uses fixed expert pools for controlled comparisons, so knowledge graph based model selection is outside our scope. We already evaluate TA and TIES, the merging methods considered in this paper. We will add this citation and include a further discussion of this graph-based model fusion approach in the appendix.
W-6: Open model terminology
In the paper, the term "open model" is not complete. Is it "open-source" model?
We will use "open-source model" to refer to models with publicly available weights.
References
[1] Cai, Shuo, et al. "From Parameters to Behaviors: A Survey of Model Fusion for Large Language Models." Findings of EMNLP 2026.
[2] Teguimene Yendji, Furel De Consol, et al. "Towards an Approach Combining Knowledge Graphs and Prompt Engineering for Merging Large Language Models." NeurIPS 2024 LLM Merging Competition.
Thanks again for reviewing our work. We have posted our responses and additional results under your review. With the discussion period ending soon, please let us know if our replies address your concerns or if anything else needs clarification.
Response to gLm6 (1/2)
Thank you for the helpful review. We respond to each of your points below.
W-1/Q-1/Q-2/Q-3: Definition of model fusion and distinction between merging and distillation
Rather than conflating the two, we use model fusion as the umbrella term for combining multiple specialized experts into a single standalone model without inference routing [1]. Under this taxonomy, model merging (such as TA and TIES) combines weights directly in parameter space, while multi teacher distillation (such as Seq-KD and MOPD) transfers expert capabilities in behavior space. MOPD is a behavior level fusion method because it unifies specialized teachers through on policy rollout supervision. Evaluating both paradigms side by side in FusionEval is intentional, as they represent the main competing ways practitioners build a single unified model from domain experts. We will sharpen this taxonomy early in the paper to avoid any confusion.
W2/Q4: Method Mechanisms and Teacher–Student Roles
Every method within a model pool (Qwen3-4B or Qwen3-8B) builds on the same base model and the exact same five domain experts (Math, Code, Science, Instruction Following, and Agent).
The ten merging methods (Soup, TA, TIES, DARE, DELLA, Fisher, L&S, Ada, RegMean, and LOT) combine weights or task vectors directly in parameter space without a student model (Table 3). For distillation, Seq-KD and MOPD use all five domain experts as teachers to train one shared student initialized from the base model via mixed domain SFT. Seq-KD trains offline on teacher generated responses, while MOPD trains online on student rollouts using token level supervision from the relevant domain teacher. Both output a single standalone checkpoint for inference. We will provide a more detailed explanation in the appendix of the next version.
W-3/Q-5/Q-6: Expert pools, model selection, and experimental setup
We construct three expert pools based on Qwen3-4B-Base, Qwen3-8B-Base, and Llama-3.2-3B. Starting from base rather than instruct models avoids prior alignment confounders and keeps capability attribution clean, while freezing the five shared architecture experts across methods guarantees controlled comparisons. Qwen3-4B-Base and Qwen3-8B-Base test size scaling across both merging and distillation, whereas Llama-3.2-3B tests cross family generalization for merging and calibration.
Table. Overview of expert pools, architectures, methods, and evaluation benchmarks.
The nine merging methods are Soup, TA, TIES, DARE, DELLA, Fisher, L&S, Ada, and RegMean. All resulting standalone models are evaluated on the eight benchmarks above using identical prompts, decoding parameters, and scoring scripts within each pool (Table 4 and Appendix B.4). We will include this summary table in Section 3.
W-4: Undefined technical abbreviations
We will define missed technical terms and abbreviations at their first occurrence and ensure consistent terminology throughout the paper.