acceptodds
Under review as a conference paper at ICLR 2027

When the Whole Exceeds the Parts: Constructing Multi-Ability LLMs via Component-wise Weighted Fusion

Abstract

Specialized language models often perform poorly on tasks that require capabilities from different domains. We introduce Component-wise Weighted Fusion (CWF), a model-merging method that learns how much each expert contributes to individual feed-forward network (FFN) neurons and attention heads. An FFN coefficient is shared across the neuron’s Gate, Up, and Down projections. Attention coefficients are shared by corresponding query and output slices, with group-averaged coefficients for shared key/value heads. The expert weights remain frozen while the coefficients are trained on mixed-domain data; the resulting weights form one dense model with no additional inference-time experts. We evaluate mathematics in Indonesian and Thai, coding with mathematics, and coding with multilingual knowledge using Llama and Qwen models at the 8B scale. Without target-language mathematics examples in mask training, CWF improves Indonesian and Thai MATH accuracy over the stronger input expert by 13.98 and 15.70 percentage points. On Qwen Code+Math, it matches the best reported merging baseline on each coding benchmark while retaining strong mathematical performance. We jointly evaluate target-language gains and source-domain retention. Component and learning-rate ablations favor joint FFN/attention fusion with separate learning rates, while mask and activation analyses expose heterogeneous expert contributions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
model merging, large language models, capability composition, cross-lingual transfer, parameter-efficient learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_igEBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Component-wise Weighted Fusion (CWF), which learns convex mixtures of corresponding components from two experts derived from a shared backbone. Each FFN neuron shares its mixture coefficients across Gate, Up, and Down. Attention shares coefficients between each query head’s Q/O slices and uses group-averaged coefficients for shared K/V heads. Expert parameters remain frozen while the mixing logits are optimized using mixed-domain cross-entropy. The resulting projections are materialized into one dense model. Experiments study mathematics in Indonesian and Thai, as well as code–math and code–multilingual combinations, with additional retention, ablation, and neuron-removal analyses.

Critical strengths

1. A clear and practical structural parameterization. CWF specifies a coherent coefficient-sharing scheme for both gated FFNs and GQA while preserving the backbone’s inference architecture (Section 3). The evaluated models require fewer than one million trainable mixing logits, and the learned projections can be stored in a standard checkpoint without additional expert forward passes at inference. This is a practical way to reuse specialized checkpoints while retaining single-model deployment.

2. Meaningful reported gains on a relevant composition setting. The cross-lingual mathematics experiments train the masks on English mathematics and target-language conversation, without target-language mathematics in mask training. CWF reaches 43.12% MATH-Id and 27.68% MATH-Th, improving over the stronger input expert by 13.98 and 15.70 percentage points, and over the strongest competing merger by 4.46 and 6.90 points, respectively (Table 1). This is a useful evaluation of complementary capabilities within the same task. Reporting English and general-capability retention alongside these gains also makes the resulting trade-offs visible.

Critical weaknesses

1. CWF’s advantage over the closely related DAM method is not established. DAM already learns column-wise merging coefficients and studies language–mathematics and code–multilingual combinations (Gauthier-Caron et al., 2024). The paper explains how CWF differs: it ties the Gate, Up, and Down coefficients within each neuron, introduces sharing rules for Q/O and shared K/V, and replaces DAM’s expert-output matching and regularization with mixed-domain cross-entropy. These are concrete methodological changes, but the main results do not include a direct comparison with DAM. Table 7 compares which modules are merged and does not isolate the benefit of coefficient binding. Consequently, the size of CWF’s improvement over this prior method, and the relative contributions of structural binding and the changed training objective, remain unclear.

2. There is no comparison with ordinary joint fine-tuning on the same data. CWF trains its mixture coefficients on English mathematics and target-language conversation. The same examples could also be mixed together to fine-tune a model directly. The paper does not provide this comparison: Seq-FT in Table 1 is sequential fine-tuning and does not establish how joint training would perform. A useful control would start from CWF’s initial merged checkpoint and apply LoRA fine-tuning on the same mixed data, then compare both new-task performance and retention of existing abilities. Without this experiment, CWF’s practical advantage over ordinary fine-tuning remains uncertain. Appendix C acknowledges the absence of this comparison.

Questions for authors
  1. What are CWF’s concrete advantages over DAM? The paper describes differences in coefficient binding and the training objective. On one existing setting, please compare CWF and DAM using the same experts and training data with comparable tuning budgets, reporting task performance, retention of existing abilities, and training cost. Is there evidence showing whether the gains mainly come from coefficient binding or from changing the training objective?
  2. How well does direct fine-tuning on the same mixed data perform? Starting from CWF’s initial merged checkpoint before mask training, please apply LoRA fine-tuning to exactly the same mixed data with comparable training and tuning budgets. How does it compare with CWF on target-language mathematics, English mathematics, general-capability retention, and training cost?
  3. Does expert merging retain an additional benefit as models become larger? (Scope clarification.) With greater model capacity and appropriate multi-domain training, a single model may more readily acquire knowledge and skills from multiple domains, potentially reducing the need to combine specialized experts. Could this reduce CWF’s additional benefit over joint fine-tuning or other learned merging methods at the same model size? The current experiments cover only 8B models. If results at other scales are already available, please include them; otherwise, please clarify the scope of the current conclusions and discuss when expert merging would remain useful as model size increases.
Reviewer_4K4AOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Component-wise Weighted Fusion (CWF), a model merging method designed to combine complementary capabilities from specialized LLMs. Instead of applying uniform or layer-wise merging coefficients, CWF learns fine-grained fusion weights for individual FFN neurons and attention heads. Specifically, it shares coefficients across the Gate, Up, and Down projections of each FFN neuron and introduces a coefficient-sharing mechanism for attention heads under Grouped-Query Attention (GQA). The original expert parameters remain frozen, and only the fusion coefficients are optimized using mixed-domain data. Experiments on cross-lingual mathematical reasoning, Code + Math, and Code + Multilingual tasks demonstrate improvements over several existing merging baselines, while producing a single dense model without additional inference-time overhead.

Critical strengths

Strengths

  1. The paper provides an intuitive motivation for incorporating Transformer component structures into model merging.
  2. The coefficient-sharing mechanisms for FFN neurons and attention heads are clearly formulated and reasonably motivated.
  3. Promising results on capability composition.
Critical weaknesses

Weakness

  1. Although Differentiable Adaptive Merging (DAM) is discussed in the Related Work section, it is not included in the experimental comparisons. A direct comparison with DAM would help establish whether CWF provides meaningful improvements over existing adaptive merging approaches.
  2. The current ablations show the benefits of joint FFN and attention fusion but do not directly validate the proposed coefficient-sharing mechanisms. Comparisons with independently learned Gate/Up/Down, Q/O, and K/V coefficients, ideally under matched parameter budgets, would help clarify whether the improvements come from structural design or simply fewer trainable parameters.
Questions for authors

Please refer to the weaknesses above.

Reviewer_Ju1uOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper describes a method for model merging for multi-task learning. The method works at a very granular level: the neuron-level in FFN networks and in GQA. It defines a way to use a mixed-domain dataset to identify per-neuron weights, learned through gradient-based optimization. Once these weights are learned, the merging is simply weighting the components of each.

Critical strengths
  1. High-performing model merging idea in comparison to alternatives
  2. Experimental breath, across different types of multi-task scenarios and two models.
Critical weaknesses
  1. Writing is very difficult to parse/understand:
    1. Intro is way way too concise, background & motivation is not properly expressed. L48 coefficients ? L81 how do they motivate ? L86 masks ?
    2. Related Work reads a list, doesn’t properly contextualize the work in the research landscape.
    3. Methodology doesn’t define terms, has half-written sentences. Not sure what is going on, what is being “fused” when.
    4. No justification provided for checkpoints selected, why is only one checkpoint trained when the rest are off-the-shelf ? How is it trained? Even appendix doesn’t provide much more info. The math expert’s results are pretty wild in Table 1, seems like the model is just learning to follow instructions (since base is not instruction-tuned).
    5. Section 4.2 is useless, it just lists out what’s written in the tables. Results section should interpret and explain results, not describe them.
    6. No conceptual or scientific discussion/narrative at all. The method is not motivated in any way, and experimental design is not justified (models, training, evaluation). Paper is descriptive and detailed, but is not clear or explanatory. There’s also no experimental ablations to understand the importance of all the detailed design choices.
  2. Low reproducibility and reliability. Reproducibility Statement points to Section 3, which itself just defines equations. L248-255 is incomprehensible, and very low detail. In addition, all these datasets have at most a few hundred samples, so using 4 significant figures and no discussion of uncertainty/variation feels quite misleading. For example, a one-question different on the 30-question AIME dataset is a 3.33% difference. I have very low faith in the overall empirical conclusions.
  3. Unlike many of the baselines, CWF requires gradient-based training of parameters. And compared to the others that do training, none require learning millions of vlaues (one for each neuron/head). Even with the results displayed, I have a hard time seeing such a complicated method being adopted when other, basic ideas perform almost as well. Merging, as I understand, is super useful/common because it allows the creation of new checkpoints to test with little cost.
Questions for authors
  1. Module 2 is specific to GQA, when many modern LLMs use MLA, gated GQA, or linear/recurrent forms (especially 2026 onwards). For which would this method work ?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.