- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Component-wise Weighted Fusion (CWF), which learns convex mixtures of corresponding components from two experts derived from a shared backbone. Each FFN neuron shares its mixture coefficients across Gate, Up, and Down. Attention shares coefficients between each query head’s Q/O slices and uses group-averaged coefficients for shared K/V heads. Expert parameters remain frozen while the mixing logits are optimized using mixed-domain cross-entropy. The resulting projections are materialized into one dense model. Experiments study mathematics in Indonesian and Thai, as well as code–math and code–multilingual combinations, with additional retention, ablation, and neuron-removal analyses.
Critical strengths
1. A clear and practical structural parameterization. CWF specifies a coherent coefficient-sharing scheme for both gated FFNs and GQA while preserving the backbone’s inference architecture (Section 3). The evaluated models require fewer than one million trainable mixing logits, and the learned projections can be stored in a standard checkpoint without additional expert forward passes at inference. This is a practical way to reuse specialized checkpoints while retaining single-model deployment.
2. Meaningful reported gains on a relevant composition setting. The cross-lingual mathematics experiments train the masks on English mathematics and target-language conversation, without target-language mathematics in mask training. CWF reaches 43.12% MATH-Id and 27.68% MATH-Th, improving over the stronger input expert by 13.98 and 15.70 percentage points, and over the strongest competing merger by 4.46 and 6.90 points, respectively (Table 1). This is a useful evaluation of complementary capabilities within the same task. Reporting English and general-capability retention alongside these gains also makes the resulting trade-offs visible.
Critical weaknesses
1. CWF’s advantage over the closely related DAM method is not established. DAM already learns column-wise merging coefficients and studies language–mathematics and code–multilingual combinations (Gauthier-Caron et al., 2024). The paper explains how CWF differs: it ties the Gate, Up, and Down coefficients within each neuron, introduces sharing rules for Q/O and shared K/V, and replaces DAM’s expert-output matching and regularization with mixed-domain cross-entropy. These are concrete methodological changes, but the main results do not include a direct comparison with DAM. Table 7 compares which modules are merged and does not isolate the benefit of coefficient binding. Consequently, the size of CWF’s improvement over this prior method, and the relative contributions of structural binding and the changed training objective, remain unclear.
2. There is no comparison with ordinary joint fine-tuning on the same data. CWF trains its mixture coefficients on English mathematics and target-language conversation. The same examples could also be mixed together to fine-tune a model directly. The paper does not provide this comparison: Seq-FT in Table 1 is sequential fine-tuning and does not establish how joint training would perform. A useful control would start from CWF’s initial merged checkpoint and apply LoRA fine-tuning on the same mixed data, then compare both new-task performance and retention of existing abilities. Without this experiment, CWF’s practical advantage over ordinary fine-tuning remains uncertain. Appendix C acknowledges the absence of this comparison.
Questions for authors
- What are CWF’s concrete advantages over DAM? The paper describes differences in coefficient binding and the training objective. On one existing setting, please compare CWF and DAM using the same experts and training data with comparable tuning budgets, reporting task performance, retention of existing abilities, and training cost. Is there evidence showing whether the gains mainly come from coefficient binding or from changing the training objective?
- How well does direct fine-tuning on the same mixed data perform? Starting from CWF’s initial merged checkpoint before mask training, please apply LoRA fine-tuning to exactly the same mixed data with comparable training and tuning budgets. How does it compare with CWF on target-language mathematics, English mathematics, general-capability retention, and training cost?
- Does expert merging retain an additional benefit as models become larger? (Scope clarification.) With greater model capacity and appropriate multi-domain training, a single model may more readily acquire knowledge and skills from multiple domains, potentially reducing the need to combine specialized experts. Could this reduce CWF’s additional benefit over joint fine-tuning or other learned merging methods at the same model size? The current experiments cover only 8B models. If results at other scales are already available, please include them; otherwise, please clarify the scope of the current conclusions and discuss when expert merging would remain useful as model size increases.