- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Selective Modular Merging (SMM), a training-free and data-free method for merging heterogeneous language models with different widths and depths. The method uses orthogonal Procrustes to align the source model's FFN weights to the target space, and a reference model to construct task vectors for selecting matched FFN neurons based on cosine similarity. It then selectively transfers FFN weights under a norm constraint while keeping the target attention parameters unchanged. Experiments on Qwen and Llama model families show improvements on several downstream tasks while largely preserving the target model's general capabilities.
Critical strengths
- The paper studies an interesting and practical problem of merging models with different sizes without requiring additional training or calibration data. This can be useful when training resources or data are unavailable.
Critical weaknesses
- Although the paper studies heterogeneous model merging, the proposed method is restricted to models within the same family with shared token embeddings. It is unclear how well the method would generalize to more heterogeneous settings, such as models from different families. While the training-free and data-free properties are appealing, I think this advantage should also be considered relative to methods that require only minimal training or a small amount of data, especially when such data can be relatively cheap to obtain.
- In Tables 1 and 2, SMM often performs very close to the original target model, and there are also cases where the performance decreases (e.g., GSM8K and HumanEval+). Given the relatively limited applicability of the method, I am not fully convinced that the performance improvement is strong enough compared with existing approaches that allow a small amount of training or calibration data. I would also suggest reporting average performance or improvement over the target for each transfer setting, which would make the overall gains easier to evaluate.
- I'm aware of the gain on Table 3 but the gain seems to be on a less popular benchmark.
- The justification for retaining target attention is not fully convincing. Section 3.4 argues that attention updates are context-dependent and their effect on the loss cannot be determined without data, based on Eqs. 5–6. However, I think the same argument can also apply to FFN updates, since their effects on downstream loss are also input-dependent. The analysis does not clearly explain why the proposed FFN updates should be safer than attention updates. Since much of the target model is preserved, the strong retention performance seems somewhat expected, and I would like to see stronger evidence that the selective FFN transfer provides meaningful benefits.
- Eq. 4 uses cosine similarity between the target task vector and the aligned source-reference difference to select FFN neurons. While I understand the intuition, it is unclear why this specific criterion should identify the most beneficial neurons to transfer. I think the paper needs a more direct comparison with simpler neuron matching or selection strategies to demonstrate the benefit of this design choice.
Questions for authors
See weaknesses above.