acceptodds
Withdrawn from ICLR 2027

Selective Modular Merging of Heterogeneous Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Model merging offers a training-free strategy for model adaptation and knowledge transfer, with a significant efficiency advantage over existing post-training methods (such as reinforcement learning-based distillation and supervised fine-tuning). However, current merging methods either apply to homogeneous models only or require additional data and training costs, limiting their feasibility and practical usefulness. In this study, we propose Selective Modular Merging (SMM), a simple yet effective training-free method for merging heterogeneous language models. In principle, given a task-specific student model and its corresponding task vector, SMM further improves performance on the task by selectively merging a heterogeneous but more powerful source model into the target module by module. For each of the target's FFN layers, SMM aligns the source's grouped FFN layers to it via orthogonal Procrustes, then selects the most informative one to merge in an optimization framework based on the task vector. For the target's attention layers, SMM retains them because of their contextual dependency, a choice supported by the same framework. We evaluate SMM on merging heterogeneous Llama and Qwen models of different sizes. Experiments across language and mathematical reasoning tasks show that SMM helps target models approach source performance while keeping their small model sizes, demonstrating that independently trained heterogeneous language models can serve as reusable carriers for training-free knowledge transfer and enhancement.Our code is available at https://anonymous.4open.science/r/SMM-code-97BC

Primary area:
foundation or frontier models, including LLMs
Keywords:
Knowledge transfer, Model Merging, Large Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zPHQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Selective Modular Merging (SMM), a training-free and data-free method for merging heterogeneous language models with different widths and depths. The method uses orthogonal Procrustes to align the source model's FFN weights to the target space, and a reference model to construct task vectors for selecting matched FFN neurons based on cosine similarity. It then selectively transfers FFN weights under a norm constraint while keeping the target attention parameters unchanged. Experiments on Qwen and Llama model families show improvements on several downstream tasks while largely preserving the target model's general capabilities.

Critical strengths
  1. The paper studies an interesting and practical problem of merging models with different sizes without requiring additional training or calibration data. This can be useful when training resources or data are unavailable.
Critical weaknesses
  1. Although the paper studies heterogeneous model merging, the proposed method is restricted to models within the same family with shared token embeddings. It is unclear how well the method would generalize to more heterogeneous settings, such as models from different families. While the training-free and data-free properties are appealing, I think this advantage should also be considered relative to methods that require only minimal training or a small amount of data, especially when such data can be relatively cheap to obtain.
  2. In Tables 1 and 2, SMM often performs very close to the original target model, and there are also cases where the performance decreases (e.g., GSM8K and HumanEval+). Given the relatively limited applicability of the method, I am not fully convinced that the performance improvement is strong enough compared with existing approaches that allow a small amount of training or calibration data. I would also suggest reporting average performance or improvement over the target for each transfer setting, which would make the overall gains easier to evaluate.
  • I'm aware of the gain on Table 3 but the gain seems to be on a less popular benchmark.
  1. The justification for retaining target attention is not fully convincing. Section 3.4 argues that attention updates are context-dependent and their effect on the loss cannot be determined without data, based on Eqs. 5–6. However, I think the same argument can also apply to FFN updates, since their effects on downstream loss are also input-dependent. The analysis does not clearly explain why the proposed FFN updates should be safer than attention updates. Since much of the target model is preserved, the strong retention performance seems somewhat expected, and I would like to see stronger evidence that the selective FFN transfer provides meaningful benefits.
  2. Eq. 4 uses cosine similarity between the target task vector and the aligned source-reference difference to select FFN neurons. While I understand the intuition, it is unclear why this specific criterion should identify the most beneficial neurons to transfer. I think the paper needs a more direct comparison with simpler neuron matching or selection strategies to demonstrate the benefit of this design choice.
Questions for authors

See weaknesses above.

Reviewer_CxEnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Selective Modular Merging, which transfers selected feed-forward weights from a larger language model into a smaller target without additional training. It aligns their representations using shared vocabulary embeddings and selects source contributions using the target-reference weight difference. The smaller model retains its structure, parameter count, and attention weights.

Experiments transfer from Qwen2.5 7B models into Qwen2.5-1.5B-Instruct and from Llama-3.1-8B-Instruct into a Malay-adapted Llama-3.2-1B model. Several benchmarks improve, but substantial gaps remain relative to the larger sources. On MATH-500, merging improves the target from 49.40% to 53.00%, compared with the source’s 84.20%, closing only about 10% of the original performance gap. On IFEval, the merged target reaches 42.96%, compared with 70.21% for the source. GSM8K and HumanEval+ performance decrease after merging. Although some gap is expected because the target is smaller, these results provide limited support for the claim that merged targets approach source performance.

Critical strengths
  1. The method addresses a useful efficiency problem. For a fixed configuration, it constructs a merged checkpoint without calibration forward passes or gradient updates, while preserving the target architecture and inference cost. The reported core merging times of approximately 23 to 35 seconds make the approach potentially practical, although these times exclude checkpoint I/O, configuration selection, and evaluation.

  2. The experiments provide evidence that direct parameter transfer can improve selected capabilities across model sizes. The MATH-500 improvement from 49.40% to 53.00% and the MalayMMLU improvement from 41.91% to 47.98% are encouraging. The aggregate general-capability scores also remain close to those of the original Qwen target.

Critical weaknesses
  1. The experiments do not sufficiently isolate the proposed mechanism or establish its generality.

The SVD solution is valid for the stated objective, but this does not establish that a single map derived from vocabulary anchors provides useful correspondences throughout the network. The authors motivate this choice using prior work and residual-coordinate consistency, but direct experimental validation remains limited. I found no reported comparison against a random semi-orthogonal map or a map computed from shuffled token correspondences. These controls would help determine whether meaningful vocabulary alignment drives the improvements.

Similarly, controlled comparisons against random neuron matching and direct weight-similarity matching would help establish the contribution of task-vector selection. These comparisons should use equivalent coverage, comparable update magnitudes, and equal configuration-search budgets. The module ablation in Table 4 is informative, but the variants use different perturbation budgets, as the authors acknowledge. It therefore supports conclusions about the evaluated configurations rather than a general advantage of FFN-only transfer. Evaluation also remains restricted to transfers within the Qwen and Llama families, leaving applicability across model families and substantially different architectures unverified.

  1. The evidence for reliable improvement is limited by mixed results and the evaluation protocol.

SMM improves five of seven Qwen capability benchmarks, but GSM8K decreases from 72.48% to 71.65% and HumanEval+ decreases from 47.56% to 45.73%. Most gains are modest, and the reported point estimates do not establish their statistical reliability. Substantial gaps also remain relative to the larger sources. For example, MATH-500 improves from 49.40% to 53.00%, while the source achieves 84.20%. Although the smaller target’s capacity may explain some of this gap, the results provide limited support for the claim that merged targets approach source performance.

Each domain uses a separately selected checkpoint, so the results do not demonstrate one merged model with all reported improvements. The authors also acknowledge retrospective evaluation and prior inspection of the Malay benchmark. Configuration selection uses development performance, meaning that the data-free property applies to constructing a fixed merge, while selecting its configuration still uses evaluation data. This distinction should be reflected in the claims and computational cost reporting. Evaluation on previously untouched benchmarks, with configurations fixed in advance and uncertainty estimates for the score differences, would strengthen the evidence for reliable improvements.

Questions for authors
  1. How much do the gains depend on meaningful vocabulary alignment? Do you have comparisons against a random semi-orthogonal map or a map computed from shuffled token correspondences, while keeping the remaining merging procedure and update budgets comparable?

  2. What evidence isolates the benefit of task-vector selection? With the alignment map fixed, how does the proposed selection rule compare with random neuron matching and direct weight-similarity matching under equivalent coverage and comparable update magnitudes?

  3. What evidence supports applying the same alignment map across all layers? Beyond the cited prior work and residual-coordinate argument, have you examined whether the map aligns intermediate representations in the evaluated models, and whether that alignment relates to downstream improvements or regressions?

  4. How dependent are the results on the target-reference pair? The Qwen task vector captures instruction tuning rather than a specific mathematical or coding adaptation. Have you tested whether a more domain-specific target-reference pair improves transfer, particularly on GSM8K and HumanEval+, where the current merges reduce performance?

Reviewer_XxvuOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper aims to proposes a training-free and data-free approach for LLMs merging. Existing approaches only merge models of same architecture and the paper claims to merge models of heterogeneous architectures.

Critical strengths
  • The paper addresses relevant challenges of transferring capabilities between heterogeneous models
  • Related work is well written, definition of model merging and existing work is provided
Critical weaknesses
  • The distinction between model merging and knowledge distillation is unclear, making it difficult to assess the proposed method
  • The paper appears to conflate model merging with knowledge distillation.
  • The paper claims to address heterogeneous model merging, but the evaluated models include variant from the same family
  • Tables 1-3 do not provide sufficient information on what is the source, the target, the reference, etc.
  • Technical abbreviations such as FFN are not defined before first use (including in the abstract)
Questions for authors
  • What precisely does model merging mean in the paper?
  • How do the paper distinguish model merging from knowledge distillation?
  • Could the paper evaluate the merging of models of different families (e.g., Qwen and Llama)?
  • Line 085: "Instead of directly interpolating incompatible parameters ... transfer": How is this possible given that the models has different architecture

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.