acceptodds
Under review as a conference paper at ICLR 2027

Unified Merging of Heterogeneous Pretrained Models: From Layer Selection to Semantic Alignment

Abstract

As a novel technique for large pretrained models, model merging integrates multiple independently pretrained or fine-tuned models to construct a unified and efficient model without accessing the original training data or incurring an expensive computational cost. Existing model merging methods predominantly focus on parameter-level fusion within a fixed model architecture, facing inherent limitations in scalability and performance enhancement. While merging models at the architecture level shows promising potential, current approaches lack a clear and unified problem formulation, and suffer from exponential growth of the search space, and incompatibility of inter-layer feature representations. To address these challenges, we propose a unified architecture-parameter merging framework named UniMerge. Specifically, UniMerge formulates model merging as a Markov decision process and incorporates parameter-level merged candidates into an architecture-level search pool. To solve the formulated problem, UniMerge designs a state-enhanced embedding network to capture inter-layer parameter correlations and introduces a contrastive self-supervised learning-assisted actor-critic policy to improve exploration efficiency. To mitigate inter-layer incompatibility, we develop a glue layer based on dual knowledge distillation to ensure inter-layer semantic alignment and cross-layer feature representation. Extensive experiments on language and vision models demonstrate strong performance and search efficiency, while cross-scale experiments further show that UniMerge can stitch compatible models with different widths and depths, extending architecture-level merging from fine-tuning-induced heterogeneity to explicit structural heterogeneity.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Model merging, architecture merging, large pretrained model, knowledge distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5VanOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes UniMerge, a framework for architecture-level merging of heterogeneous pretrained models. It formulates layer selection as an MDP, uses parameter-derived layer embeddings and contrastive actor-critic search to select inference paths, and introduces knowledge-distillation-based glue layers to align representations across adjacent layers. Experiments cover language and vision models, including models with different widths and depths.

Critical strengths
  • The paper addresses two important challenges in architecture-level merging: searching over layer combinations and maintaining compatible representations when connecting layers from different models.

  • The experiments cover multiple model families and tasks, and the cross-scale experiments are a useful attempt to extend architecture-level merging beyond models with identical structures.

Critical weaknesses
  • Although UniMerge is compared with alternative search strategies under a matched candidate-evaluation budget, this does not ensure equal end-to-end computational cost. UniMerge incurs additional overhead from SVD preprocessing, representation learning, and PPO training, while its hyperparameters are selected through grid search on smaller models. A wall-clock-matched comparison that accounts for preprocessing, hyperparameter tuning, and search would better establish whether UniMerge provides a genuine efficiency advantage.

  • Generalization of the learned search policy. The PPO policy selects candidate layers based on their embeddings and previously selected layers, but it remains unclear whether a trained policy can generalize to unseen model architectures or candidate pools without retraining.

Questions for authors

Q1.Can the authors compare UniMerge with alternative search strategies under the same end-to-end wall-clock or compute budget, including preprocessing and hyperparameter tuning? Does UniMerge retain its performance advantage under such a comparison?

Q2.Does the learned PPO policy generalize to unseen model combinations without retraining, or must the policy and Glue layers be optimized separately for each merging task? Can the authors evaluate cross-task or cross-model transfer of a trained policy and compare its performance and computational cost with training a new policy from scratch?

Reviewer_87q9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Parameter-level merging only combines models with the same architecture and does not extend to heterogeneous ones. Architecture-level merging instead builds a new model by stacking layers taken from several models into a path from the first layer to the last, but the search space grows exponentially with depth and adjacent layers from different models mismatch in dimension, distribution and semantics. UniMerge represents each candidate layer by an embedding computed from the singular values of its weights and refined with a contrastive loss, and trains a policy with PPO that picks the next layer based on these embeddings. Between two layers from different models it inserts a small glue layer, trained on random Gaussian inputs so that its output matches what the next layer normally receives, both before and after passing through that layer. On three LLM families (Llama-2-7B, Qwen2.5-1.5B, Llama-2-13B) and two CLIP ViTs, UniMerge has the best average on every backbone, 1.09–1.58 points higher than HM3 on the LLMs and about 0.2 points higher than PCB Merging on the ViTs. When the selected layers are kept fixed and only the connector is changed, the glue layer improves task performance by 5.40 points over direct connection. When layers from a smaller and a larger model of the same family (e.g., Qwen2.5-Math-1.5B and 7B) are stitched together, the result is 1.27–3.47 points better than the smaller model alone on translation, but 3.03–24.68 points worse than it on general understanding, math and code (Table 9).

Critical strengths
  1. The problem is worth studying. Parameter-level merging only works for models with the same architecture, and merging at the architecture level is a natural way to go beyond this limitation. The paper also tries to extend it to models with different widths and depths, a setting that prior work such as HM3 [1] has not considered.

  2. The experiments are extensive. They cover three LLM families and two CLIP ViTs, compare against a dozen parameter-level and architecture-level baselines, and include additional analyses on out-of-domain tasks, stability and hyperparameter sensitivity (Appendix D.8–D.10).

  3. The paper is generally easy to follow. Splitting the problem into which layers to merge and how to connect them gives a clear structure, and Fig. 1 summarizes the challenges and the framework well.

Critical weaknesses
  1. Too many components for a small gain. The method stacks SVD-based layer embeddings, a Transformer policy trained with PPO, an LSTM contrastive loss and a distilled glue layer. Still, the improvement over the best baseline is 1.09–1.58 points on the LLMs and about 0.2 on the ViTs (Table 1), looks like some random fluctuation, and no variance is given. In the controlled comparison with HM3 [1] (Table 2, right), the full pipeline adds 1.71 points in total. The SVD state, which is sold as a main contribution, adds 0.29 over simple architecture metadata. Also, casting path search as an MDP solved by RL is not new; HM3 [1] already does this (L1334–1338). I am not convinced all these parts are needed and effective.

  2. Glue layer training is confusing. If I read Eq. 25–28 correctly, the glue input zl(s)=Qlmerged(ξa(s))z_l^{(s)} = Q_l^{\text{merged}}(\xi_a^{(s)}) and the target ubsource,(s)=ξb(s)u_b^{\text{source},(s)} = \xi_b^{(s)} are built from ξa(s)∼N(0,σa2I)\xi_a^{(s)} \sim \mathcal{N}(0, \sigma_a^2 I) and ξb(s)∼N(0,σb2I)\xi_b^{(s)} \sim \mathcal{N}(0, \sigma_b^2 I), which are sampled independently. Then zlz_l is not correlated with ubu_b, and the minimizer of Lcross=E[∥Gl(zl)−ub∥22]\mathcal{L}_{\text{cross}} = \mathbb{E}\big[\lVert \mathcal{G}_l(z_l) - u_b \rVert_2^2\big] is simply the constant Gl(zl)=E[ub∣zl]=E[ub]=0\mathcal{G}_l(z_l) = \mathbb{E}[u_b \mid z_l] = \mathbb{E}[u_b] = 0. But Table 10 shows the glue layer helps, so I suspect the implementation differs from what is written.

  3. Concerns about the experimental results.. Llama-2-7B and Qwen2.5-1.5B are different base models, yet their Pretrained rows share three identical values: 21.64 on WMT&IWS, 16.34 on GSM8K and 15.36 on HumanEval (Table 3 vs. Table 4). On ViT-B/32, TSV-Merge [4], KnOTS-Ties [5] and RobustMerge [6] are all worse than Task Arithmetic [3] (Table 6), even though the paper says they follow “their reported ViT configurations” (L357–358). I doubt these baselines were reproduced correctly.

  4. Over-stated cross-scale claims. The text says UniMerge “remains effective on math and code” (L414–415). In Table 9, however, the stitched model is 3.03–24.68 points below the smaller model alone on general understanding, math and code, and only beats it on translation. The larger model’s own scores are not shown. So I don’t think the claim of cross-scale is well supported.

References

[1] HM3: Hierarchical Multi-Objective Model Merging for Pretrained Models. Yu Zhou, Xingyu Wu, Jibin Wu, Liang Feng, Kay Chen Tan. NeurIPS 2025.

[2] Parameter Competition Balancing for Model Merging. Guodong Du, Junlin Lee, Jing Li, Runhua Jiang, Yifei Guo, Shuyang Yu, Hanting Liu, Sim Kuan Goh, Ho-Kin Tang, Daojing He, et al. NeurIPS 2024.

[3] Editing Models with Task Arithmetic. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, Ali Farhadi. ICLR 2023.

[4] Task Singular Vectors: Reducing Task Interference in Model Merging. Antonio Andrea Gargiulo, Donato Crisostomi, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, Emanuele Rodolà. CVPR 2025.

[5] Model Merging with SVD to Tie the KnOTS. George Stoica, Pratik Ramesh, Boglarka Ecsedi, Leshem Choshen, Judy Hoffman. ICLR 2025.

[6] RobustMerge: Parameter-Efficient Model Merging for MLLMs with Direction Robustness. Fanhu Zeng, Haiyang Guo, Fei Zhu, Li Shen, Hao Tang. NeurIPS 2025.

Questions for authors

Q1. Fig. 4 shows a clear depth-wise pattern: the base model in early blocks, task experts in the middle, and parameter-level merged models near the. How would path that simply follows this pattern perform? If it comes close to UniMerge, the RL search may not be necessary.

Q2. What data is used to compute the reward during search, how large is it, and does it overlap with the test sets? The hyperparameters are chosen on Qwen2.5-1.5B and ViT-B/32 according to the "Benchmark average" in Fig. 10–11. Is this computed on validation or test data? Did the parameter-level baselines get a comparable budget of data and training resources?

Q3. What does the final merged model look like? Specifically, how is the path length decided (is there a stop action?) how many glue layers are inserted?

Reviewer_3NZEOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2(−1)
Read review
Paper summary

The paper formulates architecture-level model merging as an MDP over fine-tuned and parameter-merged candidates. It uses SVD spectral states with contrastive actor-critic search for layer selection, and dual-distillation glue layers for layer alignment. Experiments cover language and vision backbones, including cross-scale width/depth heterogeneity.

Critical strengths

I find the combination of AutoML/NAS-style search and explicit layer alignment interesting for pretrained model merging. The distinction between selecting layers and making them work together makes the design easy to follow. The dual-distillation glue layers are well motivated, and the fixed-path experiments help demonstrate their contribution independently of the search. The optimal-transport analysis also offers a useful interpretation of the alignment losses. I particularly appreciate the cross-scale experiments, which go beyond differently fine-tuned models with the same architecture and examine compatible models with different widths and depths.

Critical weaknesses

The method reminds me of earlier AutoML/NAS approaches, so I would like to better understand what the more elaborate search adds. The grid and evolutionary search comparisons are useful, but how would simple random search perform with the same candidate pool, glue layers, and total compute budget?

I am still unsure how well alignment learned from Gaussian inputs carries over to real data. Could the authors clarify what the theoretical bounds tell us about downstream performance, and whether using real inputs for calibration would change the results?

The math and code failure analysis is helpful, but I would like a clearer explanation of the remaining performance loss. Is it mainly caused by the selected layers, imperfect alignment, or an interaction between the two?

The cross-scale experiments are interesting, although they focus on compatible models within the same family. What are the main obstacles to extending the method to more different models, such as Qwen and Llama?

Questions for authors

The residual-alignment results are helpful, but I am still unsure how they relate to the performance loss on math and code tasks. Do examples with larger alignment errors tend to fail more often, and could this explain the remaining performance gap between UniMerge and the source experts?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.