acceptodds
Under review as a conference paper at ICLR 2027

Training-Free Cross-Architecture Merging for Graph Neural Networks

Abstract

Model merging has emerged as a powerful paradigm for combining the capabilities of distinct expert models without the high computational cost of retraining, yet current methods are fundamentally constrained to homogeneous architectures. For GNNs, however, message passing is topology-dependent and sensitive to misalignment, making direct parameter-space merging unreliable. To bridge this gap, we introduce MOSAIC (Model Operator Space Alignment for Integration & Composition), a training-free framework that lifts merging from parameter space to operator space. We formalize Universal Message Passing Mixture (UMPM), a shared operator family that expresses heterogeneous GNN layers in a common functional language. MOSAIC enables cross-architecture GNN merging (e.g., GCNGAT) without retraining, retaining high specialist accuracy in most cases in compatible depth settings and achieving inference speedups of – over ensembles.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
graph neural networks, model merging, heterogeneous architectures, representation alignment, training-free learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_it3wOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors discuss the problem of merging two independently trained GNNs on the same graph, with varying architecture hyperparameters and message-passing operations to a single GNN in the forward pass phase. They propose a framework that matches layers from one model with another for interpolation in the forward pass. This is done by computing similarities between pairs of layer representations and finding an optimal similarity preserving path via dynamic programming, without any training. Alignment of one GNN layer to the coordinates of another is done via SVD, and results are interpolated linearly. The authors present experiments on both heterogenous merging (when two architectures differ in MP operations) and homogenous merging (two architectures based on the same operation, with varying hyperparameters).

Critical strengths

S1: The biggest strength of the paper is that it offers a training-free method. It uses only artifacts that can be obtained in two forward passes (one for each parent), which puts it ahead of methods that require actual training.

S2: The use of dynamic programming on the similarity between layers via CKA is a nice application of the technique.

Critical weaknesses

W1: The paper is notation-heavy, without any need for it, as it is not a theoretical paper and the techniques used in the method are well-known in ML literature. I would greatly advise the authors to revise the method part to improve the flow of the reading, omitting some technical stuff to the appendix in favor of more intuitive explanations with motivations for their choices.

W2: Literature review is insufficient. It generally cites existing work without actually explaining the challenges, pros and cons of these methods beyond generic one-liners, nor does it connect these previous work to the paper.

W3: For the experiments - there is no (“no-fusion”) baseline. The reader needs to know the capabilities of the two parent models independently to judge if the fusion is actually worth it. I would also advise adding some late-fusion methods based on the final logits of the parent models. This will help fill the heterogeneous baseline table as right now it is uninformative (while the claim of heterogeneous fusion is an important property of the method). I would also like to see edge-cases being tested. Let’s say one of the parents is heavily oversmoothed - does the merging method overcome this? Does it know to reduce itself to rely on the other parent? These are interesting properties to check empirically.

Questions for authors

see W1-W3 above

Reviewer_aAGPOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes MOSAIC, a training-free framework for merging GNNs with different architectures, depths, and widths. It introduces a unified message-passing operator space and combines layer alignment, closed-form parameter fusion, and kernel-based correction to produce a single merged model.

Critical strengths
  1. The paper addresses a meaningful problem by exploring how GNNs with different message-passing mechanisms can be merged without additional training or labeled data.

  2. It proposed unified message-passing representation provides a systematic way to handle architectural differences, while the closed-form merging procedure avoids gradient-based optimization.

Critical weaknesses
  1. Table 2 reports results for several existing model merging methods only under the same-architecture setting, while the cross-architecture evaluation includes only PLeaS. Moreover, the experiments primarily focus on the authors' constructed class-specialist setting, without evaluating more general model merging scenarios. This makes it difficult to assess the effectiveness and generalizability of MOSAIC.

  2. The manuscript repeatedly refers to Appendix A.4, Appendix Tables 6–7, and Algorithm 1, but none of these materials are included in the current manuscript.

Questions for authors
  1. Table 2 reports results for several existing model merging methods only under the same-architecture setting, while the cross-architecture evaluation includes only PLeaS. Moreover, the experiments primarily focus on the authors' constructed class-specialist setting, without evaluating more general model merging scenarios. This makes it difficult to assess the effectiveness and generalizability of MOSAIC.

  2. The manuscript repeatedly refers to Appendix A.4, Appendix Tables 6–7, and Algorithm 1, but none of these materials are included in the current manuscript.

Reviewer_xhs5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates graph model merging—specifically, how to merge two pre-trained GNNs with different architectures into a single model while preserving as much of their respective capabilities as possible, without retraining, using labels, or performing gradient optimization. The authors propose MOSAIC, which aims to achieve cross-architecture model merging through unified message-passing representations, network layer matching, hidden space alignment, closed-form parameter fusion, and message distribution correction. However, this paper appears to overlap significantly with GFFMERGE, published at ICML 2026, yet it lacks a direct comparison with GFFMERGE.

Critical strengths
  1. GNNs with different messaging architectures cannot be directly combined in the same way as networks in a homogeneous parameter space; this research question is significant.
  2. Describing GCN, GAT, GraphSAGE, and GIN within a common framework provides a practical implementation for combining parameters across different architectures.
  3. The experiments cover a broad range of scenarios. The authors not only tested isomorphic and heterogeneous architectures but also considered depth, width, node classification, and certain graph-level tasks.
Critical weaknesses
  1. Regarding the cross-architecture issues present in GNN merging, related work has already addressed this topic [1, 2, 3]. In particular, [1] is directly relevant to this paper but was not cited or compared. Additionally, the contributions of this paper require corresponding revisions.

[1] GFFMERGE: Efficient Merging of Graph Neural Force Fields and Beyond. Verma et al. ICML 2026. [2] Out-of-Distribution Graph Models Merging. Wang et al. ICLR 2026. [3] G-Merging: Graph Models Merging for Parameter-Efficient Multi-Task Knowledge Consolidation. Chen et al. ICLR 2026.

  1. The authors determine the weight assigned to each parent during merging based on its operator reconstruction residual. However, the reconstruction objective in Equation (6) is the sum of the parent’s own operator outputs. It remains unclear how this optimization method reflects the parent’s actual reliability, and it needs comparison against uniform fusion.

  2. Graph-level Peptides-struct is a regression task, but Appendix Table 5 still reports Retention; it remains unclear how MAE is converted to this metric.

Questions for authors

Please refer to the Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.