- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors discuss the problem of merging two independently trained GNNs on the same graph, with varying architecture hyperparameters and message-passing operations to a single GNN in the forward pass phase. They propose a framework that matches layers from one model with another for interpolation in the forward pass. This is done by computing similarities between pairs of layer representations and finding an optimal similarity preserving path via dynamic programming, without any training. Alignment of one GNN layer to the coordinates of another is done via SVD, and results are interpolated linearly. The authors present experiments on both heterogenous merging (when two architectures differ in MP operations) and homogenous merging (two architectures based on the same operation, with varying hyperparameters).
Critical strengths
S1: The biggest strength of the paper is that it offers a training-free method. It uses only artifacts that can be obtained in two forward passes (one for each parent), which puts it ahead of methods that require actual training.
S2: The use of dynamic programming on the similarity between layers via CKA is a nice application of the technique.
Critical weaknesses
W1: The paper is notation-heavy, without any need for it, as it is not a theoretical paper and the techniques used in the method are well-known in ML literature. I would greatly advise the authors to revise the method part to improve the flow of the reading, omitting some technical stuff to the appendix in favor of more intuitive explanations with motivations for their choices.
W2: Literature review is insufficient. It generally cites existing work without actually explaining the challenges, pros and cons of these methods beyond generic one-liners, nor does it connect these previous work to the paper.
W3: For the experiments - there is no (“no-fusion”) baseline. The reader needs to know the capabilities of the two parent models independently to judge if the fusion is actually worth it. I would also advise adding some late-fusion methods based on the final logits of the parent models. This will help fill the heterogeneous baseline table as right now it is uninformative (while the claim of heterogeneous fusion is an important property of the method). I would also like to see edge-cases being tested. Let’s say one of the parents is heavily oversmoothed - does the merging method overcome this? Does it know to reduce itself to rely on the other parent? These are interesting properties to check empirically.
Questions for authors
see W1-W3 above