- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper challenges the conventional view that aggregation is the main cause of performance degradation in deep GNNs. Through comparisons with MLPs, component ablations, and initialization studies, it argues that some effects attributed to aggregation instead arise from other network components and initialization choices. It also emphasizes that existing oversmoothing bounds are conditional on learned parameters and recommends exploring deeper GNNs for tasks requiring long-range information.
Critical strengths
- I agree with the paper’s central argument that aggregation should not be the default explanation for performance degradation in deep GNNs. Greater depth can help when a larger receptive field provides additional predictive information, but may hurt performance when it offers no benefit/noise for the task.
- The paper connects this argument to the transformation and activation components shared by GNNs and MLPs, as well as weight initialization where these factors are often overlooked or treated as minor implementation details in graph learning. Demonstrating their importance provides useful guidance for researchers training deep GNNs.
Critical weaknesses
- Showing that existing theory does not imply inevitable oversmoothing does not, by itself, rule out aggregation as a contributor to performance degradation. Although He initialization helps train deeper models, the paper’s experiments do not establish a predictive advantage over shallow alternatives, leaving the practical benefits of its recommendations unclear.
- The observation that aggregation helps preserve test performance as depth increases is interesting, but the paper offers limited explanation of why this occurs.
- Given these unresolved questions, it remains unclear whether the claimed “systematic re-examination” provides sufficient new insight beyond prior work to warrant publication at ICLR.
Questions for authors
Responses to the critical weaknesses raised above would help clarify my assessment. Additionally,
- What explains the greater robustness of GNN test performance to depth compared with MLPs?
- Appendix F states that learning rates are selected by training accuracy, while checkpoints are selected by validation accuracy. Is this intentional, or is it a typo?
- For the datasets using a 60% training split, do the main observations (particularly the greater performance robustness of deeper GNNs compared with MLPs) also hold at lower label rates which is common node classification setting (semi-supervised)?
4 comments
Q2. Appendix F states that learning rates are selected by training accuracy, while checkpoints are selected by validation accuracy. Is this intentional, or is it a typo?
A5. Statement in Appendix F (now Appendix G.1 in the revised version) was intentional. In the initial submission, the learning rate was selected by training accuracy in order to compare how well GNNs and MLPs can be optimized, by reporting the best training accuracy that each model can achieve, while the checkpoint for test accuracy was selected by validation accuracy. We agree that this was not explained clearly. In the revision, we select the learning rate by validation accuracy. The main reason is consistency with the new Section 4, where the hyperparameters include the dropout rate and weight decay, for which selection by training accuracy is not appropriate. The conclusion is unchanged: in both versions, GNNs maintain higher test accuracy than MLPs at large depths on most datasets. For a transparent comparison, the figures of the initial submission are preserved without any change to the results in Appendix J.
Q3. For the datasets using a 60% training split, do the main observations (particularly the greater performance robustness of deeper GNNs compared with MLPs) also hold at lower label rates which is common node classification setting (semi-supervised)?
A6. We added these experiments in Appendix E, with 20 labeled nodes per class on all nine datasets (the public split for Cora, CiteSeer, and PubMed), both in the vanilla setting (Figure 14) and with standard techniques (Figure 15). The observation holds. In the vanilla setting, the test accuracy of GCN and GAT degrades less than that of the MLP up to 32 layers on the homophilic datasets, while all models degrade at 64 layers. With standard techniques, the GNNs remain stable up to 64 layers and degrade no more than the MLP. As in the fully supervised setting, Roman-Empire is the exception in the vanilla setting, and on Squirrel and Amazon-Ratings all models stay close to the accuracy of predicting the majority class, so the comparison there is not informative.
Dear Reviewer,
Thank you again for your constructive feedback, which helped us significantly strengthen our manuscript. As the discussion period closes in about two days, we would greatly appreciate it if you could review our response and let us know whether it addresses your concerns. We remain happy to provide further clarification if needed.
Thanks for the additional experiments. They address several of my empirical concerns. I still think the paper falls short of the bar for ICLR in terms of what it adds to the community’s understanding and what it offers in practice. I’m keeping my original score of 2.
We sincerely thank the reviewer for the thoughtful review and for the specific questions, which helped us clarify and strengthen the paper. We have uploaded a revised version with additional experiments, and we address each of the reviewer’s concerns below.
W1-1. Showing that existing theory does not imply inevitable oversmoothing does not, by itself, rule out aggregation as a contributor to performance degradation.
A1-1. We agree. Section 3.2 only shows that the existing bounds do not make oversmoothing inevitable, and we do not use it as evidence that aggregation is harmless. That evidence is empirical: aggregation does not make the test accuracy of GNNs degrade more than that of MLPs on most datasets (Section 3.1), the rapid exponential decay of node similarity is driven by transformation and activation, not by aggregation (Section 3.3), and with standard techniques GNNs remain stable up to 64 layers (the new Section 4). We also state the remaining concern explicitly: in vanilla models, the training accuracy of GNNs decreases with depth earlier than that of MLPs (Section 3.1). The new Section 4 shows that this is resolved by standard techniques.
W1-2. Although He initialization helps train deeper models, the paper’s experiments do not establish a predictive advantage over shallow alternatives, leaving the practical benefits of its recommendations unclear.
A2. We added Section 4 to address this point. With skip connections, batch normalization, dropout, and weight decay, depth brings a clear advantage on tasks that require it. On Roman-Empire, the test accuracy of GCN improves from 77.2% with a single layer to 91.3% with 32 layers, while that of the MLP remains flat at every depth. On Minesweeper, the ROC-AUC of GCN improves from 88.3 to 98.4 with 64 layers. On the other datasets the benefit is mild, but deep models are not worse than shallow ones.
We also note that He initialization is not proposed as a way to improve accuracy. Its role is in the analysis of vanilla models, where Glorot initialization with ReLU causes zero-collapsing and confounds the effect of aggregation. With normalization, this problem does not arise, and the experiments in the new Section 4 use the default initialization.
W2&Q1. The observation that aggregation helps preserve test performance as depth increases is interesting, but the paper offers limited explanation of why this occurs. / What explains the greater robustness of GNN test performance to depth compared with MLPs?
A3. We do not have an established explanation for this observation. However, given that the concentration of representations within the same class helps generalization (Chuang et al., 2021), we hypothesize that aggregation contributes to this robustness by reducing the distance between the representations of neighboring nodes. At the same time, since concentration across different classes harms generalization (Chuang et al., 2021), whether this smoothing helps is likely to depend on the level of heterophily. We leave this direction as an open question in the new Section 5.
W3. Given these unresolved questions, it remains unclear whether the claimed “systematic re-examination” provides sufficient new insight beyond prior work to warrant publication at ICLR.
A4. Prior works question the severity of oversmoothing or build deep GNNs, but they do not reexamine the evidence on which the narrative was first built (we discuss this in the new Section 5 and review these works in Appendix A). Our contributions in this respect are: (i) a comparison with MLPs at every depth on nine datasets, which shows that the degradation attributed to aggregation is shared with or larger in MLPs; (ii) a reproduction of the depth experiment of Kipf & Welling (2017), showing that the reported drop appears only under particular seeds, where MLPs degrade even earlier (Appendix C); (iii) the finding that the exponential oversmoothing observed in previous empirical validations comes from zero-collapsing under Glorot initialization with ReLU, which also occurs in MLPs (Section 3.3); and (iv) the new Section 4, which extends the analysis to well trained models with depths of up to 64 layers.