acceptodds
Under review as a conference paper at ICLR 2027

Stop Blaming Aggregation in Deep GNNs Based on Incomplete Evidence

31 views

Abstract

This paper argues that aggregation mechanisms in Graph Neural Networks (GNNs) have been unfairly blamed for limiting network depth based on incomplete evidence. The prevailing narrative has led researchers to view aggregation as fundamentally problematic for deep GNNs. However, through reexamining the experimental evidence and theoretical explanations by proper ablations, we show that aggregation's negative effects have been exaggerated while its benefits have been overlooked. Based on our findings, we advocate two actions: First, practitioners should actively deploy deep GNNs on tasks that require large capacity or long-range dependencies, whose potential has already been demonstrated yet remains overlooked. Second, researchers should rigorously investigate the true effects of aggregation on deep GNNs rather than accepting conventional wisdom.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
deep gnn, aggregation, oversmoothing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Xg3EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper challenges the conventional view that aggregation is the main cause of performance degradation in deep GNNs. Through comparisons with MLPs, component ablations, and initialization studies, it argues that some effects attributed to aggregation instead arise from other network components and initialization choices. It also emphasizes that existing oversmoothing bounds are conditional on learned parameters and recommends exploring deeper GNNs for tasks requiring long-range information.

Critical strengths
  • I agree with the paper’s central argument that aggregation should not be the default explanation for performance degradation in deep GNNs. Greater depth can help when a larger receptive field provides additional predictive information, but may hurt performance when it offers no benefit/noise for the task.
  • The paper connects this argument to the transformation and activation components shared by GNNs and MLPs, as well as weight initialization where these factors are often overlooked or treated as minor implementation details in graph learning. Demonstrating their importance provides useful guidance for researchers training deep GNNs.
Critical weaknesses
  • Showing that existing theory does not imply inevitable oversmoothing does not, by itself, rule out aggregation as a contributor to performance degradation. Although He initialization helps train deeper models, the paper’s experiments do not establish a predictive advantage over shallow alternatives, leaving the practical benefits of its recommendations unclear.
  • The observation that aggregation helps preserve test performance as depth increases is interesting, but the paper offers limited explanation of why this occurs.
  • Given these unresolved questions, it remains unclear whether the claimed “systematic re-examination” provides sufficient new insight beyond prior work to warrant publication at ICLR.
Questions for authors

Responses to the critical weaknesses raised above would help clarify my assessment. Additionally,

  • What explains the greater robustness of GNN test performance to depth compared with MLPs?
  • Appendix F states that learning rates are selected by training accuracy, while checkpoints are selected by validation accuracy. Is this intentional, or is it a typo?
  • For the datasets using a 60% training split, do the main observations (particularly the greater performance robustness of deeper GNNs compared with MLPs) also hold at lower label rates which is common node classification setting (semi-supervised)?
4 comments

We sincerely thank the reviewer for the thoughtful review and for the specific questions, which helped us clarify and strengthen the paper. We have uploaded a revised version with additional experiments, and we address each of the reviewer’s concerns below.


W1-1. Showing that existing theory does not imply inevitable oversmoothing does not, by itself, rule out aggregation as a contributor to performance degradation.

A1-1. We agree. Section 3.2 only shows that the existing bounds do not make oversmoothing inevitable, and we do not use it as evidence that aggregation is harmless. That evidence is empirical: aggregation does not make the test accuracy of GNNs degrade more than that of MLPs on most datasets (Section 3.1), the rapid exponential decay of node similarity is driven by transformation and activation, not by aggregation (Section 3.3), and with standard techniques GNNs remain stable up to 64 layers (the new Section 4). We also state the remaining concern explicitly: in vanilla models, the training accuracy of GNNs decreases with depth earlier than that of MLPs (Section 3.1). The new Section 4 shows that this is resolved by standard techniques.


W1-2. Although He initialization helps train deeper models, the paper’s experiments do not establish a predictive advantage over shallow alternatives, leaving the practical benefits of its recommendations unclear.

A2. We added Section 4 to address this point. With skip connections, batch normalization, dropout, and weight decay, depth brings a clear advantage on tasks that require it. On Roman-Empire, the test accuracy of GCN improves from 77.2% with a single layer to 91.3% with 32 layers, while that of the MLP remains flat at every depth. On Minesweeper, the ROC-AUC of GCN improves from 88.3 to 98.4 with 64 layers. On the other datasets the benefit is mild, but deep models are not worse than shallow ones.

We also note that He initialization is not proposed as a way to improve accuracy. Its role is in the analysis of vanilla models, where Glorot initialization with ReLU causes zero-collapsing and confounds the effect of aggregation. With normalization, this problem does not arise, and the experiments in the new Section 4 use the default initialization.


W2&Q1. The observation that aggregation helps preserve test performance as depth increases is interesting, but the paper offers limited explanation of why this occurs. / What explains the greater robustness of GNN test performance to depth compared with MLPs?

A3. We do not have an established explanation for this observation. However, given that the concentration of representations within the same class helps generalization (Chuang et al., 2021), we hypothesize that aggregation contributes to this robustness by reducing the distance between the representations of neighboring nodes. At the same time, since concentration across different classes harms generalization (Chuang et al., 2021), whether this smoothing helps is likely to depend on the level of heterophily. We leave this direction as an open question in the new Section 5.


W3. Given these unresolved questions, it remains unclear whether the claimed “systematic re-examination” provides sufficient new insight beyond prior work to warrant publication at ICLR.

A4. Prior works question the severity of oversmoothing or build deep GNNs, but they do not reexamine the evidence on which the narrative was first built (we discuss this in the new Section 5 and review these works in Appendix A). Our contributions in this respect are: (i) a comparison with MLPs at every depth on nine datasets, which shows that the degradation attributed to aggregation is shared with or larger in MLPs; (ii) a reproduction of the depth experiment of Kipf & Welling (2017), showing that the reported drop appears only under particular seeds, where MLPs degrade even earlier (Appendix C); (iii) the finding that the exponential oversmoothing observed in previous empirical validations comes from zero-collapsing under Glorot initialization with ReLU, which also occurs in MLPs (Section 3.3); and (iv) the new Section 4, which extends the analysis to well trained models with depths of up to 64 layers.

Q2. Appendix F states that learning rates are selected by training accuracy, while checkpoints are selected by validation accuracy. Is this intentional, or is it a typo?

A5. Statement in Appendix F (now Appendix G.1 in the revised version) was intentional. In the initial submission, the learning rate was selected by training accuracy in order to compare how well GNNs and MLPs can be optimized, by reporting the best training accuracy that each model can achieve, while the checkpoint for test accuracy was selected by validation accuracy. We agree that this was not explained clearly. In the revision, we select the learning rate by validation accuracy. The main reason is consistency with the new Section 4, where the hyperparameters include the dropout rate and weight decay, for which selection by training accuracy is not appropriate. The conclusion is unchanged: in both versions, GNNs maintain higher test accuracy than MLPs at large depths on most datasets. For a transparent comparison, the figures of the initial submission are preserved without any change to the results in Appendix J.


Q3. For the datasets using a 60% training split, do the main observations (particularly the greater performance robustness of deeper GNNs compared with MLPs) also hold at lower label rates which is common node classification setting (semi-supervised)?

A6. We added these experiments in Appendix E, with 20 labeled nodes per class on all nine datasets (the public split for Cora, CiteSeer, and PubMed), both in the vanilla setting (Figure 14) and with standard techniques (Figure 15). The observation holds. In the vanilla setting, the test accuracy of GCN and GAT degrades less than that of the MLP up to 32 layers on the homophilic datasets, while all models degrade at 64 layers. With standard techniques, the GNNs remain stable up to 64 layers and degrade no more than the MLP. As in the fully supervised setting, Roman-Empire is the exception in the vanilla setting, and on Squirrel and Amazon-Ratings all models stay close to the accuracy of predicting the majority class, so the comparison there is not informative.

Dear Reviewer,

Thank you again for your constructive feedback, which helped us significantly strengthen our manuscript. As the discussion period closes in about two days, we would greatly appreciate it if you could review our response and let us know whether it addresses your concerns. We remain happy to provide further clarification if needed.

Thanks for the additional experiments. They address several of my empirical concerns. I still think the paper falls short of the bar for ICLR in terms of what it adds to the community’s understanding and what it offers in practice. I’m keeping my original score of 2.

Reviewer_WeF2OpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
1
Presentation
2(−1)
Soundness
1
Read review
Paper summary

This work studies the causes of depth degradation in graph neural networks. Particularly, they bring to attention that the effects are also shared by multilayer perceptrons (that is, by removing the aggregation part), and even that they are less performant for larger depths with respect to graph models. They also challenge the assumption that oversmoothing is exponential with respect to the number of layers; particularly, that there is less smoothing in linear architectures than in nonlinear ones, and thus the smoothing does not come from aggregation. Finally, they urge other researchers to study depth with respect to modern optimization techniques.

Critical strengths

I agree with some conclusions of the paper, particularly, that GNNs have less trouble with depth than usually reported, and that oversmoothing is intrinsically linked to learning similar representations. I also agree with the final conclusion that researchers should focus on studying depth with respect to other architectural components in tasks that require it provably, and that there are not many works on it yet, though there are prior benchmarks that call for the same thing.

Critical weaknesses

I do not agree with the empirical support of the paper. All the models are undertrained, and all the reported failures seem to come from worse or not enough optimization. See Figure 5, where training degrades with depth, I assume that the larger models are not given proportionally more epochs or more width to be able to fit the training samples. See Roman Empire, which, for instance, in an established benchmark (Luo et al., NeurIPS 2024) has around 90% test accuracy at around 10 layers, and here it is around 20%, with low training accuracy. The same goes for MLPs, which are known to require different hyperparameters than GNNs on the same benchmarks (see e.g. Rubio-Madrigal et al., ICML 2026), but can get much higher training accuracy (and test) than reported here.

Moreover, a big part of the paper is a discussion of opinions with little actionable points: for instance, asking researchers to study "when and why depth helps", while I would have expected this paper to have a better conclusion to this question, rather than: it doesn't hurt as much as sometimes stated.

Questions for authors

How do the authors think that the effects of aggregation on optimization can be disentangled in a way that remains competitive? That is, if a model is restricted from an element that makes it competitive on that benchmark.

I did not entirely get the stance of the authors on whether they agree with repeated aggregation being problematic in any sense, or if depth is problematic in any sense, and if they mean propagation depth, or computational depth, which I feel are entangled here.

7 comments

We sincerely thank the reviewer for the careful and constructive review. The comments on the empirical support and on the clarity of our position led us to add new experiments and to restructure the paper, and we believe that the paper has become considerably stronger. We have uploaded a revised version, and we address each of the reviewer’s concerns below.


W1. I do not agree with the empirical support of the paper. All the models are undertrained, [...] can get much higher training accuracy (and test) than reported here.

A1. Motivated by the reviewer's comment, we have added a new section (Section 4) with experiments that build on the settings of the works the reviewer referred to (Luo et al., 2024; Rubio-Madrigal & Burkholz, 2026), which can be found in the uploaded revision. These experiments show that the observations of Section 3.1 still hold with well trained models. We also clarify that the low test accuracy in Section 3.1 (Figure 5 of the initial submission, now merged into Figure 1) reflects the limits of vanilla models, rather than insufficient training due to a lack of epochs or width.

New experiments with well trained models (new Section 4). We repeated the depth experiments after applying skip connections, batch normalization, and dropout, which are also adopted in Luo et al. (2024). While they explore up to 20 layers on Roman-Empire and Minesweeper and up to 10 layers on the other datasets, we extend the depth to 64 layers on all datasets and compare the GNNs with MLPs at every depth. For the MLP, we additionally apply weight decay, as in Rubio-Madrigal & Burkholz (2026), and use a larger hyperparameter search space than for the GNNs, following the reviewer's comment that MLPs require different hyperparameters. The results are shown in Figures 4 and 5 in the Section 4 of the revised version, and the detailed experimental setups, including the architecture and the hyperparameter search spaces, are provided in Appendix G.3.

  • The observations of Section 3.1 hold with well trained models. Aggregation still does not cause performance degradation as depth increases: the test accuracy of GCN and GAT remains stable up to 64 layers on all nine datasets, and the GNNs are at least as stable as the MLP. As in Section 3.1, the GNNs also maintain higher test accuracy than the MLP at large depths on most datasets.

  • These experiments further strengthen our argument. Beyond confirming the observations of Section 3.1, they resolve the concerns that remained in the vanilla setting. First, the optimization problem of deep vanilla GNNs disappears: the training accuracy of GCN and GAT no longer decreases with depth. It stays above 96% with two or more layers on eight datasets, and on Roman-Empire it increases with depth and exceeds 99% with eight or more layers. Second, depth becomes beneficial on some tasks. On Roman-Empire, GCN and GAT reach 91.3% and 91.0% at 32 layers with the same hidden dimension of 64 as in Section 3.1, matching Luo et al. (2024), while the MLP remains flat, and on Minesweeper GCN attains its best performance at 64 layers.

Why the accuracy in Section 3.1 is lower than in the benchmarks. All models in Section 3.1 are vanilla architectures without skip connections, normalization, dropout, or weight decay, whereas the works the reviewer compares with use such standard techniques: Luo et al. (2024) use residual connections, normalization, and dropout, and Rubio-Madrigal & Burkholz (2026) use normalization, dropout, and weight decay for their MLPs. The training budget itself is not small. Each model is trained for up to 2,000 epochs, with early stopping only after 1,000 epochs without improvement in validation accuracy, and the learning rate is selected among four values at every depth. On Roman-Empire, for example, the validation accuracy of vanilla GCN and GAT with 8 and 16 layers peaks within the first 260 epochs and then decreases, while their training accuracy keeps increasing for more than 1,000 additional epochs (Appendix I). Longer training would therefore raise their training accuracy further, but would not improve their test accuracy. Moreover, the new Section 4 uses the same hidden dimension, and the training accuracy recovers once the standard techniques are applied. The gap to the benchmark numbers therefore comes from the vanilla architecture, not from epochs or width.

Why we keep Section 3.1. Experiments with standard techniques alone would leave open the concern of whether the negative effects of aggregation are merely masked by those techniques. Section 3.1 addresses this concern. By removing every other component that affects the behavior of deep models, the comparison between GNNs and MLPs becomes a controlled ablation of aggregation alone. We therefore kept Section 3.1 and added the new Section 4 next to it, instead of replacing one with the other.

W2. Moreover, a big part of the paper is a discussion of opinions with little actionable points: for instance, asking researchers to study "when and why depth helps", while I would have expected this paper to have a better conclusion to this question, rather than: it doesn't hurt as much as sometimes stated.

A2. We agree that the action items were not clearly visible in the initial version. Following the reviewer's comment, we organized our action items, stated implicitly and scattered across the paper, explicitly in a single section (new Section 5) of about one page. The space gained is used for the new Section 4, which reports the experiments motivated by the reviewer's comment in W1.

For example, the new Section 5 states the following action items. First, understanding the actual effects of aggregation requires proper ablations with MLPs as a control, since effects that GNNs share with MLPs can otherwise be attributed to aggregation, as Section 3 shows for the performance degradation and the rapid collapse of node similarity. Second, deep GNNs should be explored and studied on top of standard techniques, since these techniques resolve the optimization problem of deep vanilla GNNs up to 64 layers. Third, smoothing should be analyzed in connection with optimization and generalization, since current node similarity measures only capture the complete convergence of representations.

Regarding "when and why depth helps", the revised version offers more on this question than "it doesn't hurt as much as sometimes stated". On when, the new Section 4 shows that, with standard techniques, depth is clearly beneficial on Roman-Empire and Minesweeper, and the comparison with the MLP at every depth shows that this benefit arises only in the presence of aggregation. On why, the new Section 5 suggests a clue: smoothing should be connected to optimization and generalization, possibly through the facts that the cross-entropy loss encourages representations of the same class to cluster, and that generalization is linked to concentration within classes and separation between classes (Chuang et al., 2021), which we leave as an open question.


Q1. How do the authors think that the effects of aggregation on optimization can be disentangled in a way that remains competitive? That is, if a model is restricted from an element that makes it competitive on that benchmark.

A3. This is the design of the new Section 4. All models share the same architecture with standard techniques, and the MLP is obtained by removing only aggregation. Each model is then tuned separately at each depth, with a larger search space for the MLP, so no model is restricted in its optimization. In this setting the GNNs are competitive (91.3% on Roman-Empire, matching Luo et al. (2024)), and the comparison with the MLP at the same depth isolates the effect of aggregation. For example, on Roman-Empire the training accuracy of the GNNs exceeds that of the MLP at every depth, which shows that aggregation does not hinder optimization once these techniques are applied.


Q2. I did not entirely get the stance of the authors on whether they agree with repeated aggregation being problematic in any sense, or if depth is problematic in any sense, and if they mean propagation depth, or computational depth, which I feel are entangled here.

A4. Our stance is that we disagree with the completeness of the evidence that repeated aggregation is harmful, rather than with the claim itself. The evidence is incomplete precisely because of the entanglement that the reviewer points out: it was obtained in settings where propagation depth and computational depth grow together, yet its effects were attributed to propagation depth alone.

As for whether repeated aggregation or depth is problematic in any sense, our answer is yes, in some sense. In Section 3.1, we observe that aggregation can make deep vanilla models harder to optimize, and computational depth also causes the degradation and the rapid collapse that GNNs share with MLPs (Sections 3.1 and 3.3). However, these problems are resolved by standard techniques, so they are not a fundamental limitation. Our stance is therefore that deep GNNs should be explored and studied on top of these standard techniques. This was argued in the former Section 5.2, which is now the last paragraph of the new Section 5, and it is additionally supported by the experiments in the new Section 4.

Dear Reviewer,

Thank you again for your constructive feedback, which helped us significantly strengthen our manuscript. As the discussion period closes in about two days, we would greatly appreciate it if you could review our response and let us know whether it addresses your concerns. We remain happy to provide further clarification if needed.

Reviewer_63G8OpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper argues that aggregation does not limit depth in GNNs. They compare using three datasets - Cora, CiteSeer, and PubMed (and have more datasets in the appendix) and show that Activation + Transformation operators are the ones that limit the utility of deep GNNs using ablation studies. I believe this is an interesting addition to our understanding of deep GNNs.

Critical strengths

The insight that aggregation is not the issue wrt deep GNNs is very useful knowledge. Deep GNNs are very useful for large graphs as shown by their protein application example. Identifying what is causing issues and taking the blame off aggregation is useful. The paper improves our understanding of GNNs and thus I believe it make a novel contribution worthy of publishing. They validate this with ablation studies over several datasets.

Critical weaknesses
  1. In the paper, there are only three datasets - all from the digital library domains. I think there are other domains they have experimented in the appendix. These should be brought into the main paper. A lot of the paper is motivating or selling the worth of the results, etc. The protein example of deep GNNs etc. is repeated. This information can be summarized and consolidated and some of the actual results from other datasets and the initializations should be brought into the main paper. That said, I think this can be done and so I am not arguing the paper not be published because of this.
Questions for authors

Not much except if they can address the issue mentioned in the weakness.

4 comments

We thank the reviewer for the positive assessment and the constructive suggestions on the presentation. We have revised the paper accordingly and uploaded the revised version, in which the changes described below can be checked.

1. Results on other datasets are now in the main paper. Figure 1 and Section 3.1 now cover nine datasets instead of three: five homophilic datasets (Cora, CiteSeer, PubMed, Photo, Computers) and four heterophilic datasets (Squirrel, Chameleon, Amazon-Ratings, Roman-Empire). The results that were previously in the appendix are discussed in the main text, including the exception of Roman-Empire.

2. Results on initializations are now in the main paper. The node similarity measures under He initialization, previously in the appendix, are now reported as Figure 3 in Section 3.3, next to the results under Glorot initialization in Figure 2.

3. The motivating content has been consolidated. We merged the former Sections 4 and 5 into a single Discussion section (new Section 5) of about one page. The repeated description of successful deep GNNs, including the example on ogbn-proteins, has been removed from this part.

4. Additional results. The space gained was used for a new section with experiments (new Section 4). There, we repeat the depth experiments after applying standard techniques (skip connections, batch normalization, dropout, and weight decay) to all models, with depths of up to 64 layers. The decrease in training accuracy observed in vanilla GNNs is resolved, and depth brings a clear benefit on Roman-Empire and Minesweeper. We also added depth experiments in the semi-supervised setting (Appendix E), which show that our observations on the robustness of deep GNNs relative to MLPs also hold when labels are scarce.

We hope that these revisions address the concern on presentation.

Dear Reviewer,

Thank you again for your positive assessment and constructive feedback, which helped us significantly strengthen our manuscript. As the discussion period closes in about two days, we would greatly appreciate it if you could review our response and let us know whether it addresses your concerns. We remain happy to provide further clarification if needed.

I have read the comments by the authors. Thank you. I already gave this paper the highest in my lot. I am happy it is better but I think my rating is still the same.

Thank you very much for taking the time to read our response. We are glad to hear that the revisions improved the paper and that our rebuttal addressed your concerns. We sincerely appreciate your strong support of our work.

Comments on the paper

1 comment

We sincerely thank all reviewers for their careful and constructive reviews. The comments helped us identify where the empirical support and the presentation of the paper needed to be strengthened, and we have revised the paper accordingly. We have uploaded the revised version, and we summarize the main changes below for the reviewers' convenience.

Summary of changes in the revised version

  • Section 3.1 and Figure 1: extended from three to nine datasets (the results previously in the appendix are merged into Figure 1).
  • Section 3.3 and Figure 3: the results under He initialization are moved from the appendix to the main text.
  • Section 4 (new): depth experiments with standard techniques, up to 64 layers (Figures 4 and 5).
  • Section 5 (new): the former Sections 4 and 5 are condensed into a single Discussion section.
  • Abstract, Introduction, and Conclusion: updated to reflect the changes above.
  • Appendix E (new): depth experiments in the semi-supervised setting.
  • Appendix G.3 (new): experimental details for Section 4.
  • Appendix I (new): training dynamics of vanilla models on Roman-Empire.
  • Appendix J (new): figures of the initial submission.

Since several sections and figures have been renumbered, we refer to the numbering of the revised version in our responses, and we explicitly mark the sections of the initial submission as "former".

We hope that the revision addresses the reviewers' concerns, and we would be glad to answer any further questions during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.