- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether the quality of graph neural network (GNN) representations can be assessed using a single evaluation metric, a question of theoretical significance for graph representation learning. Focusing primarily on GIN and GraphSAGE on Cora, the paper considers three readout mechanisms for evaluating representations: unsupervised prototype recovery and supervised centroid-based and linear-probe evaluation. It introduces the perspective of readout-dependent representation accessibility and further attempts to explain the performance discrepancies across readout mechanisms through subspace analysis, multi-prototype experiments, analyses of degree-related patterns, and training-time interventions that scale the target nodes' degrees in the message-passing operators. Based on its empirical findings, the paper draws some constructive conclusions:
- The prototype recovery deficiency of GIN on Cora is associated with the capacity limitations of a single prototype.
- Scaling target-node degrees during training can causally alter representation accessibility.
Overall, the paper is primarily an empirical study rather than a methodological contribution.
Critical strengths
-
The paper addresses a clearly defined research question, follows a coherent experimental design, and presents a complete chain of reasoning.
-
The experiments report results across multiple runs and use stratified Bootstrap to estimate confidence intervals. These practices enhance the credibility of the experimental findings and make the statistical comparisons more convincing than those based on single-run results.
-
The paper acknowledges several important limitations and carefully delineates the scope of its conclusions, avoiding the overinterpretation of metric changes as a comprehensive causal explanation of the underlying mechanisms.
-
The findings offer useful insights into the diagnosis and evaluation of GNN representation learning.
Critical weaknesses
-
Would the gap in prototype recovery between GIN and GraphSAGE persist if K-Means were replaced with alternative clustering methods, if different distance metrics were used, or if graphs with different characteristics and scales were considered?
-
The current intervention demonstrates that modifying degree scaling can alter representation accessibility. However, is node degree merely a correlated factor, or does it help explain the underlying failure mechanism?
Questions for authors
-
The paper's primary empirical evidence comes from two homophilous graphs, Cora and Amazon-Computers. It remains unclear whether the same phenomena persist in heterophilous graphs, heterogeneous graphs, or large-scale graphs with millions of nodes. Moreover, the current study focuses exclusively on node classification. It is unclear whether similar findings would hold for node regression, edge-level tasks, or graph-level tasks. If these settings are beyond the scope of the paper, the authors should frame their work as identifying and diagnosing a phenomenon in specific graph-based node classification settings, rather than implying broader applicability to GNN representation learning in general.
-
I am curious why the authors chose K-Means as the sole clustering algorithm, despite its popularity. Given that increasing the number of prototypes improves performance, the authors should consider whether non-convex clustering algorithms might be more appropriate. Comparisons with alternative clustering algorithms, different distance metrics, and nonlinear unsupervised readout methods would strengthen the evaluation. If the observed discrepancies persist under these settings, the paper's claims regarding readout dependence would be considerably more convincing.
-
The paper demonstrates how to identify and localize problems in representations, but its contributions remain primarily at the level of diagnosis and evaluation. These findings have not yet been sufficiently translated into practical algorithmic improvements, which may limit the paper's immediate practical value.
-
In homophilous graphs, nodes with higher degrees are expected to aggregate information from more same-label neighbors, which should, in principle, benefit node classification. Therefore, it is unsurprising that modifying degree scaling may affect predictive performance. The paper lacks intervention experiments covering a broader range of intermediate intervention strengths and a more diverse set of graph types, which limits the strength of the causal interpretation of the degree-scaling intervention.