- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates memorization in graph neural networks (GNNs) and its relationship with graph topology and model architecture. Specifically, it seeks to answer three questions: How do topological properties, such as node degree and heterophily, affect memorization in GNNs? Does the way in which ego-node and neighborhood representations are integrated alter a GNN's ability to memorize noisy labels? Do architectural differences further affect the model's vulnerability to membership inference attacks (MIAs)?
The main contributions are as follows: the interplay between node degree, label homophily, and feature homophily affects memorization rates; separation-based architectures are generally more prone to memorizing noisy labels; and separation-based architectures exhibit higher membership inference attack success rates across multiple datasets. These findings are supported by experiments on both synthetic and real-world graph datasets, with the aim of explaining the observed differences through the lens of architectural inductive biases.
Critical strengths
-
Rather than limiting the analysis to data characteristics that make certain nodes more susceptible to memorization, the paper further investigates how the architectural inductive biases of GNNs influence node memorization. It connects two questions: how GNNs leverage neighborhood information to make predictions, and how they fit anomalous labels in the training data.
-
The experiments on synthetic datasets are well designed, enabling effective control of variables and analysis of nonlinear relationships. The paper also employs layer rewinding and layer retraining experiments to investigate the respective contributions of the ego-weight layer and the neighborhood convolution layer to the memorization of noisy labels.
-
The paper further connects memorization with privacy risks by evaluating membership inference attacks using online LiRA (Likelihood Ratio Attack). This analysis examines whether the observed memorization behavior is associated with measurable privacy risks, thereby adding practical relevance to the study.
Critical weaknesses
1. Are the GNNs sufficiently trained?
2. Do these phenomena persist under commonly used GNN training techniques and tricks? This question is important for establishing the scope and applicability of the paper's conclusions.
Questions for authors
-
The authors broadly categorize GNN architectures into two types: integration-based and separation-based architectures. This distinction appears to depend on whether the target node and its neighborhood share the same set of learnable parameters. However, this taxonomy seems overly simplistic. The paper does not appear to discuss whether mechanisms such as residual connections (which introduce additional learnable parameters) and gating are used, although these mechanisms may also affect how ego-node information is preserved. Therefore, the authors should further analyze the relationship between independent ego-node weight layers and mechanisms such as residual connections and gating, and validate their conclusions on a broader range of GNN architectures with different information integration mechanisms.
-
The authors match the parameter size by increasing the hidden dimensionality of the neighbor-processing layer in the integration-based models. However, two models with the same number of parameters but different architectures may still differ in their effective expressive capacity, gradient pathways, feature interaction patterns, and optimization difficulty. Models with unshared parameters may theoretically have greater capacity. Could the stronger memorization of noisy labels be attributed to the additional capacity introduced by the independent ego-node weight layer, rather than to the proposed architectural distinction itself?
-
In Table 3, the separation-based models outperform their integration-based counterparts on PubMed, Computers, Chameleon, and Squirrel, which appears to contradict the evaluation results reported in [1]. Could the authors analyze this discrepancy? When training these GNNs, did the authors consider mechanisms such as residual connections (which introduce additional learnable parameters), dropout, and normalization, and appropriately tune their configurations? Appendix D.2 appears to indicate that a fixed configuration was used. More thorough tuning to better realize the potential of GNNs would make the conclusions more convincing.
-
Table 4 provides statistics for the ogbn-arXiv dataset, but I could not find the corresponding experimental results. Did I miss them?
[1] Classic GNNs are Strong Baselines: Reassessing GNNs for Node Classification, NeurIPS 2024.