acceptodds
Withdrawn from ICLR 2027

The Impact of Ego-Neighbor Separation and Graph Topology on Graph Neural Network Memorization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent studies on memorization in Graph Neural Networks (GNNs) indicate that nodes with high label inconsistency within their neighborhoods are more susceptible to being memorized. However, this perspective overlooks the interplay between diverse topological factors and the structural inductive biases that dictate how ego-node and neighborhood representations are integrated during the aggregation phase. In this work, we systematically dissect GNN memorization by categorizing GNNs into two paradigms: Ego-Neighbor Integration GNNs, which mix ego-node and neighborhood representations during aggregation (e.g., GCN), and Ego-Neighbor Separation GNNs, which utilize independent ego-weight layers to process ego-node representations without mixing them during aggregation (e.g., GraphSAGE). Our findings reveal that topological factors like node degree can suppress memorization in both paradigms; however, Ego-Neighbor Separation GNNs still retain their memorization ability more effectively than Integration GNNs. Finally, bridging these architectural insights with privacy concerns, we show that GNNs equipped with ego-weight layers exhibit substantially increased susceptibility to Membership Inference Attacks. Ultimately, our findings demonstrate that GNN memorization is not merely a topology-dependent phenomenon, but rather the result of a complex interplay between the topological environment and how ego-node and neighborhood representations are integrated.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Memorization, Graph Neural Network, Homophily

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2ya2OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates memorization in graph neural networks (GNNs) and its relationship with graph topology and model architecture. Specifically, it seeks to answer three questions: How do topological properties, such as node degree and heterophily, affect memorization in GNNs? Does the way in which ego-node and neighborhood representations are integrated alter a GNN's ability to memorize noisy labels? Do architectural differences further affect the model's vulnerability to membership inference attacks (MIAs)?

The main contributions are as follows: the interplay between node degree, label homophily, and feature homophily affects memorization rates; separation-based architectures are generally more prone to memorizing noisy labels; and separation-based architectures exhibit higher membership inference attack success rates across multiple datasets. These findings are supported by experiments on both synthetic and real-world graph datasets, with the aim of explaining the observed differences through the lens of architectural inductive biases.

Critical strengths
  1. Rather than limiting the analysis to data characteristics that make certain nodes more susceptible to memorization, the paper further investigates how the architectural inductive biases of GNNs influence node memorization. It connects two questions: how GNNs leverage neighborhood information to make predictions, and how they fit anomalous labels in the training data.

  2. The experiments on synthetic datasets are well designed, enabling effective control of variables and analysis of nonlinear relationships. The paper also employs layer rewinding and layer retraining experiments to investigate the respective contributions of the ego-weight layer and the neighborhood convolution layer to the memorization of noisy labels.

  3. The paper further connects memorization with privacy risks by evaluating membership inference attacks using online LiRA (Likelihood Ratio Attack). This analysis examines whether the observed memorization behavior is associated with measurable privacy risks, thereby adding practical relevance to the study.

Critical weaknesses

1. Are the GNNs sufficiently trained?

2. Do these phenomena persist under commonly used GNN training techniques and tricks? This question is important for establishing the scope and applicability of the paper's conclusions.

Questions for authors
  1. The authors broadly categorize GNN architectures into two types: integration-based and separation-based architectures. This distinction appears to depend on whether the target node and its neighborhood share the same set of learnable parameters. However, this taxonomy seems overly simplistic. The paper does not appear to discuss whether mechanisms such as residual connections (which introduce additional learnable parameters) and gating are used, although these mechanisms may also affect how ego-node information is preserved. Therefore, the authors should further analyze the relationship between independent ego-node weight layers and mechanisms such as residual connections and gating, and validate their conclusions on a broader range of GNN architectures with different information integration mechanisms.

  2. The authors match the parameter size by increasing the hidden dimensionality of the neighbor-processing layer in the integration-based models. However, two models with the same number of parameters but different architectures may still differ in their effective expressive capacity, gradient pathways, feature interaction patterns, and optimization difficulty. Models with unshared parameters may theoretically have greater capacity. Could the stronger memorization of noisy labels be attributed to the additional capacity introduced by the independent ego-node weight layer, rather than to the proposed architectural distinction itself?

  3. In Table 3, the separation-based models outperform their integration-based counterparts on PubMed, Computers, Chameleon, and Squirrel, which appears to contradict the evaluation results reported in [1]. Could the authors analyze this discrepancy? When training these GNNs, did the authors consider mechanisms such as residual connections (which introduce additional learnable parameters), dropout, and normalization, and appropriately tune their configurations? Appendix D.2 appears to indicate that a fixed configuration was used. More thorough tuning to better realize the potential of GNNs would make the conclusions more convincing.

  4. Table 4 provides statistics for the ogbn-arXiv dataset, but I could not find the corresponding experimental results. Did I miss them?

[1] Classic GNNs are Strong Baselines: Reassessing GNNs for Node Classification, NeurIPS 2024.

Reviewer_VcjjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how node degree, label homophily, and feature homophily affect GNN memorization.

It measures memorization through fitting deliberately corrupted training labels and compares architectures that jointly transform ego and neighbor features against those with separate transformations.

Synthetic experiments suggest that ego-neighbor separation facilitates noisy-label fitting, particularly where neighborhood mixing suppresses it.

Critical strengths
  • S1. The studied problem is practical and important in use cases of GNNs.

  • S2. Parameter-count-control analysis is reasonable and intuitive.

  • S3. Empirical evidence is suggested from diverse aspects, such as learning trajectories and membership inferences.

Critical weaknesses
  • W1. Unclear architecture-specific memorization–privacy trade-off. Differences in membership leakage coincide with substantial differences in training accuracy, making it difficult to attribute the privacy risks specifically to ego-neighbor separation rather than different degrees of model fitting. Could the authors have more controlled analysis for this?

  • W2. Limited real-world memorization analysis. Real-world experiments primarily evaluate membership inference, while the central findings on memorization and its interaction with graph topology rely on synthetic graphs. Their generality therefore remains insufficiently established. Could the authors further analyze the memorization in the real-world datasets as well?

  • W3. Gap between analysis and architectural implications. The analysis does not clearly establish whether independent ego weights, ego-feature scaling, or changes in aggregation normalization drive the observed effects. Consequently, the architectural guidance supported by the findings remains unclear. Could the authors better clarify the (potential) underlying factor?

Questions for authors

See Weakness.

Reviewer_sHSVOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates GNN memorization by categorizing GNNs into two paradigms: Ego-Neighbor Integration GNNs, and Ego-Neighbor Separation GNNs. It finds that in addition to label homophily, other topological factors such as feature homophily and node degree, also influence memorization in GNNs. And Ego-Neighbor Separation GNNs exhibit higher GNN memorization.

Critical strengths

A controlled experiment design and interesting finding.

The paper uses CSBM-3H to vary node degree, label homophily and feature homophily independently, and analyzes how they influence GNN memorization. The stated finding, that Ego-Neighbor Separation GNNs memorize noisy labels more than their Integration counterparts, is interesting. Ego/neighbour separation is built into widely used architectures such as GraphSAGE. A hidden memorization cost of this design would matter for practitioners working with sensitive, heterophilous graphs.

Critical weaknesses

1. The memorization measure departs from the work the paper builds on, and the main comparisons are confused by under-training and fitting ability.

The paper positions itself as extending Jamadandi et al. (2025), yet measures memorization differently: it uses the fraction of fitted label-corrupted nodes (MR) instead of the counterfactual, node-level memorization score (NCMemo) used there. NCMemo is designed specifically for graph data. The switch is not justified, and it has consequences.

  • The topology findings cannot be compared with the result the paper claims to broaden.

  • MR captures fitting of random noise, not memorization of atypical but correctly labelled nodes.

  • A low MR cannot distinguish "the model does not memorize this node" from "the model cannot fit the corrupted label". This is a serious ambiguity here, because several models have not converged, e.g., in figure 1, GCN and GAT are still learning at epoch 100 with Deg 26-30 | h_F=0.0 | h_L=0.0, and in figure 2, GraphSAGE is still learning at epoch 100 with Deg 26-30 | h_F=0.9 | h_L=1.0.

  • Corrupting labels itself changes the effective label homophily under study.

2. The contributions of this work are limited, and the attribution of memorization to the ego-weight, and to "Separation GNNs" as a class, is not established.

Limited contribution. Most findings follow from known results or from the model equations:

  • That a dedicated ego pathway preserves node-specific signal against a contradictory neighbourhood is the established motivation for GraphSAGE and H2GCN.

  • That high-degree nodes in label-homophilous graphs are classified more robustly, and that feature and label homophily interact, are reported by Subramonian et al. (2024) and Zheng et al. (2024).

  • More capacity to fit node-specific signal leading to more fitting of noisy labels, and in turn to higher membership leakage, is the expected consequence of these, not a separate discovery.

  • The claimed "interplay" between degree and architecture follows largely from the update rule. In GCN with a self-loop, the ego contribution is scaled by 1/(dv+1)1/(d_v+1), about 3% at degree 28, whereas Eq. 11 gives the ego term an undiluted path. "Degree suppresses memorization in Integration but not Separation models" is therefore largely one mechanism, not an empirical interaction of two factors.

  • "Node Support", presented as a broader lens, is never operationalized as a node-level quantity or shown to predict which nodes are memorized.

  • Moreover, the paper does not translate its observations into practical guidance: no privacy–utility comparison or mitigation is provided.

The contribution therefore lies mainly on the empirical isolation of the ego-weight effect, which is not established:

  • Section 4 rests on one Separation model. GraphSAGE is the only Separation model there, and it also differs from GCN and GAT in aggregation (mean vs normalized sum vs attention), normalization and self-loop handling.

  • Section 5 does not resolve this. Its variants remain limited to three message-passing backbones, and each Sep variant changes two components at once (it removes the self-loop and adds WegoW_{ego}). The Integration variants are widened to match parameter count, which changes capacity in its own right.

  • The paper's own tables contradict the central claim. At the same high-degree setting (d∈[26,30]d \in [26,30], hF=0.9h_F = 0.9), last-epoch MR for the three Separation variants differs by up to about 40× (Table 2, 5 and 6):

hLh_LSep-GCNSep-GATSep-GraphSAGE
0.54.8622.0069.43
1.01.7117.7168.00

All three have an ego-weight, so the ego-weight alone cannot explain GraphSAGE's behaviour. The aggregator or normalization evidently plays a major role.

Questions for authors
  1. Why was the counterfactual memorization score of Jamadandi et al. (2025), on which this work builds, not used, or at least reported alongside MR? Do the main findings (degree and feature-homophily effects, Sep > Int) hold under that measure?

  2. Do the conclusions hold when all models are trained to convergence?

  3. Are the observations and findings also applicable to other graph tasks, e.g., graph classification tasks, link prediction tasks?

  4. What are the concrete practical implications of these findings?

  5. The MIA setting appears to reuse the label-corrupted, last-epoch models of Sec. 5. The Train Clean/Test accuracies in Table 2 are identical to the last columns of Table 1, contradicting the best-validation checkpointing stated in Appendix E. If so, the higher TPR@1%FPR of Separation models may be driven largely by the 10% corrupted members, which are extreme outliers and are fully memorized by Separation models. How is the final model selected exactly for LiRA?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.