acceptodds
Under review as a conference paper at ICLR 2027

Localizing Memorization in Graph Neural Networks

Abstract

Graph Neural Networks (GNNs) have been shown to memorize individual training examples, with recent work quantifying their memorization. However, where memorization occurs within the network, i.e., which layers and weights are responsible, remains an open question. While studies in Deep Neural Networks (DNNs) have established that memorization often concentrates on a small number of model weights across all model layers, no such analysis has been explored for GNNs. This paper presents the first systematic study of localizing memorization in GNNs at both the layer and weight levels. At the layer level, we employ gradient accounting to measure per-layer gradient norm contributions from memorized and non-memorized nodes. At the weight level, we analyze memorization of a training node by measuring the minimum number of important weights whose ablation leads to the node's prediction flip. By localizing memorization in multiple GNN architectures trained on diverse datasets of varying homophily levels, we find that (1) memorized nodes dominate the gradient contributions over non-memorized nodes at every layer, with the gap generally increasing with layer depth, (2) the graph homophily determines whether memorized nodes produce gradients aligned with or opposed to those of non-memorized nodes, revealing a GNN-specific memorization mechanism, (3) the memorization of individual nodes in GNNs is concentrated on a few model weights, and (4) finally, the memorization localization analysis in GNNs enables targeted mitigation of privacy leakage. Overall, our work contributes to the fundamental understanding of memorization in GNNs.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
graph neural networks, memorization, membership inference attack

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DTzdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates the memorisation problem in GNNs via gradient behaviour, extending the existing understanding in the memorisation problem beyond NCMemo, and investigates the implications on privacy. The investigation includes experiments on common architectures on both synthetic and real graph datasets.

Critical strengths

The experimentations cover common GNN architectures with extensive ablation studies in the appendices

Critical weaknesses
  1. The authors "defines" and "introduces" LMS and AWS metrics without properly crediting their similarities to the prior works [1]. LMS is a rewritten from Section 4.1 p.4 in [1], and AWS follows the procedure from Section 6.1, p. 7 in [1]

This limits the novelty of the general work as the paper mostly uses or adapts the existing measures and experimentation procedures.

  1. The theoretical front is weak, where the propositions are just restatements of the empirical observations.

[1] Maini, Pratyush, et al. "Can neural network memorization be localized?." arXiv:2307.09542 (2023).

Questions for authors

Can the authors please clarify what the "GNN-specific mechanism" is?

the gap between them increases with layer depth, indicating that in GNNs, memorization is distributed across all layers but becomes increasingly dominant in deeper layers.

Can the authors also clarify the causal relation between the widening LMS node gap and the memorisation. Could that be e.g. empirically observed through the retrievability of the node embedding at shallow vs deep layers?

3 comments
AuthorsOpenReview ↗

Response to Reviewer DTzd (1/2)

We thank the Reviewer for acknowledging that our experimentations cover common GNN architectures with extensive ablation studies. We also appreciate the constructive comments, which have helped us better sharpen the GNN-specific contributions of our work.

W1: Novelty of the work.

To the best of our knowledge, our work presents the first systematic study of memorization localization in GNNs. Specifically, we adapted gradient accounting and greedy neuron removal in [1] to the graph setting. More importantly, this study reveals several previously unexplored, GNN-specific properties of memorization in GNNs:

  1. Layer-wise gradient contributions: We find that, in GNNs, memorized nodes exhibit larger gradient contributions than non-memorized nodes across all layers, with the gap increasing with layer depth. This is different from the findings of Maini et al., (2023) [1], who show that in DNNs "no layer receives significantly more contribution from memorized examples than from clean examples" (Figure 1 in [1]). We have conducted additional experiments using an SGC-style graph neural network and a standard MLP to demonstrate that this observation is specific to GNNs with learnable message-passing layers (see Section 5.2 of the revised paper). Specifically, in these GNNs, graph propagation is integrated into the learnable layers, resulting in graph-coupled computations which influences how memorized information is distributed across layers.

  2. Homophily-dependent gradient alignment: We find that the gradients of memorized and non-memorized nodes are more positively aligned in heterophilic graphs, but become increasingly opposed as graph homophily increases. This reveals a relationship between a fundamental graph property, i.e., homophily, and the gradient alignment of memorized and non-memorized nodes. We have further validated this finding across a range of settings: (i) general GNN architectures, such as GCN and GAT; (ii) a heterophilic-aware GNN architecture, H2GCN (see Appendix J); (iii) large-scale graph datasets, including ogbn-arxiv, Coauthor-CS, and Coauthor-Physics (see Appendix N); and (iv) a more advanced GNN architecture, Graph Transformer (see Appendix N).

  3. Concentration of memorization: We observe that memorized nodes exhibit concentrated memorization in GNNs, with memorization-critical weights located primarily in the final layer. We also provide a theoretical analysis to explain this observation (see Appendix G). Furthermore, we have conducted additional experiments on layer-wise ablation sensitivity and layer-wise collateral prediction changes (see Appendix H) to verify that the final-layer memorization concentration is GNN-specific due to the different graph-coupled collateral effects across GNN layers.

Finally, building on these insights, we develop a targeted defense against privacy leakage. Our experiments demonstrate that this localization-guided defense achieves a stronger privacy-utility trade-off than existing baseline defenses.

Overall, our contributions go beyond adapting existing techniques to GNNs. We provide a systematic study of memorization localization in GNNs, reveal graph-specific memorization phenomena that distinguish their behavior from that of conventional DNNs, and translate these findings into a practical defense. We have added these clarifications in Section 5.2, Appendix N, Appendix J, Appendix G, Appendix H of the revised paper.

W2: The theoretical front is weak, where the propositions are just restatements of the empirical observations.

To provide theoretical support for our empirical findings, we present formal proofs of Propositions 1 and 2 in Appendices F and G, respectively. Following the Reviewer's suggestion, we have revised the Proposition 1 and 2 in the main paper to distinguish them from the empirical observations:

Proposition 1 (Gradient alignment decreases with homophily). Consider a non-memorized node and a memorized node sharing the same true class but assigned different training labels. Under the assumptions specified in Appendix F, the expected inner product between their gradients decreases as graph homophily increases, becoming negative in the high-homophily regime. See Appendix F for the formal statement, definitions, assumptions, and proof.

Proposition 2 (Final-layer weights provide a low-cost pathway for memorization in GNNs). Consider an ll-layer GNN and a target node uu. A perturbation δWk\delta\mathbf{W}_k to the weights of layer kk changes the logit of node uu through the layer-(kk-1) embeddings of uu's (ll-kk)-hop neighbourhood. In contrast, a perturbation δWl\delta\mathbf{W}_l to the final layer changes the logit of node uu only through uu's own embedding. Consequently, a low-cost way to memorize a node is to adjust the final layer's weights, leading to concentrated memorization-critical weights in the final layer.

AuthorsOpenReview ↗

Response to Reviewer DTzd (2/2)

Q1: What the "GNN-specific mechanism" is.

By the "GNN-specific mechanism", we refer to the role of message passing and graph-coupled computation in shaping how memorization occurs and is distributed within GNNs. Specifically, our findings reveal three phenomena of this mechanism, including (i) Layer-wise gradient contributions, (ii) Homophily-dependent gradient alignment, and (iii) Concentration of memorization. The detailed clarification of these three phenomena caused by the "GNN-specific mechanism" is presented in the response to the W1.

Q2: Causal relation between the LMS gap and memorization.

Following the Reviewer's suggestion, we additionally conducted a retrievability experiment to empirically investigate how the memorization relates to the layer-wise LMS gap. The experiment follows these steps: (1) We use the same GCN as in the paper to train and freeze the model (syn-cora, memorization rate ρ=0.01\rho=0.01); (2) We run a forward pass on the model and extract the representations of all nodes at each layer: input features - XX, the output of Layer 0 - H(0)H^{(0)}, the output of Layer 1 - H(1)H^{(1)}, and the output of Layer 2 - ZZ (the logits); (3) For each layer, we train a linear probe using each layer's representations, only on clean training nodes, thereby preventing the probe from directly memorizing the noisy labels; (4) For all mislabeled nodes, we calculate the fraction of fitted memorized nodes which the corresponding probe predicts the assigned (noisy) label.

Homophily levelInput XOutput of Layer 0Output of Layer 1Output of Layer 2
h=0.000.0140.0350.0720.923
h=0.500.0090.0130.1840.914
h=1.000.0070.0210.3040.966

The results show that the memorized label becomes increasingly retrievable across the layer depths, e.g., in the h=0.00h=0.00 setting, the fraction increases from 1.4% at the input features to 92.3% at the final layer. A similar increasing trend is observed across all three homophily levels. These findings indicate that memorized information becomes more accessible in deeper GNN layers. From Figure 1 in the paper, we can observe that the layer-wise LMS gap also increases with layer depth. Together, the two results support the connection between the memorization and the LMS gap. We have added these results and analysis in Appendix E of the revised paper.

If these clarifications address the reviewer's comments, we would be grateful if they could update their score accordingly.

AuthorsOpenReview ↗

Request to respond to the rebuttal

As the rebuttal period is coming to an end, we would like to thank the Reviewer again for their constructive feedback. We would like to highlight the additional experiments, theoretical clarifications, and analyses we have provided in response to address their concerns:

  1. Novelty and GNN-specific contributions (W1, Q1): We have further clarified three GNN-specific memorization phenomena: (i) layer-wise gradient contributions, which distinguish GNNs from conventional DNNs (Maini et al., (2023) [1]); (ii) homophily-dependent gradient alignment; and (iii) concentration of memorization-critical weights in the final layer.

We have also conducted additional experiments to validate these findings across different architectures and settings (see Section 5.2, Appendix N, Appendix J, Appendix G, Appendix H of the revised paper).

  1. Theoretical contributions (W2): We have provided formal proofs of Propositions 1 and 2 in Appendices F and G, respectively, and revised their statements in the main paper to distinguish them from the empirical observations clearer. In particular, these propositions characterize how graph homophily affects gradient alignment and explain why the final layer provides a low-cost pathway for memorization in GNNs.

  2. Causal relationship between the LMS gap and memorization (Q2): We have conducted an additional retrievability experiment using linear probes trained on clean training nodes. The results show that the memorized labels become increasingly retrievable across GNN layers, supporting the connection between memorization and the layer-wise LMS gap. We have included the experimental results and analysis in Appendix E.

We believe these additional experiments, analyses, and theoretical clarifications have further demonstrated the novelty and significance of our contributions. We would be grateful if the Reviewer could review our responses and consider updating their rating accordingly. We would also be happy to address any remaining questions.

Reviewer_hMJdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper conducts a study of where memorization happens in graph neural networks, analogus to [1] for non-graph DNNs and extending the analysis that measured the amount but not location of memorization in GNNs [2]. Empirical findings are flagged when they resemble what has been discovered for non-graph data, and when they differ. For example, the paper presents some empirical and theoretical analysis to argue that graph homophily may affect how much memorization occurs and how the gradients of memorized nodes (nodes for which the GNN learns to fit an arbitrary label) compare in sign to those of non-memorized nodes. As a practical application, the authors consider a graph rewiring strategy to reduce memorization and preserve privacy of memorized nodes.

[1] Maini et al. Can Neural Network Memorization Be Localized? ICML 2023. [2] Jamadandi et al. Memorization In Graph Neural Networks. NeurIPS 2025.

Critical strengths

S1. The paper both connects its findings to works outside the graph domain and develops graph-specific intuition, theory, and empirical results. S2. The contributions include a mix of theory, empirical findings, and new methodology. Theory is connected to empirical findings.

Critical weaknesses

W1. The claims seem to have some errors / unjustified asssertions. The intuition of Proposition 1 is that "in a heterophilic graph, neighbouring aggregation yields a class-uninformative signal". Perhaps this is true of vanilla GNNs like GCN or GAT, but it does not seem to be generally true for all GNN models; many methods have been developed to perform class-informative aggregation in heterophilic graphs, starting with Zhu et al. (2020) whose analysis the authors build on. This intuition is repeated in Appendix D to explain the claim that "the error alignment term is weakly positive" at low homophily levels, which does not seem to be always true. Furthermore, in Appendix E.2, the authors try to localize the effects of final-layer perturbations to a single node. But the weight matrix WlW_l transforms the feature dimensions of all nodes, so a perturbation of WlW_l does change the embeddings of all nodes. Remark 2 claims that this is specific to GNNs because MLPs process each input differently, but note that graph convolutions have been approximated by logistic regression after transforming the features with the graph structure [3].

W2. Most of the analysis of gradient signs is done on vanilla architectures and tiny synthetic or real benchmarks. Analysis of larger graphs / more advanced architectures is not presented till Appendix K, and the gradient sign analysis is not conducted. Given that as stated above, the gradient sign analysis seem to an empirical finding of specific models and datasets rather than theoretically proven to hold universally, it is necessary to conduct a more thorough empirical study to support this claim for more state of the art graph tasks.

[3] Wu et al. Simplifying Graph Convolutional Networks. ICML 2019.

Questions for authors

Let's say that your "graph neural network" was just an MLP applied to ALXA^{L} X, similar to Simple Graph Convolutional Networks [3]. Would you expect the behavior to be explained by the analysis for DNNs in [1]? (e.g. memorization not necessarily being mainly located primarily at the final layer).

6 comments
AuthorsOpenReview ↗

Response to Reviewer hMJd (1/2)

We thank the Reviewer for recognizing the significance of this work, which both connects its findings to works outside the graph domain and develops graph-specific intuition, theory, and empirical results. We appreciate the acknowledgment of our work's contribution as a mix of theory, empirical findings, and new methodology.

W1-a: The intuition of Proposition 1 does not seem to be generally true for all GNN models.

To investigate whether this intuition extends to architectures specifically designed for heterophilic graphs, we have additionally conducted an experiment for H2GCN (Zhu et al., 2020), the heterophilly-aware GNN the Reviewer refers to. We measure the cosine similarity between the average gradients of memorized and non-memorized nodes across training epochs.

H2GCN (syn-cora, $\rho$=0.01):

HomophilyLayer 0Layer 1Layer 2
0.000.25380.26240.2748
0.300.20850.21840.2295
0.500.00230.07320.1012
0.70-0.1942-0.1035-0.0732
1.00-0.7093-0.5631-0.5032

H2GCN exhibits lower gradient cosine similarity for heterophilic graphs (h=0.00h=0.00) than GCN, suggesting that H2GCN performs a more class-informative aggregation in heterophilic graphs than vanilla GNNs like GCN. Nevertheless, we can still observe an inverse relationship between the gradient cosine similarity and homophily. These results indicate that the observed homophily-dependent gradient-alignment phenomenon is not limited to GCN and GAT, but also extends to H2GCN, a representative heterophily-aware GNN. See Appendix J for these resutls.

W1-b: Final-layer perturbations and Remark 2.

We would like to clarify what Appendix E.2 (now Appendix G.2 in the revised paper) intends to show. We agree that WlW_l is shared across all nodes, so a perturbation to WlW_l changes the logits of all nodes. However, Equation 20 does not claim otherwise. Rather, its claim concerns the pathway through which a perturbation affects node uu's logit. Specifically, at the final layer, the change node uu's logit depends only on its own pre-computed embedding and on no other node's embedding. In contrast, at an earlier layer k<lk<l (Equation 18), the change node uu's logit also depends on the layer-(k-1) embeddings of its (l-k)-hop neighbourhood. Therefore, what Appendix E.2 emphasizes is not whether a perturbation to WW affects other nodes, but the graph-coupled collateral, which exists before the final layer.

To further verify the lower collateral cost at the final layer, we have conducted two additional experiments:

(1) Layer-wise ablation sensitivity. We restrict greedy ablation to one layer at a time and measure the fraction of that layer's weights needed to flip the node's prediction, defined by AWSkAWS_k (kk indicates the layer). We then compute RkR_k, the ratio between the AWS of memorized nodes and that of the non-memorized nodes at layer kk: Rk=AWSkmem/AWSknonR_k=AWS_k^{mem}/AWS_k^{non}. A lower RkR_k means a bigger memorization-specific gap at layer kk. The results for GCN, syn-cora, ρ=0.01\rho=0.01 are shown below.

Homophily levelR0R_0R1R_1R2R_2
0.000.820.860.38
0.300.870.880.37
0.500.900.890.45
0.700.910.870.65
1.000.950.910.61

We see a substantially larger memorization-specific gap, i.e., 1−Rk1-R_k, in the final layer than in earlier layers. For example, at h=0.00h=0.00, the gap is 0.62 in the final layer, compared with 0.18 and 0.14 in Layers 0 and 1, respectively. This indicates that the observed layer preference is specific to memorization, rather than simply reflecting general prediction sensitivity.

(2) Layer-wise collateral prediction changes. We also measure the collateral prediction flip rates per layer on other nodes rather than the target node (memorized nodes). These nodes are separated into graph neighbors and non-neighbors. The results on GCN, syn-cora, ρ=0.01\rho=0.01, h=0.00h=0.00, are shown below.

Collateral prediction flip rate (%):

Layer ablatedNeighbor flip rateNon-neighbor flip rate
Layer 015.425.22
Layer 114.234.57
Layer 23.454.03

The results show that the layers differ in graph-coupled collateral effects. Specifically, ablating earlier layers (Layers 0 or 1) changes the predictions of the target node’s neighbors approximately three times as often as those of non-neighbors (e.g., 15.42% vs. 5.22% for Layer 0). In contrast, ablating the final layer exhibits comparable flip rates for neighbors and non-neighbors (3.45% vs. 4.03%). We have revised the Appendix E.2 (now Appendix G.2 in the revised paper) accordingly and added these experiments in Appendix H.

Under this pathway-based definition, Remark 2 also holds. In an MLP, no layer is graph-coupled, so the argument gives no preference for the final layer. To empirically examine whether the claim in Remark 2 is specific to GNNs, we have conducted additional experiments on an MLP and an SGC-style graph neural network, as detailed in our response to the "Questions For Authors" below.

AuthorsOpenReview ↗

Response to Reviewer hMJd (2/2)

W2: Most of the analysis of gradient signs is done on vanilla architectures and tiny synthetic or real benchmarks.

We appreciate the Reviewer's concern regarding the scalability and generalisability of our gradient sign analysis. To address this, we have conducted additional experiments along two dimensions.

(1) Gradient sign analysis on larger graph datasets. We extended our gradient cosine similarity analysis to three large graph datasets, following the Reviewer’s suggestion: ogbn-arxiv, Coauthor-CS, and Coauthor-Physics, ranging from 18,000 to 169,000 nodes. The average cosine similarity between the memorized and non-memorized gradients across training epochs are reported below.

Average cosine similarity between memorized and non-memorized gradients across epochs (large graphs, $\rho=0.01$):

DatasetHomophily levelLayer 0Layer 1Layer 2
ogbn-arxiv0.58830.01530.07320.1043
Coauthor-CS0.7851-0.2034-0.1032-0.1065
Coauthor-Physics0.8722-0.4531-0.3125-0.3154

(2) Gradient sign analysis on a more advanced GNN architecture. We further extended our gradient sign analysis to Graph Transformer, a more advanced GNN backbone. The results are reported below.

Average cosine similarity between memorized and non-memorized gradients across epochs (Graph Transformer, $\rho=0.01$):

DatasetHomophily levelLayer 0Layer 1Layer 2
Actor0.00620.43520.40410.3841
Chameleon0.03390.40320.39810.4123
Cora0.7637-0.2043-0.0942-0.0713

Overall, these additional experiments extend our analysis beyond vanilla GNN architectures and small benchmarks. The results demonstrate that the gradient alignment patterns, i.e., inverse relationship between gradients cosine similarity and homophily, can also be observed on larger graphs and with a more advanced GNN architecture. We have incorporated these additional results into Appendix N of the revised paper.

Questions For Authors: For a graph neural network, which is an MLP applied to ALXA^L X, similar to SGC, would you expect the behavior to be explained by the analysis for DNNs in [1]?

Following the suggestion from the reviewer, we conducted additional experiments on (1) a 3-layer MLP applied on XX, and (2) a SGC-style graph neural network which applies a 3-layer MLP to A~LX\tilde{A}^L X. Both models are trained on syn-cora with memorization rate of 0.01. We report the LMS gap (LMS\texttt{LMS}-gap) between memorized and non-memorized nodes, defined as LMS\texttt{LMS}-gap = LMS_{mem} - LMS_{non} per layer. The results are shown below.

LMS-gap between memorized and non-memorized nodes (x10^-2)

ModelHomophily levelLayer 0Layer 1Layer 2
MLP0.00.2360.2370.232
MLP0.50.3690.3230.304
MLP1.00.3910.3390.360
SGC (MLP on A~LX\tilde{A}^L X)0.00.4860.5400.451
SGC (MLP on A~LX\tilde{A}^L X)0.50.5350.4920.461
SGC (MLP on A~LX\tilde{A}^L X)1.00.5680.5730.478
GCN0.00.4580.9903.529
GCN0.50.5581.5555.870
GCN1.00.7122.3619.708

The results show that, for both the standard MLP and the SGC-style graph neural network, memorized nodes exhibit higher LMS values than non-memorized nodes at every layer but the LMS gap does not increase with depth. This is consistent with the findings of Maini et al., (2023) [1], who show that in DNNs "no layer receives significantly more contribution from memorized examples than from clean examples" (Figure 1 in [1]). In contrast, for GCN, the LMS gap between the memorized and non-memorized nodes increases with depth. This distinction between the SGC-style model and GCN can be explained by their architectures. Specifically, in the SGC-style model, graph propagation (A~LX\tilde{A}^L X) is performed as fixed preprocessing, before the learning MLP layers. Consequently, no learnable layer is graph-coupled, and the model exhibits behaviour similar to a standard MLP. In GCN, however, graph propagation is integrated into the learnable layers, leading to graph-coupled computations which influences how memorized information is distributed across layers.

These results also verify that the observations we reported in the paper, i.e., memorization becomes increasingly dominant in deeper layers, is specific to GNNs with learnable message-passing layers. We have added these results and analysis into the revised paper (see Section 5.2).

If the additions address the reviewer's concerns, we would be happy if they could increase their score.

Reviewer_hMJdOpenReview ↗

Empirical scope improved; remaining questions about theory

Thank you for the detailed response. With more advanced architectures particularly for heterophily and larger datasets, the empirical contribution of the work is appreciably stronger. I also appreciate the clarification on the final-layer perturbations argument. The proof, however, relies on a linearized GNN where effectively all graph propagation can happen before the learned layers (hence my comment about SGC). This theory thus doesn't seem sufficient to support the claim that an early-layer change can propagate through subsequent graph operations and thus lead GNNs to prefer memorization in later layers.

I also still don't buy the claim in the low-homophily proof step that near-uniform prediction implies a positive error alignment term. Consider binary classification with a clean node and a mislabeled node both of true class 1, but with the latter’s training label flipped to class 2. Taking their predictions as �0�,�1��0�, �1�, and �2��2�, their error-vector inner product is �3��3�, which seems to be a counterexample.

Reviewer_YWueOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates which GNN layers and weights support predictions on memorized training nodes. It identifies nodes through controlled label corruption and NCMemo, uses normalized layerwise gradient norms during training, and measures the fraction of weights removed by a greedy procedure before a target prediction changes. The experiments report larger gradients and lower ablation fractions for memorized nodes, homophily-dependent gradient alignment, and concentration of ablated weights in the final layer. A proof-of-concept pruning defense uses localization to reduce membership-inference attack success.

Critical strengths
  1. The study addresses a worthwhile question with complementary measurements and node definitions. Label corruption supplies a controlled intervention, while NCMemo examines memorization without deliberately changing labels. Training-gradient accounting and post-training ablation probe different aspects of model behavior. Figure 2's homophily-dependent gradient-alignment pattern is an interesting empirical finding in the studied setting, independently of the privacy application.

  2. Appendix J strengthens the evidence beyond an unrestricted clean-node comparison. It selects clean controls by training loss and prediction margin. Table 6 reports lower ablation scores for fitted mislabeled nodes than for either control set across five real datasets. These results make the concentration finding more persuasive, although the particular matching procedures cannot rule out every difference in difficulty or establish where information is stored.

Critical weaknesses
  1. Final-layer ablation sensitivity does not yet establish a memorization-specific storage mechanism. Section 5.2 and Appendix G.5 report concentration of ablated weights in the final layer for both memorized and clean nodes. In my assessment, this leaves open how much of the observed layer preference is specific to memorization rather than general prediction sensitivity. Appendix E offers a collateral-cost explanation, but Eq. 20 uses a classifier matrix shared across nodes: changing that matrix can alter several nodes' logits through their respective embeddings. In my reading, the equation alone therefore does not establish that final-layer edits are node-specific or have lower collateral cost. This qualification matters because final-layer localization is a central conclusion and motivates the defense. It does not negate the lower ablation scores of memorized nodes.

  2. The pruning defense needs an operational specification. Section 6 describes using AWS to remove memorization-critical weights in the final layer, but I could not locate the rule aggregating per-node attributions into a single ranking of final-layer weights or the exact node set used by the defender. Those details determine reproducibility and the required information; their absence does not establish that corruption identities were used. Table 2 reports an encouraging operating point on Cora/GCN at corruption rate 0.01 and masking budget 10% of final-layer weights, with MIA AUC 0.51 and test accuracy 0.81. Figure 4 also sweeps masking budgets for three methods. Weight decay and rewiring appear at single settings, however, so I read the evidence as supporting these operating-point comparisons more directly than a general claim of the best privacy–utility tradeoff.

Questions for authors
  1. When interventions are restricted to each layer at comparable cost, is the difference between memorized and matched clean nodes largest in the final layer? Measuring collateral prediction changes would also help assess Appendix E's explanation. Alternatively, can the localization claim be scoped more closely to what the present intervention establishes?

  2. What node set, attribution-aggregation rule, and ranking procedure define the pruning defense, and does it require knowledge of corruption identities? With that procedure specified, how do the reported gains compare with weight decay and rewiring over comparable utility ranges?

4 comments
AuthorsOpenReview ↗

Response to Reviewer YWue (1/2)

We thank the Reviewer for recognizing the significance of this work as addressing a worthwhile question with complementary measurements and node definitions, for finding Figure 2's homophily-dependent gradient-alignment pattern to be an interesting empirical finding in the studied setting. We address the Reviewer’s concerns below and have revised the paper accordingly.

W1/Q1: Final-layer ablation sensitivity does not yet establish a memorization-specific storage mechanism.

We would like to clarify what Appendix E.2 (now as Appendix G.2 in the revised paper) intends to show. We agree that WlW_l is shared across all nodes, so a perturbation to WlW_l changes the logits of all nodes. However, Equation 20 does not claim otherwise. Rather, its claim concerns the pathway through which a perturbation affects node uu’s logit. Specifically, at the final layer, the change in node uu’s logit depends only on its own pre-computed embedding and on no other node’s embedding. In contrast, at an earlier layer k<lk < l (Equation 18), the change in node uu’s logit also depends on the layer-(k-1) embeddings of its (l-k)-hop neighbourhood. Therefore, what Appendix E.2 emphasizes is not whether a perturbation to WW affects other nodes, but the graph-coupled collateral, which exists only before the final layer.

To further verify the lower collateral cost at the final layer, we have conducted two additional experiments, following the Reviewer’s suggestion.

(1) Layer-wise ablation sensitivity. We restrict greedy ablation to one layer at a time and measure the fraction of that layer’s weights needed to flip the node’s prediction, defined by AWSk\texttt{AWS}_k (kk indicates the layer). We then compute RkR_k, the ratio between the AWSk\texttt{AWS}_k of memorized nodes and that of the non-memorized nodes, i.e., Rk=AWSkmem/AWSknonR_k = \texttt{AWS}^{mem}_k / \texttt{AWS}^{non}_k. A lower RkR_k means a bigger memorization-specific gap at layer kk. The results for GCN on syn-cora with a memorization rate of ρ=0.01\rho=0.01 are shown below.

Homophily levelR0R_0R1R_1R2R_2
0.000.820.860.38
0.300.870.880.37
0.500.900.890.45
0.700.910.870.65
1.000.950.910.61

We can see a substantially larger memorization-specific gap, i.e., 1-RkR_k, in the final layer than in earlier layers. For example, at h=0.00h=0.00, the gap is 0.62 in the final layer, compared with 0.18 and 0.14 in Layers 0 and 1, respectively. This indicates that the observed layer preference is specific to memorization, rather than simply reflecting general prediction sensitivity.

(2) Layer-wise collateral prediction changes. We also measure the collateral prediction flip rates per layer on other nodes rather than the target node (memorized nodes). These nodes are separated into graph neighbors and non-neighbors. The results for GCN on syn-cora with a memorization rate of ρ=0.01\rho=0.01 and homophily level h=0.50h=0.50 are shown below.

Collateral prediction flip rate (%):

Layer ablatedNeighbor flip rateNon-neighbor flip rate
Layer 015.425.22
Layer 114.234.57
Layer 23.454.03

The results show that the layers differ in graph-coupled collateral effects. Specifically, ablating earlier layers (i.e., Layers 0 or 1) changes the predictions of the target node’s neighbors approximately three times as often as those of non-neighbors (e.g., 15.42% vs. 5.22% for Layer 0). In contrast, ablating the final layer exhibits comparable flip rates for neighbors and non-neighbors (i.e., 3.45% vs. 4.03%). This indicates that final layer perturbations largely avoid the neighborhood-wide collateral effects that earlier layers bring.

Together, these two additional experiments show that the final-layer localization is associated with memorization, rather than merely reflecting general prediction sensitivity. We have revised the Appendix E.2 (now as Appendix G.2 in the revised paper) accordingly and added these experiments in Appendix H.

AuthorsOpenReview ↗

Response to Reviewer YWue (2/2)

W2/Q2: The pruning defense needs an operational specification.

Our pruning defense does not require prior knowledge of corruption identities. The full procedure is as follows: (1) The node set used in the pruning defense is the full training set D\mathcal{D}. The defender needs only the trained model and its training data. (2) For each node v∈Dv \in \mathcal{D}, we run the greedy ablation of Algorithm 1 and record the final-layer weights it removes. (3) Each final-layer weight is scored by its summed importance (Equation 3) across all nodes. (4) We zero the top γ\gamma fraction of final-layer weights ranked by their summed importance scores. Aggregating over all nodes still targets memorization, because memorized nodes depend on few, specific weights, and those weights have significantly higher importance scores than those associated with non-memorized nodes. The “AWS (Mislabeled)” column of Table 2 reports how the AWS of memorized nodes changes after each defense, indicating whether the defense successfully removes weights on which their predictions depend.

To compare our localization-guided pruning defense with weight decay and rewiring at comparable utility levels, we conducted an additional experiment below:

Comparison of localization-guided pruning, weight decay and rewiring (GCN, Cora, $\rho=0.01$, $\gamma=10%$):

DefenseTrain AccTest AccMIA AUCTPR@1%FPR
None0.890.860.660.11
Weight Decay0.810.790.640.10
Rewiring0.800.780.620.09
Localization-guided Pruning0.830.810.510.03

We can observe that under comparable test accuracy, our localization-guided pruning still achieves a lower MIA AUC, and TPR@1%FPR than weight decay and rewiring defenses. We have added the complete operational procedure of our pruning defense to Section 6 of the paper and provided Algorithm 2 in Appendix D. We have also updated the results for weight decay and rewiring defenses in Table 3, Section 6 to compare our defense with them under comparable utility levels.

If these additional experiments and clarifications address the reviewer's concerns, we would be grateful if they could update their score accordingly.

Reviewer_YWueOpenReview ↗

Response to rebuttal

Thank you for running the additional experiments. They address my original concerns, I will raise my rating to 3.

AuthorsOpenReview ↗

Thanks for acknowledging the rebuttal

We kindly thank the Reviewer for the recognition of our additional results and for raising their score to positive. We remain available shall any additional questions arise.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their constructive feedback and valuable suggestions. In response to the reviews, we have conducted additional experiments and analyses to further validate our findings, investigate the generalisability of our observations, and strengthen the theoretical and practical contributions of our work. We believe that these additional experiments and addressing the individual concerns raised by the reviewers have helped us further improve our work significantly.

Specifically, we have extended our experiments as below:

(1) Generalisability across GNN architectures and graph datasets. We extended our gradient alignment analysis to H2GCN (see Appendix J), a representative heterophily-aware GNN architecture, and Graph Transformer (see Appendix N), a more advanced GNN backbone. We also evaluated the gradient alignment patterns on three larger graph datasets (see Appendix N), i.e., ogbn-arxiv, Coauthor-CS, and Coauthor-Physics, containing up to 169,000 nodes. These experiments demonstrate that the observed homophily-dependent gradient alignment is not limited to vanilla/general GNN architectures or small synthetic benchmarks.

(2) Identifying the GNN-specific phenomena of layer-wise memorization. We conducted additional experiments using a standard MLP and an SGC-style GNN (see Section 5.2), where graph propagation is performed as fixed preprocessing before the learnable layers. Unlike GCN, neither model exhibits an increasing LMS gap with layer depth. This provides further evidence that the increasing concentration of memorization in deeper layers is associated with learnable graph-coupled message passing.

(3) Further validating final-layer memorization concentration. We conducted two additional layer-wise analyses (see Appendix H): (i) layer-wise ablation sensitivity, which quantifies the memorization-specific difference in the number of weights required to flip predictions, and (ii) layer-wise collateral prediction changes, which compare the effects of weight ablation on neighbouring and non-neighbouring nodes. The results provide further empirical evidence that the final-layer has substantially less graph-coupled collateral than earlier layers, and is more associated with memorization.

(4) Investigating the relationship between memorization and LMS gap. We conducted an additional linear-probe retrievability experiment (see Appendix E) to investigate how the memorization relates to the layer-wise LMS gap. We show that the memorized label becomes increasingly retrievable across the layer depths. This provides additional empirical evidence supporting the relationship between the layer-wise LMS gap and memorization.

(5) Evaluating the practical effectiveness of our defense. We additionally compared our localization-guided pruning defense against weight decay and graph rewiring under comparable utility levels (see Table 3, Section 6). Our defense achieves a more favourable privacy-utility trade-off, with lower membership inference attack MIA AUC and TPR@1%FPR while maintaining comparable test accuracy.

Overall, these additional experiments strengthen our work in three important aspects: (i) generalisability, by extending the findings to diverse GNN architectures and larger graph datasets; (ii) GNN-specific mechanism understanding, by distinguishing memorization behaviour in GNNs from that in conventional DNNs and fixed-propagation graph models, and by further validating final-layer memorization concentration; and (iii) practical defense, by providing further evidence for the privacy-utility benefits of our proposed defense.

We have incorporated the corresponding experimental results and discussions into the revised manuscript and its appendices. We believe these additions substantially strengthen the empirical and theoretical support for our findings and clarify the GNN-specific contributions of this work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.