- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents an interpretability study of graph language models (GLMs) and identifies "graph sink tokens". Graph‑derived tokens exhibit extreme outlier activations on a small set of hidden dimensions. The authors test two representative GLM architectures (LLaGA, TEA‑GLM) across 7 LLM backbones and 7 graph datasets on node classification and link prediction, using pruning, embedding swapping, activation patching, and linear probing. The central empirical observation is a dissociation. High‑activation sink tokens, despite carrying probe‑readable task information at late layers, have negligible causal impact on model predictions.
Critical strengths
- The core observation is a useful cautionary note for the GLM community. The paper explicitly challenges the implicit field-wide heuristic that activation magnitude or linear-probe accuracy proxies for a token's causal contribution to prediction. The demonstration that "probe-readable" information is not necessarily "causally used" by the model is a relevant warning for interpretability work on graph-LLM systems.
- Empirical scope and intervention design are solid. The study covers two structurally distinct, widely adopted GLM paradigms (neighborhood-serialization with padding, and GNN-compressed learnable graph tokens) across diverse LLM families and graph datasets. The authors include matched non-sink control groups for every pruning/patching intervention, and run extensive threshold-sensitivity and rank-based ablation checks to rule out hyperparameter-driven artifacts.
Critical weaknesses
- The paper advertises mechanistic analysis yet remains purely phenomenological. While the authors thoroughly characterize graph‑sink‑token behaviours, they do not investigate why sink tokens emerge. It is unclear whether sinks stem from padding embeddings, graph‑projector artifacts, or intrinsic LLM Transformer dynamics. The origin of sink tokens in padding‑free TEA‑GLM is also left unexplained. Without root‑cause investigation, the submission does not deliver on its core title promise.
- TEA‑GLM results suffer from a critical evaluation confound. Most TEA‑GLM experiments rely on zero‑shot cross‑dataset transfer, where models often collapse to near‑constant predictions. Under this setting, weak performance changes after graph‑token ablation cannot reliably distinguish low token importance versus a fully inert graph module dominated by text prompts. No supervised fine‑tuned baseline is provided to resolve this ambiguity. Combined with its tiny fixed token budget of only five graph tokens, TEA‑GLM results cannot support general conclusions about graph‑sink phenomena.
Questions for authors
-
This paper is positioned as a mechanistic analysis, yet the root cause of graph sink tokens remains unknown. Could you design controlled experiments to identify whether graph‑sink behaviour stems from padding embeddings, graph‑projector artifacts, or intrinsic LLM dynamics? Please also address why sinks still emerge in padding‑free TEA‑GLM.
-
Most TEA‑GLM evaluations use zero‑shot cross‑dataset transfer, where models may collapse to near‑constant predictions. Can you provide results from supervised fine‑tuned TEA‑GLM on target datasets to validate whether your conclusions hold beyond this setting?
-
Linear probes can decode node labels from sink tokens but pruning sinks barely alters predictions. How do you rule out the alternative that probe signals come from global residual‑stream information leakage instead of locally stored task information within sink tokens?
-
Please explicitly state the generalization boundary of your findings in the main text, given that experiments are only conducted on two GLM architectures.
5 comments
Response to Reviewer ZvBW (Part 2/3)
[Continued] [W1, Q1] The root cause of graph sink tokens.
Graph projector. We replace the trained projector of each GLM with an untrained projector that has the same layers but random weights, and keep everything else fixed. We also use the released padding-free LLaGA hop-view (HO) checkpoint, whose graph tokens have large norms, and give its graph slots vectors at the norm of text tokens.
| GLM | Change | #Sinks: original->modified |
|---|---|---|
| LLaGA | Random projector | ×3.6-4.6 |
| TEA-GLM (Qwen2.5-7B) | Random projector | 2.15->2.24 |
| TEA-GLM (Llama-3.1-8B) | Random projector | 1.83->1.74 |
| LLaGA-HO | Text-norm inputs | 0->2.56 |
A trained projector is unnecessary for sink formation, as random LLaGA projectors produce 3.6–4.6 times more sinks, random TEA-GLM projectors maintain stable number of sinks. Padding-free LLaGA-HO yields 2.56 sinks per example and layer only after graph tokens are rescaled to text-token norms, demonstrating that projectors impact sinks exclusively through the norm of their outputs.
Intrinsic LLM dynamics. We learn from figure 1 that each LLM has its own sink dimensions, and the dimension values stay fixed regardless of input data. To test whether the LLM produces these massive activations by itself, we ask the following: whether the same dimensions appear without any graph tokens, which components of the LLM write the massive activations into graph tokens, and whether zeroing these components removes the sinks.
| Test | Result |
|---|---|
| Text-only prompts | Same dimensions, sinks at first token or delimiter |
| Components that create sinks | First-block MLP; attention heads in later blocks |
| Zeroing these heads | Sinks drop by up to 33%; random heads: no change |
The table shows that the same dimensions carry massive activations even without graph tokens, on the first token or delimiter. Two components of the LLM write these activations into graph tokens: the first-block MLP, into the [PAD] tokens of LLaGA, and attention heads in later blocks, whose zeroing lowers the number of sinks by up to 33% while zeroing random heads has no effect. Thus, graph sinks are the LLM's own massive activations, written into graph tokens by its MLP and attention heads, as into <BOS> and punctuation tokens.
Why sinks emerge in padding-free TEA-GLM. Sinks do not require [PAD] tokens: as shown in response part (1/3), real-node tokens that carry node information also become sinks when their norm is low. TEA-GLM’s final LayerNorm keeps its graph tokens at a fixed norm (60-64). We note that they do not form as sinks in early layers: in Vicuna-7B and Llama-3.1-8B, none is a sink in the first 25th quantile of the layers. They become sinks at later layers, through attention: in the last 25th quantile of the layers, 75-92% of the TEA-GLM graph tokens are sinks, written by the attention heads in the table above. Zeroing these heads at graph-token positions lowers the number of TEA-GLM sinks by up to 33%, while zeroing random heads has no effect, and accuracy stays within 1.33%. TEA-GLM sinks thus come from the LLM's own attention, not from padding.
[1] Darcet et al. Vision Transformers Need Registers. ICLR, 2024.
[2] Kang et al. See what you are told: Visual attention sink in large multimodal models. ICLR, 2025.
[3] Luo et al. To sink or not to sink: Visual information pathways in large vision-language models. ICLR, 2026.
[4] Choi et al. When sinks help or hurt: Unified framework for attention sink in large vision-language models. ECCV, 2026.
[5] Jiang et al. Vision Transformers Don’t Need Trained Registers. NeurIPS, 2025.
[6] Sun et al. Massive activations in large language models. Arxiv, 2024.
[7] Barbero et al. Why do llms attend to the first token?. Arxiv, 2025.
[8] Gu et al. When attention sink emerges in language models: An empirical view. ICLR, 2025.
[9] Sun et al. The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks. Arxiv, 2026.
[10] Ruscio et al. What are you sinking? a geometric approach on attention sink. Neurips, 2026.
[11] Heimersheim and Nanda, How to Use and Interpret Activation Patching. ArXiv, 2024.
Response to Reviewer ZvBW (Part 3/3)
[W2, Q2] Supervised TEA-GLM on the target datasets.
We thank the reviewer for this question. TEA-GLM is designed as a zero-shot graph learner: its projector is trained once and then transferred to new datasets and tasks. In the paper, we therefore analyze graph sinks in its original setting, so that the analysis reflects TEA-GLM as it is intended to be used. We do agree that a supervised setting, in which the graph tokens are trained on the evaluated dataset, gives a complementary test of our conclusions, and we report it below.
We train the TEA-GLM projector on the training split of each evaluated dataset, using the same prompt template and subgraph sampler as the test prompts, with Qwen2.5-7B on five settings (NC on Cora and PubMed, LP on Arxiv, Cora, and PubMed). We also double the number of graph tokens to K = 10, and evaluate on the same 300 test examples as in the paper. We repeat the pruning experiments of Table 3, where we report the accuracy of pruning sinks and pruning random non-sinks.
| Intervention | NC Cora | NC PubMed | LP Arxiv | LP Cora | LP PubMed |
|---|---|---|---|---|---|
| Baseline | 31.33 | 90.00 | 90.00 | 81.67 | 87.67 |
| Prune sinks | 31.00 | 86.67 | 80.00 | 65.00 | 83.00 |
| Non-sink | 20.87±2.43 | 85.33±0.60 | 65.27±1.65 | 62.53±1.15 | 85.27±1.25 |
The table shows that pruning sinks do not lower accuracy significantly more than pruning the same number of non-sinks. In NC Cora and LP Arxiv, non-sink pruning lowers accuracy by 10.5 and 24.7%, compared with 0.3 and 10.0% for sink pruning.Thus, with supervised training, the graph tokens with large activations are still not the tokens that the model relies on most.
[Q3] Is the probe signal stored in the sink tokens, or does it come from other positions?
We thank the reviewer for this question. As the reviewer suggests, the label information in the sink tokens comes from the other graph tokens through attention. This explains a result in the paper: a probe can read node labels from the sink tokens, yet the model does not rely on these tokens for prediction. We show both below.
Attention knockout. We apply attention knockout to the attention from the sink tokens to all other graph tokens: at every layer, the sink tokens cannot attend to any other graph token, while every other attention entry stays unchanged, so the sink tokens still attend to themselves and to the text before the graph.
The table shows how attention knockout changes two things on each backbone: how well the Table 5 probe predicts node labels from the final-state representations of three groups of graph tokens (the "Final" column of Table 5), and the model's own prediction accuracy (%). Each cell gives the value before->after the knockout, and a probe with no label information gives 19.5%, the rate of the most frequent class.
| Backbone | [PAD] sinks | non-[PAD] tokens | All graph tokens | Accuracy |
|---|---|---|---|---|
| Vicuna-7B | 59.6->16.6 | 52.8->52.2 | 60.2->61.2 | 77.3->77.0 |
| Llama-3.1-8B | 61.6->16.9 | 62.4->62.6 | 61.4->61.0 | 66.7->67.0 |
| Qwen2.5-7B | 59.5->15.8 | 60.6->60.3 | 60.8->60.8 | 60.7->60.7 |
The table shows that the knockout removes the label information from the sink tokens: probe accuracy falls to or below chance, while the probes on the other graph tokens change by at most 1.0% and the model's prediction accuracy by at most 0.3%.
This result supports the reviewer’s point: the label information leaks into the sink tokens from the other graph tokens through attention, and the model does not need it for prediction. We clarify that our probe experiment shows only that late sink tokens contain label information that a linear probe can read. It does not claim that the sink tokens produce or store this information themselves, and we agree that it comes from other positions. We thank the reviewer for this suggestion, which makes our statement more precise and rigorous.
Revision. We will change “sink states contain node-label information” to “sink states receive node-label information from other graph tokens through attention,” and add the attention knockout test in the revision.
[Q4] Generalization boundary of the findings.
We study two of the widely used GLM designs with seven backbones from four LLM families (7B to 72B), seven text-attributed graphs from the citation and e-commerce domains, and two tasks, node classification and link prediction. As discussed in [W1, Q1], this paper uses causal interventions and mechanistic analyses to study the properties of graph sinks in both designs. The overarching goal of our work is to lay the foundation of defining and analyzing graph sinks across different thrusts. Future work analyzing specialized GLM architectures with complex graph encoders, alignment objectives, or graph-token layouts, can use our tests as a recipe to understand, identify, and mitigate sink behavior in GLMs.
Response to Reviewer ZvBW (Part 1/3)
We sincerely appreciate the reviewer for recognizing that our core observation is
a useful cautionary note for the GLM community, that the paperchallenges the implicit field-wide heuristic, and that theempirical scope and intervention design are solid, withmatched non-sink control groups for every pruning/patching intervention. We address each comment below.We thank the reviewer for this important comment. We first clarify the scope of our mechanistic analysis, and then report controlled experiments on the origin of graph sinks, including in padding-free TEA-GLM.
Two lines of work on sink tokens. Research on sink tokens follows two related but distinct lines. The first studies what sink tokens represent and how they affect the model’s computation, using causal interventions and mechanistic analyses in language and vision models [1-5]. The second asks why sinks emerge [6-10]. Our work belongs to the first line and extends it to the graph modality. We ask whether graph tokens that carry massive activations are useful for prediction, and answer it with the causal interventions and probes of mechanistic interpretability [11]: pruning, swapping, activation patching, label probing, and logit lens. These analyses separate activation magnitude, readable label information, and predictive contribution, and this finding does not depend on how sinks form.
We agree that the origin of sinks is the natural next question, and we thank the reviewer for suggesting the experiments below.
Controlled experiments. We use Arxiv NC (300 examples per configuration) with Vicuna-7B, Qwen2.5-7B, and Llama-3.1-8B in both GLMs, change only the vectors at graph-token positions, and report sink tokens per example and layer.
Padding embeddings. LLaGA sets every
[PAD]embedding to a zero vector. Removing the[PAD]tokens without retraining removes almost all sinks, while accuracy changes by at most 3.0% (kindly see Reviewer xyke [Q1]):[PAD]To find which property of
[PAD]causes the sinks, we conduct intervention experiments to change the[PAD]and node vectors directly.[PAD]tokens are identical to each other and have zero norm. We add small, distinct noise to each[PAD]vector, which keeps the norm low, replace[PAD]with vectors at the norm of real-node embeddings (300-1,400), and rescale real-node embeddings down to the norm of text tokens:[PAD][PAD]with real-node norm vectorsThe table shows that the sinks come from the low norm of the
[PAD]vectors. Different noise does not change the number of sink tokens, but replacing[PAD]with real-node norm removes them, and real nodes, which carry node features, become sinks when rescaling its norm to that of text tokens. A graph token therefore becomes a sink when its input norm is low, whether it is a[PAD]token or a real node.