acceptodds
Under review as a conference paper at ICLR 2027

When Graph Tokens Sink: A Mechanistic Analysis of Graph Language Models

Abstract

Graph Language Models (GLMs) have become a promising direction for adapting Large Language Models (LLMs) to graph learning tasks. By transforming graph topology and node information into graph tokens, GLMs allow LLMs to jointly process structured graph inputs and textual instructions. Yet, it remains unclear how the internal behavior of these tokens relates to the task information they contain and their contribution to prediction. In this work, we analyze how LLMs process graph information through graph-token behavior in two representative GLM architectures, LLaGA and TEA-GLM, across seven LLM backbones and seven datasets. **Findings.** We find that large graph-token activations do not reliably indicate greater contribution to prediction. Graph sink tokens emerge as activation-level outliers, with massive activation values along a small set of hidden-state dimensions, but do not necessarily attract the largest attention weights from query tokens. Their positional patterns differ between architectures: LLaGA sinks are overrepresented among padding tokens and rarely occur at target-node positions, whereas TEA-GLM shows no consistent positional preference across backbones and datasets. Pruning and activation patching generally affect LLaGA's node-classification predictions less when applied to sink tokens than to matched non-sink tokens. This distinction is weaker in link prediction and TEA-GLM, while swapping sink and non-sink input embeddings usually causes small accuracy changes in both architectures. Probing further shows that later LLaGA sink states contain node-label information that a simple classifier can read. **Implications.** Together, these results reveal a decoupling between activation-level saliency and graph-semantic utility. Large activations, readable label information, and predictive contribution do not necessarily coincide in the same graph tokens. This distinction provides a basis for assessing graph-token construction, placement, and alignment mechanisms beyond activation magnitude alone.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Graph Language Models, Attention Sink, Multimodal Models

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_ZvBWOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents an interpretability study of graph language models (GLMs) and identifies "graph sink tokens". Graph‑derived tokens exhibit extreme outlier activations on a small set of hidden dimensions. The authors test two representative GLM architectures (LLaGA, TEA‑GLM) across 7 LLM backbones and 7 graph datasets on node classification and link prediction, using pruning, embedding swapping, activation patching, and linear probing. The central empirical observation is a dissociation. High‑activation sink tokens, despite carrying probe‑readable task information at late layers, have negligible causal impact on model predictions.

Critical strengths
  1. The core observation is a useful cautionary note for the GLM community. The paper explicitly challenges the implicit field-wide heuristic that activation magnitude or linear-probe accuracy proxies for a token's causal contribution to prediction. The demonstration that "probe-readable" information is not necessarily "causally used" by the model is a relevant warning for interpretability work on graph-LLM systems.
  2. Empirical scope and intervention design are solid. The study covers two structurally distinct, widely adopted GLM paradigms (neighborhood-serialization with padding, and GNN-compressed learnable graph tokens) across diverse LLM families and graph datasets. The authors include matched non-sink control groups for every pruning/patching intervention, and run extensive threshold-sensitivity and rank-based ablation checks to rule out hyperparameter-driven artifacts.
Critical weaknesses
  1. The paper advertises mechanistic analysis yet remains purely phenomenological. While the authors thoroughly characterize graph‑sink‑token behaviours, they do not investigate why sink tokens emerge. It is unclear whether sinks stem from padding embeddings, graph‑projector artifacts, or intrinsic LLM Transformer dynamics. The origin of sink tokens in padding‑free TEA‑GLM is also left unexplained. Without root‑cause investigation, the submission does not deliver on its core title promise.
  2. TEA‑GLM results suffer from a critical evaluation confound. Most TEA‑GLM experiments rely on zero‑shot cross‑dataset transfer, where models often collapse to near‑constant predictions. Under this setting, weak performance changes after graph‑token ablation cannot reliably distinguish low token importance versus a fully inert graph module dominated by text prompts. No supervised fine‑tuned baseline is provided to resolve this ambiguity. Combined with its tiny fixed token budget of only five graph tokens, TEA‑GLM results cannot support general conclusions about graph‑sink phenomena.
Questions for authors
  1. This paper is positioned as a mechanistic analysis, yet the root cause of graph sink tokens remains unknown. Could you design controlled experiments to identify whether graph‑sink behaviour stems from padding embeddings, graph‑projector artifacts, or intrinsic LLM dynamics? Please also address why sinks still emerge in padding‑free TEA‑GLM.

  2. Most TEA‑GLM evaluations use zero‑shot cross‑dataset transfer, where models may collapse to near‑constant predictions. Can you provide results from supervised fine‑tuned TEA‑GLM on target datasets to validate whether your conclusions hold beyond this setting?

  3. Linear probes can decode node labels from sink tokens but pruning sinks barely alters predictions. How do you rule out the alternative that probe signals come from global residual‑stream information leakage instead of locally stored task information within sink tokens?

  4. Please explicitly state the generalization boundary of your findings in the main text, given that experiments are only conducted on two GLM architectures.

5 comments
AuthorsOpenReview ↗

Response to Reviewer ZvBW (Part 1/3)

We sincerely appreciate the reviewer for recognizing that our core observation is a useful cautionary note for the GLM community, that the paper challenges the implicit field-wide heuristic, and that the empirical scope and intervention design are solid, with matched non-sink control groups for every pruning/patching intervention. We address each comment below.

[W1, Q1] The root cause of graph sink tokens.

We thank the reviewer for this important comment. We first clarify the scope of our mechanistic analysis, and then report controlled experiments on the origin of graph sinks, including in padding-free TEA-GLM.

Two lines of work on sink tokens. Research on sink tokens follows two related but distinct lines. The first studies what sink tokens represent and how they affect the model’s computation, using causal interventions and mechanistic analyses in language and vision models [1-5]. The second asks why sinks emerge [6-10]. Our work belongs to the first line and extends it to the graph modality. We ask whether graph tokens that carry massive activations are useful for prediction, and answer it with the causal interventions and probes of mechanistic interpretability [11]: pruning, swapping, activation patching, label probing, and logit lens. These analyses separate activation magnitude, readable label information, and predictive contribution, and this finding does not depend on how sinks form.

We agree that the origin of sinks is the natural next question, and we thank the reviewer for suggesting the experiments below.

Controlled experiments. We use Arxiv NC (300 examples per configuration) with Vicuna-7B, Qwen2.5-7B, and Llama-3.1-8B in both GLMs, change only the vectors at graph-token positions, and report sink tokens per example and layer.

Padding embeddings. LLaGA sets every [PAD] embedding to a zero vector. Removing the [PAD] tokens without retraining removes almost all sinks, while accuracy changes by at most 3.0% (kindly see Reviewer xyke [Q1]):

#Sinks w/ and w/o [PAD]ArxivCoraPubMed
Vicuna-7B9.76/0.0021.07/0.0022.12/0.00
Qwen2.5-7B6.84/0.1617.72/0.1815.28/0.02
Llama-3.1-8B9.86/0.0012.81/0.0010.84/0.00

To find which property of [PAD] causes the sinks, we conduct intervention experiments to change the [PAD] and node vectors directly. [PAD] tokens are identical to each other and have zero norm. We add small, distinct noise to each [PAD] vector, which keeps the norm low, replace [PAD] with vectors at the norm of real-node embeddings (300-1,400), and rescale real-node embeddings down to the norm of text tokens:

InterventionBackbone#Sinks: original->modified
Add small, distinct noise to [PAD]Vicuna-7B9.77->9.49
Replace [PAD] with real-node norm vectorsVicuna-7B9.77->0.00
Llama-3.1-8B9.86->0.00
Rescale real nodes to text-token normVicuna-7B0.00->28.65
Qwen2.5-7B0.17->19.54
Llama-3.1-8B0.00->20.74

The table shows that the sinks come from the low norm of the [PAD] vectors. Different noise does not change the number of sink tokens, but replacing [PAD] with real-node norm removes them, and real nodes, which carry node features, become sinks when rescaling its norm to that of text tokens. A graph token therefore becomes a sink when its input norm is low, whether it is a [PAD] token or a real node.

AuthorsOpenReview ↗

Response to Reviewer ZvBW (Part 2/3)

[Continued] [W1, Q1] The root cause of graph sink tokens.

Graph projector. We replace the trained projector of each GLM with an untrained projector that has the same layers but random weights, and keep everything else fixed. We also use the released padding-free LLaGA hop-view (HO) checkpoint, whose graph tokens have large norms, and give its graph slots vectors at the norm of text tokens.

GLMChange#Sinks: original->modified
LLaGARandom projector×3.6-4.6
TEA-GLM (Qwen2.5-7B)Random projector2.15->2.24
TEA-GLM (Llama-3.1-8B)Random projector1.83->1.74
LLaGA-HOText-norm inputs0->2.56

A trained projector is unnecessary for sink formation, as random LLaGA projectors produce 3.6–4.6 times more sinks, random TEA-GLM projectors maintain stable number of sinks. Padding-free LLaGA-HO yields 2.56 sinks per example and layer only after graph tokens are rescaled to text-token norms, demonstrating that projectors impact sinks exclusively through the norm of their outputs.

Intrinsic LLM dynamics. We learn from figure 1 that each LLM has its own sink dimensions, and the dimension values stay fixed regardless of input data. To test whether the LLM produces these massive activations by itself, we ask the following: whether the same dimensions appear without any graph tokens, which components of the LLM write the massive activations into graph tokens, and whether zeroing these components removes the sinks.

TestResult
Text-only promptsSame dimensions, sinks at first token or delimiter
Components that create sinksFirst-block MLP; attention heads in later blocks
Zeroing these headsSinks drop by up to 33%; random heads: no change

The table shows that the same dimensions carry massive activations even without graph tokens, on the first token or delimiter. Two components of the LLM write these activations into graph tokens: the first-block MLP, into the [PAD] tokens of LLaGA, and attention heads in later blocks, whose zeroing lowers the number of sinks by up to 33% while zeroing random heads has no effect. Thus, graph sinks are the LLM's own massive activations, written into graph tokens by its MLP and attention heads, as into <BOS> and punctuation tokens.

Why sinks emerge in padding-free TEA-GLM. Sinks do not require [PAD] tokens: as shown in response part (1/3), real-node tokens that carry node information also become sinks when their norm is low. TEA-GLM’s final LayerNorm keeps its graph tokens at a fixed norm (60-64). We note that they do not form as sinks in early layers: in Vicuna-7B and Llama-3.1-8B, none is a sink in the first 25th quantile of the layers. They become sinks at later layers, through attention: in the last 25th quantile of the layers, 75-92% of the TEA-GLM graph tokens are sinks, written by the attention heads in the table above. Zeroing these heads at graph-token positions lowers the number of TEA-GLM sinks by up to 33%, while zeroing random heads has no effect, and accuracy stays within 1.33%. TEA-GLM sinks thus come from the LLM's own attention, not from padding.

[1] Darcet et al. Vision Transformers Need Registers. ICLR, 2024.

[2] Kang et al. See what you are told: Visual attention sink in large multimodal models. ICLR, 2025.

[3] Luo et al. To sink or not to sink: Visual information pathways in large vision-language models. ICLR, 2026.

[4] Choi et al. When sinks help or hurt: Unified framework for attention sink in large vision-language models. ECCV, 2026.

[5] Jiang et al. Vision Transformers Don’t Need Trained Registers. NeurIPS, 2025.

[6] Sun et al. Massive activations in large language models. Arxiv, 2024.

[7] Barbero et al. Why do llms attend to the first token?. Arxiv, 2025.

[8] Gu et al. When attention sink emerges in language models: An empirical view. ICLR, 2025.

[9] Sun et al. The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinks. Arxiv, 2026.

[10] Ruscio et al. What are you sinking? a geometric approach on attention sink. Neurips, 2026.

[11] Heimersheim and Nanda, How to Use and Interpret Activation Patching. ArXiv, 2024.

AuthorsOpenReview ↗

Response to Reviewer ZvBW (Part 3/3)

[W2, Q2] Supervised TEA-GLM on the target datasets.

We thank the reviewer for this question. TEA-GLM is designed as a zero-shot graph learner: its projector is trained once and then transferred to new datasets and tasks. In the paper, we therefore analyze graph sinks in its original setting, so that the analysis reflects TEA-GLM as it is intended to be used. We do agree that a supervised setting, in which the graph tokens are trained on the evaluated dataset, gives a complementary test of our conclusions, and we report it below.

We train the TEA-GLM projector on the training split of each evaluated dataset, using the same prompt template and subgraph sampler as the test prompts, with Qwen2.5-7B on five settings (NC on Cora and PubMed, LP on Arxiv, Cora, and PubMed). We also double the number of graph tokens to K = 10, and evaluate on the same 300 test examples as in the paper. We repeat the pruning experiments of Table 3, where we report the accuracy of pruning sinks and pruning random non-sinks.

InterventionNC CoraNC PubMedLP ArxivLP CoraLP PubMed
Baseline31.3390.0090.0081.6787.67
Prune sinks31.0086.6780.0065.0083.00
Non-sink20.87±2.4385.33±0.6065.27±1.6562.53±1.1585.27±1.25

The table shows that pruning sinks do not lower accuracy significantly more than pruning the same number of non-sinks. In NC Cora and LP Arxiv, non-sink pruning lowers accuracy by 10.5 and 24.7%, compared with 0.3 and 10.0% for sink pruning.Thus, with supervised training, the graph tokens with large activations are still not the tokens that the model relies on most.

[Q3] Is the probe signal stored in the sink tokens, or does it come from other positions?

We thank the reviewer for this question. As the reviewer suggests, the label information in the sink tokens comes from the other graph tokens through attention. This explains a result in the paper: a probe can read node labels from the sink tokens, yet the model does not rely on these tokens for prediction. We show both below.

Attention knockout. We apply attention knockout to the attention from the sink tokens to all other graph tokens: at every layer, the sink tokens cannot attend to any other graph token, while every other attention entry stays unchanged, so the sink tokens still attend to themselves and to the text before the graph.

The table shows how attention knockout changes two things on each backbone: how well the Table 5 probe predicts node labels from the final-state representations of three groups of graph tokens (the "Final" column of Table 5), and the model's own prediction accuracy (%). Each cell gives the value before->after the knockout, and a probe with no label information gives 19.5%, the rate of the most frequent class.

Backbone[PAD] sinksnon-[PAD] tokensAll graph tokensAccuracy
Vicuna-7B59.6->16.652.8->52.260.2->61.277.3->77.0
Llama-3.1-8B61.6->16.962.4->62.661.4->61.066.7->67.0
Qwen2.5-7B59.5->15.860.6->60.360.8->60.860.7->60.7

The table shows that the knockout removes the label information from the sink tokens: probe accuracy falls to or below chance, while the probes on the other graph tokens change by at most 1.0% and the model's prediction accuracy by at most 0.3%.

This result supports the reviewer’s point: the label information leaks into the sink tokens from the other graph tokens through attention, and the model does not need it for prediction. We clarify that our probe experiment shows only that late sink tokens contain label information that a linear probe can read. It does not claim that the sink tokens produce or store this information themselves, and we agree that it comes from other positions. We thank the reviewer for this suggestion, which makes our statement more precise and rigorous.

Revision. We will change “sink states contain node-label information” to “sink states receive node-label information from other graph tokens through attention,” and add the attention knockout test in the revision.

[Q4] Generalization boundary of the findings.

We study two of the widely used GLM designs with seven backbones from four LLM families (7B to 72B), seven text-attributed graphs from the citation and e-commerce domains, and two tasks, node classification and link prediction. As discussed in [W1, Q1], this paper uses causal interventions and mechanistic analyses to study the properties of graph sinks in both designs. The overarching goal of our work is to lay the foundation of defining and analyzing graph sinks across different thrusts. Future work analyzing specialized GLM architectures with complex graph encoders, alignment objectives, or graph-token layouts, can use our tests as a recipe to understand, identify, and mitigate sink behavior in GLMs.

Reviewer_vRryOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper analyzes graph tokens with unusually large activations in LLaGA and TEA-GLM, examining the relationship between activation magnitude, task information, and predictive contribution. The experiments cover seven LLM backbones and seven datasets and use pruning, swapping, and activation patching to examine the role of these tokens. The main finding is that large activations do not necessarily indicate greater predictive contribution. In LLaGA node classification, interventions on sink tokens generally have less effect on predictions than interventions on non-sink tokens, even though later sink states contain node-label information that can be decoded.

Critical strengths
  1. The paper asks a worthwhile question. Graph tokens are intended to convey graph structure to an LLM, but overall task performance alone does not establish whether or how the model uses these tokens for prediction. Separating activation magnitude, readable label information, and predictive contribution provides a useful way to examine how graph language models process graph inputs.
  2. The authors are careful about the scope of their findings. They explicitly note that the distinction between sink and non-sink tokens is most evident in LLaGA node classification. They also flag TEA-GLM link-prediction configurations with constant or near-constant predictions and exclude them when interpreting token importance, rather than treating unchanged predictions in degenerate settings as evidence that the tokens are unimportant. The appendix includes threshold sensitivity tests and rank-based controls without a token threshold, providing additional support for the observations.
Critical weaknesses
  1. The main pruning control in LLaGA does not fully separate the effects of sink status from those of padding. Table 1 shows that 70.5%–100% of detected sinks on the citation datasets occur at [PAD] positions. The random non-sink control explicitly matches the number of deletions at each block, but the composition of the deleted tokens in the two groups is not reported. Table 12 adds a random-padding control, but its coverage is limited and it uses rank-based exclusion rather than directly reproducing the cumulative deletion setting in the main experiments. The performance difference may therefore partly reflect differences in the node information carried by the deleted tokens. This does not invalidate the observation that large activations need not imply greater predictive contribution, but it leaves the role of sink status itself less clear. This is my main reservation about this part of the analysis.
Questions for authors
  1. Which transformer blocks were used for activation patching? If multiple blocks were tested, how were their results aggregated in Table 4, and was the difference between sink and non-sink patches consistent across depths? This matters because a small effect at later layers could reflect information having already reached other token states.
  2. In the pruning experiment, tokens are selected at each layer based on the intervened states, and deletions accumulate across layers. Could deletions in earlier layers change which tokens are identified as sinks in later layers? Have the authors compared these selections with those from an unmodified run? It would help to understand whether such changes affect the comparison between sink and non-sink pruning.
4 comments
AuthorsOpenReview ↗

Response to Reviewer vRry (Part 1/2)

We sincerely appreciate the reviewer for recognizing that the paper asks a worthwhile question that separating activation magnitude, readable label information, and predictive contribution provides a useful way to examine how GLMs process graph inputs and the paper is careful about the scope of its findings, including the flagged TEA-GLM configurations and the threshold and rank-based controls in the appendix. We address each comment below.

[W1] The pruning control does not separate sink status from padding.

We thank the reviewer for this thoughtful comment. As suggested, we report the composition of the pruned tokens for both sink and non-sink tokens. We rerun the pruning experiments of Table 3 on five backbones (Vicuna-7B, Qwen2.5-7B, Llama-3.1-8B, Yi-1.5-9B, Qwen2.5-14B) and in total of 50 LLaGA configurations (NC on three citation and four e-commerce datasets, LP on three citation datasets), using the first 100 evaluation examples per configuration.

Composition of the pruned tokens.

For each config, we report the percentage of [PAD] tokens and also the mean number of real-node tokens (non-[PAD] tokens).

ConfigsPruned tokens[PAD]%Real-node tokens
NC citationSinks99.0%0.3
Non-sinks46.8%12.3
NC e-commerceSinks73.0%11.8
Non-sinks37.9%18.1
LP citationSinks99.8%0.1
Non-sinks30.9%12.2

Results show that for NC and LP citation datasets sink token pruning, 99.0% and 99.8% of the sink tokens are [PAD] tokens, which follows our results in Table 1. For random non-sink token pruning, the results show that the majority of the token composition are real-node, non-[PAD] graph tokens. In all configurations tested here, It removes an average of 14.2 real-node tokens per example.

Pruning sinks without [PAD] tokens. To separate the effects of sink tokens from those of [PAD] tokens, we re-conduct the original pruning intervention for Qwen2.5-7b on e-commerce datasets, using the exact evaluation setup from the paper (line 269-283). Instead of identifying and pruning sink tokens starting at layer l=1l=1, we locate the layer l∗l^* where real-node tokens make up the largest fraction of all sink tokens.

Datasetl∗l^*Real-node fraction# sinks prunedBaseline %Prune Sinks %
Computer90.922.6790.6790.67
History90.912.5068.3368.67
Photo90.973.3280.6779.33
Sports230.5214.8992.0092.00

Results above show that even if we prune sink tokens that are non-pad, actual graph node tokens, the accuracy rarely changes. The same phenomenon was also shown in our response to Reviewer xyke [Q1], where we completely remove the [PAD] tokens from the LLaGA input and rerun sink identification and pruning on NC citation datasets. Together, these results strengthen our conclusion that the effect of sink pruning does not depend on the sinks being [PAD] tokens: sinks that carry node features can also be removed without changing predictions.

AuthorsOpenReview ↗

Response to Reviewer vRry (Part 2/2)

[Q1] Which transformer blocks were patched, how Table 4 aggregates them, and whether the sink vs. non-sink difference holds across depth.

We appreciate the reviewer for pointing out this question. We clarify it below.

Transformer blocks and results aggregation for activation patching.

Backbone (#blocks)Patched blocks
Vicuna-7B (32), Llama-3.1-8B (32)1,4,8,...,30
Qwen2.5-7B (28)1,4,8,...,26
Yi-1.5-9B (48)1,4,7,...,46
Qwen2.5-14B (48)1,4,8,...,46
Llama-3.1-70B, Qwen2.5-72B (80)1,8,16,...,72,78

The table shows the transformer blocks we patch for each LLM backbone (excluding block 0 and last block). For every configuration, we compute its value in Table 4 in three steps:

  • Compute TT by replacing selected graph token states at a block's input with states from a paired example assigned to a different class, repeated across three pairs and every patched block.
  • Average TT over the three pairs and all patched blocks to get one value per example, excluding examples where denominators in Eq. 3 are close to zero.
  • Average these per-example values across the configuration's examples, with Table 4 reporting the median over configurations.
  • Because averaging across all patched blocks combines early blocks (where graph tokens have large effects) with late blocks (where effects are small), results are additionally broken down by depth.
LLaGA NCEarlyMiddleLate
Whole graph0.9680.7310.261
Random non-sinks0.0740.0320.015
Sinks0.0130.0060.003
#random non-sinks > sinks46/4941/4949/49

We split the patched blocks of each configuration into three groups of equal size by depth (early, middle, and late), compute T for each group with the steps above, and report the median over configurations. We also count the configurations in which non-sink patches give a larger T than sink patches.

The results confirm the reviewer’s point: the effect of graph tokens drops from early blocks (T=0.968T = 0.968) to late blocks (T=0.261T = 0.261) as attention transfers graph information into text tokens. We attribute this decrease to how the LLM moves information across layers: as depth increases, attention transfers graph information into the text tokens that produce the prediction, so later graph-token states determine less of the output. Since we patch sink and non-sink tokens at the same blocks, this decrease affects both. Although the absolute effect of patching decreases with depth, the order between the two stays the same. At every depth, the median TT of random non-sinks is ~5-6 times that of sinks (0.074 vs. 0.013, 0.032 vs. 0.006, and 0.015 vs. 0.003), and non-sinks give the larger effect in most of 49 configurations. In particular, the gap already appears in the early blocks, where the graph tokens still determine almost the entire prediction. Thus, the small effect of sinks does not come from graph information having already reached other tokens at later blocks.

[Q2] Whether earlier deletions change which tokens are selected as sinks at later blocks, and whether this affects the sink vs. non-sink comparison.

To determine whether earlier deletions change which tokens are selected at later blocks, we record the tokens pruned at every block and compare them with the original model without pruning, under the same settings as in [W1]. We report three measures for each type of pruning. Sink at the same block is the percentage of pruned tokens that the original model also identifies as sinks at that block, and sink at any block is the percentage that it identifies as sinks at one or more blocks. Original top-ranked tokens removed is the percentage of the original model's top-ranked tokens, the two highest-scoring tokens at any block, that pruning has removed by the last block.

ConfigsPruned tokensSink at the same blockSink at any blockTop-ranked tokens original removed
NC citationSinks89.4%99.9%99.5%
Non-sink4.3%57.9%11.9%
NC e-commerceSinks89.8%100.0%99.5%
Non-sink2.1%69.9%10.3%
LP citationSinks94.9%100.0%97.2%
Non-sink1.3%57.6%5.6%

Sink pruning removes the tokens that the original model identifies as sinks: in NC citation, 89.4% of the pruned tokens are sinks at the same block and 99.9% at some block, and by the last block, sink pruning has removed 99.5% of the original model's top-ranked tokens. For control experiments, 95.7% of the tokens it removes are non-sinks at the same block. Note that values in "Sink at any block" (57.6-69.9%) come from tokens that become sinks at other blocks, such as most [PAD] token in the citation datasets, and at the block where non-sink pruning removes them, these tokens are not sinks. The same pattern holds in NC e-commerce and LP citation. Thus, earlier deletions change the selected sinks very slightly, and the comparison in Table 3 remains a comparison between the sinks and non-sinks of the original model.

Thank you for the response, which mitigates my concerns to some degree. Nevertheless, I will keep my original score unchanged.

We sincerely thank you for the positive assessment and for your valuable suggestions, which helped us improve our work!

Reviewer_xykeOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a mechanistic analysis of graph sink tokens in Graph Language Models (GLMs). The authors study LLaGA and TEA-GLM across seven LLM backbones, seven graph datasets, and node-classification and link-prediction tasks. They identify graph sinks as tokens with unusually large activations along a small set of hidden dimensions and analyze their positions, attention patterns, information content, and causal contribution. The results show that activation magnitude is not a reliable indicator of predictive importance: LLaGA sinks occur predominantly at padding positions, rarely correspond to target nodes, and generally affect node-classification predictions less than non-sink tokens under pruning and activation patching. Nevertheless, later sink representations contain node-label information detectable by a linear probe. Overall, the paper distinguishes activation saliency, decodable information, and causal contribution as separate properties of graph-token representations.

Critical strengths

1.The paper studies an important and underexplored question: whether graph tokens with extreme activations actually carry or contribute useful graph information. 2.The evaluation is extensive, covering two distinct GLM architectures, seven LLM backbones, seven datasets, and both node-classification and link-prediction tasks. The distinction between activation magnitude, decodable label information, and causal predictive contribution is conceptually useful and 3.supported by several complementary experiments.

Critical weaknesses

1.Sink identification depends on calibrated dimensions and fixed activation thresholds. Although sensitivity analyses are provided, the definition remains architecture-dependent, and it is unclear whether it captures one coherent phenomenon across all backbones and GLM designs. 2.The cumulative pruning intervention changes sequence length and attention normalization at every layer. It may therefore introduce effects beyond removing the information contained in the selected token, complicating the causal interpretation.

Questions for authors

Could you evaluate a variable-length LLaGA representation without [PAD] tokens to determine whether sink behavior is caused by fixed-length padding?

2 comments
AuthorsOpenReview ↗

Response to Reviewer xyke (Part 1/2)

We sincerely appreciate the reviewer for recognizing that the paper addresses an important and underexplored question, that the evaluation is extensive, and that our analyses are conceptually useful and supported by several complementary experiments. We address each comment below.

[W1] Sink identification depends on calibrated dimensions and fixed activation thresholds.

Sink dimensions are a property of the LLM backbone. Massive activations occur in a few fixed hidden dimensions specific to each LLM and is consistent across inputs [1]. For example, work on VLMs defines sink tokens with these backbone-specific dimensions and a fixed threshold (LLaMA2-7B has sink dimensions 1415 and 2533 [2, 3], with τtok≥20\tau_{tok} \geq 20). Our experiments replicate the same dimensions, e.g., {2533} for Vicuna-7B (fine-tuned from LLaMA2-7B), and {458, 2570} for Qwen2.5-7B. Thus, the sink dimensions change with the backbone, while for each backbone the same one or two dimensions dominate across both GLMs, all seven datasets, and both tasks (Fig. 1, Figs. 4-9). We apply the same thresholds to every configuration.

Calibration-free sink identification. To test whether the calibrated dimensions shape the results, we replace Eq. 2 with a score that takes the maximum over all dd dimensions of the backbone:

ϕ~(xjl−1)=max⁡r∈{1,…,d}∣RMSNorm⁡(xjl−1)r∣\widetilde{\phi}(x_j^{l-1}) = \max_{r \in \{1,\ldots,d\}} \left| \operatorname{RMSNorm}(x_j^{l-1})_r \right|.

This equation does not use DsinkD_{\mathrm{sink}} or τdim\tau_{\mathrm{dim}}. We run two tests on the original 300 calibration examples per config, with the same checkpoints and prompts used in the paper:

  • Dimension overlap: For the top-ranked graph token in each example and layer, we check whether the dimension that attains ϕ~\widetilde{\phi} belongs to DsinkD_{\mathrm{sink}}.

  • Sink token overlap: We rank graph tokens by ϕ~\widetilde{\phi} and keep the top kk in each example and layer, where kk is the number of sinks that Eq. 1 identifies, so that both sets have the same size. We report the overlap percentage of calibrated sinks that the top-kk set contains below.

LLaGA NCLLaGA LPTEA-GLM NCTEA-GLM LP
Sink dimension overlap (%)100.0100.0100.0100.0
$\text{Top-}k$ sink overlap (%)99.9599.9299.98100.00

Results show that in every case, the largest activation of the top-ranked graph token is in the calibrated sink dimension, and the calibration-free ranking replicates an average of at least 99.92% of the identified graph sinks using the previous calibrated rules in each setting. Thus, the calibration selects the same dimensions and the same sink tokens using uncalibrated maximum rules.

[Q1] Could you evaluate a variable-length LLaGA representation without [PAD] tokens to determine whether sink behavior is caused by fixed-length padding?

We thank the reviewer for this thoughtful question. We address your question below.

We evaluate a variable length LLaGA representation using Qwen2.5-7b where we explicitly exclude the [PAD] tokens from each sample input. We conduct the experiments on the three citation datasets (Arxiv, Cora, and Pubmed) on the NC task under the exact same settings as stated in the paper and report their accuracies (%). We conduct the same graph sink token identification and sink token pruning experiments as in the paper and report the results below.

ArxivCoraPubMed
#Mean Graph Token74.027.126.8
DsinkD_{sink}257025702570
#Mean Sink Tokens0.160.180.02
Baseline%63.6781.3392.33
Prune sinks%63.3383.0092.00

The table shows that removing the [PAD] token significantly reduces the percentage of sink tokens of each example. For the identified sink tokens, the sink dimension stays the same as stated in the paper. Pruning experiments show that pruning sink tokens do not significantly degrade the baseline performance, which is consistent with our findings presented in the paper. Additionally, most of the remaining sinks are 2-hop nodes (81.1-91.9%). Without [PAD] tokens, the few remaining graph sink tokens are mainly located on the outer neighbors rather than on the target node, which matches the positional pattern of graph sinks in the paper.

AuthorsOpenReview ↗

Response to Reviewer xyke (Part 2/2)

[W2] Cumulative pruning changes sequence length and attention normalization, which may add effects beyond removing the selected tokens’ information.

We thank the reviewer for raising this concern. We address it below.

Pruning a graph token only affects the attention perspective of remaining tokens because RMSNorm, QKV projections, and MLP layers act independently on each token, while original position IDs are retained. Pruning a graph token before block ll is equivalent to an attention knockout [4] that prevents subsequent tokens from attending to it at block ll and beyond. Although sequence shortening renormalizes softmax weights over remaining keys by 1/(1−aij)1/(1-a_{ij}), experiments across 60 configurations using five backbones (Vicuna-7B, Qwen2.5-7B, Llama-3.1-8B, Yi-1.5-9B, Qwen2.5-14B) on LLaGA NC (3 citation, 4 e-commerce datasets), LLaGA LP (3 citation datasets), and TEA-GLM NC (Arxiv, Computer) with 100 evaluation samples over 5 random seeds confirm that attention normalization does not introduce extra confounding effects beyond intended information removal.

Key masking compared with pruning. To determine whether shorter sequences change outputs, key masking experiments maintaining original sequence lengths were conducted by recording pruned token positions for sinks and random non-sinks, constructing a mask to set attention scores of pruned tokens to -inf before softmax at layer ll or earlier, and running forward passes on 50 examples using Vicuna-7b and Qwen2.5-7b on Arxiv NC to compare the resulting highest logit tokens and predicted labels against token pruning.

Backbone (dtype)Token PrunedMean logit difference%Highest logit token%Label
Vicuna-7bsinks0.020100%100%
Vicuna-7bnon-sinks0.018100%98%
Qwen2.5-7bsinks0.74100%96%
Qwen2.5-7bnon-sinks0.79100%96%

Results show that key masking reproduces the highest logit token of pruning in every case for both models, where every example gives the highest logit. The predicted labels also agrees in 96-100%, and the one or two examples that change per run are within numerical noise.

Value-zeroing compared with pruning. To evaluate normalization effects while maintaining sequence length and attention normalization, we perform value-zeroing experiments that set value vectors of pruned sink and non-sink tokens to zero from layer ll onward for all attention heads across 100 evaluation samples using five backbones (Vicuna-7b, Qwen2.5-7b, Llama-3.1-8b, Yi-1.5-9b, Qwen2.5-14b) on LLaGA NC (citation and e-commerce), LLaGA LP (citation), and TEA-GLM NC (Arxiv, Computer).

ConfigsBaselinesinksnon-sinksValue-zero sinksValue-zero non-sinks
LLaGA NC citation85.585.5(+0.0)65.4(−20.1)83.5(−2.0)59.1(−26.4)
LLaGA NC e-commerce85.384.2(−1.1)73.1(−12.2)81.7(−3.6)68.8(−16.5)
LLaGA LP citation80.779.8(−0.9)76.4(−4.3)77.4(−3.3)71.7(−9.0)
TEA-GLM NC Arxiv52.855.2(+2.4)54.1(+1.3)55.8 (+3.0)54.3(+1.5)
TEA-GLM NC Computer43.242.4(−0.8)43.8(+0.6)42.4(−0.8)43.6(+0.4)

Results show that value zeroing also reproduces the pattern of pruning. In both LLaGA NC and LP, removing sinks changes accuracy slightly, while removing the same number of non-sinks tokens drops the accuracy significantly. TEA-GLM changes little under both datasets, as with pruning. This supports that renormalization does not add additional effects than removing the token’s information. The difference between sink and non-sink tokens therefore holds when the sequence length and the attention normalization stay as in the unmodified model.

[1] Sun et al. Massive activations in large language models. Arxiv, 2024.

[2] Kang et al. See what you are told: Visual attention sink in large multimodal models. ICLR, 2025.

[3] Luo et al. To sink or not to sink: Visual information pathways in large vision-language models. ICLR, 2026.

[4] Kaduri et al. What’s in the Image? A Deep-Dive into the Vision of Vision Language Models. CVPR, 2025.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary: To all Reviewers

We sincerely appreciate the thoughtful feedback from Reviewers xyke, vRry, and ZvBW, as well as their valuable suggestions for improving the paper. We are encouraged that the reviewers found the paper to address an important and underexplored question (xyke), ask a worthwhile question about how GLMs use graph tokens for predictions (vRry), and provide a useful cautionary note for the GLM community (ZvBW). Reviewers also noted that the paper is clear and easy to follow (xyke, vRry, ZvBW), provides an extensive evaluation across architectures, backbones, datasets, and tasks (xyke), carefully separates activation magnitude, readable label information, and predictive contribution (xyke, vRry), careful about the scope of our findings (vRry), and includes solid empirical scope and matched intervention controls (ZvBW).

We would like to provide an overall summary of our rebuttal. We have updated the manuscript and marked the corresponding additions in blue.

  • Sink identification and robustness checks (App.B.3): We check whether the results change when we vary the token threshold or remove the fixed sink-dimension set. The positional patterns remain stable around the threshold used in the paper, and using all hidden dimensions selects almost the same sink tokens as our original rule. This suggests that the main results are not an artifact of calibrated dimensions or one fixed threshold choice.

  • Origin of graph sink tokens (App.A.4): We add experiments that examine padding removal, variable-length LLaGA inputs, input-vector norms, random projectors, padding-free LLaGA-HO, and LLM components. These results suggest that graph sinks arise from an interaction between graph-token input scale and activation behavior already present in the LLM, rather than from padding alone.

  • Pruning interpretation and sequence-length effects (App.A.6): We further analyze which tokens are removed during cumulative pruning intervention, including whether they are padding tokens or real graph-node tokens. We also compare pruning with key masking and value zeroing, and examine whether earlier pruning changes later sink selection.

  • TEA-GLM beyond zero-shot transfer (App.A.7): Because TEA-GLM is originally evaluated as a zero-shot graph learner, we add a supervised target-dataset experiment to test whether our conclusion holds when the graph projector is trained on the evaluated dataset.

  • Activation patching and label information (App.A.9): We clarify which transformer blocks are patched and how the transfer score is aggregated. We also break the patching results down by block depth and show that the sink and non-sink gap is already visible in early blocks. In addition, we add an attention-knockout experiment showing that the label information read from [PAD] sink tokens mainly comes from other graph tokens through attention.

  • Scope, limitations, and implementation details (App.C): We revise the manuscript to state the tested settings and the boundary of our conclusions more clearly. The current study covers two representative GLM designs, seven LLM backbones, seven text-attributed graph datasets, and two graph tasks, while leaving more specialized GLM architectures and graph-token layouts for future work.

We thank the reviewers again for their constructive feedback and suggestions. We would be happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.