acceptodds
Withdrawn from ICLR 2027

Reverse Engineering Attention Sinks in LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In autoregressive large language models (LLMs), the beginning-of-sequence (BOS) token tends to receive disproportionate amounts of attention. While past research has theorized about why these attention sinks form, we instead focus on characterizing what attention sinks are and how they're implemented. We posit that attention sinks are used to implement linear threshold-based gates to filter attention logits, and from this, a Heisenberg uncertainty principle follows: an attention head can either observe an attention sink, or observe its semantic meaning and position, but not both simultaneously. Inspired by this principle, we find that LLMs trained to accept sequences without a BOS token, which have been shown to form an attention sink at their first token, use Layer 0 to remove semantic and positional information from the first token, turning it into a replacement BOS so that it may function as the attention sink. We show how malicious prompts can fool these LLMs into acting as if each of multiple tokens are the unique attention sink. Finally, motivated by our findings, we propose minimal architectural modifications to alleviate these issues, resulting in LLMs that are comparable-or-better in performance, no longer form first-token sinks, and are more robust to repeated-token attacks.

Primary area:
interpretability and explainable AI
Keywords:
attention sink, LLM, interpretability, massive activations

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_HN9rOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies attention sinks in pretrained language models, with the goal of mechanistically describing their functions. The authors collect empirical evidence across several <=10B open-source LLMs for several claims: (1) "attention sinks implement thresholds used to filter out other attention logits", (2) "the sink token’s key is stashed in low-frequency RoPE subspaces, such that it doesn’t experience significant rotation in different in-distribution context lengths", and (3) a description of how attention sinks manifest in Qwen3-8B (an open LLM with no BOS token). Based on intuitions especially (1), the authors propose architectural interventions to combat attention sinks and compare the modified architectures with baselines on LLM pretraining tasks.

Critical strengths
  • The mechanistic/observable analysis is relatively thorough. Not much past work has gone into this much mechanistic detail about attention sink formation over so many models, so this could be an interesting contribution to the literature.
  • The writing is clear and the paper is easy to read.
Critical weaknesses
  • The examples don't seem standardized over models; it's not clear whether a lot of the figures or the mechanistic understanding thereof are cherrypicked to fit a model to the desired intuition. So it's not immediately clear what generalizable lessons to take from this (also see below).
  • The pretraining experiments are not very convincing. If I take the plot in figure 8 (and taking "millions of nats" to really mean "millionths of nats", otherwise your pretraining loss would be approximately -600,000), then the effect size is really small and within noise. Gated attention also helps the model perform better (with, I think, a much bigger effect size) while simultaneously removing attention sinks, and I wonder if it's possible to isolate the impact of attention sinks vs the extra expressivity of gated attention, as I think the performance increase is mostly from the second factor. Since the authors explicitly remark on gated attention, I would appreciate comparisons/ablations to understand what's going on here.
  • I don't really understand the motivation behind defending against repeated token attacks.
  • The math presented in the paper is not complicated. While that's great overall as it increases the accessibility of the work, I would say it's not very nontrivial/informative.
Questions for authors
  • Do all the findings generalize across models? Which do or don't, and in what sense?
  • How do the proposed threshold interventions compete against gated attention?
  • Why would repeated token attacks be serious or substantial to defend against? (And how do e.g. gated attention architectures deal with those?)
Reviewer_uwCoOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper regards attention sinks as query-dependent thresholds that gate attention outputs. Through a Qwen3-8B case study, it describes how large MLP activations create a nearly token-independent first-token representation and how unusual or repeated prefixes disrupt this mechanism. Motivated by these observations, it introduces QCT-Softmax, an explicit query-conditioned sink logit with zero value contribution. Experiments show comparable language-modeling performance, substantially reduced first-token attention concentration, and lower prediction divergence under repeated prefixes.

Critical strengths
  1. The paper provides some useful evidence about how attention sinks form. It analyzes RoPE frequencies and identifies several neurons related to first-token dominance. These observations are interesting and help explain where the sink behavior may come from.

  2. The proposed modification is simple and work in the reported experiments. QCT-Softmax reduces attention concentration on the first token and improves the repeated-prefix behavior, while keeping language-modeling loss roughly unchanged.

Critical weaknesses
  1. The paper argues that low-frequency features become position-insensitive and then overwrite semantic information, but the current evidence does not fully support this causal story. For example, cosine similarity only shows that one direction dominates the representation, not that semantic information is actually erased. It is suggested that more ablation or analysis experiments could be added to better analyse the problem.

  2. Previous work has already studied key-only sink tokens, gating interpretations, and sink-related failure circuits, such as [1][2]. QCT-Softmax is a reasonable variation, but it is suggested to compare against the closest existing baselines. The evaluation is also limited to one relatively small training setting, and broader downstream tests is recommended. It is suggested therefore clarify what new mechanism is established beyond these closely related explanations and include stronger comparisons against these baselines.

[1] Itay Yona, et al. "Interpreting the Repeated Token Phenomenon in Large Language Models." Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:72535-72555, 2025.

[2] Fu, Zizhuo, et al. "Attention sink forges native moe in attention layers: Sink-aware training to address head collapse." arXiv preprint arXiv:2602.01203 (2026).

Questions for authors
  1. The paper argues that low-frequency RoPE components become position-insensitive and eventually lead to semantic overwrite. Could the authors provide more ablation or causal evidence for this mechanism?

  2. How does the proposed mechanism differ from prior explanations of attention sinks? It would be helpful to clarify which part of the proposed explanation is genuinely new relative to these works. Could the authors compare QCT-Softmax with closer baselines that explicitly address attention sinks or introduce gating/key-only mechanisms?

  3. The current evaluation appears limited to a relatively small training setting. Do the authors have evidence that the method generalizes to larger models, or more settings, or downstream tasks?

Reviewer_7E5bOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes that attention sink tokens may act as thresholds which other attention logits must exceed, and argues that the attention sink token's keys must live in the lower frequency pairs in models using RoPE, so that it is stable across positions. This implies that a head cannot both use a token as a sink and read its content at the same time. Analysis of Qwen3-8B, which is a model that doesn't use a BOS token, reveals that layer-6 neurons overwrite the first token to create a temporary BOS, which the last layer undoes. The paper proposes QCT-softmax, which uses a learned, query-conditioned threshold logit (learned key plus bias) attached to each head's softmax in the model. In a 59M non embedding parameter QCT-softmax model trained on 20B tokens, the paper reports comparable or lower loss, removal of first-token sinks, and better robustness to repeated first token attacks.

Critical strengths
  • Sinks as thresholds is an intuitive framing.
  • The QCT-softmax proposal follows directly from the analysis.
Critical weaknesses
  • Appx A.3 covers the stashing claim broadly, but the threshold claim relies on a heuristic derivation and on Fig. 2 which reports only 4 heads. The Qwen3-8B mechanism is a single model case study, and there's no intervention on the identified neurons, for instance ablating the three layer 6 neurons to check whether the sink disappears. In addition, for most analyses, more extensive validation is cheap (single forward passes), so I'd like to see a lot more data.
  • QCT-softmax is very weakly evaluated with a small model on 20B tokens without any hyperparameter tuning (although using matched initialization and data order). The performance is reported only as a change in total nats, and there's no absolute perplexity data for the trained models, per-token loss, or downstream tasks. Moreover, only a single seed is used, and at this scale pre-training noise is likely to be a sizable factor in performance differences.
  • It is unclear why QCT-softmax is better than simply training the model with a BOS token. The gpt-oss style scalar baseline is included, but there's no BOS baseline, and no comparison with the closest existing method (Gu et al., which this paper itself describes as the most similar). And why this method is advantageous over BOS with regard to robustness to repetition or the need for stashing is not detailed or tested.
  • This doesn't significantly affect my score, but the paper uses an unconventional structure and is very hard to follow. There's no dedicated introduction or related works section, and every section has its own "conclusion" subsection, which is confusing in my opinion. Other noticeable issues include the abstract saying "use Layer 0 to remove semantic and positional information" whereas other sections say layer 6, and section 4.3 saying "hypothesis 4.1" three times for different things.
Questions for authors
  • Could the authors provide broader empirical evidence for the threshold interpretation?
  • Does removing the three layer-6 neurons remove the sink, and does this appear in other models that don't use a BOS token?
  • How does QCT-softmax empirically compare to just using a BOS token, and to Gu et al's learned key?
  • Can the authors perform at least a basic learning rate sweep on the models that compare methods, pre-train them with multiple seeds, and report perplexity in a table along with Holm-corrected p-values?
  • Does the 59M baseline model form its first token sink via the same overwrite mechanism as Qwen and does QCT-softmax remove that mechanism?
  • Do the results hold at the full 2048 context length (Fig. 9 only measures at length 64, 128, and 256, even though training uses 2048), and at larger scale?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.