acceptodds
Withdrawn from ICLR 2027

Attention Sinks Beyond the First Position: Heterogeneous Pathways

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Attention sinks, in which a small subset of tokens in a sentence receives disproportionately large attention mass, are commonly associated with the first position. However, tokens at non-initial positions can also become attention sinks, yet the circuit mechanisms underlying these secondary sinks remain poorly understood. In this work, we investigate their formation in the first Transformer block using complementary activation-patching experiments. Across the evaluated models and token types, secondary sinks arise through different contributions from word embeddings and self-attention outputs. When the "bos" token appears at a non-initial position, its sink behavior is driven predominantly by its embedding in some models and by a combination of its embedding and attention output in others. For the other tested secondary sink tokens, their sink behavior relies mainly on the strong self-sinking within a small subset of critical heads in the first-block attention output. Further experiments reveal a corresponding difference in sensitivity to contextual mixing: embedding-dominated sinks persist under contextual mixing, even in the presence of competing secondary sink tokens, whereas attention-output-dominated sinks can disappear when competing tokens occur earlier in the sequence. These findings suggest multiple first-block pathways to secondary sink formation and show how these pathways shape the sink behavior under different contexts. More broadly, our analysis provides a circuit-level framework for understanding and distinguishing attention-sink mechanisms in large language models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
attention sink mechanism, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_tT4wOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method for eliciting attention sinks in pre-LN style transformer models. Its scope is limited to the first layer, token embeddings, and self-attention outputs. Given pre-defined tokens (BOS, different punctuation), patching is used to help identify the cause of secondary sinks (tokens not in the first position), whether due to a token's embedding or self-attention.

The main contribution of the paper is the patching methodology: a way of establishing sufficient and necessary conditions for identifying pathways (token embedding, self-attention) that contribute to secondary sink formation.

Experimental results show the formation of secondary sinks across a number of LLMs, sink attribution (embedding vs. self-attention) in LLMs, identification of self-attention heads that contribute to sinks, and a study of sink token mixing.

Critical strengths

The main strength of the paper is the methodology proposed in eliciting secondary attention sinks. To the best of my understanding, the patching approaches for establishing sufficient and necessary conditions in sink formation are (1) novel, (2) useful in attributing specific transformer modules to sinks, and (3) could help inform other types of sink formation studies in the future.

Critical weaknesses

The limited scope of the paper - a focus on just the first layer, token embeddings, self-attention - leads to an incremental contribution. Such a limited scope makes it less clear whether the findings made in the paper could be achieved with prior work, in place of patching, where attention sinks are studied in more detail. Namely, existing work (all largely cited in the paper) has focused on how positional encodings, query-key relations, and feedforward blocks lead to attention sinks, and not necessarily limited to first-position tokens. From this perspective, it is also unclear whether token embeddings and self-attention are the primary drivers for secondary sinks, or if other parts of the model are more important.

Additionally, there is an unclear significance of the findings made in the paper. This stands in contrast with prior work, where the focus is not just on discovery of attention sinks, but also, (1) emergence of attention sinks during pretraining, (2) importance of attention sinks for minimizing LM loss, (3) role of attention sinks in limiting context mixing, and (4) how transformer design leads to attention sinks, and alternative designs for sink suppression. Without a more detailed investigation into the emergence, or implications, of secondary attention sinks, it is difficult to appreciate their importance.

Questions for authors

In Sec. 3.2, it is claimed that "the first block is crucial for identifying the patterns that initiate secondary sink behavior", with additional arguments raised in A.6. However, attention sinks can form in later layers, and not necessarily as a consequence of representations formed in the first layer, see [Sun et al. 2026] for evidence of massive activations formed beyond the first layer. As a result, the arguments in A.6 are not convincing, and it would strengthen the paper to experimentally validate these sorts of claims.

As the results in Sec. 5 are anecdotal, are there metrics that could be employed to capture sensitivity of mixing multiple sink tokens, analogous to what is done in Sec. 4 (Eq. 6)? This would make the results more convincing.

Reviewer_mmR2OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper empirically studies the mechanisms underlying attention sinks at non-initial token positions, which the authors call secondary sinks. Through controlled experiments, the authors identify two pathways for secondary-sink formation: the X-path, in which sink formation is primarily driven by the sink token's embedding contribution through the residual connection, and the V-path, in which it is primarily driven by the first Transformer block’s attention output. Their experiments examine BOS, period, and newline tokens in Llama2-7B, Llama3-8B, Mistral-7B, and Gemma2-2B.

Critical strengths

None.

Critical weaknesses
  1. Although I am not an expert in the literature on attention sink, the authors cite many studies of primary sinks whose proposed mechanisms may share similarities with those underlying secondary sinks. A thorough comparison with these earlier works is needed to help readers understand the novelty of the present paper. It is also unclear to me what additional significance studying secondary sink offers, given that the mechanisms underlying primary sinks have already been studied extensively. Also, the models evaluated in this paper are all old models.

  2. Although the paper is generally readable, its presentation makes the experiments and findings unnecessarily difficult to follow. For example, the distinctions among the different SS quantities are not explained clearly enough for readers to distinguish. While I appreciate the authors' effort in experiments, the discussion of the results lacks depth and offers limited insight into the findings.

Questions for authors

Could the authors clarify how their findings differ from existing results on primary attention sinks and explain the significance of secondary attention sinks? I would also appreciate a discussion of attention sink in the latest models.

Reviewer_N6bCOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The main contribution of the paper is an empirical analysis of the emergence of secondary sink tokens in large language models. Specifically, the authors introduced two pathways (X-path and V-path) when these sinks may emerge, and designed causal tests to demonstrate the mechanism and robustness for each of them.

Critical strengths
  • Secondary attention sinks are under-explored in the literature so the analysis in the paper is welcome.
  • The paper is generally well-written and the ideas and experiments are easy to follow.
  • Causal tests of mechanisms and robustness are a certain plus.
Critical weaknesses
  • The main motivation of studying secondary sinks should be emphasised more. Why are they important, and what functional roles do they plan? Are these important for specific downstream tasks?
  • Related to the above, it is somewhat unclear what the implications of the empirical findings are: can they inform interventions that affect model behaviour in certain ways?
  • The entire sink analysis depends on the single factor of sink rate with a predefined threshold 0.3, which is rather arbitrarily chosen. The robustness and generalisation of the results are therefore not clear. The authors should test alternative measures of sink behaviour if possible.
  • Similarly, the identification of a sink token following the X- and V-path also depends on ad-hoc threshold parameters. The authors should test the robustness of the main findings against these choices.
Questions for authors
  • Some recent work (such as [3] below) also discussed attention sinks beyond first token, can the authors position their work against these?
  • Can the authors test other alternative measures of sink behaviour such as sink scores [1,2]?
  • Can the authors test alternative thresholds for identifying the X- and V-path? In particular, would these change the “Neither" cases?
  • (minor) There are compilation errors in A.5, please fix those.

References

[1] Gu et al., “When Attention Sink Emerges in Language Models: An Empirical View,” ICLR, 2025.

[2] Queipo-de-Llano et al., “Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coin,” ICLR, 2026.

[3] Zhang et al., “Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings,” NeurIPS, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.