acceptodds
Under review as a conference paper at ICLR 2027

What Makes Position Zero Special? A Mechanistic Study of Position Zero Attention Sinks in LLMs

Abstract

Transformers frequently allocate disproportionate attention to specific tokens, a phenomenon known as **attention sinks**. Causal large language models reliably form one at **position zero**, though its role remains debated. We approach this question from a mechanistic perspective, tracing how the position-zero sink arises from the model's internal computation. We identify a two-block subnetwork responsible for this behavior, which we term the **P0-Sink Circuit**, and show it arises purely from the structural properties of causal attention, requiring no semantic content. We further validate through from-scratch pre-training experiments that two proposed parameter-free methods effectively accelerate P0 sink formation, and find that earlier sink formation benefits pre-training and improves downstream performance. Both methods outperform the Transformer baseline and achieve performance comparable to Gated Attention across comprehensive settings. Code is available now at https://anonymous.4open.science/r/flash-linear-attention-6220/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Mechanistic Interpretability, Attention Sink

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZzQWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the mechanism behind position-zero (P0) attention sinks in causal language models. The authors propose the P0-Sink Circuit, a two-block mechanism in which the asymmetry of causal attention produces a distinctive representation at position zero, which is subsequently amplified by MLP layers into a stable, high-norm representation. Based on this analysis, the paper introduces two parameter-free training methods: Transformer-Native LM (TNLM) Loss, which removes the prediction loss at the BOS position, and Force-sink Masking, which partitions hidden dimensions between position zero and other tokens.

The authors study the formation of P0 sinks in pretrained LLMs and evaluate the proposed methods through from-scratch pretraining experiments under different learning rates and data budgets. The results suggest that both methods can improve downstream performance over a standard Transformer baseline under certain settings, with Force-sink Masking achieving results comparable to Gated Attention.

Critical strengths

1. Simple and practical training modifications.

I appreciate the simplicity of the two proposed methods, particularly Force-sink Masking. Both are straightforward to implement and introduce no additional trainable parameters. The improvements over the standard Transformer baseline under the 20B-token setting are encouraging, and the method achieves competitive performance against Gated Attention in several configurations. This makes the practical side of the paper worth considering, despite my concerns about its novelty.

Critical weaknesses

1. Limited theoretical novelty and an insufficient connection between the mechanistic analysis and the proposed methods.

I am not fully convinced by the novelty of the proposed P0-Sink Circuit. The central observation that P0 sinks can emerge from the structural properties of causal attention, rather than the semantic content of the first token, has already been investigated in prior work. It is therefore unclear what fundamentally new mechanism is established here beyond these existing results.

Moreover, although the paper presents a two-block circuit, much of the evidence is based on activation statistics and representation visualizations. These observations are interesting, but they do not fully establish the causal necessity of the proposed circuit. I also found the connection between the analysis in Section 3 and the two methods in Section 4 somewhat loose. The methods appear to be reasonable design heuristics, but it is not sufficiently demonstrated that they follow from, or specifically validate, the proposed mechanistic explanation. This weakens the paper's main conceptual contribution.

2. The practical benefits and the claimed "free lunch" are not sufficiently established.

While the proposed methods are appealingly simple, I think their practical advantages need more careful evaluation. Force-sink Masking introduces no additional parameters, but it is not entirely free: at a fixed hidden size, it permanently reserves part of the representation space for P0, leaving fewer dimensions available to other tokens. Alternatively, preserving the original representational capacity by adding dedicated sink dimensions would introduce additional computational and parameter costs. This trade-off may become important in larger models or long-context settings, which are not adequately evaluated.

I am also concerned about how well the improvements persist with more training. In Appendix E, the gains become much less convincing under the 100B-token setting, and several proposed variants perform slightly worse than the baseline at the final checkpoint. This makes it difficult to conclude that the methods provide a consistent benefit beyond relatively undertrained settings. Finally, the paper lacks a direct comparison with recent explicit attention-sink mechanisms, such as the learnable sink logits used in gpt-oss. Such alternatives are particularly relevant because they provide a content-free attention destination without requiring a real token to serve as the sink.

Questions for authors

1. What is the main mechanistic novelty relative to existing work?

Could the authors clarify how the P0-Sink Circuit differs from the explanation in Li et al. (2026), particularly regarding causal masking, value aggregation, and FFN amplification? It would also be helpful to provide targeted interventions on the proposed circuit, rather than relying mainly on representation statistics, to establish which components are actually necessary for P0 sink formation.

2. How does the method compare with explicit learnable attention sinks?

Recent architectures such as gpt-oss introduce learnable sink logits directly into the attention softmax, allowing attention mass to be assigned to a content-free destination. This seems to offer a natural alternative to forcing a real token to become an attention sink. Could the authors compare against this design under comparable training budgets? It would also be interesting to discuss its relationship with TNLM Loss: an explicit sink has no token-level prediction target, although it still receives gradients through downstream predictions. How do the two approaches differ in practice?

3. What is the actual capacity and efficiency trade-off of Force-sink Masking?

Since the method reserves kk hidden dimensions exclusively for P0, ordinary tokens effectively operate in a smaller representation space. How sensitive is performance to the choice of kk, especially for larger models and longer contexts? A more systematic study of kk, together with long-context evaluations and measured training/inference costs, would help establish whether the method is genuinely a free lunch.

4. Is earlier or stronger P0 sink formation actually responsible for the performance improvements?

The paper suggests that earlier sink formation benefits pretraining, but the experimental results do not seem to support a consistent relationship. For example, TNLM Loss sometimes improves downstream performance despite reducing the measured Sink Rate. The 100B-token results also show cases where a stronger sink does not translate into better performance. Could the authors provide more direct evidence for the proposed causal relationship, rather than relying mainly on correlations across training methods?

Reviewer_4XQzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigate the phenomenon of attention sink on (causal) LLMs at position 0 (the disproportional attention amount given to the [BOS] token). The authors identify and propose an hypothesis\mechanism that makes the first token attract such large amount of attention which they named "P0-sink circuit" : (1) causal attention lets it attend only to itself (2) using two transformer block, they produce a representation at position 0 that is largely independent of the token's semantics (3) later attention heads target this representation and position 0 become an attention sink.

They validate their hypothesis by proposing using two pre-training methods that promote P0 sink formation:

  1. Transformer-Native LM (TNLM) Loss - remove next token prediction of BOS
  2. Force sink masking - reversing the residual stream dimensions of P0
Critical strengths
  1. The proposed methods are simple, parameter free, easy to follow and implement. The Force-sink masking especially shows promising results in table 4 (~2% improvement without any additional parameters comparing to GA)

  2. The topic of attention sink is still very relevant and have practical implementations. better understanding of this phenomenon can increase interpretability and stability of modern LLMs.

  3. The authors conducted extensive experimentations that was able to isolate the positional effect from token identity (in LLaMa models).

Critical weaknesses
  1. i am not clear on the theoretical analysis. Equation 10 rely on implicit assumptions (about attention weights/value vectors and sparsity) that do not fully explain why P0 representations are uniquely stable.

  2. The claim that earlier sink formation causes downstream improvements is not supported by training data. The results in Table 4 occasionally show inverse relationships between reported sink rates and accuracy.

  3. in general there is an overstate robustness, while the experiment relay on a single seed the 100B tokens expriments show mixed results where the baseline outperforms the proposed methods. is it possible that the benefit from those methos is limited to the smaller 20B-token?

Questions for authors

aside from questions raised in the weaknesses, did the authors preform any ablation of the early layers signal?

Reviewer_JQeqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a mechanistic account of P0 attention sinks: causal masking creates a distinctive signal that early MLPs amplify into a stable representation. It introduces two parameter-free training modifications, TNLM Loss and Force-sink Masking. Experiments show downstream improvements comparable to Gated Attention at 20B training tokens, while results at 100B tokens are mixed.

Critical strengths
  1. Layer-wise BOS ablations show that LLaMA recovers P0 sinks beyond the earliest layers without BOS, supporting a structural explanation beyond token identity.

  2. Force-sink Masking improves the six-task average by approximately 1.9–2.3 percentage points across three learning rates at 20B tokens, matching Gated Attention without additional parameters. It also demonstrates that strong sinks can exist without large activation outliers.

Critical weaknesses
  1. Section 3 supports the proposed P0-Sink Circuit through representation statistics, visualizations, and simplified theoretical analysis: attention creates positional differences that MLPs detect and amplify. Section 4 further demonstrates performance gains from two training modifications and suggests that stable P0 representations improve QK optimization. However, the evidence leaves two distinct questions: a) The analysis reveals representation patterns consistent with the proposed mechanism, but what targeted intervention evidence establishes the causal contribution of the described MLP computation to the fixed-direction P0 representation and subsequent sink formation? b) TNLM also changes the prediction objective, while Force-sink Masking changes channel allocation and suppresses activation outliers. What evidence separates the contribution of sink formation from these accompanying changes and directly supports the proposed improvement in QK optimization? The performance comparisons support the effectiveness of the methods, but do not establish whether the gains arise from promoting sink formation.

  2. The improvements at 20B tokens are convincing within the tested settings, but the evidence for benefits at 100B tokens is much weaker. In Table 10, TNLM outperforms the baseline at two checkpoints but underperforms at the final checkpoint; Force-sink+ also alternates between small gains and a small loss. Thus, the claim of consistent improvements in Appendix E needs a clearly specified aggregation criterion and uncertainty assessment. Only one seed is reported, and Appendix B.1 states that k was increased from 16 to 32 after k=16 gave suboptimal results in the longer-training setting. Without a clearer selection procedure or sensitivity analysis, it remains uncertain how robustly the limited-budget gains extend to longer training.

Questions for authors
  1. What evidence separates the effect of P0 sink formation or stability from the effects of changing the objective, allocating separate channels, or suppressing outliers? Do you have controlled comparisons or optimization measurements supporting the proposed connection to QK stability? If not, please clarify which parts of this explanation should be treated as hypotheses.

  2. What evaluation or aggregation criterion underlies “consistently outperform” in Appendix E? Please clarify how k and the reported checkpoints were selected, and whether repeated-run results or uncertainty estimates are available to assess the small differences at 100B tokens.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.