- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the mechanism behind position-zero (P0) attention sinks in causal language models. The authors propose the P0-Sink Circuit, a two-block mechanism in which the asymmetry of causal attention produces a distinctive representation at position zero, which is subsequently amplified by MLP layers into a stable, high-norm representation. Based on this analysis, the paper introduces two parameter-free training methods: Transformer-Native LM (TNLM) Loss, which removes the prediction loss at the BOS position, and Force-sink Masking, which partitions hidden dimensions between position zero and other tokens.
The authors study the formation of P0 sinks in pretrained LLMs and evaluate the proposed methods through from-scratch pretraining experiments under different learning rates and data budgets. The results suggest that both methods can improve downstream performance over a standard Transformer baseline under certain settings, with Force-sink Masking achieving results comparable to Gated Attention.
Critical strengths
1. Simple and practical training modifications.
I appreciate the simplicity of the two proposed methods, particularly Force-sink Masking. Both are straightforward to implement and introduce no additional trainable parameters. The improvements over the standard Transformer baseline under the 20B-token setting are encouraging, and the method achieves competitive performance against Gated Attention in several configurations. This makes the practical side of the paper worth considering, despite my concerns about its novelty.
Critical weaknesses
1. Limited theoretical novelty and an insufficient connection between the mechanistic analysis and the proposed methods.
I am not fully convinced by the novelty of the proposed P0-Sink Circuit. The central observation that P0 sinks can emerge from the structural properties of causal attention, rather than the semantic content of the first token, has already been investigated in prior work. It is therefore unclear what fundamentally new mechanism is established here beyond these existing results.
Moreover, although the paper presents a two-block circuit, much of the evidence is based on activation statistics and representation visualizations. These observations are interesting, but they do not fully establish the causal necessity of the proposed circuit. I also found the connection between the analysis in Section 3 and the two methods in Section 4 somewhat loose. The methods appear to be reasonable design heuristics, but it is not sufficiently demonstrated that they follow from, or specifically validate, the proposed mechanistic explanation. This weakens the paper's main conceptual contribution.
2. The practical benefits and the claimed "free lunch" are not sufficiently established.
While the proposed methods are appealingly simple, I think their practical advantages need more careful evaluation. Force-sink Masking introduces no additional parameters, but it is not entirely free: at a fixed hidden size, it permanently reserves part of the representation space for P0, leaving fewer dimensions available to other tokens. Alternatively, preserving the original representational capacity by adding dedicated sink dimensions would introduce additional computational and parameter costs. This trade-off may become important in larger models or long-context settings, which are not adequately evaluated.
I am also concerned about how well the improvements persist with more training. In Appendix E, the gains become much less convincing under the 100B-token setting, and several proposed variants perform slightly worse than the baseline at the final checkpoint. This makes it difficult to conclude that the methods provide a consistent benefit beyond relatively undertrained settings. Finally, the paper lacks a direct comparison with recent explicit attention-sink mechanisms, such as the learnable sink logits used in gpt-oss. Such alternatives are particularly relevant because they provide a content-free attention destination without requiring a real token to serve as the sink.
Questions for authors
1. What is the main mechanistic novelty relative to existing work?
Could the authors clarify how the P0-Sink Circuit differs from the explanation in Li et al. (2026), particularly regarding causal masking, value aggregation, and FFN amplification? It would also be helpful to provide targeted interventions on the proposed circuit, rather than relying mainly on representation statistics, to establish which components are actually necessary for P0 sink formation.
2. How does the method compare with explicit learnable attention sinks?
Recent architectures such as gpt-oss introduce learnable sink logits directly into the attention softmax, allowing attention mass to be assigned to a content-free destination. This seems to offer a natural alternative to forcing a real token to become an attention sink. Could the authors compare against this design under comparable training budgets? It would also be interesting to discuss its relationship with TNLM Loss: an explicit sink has no token-level prediction target, although it still receives gradients through downstream predictions. How do the two approaches differ in practice?
3. What is the actual capacity and efficiency trade-off of Force-sink Masking?
Since the method reserves hidden dimensions exclusively for P0, ordinary tokens effectively operate in a smaller representation space. How sensitive is performance to the choice of , especially for larger models and longer contexts? A more systematic study of , together with long-context evaluations and measured training/inference costs, would help establish whether the method is genuinely a free lunch.
4. Is earlier or stronger P0 sink formation actually responsible for the performance improvements?
The paper suggests that earlier sink formation benefits pretraining, but the experimental results do not seem to support a consistent relationship. For example, TNLM Loss sometimes improves downstream performance despite reducing the measured Sink Rate. The 100B-token results also show cases where a stronger sink does not translate into better performance. Could the authors provide more direct evidence for the proposed causal relationship, rather than relying mainly on correlations across training methods?