What Makes Position Zero Special? A Mechanistic Study of Position Zero Attention Sinks in LLMs

Transformers frequently allocate disproportionate attention to specific tokens, a phenomenon known as **attention sinks**. Causal large language models reliably form one at **position zero**, though its role remains debated. We approach this question from a mechanistic perspective, tracing how the position-zero sink arises from the model's internal computation. We identify a two-block subnetwork responsible for this behavior, which we term the **P0-Sink Circuit**, and show it arises purely from the structural properties of causal attention, requiring no semantic content. We further validate through from-scratch pre-training experiments that two proposed parameter-free methods effectively accelerate P0 sink formation, and find that earlier sink formation benefits pre-training and improves downstream performance. Both methods outperform the Transformer baseline and achieve performance comparable to Gated Attention across comprehensive settings. Code is available now at https://anonymous.4open.science/r/flash-linear-attention-6220/.