Understanding Context Utilization via Neuron Sinks

Transformer language models apply the same computational blocks at every position, although causal masking limits the context available near sequence beginnings. We identify neuron sink, a phenomenon across diverse language models in which a sparse subset of MLP neurons exhibits disproportionately frequent strong activations near sequence beginnings. This pattern reappears at the start of a passage following an irrelevant prefix, suggesting that neuron sink depends on the availability of useful context rather than absolute position or context length. Ablating sink neurons selectively impairs prediction near sequence beginnings, demonstrating their functional importance for low-context processing. Beyond this role, we find evidence that neuron sink contributes to behavioral position bias. In a controlled model, transplanting the sequence-initial sink-neuron state to middle positions reduces first-over-middle retrieval bias, causally linking this activation bias to the primacy component of the lost-in-the-middle phenomenon. These findings uncover a mechanism for low-context processing and identify a potential target for mitigating position bias in language models.