- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies attention sinks in pretrained language models, with the goal of mechanistically describing their functions. The authors collect empirical evidence across several <=10B open-source LLMs for several claims: (1) "attention sinks implement thresholds used to filter out other attention logits", (2) "the sink token’s key is stashed in low-frequency RoPE subspaces, such that it doesn’t experience significant rotation in different in-distribution context lengths", and (3) a description of how attention sinks manifest in Qwen3-8B (an open LLM with no BOS token). Based on intuitions especially (1), the authors propose architectural interventions to combat attention sinks and compare the modified architectures with baselines on LLM pretraining tasks.
Critical strengths
- The mechanistic/observable analysis is relatively thorough. Not much past work has gone into this much mechanistic detail about attention sink formation over so many models, so this could be an interesting contribution to the literature.
- The writing is clear and the paper is easy to read.
Critical weaknesses
- The examples don't seem standardized over models; it's not clear whether a lot of the figures or the mechanistic understanding thereof are cherrypicked to fit a model to the desired intuition. So it's not immediately clear what generalizable lessons to take from this (also see below).
- The pretraining experiments are not very convincing. If I take the plot in figure 8 (and taking "millions of nats" to really mean "millionths of nats", otherwise your pretraining loss would be approximately -600,000), then the effect size is really small and within noise. Gated attention also helps the model perform better (with, I think, a much bigger effect size) while simultaneously removing attention sinks, and I wonder if it's possible to isolate the impact of attention sinks vs the extra expressivity of gated attention, as I think the performance increase is mostly from the second factor. Since the authors explicitly remark on gated attention, I would appreciate comparisons/ablations to understand what's going on here.
- I don't really understand the motivation behind defending against repeated token attacks.
- The math presented in the paper is not complicated. While that's great overall as it increases the accessibility of the work, I would say it's not very nontrivial/informative.
Questions for authors
- Do all the findings generalize across models? Which do or don't, and in what sense?
- How do the proposed threshold interventions compete against gated attention?
- Why would repeated token attacks be serious or substantial to defend against? (And how do e.g. gated attention architectures deal with those?)