I think the paper is interesting but I don't think it adds anything new to the discussion. The experiments show that the proposed behaviour happens numerically in the setting the authors chosen, I think tough that said results mostly reproduce the assumptions used to derive the theory. In fact, they don't really establish that the mechanism is robust, and they also don't provide a proper explanation of the already known causal-mask bias. Plus, I don't really get the structure of the paper: the authors went heavy on the theory in the main paper, but left the results that are actually meaningful in the appendix (for example I'd put the mask-removal experiment in the main paper).
More specifically, in the paper we find the claim that “causal masking causes parameters associated with the first token to accumulate disproportionate gradients” but the paper studies a very specifc situation (the first block is fixed and computes uniform prefix averages, tokens are orthogonal and unique across sequences, initialisation is specially chosen, plus only the second block is trained). So, I think that in this case the results show that causal masking can produce a sink in this construction, but they don't establish that the same gradient mechanism explains sink formation in ordinarily trained AR transformers. (Interesting the experiment in the appendix where the authors remove the attention mask, because it shows that removing the mask eliminates the first token’s advantage in the constructed setting, but under these very specific conditions, we can't really extend the results to LLMs).
An other problem is that the paper argues that frequent tokens become attention sinks because they accumulate larger gradients. But the fact that these tokens appear more often doesn't strictly mean that they produce a stronger learning signal, since the updates from examples with different targets can partly cancel too. In the appendix the paper tries to address this issue, but the argument relies on an early training setting with many possible outputs and not enough target classes.
The frequency experiment provides additional support, but it also changes the number of sequences and their structure, and that makes the frequency contribution quite difficult to isolate. So, the results support a frequency-related mechanism in this particular setting, but they don't show that frequency is the general explanation for attention sinks.
Finally, the paper argues that attention sinks are compatible with accurate prediction. But I'd say that in this specific setting these results only show that concentrated attention doesn't prevent the correct classification, unfortunately this doesn't establish that it preserves performance on tasks that require a more general contextual understanding or generalisation.
The experiment in the appendix reports attention weights without prediction accuracy or perplexity, so it doesn't really add that much.
I think this paper would is a good starting point, but the authors need to run experiments on real LLMs, and I understand that resources can be limited, but there are few models with checkpoints, like OLMo 1b and 7b, the Pythia suite, SmoLM, Amber. At this point a toy experiment can be meaningful only if the paper also contains complementary results on proper LLMs that point in the same direction.
About the causal mask bias (the authors cite this paper):
Wu et al., “On the Emergence of Position Bias in Transformers” (2025)
And also this other paper that is quite recent:
Peng et al., “What Makes Position Zero Special? A Mechanistic Study of Position Zero Attention Sinks in LLMs” (2026)
Plus, in Ruscio et al., "What are you sinking? A geometric approach on attention sink" (2025) the authors do study a bit the evolution of attention sinks using random matrix theory on Pythia checkpoints, and they found that a strong inductive bias for model with regular RoPE is the rotational matrix of the first token, that is an identity matrix.
Thank you for your constructive review and for recognizing the well-posed research question and clear formulation. We have expanded the quantitative evidence and will make the experimental results and proof structure more accessible in the main paper. We address your three concerns below.
We have extended the empirical analysis beyond the original illustrative curves. In particular, we evaluated Pythia-160M checkpoints from three independent pretraining seeds on WikiText-2, measuring attention across all layers and heads together with language-modeling performance.
The mean and standard deviation are computed across three independent pretraining seeds. Each checkpoint is evaluated on 128 blocks from 32 articles. First position refers to the beginning of each evaluation window.
All three models show increased first-position attention and reduced NLL, although the final attention magnitude varies across seeds. This provides quantitative evidence beyond a single training run or selected attention head.
We also completed 12 paired continuation runs: four gradient-update conditions for each of the three pretrained seeds, with 1,500 updates per run. Removing the first-position value-gradient contribution reduces subsequent first-position attention in all three seeds. The mean paired difference is −0.000414, with a cross-seed 95% confidence interval of [−0.000590,−0.000239]; a norm-matched shuffled control produces almost no mean change.
We will bring the principal numerical results and causal-mask ablation into the main text, add variability bands to the multi-seed plots, and state the statistical unit for each estimate. Across-seed variation and article-bootstrap uncertainty will be reported separately. Deterministic synthetic experiments will be identified as such rather than presented as independent random repetitions.
The two-block model is chosen to separate two processes that are otherwise difficult to analyze jointly: contextual information aggregation and subsequent attention selection. The first block distributes earlier-token information across contextual representations; the second learns which representations to attend to. This allows us to trace how causal visibility affects value learning and how that learning reinforces attention concentration.
This separation is also important for weak semantic tokens. A repeated token can be uninformative in its original identity but carry predictive information after aggregating its preceding context. The model therefore lets us analyze both why such a token attracts attention and why attending to it can remain compatible with correct prediction.
The simplified architecture serves this mechanistic purpose, while the Pythia checkpoint analyses and full-parameter continuation experiments above examine related predictions in pretrained multilayer language models on natural text. We will present these two roles explicitly: the controlled setting establishes the theoretical mechanism, and the real-model experiments test its observable implications beyond that setting.
We will reorganize Section 5 around the central argument rather than the technical recurrences. For the first-token result, the roadmap is: causal contextualization creates unequal exposure; this produces a target-aligned value-projection advantage; the advantage then reinforces attention through query-key learning. For weak semantic tokens, we will explain the distinction between the shared token's class-agnostic contribution and the predictive information carried by its contextual representation before presenting the supporting lemmas.
We will keep this conceptual roadmap in the main text, move lengthy intermediate inequalities to the appendix, and add a compact notation table and a theorem-to-lemma dependency summary. We will also explain where each assumption enters the argument, helping readers distinguish the main mechanism from the technical conditions used to prove it.
These changes will make the paper easier to follow while preserving the rigor of the analysis. Together with the new multi-seed measurements and uncertainty estimates, they directly address the concerns about empirical support, model relevance, and readability.