acceptodds
Under review as a conference paper at ICLR 2027

Sink or Swim: Diagnosing and Resolving the Relational Invariance Bottleneck in Causal Attention

Abstract

Decoder-only Transformers consistently concentrate attention on the first token, a phenomenon known as attention sink. We trace its origin to a structural constraint we call the relational invariance bottleneck: causal attention has no neutral gear— row-stochasticity forces every head to move mass somewhere, while causality makes the first position the only universal sink. An attention matrix is a universal relational invariance if it preserves every input up to a token-shared shift; under this definition, we establish the exact intersection: Collapse onto token one is therefore not an incidental artifact but the only available invariance target. To quantify this mechanism, we introduce a relational energy measuring how much token-level variation a head’s output carries, and prove that it is bounded by the head’s distance to the invariance target: This separates cause from symptom: first-token concentration matters only insofar as it signals proximity to the invariance target that suppresses relational output. Motivated by this diagnosis, we propose Adaptive Diagonal Target Editing (ADTE), a lightweight per-head module that cancels the self-token component of each attention row and rescales the remaining cross-token signal, giving heads an alternative relational invariance route that no longer requires concentrating mass on the first token. Across two architectures, ADTE consistently lowers pretraining perplexity compared with all baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Attention sink, Relational invariance bottleneck, Adaptive Diagonal Target Editing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_UiP9OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper shows a phenomenon of the first-word attention aggregation in the decoder Transformer and proposes the relation invariance bottleneck as an explanation. The paper firstly proves that under the constraints of causality and row randomness, concentrating all attention on the first element is the only operator that maintains the difference between word elements before and after residual update for any Value matrix. Furthermore, it establishes a boundary between the relationship output and the distance of this operator. Based on this, the paper proposes Adaptive Diagonal Target Editing (ADTE), which combines learnable diagonal cancellation and word-wise amplification to provide another path for achieving relation invariance.

Critical strengths
  1. The author focuses on why attention usually converges on the first word element and attempts to provide an explanation based on the structural constraints of causal attention. I believe this issue has research value and also offers a meaningful perspective for understanding the attention sink.

  2. The design of ADTE is simple and the additional parameter overhead is small. The paper reports consistent improvement in perplexity under different model scales and normalized architectures, indicating that this method has certain practical value.

Critical weaknesses
  • Firstly, the title and abstract of the manuscript have affected the reading experience. The title on the front page still uses the format of a conference submission description, which does not match the content of the paper. The multiple key symbols in the abstract formula also lack necessary explanations, and readers need to refer to the main text to understand the main conclusions.
  • Secondly, the design basis of the word-by-word amplifier is not yet fully established. The author independently determined the amplification coefficient, but did not conduct experimental comparisons and directly adopted the relatively intuitive scheme of aℓ,h=(1−λℓ,h)−1a^{\ell,h}=(1-\lambda^{\ell,h})^{-1} to restore the unit row and equal comparison. The existing ablation studies indicate that the amplifier is effective, but do not fully explain the necessity of this specific design.
  • Finally, the current experiments still do not provide sufficient support for the research motivation of the paper. The experiments demonstrated the reduction of aggregation and the improvement of perplexity, but the verification of preserving the relationships between word elements was not direct enough. It is suggested to supplement the comparison of the differences in word element representations before and after the update of the baseline and ADTE, in order to more directly support the explanation of the invariance of the relationships.
Questions for authors
  • Author should revise the title of the main page (pdf file) and clarify the key symbols in the equations presented in the abstract.
  • Compared with alternatives such as directly enforcing unit row sums, what advantages does the design of a separately learned scaling factor offer? Could you provide corresponding ablation studies to support this?
  • By comparing token representations before and after the ADTE update with those from the baseline, can we obtain more direct evidence to support the invariance interpretation?
  • Could you include downstream evaluation results on 1B or even larger models to verify the practical gains and scalability of ADTE?
Reviewer_Rn8POpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigate the attention sink phenomena in Transformer based causal LLMs, specifically they attempt to understand what feature od the attention mechanism makes the first token the attention sink. They propose a structural rational based on "relation invariance bottleneck". The prove that assigning all the weight to the first token is the only causal, row-stochastic operation that achieves this invariance universally across value matrices. They propose an architectural change combining tokenwise amplification with learned diagonal subtraction which they name Adaptive Diagonal Target Editing (ADTE) and show empirical results of ADTE that improves the models perplexity across 3 model scales and normalizations.

Critical strengths
  1. the results seems impressive and promising, ADTE achieves the lowest perplexity and the component ablations also provide clear contribution.

  2. The mathematical distinction between zero-attention output and token-shared output is interesting. Theorem 1 effectively formalizes exact, value-agnostic relational invariance under causal masking constraints.

  3. the additional evaluation on llama3-8B and qwen3-8B, providing evidence across two model families beyond the smaller models trained for the architectural experiments shows impressive results.

Critical weaknesses
  1. aside from perplexity it will go good to show the benefit and performances on downstream tasks. did you get to expereiment with other metrics aside from perplexity?

  2. The baseline comparisons are largely restricted to standard attention and one scalar-gated variant. it would be good to Include additional baselines that address SoftMax constraints (e.g., clipped SoftMax and alternative gating methods)

  3. At 250M parameters, amplification alone improves perplexity from 20.89 to 20.36 (ADTE-A), while full ADTE reaches 20.20. Therefore, much of the observed improvement is already obtained without diagonal editing.

  • some formatting issues: the title is missing from the paper. some citations should be consolidated (e.g. in line 037). the figures text is too small to read (e.g. figure 4)
Questions for authors

aside from the weaknesses.

  • How can we rule out alternative mechanisms for update suppression, such as learned value or output projections, in favor of your value-agnostic mechanism?
Reviewer_XGyzOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates first-token attention sinks in causal Transformers through relational invariance, defined as preserving pairwise token differences under a residual attention update. The authors prove that the only causal row-stochastic operator achieving this property for every value matrix is the first-token sink operator, 1e1⊤\mathbf{1}e_1^\top. They further derive an upper bound relating normalized relational-output energy to the operator's distance from this target and to first-token attention mass.

Motivated by this characterization, the paper proposes Adaptive Diagonal Target Editing (ADTE), replacing PVPV with A(P−D)VA(P-D)V, where DD learns tokenwise self-subtraction and AA learns tokenwise amplification. Validation includes head-level measurements on pretrained LLaMA3-8B and Qwen3-8B, a fixed-diagonal intervention that shifts attention toward the diagonal, C4 pretraining experiments at 130M–1B scales, component ablations, and supplementary architecture and downstream evaluations. ADTE improves perplexity over the reported vanilla and gated baselines in the main PreNorm and LNS settings.

Critical strengths
  1. Theorem 1 identifies how causality and row-stochasticity jointly single out the first-token sink as the unique operator achieving relational invariance for every value matrix. The fixed-diagonal intervention complements this characterization: introducing an alternative invariance route shifts attention concentration from the first token toward the diagonal. This connects an explicit mathematical characterization to an architectural intervention and adds a useful perspective to existing no-op and anti-overmixing explanations.

  2. ADTE adds approximately 0.16–0.24% parameters while reducing validation perplexity relative to both the vanilla and gated baselines across all reported 130M–350M PreNorm and LNS configurations. The supplementary 1B experiment also reports an improvement. The component ablation supports combining the two modules: at 250M, full ADTE achieves 20.20 PPL, compared with 20.36 for amplification alone and 20.58 for diagonal editing alone. These results provide evidence that the proposed modification can improve language modeling with a small parameter overhead.

Critical weaknesses
  1. Section 3.2 states that suppressing relational output “forces” attention toward the first-token sink. Theorem 1 establishes this necessity when relational invariance holds exactly for every value matrix. Theorem 2 shows that being close to the sink limits normalized relational-output energy, but does not establish the reverse implication for approximate invariance. The conditions needed for this further claim are not specified. Figures 2 and 6 show that stronger sinks occur alongside lower normalized relational-output energy, but do not establish that heads develop sinks because they need to preserve token differences. This leaves a gap between the mathematical characterization and the explanation of why sinks form during training.

  2. In Table 2, fixed diagonal subtraction reduces sinks but worsens perplexity from 20.89 to 21.62, while full ADTE improves perplexity to 20.20. Thus, reducing sinks alone does not explain the performance improvement. Learnable DD and AA also give the model more freedom to choose how much of its own value to subtract and how strongly to scale the resulting update. The results do not clearly separate the benefit of this added flexibility from the specific relational-invariance explanation. Figure 8 could help connect the two, but it is unclear whether its ADTE output metric measures PVPV or the actual edited output A(P−D)VA(P-D)V. This ambiguity makes it difficult to assess whether the figure supports the proposed replacement of the sink's function.

Questions for authors
  1. Under what conditions does approximately suppressing relational output, independently of the particular value matrix, require attention to be close to the first-token sink? Explaining these conditions and how they relate to Theorems 1 and 2 would help clarify the scope of the “forces” statement in Section 3.2.

  2. How do the authors interpret the fact that fixed diagonal subtraction reduces sinks but worsens perplexity, whereas ADTE improves perplexity? Learnable DD allows the model to choose how much of its own value to subtract, and AA allows it to adjust the output scale. Explaining how these choices avoid the problems of fixed subtraction would help clarify why ADTE works, and whether its gains specifically support the proposed relational-invariance explanation.

  3. Section 3.2 defines Γ\Gamma using the standard attention output PVPV. For the ADTE curve in Figure 8(d), it is unclear whether the numerator still uses PVPV or instead uses the actual output A(P−D)VA(P-D)V. Stating which output is measured would help clarify whether the figure describes attention before editing or the update actually passed to the residual stream, and therefore how it supports the interpretation given in the caption.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.