acceptodds
Under review as a conference paper at ICLR 2027

A Normal Form for Softmax Attention

Abstract

We show that every softmax attention head admits a unique three-term normal form with interpretable components. An inspired choice of gauge in the score function clearly separates geometric and topological contributions, including a metric term which is the complement of a Laplacian in a head-induced geometry, a 1-cochain whose circulation encodes the area of token triangles, and a rank-one offset that biases the keys independently of the queries. This enables a precise characterization of when softmax attention is purely diffusive and when differently represented discrete Laplacian operators are the same. Applying the proposed normal form shows that softmax attention generically implements a quasilinear advection-diffusion equation satisfying a discrete maximum principle, guaranteeing stable information propagation regardless of its weights. Additional analysis shows that the metric and circulation terms in the normal form determine a guaranteed rate at which a softmax head mixes tokens, while the remaining offset term bounds the amount of attention that each key can receive. Ultimately, this leads to a reinterpretation of observed phenomena related to attention sinks and oversmoothing that may be useful for understanding deep attention networks.

Primary area:
interpretability and explainable AI
Keywords:
softmax attention, normal form, graph calculus, geometric deep learning, transformers

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pa4YOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proves that every softmax attention head admits a unique three-term normal form: a symmetric component log K (a pullback metric in the scaled dot-product case), a skew-symmetric divergence-free 1-cochain ζ (circulation, measuring deviation from pure diffusion via token-triangle areas), and a rank-one offset 1β^⊺ (a query-independent key bias, related to attention sinks). It gives a sharp characterization of when a head is the complement of a metric graph Laplacian (exactness of the balance cochain, Theorem 2.3), unifies common GNN Laplacians under one construction (Table 1), shows softmax attention implements a quasilinear advection-diffusion step satisfying a discrete maximum principle (Theorem 4.1, Corollary 4.2), and derives guaranteed smoothing/squashing rates recovering the cubic rank-collapse of Dong et al. (2021). Experiments on pretrained encoders measure normal-form energy shares, ablate coordinates, and verify exact key-bias redundancy and token-centering effects on SST-2.

Critical strengths

S1. The mathematics is clean and correct as far as I can verify. The gauge-invariance argument (Proposition 3.1), the Coulomb-gauge construction of the unique normal form (Theorem 3.4), and the advection-diffusion decomposition (Theorem 4.1) are self-contained, with proofs at an appropriate level of rigor. The connection to Kolmogorov’s criterion / detailed balance (Remark 2.4) is a nice touch.

S2. The attention-sink interpretation of β is genuinely interesting. Corollary 3.6 showing β measures alignment of centered tokens with the token centroid, the exact removal of key-projection biases (Appendix C.2), and the LayerNorm vs. token-centering contrast give a mechanistic account that connects to a real empirical literature.

S3. Unifying Table 1 (combinatorial, random-walk, symmetric, GCN, diffusion-map Laplacians as one construction) is pedagogically valuable.

Critical weaknesses

W1. No causal or interventional connection between the normal form and model behavior is established. The theory characterizes what a head is, but the paper does not show that any observed phenomenon (attention sinks, oversmoothing, loss of rank) is caused by a particular normal coordinate. The interventions in Table 3 are output-perturbation measurements on trained encoders, not manipulations of training dynamics or downstream loss; zeroing ζ changes outputs by up to 47% while holding only 6–9% of the energy, which is reported but not explained or connected back to the theory in a predictive way. The claim that this “may be useful for understanding deep attention networks” (abstract) remains speculative.

W2. The central theoretical results are largely anticipated by prior work, and the incremental part is narrow. The diffusion/Laplacian view of attention is standard (Dong et al., 2021; Sander et al., 2022; Gruber et al., 2023; Wu et al., 2023a); the key-bias component of the normal form is essentially the gauge redundancy already documented in Namazifar et al. (2023), and the invariants (G, ξ) are drawn directly from Lee (2026). The genuinely new content — the skew/circulation coordinate ζ and its area-form interpretation — is not shown to have consequences that any of the cited prior frameworks cannot already express. For a theory paper, the absence of a sharp, novel, testable prediction that could not be obtained without the normal form is a substantive weakness.

W3. Key theoretical results hold only under restrictive conditions not representative of practice. Theorem 4.1’s dynamical interpretation treats a single head with no value projection, no normalization, no multi-head composition, and no residual stream — the authors concede this (Conclusion), but it means the advection-diffusion and smoothing-rate results (Theorems 4.1/4.3, Corollaries 4.2/4.4) apply to a subsystem that does not occur in any deployed architecture. The complete-graph restriction further confines most of Section 4. The disconnect between the mathematical object and actual transformer computation is acknowledged but underweighted.

W4. Experiments are limited to small, older encoder checkpoints. All empirical validation uses BERT/RoBERTa/DistilBERT on 96 blocks of WikiText-2 (and one SST-2 checkpoint). No decoder-only LLM is analyzed, although attention sinks — the headline application of β — are phenomena documented specifically in decoder LLMs (Xiao et al., 2024; Gu et al., 2025; Barbero et al., 2025, all cited but not tested). Since the normal form is architecture-agnostic and the decomposition is cheap to compute, this omission is hard to justify.

W5. Presentation. The paper is dense and assumes substantial graph-calculus background (coboundary operators, cochain spaces, gauge fixing) that most of the ICLR audience will not have; the main text would benefit from a worked example (e.g., a 4-token head decomposed explicitly) before the general theory. Figure 1 is under-explained.

Questions for authors

Can you demonstrate one concrete prediction of the normal form that is quantitatively confirmed — e.g., predicting which heads develop sinks (large ‖β‖) before training, or predicting rank collapse depth from measured κ(Y)? How does the decomposition behave on a decoder-only LLM (e.g., GPT-2 or Llama-scale) where attention sinks were actually documented? Is the β-coordinate concentrated on BOS/first-token positions as the theory suggests? Given W3, do any of the dynamical conclusions survive with value projections and multihead composition, even heuristically?

Reviewer_t5AROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates the mathematical structure and information propagation mechanism of Softmax Attention, proposing a canonical decomposition into three terms: a symmetric geometric term, an antisymmetric cyclic term, and a query-independent Key bias term. Drawing on graph calculus, gauge invariance, and the discrete Laplacian operator, the authors derive the necessary and sufficient conditions under which Softmax Attention can be interpreted as a pure diffusion operator, further characterizing general attention updates as a quasi-linear convection-diffusion process. Building on this framework, the paper analyzes properties such as the contraction of attention propagation, token smoothing rates, and bounds on attention weights, while attempting to link these mathematical characteristics to phenomena like oversmoothing and attention sinks. Experiments analyze the distribution of these canonical components across various pre-trained language encoders and evaluate their respective impacts on attention outputs through component intervention. Although the paper offers a comprehensive mathematical analysis, it does not sufficiently distinguish certain theoretical results from existing work; furthermore, the core conclusions rely on a somewhat idealized attention structure, and the experiments fail to fully validate the proposed theoretical explanations within practical Transformer networks.

Critical strengths
  1. This paper proposes a unique and orthogonal canonical decomposition of Softmax attention that distinguishes between symmetric, antisymmetric, and query-independent components. This representation provides a unified mathematical framework for analyzing various attention structures.

  2. By integrating the graph Laplacian, discrete differential forms, and the convection-diffusion equation, the paper derives results concerning pure diffusion conditions, eigenvalue contraction properties, and token smoothing rates, offering significant theoretical value.

Critical weaknesses
  1. The core canonical decomposition presented in this paper builds upon established mathematical frameworks—specifically, the row-normalization invariance of Softmax, symmetric and skew-symmetric matrix decomposition, and graph-based decomposition theories. Notably, the invariants of row-centered attention scores have already been investigated in On the Invariants of Softmax Attention (Lee, 2026). Although this paper introduces three orthogonal canonical forms, it fails to adequately articulate the significant theoretical insights—beyond those already accessible via existing frameworks—that this decomposition yields.
  2. Corollary 3.2 characterizes the quadratic form induced by the symmetric component of general attention weights as a "pullback metric"; however, since the symmetric weight matrix in standard Scaled Dot-Product Attention is not necessarily positive semi-definite, the corresponding quadratic form does not necessarily possess the properties of a valid metric.
  3. The proposed canonical decomposition relies on attention structures with symmetric connectivity, whereas modern autoregressive Transformers predominantly employ asymmetric causal attention masks. Furthermore, the analysis of contraction and stability focuses primarily on single-head propagation, failing to fully account for practical network components such as multi-head attention, value projections, residual connections, and normalization layers. Consequently, the paper's explanatory scope regarding general Softmax attention and deep Transformers is significantly limited.
  4. The experiments rely primarily on bidirectional encoders such as BERT, RoBERTa, and DistilBERT using a limited set of text samples, lacking validation on modern autoregressive language models.
Questions for authors
  1. Given that Softmax invariance properties and row-centered representations have already been investigated in works such as On the Invariants of Softmax Attention (2026), could the authors further clarify the independent innovations and irreplaceable theoretical contributions of the three canonical decompositions proposed in this paper relative to existing theoretical frameworks?
  2. How should the pullback metric in Corollary 3.2 be interpreted when the symmetric weight matrix of Scaled Dot-Product Attention is not positive semi-definite? Could the authors further specify the mathematical conditions required for the associated geometric interpretation to hold?
  3. Could the authors explain how the proposed canonical decomposition extends to Causal Attention (which has asymmetric support), and whether the current theoretical conclusions regarding stability, contraction, and oversmoothing can be generalized to full Transformer architectures that include multi-head attention and residual connections?
  4. There appears to be an inconsistency between Appendix C.3 and Appendix D.1 regarding whether attention is dynamically updated based on features in the Cora experiments. Could the authors clarify whether the attention matrix is ​​recomputed during the actual propagation process and explain how the relevant experiments validate the proposed adaptive convection-diffusion theory?
  5. Could the authors provide additional experiments on autoregressive language models and further validate the actual links between the different canonical components and phenomena such as "attention sinks" and oversmoothing using end-to-end prediction metrics, rather than relying solely on differences in attention outputs?
Reviewer_WWtBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates that every softmax attention head can be interpreted by a unique three-term normal form: a metric term, a 1-cochain and a rank-one offset that biases the keys indepedently of the queries. Applying the proposed form guarantee stable information propagation regardless of its weights. This leads to a reinterpretation of observed phenomena related to attention sinks and oversmoothing.

Critical strengths

1 Theoretical justification The paper provide a theoretical framework for studying oversmoothing and attention sinks ( a phenomena in modern transformer training) by connecting the symmetric and circulation components to token-mixing bounds and identifying a query-independent key bias associated with attention sinks. This provide an insightful perspective to the community.

2.informative experimental analysis Well controlled experiments are conducted across several pretrained encoders and illustrate the decomposition.

Critical weaknesses

1 Limited experiment scope It would be helpful to clarify whether the proposed normal-form decomposition extends to causal attention. In language modelling, the experiments are only conducted on BERT-style models and it would be helpful to demonstrate the analysis on decoder-only language models. Also, downstream experiments are limited to SST-2 and more downstream benchmarks should be evaluated.

Questions for authors
  1. How should the “pullback metric” interpretation be understood when the symmetric weight component is not positive semidefinite?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.