acceptodds
Under review as a conference paper at ICLR 2027

On attention heads and bilinear forms

Abstract

We study the symmetric and antisymmetric parts of bilinear forms encoded by the query and key maps of attention heads, separating properties imposed by the architecture from patterns associated with training. We prove that, whenever the stacked query–key map is surjective, the symmetric part has exactly as many positive as negative eigenvalues. This condition is generic when the embedding dimension is at least twice the head dimension, and the predicted signature occurs in all heads of fourteen pretrained language models. To describe variation beyond this structural constraint, we introduce an orthogonally invariant profile map into a three-dimensional simplex. In thirteen of the fourteen models, trained profiles accumulate near the one-parameter family of profiles of rank-one forms. A limit theorem partially explains this phenomenon: profiles of symmetric matrices with proportionally distributed positive and negative eigenvalues converge to the projection of the rank-one profile family onto the symmetric face.

Primary area:
interpretability and explainable AI
Keywords:
attention, transformers, bilinear forms, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rC6mOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the bilinear forms defined by the query and key matrices of attention heads, decomposing them into symmetric and antisymmetric parts. It proves that, under specific conditions, the symmetric part has equally many positive and negative eigenvalues.

Critical strengths
  1. The paper includes both a theoretical part and observations that complement it. The observations are drawn from many different models showing a meaningful analysis of existing models.
Critical weaknesses
  1. The presentation is very bad. I can't understand the gaps that this work tries to fill, and it seems that most of the theory presented here concerns general properties of bilinear forms. The contributions paragraph is extremely short and doesn't explain the results. The related work section is also very short and doesn't provide the necessary background to explain how this paper builds upon existing work.

  2. The main theorem (according to the contributions paragraph) is the balance theorem. Its statement uses terms that are not defined anywhere in the paper (for example, “Zariski-open dense”), and the proof is not very clear.

Questions for authors
  1. Please rewrite the introduction to explain the gap that you aim to fill in your manuscript.

  2. The contributions paragraph is very short. Please elaborate on the contributions of your work.

  3. Please explain the balance theorem more clearly. Several terms in the theorem’s statement and proof, such as “Zariski-open dense” are not introduced. This applies to the entire paper: all terms used need to be introduced clearly.

  4. The discussion section appears to describe only the limitations of the work without providing insight into why the work is important. Could you clarify its significance?

Reviewer_JHPFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

In this paper the authors study the matrix L=K⊤QL=K^\top Q formed by the query and key weights of an attention head, where they consider separately its symmetric and antisymmetric parts. The authors show that when the stacked query and key matrix has full row rank, the symmetric part has exactly nn positive and nn negative eigenvalues, where nn is the head dimension. The authors then introduce four simple statistics describing the magnitude of the antisymmetric component and the relative magnitudes of the positive and negative symmetric spectra. These statistics define a profile in a three-dimensional simplex, and the profiles of rank-one matrices form two line segments. They compute the pretrained spectra of 9512 heads from fourteen language models and they show that thirteen of these models accumulate near these segments although the underlying matrices remain effectively high rank. In this paper the authors also compare trained heads with benchmarks over Gaussian query/key matrices, they examine the relationships between the spectral distributions and prove a result for symmetric profiles when the positive and negative spectra have the same distribution up to a scaling factor.

Critical strengths

I personally find this paper interesting despite its genuine simplicity. I find the simplex representation the authors provide appealing and the concentration of pretrained language models near the two rank-one segments of the simplex very interesting. This construction gives a simple way to compare many spectral properties of attention heads together, and the observation the authors found across different pretrained model families is the main reason for my positive assessment. In my opinion identifying this regularity can be useful even without a complete explanation of how training produces it in practice.

I appreciate the distinction between properties imposed by the architecture and those observed in trained weights. The balance theorem helps explain why equal positive and negative eigenvalue counts need not indicate a learned behavior. The authors also compare with Gaussian matrices, report an exception to the main empirical pattern, and explicitly check that proximity to rank-one profiles does not mean that the matrices themselves become rank one.

Critical weaknesses

My main concerns in this paper regard the clarity of the presentation and interpretation, as well as the support of the empirical observations. I personally find the underlying idea considerably simpler and more interesting than the current exposition makes it appear.

  1. I think the paper requires a substantial revision for clarity, beyond correcting individual sentences. As an example, the surjectivity assumption in theorem 1 should also be written as rank⁡(J)=2n\operatorname{rank}(J)=2n, alongside the stated condition N≥2nN\geq2n. Terms such as "Zariski-open dense," "hyperbolic," "radical," and "congruence", which are disseminated in the whole paper, should be accompanied by their concrete matrix meaning when first used. I noticed some background is provided later, but a reader like me had to work back and forth to understand fairly elementary arguments. Another example: the phrase "independent linear maps" for trained Q,KQ,K also needs clarification, since it could be read as statistical independence, which is not required by the balance theorem theorem stated by the authors. Overall, I personally think the whole manuscript needs a better rewriting, ensuring all quantities are properly defined and results cleanly explained, without directly mentioning technical terms taken from different literature fields.

  2. I also think the profile construction would benefit from an equally direct introduction. I think the authors should explain early that each point represents one head's matrix statistics, not attention probabilities over tokens. The meaning of "overlap" between sorted eigenvalue magnitudes, the three types called "parity," and the difference between a rank-one profile and a rank-one matrix should be explained in plain terms. Standard mathematical results should also be identified and referenced where they are used, so that a reader like me can distinguish established tools from the definitions and observations introduced here in this work.

  3. I think the main empirical finding, which I found very interesting, deserves more quantitative support. The figures suggest accumulation near the two segments the authors' refer as Θ\Theta, but the Gaussian reference already lies near its central point. The interesting difference is therefore the spread of these trained profiles along the segments, not proximity to the segments alone. I would appreciate if the authors could add measurements of the actual distance to the submanifold Θ\Theta, together with the corresponding Gaussian comparison and a further investigation of the opt-350m exception. I think a further useful check would be a simple random model with correlated query and key matrices, to investigate whether part of this spread can be reproduced without training.

  4. The mathematical results are relatively elementary, so in my opinion the main contribution is the representation provided by the authors and the empirical regularity along pretrained models. Moreover, the theorem 1 concerns eigenvalue counts, whereas the profile also uses magnitudes but omits eigenvectors and the input distribution. This limits what can be inferred about actual attention behavior, so I would appreciate that the authors acknowledge these limitations, and their practical meaning should be made clearer when discussing the results.

  5. In theorem 3 the authors assume that the positive and negative eigenvalues of the symmetric matrix SS follow the same distribution up to a rescaling. Under this assumption, they show that the profile of SS approaches the edges of the symmetric face of the simplex. However, the main empirical observation concerns the profiles of the full matrix L=S+TL=S+T concentrating near Θ\Theta, which also requires a particular relationship between the symmetric and antisymmetric parts (a≈ba\approx b). If I understood correctly, this does not seem to be explained by theorem 3, nor it's clear to me why training should produce such a relationship. The authors acknowledge that their explanation is partial, but I think section 5 should clarify better which part of the empirical observation is explained by the theorem and which remains open.

  6. I personally found sections 2.1 and 4.4 insufficiently connected to the main argument of this paper. In section 2.1, I think the authors should explain more clearly which contributions to the actual attention scores are considered in their analysis and which are neglected. In particular, for models using RoPE, the experiments consider only relative position zero, although the matrix describing the bilinear interactions generally depends on the relative position of the tokens. I would appreciate from the authors a clearer motivation for this choice and its implications for the results. In section 4.4, the authors fit different distributions to the spectra of the symmetric and antisymmetric parts and report several relations between their fitted parameters. However, I personally found it difficult to understand the motivation and significance of these additional observations, and in particular how they relate to the main empirical finding of concentration near the submanifold Θ\Theta. I think these connections should be better explained in the paper.

Overall, despite all the weaknesses stated, I personally lean towards weak acceptance. I find the profile construction and the empirical pattern worth reporting and interesting despite the simplicity of the algebra. However, my assessment is borderline, in the sense that a substantial improvement in clarity, motivation, and the interpretation of the measurements is important to my final recommendation. For this reason, I would appreciate a concrete response to my concerns, in particular for the support for the central concentration claim.

Questions for authors

I list my main questions below in a synthetic way. Please refer to the weaknesses section above for the more detailed discussion.

  1. Can the authors improve the presentation of the main assumptions, definitions, and mathematical results, explaining the technical terminology more clearly and providing references where appropriate? Maybe with a simple example illustrating the profile construction.
  2. Can the authors provide a quantitative analysis of the concentration near Θ\Theta, distinguishing the distance from the two segments and the spread along them? How do these measurements compare with the Gaussian reference, and do you have insights on what happens for the opt-350m exception? Can a simple random model with correlated query and key matrices reproduce a similar pattern?
  3. Can the authors clarify what the observed proximity to Θ\Theta implies about the symmetric and antisymmetric parts of the attention matrices? In particular, which aspects of the empirical observation are explained by theorem 3, and which remain unexplained?
  4. Can the authors better motivate sections 2.1 and 4.4 and explain their relevance to the main findings? Why the RoPE measurements are restricted to relative position zero, and what information do the fitted spectral distributions add more than the simplex representation?
1 comment

Thank you very much for the detailed and helpful suggestions.

We've made a careful pass through the paper in an attempt to avoid any technical terms. We've also incorporated the many specific and helpful suggestions that were in your list of weaknesses for the paper. Standard results are now cited where they are used (for example Sylvester's law of inertia in the proof of Theorem 1).

We added quantitative support for the main empirical finding. There is now a table (Table 1) and a figure showing accumulation for all 18 models (Figure 3), and Figure 5 shows the distributions of the distance from Theta and of the position along it for every model. We agree with your reading of the Gaussian reference: Gaussian heads already lie at the midpoint of Theta, and what distinguishes trained heads is their spread along the two segments. Table 1 reports this comparison: the 90th percentile of |c - d| is at most 0.0002 for Gaussian heads and between 0.04 and 0.18 for the trained heads of each model.

Following your suggestion to try a simple random model with correlated query and key matrices, we tried the model K = rho Q + sqrt(1 - rho^2) G with Q and G independent Gaussian matrices. Indeed its profiles lie on Theta at position c - d ≈ rho |rho|, so correlation between the query and key matrices reproduces the spread along Theta without training (Section 4.6 and Appendix A.6). This suggests that the explanation of the spread is some form of correlation between the trained query and key matrices, which remains to be described. We felt this was a significant finding that reframes the empirical finding of accumulation along Theta.

The opt-350m exception was an artifact of an error in our extraction code that only affected the opt models (we describe the error in a general comment of this thread). With the corrected data opt-350m accumulates near Theta like the other models.

We also explained why the restriction to relative distance zero is not a significant restriction (it has an insignificant effect on the profiles, see Figure 4).

We explained that the equality between a and b (one of the defining equations for Theta) is not addressed by Theorem 3 but that it follows from Observation 3, one of the open observations from the paper. We also explained in more detail that Theorem 3 explains the other defining equation cd = 0, subject to the assumption of approximate proportionality of the two lobes of S's eigenvalue spectrum.

We revised and reorganized sections 2.1 and 4.4 for clarity. Section 2.1 now states which parts of the attention score are measured and which are omitted. Section 4.4 became Section 4.7, and it now more clearly presents its finding (Observation 3); the fitted distributions moved to Appendix C.2. The Discussion now acknowledges that the profile omits the eigenvectors and the value and output maps.

Reviewer_m64nOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper analyzes query-key bilinear forms by decomposing the attention head matrix into symmetric and antisymmetric parts. It aims to distinguish properties determined by architecture and training. The authors argue that the negative eigenvalues reported by Migliarini (2025) do not require an explanation in terms of the head's learned behavior based on Theorem 1. They design a profile with four coordinates. Each face of the simplex admits an explicit matrix characterization. They also provide a partial explanation of Observation 2 using Theorem 3. The analysis covers models with absolute, rotary, and ALiBi positional encodings.

Critical strengths
  • The mathematical concepts are clearly presented. The profile is well-designed, with clear interpretations of the faces and an intuitive visualization.
  • The analysis has broad coverage, ranging from 70M to 24B models. Three different positional encodings and GQA are also considered.
Critical weaknesses
  • Observation 2 is not quantified, and the only evidence is a visual inspection of Figure 3.
  • The biases are ignored in the analysis but GPT-2, OPT, and BLOOM all use them. Besides, analyzing the current mainstream models with QK-norm and Muon can provide more useful insights. The opt-350m exception is not analyzed. To my knowledge, it's the only analyzed model using post-LayerNorm. It could be a natural control for separating architectural from training effects.
Questions for authors
  1. Could you report the distribution of distances from the heads to Θ\Theta for each model? Ideally, report |a - b| and cd separately.
  2. What are the values of the three statistics in Table 1 under Gaussian initialization and under a model that preserves each head’s singular values but randomizes its singular vectors? Are the Gaussian initialization profiles obtained from synthetic matrices or from actual initialization checkpoints?
  3. Is the deviation of opt-350m from Θ\Theta related to its post-LayerNorm or its projection layer?
  4. It would be better to discuss [1] and [2] and clarify what information the profile provides beyond existing scalar measures.

[1] Shafayeth Jamil and Rehan Kapadia. The routing and filtering structure of attention. arXiv preprint arXiv:2605.18826, 2026.

[2] Hengyu Li. Fingerprint, not blueprint: How positional schemes set the default spectral algebra of attention. arXiv preprint arXiv:2607.06621, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.