My main concerns in this paper regard the clarity of the presentation and interpretation, as well as the support of the empirical observations. I personally find the underlying idea considerably simpler and more interesting than the current exposition makes it appear.
-
I think the paper requires a substantial revision for clarity, beyond correcting individual sentences. As an example, the surjectivity assumption in theorem 1 should also be written as rank(J)=2n, alongside the stated condition N≥2n. Terms such as "Zariski-open dense," "hyperbolic," "radical," and "congruence", which are disseminated in the whole paper, should be accompanied by their concrete matrix meaning when first used. I noticed some background is provided later, but a reader like me had to work back and forth to understand fairly elementary arguments. Another example: the phrase "independent linear maps" for trained Q,K also needs clarification, since it could be read as statistical independence, which is not required by the balance theorem theorem stated by the authors. Overall, I personally think the whole manuscript needs a better rewriting, ensuring all quantities are properly defined and results cleanly explained, without directly mentioning technical terms taken from different literature fields.
-
I also think the profile construction would benefit from an equally direct introduction. I think the authors should explain early that each point represents one head's matrix statistics, not attention probabilities over tokens. The meaning of "overlap" between sorted eigenvalue magnitudes, the three types called "parity," and the difference between a rank-one profile and a rank-one matrix should be explained in plain terms. Standard mathematical results should also be identified and referenced where they are used, so that a reader like me can distinguish established tools from the definitions and observations introduced here in this work.
-
I think the main empirical finding, which I found very interesting, deserves more quantitative support. The figures suggest accumulation near the two segments the authors' refer as Θ, but the Gaussian reference already lies near its central point. The interesting difference is therefore the spread of these trained profiles along the segments, not proximity to the segments alone. I would appreciate if the authors could add measurements of the actual distance to the submanifold Θ, together with the corresponding Gaussian comparison and a further investigation of the opt-350m exception. I think a further useful check would be a simple random model with correlated query and key matrices, to investigate whether part of this spread can be reproduced without training.
-
The mathematical results are relatively elementary, so in my opinion the main contribution is the representation provided by the authors and the empirical regularity along pretrained models. Moreover, the theorem 1 concerns eigenvalue counts, whereas the profile also uses magnitudes but omits eigenvectors and the input distribution. This limits what can be inferred about actual attention behavior, so I would appreciate that the authors acknowledge these limitations, and their practical meaning should be made clearer when discussing the results.
-
In theorem 3 the authors assume that the positive and negative eigenvalues of the symmetric matrix S follow the same distribution up to a rescaling. Under this assumption, they show that the profile of S approaches the edges of the symmetric face of the simplex. However, the main empirical observation concerns the profiles of the full matrix L=S+T concentrating near Θ, which also requires a particular relationship between the symmetric and antisymmetric parts (a≈b). If I understood correctly, this does not seem to be explained by theorem 3, nor it's clear to me why training should produce such a relationship. The authors acknowledge that their explanation is partial, but I think section 5 should clarify better which part of the empirical observation is explained by the theorem and which remains open.
-
I personally found sections 2.1 and 4.4 insufficiently connected to the main argument of this paper. In section 2.1, I think the authors should explain more clearly which contributions to the actual attention scores are considered in their analysis and which are neglected. In particular, for models using RoPE, the experiments consider only relative position zero, although the matrix describing the bilinear interactions generally depends on the relative position of the tokens. I would appreciate from the authors a clearer motivation for this choice and its implications for the results. In section 4.4, the authors fit different distributions to the spectra of the symmetric and antisymmetric parts and report several relations between their fitted parameters. However, I personally found it difficult to understand the motivation and significance of these additional observations, and in particular how they relate to the main empirical finding of concentration near the submanifold Θ. I think these connections should be better explained in the paper.
Overall, despite all the weaknesses stated, I personally lean towards weak acceptance. I find the profile construction and the empirical pattern worth reporting and interesting despite the simplicity of the algebra. However, my assessment is borderline, in the sense that a substantial improvement in clarity, motivation, and the interpretation of the measurements is important to my final recommendation. For this reason, I would appreciate a concrete response to my concerns, in particular for the support for the central concentration claim.
Thank you very much for the detailed and helpful suggestions.
We've made a careful pass through the paper in an attempt to avoid any technical terms. We've also incorporated the many specific and helpful suggestions that were in your list of weaknesses for the paper. Standard results are now cited where they are used (for example Sylvester's law of inertia in the proof of Theorem 1).
We added quantitative support for the main empirical finding. There is now a table (Table 1) and a figure showing accumulation for all 18 models (Figure 3), and Figure 5 shows the distributions of the distance from Theta and of the position along it for every model. We agree with your reading of the Gaussian reference: Gaussian heads already lie at the midpoint of Theta, and what distinguishes trained heads is their spread along the two segments. Table 1 reports this comparison: the 90th percentile of |c - d| is at most 0.0002 for Gaussian heads and between 0.04 and 0.18 for the trained heads of each model.
Following your suggestion to try a simple random model with correlated query and key matrices, we tried the model K = rho Q + sqrt(1 - rho^2) G with Q and G independent Gaussian matrices. Indeed its profiles lie on Theta at position c - d ≈ rho |rho|, so correlation between the query and key matrices reproduces the spread along Theta without training (Section 4.6 and Appendix A.6). This suggests that the explanation of the spread is some form of correlation between the trained query and key matrices, which remains to be described. We felt this was a significant finding that reframes the empirical finding of accumulation along Theta.
The opt-350m exception was an artifact of an error in our extraction code that only affected the opt models (we describe the error in a general comment of this thread). With the corrected data opt-350m accumulates near Theta like the other models.
We also explained why the restriction to relative distance zero is not a significant restriction (it has an insignificant effect on the profiles, see Figure 4).
We explained that the equality between a and b (one of the defining equations for Theta) is not addressed by Theorem 3 but that it follows from Observation 3, one of the open observations from the paper. We also explained in more detail that Theorem 3 explains the other defining equation cd = 0, subject to the assumption of approximate proportionality of the two lobes of S's eigenvalue spectrum.
We revised and reorganized sections 2.1 and 4.4 for clarity. Section 2.1 now states which parts of the attention score are measured and which are omitted. Section 4.4 became Section 4.7, and it now more clearly presents its finding (Observation 3); the fitted distributions moved to Appendix C.2. The Discussion now acknowledges that the profile omits the eigenvectors and the value and output maps.