- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper analyzes the representation geometry evolution of transformers during training by analyzing the basis function decomposition of a single-layer transformer. With this setup, the authors theoretically identify three stages of training with qualitatively different behavior of the model parameter geometry: Condensation to a low-dimensional structure, dilution to a higher-dimensional one, and stabilization. The authors empirically evaluate these setup on both single-layer as well as larger models, finding agreement with the theory.
Critical strengths
- The paper considers the interesting question of how the model qualitatively changes throughout training. Obtaining better insights into this is certainly useful.
- The theoretical model, while simplified (a single layer), is realistic, and insights from the theoretical analysis can be observed in the experiments.
Critical weaknesses
- My biggest issue with the paper is that the complexity of the theoretical analysis in Section 2.2 obscures and makes it difficult to discern (a) what claims are specifically being made about the training dynamics that we can observe in the given model, and (b) what are straightforward consequences of the assumptions made in the process.
- For example, I imagine most of the “heft” of Thm. 2.3 comes from the two assumptions made in it; the implication follows quickly. However, it is difficult to establish how and when the assumptions would be applicable.
- I understand that the assumptions made were motivated by practical observations, but making assumptions that model the realistic behavior makes agreement with the experiments quite expected.
- On a related note, since I’m quite unfamiliar with related work in the area, I feel like the related work section does not comprehensively capture what parts of the methodology come from existing techniques and what is substantially novel. This would help put the work in perspective. I don’t think that the contributions are not novel, but I would like to see the techniques put into perspective.
- The language of the paper is imprecise. Some examples:
- The paper consistently refers to “LLMs”, while studying specifically single-layer transformers. Even when it becomes clear that the study is about transformers specifically, the emphasis on “LLMs” obscures the fact that theoretical claims are made specifically about single-layer models.
- Many quantities are used before being properly defined, e.g., A(X) in Eq. (4), M(t) in Prop. 2.3, Z in Eq. (11), etc.
- The paper refers to purely affine functions as “MLPs”.
- Some (in my eyes, non-trivial) claims are made without further justification, for example:
- Why can the number of heads be without loss of generality? (Line 233)
- Why do the stated conditions imply ? (Line 294)
- Some passages of the writing are also unclear and difficult to connect to the rest of the paper, e.g., the paragraph above Section 2.1.1.
Questions for authors
- Small typo: I think Eq. (44) is missing a derivative on .
4 comments
The language of the paper is imprecise.
Single-layer models.
For the theoretical analysis, we focus on single-layer models for simplicity. Simplified models are commonly adopted in theoretical studies of LLMs to facilitate the analysis of their underlying mechanisms. Meanwhile, our experiments demonstrate that the proposed framework can also be applied to practical multi-layer LLMs.
Many quantities are used before being properly defined and The paper refers to purely affine functions as “MLPs”.
We thank the reviewer for pointing this out. We will revise the manuscript to ensure that all quantities are properly defined before use and that purely affine functions are no longer referred to as MLPs.
Why can the number of heads be 1 without loss of generality?
We acknowledge that our explanation near Line 233 was unclear. Our original motivation was that the analyses of Stages II and III do not depend on . For Stage I, under the approximation that the Softmax output is a constant matrix, the number of attention heads only affects the dimensions of the corresponding matrices, rather than the underlying argument.
In the revised version, we will remove the statement that a single attention head can be assumed without loss of generality, and instead explicitly state that we adopt a simplified model for the theoretical analysis.
Why do the stated conditions imply ?
We clarify that is explicitly introduced as an assumption in Assumption 2.2. We further state that, under , the condition provides a favorable high-dimensional regime and a sufficient condition for . Additionally, the results in Figures 2 and 3 show that the assupmtion is practical.
Some passages of the writing are also unclear and difficult to connect to the rest of the paper.
The paragraph preceding Section 2.1.1 is important because the basis representation of the model output is not unique. Different basis representations can lead to different values of the metrics, including the effective rank. This paragraph clarifies that, throughout the subsequent analysis, we focus on the metrics induced by the specific basis representation defined in our framework.
To improve the logical flow, we will move this paragraph to the end of Section 2.1.1, where the metrics are introduced.
Small typo
Please refer to our response to the A later proof assumes that the largest eigenvalue of a Gram matrix—the matrix used to compare component directions—is negative. (Q1) of Reviewer jqk9 regarding the correction.
I thank the authors for the clarifications. I appreciate the explanations. They resolve some of my concerns, but the overarching issue regarding the usefulness of the results, given the restrictive assumptions, remains. For example, while I appreciate the more accurate setting with the single attention head, this makes the results even more restrictive. I will raise my score to a weak reject.
We thank the reviewer for acknowledging our clarifications and reconsidering the score.
We would like to clarify that the primary goal of this work is to understand the underlying training dynamics of LLMs, rather than to directly propose a practical training algorithm. From this perspective, we believe that the contribution of this manuscript should be assessed not only by practical applicability, but also by the theoretical explanation of the phenomena in LLM training.
For example,
-
NIPS2018: Neural Tangent Kernel: Convergence and Generalization in Neural Networks analyzes the training dynamics for a MLP with infinite-width.
-
NIPS2019: Implicit Regularization in Deep Matrix Factorization analyzes the training dynamics for a MLP without activation function and bias.
-
ICLR2025: A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Language analyzes LLMs through a simplified formal-language setting.
-
ICML2026: Focus and Dilution: The Multi-stage Learning Process of Attention analyzes the one-layer one-head transformer without W_V and FFN.
We hope that these accepted papers will help address your concerns and lead you to reconsider your assessment of our paper's contribution.
Furthermore, the three-stage phenomenon has also been empirically verified in real-world LLMs trained on real-world datasets.
The complexity of the theoretical analysis in Section 2.2 obscures and makes it difficult to discern ...
We first clarify the two main points raised by the reviewer.
1) Most of the “heft” of Thm. 2.3 comes from the two assumptions made in it.
We agree that the implication follows relatively directly from the two assumptions. To provide further insights into their roles and implications, we initially discussed them in Remarks E.2 and E.3 (As noted in the footnote near Line 377 of the original version), these remarks are important for understanding the theorem. To make their significance more explicit, we will move both remarks to the main text (See Remarks 2.1 and 2.2 in the revision).
2)Making assumptions that model the realistic behavior makes agreement with the experiments quite expected.
We would like to clarify that our assumptions are not formulated based on the observed behavior of LLMs. Rather, they are necessary conditions for establishing the corresponding theoretical results.
For example, in Theorem 2.3, if the high-effective-rank condition is not satisfied, the network may be able to accomplish the learning task using a representation with low effective rank, in which case an increase in effective rank is not necessary. Similarly, if the condition in Eq. (21) is not satisfied, we cannot characterize the direction of the representation update. Consequently, the effective rank may increase, decrease, or even oscillate. Therefore, these assumptions are introduced to establish the conditions under which the corresponding theoretical conclusions hold, rather than to encode the empirically observed behavior of LLMs. Our experiments independently demonstrate that these conditions are satisfied in the settings we study.
As a counterexample, consider a 10-layer MLP with 100 neurons per layer trained to approximate an affine target function. Since the target function can be represented with low effective rank, the effective rank of the learned representation does not necessarily increase during training.
Then for the discerning raised by the reviewer.
(a) What claims are specifically being made about the training dynamics that we can observe in the given model
Our main claims concerning training dynamics are as follows:
Together, these results characterize the three-stage evolution of effective rank during training.
(b) What are straightforward consequences of the assumptions made in the process.
The assumptions play different roles in our theoretical analysis:
In summary, each assumption serves a specific purpose in the theoretical analysis. The theorems together tell a coherent story about the evolution of effective rank in LLMs: it first decreases, then increases, and finally stabilizes. The only prior consideration is that LLMs are expected to learn complex, high-dimensional mappings.
Additionally, the reviewer jqk9 asks for further experimental details demonstrating whether the theoretical assumptions are satisfied (See Details of experiments for the conditions (Q3).) And we put them in Appendix D.1.
In short, we first derive the assumptions theoretically and then verify them empirically through experiments.
The related work section does not comprehensively capture what parts of the methodology come from existing techniques and what is substantially novel.
We thank the reviewer for pointing this out. We will revise the Related Work section (See Appendix A) to more clearly distinguish the existing techniques on which our methodology builds from the novel contributions of our work. We hope that this clarification will help the reviewer better assess the novelty of our contributions.