acceptodds
Under review as a conference paper at ICLR 2027

Dynamic Analysis of Large Language Models from a Basis Representation Perspective

Abstract

We introduce a basis representation framework for analyzing how the representation geometry of large language models evolves during training by characterizing the representation geometry of internal architectures through Gram matrices. By integrating gradient flow analysis, we identify three stages of pretraining: a focus stage in which representation geometry condense onto a low-dimensional structure, a dilution stage in which previously insignificant directions are activated, and a stabilization stage in which the representation geometry converges while the parameters continue to evolve. We empirically validate these predicted dynamics across multiple LLMs and datasets. In particular, we consistently observe the theoretically predicted dynamics when training open-source LLMs on real-world datasets.

Primary area:
learning theory
Keywords:
large language models, dynamic analysis, basis representation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ca9JOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper analyzes the representation geometry evolution of transformers during training by analyzing the basis function decomposition of a single-layer transformer. With this setup, the authors theoretically identify three stages of training with qualitatively different behavior of the model parameter geometry: Condensation to a low-dimensional structure, dilution to a higher-dimensional one, and stabilization. The authors empirically evaluate these setup on both single-layer as well as larger models, finding agreement with the theory.

Critical strengths
  • The paper considers the interesting question of how the model qualitatively changes throughout training. Obtaining better insights into this is certainly useful.
  • The theoretical model, while simplified (a single layer), is realistic, and insights from the theoretical analysis can be observed in the experiments.
Critical weaknesses
  • My biggest issue with the paper is that the complexity of the theoretical analysis in Section 2.2 obscures and makes it difficult to discern (a) what claims are specifically being made about the training dynamics that we can observe in the given model, and (b) what are straightforward consequences of the assumptions made in the process.
    • For example, I imagine most of the “heft” of Thm. 2.3 comes from the two assumptions made in it; the implication follows quickly. However, it is difficult to establish how and when the assumptions would be applicable.
    • I understand that the assumptions made were motivated by practical observations, but making assumptions that model the realistic behavior makes agreement with the experiments quite expected.
  • On a related note, since I’m quite unfamiliar with related work in the area, I feel like the related work section does not comprehensively capture what parts of the methodology come from existing techniques and what is substantially novel. This would help put the work in perspective. I don’t think that the contributions are not novel, but I would like to see the techniques put into perspective.
  • The language of the paper is imprecise. Some examples:
    • The paper consistently refers to “LLMs”, while studying specifically single-layer transformers. Even when it becomes clear that the study is about transformers specifically, the emphasis on “LLMs” obscures the fact that theoretical claims are made specifically about single-layer models.
    • Many quantities are used before being properly defined, e.g., A(X) in Eq. (4), M(t) in Prop. 2.3, Z in Eq. (11), etc.
    • The paper refers to purely affine functions as “MLPs”.
    • Some (in my eyes, non-trivial) claims are made without further justification, for example:
      • Why can the number of heads be 11 without loss of generality? (Line 233)
      • Why do the stated conditions imply ε0≪0\varepsilon_0 \ll 0? (Line 294)
  • Some passages of the writing are also unclear and difficult to connect to the rest of the paper, e.g., the paragraph above Section 2.1.1.
Questions for authors
  • Small typo: I think Eq. (44) is missing a derivative on MM.
4 comments

The complexity of the theoretical analysis in Section 2.2 obscures and makes it difficult to discern ...

We first clarify the two main points raised by the reviewer.

1) Most of the “heft” of Thm. 2.3 comes from the two assumptions made in it.

We agree that the implication follows relatively directly from the two assumptions. To provide further insights into their roles and implications, we initially discussed them in Remarks E.2 and E.3 (As noted in the footnote near Line 377 of the original version), these remarks are important for understanding the theorem. To make their significance more explicit, we will move both remarks to the main text (See Remarks 2.1 and 2.2 in the revision).

2)Making assumptions that model the realistic behavior makes agreement with the experiments quite expected.

We would like to clarify that our assumptions are not formulated based on the observed behavior of LLMs. Rather, they are necessary conditions for establishing the corresponding theoretical results.

For example, in Theorem 2.3, if the high-effective-rank condition is not satisfied, the network may be able to accomplish the learning task using a representation with low effective rank, in which case an increase in effective rank is not necessary. Similarly, if the condition in Eq. (21) is not satisfied, we cannot characterize the direction of the representation update. Consequently, the effective rank may increase, decrease, or even oscillate. Therefore, these assumptions are introduced to establish the conditions under which the corresponding theoretical conclusions hold, rather than to encode the empirically observed behavior of LLMs. Our experiments independently demonstrate that these conditions are satisfied in the settings we study.

As a counterexample, consider a 10-layer MLP with 100 neurons per layer trained to approximate an affine target function. Since the target function can be represented with low effective rank, the effective rank of the learned representation does not necessarily increase during training.

Then for the discerning raised by the reviewer.

(a) What claims are specifically being made about the training dynamics that we can observe in the given model

Our main claims concerning training dynamics are as follows:

  • Theorem 2.2: The effective rank of the model tends to decrease toward a low value during the focus stage.
  • Theorem 2.3: The theorem characterizes the conditions under which the effective rank increases during the dilution stage.
  • Theorem 2.4: During the stabilization stage, the effective rank remains relatively stable even as the training loss continues to decrease.

Together, these results characterize the three-stage evolution of effective rank during training.

(b) What are straightforward consequences of the assumptions made in the process.

The assumptions play different roles in our theoretical analysis:

  • Assumption 2.1: Its direct consequence is the linear independence of the basis functions.
  • Assumption 2.2: Its direct consequence is that the initial effective rank is high.
  • Assumption 2.3: It does not directly imply a specific training dynamic, but provides a necessary condition for establishing Theorem 2.2.
  • Assumption 2.4: It does not have a direct standalone consequence. Instead, it is used to distinguish the dilution and stabilization stages and to establish the existence of both stages.
  • Assumption 2.5: It does not have a direct standalone consequence either, but is introduced to analyze the behavior of the model during the stabilization stage.

In summary, each assumption serves a specific purpose in the theoretical analysis. The theorems together tell a coherent story about the evolution of effective rank in LLMs: it first decreases, then increases, and finally stabilizes. The only prior consideration is that LLMs are expected to learn complex, high-dimensional mappings.

Additionally, the reviewer jqk9 asks for further experimental details demonstrating whether the theoretical assumptions are satisfied (See Details of experiments for the conditions (Q3).) And we put them in Appendix D.1.

In short, we first derive the assumptions theoretically and then verify them empirically through experiments.

The related work section does not comprehensively capture what parts of the methodology come from existing techniques and what is substantially novel.

We thank the reviewer for pointing this out. We will revise the Related Work section (See Appendix A) to more clearly distinguish the existing techniques on which our methodology builds from the novel contributions of our work. We hope that this clarification will help the reviewer better assess the novelty of our contributions.

The language of the paper is imprecise.

Single-layer models.

For the theoretical analysis, we focus on single-layer models for simplicity. Simplified models are commonly adopted in theoretical studies of LLMs to facilitate the analysis of their underlying mechanisms. Meanwhile, our experiments demonstrate that the proposed framework can also be applied to practical multi-layer LLMs.

Many quantities are used before being properly defined and The paper refers to purely affine functions as “MLPs”.

We thank the reviewer for pointing this out. We will revise the manuscript to ensure that all quantities are properly defined before use and that purely affine functions are no longer referred to as MLPs.

Why can the number of heads be 1 without loss of generality?

We acknowledge that our explanation near Line 233 was unclear. Our original motivation was that the analyses of Stages II and III do not depend on dheadd_{\mathrm{head}}. For Stage I, under the approximation that the Softmax output is a constant matrix, the number of attention heads only affects the dimensions of the corresponding matrices, rather than the underlying argument.

In the revised version, we will remove the statement that a single attention head can be assumed without loss of generality, and instead explicitly state that we adopt a simplified model for the theoretical analysis.

Why do the stated conditions imply ϵ0≪1\epsilon_0\ll 1?

We clarify that ϵ0≪1\epsilon_0\ll 1 is explicitly introduced as an assumption in Assumption 2.2. We further state that, under σ=0.02\sigma=0.02, the condition N≪DLsdoutN\ll\mathcal{D}L_s d_{\mathrm{out}} provides a favorable high-dimensional regime and a sufficient condition for ε0≪1\varepsilon_0\ll 1. Additionally, the results in Figures 2 and 3 show that the assupmtion is practical.

Some passages of the writing are also unclear and difficult to connect to the rest of the paper.

The paragraph preceding Section 2.1.1 is important because the basis representation of the model output is not unique. Different basis representations can lead to different values of the metrics, including the effective rank. This paragraph clarifies that, throughout the subsequent analysis, we focus on the metrics induced by the specific basis representation defined in our framework.

To improve the logical flow, we will move this paragraph to the end of Section 2.1.1, where the metrics are introduced.

Small typo

Please refer to our response to the A later proof assumes that the largest eigenvalue of a Gram matrix—the matrix used to compare component directions—is negative. (Q1) of Reviewer jqk9 regarding the correction.

I thank the authors for the clarifications. I appreciate the explanations. They resolve some of my concerns, but the overarching issue regarding the usefulness of the results, given the restrictive assumptions, remains. For example, while I appreciate the more accurate setting with the single attention head, this makes the results even more restrictive. I will raise my score to a weak reject.

We thank the reviewer for acknowledging our clarifications and reconsidering the score.

We would like to clarify that the primary goal of this work is to understand the underlying training dynamics of LLMs, rather than to directly propose a practical training algorithm. From this perspective, we believe that the contribution of this manuscript should be assessed not only by practical applicability, but also by the theoretical explanation of the phenomena in LLM training.

For example,

  1. NIPS2018: Neural Tangent Kernel: Convergence and Generalization in Neural Networks analyzes the training dynamics for a MLP with infinite-width.

  2. NIPS2019: Implicit Regularization in Deep Matrix Factorization analyzes the training dynamics for a MLP without activation function and bias.

  3. ICLR2025: A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Language analyzes LLMs through a simplified formal-language setting.

  4. ICML2026: Focus and Dilution: The Multi-stage Learning Process of Attention analyzes the one-layer one-head transformer without W_V and FFN.

We hope that these accepted papers will help address your concerns and lead you to reconsider your assessment of our paper's contribution.

Furthermore, the three-stage phenomenon has also been empirically verified in real-world LLMs trained on real-world datasets.

Reviewer_jqk9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper represents a Transformer block as a sum of architecture-induced basis functions and uses their normalized Gram matrix to characterize representation geometry. It tracks effective rank, extreme eigenvalues, and average absolute off-diagonal correlation. The proposed account divides training into focus, dilution, and stabilization: effective rank initially falls, subsequently rises, and eventually changes slowly even while optimization continues. The theoretical analysis studies a simplified single-layer Transformer under gradient flow and several additional assumptions. Experiments train a small single-layer model on synthetic Markov data and reduced LLaMA2- and Qwen3-style models on WikiText-2. The figures show related spectral trajectories across these settings, token-frequency distributions, and selected intermediate layers.

Critical strengths
  1. The construction in Section 2.1 connects Gram entries to components of the attention and MLP branches. The factorization in Theorem 2.4 further separates activation correlations from output-weight correlations. This makes the proposed measurement more interpretable than an unexplained scalar summary and gives a useful language for studying geometry independently of component magnitudes.

  2. Figure 2, Appendix C's intermediate-layer results, and the five Markov token-frequency settings show related contraction-and-expansion behavior, followed by a spectral plateau while perplexity can continue improving. These observations warrant investigation independent of the proposed explanation. They establish breadth across the reported configurations, although repeated-seed uncertainty is not reported and independent reproducibility has not been demonstrated here.

Critical weaknesses
  1. Several mathematical steps do not support the guarantees as written. One claim says that random initialization and a sufficiently large data representation ensure that the component outputs are independent. But repeating the same input many times can satisfy the stated size condition without adding any independent directions. This gives a counterexample to Proposition 2.3 and Appendix E.3. A later proof assumes that the largest eigenvalue of a Gram matrix—the matrix used to compare component directions—is negative. That is impossible for this kind of matrix (Equation 44). Interpreting the expression as the matrix's rate of change does not fix it: normalization keeps the sum of its diagonal entries constant, which rules out the proposed condition. The claimed time by which independence must be lost is therefore not established. A more limited result—that independence persists for a short time if it holds initially—can still be valid.

  2. There is also an approximation problem in Appendix E.6. The method rescales each component to unit size. An error that looks small before rescaling can become large for a tiny component. The paper needs to show that these amplified errors do not undermine its bound on effective rank. Repairing the main expansion alone would not resolve this issue.

  3. The analysis assumes key ingredients of the three stages without showing that training produces them. The results rely on conditions such as weight matrices using few independent directions, a particular shift toward a more spread-out structure, and a period of small training updates. Some mathematical conclusions are valid if these conditions hold. The missing step is showing that they arise in the studied training process and explain the observed pattern. Even the assumed small weight ranks do not automatically make the proposed bound useful.

  4. The experiments also use different attention rules and optimizers from the simplified theory (Table 2). They show the pattern, but do not directly test its proposed cause. Measurements of the assumed ranks, update sizes, and changes in individual component similarities would make the connection much stronger. The aggregate statistics in Appendix B do not supply those checks.

Questions for authors
  1. What assumptions about the data and initialization make the independence claim valid and rule out repeated-input examples? How should Equation 44 be corrected? Please distinguish proving independence at the start, proving that it lasts for some interval, and proving a time by which it must be lost.

2.Can you justify the approximation across the whole dataset and bound its error after rescaling each component? What prevents errors in very small components from becoming large, and when does the resulting effective-rank bound meaningfully restrict the model?

3.Do the experiments actually satisfy the conditions used to explain the three stages? Please provide available measurements and separate experiments that match the theory from the other configurations. Checks across random seeds and learning-rate schedules would also help show whether the proposed mechanism explains the pattern.

1 comment

But repeating the same input many times can satisfy the stated size condition without adding any independent directions. (Q1)

We thank the reviewer for pointing this out. {Xk}\{X_k\} is actually pairwise distinct, we clarified it in the revision (See Setup).

We also revised Proposition 2.3, its proof, and Assumption 2.1 to make it more rigorous.

A later proof assumes that the largest eigenvalue of a Gram matrix—the matrix used to compare component directions—is negative. (Q1)

We thank the reviewer for identifying this issue. This is a typo: it should refer to the largest eigenvalue of the derivative of the Gram matrix.

Actually, the proof concerns the unnormalized Gram matrix. Since a zero determinant of the unnormalized Gram matrix also implies linear dependence among the basis functions, the same conclusion holds for the normalized Gram matrix, as normalization does not change its rank. We revised the proof to clarify this point in Appendix E.3.

An approximation problem in Appendix E.6. (Q2)

We thank the reviewer for pointing this out. We agree that a condition on the scalar α\alpha is needed to control the approximation error after normalization. We discribe it in the theorem by revising the Theorem 2.2 and its proof (Appendix E.6.).

Details of experiments for the conditions. (Q3)

We provide additional experimental details in the appendix to demonstrate how the conditions required by the theoretical results are verified in practice in Appnedix D.1. Here are the clarifications for each assumption and theorem:

  • Assumption 2.1: The linear independence of the basis functions is verified by showing that the minimum eigenvalue of the Gram matrix is strictly positive (Figure 2 and Figure 3).
  • Assumption 2.2: Figure 2 and Figure 3 show that the initial effective rank is high, which implies ϵ0≪1\epsilon_0 \ll 1 based on Eq.15.
  • Theorem 2.1: We plot the absolute mean values of the parameters associated with the attention and embedding layers (Figure 10.a and Figure 10.b).
  • Assumption 2.3: We plot the evolution of Nr(t)N_r(t) during training (Figure 10.d and Figure 10.e).
  • Theorem 2.2: We plot (ησ2+σ4)/αi(\eta\sigma^2+\sigma^4)/\alpha_i to verify the magnitude of the approximation error after normalization (Figure 10.c).
  • Assumption 2.4: The gradient norm of the loss is reflected in the corresponding loss curves (Figure 9).
  • Theorem 2.3: The required conditions are naturally satisfied in our experiments (Figure 2 and Figure 3): the effective rank increases during the dilution stage, and the training successfully learns the target task.
  • Assumption 2.5: We plot the mean values of the parameters associated with the embedding layers (Figure 10.b).

Experiments match the theorem (Q3)

We conducted additional experiments using models and optimizers that are consistent with the theoretical settings. The corresponding results are reported in the main paper (Figure 2) and appendix (Table 2).

Notably, to strictly align the experimental setting with the implicit low-rank bias considered in our theoretical analysis, we removed the SiLU activation from the MLP layer (See Eq.1) and updated the relevant proof (See Appendix E.6).

Random seeds and learning rate. (Q3)

We conducted experiments with two different random seeds and two different learning rates. The corresponding results are reported in Appendix C.3.

Reviewer_Trf2OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper writes each layer's output as a sum of "basis functions". Examples are the rank-one terms a_i w_{O,i}^T in attention, g_i w_{down,i}^T in the MLP, and the residual. It then tracks the normalized Gram matrix of these bases during training, using effective rank r_eff, λ_max, λ_min and mean off-diagonal magnitude ρ. From a gradient-flow analysis of a single-layer Transformer, it argues for three stages:

  1. Focus: r_eff drops.
  2. Dilution: r_eff rises.
  3. Stabilization: r_eff plateaus while the loss keeps improving.

Experiments use a single-layer model on synthetic Markov data, plus LLaMA2-style and Qwen3-style models on WikiText-2.

Critical strengths
  • The idea of tracking a Gram matrix of per-unit output contributions is simple and cheap to compute. It could be useful as a diagnostic.
  • The paper is honest that monotonicity is not proven. It also states its scale limitation.
  • The appendix proofs are mostly easy to follow. The calculus in Prop. 2.6 and Lemma E.1 appears correct.
  • The qualitative curves (a drop, a rise, then a plateau in r_eff) recur across the three architectures shown.
Critical weaknesses

Theory–experiment mismatch

  • The theory analyzes gradient flow, but the experiments use Adam/AdamW with a cosine schedule. This breaks the O(σ⁴) versus O(σ²) scale hierarchy in Theorem 2.1.
  • The theory assumes no causal mask, no RoPE, no Q/K-norm, a single head and last-position loss. The experiments use all of these features (causal mask, RoPE, multiple heads, RMSNorm), and the single-layer model even has gated attention.
  • Embeddings are treated as fixed in the theory but are trained in practice.
  • The multi-layer figures don't say which layer's Gram matrix is plotted.

Circular or weak theoretical results

  • Definition 2.1 defines t₁* as the time during which weights stay O(σ), so Theorem 2.1 is close to circular.
  • Theorem 2.2 depends on Assumption 2.3 (approximately low-rank weights), which is exactly what needs explaining and is never tested. The "≲" bound also ignores how sensitive entropy-based rank is to small eigenvalues.
  • Theorem 2.3 builds its conclusion into its hypothesis through an unobservable target p⋆. The result then follows directly from Schur-concavity, and the claim that practice satisfies Eq. 21 is unsupported.
  • Prop. 2.5 defines stabilization as a small gradient, so a bounded ṙ_eff is nearly trivial. It also contradicts the claim that parameters keep evolving, and the N^{3/2} constant makes the bound vacuous at experimental scale.
  • Prop. 2.6 and Theorem 2.4 are algebraic identities that don't explain why stabilization occurs.
  • The proof of Prop. 2.3 has gaps. It doesn't show det M is not identically zero, and the upper-bound argument for T is garbled.
  • Assumption 2.2 is supported only by small mean |M_ij| (ρ), which doesn't imply a small spectral norm ‖E₀‖₂.

Overstated claims

  • The bound predicts a very small r_eff, but the observed drop is moderate (thousands remain), so "condenses onto a low-dimensional structure" is overstated.
  • The Fig. 1 caption describes the large eigenvalues decreasing, which doesn't clearly match "activating previously insignificant directions".
  • The claim of applicability to deeper and larger models is unsupported and conflicts with the stated limitations.
  • "Rigorous" is too strong given the number of assumptions.

Metric interpretation

  • The Gram matrix depends on a parameterization-induced basis (it changes under a hidden-unit rotation), and r_eff is not shown to measure representation dimension.
  • Normalization discards scale information.
  • There are no controls (random bases, hidden-state rank, weight stable rank), and no quantitative comparison to prior work (Li et al. 2026, Chen et al. 2026).

Thin experimental evidence

  • There are no seeds, error bars or variance, and stage boundaries are identified by eye.
  • None of the theory's quantities (t₁*, t₂*, ε, N_r(t), p⋆) are measured.
  • The runs are short and small. D, numerical precision (bf16 with eigenvalues near 10⁻³) and warmup are not specified.
  • The cosine learning-rate decay is an uncontrolled confound for the stabilization stage.
  • The Markov datasets S1–S5 could test the claim that dilution is driven by data complexity, but no quantitative analysis is reported.
  • Stabilization evidence (E1–E3) is limited and unclear, with no direct measurement of parameter or function change.

Presentation

  • Many equations are broken (Eqs. 3, 7, 12, 30–32, 44–47), and notation is inconsistent (e.g., 3+d_attn+d_h versus N=1+d_attn+d_h).
  • The AI Use Statement has unfilled template brackets.
  • The Markov-dataset citation (MasakhaNER) is irrelevant.
  • Model sizes are given in MB rather than parameter counts, and the use of the llama-7b-hf tokenizer for Qwen3 is unexplained.
  • Parts of Appendix B appear misplaced.
Questions for authors
  1. Which layer's Gram matrix is shown for the multi-layer models? Do other layers (Fig. 4) show the same three stages with equal clarity?
  2. Do the stages persist with SGD or with a constant learning rate? How are they affected by warmup and by the cosine decay?
  3. Can you measure N_r(t) and the weight singular spectra to verify Assumption 2.3, and check whether r_eff tracks them?
  4. Is p⋆ operationalized anywhere? How would one test Eq. 21?
  5. How sensitive are the results to D, to precision, and to the choice of basis (e.g., a random rotation of attention-head outputs)?
3 comments

Critical Weaknesses

Most of the listed weaknesses can be addressed by using a proper PDF reader. For example, the issues regarding (1) broken equations: they are not broken and (2) the alleged circular reasoning in Theorem 2.1: it is the derivative rather than the value. For the remaining weaknesses, please refer to our responses in the Overall Comment.

Questions

Q1

We evaluate the main part of LLM (after embedding layer and before LM head), so the Gram matrix is constructed from the outputs of the last layer in multi-layer models. As clearly demonstrated in Figure 4 (Figure 5 in the revised version), the same three-stage phenomenon is consistently observed on other layers .

Q2

We provide additional experiments using a model aligned with the theoretical model and optimized with full-batch gradient descent. The results reported in Figure 2 of the revised version demonstrate the consistency of the three-stage phenomenon under this setting.

Q3

We provide detailed empirical verification of NrN_r and the other conditions and assumptions in Appendix D.1.

Q4

Yes. One can directly apply PCA or other methods to identify the dominant and dominated directions in the dataset and use them to approximate p∗p^*.

Q5

We further provide experimental results with different random seeds and learning rates in Appendix C.3, demonstrating the consistency of the three-stage phenomenon across different experimental settings.

Finally, we do not manually choose the basis functions. Instead, they are determined by the architecture and parameters of the model. (Please see the paragraph on Line 134 in the revised version)

I thank the authors for their response. I'll keep my current scores.

Thank you for your reply. If there are any remaining concerns that prevent you from giving a positive score, we would be more than happy to address them.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall Comment

We would like to thank all reviewers for their constructive comments and suggestions. We have highlighted all revisions in blue in the revised manuscript. The major revisions can be summarized in two parts:

For theoretical analysis

In response to the reviewers' concerns, we have revised Assumption 2.3, Proposition 2.3, Theorem 2.2, and their corresponding proofs to provide a more rigorous theoretical analysis.

For experiments

The reviewers' concerns mainly focus on three aspects: (1) the gap between the theoretical and experimental models, (2) the generalizability of the three-stage phenomenon, and (3) the empirical verification of whether the model satisfies the assumptions and conditions required by the theoretical analysis.

1) Aligning the theoretical and experimental models.

We further simplified the model analyzed in Eq. (11) to better capture the implicit low-rank bias. We then conducted additional experiments using a model whose architecture strictly matches the theoretical model in Eq. (11) and which is optimized using full-batch gradient descent. The results are reported in Figure 2, revealing that the observed three-stage phenomenon is consistent with the theoretical model.

2) Verifying the generalizability of the three-stage phenomenon.

We conducted additional experiments with different random seeds and learning rates to demonstrate that the observed three-stage phenomenon is consistent across different experimental settings.

3) Verifying the theoretical assumptions and conditions.

Importantly, the assumptions are derived from our theoretical analysis rather than introduced based on empirical observations.

We then examine their validity in the experiments to establish the connection between the theoretical conditions and the observed training dynamics. Specifically, we provide further details of the theoretical model and its correspondence to the experimental setup in Appendix D.1, demonstrating that the model satisfies the assumptions and conditions required by our theoretical analysis.

Overall, these revisions provide a more rigorous theoretical foundation, a closer alignment between the theoretical and experimental models, and further evidence for the robustness of the observed three-stage phenomenon.

Dear AC,

We kindly ask to additionally encourage reviewers to interact with us during the discussion period. It is our belief that we addressed concerns and we would like to receive additional comments or reevaluation of the score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.