acceptodds
Under review as a conference paper at ICLR 2027

A prism hierarchy of learning regimes in large linear autoencoders

Abstract

Theoretical studies of machine learning models commonly consider different limiting regimes in which the learning dynamics of gradient descent becomes theoretically tractable. It is, however, desirable to have a systematically obtained picture of qualitatively different extreme learning regimes for a particular type of models. In this paper we propose such a picture for large weight-tied linear autoencoders characterized by input and latent dimensions, initialization magnitude, and training set size. This model is nonlinear in the weights and its gradient flow does not have a general theoretical solution. We show that at the level of the formal loss-expansion hierarchy, its extreme regimes are naturally associated with faces of a triangular prism. In particular, there are five basic extreme regimes associated with the 2-faces of the prism: (1) large-data, (2) small-data, (3) mean-field, (4) narrow-latent, and (5) free. For regimes (1,2,3,4), we derive and rigorously characterize the limiting train and population dynamics under gradient flow, rigorously prove convergence to these limits, and get good agreement with experimental results. The resulting limits reveal spectral critical slowing, initialization-driven trapping away from the PCA optimum, and regimes in which population loss converges exponentially while training loss relaxes only algebraically.

Primary area:
learning theory
Keywords:
Linear networks, autoencoders, regime classification

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Bk8vOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper studies gradient-flow dynamics in large weight-tied linear autoencoders under joint scaling of input dimension, latent width, training-set size, and initialization variance. It defines learning regimes through the leading monomial supports of the Taylor coefficients of averaged training and population losses. The resulting formal phase diagram has the geometry of a triangular prism, with five distinguished extreme regimes. The authors derive limiting dynamics and finite-time convergence results for four regimes and compare several predictions with experiments on CIFAR-10 subsets. They also analyze initialization-dependent alignment, trapping in the limiting system, and different asymptotic relaxation rates for training and population losses.

Critical strengths

The formal classification gives a systematic account of how different parameter scalings affect the dominant terms in the loss expansions. The leading-support construction is mathematically explicit and avoids relying solely on a collection of separately chosen limits. The convergence results for four extreme regimes add substance to the classification.

Critical weaknesses

The main limitation is the gap between the formal phase diagram and the actual learning dynamics. Theorem 3.1 classifies dominant monomials in Taylor coefficients at initialization. It does not establish an exhaustive classification of limiting trajectories. The paper explicitly acknowledges that the relevant formal expansions need not converge and that coefficient-wise limits need not commute with summation. Consequently, the prism geometry alone cannot establish that the identified phases exhaust the distinct dynamical behaviors of the model. This substantially limits the scope of the central classification claim.

The long-time conclusions face a separate difficulty. The convergence theorems control the high-dimensional dynamics on fixed time intervals, whereas the trapping transition and relaxation-rate separation are obtained by taking the long-time limit of the resulting effective system. No quantitative control is provided for the joint large-dimension and long-time limit. In particular, the finite-time convergence theorem does not establish that finite-dimensional trajectories remain trapped as training time tends to infinity. The free regime also remains dynamically unsolved. The numerical comparisons support particular limiting predictions but cannot resolve these gaps in the claimed classification.

Questions for authors
  1. Beyond the leading-support classification, what result establishes that distinct formal phases necessarily correspond to distinct limiting learning trajectories? If no such result is available, what precise completeness claim remains justified?

  2. Can the authors provide a quantitative bound on the time interval over which finite-dimensional trajectories follow the trapped branch of the narrow-latent limiting system? Does the trapping behavior persist when dimension and training time increase jointly?

  3. What dynamical result, beyond the dominant-balance argument, establishes the free regime as a distinct learning mechanism? How does leaving this regime unsolved affect the interpretation of the five-regime classification?

9 comments

We thank the anonymous reviewer for their valuable comments! Below are our response the specific weaknesses noted and questions raised.

The main limitation is the gap between the formal phase diagram and the actual learning dynamics... & Q1:

  1. We certainly don't claim exhaustiveness/completeness of our phase diagram. This is one of the reasons why we use the term signature in the paper: different signatures suggest different regimes, but not vice versa.
  2. We believe the question of completeness to be quite subtle and be a matter of debate. One may or may not view some properties of the training dynamics as signifying a separate regime. (As an example, take convergence to either zero or strictly positive final loss.) Any claim of completeness faces a risk of someone proposing a new property subdividing the regimes.
  3. Rather, our goal is any reasonable regime classification which is systematic and produces reasonable results, consistent with common sense, past research and, where available, alternative derivations.
  4. By systematic we mean that the phase diagram should result from general rules uniformly applicable to general scaling vectors. It must not be based on arbitrary decisions like "let us consider the regime in which m≲nm\lesssim n". The regimes along with their boundaries should be derived from general principles applied to the model in question.
  5. In the present work we derive in such a systematic way a phase diagram which we find quite reasonable, in this sense. It has a natural hierarchical structure; several of its regimes have a natural interpretation consistent with past research ("free", "mean-field", etc.); most of our extreme regimes admit an analytic solution of a particular type; these types are different for different regimes.
  6. Our regime classification based on Taylor coefficients and monomial signatures is systematically and uniformly applied to all scaling vectors defining possible mutual power law relations between the hyperparameters. In contrast, we are not aware of any equally systematic and uniformly applicable method to perform the macroscopic limit on the level of loss trajectories. The limits that we are aware of are performed on a case-by-case basis.

The free regime also remains dynamically unsolved...:

As for the missing free regime, the simplest reduction we managed to get in the appropriate (p,n,m,σ)(p,n,m,\sigma) limit is a non-commutative ODE. We are aware neither of appropriate mathematical tools amenable for such non-commutative ODEs, nor whether further reduction (as in the four regimes we studied) is available.

Q3: What dynamical result, beyond the dominant-balance argument, establishes the free regime as a distinct learning mechanism?

We did not established a limit dynamics for the free regime as for the others, to certify that the resulting dynamics is qualitatively different, therefore, indeed, deserves to be called a regime. However, the free regime is the only one that ignores the target: for the other four regimes, the target does appear in the limit dynamics.

Q2: Can the authors provide a quantitative bound on the time interval over which finite-dimensional trajectories follow the trapped branch of the narrow-latent limiting system? Does the trapping behavior persist when dimension and training time increase jointly?

We thank the reviewer for pointing out this distinction. We agree that uniform convergence on fixed time intervals does not, by itself, justify transferring the long-time trapping behavior of the limiting system to finite-dimensional gradient flow. We have addressed this issue by deriving a quantitative extension of Theorem 4.3 to time intervals growing logarithmically with dimension, under isotropic Gaussian training data. In particular, this establishes that the strict trapping behavior persists in an appropriate joint large-dimension, long-time limit.

Quantitative time horizon. As in the paper, define Sp=XX⊤/m\mathbf S_p=\mathbf X\mathbf X^\top/m, Gp(t)=Up(t)Up(t)⊤\mathbf G_p(t)=\mathbf U_p(t)\mathbf U_p(t)^\top, and Hp(t)=Up(t)SpUp(t)⊤\mathbf H_p(t)=\mathbf U_p(t) \mathbf S_p \mathbf U_p(t)^\top. Consider the narrow-latent regime n/p→0n / p \to 0, p/m=ϕ>1p/m = \phi > 1, and pσ2=κp \sigma^2 = \kappa. Let νϕ\nu_\phi be the Marchenko–Pastur law, and choose a deterministic sequence bp→0b_p\to0 such that it bounds the Kolmogorov distance between the empirical spectral measure μS\mu_S and the limit measure νϕ\nu_\phi:

= O_{\mathbb P}(b_p), \qquad d_{\rm K}(\mu,\nu) = \sup_x|\mu(({-}\infty,x])-\nu(({-}\infty,x])|.$$ Define $\Delta_p = \sqrt{\frac{n+\ln p}{p}} + b_p.$ Our quantitative extension establishes that there exist dimension-independent constants $c,C>0$ such that, for every deterministic $T_p\le c\ln(1/\Delta_p)$, $$E_p := \sup_{0\le t\le T_p} (|\mathbf G_p(t)-a(t) \mathbf I_n|_{\mathrm{op}} + |\mathbf H_p(t)-r(t) \mathbf I_n|_{\mathrm{op}}) = O_{\mathbb P}(e^{CT_p}\Delta_p),$$ where $a(t),r(t)$ are the limiting quantities of Theorem 4.3. In particular, choosing $T_p=c_0\ln(1/\Delta_p)$ with $c_0>0$ sufficiently small yields an approximation error of order $O_{\mathbb P}(\Delta_p^{1-Cc_0})=o_{\mathbb P}(1)$. Thus the approximation remains valid over a time horizon that diverges with dimension. When the dimensional ratios and parameters converge at polynomial rates, this gives $T_p$ proportional to $\ln p$. **Persistence of trapping in the joint limit.** Consider the strict trapping phase identified in Proposition 4.4: $\kappa>2$, $\frac{\kappa^2}{\kappa-1} \left(1-\frac1\phi\right)^2>4.$ In this phase, the limiting system satisfies $a(t)\downarrow a_*>2$, $r(t)\to0$. Let $\delta=a_*-2>0$. Our quantitative estimate gives $E_p=o_{\mathbb P}(1)$ as long as $T_p=c_0\ln(1/\Delta_p)$ with $c_0>0$ sufficiently small. Since $a(t)\ge a_*$ for all $t\ge0$, Weyl's inequality yields $$\inf_{0\le t\le T_p}\lambda_{\min}(\mathbf G_p(t)) \ge a_*-E_p =2+\delta-o_{\mathbb P}(1).$$ Thus, with probability tending to one, the Gram matrix remains uniformly above $2 \mathbf I_n$ throughout the logarithmically growing time interval. Furthermore, for any $s_p\to\infty$ with $s_p\le T_p$, the convergence $r(t)\to0$ implies $$\sup_{s_p\le t\le T_p}\|\mathbf H_p(t)\|_{\mathrm{op}} \xrightarrow{\mathbb P}0.$$ For Gaussian data with $\phi>1$, the positive spectrum of $\mathbf S_p$ is uniformly separated from zero. Denoting by $\mathbf P_p$ the projector onto the range of $\mathbf S_p$, we consequently obtain $$\sup_{s_p\le t\le T_p}\|\mathbf U_p(t) \mathbf P_p\|_{\mathrm{op}} \xrightarrow{\mathbb P}0.$$ In particular, the null-space projection $\mathbf V_p(t)= \mathbf U_p(t) (\mathbf I - \mathbf P_p)$ satisfies $\mathbf V_p(t) \mathbf S_p = 0$ and $\mathbf V_p(t) \mathbf V_p(t)^\top \succ 2 \mathbf I_n$ with high probability throughout this late-time interval. By Proposition C.29, these projected configurations are nonglobal local minima of the finite-dimensional empirical loss. Hence the finite-dimensional trajectory approaches the manifold of nonglobal local minima uniformly over a diverging time window. This establishes the persistence of strict trapping in the joint large-dimension, long-time limit, without asserting permanent trapping at each fixed finite dimension. **The loss conclusions also persist.** Writing $\psi=p/n$, we obtain $$\psi\left(L(t_p)-\frac12\right) \xrightarrow{\mathbb P} a_*\left(\frac{a_*}{2}-1\right), \qquad \psi\left( \widehat L(t_p)-\frac{\operatorname{tr}\mathbf S_p}{2p} \right)\xrightarrow{\mathbb P}0.$$ In particular, the limiting population-loss correction remains strictly separated from the PCA-optimal value. **Proof approach.** The extension uses a quantitative, growing-time version of the conditional Gaussian Gram concentration argument in Lemma C.19, combined with quantitative spectral-transform convergence and the deterministic stability estimates of Theorem C.20 and Proposition C.23. The resulting errors grow at most exponentially in the time horizon, which yields the logarithmic bound above.

Thank you for the response. On completeness, I agree that lines 146-147 and 480-482 already note that the signature diagram is not meant to exhaust trajectory-level regimes, and I do not hold the paper to that standard. My concern is with a specific overstatement in the framing: lines 055-057 claim that "the limiting analyses show that these regimes correspond to genuinely different learning dynamics", while the abstract presents five basic extreme regimes. As your reply confirms, no limiting trajectory dynamics has been derived for the free regime. Showing that its leading Taylor coefficients omit the target is a property of the formal expansion at initialization, not of the trajectories. Please revise the abstract and introduction to state that dynamical distinctness is established for four facets, while the fifth is identified only at the formal expansion level.

More broadly, describing the goal as a classification that is systematic and consistent with common sense and prior work does not provide a testable criterion. Within the paper's own framework, a concrete test is whether signature differences predict dynamical behavior beyond the four solved facets: for instance, whether an intermediate edge or face exhibits trajectory-level behavior distinct from its adjacent facets, or whether two different signatures can collapse to the same limiting trajectory after a time reparameterization. Without such a check, the prism classifies which monomials dominate the initialization Taylor series and should be presented in those terms.

On Q2, the logarithmic-horizon extension goes in the right direction and would strengthen Section 4.3 once verified. However, the forum PDF is still the initial submission, and I cannot properly verify a multi-step proof from a comment sketch alone. If you upload a revised PDF before November 18, I will read the full proof. That revision should also make three scope conditions explicit: (1) since the error grows as eCTe^{CT}, trapping is proved on logarithmically growing windows clog⁡(1/Δp)c\log(1/\Delta_p) rather than indefinitely (matching Reviewer VAbQ's concern), which should be stated in the abstract and Section 4.3; (2) the extension assumes isotropic Gaussian training data, whereas Figure 2 and most panels of Figure 3 use whitened CIFAR-10 subsets, which are isotropic but non-Gaussian; and (3) trapping requires ϕ=p/m>1\phi = p/m > 1 (lines 411-413), i.e., a macroscopic null space of the empirical covariance, which belongs in the abstract alongside the trapping claim. On the technical side, since your argument brings the trajectory within o(1)o(1) of the nonglobal local minima in Appendix C.3.7, is a shrinking basin radius with pp the only obstruction to permanent trapping at fixed large pp, or does another step break down?

I am keeping my current score for now, since the forum manuscript has not yet changed, and will revisit soundness if the complete proof and scope revisions appear in an updated PDF.

Reviewer_VAbQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the training dynamics of weight-tied linear autoencoders under different scalings of input dimensions, latent dimensions, initialization scale, and training set size. First, they identify 5 extreme regimes of this problem and characterize the intermediate phases using the coefficient-expansion framework introudecd by Yarotsky et al. (2026). Then, they analyze the dynamics of 4 of the 5 extreme regimes.

Critical strengths
  • The authors do provide a somewhat complete characterization for the problem studied.
  • The classification of different regimes looks interesting and principled.
Critical weaknesses
  • Linear autoencoders are not a particularly interesting model.
  • The arguments in Section 4 and Appendix C (limiting solutions in extreme regimes) are, in some sense, unnecessarily complicated. Basically all those arguments are built upon finding some idealized solution with relatively clean dynamics, analyzing those dynamics, and then controlling the difference between the true and idealized dynamics. The first part leads to interesting observations, but the second part is basically brute-force estimation + Gronwall's lemma and all those approximation error bounds grow exponentially with time. I don't find this part interesting or particularly meaningful. This paper would be much shorter and more readable if the authors focused solely on those idealized dynamics.
Questions for authors

Can you respond to my second point in the Weaknesses section? In particular, what interesting insights can we gain from the analysis of the difference between the idealized and actual dynamics?

3 comments

We thank the anonymous reviewer for their valuable feedback! Below we the address the weaknesses raised.

Linear autoencoders are not a particularly interesting model.

We agree, but already this simple model (1) reveals interesting phenomena such as gradient flow trapping in a non-PCA local minimum, (2) exhibits rich regime classification, (3) cannot be easily solved for general (p,n,m)(p,n,m). To be precise, the dynamics in the proportional limit (the one corresponding to the prism as a whole) reduces to a non-commutative ODE, while we couldn't get any further reduction. We are not aware of mathematical tools amenable for analyzing such objects; neither we are aware if further reduction of the limit dynamics is possible.

The arguments in Section 4 and Appendix C (limiting solutions in extreme regimes) are, in some sense, unnecessarily complicated...

We agree that focusing on limit trajectories and their properties, while moving all of the formal convergence theorems to Appendix, would simplify the exposition. On the other hand, we suspect that a certain part of ICLR audience might argue that mere limit trajectory analysis without formal convergence results would make the whole analysis non-rigorous and mathematically incomplete. Our exposition choice was a trade-off: brief but exact formal statements in the main, with more detailed formal statements moved to Appendix and the rest of Section 4 devoted to qualitative limit trajectory analysis.

I thank the authors for the clarification and will maintain my positive score.

To be honest, I don't think the part of ICLR community that cares formal results would find an exponential error growth rate satisfactory, especially when the relevant training time can be much larger than the inverse of the coefficient in the exponent. That being said, I understand this is subjective and my personal opinion doesn't necessarily reflect that of the broader ICLR theory community

We would like to draw the reviewer's attention to a new result in the revised manuscript, which directly addresses the question of whether the convergence analysis provides meaningful conclusions beyond finite-time approximation estimates.

In the new Appendix C.6, we establish a dimension-independent attracting neighborhood of the nonglobal local minima associated with strict trapping in the narrow-latent regime (Lemma C.43). Combining this stability result with the original finite-time convergence theorem, Corollary C.44 proves permanent finite-dimensional trapping with probability tending to one as p→∞p\to\infty. Specifically, the weights Up(t)\mathbf U_p(t) converge as t→∞t\to\infty to a nonglobal local minimum, provided the nonzero spectrum of the empirical covariance has a uniform positive gap. This condition holds, in particular, for independent isotropic Gaussian training data with ϕ=p/m>1\phi=p/m>1.

Moreover, the corollary establishes uniform-in-time convergence of the Gram observables and scaled losses on the entire interval [0,∞)[0,\infty) in the strict trapped phase.

Thus, the convergence analysis does more than justify the idealized dynamics on finite time intervals: it allows us to establish a qualitative property of the actual finite-dimensional gradient flow, namely permanent trapping away from the PCA optimum.

We refer the reviewer to the revised PDF (Appendix C.6) and our latest response to Reviewer Bk8v for further details.

Reviewer_aGP9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the learning dynamics of gradient flow for weight-tied linear autoencoders when the input dimension, latent dimension, sample size, and initialization scale are jointly scaled. In particular, the authors identify five basic learning regimes (large-data, small-data, mean-field, narrow-latent, and free), which are associated with the five two-dimensional faces of a triangular prism. For all regimes except for the free regime, they also derive the limiting train and population dynamics and prove convergence to these limits on fixed finite time intervals.

Critical strengths

The idea of identifying learning regimes through the leading-support signatures seems novel and interesting. It is also interesting that the limiting dynamics show qualitatively different behaviors, such as trapping away from the PCA solution (Proposition 4.4) and different convergence rates of the train and population losses (Eq. (20)).

Critical weaknesses
  • First, the practical significance of the results is unclear. The paper only considers weight-tied linear autoencoders and it does not discuss how the results could help understand or train more complex models, such as nonlinear networks. It would be helpful to discuss whether the proposed classification of regimes can be extended beyond the linear setting, and whether the results can give any practical guidance for training linear autoencoders, such as the choice of the initialization scale or latent dimension.

  • Second, the limiting dynamics are derived for four of the five basic regimes while the phase diagram also includes the free regime and combined regimes (e.g., large-data + mean-field regimes). The learning dynamics in these regimes are still unclear and may require separate analyses. It would be more interesting to see whether it is possible to develop a unified analysis that covers different regimes rather than a separate analysis for each regime.

Questions for authors
  • In lines 151-152, what are the expectations in Ftr(t)F_{tr}(t) and Fpop(t)F_{pop}(t) taken over?

  • In Section 3, the formal phases are defined by the leading-support signatures. It would be helpful if the authors could further clarify the intuition behind this definition. Also, since there may be other ways to classify the learning regimes, why did the authors choose this particular definition?

2 comments

We thank the anonymous reviewer for their valuable feedback. We address the questions and the weaknesses raised below.

Weaknesses

The practical significance of the results is unclear...

First, the monomial-based classification approach we adopt extends naturally to neural networks of any depth, with or without weight sharing, with polynomial activation functions — we leave the corresponding classification and trajectory analysis for future work.

Second, the most practically-relevant regime is the proportional one, which corresponds to the prism as a whole. The faces correspond to first-order degeneracies; studying these degenerate regimes reveal properties one might expect from the proportional regime when certain hyperparameters (like ϕ\phi or ψ\psi) become small of large. In particular, the narrow-latent regime communicates an important warning — gradient flow might not converge to a PCA solution for autoencoders with a bottleneck (Proposition 4.4). Investigating how narrow the autoencoder should be for gradient flow not to converge to the PCA optimum, and whether it stays true for ReLU autoencoders, requires a separate analysis.

The limiting dynamics are derived for four of the five basic regimes...

One gets solutions for combined regimes (i.e. edges or vertices of the prism) simply by taking a limit of the corresponding limit trajectory: for example, one gets large-data+mean-field by taking ϕ→0\phi \to 0 in Eq. (10) (Theorem 4.1), or, alternatively, by taking ψ→0\psi \to 0 in Eq. (13) (Theorem 4.2). The two resulting formulas are consistent for any finite tt. We do not expect any qualitatively new phenomena for these more degenerate combined regimes; this is the reason why we do not study them explicitly in our work.

As for the missing free regime, the simplest reduction we managed to get in the appropriate (p,n,m,σ)(p,n,m,\sigma) limit is a non-commutative ODE. We are not aware if the corresponding dynamics can be meaningfully reduced any further. In fact, the proportional regime (the one corresponding to the prism as a whole) also reduces to a non-commutative ODE in the appropriate large-dimensional limit. Studying the dynamics of this ODE would correspond to a unified analysis you propose, but we are aware neither of appropriate mathematical tools amenable for such non-commutative ODEs, nor whether further reduction (as in the four regimes we studied) is available.

Questions

What are the expectations in FtrF_{tr} and FpopF_{pop} taken over?

As everywhere in the main, expectations are taken wrt weight initialization U(0)\mathbf U(0) and training data X\mathbf X.

The formal phases are defined by the leading-support signatures...

This choice of criteria yields a systematic, relatively universal, analytically tractable regime classification. We are not aware of any other natural classification approach possessing these properties.

Intuitively, the approach relies on several ideas:

  1. Macroscopic limits of the loss trajectories are hard to define systematically for general hyperparameter scalings. Usually, this is done only on a case-by-case basis, with a separate known limiting regime in each case.
  2. At least formally, the loss trajectory is equivalently described by the sequence of its Taylor coefficients. For a broad class of models, these coefficents are polynomials in the hyperparameters - i.e., are relatively simple, finite-dimensional objects.
  3. In contrast to the limit of the whole trajectory, it is fairly clear how to define macroscopic leading terms of these polynomial Taylor coefficients.
  4. These leading terms are linear combinations of specific monomials. The respective sets of leading monomials change abruptly at some surfaces in the space of hyperparameter scalings.
  5. This creates a natural opportunity to define discrete classes by labeling all scaling vectors by their discrete "monomial set signatures" consisting of the sets of leading monomials for all Taylor coefficients.
  6. The approach can thus be viewed as systematically converting the original continuous setting into a discrete one, necessary for the regime classification.

I thank the authors for their response. The prism organizes the different ways the model and data can scale, but the learning dynamics are still analyzed case by case. The proportional regime, which the authors describe as the most practically relevant, is also still open. It is unclear to me what the phase diagram adds to the separate analyses or how the results can inform practical training. I will therefore keep my current score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.