- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper studies gradient-flow dynamics in large weight-tied linear autoencoders under joint scaling of input dimension, latent width, training-set size, and initialization variance. It defines learning regimes through the leading monomial supports of the Taylor coefficients of averaged training and population losses. The resulting formal phase diagram has the geometry of a triangular prism, with five distinguished extreme regimes. The authors derive limiting dynamics and finite-time convergence results for four regimes and compare several predictions with experiments on CIFAR-10 subsets. They also analyze initialization-dependent alignment, trapping in the limiting system, and different asymptotic relaxation rates for training and population losses.
Critical strengths
The formal classification gives a systematic account of how different parameter scalings affect the dominant terms in the loss expansions. The leading-support construction is mathematically explicit and avoids relying solely on a collection of separately chosen limits. The convergence results for four extreme regimes add substance to the classification.
Critical weaknesses
The main limitation is the gap between the formal phase diagram and the actual learning dynamics. Theorem 3.1 classifies dominant monomials in Taylor coefficients at initialization. It does not establish an exhaustive classification of limiting trajectories. The paper explicitly acknowledges that the relevant formal expansions need not converge and that coefficient-wise limits need not commute with summation. Consequently, the prism geometry alone cannot establish that the identified phases exhaust the distinct dynamical behaviors of the model. This substantially limits the scope of the central classification claim.
The long-time conclusions face a separate difficulty. The convergence theorems control the high-dimensional dynamics on fixed time intervals, whereas the trapping transition and relaxation-rate separation are obtained by taking the long-time limit of the resulting effective system. No quantitative control is provided for the joint large-dimension and long-time limit. In particular, the finite-time convergence theorem does not establish that finite-dimensional trajectories remain trapped as training time tends to infinity. The free regime also remains dynamically unsolved. The numerical comparisons support particular limiting predictions but cannot resolve these gaps in the claimed classification.
Questions for authors
-
Beyond the leading-support classification, what result establishes that distinct formal phases necessarily correspond to distinct limiting learning trajectories? If no such result is available, what precise completeness claim remains justified?
-
Can the authors provide a quantitative bound on the time interval over which finite-dimensional trajectories follow the trapped branch of the narrow-latent limiting system? Does the trapping behavior persist when dimension and training time increase jointly?
-
What dynamical result, beyond the dominant-balance argument, establishes the free regime as a distinct learning mechanism? How does leaving this regime unsolved affect the interpretation of the five-regime classification?
9 comments
Q2: Can the authors provide a quantitative bound on the time interval over which finite-dimensional trajectories follow the trapped branch of the narrow-latent limiting system? Does the trapping behavior persist when dimension and training time increase jointly?
We thank the reviewer for pointing out this distinction. We agree that uniform convergence on fixed time intervals does not, by itself, justify transferring the long-time trapping behavior of the limiting system to finite-dimensional gradient flow. We have addressed this issue by deriving a quantitative extension of Theorem 4.3 to time intervals growing logarithmically with dimension, under isotropic Gaussian training data. In particular, this establishes that the strict trapping behavior persists in an appropriate joint large-dimension, long-time limit.
Quantitative time horizon. As in the paper, define , , and . Consider the narrow-latent regime , , and . Let be the Marchenko–Pastur law, and choose a deterministic sequence such that it bounds the Kolmogorov distance between the empirical spectral measure and the limit measure :
= O_{\mathbb P}(b_p), \qquad d_{\rm K}(\mu,\nu) = \sup_x|\mu(({-}\infty,x])-\nu(({-}\infty,x])|.$$ Define $\Delta_p = \sqrt{\frac{n+\ln p}{p}} + b_p.$ Our quantitative extension establishes that there exist dimension-independent constants $c,C>0$ such that, for every deterministic $T_p\le c\ln(1/\Delta_p)$, $$E_p := \sup_{0\le t\le T_p} (|\mathbf G_p(t)-a(t) \mathbf I_n|_{\mathrm{op}} + |\mathbf H_p(t)-r(t) \mathbf I_n|_{\mathrm{op}}) = O_{\mathbb P}(e^{CT_p}\Delta_p),$$ where $a(t),r(t)$ are the limiting quantities of Theorem 4.3. In particular, choosing $T_p=c_0\ln(1/\Delta_p)$ with $c_0>0$ sufficiently small yields an approximation error of order $O_{\mathbb P}(\Delta_p^{1-Cc_0})=o_{\mathbb P}(1)$. Thus the approximation remains valid over a time horizon that diverges with dimension. When the dimensional ratios and parameters converge at polynomial rates, this gives $T_p$ proportional to $\ln p$. **Persistence of trapping in the joint limit.** Consider the strict trapping phase identified in Proposition 4.4: $\kappa>2$, $\frac{\kappa^2}{\kappa-1} \left(1-\frac1\phi\right)^2>4.$ In this phase, the limiting system satisfies $a(t)\downarrow a_*>2$, $r(t)\to0$. Let $\delta=a_*-2>0$. Our quantitative estimate gives $E_p=o_{\mathbb P}(1)$ as long as $T_p=c_0\ln(1/\Delta_p)$ with $c_0>0$ sufficiently small. Since $a(t)\ge a_*$ for all $t\ge0$, Weyl's inequality yields $$\inf_{0\le t\le T_p}\lambda_{\min}(\mathbf G_p(t)) \ge a_*-E_p =2+\delta-o_{\mathbb P}(1).$$ Thus, with probability tending to one, the Gram matrix remains uniformly above $2 \mathbf I_n$ throughout the logarithmically growing time interval. Furthermore, for any $s_p\to\infty$ with $s_p\le T_p$, the convergence $r(t)\to0$ implies $$\sup_{s_p\le t\le T_p}\|\mathbf H_p(t)\|_{\mathrm{op}} \xrightarrow{\mathbb P}0.$$ For Gaussian data with $\phi>1$, the positive spectrum of $\mathbf S_p$ is uniformly separated from zero. Denoting by $\mathbf P_p$ the projector onto the range of $\mathbf S_p$, we consequently obtain $$\sup_{s_p\le t\le T_p}\|\mathbf U_p(t) \mathbf P_p\|_{\mathrm{op}} \xrightarrow{\mathbb P}0.$$ In particular, the null-space projection $\mathbf V_p(t)= \mathbf U_p(t) (\mathbf I - \mathbf P_p)$ satisfies $\mathbf V_p(t) \mathbf S_p = 0$ and $\mathbf V_p(t) \mathbf V_p(t)^\top \succ 2 \mathbf I_n$ with high probability throughout this late-time interval. By Proposition C.29, these projected configurations are nonglobal local minima of the finite-dimensional empirical loss. Hence the finite-dimensional trajectory approaches the manifold of nonglobal local minima uniformly over a diverging time window. This establishes the persistence of strict trapping in the joint large-dimension, long-time limit, without asserting permanent trapping at each fixed finite dimension. **The loss conclusions also persist.** Writing $\psi=p/n$, we obtain $$\psi\left(L(t_p)-\frac12\right) \xrightarrow{\mathbb P} a_*\left(\frac{a_*}{2}-1\right), \qquad \psi\left( \widehat L(t_p)-\frac{\operatorname{tr}\mathbf S_p}{2p} \right)\xrightarrow{\mathbb P}0.$$ In particular, the limiting population-loss correction remains strictly separated from the PCA-optimal value. **Proof approach.** The extension uses a quantitative, growing-time version of the conditional Gaussian Gram concentration argument in Lemma C.19, combined with quantitative spectral-transform convergence and the deterministic stability estimates of Theorem C.20 and Proposition C.23. The resulting errors grow at most exponentially in the time horizon, which yields the logarithmic bound above.Thank you for the response. On completeness, I agree that lines 146-147 and 480-482 already note that the signature diagram is not meant to exhaust trajectory-level regimes, and I do not hold the paper to that standard. My concern is with a specific overstatement in the framing: lines 055-057 claim that "the limiting analyses show that these regimes correspond to genuinely different learning dynamics", while the abstract presents five basic extreme regimes. As your reply confirms, no limiting trajectory dynamics has been derived for the free regime. Showing that its leading Taylor coefficients omit the target is a property of the formal expansion at initialization, not of the trajectories. Please revise the abstract and introduction to state that dynamical distinctness is established for four facets, while the fifth is identified only at the formal expansion level.
More broadly, describing the goal as a classification that is systematic and consistent with common sense and prior work does not provide a testable criterion. Within the paper's own framework, a concrete test is whether signature differences predict dynamical behavior beyond the four solved facets: for instance, whether an intermediate edge or face exhibits trajectory-level behavior distinct from its adjacent facets, or whether two different signatures can collapse to the same limiting trajectory after a time reparameterization. Without such a check, the prism classifies which monomials dominate the initialization Taylor series and should be presented in those terms.
On Q2, the logarithmic-horizon extension goes in the right direction and would strengthen Section 4.3 once verified. However, the forum PDF is still the initial submission, and I cannot properly verify a multi-step proof from a comment sketch alone. If you upload a revised PDF before November 18, I will read the full proof. That revision should also make three scope conditions explicit: (1) since the error grows as , trapping is proved on logarithmically growing windows rather than indefinitely (matching Reviewer VAbQ's concern), which should be stated in the abstract and Section 4.3; (2) the extension assumes isotropic Gaussian training data, whereas Figure 2 and most panels of Figure 3 use whitened CIFAR-10 subsets, which are isotropic but non-Gaussian; and (3) trapping requires (lines 411-413), i.e., a macroscopic null space of the empirical covariance, which belongs in the abstract alongside the trapping claim. On the technical side, since your argument brings the trajectory within of the nonglobal local minima in Appendix C.3.7, is a shrinking basin radius with the only obstruction to permanent trapping at fixed large , or does another step break down?
I am keeping my current score for now, since the forum manuscript has not yet changed, and will revisit soundness if the complete proof and scope revisions appear in an updated PDF.
We thank the anonymous reviewer for their valuable comments! Below are our response the specific weaknesses noted and questions raised.
As for the missing free regime, the simplest reduction we managed to get in the appropriate (p,n,m,σ) limit is a non-commutative ODE. We are aware neither of appropriate mathematical tools amenable for such non-commutative ODEs, nor whether further reduction (as in the four regimes we studied) is available.
We did not established a limit dynamics for the free regime as for the others, to certify that the resulting dynamics is qualitatively different, therefore, indeed, deserves to be called a regime. However, the free regime is the only one that ignores the target: for the other four regimes, the target does appear in the limit dynamics.