- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies population gradient flow in a two-layer quadratic network (teacher–student, M = WWᵀ). It splits the dynamics into motion of the student subspace (feature formation) and growth of amplitudes within that subspace (feature expression). For isotropic Gaussian inputs, the subspace evolves exactly as the span of e^{tM⋆}U₀, independently of the amplitudes. Overlaps with teacher eigen-subspaces follow a replicator equation, so features form in order of the teacher eigenvalues. Each alignment time is set by an eigenvalue gap and does not depend on the initialization scale, whereas expression is delayed by log(1/ε). This gives a feature-by-feature account of silent alignment. For general inputs at small initialization, the same law holds with an effective teacher H⋆ reshaped by the input covariance and fourth cumulant. As for finite amplitude, when amplitudes grow, the student’s own predictions feed back on the directions. In exactly solvable cases, anisotropic covariance makes a later feature replace the first one, directional kurtosis produces a mixed feature, and spherical fourth moments make amplitudes compete or cooperate while preserving the direction path.
Critical strengths
The paper is solid, clearly written, and technically careful. The direction/amplitude decomposition is clean, and the replicator formulation makes the hierarchy of feature formation transparent and quantitative.
The structured-input analyses are the most interesting part. They show explicitly how anisotropy and fourth moments govern the feedback-induced reorganization of early features, through replacement, mixing, and competition or cooperation. These are nontrivial nonlinear effects that linear networks cannot capture.
The ranking of features by fitted amplitude λ, loss reduction fλ, and initial growth f²λ is elegant. It connects to empirical observations on feature availability versus predictivity.
The numerical checks are thorough: independent integration of the closures, finite-data and minibatch tests, and a held-out ranking benchmark.
Critical weaknesses
Novelty of the isotropic part. The isotropic matrix Riccati solution is already known (Martin et al. 2024; Ben Arous et al. 2025; Ziyin et al. 2026), and the projector flow is Brockett/Oja. The new content there is mainly the feature-resolved reading: replicator equation, monotone cumulative capture, and mode-specific silent alignment. Sections 4–5 contain the main novelty, and the paper should say so more explicitly.
High-dimensional regime. All results are at finite d. A discussion on how the main results and conclusions change when d>>1 is important.
Scope of the reorganization results. The exactly solvable examples are low-dimensional and assume teacher and data moments diagonal in the same basis. A more general criterion for when early features get reorganized would be valuable, for example built on the stability analysis of Prop. H.2, along with an example where the two bases are not aligned.
Questions for authors
none, see the weaknesses.
3 comments
Response to Reviewer cDiv (2/2): General reorganization
3. General reorganization and nonaligned bases
We agree that this is an important question. We revisited it and derived a general sufficient condition for Gaussian inputs with arbitrary positive-definite covariance, without assuming a shared eigenbasis.
Let project onto the leading eigenvectors of , the subspace selected during early learning, and let be the final subspace projector. For and , with gaps separating the leading eigenvalues in the early and final problems below, the condition for generic initialization is
\quad\Longrightarrow\quad P_\infty\ne P_{\mathrm e}.}$$ In words, if covariance mixes the early learned subspace with directions outside it, the student must change that subspace as its features become expressed. We can see this directly by writing an orthonormal basis for the early subspace as $U_{\mathrm e}$. At this subspace, the teacher drive has no perpendicular component. For any positive definite amplitude matrix $S$, the remaining direction equation is $$\dot U_{\mathrm e} =-(I-P_{\mathrm e})\Sigma U_{\mathrm e}\\, S\\,(U_{\mathrm e}^\top\Sigma U_{\mathrm e}).$$ The first factor, $(I-P_{\mathrm e})\Sigma U_{\mathrm e}$, is the part of the covariance action that points outside the early subspace. The two remaining factors are the amplitude matrix $S$ and the covariance within that subspace, $U_{\mathrm e}^\top\Sigma U_{\mathrm e}$. Both are positive definite and therefore invertible. Multiplying by them cannot turn a nonzero first factor into zero. Hence no stationary rank-$k$ student can retain the early subspace when the stated condition holds. We can also determine the subspace to which the student converges. If we express the student weights as $\Sigma^{1/2}W$, the Gaussian loss becomes ordinary squared matrix error in approximating $T=\Sigma^{1/2}M_\star\Sigma^{1/2}$ by a rank-$k$ matrix. The best fit keeps the largest $k$ eigenvalues of $T$ and their eigenvectors. Let $V_k$ contain those eigenvectors. Mapping back to the original weight coordinates then gives the final student subspace $\operatorname{span}(\Sigma^{-1/2}V_k)$.I thank the authors for the detailed reply, and the additional material. I keep my score.
Response to Reviewer cDiv (1/2): Novelty and high dimension
Thank you for the careful review and especially for your interest and questions on how data structure shapes feature reorganization. Your questions encouraged us to derive a more general reorganization criterion and work out how alignment and cooperation depend on dimension.
1. Novelty of the isotropic results
As you mention, the feature-resolved reading is central to our contribution and goes beyond the known solutions. The fine-grained feature analysis lets us distinguish several events that might otherwise be compressed into "learning a mode", especially when one follows only its contribution to the predictor. A direction can already be formed in the student representation while its amplitude, and hence its contribution to prediction, remains negligible.
We build on the known Riccati solution, as you identified, and classical projector flow to derive the exact joint-subspace overlap law (the replicator equation), monotone cumulative capture, and feature-specific formation and expression times. For a rank-k student, the acquisition of the first m≤k teacher directions has asymptotic rate
\qquad C_m(t)=\sum_{i\leq m}v_i^\top P(t)v_i,$$ under the simple spectral-gap and generic-overlap assumptions in Appendix C. Expression instead occurs on the familiar logarithmic timescale, $$t_m^{\mathrm{expr}}=\frac{\log(1/\varepsilon)}{2\lambda_m}+O(1).$$ Comparing this expression time with the feature formation timescale we identify gives a silent interval for each feature. We will emphasize our distinct contributions beyond the known equations more explicitly in the updated manuscript. **2. High-dimensional regime** The dynamical equations hold at every finite dimension. The quantities that can change with dimension are the spectral gaps, initial overlaps, and total amplitude. For example, consider a rank-one student with isotropic Gaussian inputs. Let the teacher have one leading eigenvalue $\lambda_1$, with all remaining eigenvalues equal to $\lambda_0\geq0$. As dimension grows, hold these eigenvalues and their gap $\Delta=\lambda_1-\lambda_0>0$ fixed. Write $p_0$ for the student's initial squared overlap with the leading teacher direction. To reach a fixed squared overlap $1-\eta>p_0$, the exact direction law gives $$t_{\mathrm{align}}(\eta)= \frac{\log((1-p_0)/p_0)+\log((1-\eta)/\eta)}{2\Delta}.$$ Suppose $p_0$ is of order $1/d$, as is typical for a random initial direction. Formation then takes $(\log d)/(2\Delta)+O(1)$. This longer alignment time under this scaling comes from starting farther from the target direction, even though the gap controlling the rate is unchanged. To determine whether this alignment remains silent, we also need the amplitude at that time. Let $s(t)$ be the student's amplitude and $\varepsilon=s(0)$ its initial value. The exact amplitude solution gives $s(t_{\mathrm{align}})=O(\varepsilon d^{\lambda_0/\Delta})$. Thus alignment occurs while the amplitude still vanishes if we let initialization decrease with dimension so that $$\varepsilon d^{\lambda_0/\Delta}\longrightarrow0.$$