acceptodds
Under review as a conference paper at ICLR 2027

Theory of Feature Formation, Expression and Reorganization in Quadratic Networks

Abstract

A neural network learns a task by forming useful features and learning how strongly to use them in its predictions. Classical exact mode solutions for deep linear networks describe how expression grows from weights already aligned with the task, leaving the formation of those features unaddressed. We derive how feature directions form from unaligned weights in a quadratic network and how growing strengths affect further direction learning. Under population gradient flow, we solve the early direction dynamics in the limit of small initialization across a broad class of input distributions. The network acquires a hierarchy of directions set by the task and data while its output remains small, with alignment timescales governed by differences in growth rates. For isotropic Gaussian inputs, the direction law is exact throughout training and does not require small initialization. We quantify silent alignment feature by feature, showing that smaller initialization delays substantial use of each feature without slowing its formation. For more general inputs, growing feature strengths can feed back on direction learning through the changing prediction error and reorganize the features formed early in training. Exact solutions for structured input distributions show how anisotropy and fourth moments govern this feedback-induced reorganization: the network can replace a feature acquired early or mix it with another direction, or features can promote or suppress one another's growth. The resulting theory connects the formation of a representation to its subsequent development as the network learns to use it.

Primary area:
learning theory
Keywords:
feature learning, learning dynamics, quadratic neural networks, silent alignment, representation learning

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.50 over 4 reviews, above 99% of the others.
Reviewer_cDivOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies population gradient flow in a two-layer quadratic network (teacher–student, M = WWᵀ). It splits the dynamics into motion of the student subspace (feature formation) and growth of amplitudes within that subspace (feature expression). For isotropic Gaussian inputs, the subspace evolves exactly as the span of e^{tM⋆}U₀, independently of the amplitudes. Overlaps with teacher eigen-subspaces follow a replicator equation, so features form in order of the teacher eigenvalues. Each alignment time is set by an eigenvalue gap and does not depend on the initialization scale, whereas expression is delayed by log(1/ε). This gives a feature-by-feature account of silent alignment. For general inputs at small initialization, the same law holds with an effective teacher H⋆ reshaped by the input covariance and fourth cumulant. As for finite amplitude, when amplitudes grow, the student’s own predictions feed back on the directions. In exactly solvable cases, anisotropic covariance makes a later feature replace the first one, directional kurtosis produces a mixed feature, and spherical fourth moments make amplitudes compete or cooperate while preserving the direction path.

Critical strengths

The paper is solid, clearly written, and technically careful. The direction/amplitude decomposition is clean, and the replicator formulation makes the hierarchy of feature formation transparent and quantitative.

The structured-input analyses are the most interesting part. They show explicitly how anisotropy and fourth moments govern the feedback-induced reorganization of early features, through replacement, mixing, and competition or cooperation. These are nontrivial nonlinear effects that linear networks cannot capture.

The ranking of features by fitted amplitude λ, loss reduction fλ, and initial growth f²λ is elegant. It connects to empirical observations on feature availability versus predictivity.

The numerical checks are thorough: independent integration of the closures, finite-data and minibatch tests, and a held-out ranking benchmark.

Critical weaknesses

Novelty of the isotropic part. The isotropic matrix Riccati solution is already known (Martin et al. 2024; Ben Arous et al. 2025; Ziyin et al. 2026), and the projector flow is Brockett/Oja. The new content there is mainly the feature-resolved reading: replicator equation, monotone cumulative capture, and mode-specific silent alignment. Sections 4–5 contain the main novelty, and the paper should say so more explicitly.

High-dimensional regime. All results are at finite d. A discussion on how the main results and conclusions change when d>>1 is important.

Scope of the reorganization results. The exactly solvable examples are low-dimensional and assume teacher and data moments diagonal in the same basis. A more general criterion for when early features get reorganized would be valuable, for example built on the stability analysis of Prop. H.2, along with an example where the two bases are not aligned.

Questions for authors

none, see the weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer cDiv (1/2): Novelty and high dimension

Thank you for the careful review and especially for your interest and questions on how data structure shapes feature reorganization. Your questions encouraged us to derive a more general reorganization criterion and work out how alignment and cooperation depend on dimension.

1. Novelty of the isotropic results

As you mention, the feature-resolved reading is central to our contribution and goes beyond the known solutions. The fine-grained feature analysis lets us distinguish several events that might otherwise be compressed into "learning a mode", especially when one follows only its contribution to the predictor. A direction can already be formed in the student representation while its amplitude, and hence its contribution to prediction, remains negligible.

We build on the known Riccati solution, as you identified, and classical projector flow to derive the exact joint-subspace overlap law (the replicator equation), monotone cumulative capture, and feature-specific formation and expression times. For a rank-kk student, the acquisition of the first m≤km\leq k teacher directions has asymptotic rate

\qquad C_m(t)=\sum_{i\leq m}v_i^\top P(t)v_i,$$ under the simple spectral-gap and generic-overlap assumptions in Appendix C. Expression instead occurs on the familiar logarithmic timescale, $$t_m^{\mathrm{expr}}=\frac{\log(1/\varepsilon)}{2\lambda_m}+O(1).$$ Comparing this expression time with the feature formation timescale we identify gives a silent interval for each feature. We will emphasize our distinct contributions beyond the known equations more explicitly in the updated manuscript. **2. High-dimensional regime** The dynamical equations hold at every finite dimension. The quantities that can change with dimension are the spectral gaps, initial overlaps, and total amplitude. For example, consider a rank-one student with isotropic Gaussian inputs. Let the teacher have one leading eigenvalue $\lambda_1$, with all remaining eigenvalues equal to $\lambda_0\geq0$. As dimension grows, hold these eigenvalues and their gap $\Delta=\lambda_1-\lambda_0>0$ fixed. Write $p_0$ for the student's initial squared overlap with the leading teacher direction. To reach a fixed squared overlap $1-\eta>p_0$, the exact direction law gives $$t_{\mathrm{align}}(\eta)= \frac{\log((1-p_0)/p_0)+\log((1-\eta)/\eta)}{2\Delta}.$$ Suppose $p_0$ is of order $1/d$, as is typical for a random initial direction. Formation then takes $(\log d)/(2\Delta)+O(1)$. This longer alignment time under this scaling comes from starting farther from the target direction, even though the gap controlling the rate is unchanged. To determine whether this alignment remains silent, we also need the amplitude at that time. Let $s(t)$ be the student's amplitude and $\varepsilon=s(0)$ its initial value. The exact amplitude solution gives $s(t_{\mathrm{align}})=O(\varepsilon d^{\lambda_0/\Delta})$. Thus alignment occurs while the amplitude still vanishes if we let initialization decrease with dimension so that $$\varepsilon d^{\lambda_0/\Delta}\longrightarrow0.$$
AuthorsOpenReview ↗

Response to Reviewer cDiv (2/2): General reorganization

3. General reorganization and nonaligned bases

We agree that this is an important question. We revisited it and derived a general sufficient condition for Gaussian inputs with arbitrary positive-definite covariance, without assuming a shared eigenbasis.

Let PeP_{\mathrm e} project onto the leading kk eigenvectors of G=ΣM⋆ΣG=\Sigma M_\star\Sigma, the subspace selected during early learning, and let P∞P_\infty be the final subspace projector. For 1≤k<d1\leq k<d and k≤rank⁡(M⋆)k\leq\operatorname{rank}(M_\star), with gaps separating the leading kk eigenvalues in the early and final problems below, the condition for generic initialization is

\quad\Longrightarrow\quad P_\infty\ne P_{\mathrm e}.}$$ In words, if covariance mixes the early learned subspace with directions outside it, the student must change that subspace as its features become expressed. We can see this directly by writing an orthonormal basis for the early subspace as $U_{\mathrm e}$. At this subspace, the teacher drive has no perpendicular component. For any positive definite amplitude matrix $S$, the remaining direction equation is $$\dot U_{\mathrm e} =-(I-P_{\mathrm e})\Sigma U_{\mathrm e}\\, S\\,(U_{\mathrm e}^\top\Sigma U_{\mathrm e}).$$ The first factor, $(I-P_{\mathrm e})\Sigma U_{\mathrm e}$, is the part of the covariance action that points outside the early subspace. The two remaining factors are the amplitude matrix $S$ and the covariance within that subspace, $U_{\mathrm e}^\top\Sigma U_{\mathrm e}$. Both are positive definite and therefore invertible. Multiplying by them cannot turn a nonzero first factor into zero. Hence no stationary rank-$k$ student can retain the early subspace when the stated condition holds. We can also determine the subspace to which the student converges. If we express the student weights as $\Sigma^{1/2}W$, the Gaussian loss becomes ordinary squared matrix error in approximating $T=\Sigma^{1/2}M_\star\Sigma^{1/2}$ by a rank-$k$ matrix. The best fit keeps the largest $k$ eigenvalues of $T$ and their eigenvectors. Let $V_k$ contain those eigenvectors. Mapping back to the original weight coordinates then gives the final student subspace $\operatorname{span}(\Sigma^{-1/2}V_k)$.

I thank the authors for the detailed reply, and the additional material. I keep my score.

Reviewer_AhDPOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how feature directions form, become expressed, and subsequently reorganize in a shallow quadratic teacher–student network trained by population gradient flow. It separates the learned matrix into a feature subspace and within-subspace amplitudes. For isotropic Gaussian inputs, it gives an exact account of subspace evolution from unaligned initialization and quantifies feature-specific alignment rates and silent-alignment windows. For general input distributions, it identifies how covariance and fourth moments reshape the early growth operator and establishes controlled small-initialization alignment results. Structured examples then explain feature replacement, stable mixing, and competitive or cooperative amplitude dynamics. Derivations and numerical experiments, including selected finite-sample and minibatch checks, support the analysis.

Critical strengths

The analysis beyond isotropic Gaussian inputs provides concrete insight into representation reorganization. The controlled early-alignment results connect feature formation to an effective operator determined jointly by the target, covariance, and fourth moments. The structured examples subsequently show how changing residuals can cause an initially acquired feature to be replaced or become a stable mixture. Particularly informative is the distinction between changes in feature direction and interactions in feature strength: spherical fourth moments preserve the rescaled subspace trajectory while inducing competition or cooperation between amplitudes. These results explain mechanisms and their conditions, rather than only describing learning curves.

The paper gives a clear and coherent account of the relationship between feature formation and expression. Its feature-specific alignment rates and expression times make precise how a direction can become well represented before it contributes appreciably to prediction. The organization connects this early separation to the later feedback of amplitudes onto direction learning. The appendix supplies detailed derivations, and the numerical checks illustrate the predicted dynamics. This combination makes the work informative as a theoretical study of quadratic networks.

Critical weaknesses

None

Questions for authors

Can the authors provide a clearer positioning? Specifically, could the authors delineate which feature-specific timing and subspace-dynamics results in Section 3 refine the directional/radial decomposition and autonomous alignment dynamics already studied by Ben Arous et al. (2025)? A brief comparison with Min, Mallada & Vidal (ICLR 2024) would also help distinguish the present exact quadratic-network results from early-alignment theory for ReLU networks.

2 comments
AuthorsOpenReview ↗

Response to Reviewer AhDP

Thank you for the thoughtful review and for highlighting the connection between feature-specific formation and expression and later reorganization. Below, we clarify what new results we add compared to the works you highlight.

Positioning relative to Ben Arous et al. and Min et al.

Indeed, Ben Arous et al. also separate radial and directional variables and analyze alignment dynamics. Our contributions build on that to resolve the formation and expression of individual features. In particular, for isotropic Gaussian data, we give the exact joint-subspace overlaps over time

\qquad E_I=\sum_{i\in I}\lambda_i.$$ The formula resolves the student's overlaps with every rank-$k$ teacher eigensubspace, including contributions from directions whose amplitudes are too small to affect its prediction. Studying this result further, we show that individual captures can rise and fall due to saddles in the direction learning dynamics. The cumulative capture of the first $m$ teacher directions increases monotonically, with a rate set by $\lambda_m-\lambda_{k+1}$, where $k$ is the student rank. Comparing this formation rate with $t_m^{\mathrm{expr}}=\log(1/\varepsilon)/(2\lambda_m)+O(1)$ gives a feature-specific silent interval. Another crucial difference is the initialization regime. Under the joint initialization and high-dimensional scaling of Ben Arous et al., their result places feature alignment and the associated risk reduction at the same leading transition time, without a parametrically separated window of silent alignment in that limit. As you mention, Min, Mallada and Vidal prove early directional alignment for ReLU binary classification with well-separated data and balanced small initialization. In their picture, active ReLU neurons enter regions where they respond exclusively to one class or the other. They bound the time needed to reach these regions independently of initialization scale, followed by a logarithmic delay before fitting. Our quadratic analysis, instead, resolves the continuous subspace trajectory exactly and describes the acquisition of individual teacher features, with a separate formation rate and silent interval for each feature. Furthermore, we extend early formation to general covariance and fourth moments and analyze how finite-amplitude feedback can replace or mix the acquired directions. We will make these comparisons clearer in the related-work discussion.

Thanks for the response. I confirm my score.

Reviewer_UdBnOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper characterizes the learning dynamics of a two-layer quadratic network with a matching teacher and student, giving a quantitative description of feature formation and expression. The authors solve the isotropic case exactly, showing in what order the network acquires the target directions and how long it takes before they are used. In some structured settings, they also show how features can be reorganized, compete or cooperate with each another.

Critical strengths

The main strengths lie in the clean and quantitative description of qualitative phenomena around feature formation and survival, such as silent alignment, for which the paper’s framework provides simple and principled interpretations. The exactly solvable examples show clearly how early features can be replaced or mixed later in some anisotropic cases. This represents a genuine step beyond current solutions of quadratic networks in the literature.

Critical weaknesses

My main concern is about the centering of the student, which requires knowledge of the input covariance. While it is common in the isotropic literature, in the general covariance case it is a modeling choice that the paper does not motivate or discuss. In the anisotropic setting this choice is not simply a convention, and the results or the described phenomenology may depend on it.

Questions for authors

Can you expand the motivation behind the centering choice, whether and how it affects your results? How do you expect the feature life cycle to change without centering or with a learned bias?

2 comments
AuthorsOpenReview ↗

Response to Reviewer UdBn

Thank you for the thoughtful review and for raising the role of centering. It led us to revisit the model. We worked out its interpretation through a fitted output bias and examined which formation and reorganization results persist when the bias learns at a finite rate or is absent.

Centering, learned bias, and the feature life cycle

Centering, as used in our model, has a standard regression interpretation. It can be thought of as the learning objective obtained after fitting an output intercept. Mathematically, let gW(x)=x⊤WW⊤xg_W(x)=x^\top WW^\top x, and consider the ordinary predictor gW(x)+bg_W(x)+b. Then its loss decomposes exactly as

=\frac12\operatorname{Var}(y-g_W(x)) +\frac12\bigl(\mathbb E[y-g_W(x)]-b\bigr)^2.$$ Minimizing over $b$ here leaves exactly the centered objective in the paper. The feature-learning problem can therefore be thought of as concerning the varying part of the target, with its constant mean fitted separately. Taken this way, the learning objective does not require supplying the population covariance as an oracle. On a finite dataset, write $e_i=y_i-g_W(x_i)$. Fitting the intercept gives the empirical objective $(2N)^{-1}\sum_i(e_i-\bar e)^2$. Alternatively, for two independent examples with residuals $e$ and $e'$, we have $$\frac12\operatorname{Var}(y-g_W(x)) =\frac14\mathbb E[(e-e')^2].$$ We can therefore train using the loss $(e-e')^2/4$ for each pair. Its expected gradient is the centered gradient, without needing to know the residual mean or input covariance. If we instead learn the bias alongside the weights, the mean residual $q=\mathbb E[y-g_W(x)]-b$ gives the following dynamics: $$\dot W=\left(H_{M_\star-M}+\frac q2\Sigma\right)W, \qquad \dot b=\gamma q,\qquad \gamma>0.$$ The intercept adds a covariance-dependent contribution and can change the intermediate anisotropic trajectory. Nevertheless, the central formation and reorganization results survive. For isotropic Gaussian inputs, the extra term is proportional to the identity, so the full subspace trajectory is unchanged at any bias-learning rate. For general input moments, the same early growth matrix and alignment hold with coupled bias learning when the target is mean-zero and the bias starts at zero. At equilibrium, the stationary feature matrices coincide with those of the centered model. The B-to-A replacement also persists without an output bias in the covariance example, although the final amplitudes change.

I thank the authors for their reply. I confirm my score.

Reviewer_2HnaOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies population gradient flow for a two-layer quadratic network with unit output weights, f(x) = x^T Mx - tr(MΣ) with M = WW^T, trained on a quadratic teacher with PSD matrix M_\star. Writing M = USU^T separates feature formation (the student subspace spanned by U) from expression (the amplitude matrix S), and splits the direction dynamics into a teacher drive and a student-feedback term. For isotropic Gaussian inputs, the student subspace is span(e^{tM_\star}U_0) at all times, independent of the amplitudes and of the initialization scale ε. Overlaps with teacher eigensubspaces follow a replicator equation, so teacher directions are acquired in order of their eigenvalues. Each feature gets an alignment time set by an eigenvalue gap and independent of ε, and an expression time of order log(1/ε)/(2λ_m), which defines a feature-specific window of silent alignment. For general inputs at small initialization, the same law holds with an effective teacher H_\star reshaped by the input covariance and fourth cumulant, with a bound valid up to times of order log(1/ε). Once amplitudes grow, the student feedback can reorganize the features formed early. Three solvable families illustrate this: anisotropic covariance makes a later feature replace an early one, one-axis kurtosis yields a mixed feature, and spherical fourth moments couple the amplitudes competitively or cooperatively while leaving directions unchanged. For independent coordinates, the paper contrasts three rankings (fitted amplitude λ, loss reduction fλ, initial growth f^2λ) and connects the gap between them to the availability/predictivity dissociation reported by Hermann & Lampinen. Validation consists of numerical checks of every closed form in d \leq 7, a held-out ranking benchmark of 192 random problems, and finite-sample and minibatch runs for the covariance switch.

Critical strengths
  • The authors establish a clear separation between the timescales of feature formation annd expression individually for each feature and in a non-trivial (non-linear) setting. This makes it a very interesting contribution to the area.
  • An excellently clear analysis and demonstration of (some of) the reasons why certain “easy” features emerge early but might be de-prioritized later. The different rankings provided by λ, fλ, and f^2λ make this presentation very clear and pedagogical.
  • I would think that the weakest point is the presentation of contributions and novelty regarding Section 3. With the exception of the replicator formulation and per-feature timescales the rest of the analysis seems to be covered in prior work (Stoger & Soltanolkotabi, 2021; Jin et al. 2023). At the very least the authors should elaborate a bit more about the relationship of present work to existing work like the papers above.
Critical weaknesses
  • I would think that the weakest point is the presentation of contributions and novelty regarding Section 3. With the exception of the replicator formulation and per-feature timescales the rest of the analysis seems to be covered in prior work (Stoger & Soltanolkotabi, 2021; Jin et al. 2023). At the very least the authors should elaborate a bit more about the relationship of present work to existing work like the papers above.
  • I would also argue that the examples given in Section 5 regarding feature re-organization (which I love and I learned a lot from) are too specific to be considered a general theory of reorganization. I would recommend an even clearer presentation of the results in the abstract that clarified further that we’re talking about narrow examples.
Questions for authors

Questions that could change my assessment:

  1. Please give a short comparison, (maybe in a table or other simple formet), of what Sec. 3 adds beyond the known matrix solution and the small-initialization analyses of matrix sensing (Stöger & Soltanolkotabi, 2021; Jin et al., 2023). Which statements are new: the replicator law, the monotone cumulative capture, the per-mode expression bound, or the global silent interval?
  2. Can you state any general condition for reorganization beyond the three solvable families? For example, a sufficient condition on (Σ, K) under which the leading eigenspace of H_\star differs from the subspace retained at convergence, even perturbatively around isotropic inputs.
  3. App. E notes \bar{κ} \geq -2/(d+2) for spherical fourth moments, which implies c \geq -1/d. The cooperative regime therefore vanishes as d grows, while the competitive one does not. Please state this in Sec. 5.3. Is cooperation relevant beyond small d?
  4. Does the f^2λ growth ranking predict early feature order in a ReLU or tanh network trained on the same synthetic data? Even one such experiment would support the claims in Sec. 5.4.

Clarification requests (minor):

  1. Sec. 3, l. 156–158: “the student matrix acts within its own range, so (I-P)MU = 0” states a trivial identity (MU = US) that holds for any input law. The step that uses the isotropic assumption is that the student-feedback operator in Eq. (5) reduces to H_M = M. Please state this explicitly and point to Prop. E.1 and Thm. E.2, which give the conditions under which the covariance and cumulant parts of the feedback vanish for every subspace. The point is made more clearly in Sec. 5 (l. 300–302); making it in Sec. 3 would make the amplitude-independence of the projector self-explanatory.
  2. Fig. 3c,d use initial amplitudes of 10^{-24}. Does the plateau structure remain visible at initialization scales used in practice?
  3. Sec. 5.3: the sign of c is the sign of \bar{κ}, since 1+\bar{κ} > 0. So c > 0 is the heavy-tailed case (every Gaussian scale mixture, multivariate t) and c < 0 the light-tailed one (a concentrated radius, e.g. uniform on the sphere). Please say this where the competitive and cooperative regimes are introduced (l. 377, 395). Relatedly, the sampled experiment in App. I.2 uses only laws with c \geq 0 (Gaussian, t_{12}, a scale mixture).
3 comments
AuthorsOpenReview ↗

Response to Reviewer 2Hna (1/3): Contribution and isotropic geometry

Thank you for the detailed review and for highlighting the separation between formation and expression for individual features. In response to your questions, we derived conditions for reorganization beyond the solvable examples and examined cooperation in high dimension. We also tested ReLU learning on the same task and checked larger initialization and sampled cooperation.

Question 1. Contribution beyond matrix sensing and the known solution

The exact feature-wise analysis in our work lets us distinguish several events that are compressed into "learning a mode" when one follows only its contribution to the predictor. This forms the basis of our contribution beyond the works you highlight.

Stöger and Soltanolkotabi establish an alignment phase that enables subsequent recovery. In our work, we resolve that alignment into the acquisition of individual features. In particular, our contribution is the exact joint-overlap dynamics, monotone cumulative capture, and the formation rate λm−λk+1\lambda_m-\lambda_{k+1}, which identifies how competition for the student's limited capacity determines acquisition.

Similarly, Jin et al. establish successive recovery of approximately rank-ss predictors. But that does not determine when the directions supporting those predictors are first formed. A rank-kk student can already have its entire useful subspace aligned while its prediction remains effectively rank one due to lopsided amplitudes. Our analysis distinguishes those events and calculates their separation feature by feature.

Formation and expression depend on different quantities. Formation depends on the gap from competing directions outside the student's capacity, while expression depends on the feature's own strength and initialization scale. For a rank-kk student under the assumptions in Appendix C, the quantitative comparison is

\qquad t_m^{\mathrm{expr}}=\frac{\log(1/\varepsilon)}{2\lambda_m}+O(1).$$ Here $C_m$ is the total capture of the first $m$ teacher directions, and $\lambda_{k+1}$ is the strength of the strongest direction outside the student's capacity. For the specific statements in your question: | Statement | Relation to prior work | |-----------|------------------------| | Riccati solution and autonomous projector flow | Established mathematical foundations. | | Joint-overlap replicator law and monotone cumulative capture | Explicit laws for feature geometry derived in our analysis. | | Per-mode expression bound | All-time amplitude control from generic unaligned initialization; the logarithmic growth delay itself is established. | | Silent alignment | The global phenomenon is established; our geometry and amplitude analysis quantifies a separate interval for each feature. | Sections 4 and 5 then follow the same geometry into general input moments and amplitude-driven reorganization. **Minor clarification 1. Isotropic feedback cancellation** The distribution-specific step is $H_M=M$ for isotropic Gaussian inputs. Since $M=USU^\top$, the columns of $MU=US$ lie within the student subspace. Therefore $$(I-P)H_MU=(I-P)MU=(I-P)US=0.$$ The student's feedback changes the amplitudes without turning the subspace, leaving the teacher contribution to determine the direction dynamics. Proposition E.1 and Theorem E.2 give the broader conditions under which the covariance and fourth-cumulant contributions separately preserve every subspace. The geometric identity $MU=US$ alone would not remove the feedback for general input moments. We will state the step $H_M=M$ explicitly in Section 3 and include these appendix references there.
AuthorsOpenReview ↗

Response to Reviewer 2Hna (2/3): Reorganization and high dimension

Question 2. General conditions for reorganization

In response to your question, we have derived both a general Gaussian criterion and a quantitative condition for weak covariance and fourth-moment changes.

For Gaussian inputs, let PeP_{\mathrm e} project onto the leading kk eigenvectors of H⋆=ΣM⋆ΣH_\star=\Sigma M_\star\Sigma, and let P∞P_\infty be the final subspace projector. With Σ≻0\Sigma\succ0, 1≤k<d1\leq k<d, k≤rank⁡(M⋆)k\leq\operatorname{rank}(M_\star), and gaps in the early and final spectral problems,

\quad\Longrightarrow\quad P_\infty\ne P_{\mathrm e}$$ for generic initialization. In words, covariance mixing between the early subspace and its complement forces reorganization. Let $U_{\mathrm e}$ be an orthonormal basis for the early subspace. At $U=U_{\mathrm e}$, the teacher contribution to direction learning vanishes, leaving $$\dot U_{\mathrm e} =-(I-P_{\mathrm e})\Sigma U_{\mathrm e}\\, S(U_{\mathrm e}^\top\Sigma U_{\mathrm e}).$$ The amplitude matrix $S$ and the covariance within the subspace, $U_{\mathrm e}^\top\Sigma U_{\mathrm e}$, are positive definite for a rank-$k$ student. Multiplying by them cannot cancel a nonzero perpendicular factor $(I-P_{\mathrm e})\Sigma U_{\mathrm e}$. Hence no stationary rank-$k$ student can retain the early subspace when the stated condition holds. To find the final subspace, express the weights as $\Sigma^{1/2}W$. The fitting problem then becomes a rank-$k$ approximation of $T=\Sigma^{1/2}M_\star\Sigma^{1/2}$, whose best fit retains its leading $k$ eigenvectors $V_k$. Returning to the original coordinates gives $\operatorname{span}(\Sigma^{-1/2}V_k)$. Generic gradient-flow trajectories converge to this best fit, and we have verified the predicted reorganization numerically. Thinking more perturbatively, take a smooth family of input laws with $$\Sigma_\eta=I+\eta C+O(\eta^2),\qquad K_A^{(\eta)}=\eta K_A^{(1)}+O(\eta^2).$$ Let $M_k=\sum_{i\leq k}\lambda_i v_iv_i^\top$, with $\lambda_k>\lambda_{k+1}$ and $\lambda_k>0$. If $P_{\mathrm e}(\eta)$ and $P_{\mathrm f}(\eta)$ are the early and final projectors, then for $i\leq k<j$, $$v_j^\top(P_{\mathrm e}-P_{\mathrm f})v_i =\eta\\, \frac{\lambda_i v_j^\top C v_i+ \tfrac12v_j^\top K_{M_k}^{(1)}v_i} {\lambda_i-\lambda_j} +O(\eta^2).$$ Any nonzero numerator guarantees reorganization for sufficiently small nonzero $\eta$. The numerator is the perpendicular feedback from the expressed modes, and the denominator is the gap between retained mode $i$ and omitted mode $j$. The trajectory solutions in Section 5 remain special solvable cases for which we can determine the full trajectory analytically, while the criterion and perturbation formula we give here describe reorganization well beyond those cases. **Question 3. Cooperation in high dimension** The bound you give, $c\geq-1/d$, is correct, but it bounds the interaction between an individual pair of modes. The amplitude dynamics, though, depend on the combined feedback $c\theta$, which need not vanish. To see this, write $c=-\alpha/d$, $0<\alpha<1$, and let $\bar r=d^{-1}\sum_i(\lambda_i-s_i)$ be the average residual across modes. Then the exact equation is $$\frac{ds_j}{d\tau} =2s_j(\lambda_j-s_j-\alpha\bar r).$$ Learning one bounded-strength feature changes the shared feedback by order $1/d$. Learning a finite fraction of the features changes it by order one. Cooperation can therefore remain finite when the number of learned features grows with dimension. Such inputs can have uniformly random directions but less variation in their squared length than Gaussian inputs. With covariance $I$, write $V=\operatorname{Var}(\\|x\\|^2)$ for this variation. The coupling is $$c=\frac{V-2d}{2(d^2+V)}.$$ Gaussian inputs have $V=2d$. Smaller variation gives negative coupling, and reducing $V$ by a fixed fraction of $2d$ gives a coupling of order $-1/d$. In fact, we can check the scaling issue directly by repeating each teacher and student mode $n$ times and replacing $c$ by $c/n$. Every amplitude trajectory and release time remains unchanged, because the total residual grows by the compensating factor $n$. We also verified this numerically.
AuthorsOpenReview ↗

Response to Reviewer 2Hna (3/3): ReLU and robustness

Question 4. ReLU on the same synthetic task

Thank you for prompting us to check this. In the same Gaussian covariance task as in the paper, we found that indeed a ReLU student also forms B first and later reorganizes toward A. To check that, we kept the input covariance and quadratic teacher unchanged and trained y^=a,ReLU⁡(w⊤x)+b,\widehat y=a\\,\operatorname{ReLU}(w^\top x)+b, with all weights, the readout and the output intercept learnable (which are well beyond the assumptions of our theory).

For a unit direction uu,

=\frac{u^\top\Sigma M_\star\Sigma u} {\sqrt{2\pi\\,u^\top\Sigma u}}.$$ With balanced small initialization, the early student direction ascends this correlation. Along teacher axis $i$, the ReLU growth score is therefore proportional to $d_i^{3/2}\lambda_i$ and the best fitted axis still ranks by $d_i\lambda_i$. For the two principal features in our example, the scores are | Feature | Quadratic early score | ReLU early score | Best-fit score | |---------|-----------------------|------------------|----------------| | A | 1 | 1 | 1 | | B | 1.75 | 1.107 | 0.70 | Both activations favor B early, while A gives the better fit. In fact, for any positive-definite Gaussian covariance and positive-semidefinite quadratic teacher, the best single-unit ReLU direction coincides with the best rank-one quadratic direction. The experiment therefore supports the early-versus-final preference described in the paper but the precise $f^2\lambda$ formula is specific to the quadratic model. See [Figure R1][response-figures]. **Minor clarification 2. Initialization scale** Very small initialization separates the stages and makes the long plateaus distinct. At larger initialization, the plateaus shorten and merge, while the competitive overshoot and cooperative initial suppression followed by growth persist. The figure below keeps the teacher, student rank, and interaction strengths from Figure 3c,d of the submitted paper fixed and changes only initialization. See [Figure R2][response-figures]. **Minor clarification 3. Interaction sign and sampled cooperation** Indeed, the sign of $c$ is the sign of $\bar\kappa$. Gaussian mixtures with different scales, including the covariance-normalized $t_{12}$ inputs used in the paper, have $c>0$. Concentrated-radius inputs, such as the uniform sphere, have $c<0$. The sign has a direct interpretation: $$\operatorname{Cov}(x_i^2-1,x_j^2-1)=\bar\kappa,\qquad i\ne j.$$ Fitting one feature reduces the growth drive for the others when this covariance is positive, and increases it when the covariance is negative. These are the competitive and cooperative cases in the amplitude equation. We will explain this sign interpretation where we introduce the two regimes in Section 5.3. We also checked the cooperative case on six independent finite datasets, where both full-dataset gradient flow and minibatch SGD reproduce the initial suppression and subsequent growth shown below. See [Figure R3][response-figures]. [response-figures]: https://anonymous.4open.science/r/response-figures-40AE/response_figures.pdf

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.