acceptodds
Under review as a conference paper at ICLR 2027

Implicit Bias of SGD in ReLU Autoencoders

Abstract

How can useful features emerge without label supervision? We study this question theoretically using shallow ReLU and linear autoencoders trained on data from a 4-component Gaussian mixture model. Under stochastic gradient descent (SGD), the ReLU autoencoder learns neurons aligned with the cluster-mean directions, whereas the linear autoencoder recovers only the signal subspace, without selecting a preferred orientation. Consequently, the ReLU model can learn representations better suited to downstream classification, even though both architectures attain the same optimal reconstruction loss. We show that the implicit directional bias emerges during late-stage SGD but is absent under population gradient flow on the same ReLU model. We explain this implicit bias of SGD by characterizing the global-minimizer manifold of the ReLU autoencoder as a (NASM) for population gradient flow. Building on this geometry, we develop an analysis of SGD with a small step size over iterations and prove that stochastic gradient fluctuations induce an effective tangential drift along the global-minimizer manifold. In our setting, the high-dimensional drift reduces to a scalar angular ordinary differential equation whose stable equilibria correspond to the cluster-mean directions. This analysis reveals a mode of unsupervised feature learning during late-stage SGD, in which the representation continues to“get better” even after the unsupervised training loss has plateaued.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Autoencoder, stochastic gradient descent, implicit bias, unsupervised feature learning, normally hyperbolic invariant manifold

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_YHgrOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper is studying the implicit bias of SGD in autoencoders. What features are learned? The setting considered is where data is drawn from a Gaussian mixture model with means μ\mu, −μ-\mu, μ′\mu', and −μ′-\mu' for two orthogonal vectors μ\mu and μ′\mu' of equal length. The paper mainly analyses the training dynamics of a shallow autoencoder with ReLU activations trained with SGD. For the data above, the bottleneck width of the autoencoder is 4 (2 if using linear activation instead of ReLU), but for autoencoders that have this bottleneck width there is an entire space of equally good solutions (essentially vectors that are orthogonal and span the same space as μ\mu and μ′\mu' - which allows for arbitrary rotations as long as the span is not changed). SGD with ReLU activations however has an implicit bias to a particular solution (aligning with the data means). This is achieved in a second, relatively slow phase during which the training loss remains essentially unchanged. Linear autoencoders do not show this behaviour.

Critical strengths

I was not able to check all the proofs in detail, but I find this work conceptually very interesting that it is impressive to have such a rigorous account of the training dynamics and implicit bias in this setting. It is, of course, a bit of a toy setting but it is an interesting and meaningful one.

Critical weaknesses

On one hand, the experiments are very useful in illustrating the predicted behaviour. On the other hand, it also shows a gap in our understanding because while SGD with ReLU always did favour particular solutions in the experiments, it was not always as simple as weights go to "the mean of the gaussians". In some settings, they instead "align with the angular bisectors between neighboring cluster directions". It is not a particular strong weakness, as the authors openly point this out and is more and indication that additional work may be needed to deepen our understanding. This does not take away from the fact that this paper is making an important and substantial first step. And it may stimulate follow-up work.

It is disappointing that no code was provided.

Questions for authors

My main question is whether the authors have an explanation or conjecture on the change of the implicit bias behavior for different values of kk.

1 comment

We first thank the reviewer YHgr for the appreciation of this work and asking the question that we are also excited to answer.

Critical Weaknesses

It is disappointing that no code was provided.

We have included the code in the supplementary material to reproduce the experiments shown in Figure 8. I hope this helps.

Questions for Authors

My main question is whether the authors have an explanation or conjecture on the change of the implicit bias behavior for different values of kk

First of all, Conjecture 5.2 in the manuscript summarizes the pattern observed in our experiments with the symmetric Gaussian mixtures. In the tested regimes, the preferred feature directions depend on the number of components: for k=4nk=4n, features align with the cluster means, whereas for k=4n+2k=4n+2, they align with the angular bisectors between neighboring cluster means, where n≥1n \geq 1. Note that we observe this behavior when the signal subspace is embedded in a higher-dimensional ambient space (It is not due to the lower-dimensional happening in the 4-GMM case, see Corollary 4.4 and Remark 4.5).

One other important observation is that the sparsity emerges only in the late-stage. This means that, early in the loss plateau, the feature orientations remain dispersed (phase 1). In our minimal width setting, the phase 2 was the rotational movement. However, in this larger widths setting, they concentrate around the preferred 4n4n or 4n+24n + 2 directions, causing the sparsity, but sparsity grows with the number of clusters.

We can also describe a possible route toward extending our analysis, which could also be of separate interest: Our current proof relies on a normally attracting stationary manifold. This means that the population Hessian is uniformly positive definite in the normal directions, creating fast normal relaxation relative to the slow tangential motion. However, wider networks can contain configurations with additional Hessian degeneracies, where this uniform separation no longer applies directly.

Interestingly, if we expand the loss to even higher order, these normal variables still contribute to the loss, meaning that these normal variables are still faster than the tangent variables, thus the degenerate directions still have weakly restoring force. A possible extension would retain these weakly restoring directions and analyze their leading higher-order dynamics, together with their coupling to SGD noise, after eliminating the strongly stable directions. This could produce three(or more) distinct timescales: fast relaxation in directions with positive quadratic curvature, intermediate evolution of the degenerate variables, and slow feature selection. Establishing this separation (and deriving the resulting effective drift) is part of the proposed extension.

We believe this is a promising direction for understanding implicit bias beyond the normally nondegenerate setting. For example, analyses of interpolating models often assume that the Jacobian of the predictions on nn training samples has rank nn (Li et al., ICLR 2022). Redundant training constraints can violate this assumption, while (near) linear dependencies among the corresponding parameter gradients can lead to poor conditioning. Correlated or redundant training samples, resulting from data augmentation for instance, therefore motivate investigating such regimes (although correlation alone does not imply exact rank deficiency).

Reviewer_JYFcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies SGD training dynamics of a shallow weight-tied autoencoder on a toy 4-component Gaussian mixture. The authors compare linear and ReLU autoencoders, and take the minimal number of hidden neurons sufficient to reproduce the cluster means (2 neurons for a linear autoencoder, and 4 --- for a ReLU one). They prove that under the ReLU setting, as soon as SGD gets near the global minimum manifold, it exhibits a tangential drift along this manifold, while this phenomenon is absent under the linear setting. What is essential for this drift to occur is stochasticity: stochastic gradient noise produces normal fluctuations of order η\sqrt{\eta} around the global-minimizer manifold. Since the normal Hessian varies along this manifold, these fluctuations induce a systematic tangential drift of order η2\eta^2 per iteration. This drift causes a ReLU autoencoder to align its weights to the cluster means, while non-aligned configurations also exist among global minima. The nature of the drift is different from that of Edge of Stability [1]: there, no stochasticity is needed; the drift is caused by fluctuations of large-step updates around a global minima manifold and, again, Hessian variation.

[1] Arora, Li & Panigrahi. Understanding Gradient Descent on the Edge of Stability in Deep Learning. ICML'2022.

Critical strengths

A rigorous reduction from high-dimensional SGD dynamics to a deterministic angular ODE on the minimizer manifold, identifying which Gaussian-mixture directions are selected, seems like a valuable contribution.

Critical weaknesses
  1. Edge of Stability (EoS) is a widely-known and extensively studied class of phenomena that also cause tangential drift along a global minima manifold, but the submitted manuscript does not provide any references to it. It is also not clear if the empirical alignment phenomena observed in the experiments are caused by stochasticity and not by EoS. On top of that, the manuscript misses an important reference [2] that also study narrow autoencoders and demonstrate that GD and SGD converge to different solutions. In their setting, SGD also breaks the symmetry between global minimizers and selects data-aligned features, although through a different stochastic mechanism. Based on this, I think the mansucript needs a more extensive literature overview and comparison.
  2. Sections 3 and 4 are technically dense and difficult to read. This density obscures the nature and intuition behind the phenomena studied.

[2] Ghosh, Frei, Ha, Yu. The Effect of SGD Batch Size on Autoencoder Learning: Sparsity, Sharpness, and Feature Learning. JMLR'2025.

Questions for authors

None

3 comments

We thank the reviewer JYFc for providing review with the related references.

Critical Weaknesses

W1. Edge of Stability (EoS) ... :

First of all, as the reviewer has acknowledged, our setting is different from EoS, since in our setting, ηλmax(∇2L(W))→0\eta \lambda_{\mathrm{max}}(\nabla^2 L(W)) \rightarrow 0. This is the same in the experiments. For instance, we measured ηλmax(∇2L(W))\eta \lambda_{\mathrm{max}} (\nabla^2 L(W)) throughout the training, for Figure 8. This shows that our experiments were conducted well below the EoS regime. Note that we have used step size 0.2 but it is still much below the EoS threshold.

Stepλmax(∇2L(W))\lambda_{\mathrm{max}} (\nabla^2 L(W))
00.564363
150000.561688
300000.555628
600000.566674
1500000.56217
3000000.558246

However, we agree that this is relevant literature and have added to Appendix A.

W2. Comparison with Ghosh et al.

Thanks for pointing this out. We have added a detailed comparison in Appendix A, under ``Comparison with Ghosh et al.''. Note that we give the same answer for the reviewers who posed this question. Two distinctions clarify our contribution. First comes from the conceptual difference, and the second is a key consequence that follows from it.

First, the crucial conceptual distinction is that our starting point is the consideration of how to represent the given data. Accordingly, we allocate a sufficient number of neurons to represent this data and then investigate how this representation differs across architectures and training algorithms. In contrast, as stated in their work, Ghosh et al. focus on the optimization aspects of training rather than considering an appropriate representation. This is partly because they use only one neuron while their orthonormal data is higher-dimensional, which cannot be represented by a single neuron. This creates different training dynamics than in our setting, as detailed in the appendix.

Importantly, their result show that dynamics/bias is qualitatively similar for linear and ReLU AE. In their setup, the single-neuron constraint causes the dynamics of ReLU and linear networks to coincide, yielding the same implicit bias for both models. By contrast, we show that while linear AE has no directional bias, SGD on ReLU AE has an implicit directional bias.

W3. Sections 3 and 4 are technically dense ... :

Thank you for this comment. The density was necessary to give high-level proof idea (and what is different from the previous work) for the interested readers who know related proofs. We have tried to add more intuitive explanations in Section 4 (Corollary 4.4, Remarks 4.5 and 4.6). We hope this helps clarify any remaining points of confusion.

Thank you for your response!

I appreciate the new sections on EoS and on the paper of Ghosh e al. in Additional Related Work. Still, I think the technical part of the main is heavy and difficult to read. Therefore, I maintain my current score.

Thank you for pointing out the readability issue of the technical sections. We have rewritten Section 4 to clarify the main argument and importantly, added Figure 3 to illustrate how normal fluctuations in the late phase accumulate into tangential drift and angular selection. We hope these revisions make the analysis easier to follow and would appreciate your assessment of the revised section.

Reviewer_w7GUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the implicit bias of SGD in tied-weight ReLU and linear autoencoders trained on data from a 4-component Gaussian mixture. In particular, the authors show that both autoencoders attain the same optimal reconstruction loss, but SGD rotates the features learned by the ReLU autoencoder toward the cluster-mean directions, whereas the linear autoencoder only recovers the signal subspace without selecting a preferred orientation. The key step in the proof is to characterize the manifold of global minimizers as a normally attracting stationary manifold (NASM) and to reduce the SGD dynamics near this manifold to a scalar angular ODE. The authors further show that the mean-aligned features learned by the ReLU autoencoder are more favorable for downstream XOR classification than those learned by the linear autoencoder. Finally, the paper conjectures that a similar feature selection occurs for wider ReLU autoencoders and Gaussian mixtures with more components, although the selected directions may not always align with the cluster means.

Critical strengths
  • The paper studies a well-motivated question of which solution SGD selects among many equally optimal ones. This question is important for understanding why SGD often learns useful representations, and it has been much less studied in unsupervised learning than in supervised learning.

  • The theoretical approach is interesting, and the techniques developed in the paper could be useful for analyzing SGD near manifolds of global minimizers in other ReLU-based models whose losses do not satisfy the smoothness.

Critical weaknesses
  • The results are limited to the very specific setting of a symmetric 4-component Gaussian mixture, and since real-world data are much more complicated, their practical implications are unclear. The convergence result is also local and does not cover random initialization, which is widely used in practice. Moreover, Proposition 5.1 suggests that at high SNR, almost any ReLU solution (any θ≠π/4\theta\neq\pi/4) already achieves small XOR error, not only the mean-aligned solutions selected by SGD. This makes it unclear whether the implicit bias of SGD itself is important for downstream performance, as the paper suggests.

  • The paper essentially assumes infinitely many samples. In practice, models are trained on a finite dataset, where the characterization of the global minimizers in Proposition 2.2 may no longer hold, so it is unclear whether the same implicit bias still arises. It would be more interesting to see the results in the finite-sample setting theoretically, or if this is difficult, to at least verify more thoroughly in experiments whether a similar phenomenon occurs with finite samples.

Questions for authors

Additional questions are as follows:

  • The notation is often inconsistent or undefined, which makes the paper difficult to follow. The paper does not appear to have been carefully proofread before submission. The following are some of the issues I found:

    • RθR_\theta is used in line 234 without being defined.
    • The parameter is denoted by ϑ\vartheta in the SGD setting of Section 3, but by zz in Theorem 3.1 and Section 4.
    • The gradient-flow projection Φ\Phi appears in Theorem 3.1, but it is only defined in Section 4.
    • The loss is written as L△(W△)L_\triangle(W_\triangle), L(ϑ)L(\vartheta), and L(z)L(z) in different places, and the single-sample loss ℓ\ell is never explicitly defined, so it is unclear whether it includes the weight-decay penalty.
    • The paper emphasizes the NASM, but its formal definition only appears in Appendix C, where it is instead called a normally attracting invariant manifold (NAIM). A formal definition in the main text and consistent terminology would help readers better understand the concept.
  • In Fact 2.1, ρ2=1−λ/λs\rho^2=1-\lambda/\lambda_s does not seem correct for the linear autoencoder under the loss in Section 2.1. Could the authors check this?

  • It would be helpful for the authors to clarify whether the condition P(Φ(z0)=v(π/4))=0P(\Phi(z_0)=v(\pi/4))=0 in Theorem 3.1 is necessary or an artifact of the proof. If it is necessary, could the authors provide more intuition for why it is needed?

  • For the ReLU autoencoder, Theorem 3.1 only provides a guarantee at iteration T/η2T/\eta^2. Does the iterate remain close to M∗\mathcal M^* if SGD runs for more than T/η2T/\eta^2 iterations? Also, does the result for the linear autoencoder hold for all t≥t0t\geq t_0, rather than only for t∈[t0,T′]t\in[t_0,T']? This seems important, since we would like to understand the behavior of SGD in the long run, not just at a particular time point.

  • This paper seems also related to Ghosh et al. (The Effect of SGD Batch Size on Autoencoder Learning: Sparsity, Sharpness, and Feature Learning, JMLR 2025), which also studied the implicit bias of SGD in tied-weight autoencoders. The authors may find it useful to compare their results with this work.

6 comments

We first thank the reviewer w7GU for thoughtful questions and pointing out where we can improve the manuscript.

Critical Weaknesses

W1. The results are limited to the very specific setting of a symmetric 4-component Gaussian mixture, and since real-world data are much more complicated, their practical implications are unclear. The convergence result is also local and does not cover random initialization, which is widely used in practice.

  • We note that 4-GMM is a simple, but rich, data model that has been used to explain different phenomena in supervised deep learning (Refinetti et al., 2021, Frei et al., 2023, Xu et al., 2024), reflecting real-world settings involving clustered data. Indeed, our experiments on Fashion-MNIST --- a real-world dataset that does not satisfy the Gaussianity assumption made in our theoretical analysis --- also shows that ReLU AE has directional bias that is missing in linear AE (Figure 3).

    We would like to also point out that, even in this 4-GMM setting, no prior work has provided a complete characterization of its implicit bias in an unsupervised setting. We believe our work serves as an important first step toward a deeper understanding of this phenomenon. Furthermore, we note that our proof can be extended to a 2r2r-GMM lying in an rr-dimensional subspace. In Figure 3 and Conjecture 5.2, we present a conjecture that generalizes to broader settings and exhibits an important deviating behavior.

    In addition, after reading your comment, we adjusted the introduction to better motivate this discussion. Please take a look, and we hope it conveys the right message.

  • Second, regarding the comment on random initialization, we emphasize that our result covers random initialization. The only requirement is that training brings the network sufficiently close to the manifold, but this could be any point in the tubular neighborhood of the manifold the network approaches, allowing it to be any random location. Therefore SGD will almost deterministically leads ReLU networks to the same solution once it gets near. Furthermore, in the related literature, this O(η−2)O(\eta^{-2}) guarantee is known as a ``global'' analysis, as it starts from an initialization that is O(1)O(1) (independent of η\eta) away from the manifold. Our NASM then induces a fast contraction toward an O(η)O(\sqrt{\eta}) local region, where the true local analysis begins. This type of local analysis was done, for example in arXiv:1904.09080.

    If your question concerns how to reach close to this global minimizer manifold in the first place, one simple way is to use symmetric random initialization (also a practical initialization scheme). In Lemma E.1 of Appendix E, we show that this creates an invariant manifold under gradient flow. In this case, the dynamics reduce to those of a linear network --- whose loss landscape is already well-studied and known to have no spurious local minima (Baldi, Hornik 1989)--- extending the analysis to this setting is straightforward (since in this phase, SGD tracks gradient flow). We omitted this extension because it is straightforward and falls outside the main focus of our work, which centers on late-stage SGD dynamics, where the two architectures that we consider show a diverging behavior.

W2. Moreover, Proposition 5.1 suggests that at high SNR, almost any ReLU solution (any θ≠π/4\theta \neq \pi / 4) already achieves small XOR error, not only the mean-aligned solutions selected by SGD. This makes it unclear whether the implicit bias of SGD itself is important for downstream performance, as the paper suggests.

We understand the source of this confusion and have rewritten Proposition 5.1. The revised version claims that at the mean-aligned directions where θ=0,π/2\theta = 0, \pi/2, the chosen linear-probe classifier achieves Bayes optimality. In contrast to this, as θ→π/4\theta \rightarrow \pi / 4, the upper-bound of the classification error drops to 00. Therefore, unless the SNR to increases to ∞\infty as θ\theta approaches 00, high classification accuracy cannot be achieved. We hope this clarifies your confusion.

W3. The paper essentially assumes infinitely many samples. In practice, models are trained on a finite dataset, where the characterization of the global minimizers in Proposition 2.2 may no longer hold, so it is unclear whether the same implicit bias still arises. It would be more interesting to see the results in the finite-sample setting theoretically, or if this is difficult, to at least verify more thoroughly in experiments whether a similar phenomenon occurs with finite samples.

We respectfully clarify that our result does not require infinitely many samples, so this comment is not true in our setting. Our result applies over an O(η−2)O(\eta^{−2}) horizon of SGD with batch size of 1. Hence, the result uses O(η−2)O(\eta^{−2}) random samples. Also, we point out that all the experiments in the main text were conducted with the finite number of training samples. For instance, 2,000 samples in Figure 1 (as explicitly stated in the Figure 1 caption). We have also included our code in the supplementary material, which uses these 2,000 samples. Here, we used a fairly large learning rate (0.2) because the curvature of the minimizer manifold is strong, which requires fewer data points. Characterizing the relationship between the attraction of this normally attracting stationary manifold (NASM) and the required number of data points is an interesting direction for future work. We thank the reviewer for pointing this out.

Questions for Authors

Q1. The notation is often inconsistent or undefined, which makes the paper difficult to follow. The paper does not appear to have been carefully proofread before submission. The following are some of the issues I found:

Thanks for pointing out. We revised the notations based on the reported issue.

Q2. In Fact 2.1, ρ2=1−λ/λs\rho^2 = 1 - \lambda / \lambda_s does not seem correct for the linear autoencoder under the loss in Section 2.1. Could the authors check this?

You are correct about this. The ρ2=1−λ/λs\rho^2 = 1 - \lambda / \lambda_s is correct, under the regularization λ∥Wlin∥F2\lambda\lVert W_{\mathrm{lin}}\rVert_F^2 for the linear network (since they are half the size of the ReLU network). We appreciate you pointing this out.

Q3. It would be helpful for the authors to clarify whether the condition P(Φ(z0)=v(π/4))=0\mathbb P(\Phi(z_0) = v(\pi / 4)) = 0 in Theorem 3.1 is necessary or an artifact of the proof. If it is necessary, could the authors provide more intuition for why it is needed?

The condition matters for the O(η−2)O(\eta^{-2}) step guarantee if arbitrary initializations are allowed. As noted in Corollary 4.4, θ=π/4\theta=\pi/4 is an equilibrium of the limiting angular ODE. If Φ(z0)=v(π/4)\Phi(z_0)=v(\pi/4), the limiting trajectory starts at the bisector and stays there, so it does not move toward a mean direction on this time scale. This is the reason that we avoid this point.

This assumption can even be omitted with a practical assumption. As noted in the footnote 4, the condition holds automatically when z0z_0 is drawn from a distribution with a Lebesgue density supported in the neighborhood covered by the theorem. The initializations that project to v(π/4)v(\pi/4) form a lower-dimensional stable fiber and therefore have probability zero under such a distribution. Thus, the assumption can be removed under this random initialization assumption.

Q4. For the ReLU autoencoder, Theorem 3.1 only provides a guarantee at iteration T/η2T / \eta^2. Does the iterate remain close to M∗\mathcal M^* if SGD runs for more than T/η2T / \eta^2 iterations? Also, does the result for the linear autoencoder hold for all t≥t0t \geq t_0, rather than only for t∈[t0,T′]t \in [t_0, T']? This seems important, since we would like to understand the behavior of SGD in the long run, not just at a particular time point.

This is a good question about the scope of the guarantee. However, we first point out that our analysis applies to any prescribed finite observation horizon. After choosing that horizon and the desired accuracy and confidence (ϵ,δ\epsilon, \delta), we can always take the step size sufficiently small enough to obtain a guarantee throughout the horizon. In this case, the admissible step size may depend on the horizon.

We also clarify the time scaling. Both t0t_0 and T′T' are constants independent of η\eta in the rescaled time t=kη2t = k\eta^2. Thus, the interval [t0,T′][t_0,T'] corresponds to iterations from approximately t0/η2t_0/\eta^2 to T′/η2T'/\eta^2, and the guarantee holds simultaneously throughout this interval. The result therefore describes dynamics over O(η−2)O(\eta^{-2}) iterations, extending well beyond the O(η−1)O(\eta^{-1}) gradient-flow regime. For this reason, we respectfully disagree that this does not constitute a long-run analysis. Moreover, this regime is widely accepted in the literature as a long time, small-step analysis, where the distinct effects of SGD emerge. As you pointed out, analyzing behavior even beyond O(η−2)O(\eta^{-2}) iterations at a fixed step size requires additional analysis.

Q5. This paper seems also related to Ghosh et al. (The Effect of SGD Batch Size on Autoencoder Learning: Sparsity, Sharpness, and Feature Learning, JMLR 2025), which also studied the implicit bias of SGD in tied-weight autoencoders. The authors may find it useful to compare their results with this work.

Thanks for pointing this out. We have added a detailed comparison in Appendix A, under ``Comparison with Ghosh et al.''. Note that we give the same answer for the reviewers who posed this question.

Two distinctions clarify our contribution. First comes from the conceptual difference, and the second is a key consequence that follows from it. First, the crucial conceptual distinction is that our starting point is the consideration of how to represent the given data. Accordingly, we allocate a sufficient number of neurons to represent this data and then investigate how this representation differs across architectures and training algorithms. In contrast, as stated in their work, Ghosh et al. focus on the optimization aspects of training rather than considering an appropriate representation. This is partly because they use only one neuron while their orthonormal data is higher-dimensional, which cannot be represented by a single neuron. This creates different training dynamics than in our setting, as detailed in the appendix.

Importantly, their result show that dynamics/bias is qualitatively similar for linear and ReLU AE. In their setup, the single-neuron constraint causes the dynamics of ReLU and linear networks to coincide, yielding the same implicit bias for both models. By contrast, we show that while linear AE has no directional bias, SGD on ReLU AE has an implicit directional bias.

Reviewer_NhffOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies tied-weight, bias-free shallow autoencoders trained on a symmetric 4-component Gaussian mixture with means ±μ1,±μ2\pm\mu_1,\pm\mu_2 (μ1⊥μ2\mu_1\perp\mu_2, ∥μi∥=s\|\mu_i\|=s, covariance σ2Id\sigma^2I_d). The loss includes a weight decay λ∈(σ2,λs)\lambda\in(\sigma^2,\lambda_s). The ReLU autoencoder has width 4 and the linear one width 2.

  • Minimizers (Prop. 2.2, Prop. B.6). The global minimizers of the ReLU model are the antipodal pairs. On this set the reconstruction is invariant under rotation.
  • General reduction (Thm C.6). For a normally attracting manifold of stationary points and a population loss that is only C2C^2 near it, online SGD with step η\eta satisfies, on the η−2\eta^{-2} time scale, v˙=−12b(v)\dot v=-\frac12 b(v). Here bb is computed from the normal Lyapunov covariance Π\Pi and the tangential derivative of the normal Hessian. The proof assumes in particular that the sample gradient has no tangential component on the manifold (S3).
  • Main result (Lemmas 4.2–4.3, Thm 3.1).
    • For the ReLU autoencoder, bb reduces to an angular ODE θ˙=−(κB+κC) uy′(θ)\dot\theta=-(\kappa_B+\kappa_C)\,u_y'(\theta), with uy(θ)=E∣y1y2∣u_y(\theta)=\mathbb E|y_1y_2|.
    • For σ≤σ0\sigma\le\sigma_0, κB+κC>0\kappa_B+\kappa_C>0, so the hidden weights converge to the cluster-mean directions with high probability.
    • For the linear autoencoder there is no drift.
  • Further results. Prop. 5.1 bounds the XOR error of a linear probe on the ReLU features. Experiments cover larger mixtures and Fashion-MNIST. Conjecture 5.2 concerns larger widths.
Critical strengths
  • The question is well posed. Among minimizers with the same reconstruction loss, which representation does SGD select in an unsupervised model? Figure 1 isolates it: the reconstruction loss stays flat while the features rotate.
  • The model-specific computations are explicit. These include the characterization of the minimizers (Prop. B.6), the block decomposition of HNH_N and ΣN\Sigma_N into pair-preserving (S, K) and pair-breaking (B, C) directions, and the vanishing of the S, K contributions (Lemma 4.2). Together they locate the possible source of the drift in the pair-breaking directions.
Critical weaknesses
  1. The main result appears to be incorrect as stated. In Lemma 4.3, the derivative of the pair-breaking (B) block is taken in coordinates that rotate with the manifold, whereas the normal vector is parallel-transported, i.e. constant in ambient space (p. 65–66). A connection term is thus dropped. It vanishes only if the noise covariance is block-diagonal, which it is not. With this term restored, mean alignment is no longer stable at fixed dd and small σ\sigma, i.e. precisely in the regime of Theorem 3.1. Direct online-SGD simulations started on the manifold agree with the corrected drift: for d∈{2,3,8}d\in\{2,3,8\} and σ/s≤0.2\sigma/s\le0.2 the angle converges to an interior value, and alignment is recovered only for large dd (e.g. d=32d=32). Since the equilibria θ∈{0,π/4,π/2}\theta\in\{0,\pi/4,\pi/2\} are imposed by the symmetry of the data, the stability of θ=0\theta=0 is the whole content of the theorem. [Note that this flaw has been seen by a Opus 5.5]
  2. The contribution relative to existing work is limited and insufficiently positioned. The limiting drift is that of Li, Wang & Arora (ICLR 2022) restricted to normal noise. That SGD noise selects among equivalent minimizers of an autoencoder was already shown by Ghosh, Frei, Ha & Yu (JMLR 2025). The tied-weight objective is the nonlinear PCA criterion (Oja, 1997), for which rotation selection and its link to ICA are classical. The literature on noise-induced drift along symmetry directions (Ziyin et al., 2024; Chen et al., 2023) and on manifold reductions of noisy GD (Shalova, Schlichting & Peletier, 2026) is not discussed.
  3. The setting is narrow and the analysis is local. The results require minimal width, tied weights, no biases, orthogonal means of equal norm, a weight decay larger than the noise variance, and an initialization near the manifold; the feature-learning phase of Figure 1 is not covered. Two of the stated separations are direct: the absence of drift for the linear autoencoder follows from the exact O(2)O(2) invariance of its loss, and Prop. 5.1 is a Gaussian tail bound.
Questions for authors
  1. Once Lemma 4.3 is corrected, in which regime of (d,σ,s)(d,\sigma,s) is mean alignment stable, and what does SGD select outside it?
  2. Is there a general selection principle behind the angular ODE — for instance, a functional on the manifold that SGD minimizes — and does it extend to unequal or non-orthogonal means, untied weights, or larger widths (Conjecture 5.2)?
  3. How does the mechanism relate to nonlinear PCA/ICA, where the loss itself selects the rotation? Is stochasticity essential, or would a small perturbation of the model (e.g. biases, untied weights) already break the degeneracy through the loss?
4 comments

Thank you very much to Reviewer Nhff for the detailed review. We have revised the manuscript based on this discussion.

Critical Weaknesses

W1. The main result appears to be incorrect as stated

We thank the reviewer (and Opus 5.5) for pointing out this error. We revised the manuscript to incorporate this change. Specifically, a term (the one involving sin⁡(4θ)\sin(4\theta)) has now been added to the existing limiting flow:

θ˙t=−(κB(θt)+κC(θt))uy′(θt)+(1−ρ2)2s412sin⁡(4θt).\dot{\theta}_t =-(\kappa_B(\theta_t)+\kappa_C (\theta_t))u_y'(\theta_t) + \frac{(1 - \rho^2)^2 s^4}{12}\sin(4\theta_t).

Additionally, we have updated Theorem 3.1 by introducing a threshold on the dimension d, reflecting that the mean-aligned behavior is observed in the high-dimensional regime. This restriction is made to keep the narrative of Theorem 3.1 simple. Lemma 4.3 and Corollary 4.4 provide a more detailed picture of what happens for large or small dd (see Figure 3). In particular, Corollary 4.4 shows that mean alignment happens if d exceeds a threshold dmeand_{mean}, but if d is below a threshold dbisd_{bis} then SGD converges to bisectors. The picture is more nuanced between the two thresholds. Remark 4.5, and Figure 8 (in appendix) include further discussions on this topic.

W2. The contribution relative to existing work is limited and insufficiently positioned. The limiting drift is that of Li, Wang, Arora (ICLR 2022) restricted to normal noise. That SGD noise selects among equivalent minimizers of an autoencoder was already shown by Ghosh, Frei, Ha & Yu (JMLR 2025). The tied-weight objective is the nonlinear PCA criterion (Oja, 1997), for which rotation selection and its link to ICA are classical. The literature on noise-induced drift along symmetry directions (Ziyin et al., 2024; Chen et al., 2023) and on manifold reductions of noisy GD (Shalova, Schlichting & Peletier, 2026) is not discussed.

  • Regarding the comparison with Ghosh et al., two distinctions clarify our contribution. First comes from the conceptual difference, and the second is a key consequence that follows from it.

    First, the crucial conceptual distinction is that our starting point is the consideration of how to represent the given data. Accordingly, we allocate a sufficient number of neurons to represent this data and then investigate how this representation differs across architectures and training algorithms. In contrast, as stated in their work, Ghosh et al. focus on the optimization aspects of training rather than considering an appropriate representation. This is partly because they use only one neuron while their orthonormal data is higher-dimensional, which cannot be represented by a single neuron. This creates different training dynamics than in our setting, as detailed in the appendix.

    Importantly, their result show that dynamics/bias is qualitatively similar for linear and ReLU AE. In their setup, the single-neuron constraint causes the dynamics of ReLU and linear networks to coincide, yielding the same implicit bias for both models. By contrast, we show that while linear AE has no directional bias, SGD on ReLU AE has an implicit directional bias.

  • We have expanded our discussion of these related works in Appendix A and noted the connection to PCA and ICA in the introduction (lines 103–105).

  • We disagree with the reviewer that the analysis follows from Li-Wang-Arora analysis. Analysis of late-stage SGD accommodating the non-smooth ReLU nonlinearity is not covered by the assumptions in Shalova, Schlichting, and Peletier (2026) nor in Li, Wang, and Arora (2022). The distinction is clearly discussed in Remark 4.7, and we believe this framework provides a foundation for analyzing ReLU networks in more complex settings, such as the one discussed in Conjecture 5.2 (whose proof idea we briefly outline in our response to Reviewer YHgr).

  • Finally, while Ziyin et al. (2024) and Chen et al.(2023) are relevant works, their frameworks are not directly applicable to our setting. We have included a detailed discussion of these differences in Appendix A.

W3. The setting is narrow and the analysis is local. The results require minimal width, tied weights, no biases, orthogonal means of equal norm, a weight decay larger than the noise variance, and an initialization near the manifold; the feature-learning phase of Figure 1 is not covered. Two of the stated separations are direct: the absence of drift for the linear autoencoder follows from the exact O(2)O(2)-invariance of its loss, and Prop. 5.1 is a Gaussian tail bound

  • We first point out that the absence of drift for the linear autoencoder does not directly follow from the exact O(2)O(2) invariance of its loss. In fact, the ReLU autoencoder also exhibits O(2)O(2) invariance along its minimizer manifold, yet SGD still induces a distinct drift in the ReLU setting due to the structure of the gradient noise. Additionally, we have revised Proposition 5.1 to explicitly state that Bayes optimality can be achieved in the mean-aligned directions.

  • Second, while we agree that our setting is clean and rather simple, no prior work has provided a complete characterization of the implicit bias of SGD even in this 4-GMM setting. We believe our work serves as an important first step toward a deeper understanding of this phenomenon. Notably, even in the well-studied supervised learning literature, the 4-GMM is a standard benchmark data model. Furthermore, our proof extends naturally to a 2r2r-GMM lying in an rr-dimensional subspace, and analyzing the minimal width regime provides an essential foundation for extending the theory to wider networks. Whereas prior theoretical work on ReLU autoencoders has largely relied on mean-field limits (arXiv:2102.08373), our work approaches the problem from the opposite end by starting from the minimal sufficient representation, which we believe is an important direction to understand how neural networks work.

  • Third, Phase 1 is already addressed in our paper: it consists of a large O(1)O(1) displacement (independent of η\eta) during which SGD tracks gradient flow before entering the O(n)O(\sqrt{n}) local regime. If your question concerns how the iterates reach a neighborhood of this global minimizer manifold in the first place, we omitted that analysis because it is straightforward and not interesting compared to the late-stage SGD dynamics, where the two architectures show a diverging behavior. Specifically, in Lemma E.1 in Appendix E, we show that symmetric random initialization creates an invariant manifold under gradient flow. In this regime, the dynamics reduce to those of a linear network (whose loss landscape is already well-studied and known to have no spurious local minima (Baldi and Hornik, 1989)). Because SGD tracks gradient flow during this phase, extending the analysis to cover convergence to the manifold is straightforward and not the main interest of our work, which centers on late-stage SGD dynamics.

Questions for Authors

Q1. Once Lemma 4.3 is corrected, in which regime of (d,σ,s)(d, \sigma, s) is mean alignment stable, and what does SGD select outside it?

Thanks for this question. This perspective has prompted us to revise the manuscript to explicitly distinguish between the different regimes. We now interpret this phenomenon by introducing the dimensional threshold, which is determined by the mixture parameter and the regularization strength (specifically, once s,σ,λs, \sigma, \lambda are chosen. We also provide a script in the supplementary material to compute this threshold numerically). In the typical high-dimensional regime above this threshold, SGD selects the mean-aligned solution. Conversely, when the dimension dd falls below this threshold, SGD selects the bisector alignment (corresponding to the ICA criterion). You can find more details in Corollary 4.4 and Remark 4.6.

Q2. Is there a general selection principle behind the angular ODE — for instance, a functional on the manifold that SGD minimizes — and does it extend to unequal or non-orthogonal means, untied weights, or larger widths (Conjecture 5.2)?

The general selection mechanism in our analysis is determined jointly by normal curvature and gradient-noise covariance. Specifically, the covariance of the normal fluctuations (at stationarity) satisfies HN(v)Πv+ΠvHN(v)=ΣN(v)H_N(v)\Pi_v+\Pi_vH_N(v)=\Sigma_N(v), and the effective motion satisfies ⟨v˙,τ⟩=−12Tr⁡[(∇τ⊥HN)(v)Πv].\langle\dot v,\tau\rangle=-\frac12\operatorname{Tr}\left[(\nabla^\perp_\tau H_N)(v)\Pi_v\right]. Thus, normal fluctuations induce motion according to how the curvature they experience varies along the minimizer manifold. This is more general then our 4-GMM setting and applies whenever a manifold is normally attracting stationary manifold with the stated assumptions in the main text.

Regarding extensions:

  • Unequal or non-orthogonal means. With central symmetry of the data, Proposition B.6 already characterizes the minimizers for a general signal covariance as W=[QrDR,−QrDR]W=[Q_rDR,-Q_rDR], with R∈O(r)R\in O(r), for rr the dimension of subspace that the signal lives in. Thus, equal norms and orthogonality of the mixture means are not required for that geometric characterization. However, extending the SGD selection result requires (1) verifying normal attraction and the regularity assumptions needed to apply our NASM-based framework, and (2) recomputing the normal Hessian and noise covariance to identify the resulting implicit bias. Therefore, unless these are all clarified, the general theoretical principle in this regime remains open. We however believe this can be extended, since empirical results on Fashion-MNIST in Figure 3 (right panel) ---whose class means are unequal and non-orthogonal---demonstrate that the same trend still holds.

  • Untied weights. With our data model, the optimal reconstruction realized by the tied model is also optimal in the untied ReLU autoencoder (see the proof in the Appendix B). Therefore the minimizer manifold is simply Muntied={ρ[QR,−QR],ρ[QR,−QR]⊤∣R∈O(2)},ρ2=1−λλs\mathcal{M}_{\mathrm{untied}} = \lbrace \rho[QR,-QR], \rho[QR,-QR]^\top \mid R\in O(2) \rbrace, \rho^2 =1-\frac{\lambda}{\lambda_s}. Thus the remaining step is to verify the normal attraction, including in the additional directions introduced by untying the weights. Once these properties are established, the normal decomposition and stochastic reduction can be developed analogously to the tied-weight setting. We do not carry out this additional analysis in the present work. We still provide the code for the experiment in the supplementary material, where we simulate the untied model, which finds the same mean-aligned solutions.

  • Larger widths. Additional degrees of freedom can introduce flat directions or singularities that obstruct the uniform normal attraction required by our theorem. Therefore the larger widths setting cannot not directly be explained by our theory. However, we have verified experimentally that for widths of 4n4n, n≥1n \geq 1, the mean-aligned directions coincide with the minimum Hessian directions (measured by Frobenius norm), whereas for 4n+24n + 2, this is no longer the case. Resolving this remains an open question, and we outline a rough proof idea in our response to Reviewer YHgr. Our framework provides a starting point of this.

Q3. How does the mechanism relate to nonlinear PCA/ICA, where the loss itself selects the rotation? Is stochasticity essential, or would a small perturbation of the model (e.g. biases, untied weights) already break the degeneracy through the loss?

We discuss the connection to nonlinear PCA/ICA in Remark 4.6 of the main text. Notably, in our setting, the ICA criterion selects the bisector directions rather than the mean-aligned directions, which contrasts to our ReLU AE in high-dimension. Regarding model perturbations, we did not include biases because they are unnecessary for representing a centered dataset, so their effect remains an open question. Untying the weights alone, under the symmetric regularization specified above, preserves the degeneracy (see the global minimizer manifold Muntied\mathcal M_{\mathrm{untied}} above, where the degeneracy is maintained), meaning stochasticity remains essential for selecting the rotation.

Comments on the paper

1 comment
Area_Chair_HT6iOpenReview ↗

Discussion phase coming to an end

Dear reviewers YHgr, w7GU and Nhff,

As we are approaching the deadline of the discussion phase, please read the authors’ responses and share your thoughts in the discussion as soon as possible. In particular, please clarify whether the responses have addressed your main concerns. Please also update your review and scores if appropriate.

Your timely input will help us reach a well-informed decision. Thank you for your time and effort!

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.