Implicit Bias of SGD in ReLU Autoencoders

How can useful features emerge without label supervision? We study this question theoretically using shallow ReLU and linear autoencoders trained on data from a 4-component Gaussian mixture model. Under stochastic gradient descent (SGD), the ReLU autoencoder learns neurons aligned with the cluster-mean directions, whereas the linear autoencoder recovers only the signal subspace, without selecting a preferred orientation. Consequently, the ReLU model can learn representations better suited to downstream classification, even though both architectures attain the same optimal reconstruction loss. We show that the implicit directional bias emerges during late-stage SGD but is absent under population gradient flow on the same ReLU model. We explain this implicit bias of SGD by characterizing the global-minimizer manifold of the ReLU autoencoder as a $\text{normally attracting stationary manifold}$ (NASM) for population gradient flow. Building on this geometry, we develop an analysis of SGD with a small step size $\eta$ over $O({\eta }^{-2})$ iterations and prove that stochastic gradient fluctuations induce an effective tangential drift along the global-minimizer manifold. In our setting, the high-dimensional drift reduces to a scalar angular ordinary differential equation whose stable equilibria correspond to the cluster-mean directions. This analysis reveals a mode of unsupervised feature learning during late-stage SGD, in which the representation continues to“get better” even after the unsupervised training loss has plateaued.