- Rating
- 4
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper is studying the implicit bias of SGD in autoencoders. What features are learned? The setting considered is where data is drawn from a Gaussian mixture model with means , , , and for two orthogonal vectors and of equal length. The paper mainly analyses the training dynamics of a shallow autoencoder with ReLU activations trained with SGD. For the data above, the bottleneck width of the autoencoder is 4 (2 if using linear activation instead of ReLU), but for autoencoders that have this bottleneck width there is an entire space of equally good solutions (essentially vectors that are orthogonal and span the same space as and - which allows for arbitrary rotations as long as the span is not changed). SGD with ReLU activations however has an implicit bias to a particular solution (aligning with the data means). This is achieved in a second, relatively slow phase during which the training loss remains essentially unchanged. Linear autoencoders do not show this behaviour.
Critical strengths
I was not able to check all the proofs in detail, but I find this work conceptually very interesting that it is impressive to have such a rigorous account of the training dynamics and implicit bias in this setting. It is, of course, a bit of a toy setting but it is an interesting and meaningful one.
Critical weaknesses
On one hand, the experiments are very useful in illustrating the predicted behaviour. On the other hand, it also shows a gap in our understanding because while SGD with ReLU always did favour particular solutions in the experiments, it was not always as simple as weights go to "the mean of the gaussians". In some settings, they instead "align with the angular bisectors between neighboring cluster directions". It is not a particular strong weakness, as the authors openly point this out and is more and indication that additional work may be needed to deepen our understanding. This does not take away from the fact that this paper is making an important and substantial first step. And it may stimulate follow-up work.
It is disappointing that no code was provided.
Questions for authors
My main question is whether the authors have an explanation or conjecture on the change of the implicit bias behavior for different values of .
We first thank the reviewer YHgr for the appreciation of this work and asking the question that we are also excited to answer.
Critical Weaknesses
We have included the code in the supplementary material to reproduce the experiments shown in Figure 8. I hope this helps.
Questions for Authors
First of all, Conjecture 5.2 in the manuscript summarizes the pattern observed in our experiments with the symmetric Gaussian mixtures. In the tested regimes, the preferred feature directions depend on the number of components: for k=4n, features align with the cluster means, whereas for k=4n+2, they align with the angular bisectors between neighboring cluster means, where n≥1. Note that we observe this behavior when the signal subspace is embedded in a higher-dimensional ambient space (It is not due to the lower-dimensional happening in the 4-GMM case, see Corollary 4.4 and Remark 4.5).
One other important observation is that the sparsity emerges only in the late-stage. This means that, early in the loss plateau, the feature orientations remain dispersed (phase 1). In our minimal width setting, the phase 2 was the rotational movement. However, in this larger widths setting, they concentrate around the preferred 4n or 4n+2 directions, causing the sparsity, but sparsity grows with the number of clusters.
We can also describe a possible route toward extending our analysis, which could also be of separate interest: Our current proof relies on a normally attracting stationary manifold. This means that the population Hessian is uniformly positive definite in the normal directions, creating fast normal relaxation relative to the slow tangential motion. However, wider networks can contain configurations with additional Hessian degeneracies, where this uniform separation no longer applies directly.
Interestingly, if we expand the loss to even higher order, these normal variables still contribute to the loss, meaning that these normal variables are still faster than the tangent variables, thus the degenerate directions still have weakly restoring force. A possible extension would retain these weakly restoring directions and analyze their leading higher-order dynamics, together with their coupling to SGD noise, after eliminating the strongly stable directions. This could produce three(or more) distinct timescales: fast relaxation in directions with positive quadratic curvature, intermediate evolution of the degenerate variables, and slow feature selection. Establishing this separation (and deriving the resulting effective drift) is part of the proposed extension.
We believe this is a promising direction for understanding implicit bias beyond the normally nondegenerate setting. For example, analyses of interpolating models often assume that the Jacobian of the predictions on n training samples has rank n (Li et al., ICLR 2022). Redundant training constraints can violate this assumption, while (near) linear dependencies among the corresponding parameter gradients can lead to poor conditioning. Correlated or redundant training samples, resulting from data augmentation for instance, therefore motivate investigating such regimes (although correlation alone does not imply exact rank deficiency).