- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies why different optimizers converge to different solutions even when they achieve similar training losses. It studies this question using the "gauge" symmetry within matrix factorizations as a novel perspective. Specifically, the authors consider the matrix factorization model . For any orthogonal matrix , replacing the two factors with and leaves their product unchanged. Therefore, if an optimizer is gauge-equivariant, training from these equivalent parameter representations should produce identical matrix trajectories; otherwise, the optimization process may be affected by the choice of internal coordinates. Building on this, the authors establish a theoretical framework that characterizes the structure of equivariant updates and provides a norm-based test for equivariance. This framework distinguishes equivariant methods such as GD, scalar-Adam, and Muon from coordinate-wise methods such as Adam and RMSProp. The authors also analyze how uniformly adjusting the learning rate affects gradient descent. This adjustment changes only the speed of optimization without changing its path, so some existing conclusions about the path and final solution still apply. The authors also show that equivariance alone does not determine the final solution, and the result also depends on how quickly small singular values grow compared with large ones. The experiments show that equivariant methods generally produce lower effective ranks and better recovery of the ground-truth matrix. Thus, how an optimizer uses coordinate information affects not only its convergence speed but also its implicit bias.
Overall, I find the gauge symmetry perspective interesting, and the theoretical analysis is well supported by the experiments. However, I would like to see a clearer explanation of why gauge-equivariant optimizers tend to produce lower-rank solutions, especially since equivariance alone does not guarantee this behavior.
Critical strengths
S1:The paper provides a systematic framework for understanding gauge equivariance and classifying optimizers according to whether they preserve it.
S2: The theoretical analysis is solid, and the experiments are thorough, covering matrix sensing, real data, attention, and LoRA.
Critical weaknesses
W1: The experiments connects gauge equivariance with low-rank bias, but do not fully explain why breaking this symmetry leads to higher-rank solutions.
W2: The attention and LoRA experiments show that Adam breaks the symmetry, but do not check whether the rank differences observed in matrix sensing also appear in these models.
Questions for authors
Please see weaknesses for my main questions. I also have two additional questions, but I do not consider them as weaknesses:
- The Introduction section states that the goal is "to describe which solution an optimizer chooses, rather than which optimizer to use". Building on Appendix E.22.1, do the analysis or experiments offer any preliminary insights into optimizer choice in practice?
- Some studies also link low-rank bias to weight decay [1][2]. Have the authors investigated this connection? Does weight decay fit within the proposed theoretical framework, or does it conflict with it? How could the theory be extended to account for weight decay?
[1] Weight decay induces low-rank attention layers. [2] SGD with Weight Decay Secretly Minimizes the Ranks of Your Neural Networks.
6 comments
Response to Reviewer 3XtS (2/2)
...do the analysis or experiments offer any preliminary insights into optimizer choice in practice?
Our results provide practitioners with the following insights when choosing an optimizer:
- Corollary 3.3 provides a simple test to determine whether an optimizer preserves symmetries of factorized models: check whether the optimizer’s update rule is based on a norm that is invariant to rotations from the right.
- If the goal is to employ an optimizer with adaptive learning rates, but preserve GD’s solutions, one could pool Adam’s second moment estimator (scalar-Adam, see Theorem 3.4 and Section 6).
- Muon outperforms GD if the ground-truth matrix is (close to) exactly low-rank. However, if more than ≈4% of the signal’s energy is explained by singular values outside the target’s top singular directions, GD outperforms Muon (see Figure 3, left).
Does weight decay fit within the proposed theoretical framework, or does it conflict with it?
Yes, weight decay is invariant to rotations (the penalty does not change) and commutes with rotations (in case of decoupled weight decay). Hence, adding weight decay does not change which of the optimizers we consider are invariant/equivariant: GD (with weight decay) passes our test, whereas AdamW fails our test. Weight decay adds an explicit pull toward low-rank solutions (the mechanism studied in the papers you mention), on top of the implicit bias studied in our work. We did not include weight decay in our experiments to better understand implicit bias in isolation. Thank you for bringing these papers to our attention. We have cited them and added a brief discussion in Section 7.
We hope that we were able to address your comments and questions, and politely ask you to reconsider your score.
References. Li, Luo and Lyu, Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning, ICLR 2021. Galanti, Siegel, Gupte and Poggio, SGD with weight decay secretly minimizes the ranks of your neural networks, Conference on Parsimony and Learning (CPAL), 2025. Kobayashi, Akram and von Oswald, Weight decay induces low-rank attention layers, NeurIPS 2024.
I thank the authors for their detailed response. The additional experiments and explanations made the arguments in the paper more solid, and I would keep my current positive rating. I did not give a 4 because I still believe the theoretical results here in this paper are somewhat restricted and not immediately applicable across different settings.
Response to Reviewer 3XtS (1/2)
Thank you for the careful summary and the constructive questions.
Adam’s per-coordinate scaling does two things at once: it breaks the symmetry, and it changes the order in which directions are learned. We agree that equivariance alone does not guarantee low-rank solutions. The second ingredient is the spectral schedule, i.e., how fast an update grows small singular directions compared with large ones.
In particular, for GD, the relative rate at which the network learns different directions is given by the corresponding singular values of the model (see, e.g., Li et al., 2021). That is, starting from small initialization, GD will first learn directions corresponding to large singular values, and directions corresponding to small singular values will be learned last. This ordering explains the observed bias towards low-rank solutions.
In contrast, Adam divides each coordinate by its own recent gradient scale. Coordinates with small gradients therefore get relatively large steps, which weakens GD’s large-first ordering. The same per-coordinate scaling also breaks the symmetry, since a rotation changes the coordinates of the gradient but not its singular values. These two effects can be disentangled in two ways.
First, we consider an algorithm that pools Adam’s second moment estimates across rows (row-pooled Adam), which restores equivariance, but still applies a different scaling to each row. This variant closes ≈30% of Adam’s gap to scalar-Adam. It also cuts effective rank from 14.5 to 9.0, against 5.4 for scalar-Adam (Table 18).
Second, we consider an algorithm that pools Adam’s second moment estimates across all coordinates (referred to as “scalar-Adam”), which restores equivariance and also GD’s learning dynamics since all coordinates receive the same scaling. We empirically observe that scalar-Adam closely follows GD’s optimization path (Theorem 3.4 states this formally for the idealized flow), and closes ≈84% of Adam’s gap to GD (recovery error 0.573 for Adam, 0.201 for scalar-Adam, 0.131 for GD). We have added a discussion to Section 4. We also provide a theoretical result (Proposition A.9) which supports our intuition in a solvable two-mode model.
Yes, in attention.
As you suggested, we have run additional experiments to compute the effective rank of learned attention weights (i.e., the query-key product WQ⊤WK of each head). Note that in contrast to low-rank matrix sensing, attention weights are full-rank at initialization. Hence, we compute the effective rank of the change in WQ⊤WK from initialization. The resulting effective ranks (each within 0.9 across 3 seeds, and the same when all methods are compared at equal training loss) are:
These results are consistent with our observations for low-rank matrix sensing: Adam results in higher effective ranks compared to SGD, and pooling Adam’s second moment estimator (scalar-Adam) closes ≈60% of Adam’s gap to SGD. We have added these results to Section 5 (changes in blue) and report additional details and the spread across seeds in Appendix D.4.
We also report experiments for LoRA. Since the rank of the learned update is capped at r there, we ask whether SGD/Adam/scalar-Adam result in equivalent solutions when training twins with different random rotations. We observe that SGD and scalar-Adam result in identical solutions (up to numerical precision), whereas solutions trained with Adam differ significantly (their products BA end 28% to 97% apart, see Table 17).