acceptodds
Under review as a conference paper at ICLR 2027

The Loss Does Not See the Basis, but Adam Does

Abstract

Different optimization algorithms can find very different solutions to the same problem. Gradient descent (GD) exhibits a low-rank bias in overparameterized problems when initialized near zero, while Adam does not. GD is rotationally equivariant: rotating both factors of a factored model by an orthogonal matrix leaves the loss unchanged, and GD's solution rotates accordingly. In contrast, Adam is not. We prove that any algorithm using only the current gradient and respecting rotational symmetry amounts to a left-preconditioner depending only on the gradient's Gram matrix. This gives a norm-based test, which passes for the spectral norm in Muon and fails for Adam's norm. Empirically, nine optimizers fall into groups consistent with the test on underdetermined matrix sensing. Sharing Adam's second moment across coordinates gradually recovers a low-rank bias, and its solutions approach GD's as the stepsize decreases. On four hyperspectral scenes, Adam's solutions are hundreds to thousands of times further from GD's than two GD runs with different step sizes are from each other. In transformers and LoRA adapters, Adam separates two rotationally equivalent models from the first iteration. Overall, recovering particular solutions depends on the optimizer's invisible coordinate basis, as well as how fast it grows small singular values relative to large ones.

Primary area:
optimization
Keywords:
implicit bias, adaptive methods, Adam, Muon, preconditioning, matrix factorization, low-rank recovery, symmetry, equivariance

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3XtSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies why different optimizers converge to different solutions even when they achieve similar training losses. It studies this question using the "gauge" symmetry within matrix factorizations as a novel perspective. Specifically, the authors consider the matrix factorization model W=UV⊤W=UV^\top. For any orthogonal matrix QQ, replacing the two factors with UQUQ and VQVQ leaves their product unchanged. Therefore, if an optimizer is gauge-equivariant, training from these equivalent parameter representations should produce identical matrix trajectories; otherwise, the optimization process may be affected by the choice of internal coordinates. Building on this, the authors establish a theoretical framework that characterizes the structure of equivariant updates and provides a norm-based test for equivariance. This framework distinguishes equivariant methods such as GD, scalar-Adam, and Muon from coordinate-wise methods such as Adam and RMSProp. The authors also analyze how uniformly adjusting the learning rate affects gradient descent. This adjustment changes only the speed of optimization without changing its path, so some existing conclusions about the path and final solution still apply. The authors also show that equivariance alone does not determine the final solution, and the result also depends on how quickly small singular values grow compared with large ones. The experiments show that equivariant methods generally produce lower effective ranks and better recovery of the ground-truth matrix. Thus, how an optimizer uses coordinate information affects not only its convergence speed but also its implicit bias.

Overall, I find the gauge symmetry perspective interesting, and the theoretical analysis is well supported by the experiments. However, I would like to see a clearer explanation of why gauge-equivariant optimizers tend to produce lower-rank solutions, especially since equivariance alone does not guarantee this behavior.

Critical strengths

S1:The paper provides a systematic framework for understanding gauge equivariance and classifying optimizers according to whether they preserve it.

S2: The theoretical analysis is solid, and the experiments are thorough, covering matrix sensing, real data, attention, and LoRA.

Critical weaknesses

W1: The experiments connects gauge equivariance with low-rank bias, but do not fully explain why breaking this symmetry leads to higher-rank solutions.

W2: The attention and LoRA experiments show that Adam breaks the symmetry, but do not check whether the rank differences observed in matrix sensing also appear in these models.

Questions for authors

Please see weaknesses for my main questions. I also have two additional questions, but I do not consider them as weaknesses:

  1. The Introduction section states that the goal is "to describe which solution an optimizer chooses, rather than which optimizer to use". Building on Appendix E.22.1, do the analysis or experiments offer any preliminary insights into optimizer choice in practice?
  2. Some studies also link low-rank bias to weight decay [1][2]. Have the authors investigated this connection? Does weight decay fit within the proposed theoretical framework, or does it conflict with it? How could the theory be extended to account for weight decay?

[1] Weight decay induces low-rank attention layers. [2] SGD with Weight Decay Secretly Minimizes the Ranks of Your Neural Networks.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 3XtS (1/2)

Thank you for the careful summary and the constructive questions.

W1: The experiments connects gauge equivariance with low-rank bias, but do not fully explain why breaking this symmetry leads to higher-rank solutions.

Adam’s per-coordinate scaling does two things at once: it breaks the symmetry, and it changes the order in which directions are learned. We agree that equivariance alone does not guarantee low-rank solutions. The second ingredient is the spectral schedule, i.e., how fast an update grows small singular directions compared with large ones.

In particular, for GD, the relative rate at which the network learns different directions is given by the corresponding singular values of the model (see, e.g., Li et al., 2021). That is, starting from small initialization, GD will first learn directions corresponding to large singular values, and directions corresponding to small singular values will be learned last. This ordering explains the observed bias towards low-rank solutions.

In contrast, Adam divides each coordinate by its own recent gradient scale. Coordinates with small gradients therefore get relatively large steps, which weakens GD’s large-first ordering. The same per-coordinate scaling also breaks the symmetry, since a rotation changes the coordinates of the gradient but not its singular values. These two effects can be disentangled in two ways.

First, we consider an algorithm that pools Adam’s second moment estimates across rows (row-pooled Adam), which restores equivariance, but still applies a different scaling to each row. This variant closes ≈30% of Adam’s gap to scalar-Adam. It also cuts effective rank from 14.5 to 9.0, against 5.4 for scalar-Adam (Table 18).

Second, we consider an algorithm that pools Adam’s second moment estimates across all coordinates (referred to as “scalar-Adam”), which restores equivariance and also GD’s learning dynamics since all coordinates receive the same scaling. We empirically observe that scalar-Adam closely follows GD’s optimization path (Theorem 3.4 states this formally for the idealized flow), and closes ≈84% of Adam’s gap to GD (recovery error 0.573 for Adam, 0.201 for scalar-Adam, 0.131 for GD). We have added a discussion to Section 4. We also provide a theoretical result (Proposition A.9) which supports our intuition in a solvable two-mode model.

W2: The attention and LoRA experiments show that Adam breaks the symmetry, but do not check whether the rank differences observed in matrix sensing also appear in these models.

Yes, in attention.

As you suggested, we have run additional experiments to compute the effective rank of learned attention weights (i.e., the query-key product WQ⊤WKW_Q^\top W_K of each head). Note that in contrast to low-rank matrix sensing, attention weights are full-rank at initialization. Hence, we compute the effective rank of the change in WQ⊤WKW_Q^\top W_K from initialization. The resulting effective ranks (each within 0.9 across 3 seeds, and the same when all methods are compared at equal training loss) are:

SGDscalar-AdamAdam
12.314.217.1

These results are consistent with our observations for low-rank matrix sensing: Adam results in higher effective ranks compared to SGD, and pooling Adam’s second moment estimator (scalar-Adam) closes ≈60% of Adam’s gap to SGD. We have added these results to Section 5 (changes in blue) and report additional details and the spread across seeds in Appendix D.4.

We also report experiments for LoRA. Since the rank of the learned update is capped at rr there, we ask whether SGD/Adam/scalar-Adam result in equivalent solutions when training twins with different random rotations. We observe that SGD and scalar-Adam result in identical solutions (up to numerical precision), whereas solutions trained with Adam differ significantly (their products BA end 28% to 97% apart, see Table 17).

AuthorsOpenReview ↗

Response to Reviewer 3XtS (2/2)

...do the analysis or experiments offer any preliminary insights into optimizer choice in practice?

Our results provide practitioners with the following insights when choosing an optimizer:

  • Corollary 3.3 provides a simple test to determine whether an optimizer preserves symmetries of factorized models: check whether the optimizer’s update rule is based on a norm that is invariant to rotations from the right.
  • If the goal is to employ an optimizer with adaptive learning rates, but preserve GD’s solutions, one could pool Adam’s second moment estimator (scalar-Adam, see Theorem 3.4 and Section 6).
  • Muon outperforms GD if the ground-truth matrix is (close to) exactly low-rank. However, if more than ≈4% of the signal’s energy is explained by singular values outside the target’s top singular directions, GD outperforms Muon (see Figure 3, left).

Does weight decay fit within the proposed theoretical framework, or does it conflict with it?

Yes, weight decay is invariant to rotations (the penalty does not change) and commutes with rotations (in case of decoupled weight decay). Hence, adding weight decay does not change which of the optimizers we consider are invariant/equivariant: GD (with weight decay) passes our test, whereas AdamW fails our test. Weight decay adds an explicit pull toward low-rank solutions (the mechanism studied in the papers you mention), on top of the implicit bias studied in our work. We did not include weight decay in our experiments to better understand implicit bias in isolation. Thank you for bringing these papers to our attention. We have cited them and added a brief discussion in Section 7.

We hope that we were able to address your comments and questions, and politely ask you to reconsider your score.

References. Li, Luo and Lyu, Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning, ICLR 2021. Galanti, Siegel, Gupte and Poggio, SGD with weight decay secretly minimizes the ranks of your neural networks, Conference on Parsimony and Learning (CPAL), 2025. Kobayashi, Akram and von Oswald, Weight decay induces low-rank attention layers, NeurIPS 2024.

I thank the authors for their detailed response. The additional experiments and explanations made the arguments in the paper more solid, and I would keep my current positive rating. I did not give a 4 because I still believe the theoretical results here in this paper are somewhat restricted and not immediately applicable across different settings.

Reviewer_JRmaOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how the choice of optimizer influences the implicit bias of overparameterized matrix factorization through gauge equivariance. The authors show that different optimization algorithms may behave differently under orthogonal transformations of the factor matrices, even though these transformations leave the represented matrix and loss unchanged. They establish a structural characterization of gauge-equivariant memoryless update rules and analyze several widely used optimizers, including GD, Adam, Muon, and Shampoo. Extensive experiments on matrix sensing, hyperspectral data, transformers, and LoRA further illustrate the relationship between optimizer symmetry, optimization dynamics, and the solutions obtained.

Critical strengths
  1. The paper provides a clear characterization of gauge-equivariant optimization rules, offering a useful perspective on optimizer-dependent implicit bias.
  2. Extensive experiments across multiple optimizers, matrix sensing, hyperspectral data, transformers, and LoRA effectively support the main findings and demonstrate their broader relevance.
Critical weaknesses
  1. While the paper provides a rigorous characterization of gauge-equivariant update rules, the relationship between equivariance and the resulting low-rank implicit bias is primarily investigated empirically. The authors appropriately discuss the limitations of equivariance as a predictor of recovery performance. Nevertheless, a more detailed theoretical understanding of this connection would further strengthen the work.
  2. The experiments on transformers and LoRA provide interesting evidence that optimizer-dependent symmetry breaking also occurs in modern architectures. However, most of these investigations focus on differences in optimization trajectories or learned representations. Additional experiments examining the implications for downstream performance or generalization would further highlight the practical significance of the findings.
Questions for authors
  1. Under what additional assumptions might gauge equivariance lead to a theoretically guaranteed low-rank implicit bias? Could the proposed spectral transfer function provide a starting point for such an analysis?

  2. The experiments demonstrate that different optimizers may select substantially different solutions even when they achieve similar training losses. How closely are these differences related to the generalization performance of the resulting models?

  3. Could the authors provide further discussion or experimental evidence regarding the practical implications of gauge symmetry breaking in larger transformers or pretrained LoRA-based models?

2 comments
AuthorsOpenReview ↗

Response to Reviewer JRma

Thank you for your positive assessment and insightful questions. We have made changes (in blue) to address your comments and questions.

Q1. Under what additional assumptions might gauge equivariance lead to a theoretically guaranteed low-rank implicit bias? Could the proposed spectral transfer function provide a starting point for such an analysis?

Yes. The spectral transfer function hh of Section 3, which we also call the spectral schedule, characterizes how fast an optimizer will amplify small singular directions compared to large singular directions: GD will preferentially amplify large singular directions, while Muon amplifies all singular directions equally.

Two statements that are already proved in the paper:

  1. For any optimizer that scales the gradient with a rotation-invariant scalar (e.g. the idealized form of scalar-Adam), the trajectory is the same as gradient flow (Theorem 3.4). Therefore, such optimizers exhibit the same low-rank bias as gradient flow (e.g. under small initialization; see Li et al., 2021).

  2. For a two-mode setup, GD’s spectral schedule results in learning the large mode first, while Muon’s uniform spectral schedule results in learning both modes simultaneously (Proposition A.9).

Empirically, GD and Muon, the two ends of the spectral schedule, also trade places: Muon recovers better on exactly low-rank targets, and GD once more than about 4% of the target’s energy lies outside its top directions (Figure 3, left).

Q2. How closely are these differences related to the generalization performance of the resulting models?

The different solutions also generalize differently. For matrix sensing, we know the ground truth and measure the error against that: Adam has 57% error, GD has 13%, and scalar-Adam (which is Adam except that it uses a shared second moment instead of a per-coordinate second moment) has 20% error (Figure 1), indicating that most of the gap is attributable to Adam’s per-coordinate rescaling, the same operation that both breaks gauge symmetry and flattens GD’s large-first ordering. For hyperspectral completion, on the two main scenes we consider, GD achieves lower test error than Adam across all cells and random seeds at the training-selected rates of Table 9.

Q3. Could the authors provide further discussion or experimental evidence regarding the practical implications of gauge symmetry breaking in larger transformers or pretrained LoRA-based models?

Training with Adam has two practical consequences: (i) the trained model depends on an arbitrary choice of basis that the loss cannot see, and (ii) the learned attention updates have higher effective rank.

  • Gauge symmetry is present in standard pretrained transformers, including ones larger than those we train. For example, we show that in the publicly available GPT-2-small model, rotating the query and key weights within each attention head does not change the model outputs (held-out loss moves by 1.2×10−71.2\times10^{-7} nats, Appendix E.10). Consequently, gauge symmetry breaking by Adam also applies to training such models.
  • In our experiments with up to 6 layers and width 256 (Table 8), we observe symmetry breaking by Adam, even after 300 shared training steps (Appendix E.10). We expect that this phenomenon persists as model and dataset size increase since Adam breaking gauge symmetry follows from its update rule (Proposition A.2).
  • Properties shown to hold for GD-trained attention weights, e.g. low-rank bias, may not hold for Adam-trained attention weights but do hold under the idealized scalar-Adam flow (Theorem 3.4). This could be practically relevant e.g. when compressing attention weights to be low-rank. In response to Reviewer 3XtS, we added an experiment showing that attention updates learned by Adam have higher effective rank (17.1) than those learned by SGD (12.3; averaged over three random seeds), echoing our observation in matrix sensing.
  • Our proposed twin test for symmetry breaking requires only a few training runs, so it could be applied to any pretrained transformer or LoRA.

We hope that the above addresses your concerns and questions, and we would appreciate it if you would reconsider your score.

References. Li, Luo and Lyu, Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning, ICLR 2021.

Thank you for the detailed responses and additional experiments. The clarifications have adequately addressed my main concerns and strengthened the paper. I will maintain my positive assessment and current rating.

Reviewer_YSfTOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how orthogonal reparameterizations affect optimization in factored models. It classifies optimizers by gauge equivariance, characterizes memoryless equivariant updates through the gradient Gram matrix, and examines matrix recovery, hyperspectral completion, attention, and LoRA. It also applies the symmetry criterion to modify FlowAdam.

Critical strengths
  1. The connection between optimizer symmetry and implicit bias is an interesting question. Extending the analysis to attention and LoRA gives the work relevance beyond matrix sensing.
  2. The experiments include various ablation settings, such as matched training losses, learning-rate and spectral-tail sweeps, and noise twins.
Critical weaknesses
  1. The optimizer classification in contribution (1) has limited novelty. The distinction between GD's rotational equivariance and Adam's coordinate dependence, as well as scalar-moment modifications, is already established by Ling et al. (2022) and Xie et al. (2025). The paper acknowledges these works, and applying the same properties to orthogonal changes of the factor basis seems straightforward.
  2. Theorem 3.2 also appears to follow directly from existing results. Lemma 1 and Proposition 4 of Villar et al. (2021) express an orthogonally equivariant vector output as a linear combination of input vectors, with coefficients depending on their pairwise inner products. Applying this result to each output row gives exactly Φ(G)=H(GG⊤)G\Phi(G)=H(GG^\top)G, including rank-deficient inputs. The authors should clarify what their characterization adds on top.
  3. The stated goal is to describe “which solution a given optimizer will choose” (lines 112–115), but the results do not characterize this choice very precisely. GD is already known to favor low-rank solutions when initialized near zero, so good recovery of a low-rank ground truth is consistent with existing knowledge rather than a more specific characterization of the selected solution. Figure 1 (§4) does not establish that equivariance explains the observed low-rank bias or recovery advantage. The equivariant factor-aware preconditioner has higher recovery error than Adam (0.609 versus 0.573 in Appendix E.16), while non-equivariant signum reaches near-exact recovery with longer annealing (Appendix D.11). As acknowledged in Remark 3.5, these examples show that the classification alone cannot predict which optimizer recovers the ground truth better.
  4. The attention and LoRA experiments mainly establish basis dependence, while its broader significance remains unclear. The attention-merging penalty from training twins from scratch disappears in the reported common-prefix and pretrained settings. LoRA twins have similar held-out losses and negligible product-merging cost. These extensions therefore provide limited evidence that the matrix-recovery explanation predicts meaningful outcomes in the broader applications.
  5. The FlowAdam repair has limited demonstrated value. Replacing entrywise clipping with norm clipping is motivated by symmetry, but the validation remains focused on matrix recovery. Its advantage over the simpler scalar-Adam intervention is no longer obvious with finer learning-rate tuning; e.g., at n=40n=40, the reported errors are 0.148 for FlowAdam-p and 0.143 for the simpler intervention (Appendix E.22). It is unclear what additional benefit this application provides.
  6. The writing of the abstract and contributions are hard to follow and need a clearer logical flow. Known observations about low-rank bias and equivariance are listed without clearly explaining the new claim connecting them. The large distances from GD establish different fitted solutions, but do not themselves establish worse recovery or generalization.

References:

  • Ling et al. “VectorAdam for Rotation Equivariant Geometry Optimization.” NeurIPS 2022.
  • Xie et al. “Adam Exploits ℓ∞-geometry of Loss Landscape via Coordinate-wise Adaptivity.” ICLR 2025.
  • Villar et al. “Scalars are universal: Equivariant machine learning, structured like classical physics.” NeurIPS 2021.
Questions for authors
  1. Which specific properties of the selected solution can the framework predict beyond known low-rank bias and basis dependence?
  2. Does the FlowAdam repair offer a benefit over scalar-Adam under comparable tuning, and does that benefit extend beyond matrix recovery?
5 comments
AuthorsOpenReview ↗

Response to Reviewer YSfT (1/2)

Thank you for the careful reading of our paper and helpful comments and questions.

Theorem 3.2 also appears to follow directly from existing results.

Agreed, and it is now credited in the theorem, its proof and the contributions. Our characterization of optimizers preserving symmetry (Theorem 3.2) does build upon Villar et al.’s characterization of equivariant functions. However, our results go beyond theirs by applying their characterization to gradient-based optimization algorithms (which Villar et al. do not consider), yielding new insights for practitioners about popular gradient-based optimizers’ behaviors under symmetry. Specifically, our paper shows:

  • Gradient-based optimizers that rescale the gradient by rotation-invariant scalars (e.g., the idealized form of scalar-Adam) preserve symmetry, taking the same trajectory as gradient flow under infinitesimal step sizes (Theorem 3.4), hence converging to GD’s solution.
  • Optimizers performing steepest descent w.r.t. a norm preserve symmetry if the norm is invariant under right rotation (Corollary 3.3). This result immediately characterizes optimizers like GD and Muon, as well as, through its stateful part, less straightforward optimizers like damped Shampoo.
  • Exactly how two Adam runs differ after one step, when one starts from a rotated copy of the other’s initialization (Proposition A.2).

The optimizer classification in contribution (1) has limited novelty.

They study a different question. Unlike Ling et al. (2022) and Xie et al. (2025), which respectively focus on artifacts and convergence in geometry optimization and on training speed, our work aims to understand optimizers’ effects on learned solutions, especially in factored models where the symmetry is within a single model (i.e., two rotation-related initializations correspond to the same function). For example, our theory predicts scalar-Adam will follow the same trajectory as GD at small step sizes (Theorem 3.4), confirmed by experiments showing scalar-Adam and GD agree on held-out error and effective rank in all 16 seed-paired runs on real data, while Adam’s solutions are hundreds to thousands of times farther from GD’s (Section 6).

Which specific properties of the selected solution can the framework predict beyond known low-rank bias and basis dependence?

Besides characterizing which optimizers preserve symmetry, our framework can also:

  • Predict when two optimizers find the same solution. For example, scalar-Adam finds GD’s solution (its distance to GD’s solution shrinks steadily as the step size decreases), while Adam’s solution stays far from GD’s at every step size (Figure 3, right).
  • Predict which symmetries will be preserved by optimizers that do not generally preserve symmetry. For example, Adam always finds the same solution under sign/permutation symmetries but can find different solutions under general rotations of the same size (fitted matrices agree to 10−1610^{-16} in one case and sit 33% apart in the other, as predicted by our norm-based characterization in Appendix E.7).
  • Provide insights for practitioners about when to use symmetry-preserving optimizers. For example, although both Muon and GD preserve symmetry, Muon can have advantages and disadvantages compared to GD depending on problem structures: Muon excels when the target matrix is (approximately) low-rank but underperforms GD when more than ~4% of the target matrix’s energy lies outside its leading principal directions (Figure 3, left). They sit at the two ends of the spectral schedule, i.e., how fast an optimizer grows small singular directions compared with large ones. Such insights help reconcile recent seemingly contradictory results: Kang et al. (2026) showed Muon grows all singular values at the same rate (which suits exactly low-rank targets), while Dragutinović et al. (2026) showed Muon can forgo GD’s simplicity bias depending on the data. Our results quantify when the former insight dominates and when the latter does.

Figure 1 (§4) does not establish that equivariance explains the observed low-rank bias or recovery advantage.

Remark 3.5 discusses both cases you raise, and neither contradicts our two axes (symmetry and spectral schedule). The former (the factor-aware preconditioner) still preserves symmetry, but has a poor spectral schedule and poor empirical performance; the latter (signum with a long annealing phase) recovers well but carries momentum, so it does not fall into the class of memoryless update rules covered by Theorem 3.2. Its recovery also needs an exactly low-rank target, and degrades as soon as a small spectral tail is added (Appendix D.11). The symmetry axis also has a direct control: row-pooled Adam restores only the symmetry, keeping a separate scale per row, and this alone cuts effective rank from 14.5 to 9.0, against 5.4 for scalar-Adam (Table 18).

AuthorsOpenReview ↗

Response to Reviewer YSfT (2/2)

The attention and LoRA experiments mainly establish basis dependence, while its broader significance remains unclear.

We have added an experiment (over three seeds) showing that the updates Adam learns for the query-key product in attention have a much higher effective rank (17.1) than those SGD learns (12.3), and that scalar-Adam closes ~60% of this gap. We discuss this new experiment in Section 5 (blue).

This adds to the basis dependence we show with rotated twins. These are two runs of the same model trained on the same dataset, except one of them starts from the other run’s initial weights with the query and key weights rotated. Since this rotation does not change the loss, these are equivalent starting points. We show that SGD and scalar-Adam always converge to (essentially) the same solution from these starting points, whereas Adam converges to noticeably different solutions (for example, the query-key products it converges to are 56% different from each other). Even after 300 shared training steps on text, the twins’ query-key products remain 28% apart (Appendix E.10). So results proven for GD on these layers cannot be assumed for Adam.

The large distances from GD establish different fitted solutions, but do not themselves establish worse recovery or generalization.

Agreed, distance alone only shows that the solutions differ. Our experiments show that Adam converges to worse solutions than GD: higher recovery error when the ground truth is known (57% for Adam, 13% for GD, see Figure 1).

Does the FlowAdam repair offer a benefit over scalar-Adam under comparable tuning, and does that benefit extend beyond matrix recovery?

We recommend scalar-Adam instead of FlowAdam. FlowAdam was meant to show that our test diagnoses what’s wrong with an existing, practical optimizer (in this case pinpointing that coordinate-wise clipping is the problematic operation, and replacing it with global norm clipping reduces recovery error). FlowAdam has been removed from the contributions and relegated to the appendix.

The writing of the abstract and contributions are hard to follow and need a clearer logical flow.

The paper now makes one argument, in five steps:

  1. Since rotating the factors does not change the loss, optimizers which are not equivariant under such a rotation can converge to different solutions from two equivalent starting points.
  2. We provide a characterization of which optimizers are rotation-equivariant (Theorem 3.2, Corollary 3.3), and show that Adam is not, due to its coordinate-wise scaling. In particular, on a matrix sensing task, we show Adam converges to a much worse solution than GD (57% vs. 13% recovery error, see Figure 1).
  3. Removing Adam’s coordinate-wise scaling (scalar-Adam) makes it follow GD’s trajectory (Theorem 3.4 proves this for the idealized flow), and it empirically matches GD on held-out error and effective rank (in all 16 seed-paired runs on real data).
  4. Among rotation-equivariant optimizers, we characterize how the spectral schedule impacts which solution the optimizer converges to (comparing GD and Muon).
  5. We show that our symmetry-breaking characterization also applies to attention and LoRA, and that Adam again learns higher-rank updates in attention (new experiment, Section 5).

We hope the corrected attribution and these answers address your concerns, and we would be grateful if you would reconsider your score.

References. Kang et al., Uniform spectral growth and convergence of Muon in LoRA-style matrix factorization, arXiv:2602.06385, 2026. Dragutinović, Zhang and Ranganath, To use or not to use Muon: How simplicity bias in optimizers matters, arXiv:2603.00742, 2026.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer YSfT,

Thank you again for your review. As the discussion period ends in a few days, we wanted to make sure our response reached you. It answers your question on what the framework predicts beyond low-rank bias, and credits Villar et al. for Theorem 3.2 while stating the new results on the optimizer side. On your point that the significance of our attention results was unclear, our response shows why the basis dependence matters there too, with a new experiment showing that Adam learns higher-rank attention updates than SGD.

You also noted that a large distance from gradient descent shows a different solution, but not a worse one. Figure 1 shows that Adam's solution is both different and worse. When the ground truth is known, Adam's recovery error is 57%, against 13% for gradient descent.

On the writing, Section 3 now defines the spectral schedule, Section 4 explains why Adam loses the low-rank bias of gradient descent, and FlowAdam is removed from the contributions, along with other clarity edits so that the paper makes one clear point.

If our response addresses your concerns, we would be grateful if you would consider increasing your score. We are happy to answer any remaining questions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their insightful comments.

Strengths

Reviewers found our question to be interesting (YSfT, 3XtS), our theory to be solid (3XtS) and our characterization of optimizers’ equivariant update rules to be clear and rigorous (JRma). All three reviewers appreciated the breadth and thoroughness of our experiments.

The claim

Rotating both factors of W = UVᵀ by the same orthogonal matrix leaves the loss unchanged. Which solution an optimizer returns depends on two things:

  1. Symmetry: can the update see this rotation? We give an exact test. GD, scalar-Adam, Muon and Shampoo cannot see it. Adam, RMSProp, signSGD, Lion and Adafactor can.
  2. Spectral schedule: how fast does the update grow small singular directions compared with large ones? GD grows large ones first. Muon grows all at the same rate.

Changes in the revision (in blue)

  • Theorem 3.2 now cites Villar et al. (2021). [YSfT]
  • Sec. 3 defines the notion of “spectral schedule”, and later sections refer back to it. [YSfT]
  • Sec. 4 explains the reason for Adam not having the same low-rank bias as GD, and how our proposed controls isolate these two effects. [3XtS, JRma, YSfT]
  • Sec. 5 is renamed “The gauge in transformers and LoRA”. [YSfT]
  • Sec. 5 adds a new experiment in attention layers. Averaged over three seeds, the updates Adam learns have higher effective rank (17.1) than those SGD learns (12.3), with scalar-Adam in between (14.2). Details are in App. D.4. [3XtS, YSfT]
  • The abstract now also states the new attention result. [3XtS, YSfT]
  • Sec. 7 adds a discussion about weight decay and the next theoretical step. [3XtS, JRma]
  • The transfer theorem states its rotation condition as a separate (last) item in its conclusions, similarly to how it appears in its proof. [YSfT]
  • Fig. 1 is retitled “Recovery error of nine optimizers”. [YSfT]
  • FlowAdam is removed from our list of contributions and now appears only in the appendix. [YSfT]
  • A few informal words are replaced by more straightforward ones (not shown in blue). [YSfT]
AuthorsOpenReview ↗

Summary of the discussion for the AC

Dear AC,

We thank the reviewers for their thoughtful reviews and an engaged discussion. All three reviewers replied to our responses. Reviewers JRma and 3XtS kept their positive ratings, and reviewer YSfT raised their score from 1 to 2. The main changes made during the discussion are in blue in the revised PDF. Below is a brief summary per reviewer, with links to the full replies.

Reviewer YSfT (2, raised from 1)

  • Strengths: found the connection between optimizer symmetry and implicit bias "an interesting question", and noted that "extending the analysis to attention and LoRA gives the work relevance beyond matrix sensing".
  • Novelty: Theorem 3.2 now cites Villar et al. YSfT agreed that our setting differs from Ling et al. and Xie et al. However, they note that equivariant optimizers reaching the same solution from rotated starts follows directly from update equivariance. We agree. Our contributions are on the optimizer side: a one-line test for which optimizers preserve the symmetry (Corollary 3.3), which optimizers follow the path of gradient descent (Theorem 3.4), and exactly how two Adam runs started from rotated copies of the same model differ after one step (Proposition A.2).
  • How the analysis explains the rank difference (their remaining question): answered in our Nov 16 reply. Adam's per-coordinate scaling breaks the symmetry, and it also disrupts the order in which gradient descent learns directions, largest first. Our controls separate the two. Restoring only the symmetry closes about 60% of Adam's effective-rank gap to scalar-Adam (Adam with one shared scale for the whole gradient rather than one per coordinate), and about 30% of its recovery-error gap (Sec. 4, Table 18).
  • What the analysis predicts (their question): it predicts that scalar-Adam follows the path of gradient descent (Theorem 3.4 proves this for its idealized flow). On real data this holds: scalar-Adam matches gradient descent on held-out error and effective rank in all 16 seed-paired runs, while Adam's solutions are hundreds to thousands of times farther from gradient descent's than scalar-Adam's are (Sec. 6). The test also predicts that Adam ignores sign flips and permutations but not rotations, and the experiments confirm it (Appendix E.7).
  • FlowAdam: we agree that the simpler scalar-Adam is the better recommendation, so FlowAdam now appears only in the appendix and not as a contribution.
  • Writing: Section 3 now defines the spectral schedule, Section 4 explains why Adam loses the low-rank bias of gradient descent, and the abstract states the new attention result.

Reviewer 3XtS (3)

  • Strengths: found the gauge symmetry perspective "interesting", the theory "solid" and the experiments "thorough".
  • Why breaking the symmetry leads to higher rank: answered by the same mechanism and controls as in the YSfT summary above (Sec. 4, Table 18).
  • Whether the rank gap exists beyond matrix sensing: yes, in attention. We ran the experiment they suggested. Adam learns higher-rank attention updates than SGD, and the ordering holds at equal training loss (Sec. 5, Appendix D.4).
  • 3XtS kept their positive rating, noting that "the additional experiments and explanations made the arguments in the paper more solid". They then asked which optimizer to use in practice, Muon or AdamW. We gave a concrete recommendation, supported by a new run with weight decay (reply).

Reviewer JRma (3)

  • Strengths: found our characterization of equivariant optimization rules "clear", and the experiments to "effectively support the main findings and demonstrate their broader relevance".
  • Replied that our responses "adequately addressed my main concerns and strengthened the paper".

Thank you for your time and effort as AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.