acceptodds
Under review as a conference paper at ICLR 2027

Optimizer Geometry Sets the Pace: Spectral Learning Dynamics in Matrix Factorization

Abstract

Recent successes of matrix- and curvature-based optimizers have renewed interest in how update geometry shapes learning. These methods normalize or precondition updates, changing how different components progress during training. In deep matrix factorization, the geometry that slows gradient descent (GD) favors low-rank solutions by delaying the emergence of small singular modes. This raises the question of what remains of that spectral bias when normalization or curvature correction weakens or removes the slowdown. We study how optimizer geometry and depth jointly govern this behavior through a common framework for singular mode dynamics. Under explicit balance and alignment assumptions, we derive birth, saturation, and decay laws for Euclidean GD, coordinate-wise updates of SignGD and an instantaneous Adam approximation, spectral updates of Muon and cumulative Shampoo, and a block curvature model of K-FAC. The resulting picture is not a simple ordering from stronger to weaker low-rank bias. To highlight, SignGD and ideal Muon eliminate the divergent birth barrier and drive unsupported modes to zero in finite time. Cumulative Shampoo initially retains GD's depth-dependent barrier, then accumulated gradients produce a catch-up phase while making previously active modes increasingly persistent. Undamped K-FAC cancels the factorization-induced slowdown while preserving the ordering of the target singular values, whereas positive damping introduces a spectral threshold below which the slow GD phase laws reappear. These results give normalization, accumulated state, damping, and depth a direct interpretation as controls determining when modes emerge, persist, and disappear during training.

Primary area:
learning theory
Keywords:
Deep matrix factorization, implicit bias, optimizer geometry, spectral learning dynamics, spectral optimization, adaptive optimization, curvature-aware optimization, Muon, Shampoo, K-FAC

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_5nPFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how optimizer geometry and network depth shape spectral learning dynamics in deep matrix factorization. Under balance and alignment assumptions, it develops a unified framework for comparing multiple optimizers by separating the birth, decay, and saturation of singular modes. The analysis characterizes how normalization, accumulated optimizer state, and damping affect the emergence, removal, and convergence of these modes. The authors also conduct toy experiments in deep matrix factorization to numerically verify the predicted phase laws and transitions.

Critical strengths

The paper analyzes multiple optimizers within a unified deep matrix factorization framework, enabling a systematic comparison of their spectral learning dynamics. Separating these dynamics into birth, decay, and saturation provides a useful perspective on how different optimizers govern the emergence, removal, and convergence of singular modes. This distinction highlights that an optimizer can affect different stages of learning in different ways.

Critical weaknesses
  • The theoretical analysis relies on strong assumptions that substantially simplify the dynamics. In particular, Assumption 2 initializes the factors in singular bases aligned with the target, while Assumption 3 requires the joint evolution of the factors and optimizer states to preserve this aligned structure. These assumptions eliminate singular-vector dynamics in every weight matrix, leaving only scalar mode amplitudes to evolve. Consequently, Theorem 1 follows directly from substituting the assumed factorization into the gradients and differentiating the product amplitudes, which limits its substantive theoretical contribution. Several additional conditions are also insufficiently specified in the main text or theorem statements. The Muon analysis excludes momentum and replaces the Newton–Schulz approximation with exact polar normalization, with the treatment of rank-deficient gradients specified only in the appendix. Theorem 2’s Adam results require the instantaneous approximation m=g, v=g2m=g,\ v=g^2, rather than the finite-memory dynamics defined in Table 1, but this restriction is absent from the theorem statement. Likewise, the undamped K-FAC update requires invertibility or a specified pseudo-inverse convention. These conditions should be stated explicitly alongside the results they support.

  • The paper’s main contribution remains unclear to me. Much of the core GD and Muon analysis recovers phenomena already established in prior work, including depth-dependent spectral learning and low-rank bias under GD [1,2], and uniform factor growth and the disappearance of slow sequential learning under simplified Muon [3–5]. More broadly, describing how singular values emerge, saturate, and decay under different optimizers leaves the consequences of those differences insufficiently resolved. The discussion and nonaligned regression experiment provide some evidence of relevance, but do not establish when these dynamics improve or harm generalization, affect solution selection, or inform optimizer and hyperparameter choices.


References

[1] Arora et al., Implicit Regularization in Deep Matrix Factorization. NeurIPS 2019.

[2] Li et al., Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learning. ICLR 2021.

[3] Kang et al., Uniform Spectral Growth and Convergence of Muon in LoRA-Style Matrix Factorization. arXiv:2602.06385.

[4] Rhee et al., Muon Learns Balanced Solutions in Matrix Factorization without Slow Saddle-to-Saddle Dynamics. arXiv:2606.30509.

[5] Vasudeva et al., How Muon’s Spectral Design Benefits Generalization: A Study on Imbalanced Data, ICLR 2026.

Questions for authors
  1. What are the main technical advances over prior analyses? In particular, what new insights does the unified framework provide?
  2. What consequences follow from the proposed phase laws beyond characterizing singular-value trajectories? Can the authors connect these dynamics to concrete outcomes, such as solution selection and optimizer and hyperparameter choices, and clarify which implications are supported by the experiments?
  3. Could the authors explicitly state the optimizer-specific conditions required by the theorems, including momentum-free exact-polar Muon, instantaneous Adam, and the inverse or pseudoinverse convention for undamped K-FAC?
3 comments
AuthorsOpenReview ↗

Response to Q1 main technical advances

We thank the reviewer for the detailed feedback. We have addressed the comments and questions in the revised version. All updates are marked in magenta. Below we clarify the changes made.

Q1: Main technical advances over prior analyses and new insights

We agree that sequential mode learning under GD and uniform growth under ideal exact Muon without momentum are established results. We use these as reference cases as stated in line 113. The main technical advances are the exact laws for how accumulated state of Shampoo, additive damping of K-FAC and stabilization parameter of Adam change spectral learning. Furthermore, while the dynamics of GD and ideal Muon are established, their decay law is also a unique contribution of our work. Most work focus only on the emergence of small modes.

Technical advances over prior analyses and insights

  1. Cumulative Shampoo links optimizer history to all three phases. Proposition 8 derives the evolving preconditioner denominator exactly in terms of the current mode amplitude (eq. 8, page 34). This gives three predictions from the same identity. Growth initially follows GD scaling and then becomes independent of target strength. Near a positive target, the local saturation rate can decrease with target strength, reversing the GD ordering. For a zero target, the accumulated state increases the time required to remove a mode, even when its starting amplitude and removal tolerance are held fixed. This quantifies how optimizer history can change the growth, convergence, and persistence of individual modes.

  2. Adam’s stabilizer and K-FAC damping add the same GD time penalty through different mechanisms. Propositions 7 on page 31 and 9 on page 38 give the following identities for the same passage endpoints in each of the three phases.

\qquad T_{\mathrm{KF}}=T_{\mathrm{KF},\mathrm{undamped}}+\lambda_{\mathrm{KF}}T_{\mathrm{GD}}.$$ At any fixed positive stabilizer, the added term makes birth and decay times diverge similar to GD at depth two or greater. The crossover explains where the two methods differ. Instantaneous Adam compares $\epsilon$ with the layer gradient magnitude $|\mu-\sigma|\sigma^q$, where $q=1-1/L$ (line 367), so even a strong mode returns to scaled GD near its target as the residual vanishes. Additive K-FAC damping competes with the block curvature $\sigma^{2q}$ (line 406), so curvature cancellation can remain effective near a sufficiently large target. At same initialization and a common threshold, Adam’s preference for larger targets is exactly $\epsilon$ times the corresponding GD birth time gap. Undamped KFAC already favors larger targets. 3. **What Muon and undamped K-FAC comparison reveals.** Ideal Muon and undamped K-FAC both give *bounded birth times* as the initialization tends to zero. From Theorem 2, at same initialization, ideal Muon reaches a common absolute threshold at the same time across supported modes, while undamped block curvature still favors larger targets. Ideal Muon also *removes unsupported modes in finite time*, while undamped block curvature removes them *exponentially*. The common analysis identifies which of these properties follows from each update rule under the same objective and initialization.
AuthorsOpenReview ↗

Response to Q2 consequences of derived phase laws and Q3 optimizer-specific conditions

Q2 and W2: Consequences of the phase laws in solution selection, optimizer and hyperparameter choices

Our results show that optimizer settings can change which components a model learns or removes during training. This matters when training stops early or the objective changes during fine tuning. Based on this, we have now included a new section on the 'Practical Implications of Phase Laws' (pages 9-10) going beyond the singular value trajectories and new experiments using nonlinear classifiers in G.3.

  1. Optimizer state determines how much of the previously learned features survives adaptation. The Shampoo phase law predicts that retaining accumulated state slows removal of a previously learned mode. We have now added a numerical validation of this result in a controlled setting in Section 7, shown in Figure 4. What this means in practice is that retaining optimizer state can therefore affect how quickly a previously useful component is removed, even when training resumes from identical weights. This has direct implications in sequential learning. We have now evaluated three sequential tasks on permuted MNIST using a nonlinear MLP, and observe that Shampoo with retained state has the smallest accuracy drop on Task 1 after three tasks among SGD with momentum, Adam with momentum, and Muon using the Newton-Schulz update (please refer to G.3.1, Figure 27). This also implies that persistence of learned features can prolong reliance on an obsolete feature. We verify this by training a nonlinear MLP on fixed predictive patches (shortcuts) for MNIST classes for 2 epochs, followed by randomizing the patches. We measure how long the patch effect lasts and observe that retaining Shampoo state has the highest mean time to halve the patch effect (please refer to G.3.2, Figure 29). This effect of Shampoo is also observed in other works in catastrophic forgetting and stable adaptation under visual distribution shifts [1,2,3].

  2. Stabilization parameters change the spectral bias. In underdetermined regression, we now validate

    • increasing Adam’s ϵ\epsilon shifts the spectral trajectories from SignGD toward GD (see G.2.1)
    • increasing additive K-FAC damping λKF\lambda_\mathrm{KF} moves them toward GD (see G.2.2). These constants control how strongly the optimizer rescales updates along weak directions within the available training budget and can shift the bias.
  3. Solution selection. While our theoretical result corresponds to the full observation setting where the solution is unique, we observe that the results explain our observation in underdetermined regression results (G.2). Although all methods approach similar final singular spectra, their transient spectrum differ substantially. In addition to the known sequential mode learning of GD and more uniform growth in ideal Muon, we also observe that cumulative Shampoo accelerates weaker modes and K-FAC produces comparatively smooth growth across modes. These effects remain visible even though the singular vectors are not fixed, and the data covariance affects the modes. Selection among converged interpolants remains open.

Q3 and W1: Optimizer-specific conditions in Theorem 2

We have updated Table 1 and Theorem 2 with the optimizer-specific conditions: undamped K-FAC uses pseudoinverse, Muon applies exact polarization without momentum, Adam is with instantaneous state updates.

[1] Pascanu et al. Optimizers qualitatively alter solutions and we should leverage this. arXiv:2507.12224 2025

[2] Mei et al. Kradagrad: kronecker approximation-domination gradient preconditioned stochastic optimization. UAI 2023

[3] Sarkar et al. Minimax Optimality of Shampoo for Nonstationary Stochastic Optimization. arXiv:2610.16923 2026

AuthorsOpenReview ↗

Response clarifying the assumptions

W1: Clarification on the assumptions

We understand the current exposition seems to suggest that our assumptions are strong. However, we clarify the following:

  • Assumption 1: Balanced initialization

It follows the exact mode analysis of deep linear gradient flow [1], and subsequent work on linear networks [2,3,4,5]. Its role here is to give each mode the same initial amplitude in every layer so that the comparison starts from the same factorization geometry. Under gradient flow, the difference between adjacent Gram matrices is conserved [1]. For the other optimizers, Appendix D.4 verifies preservation directly from the specified updates.

  • Assumption 2: Alignment with the target

It follows the fixed basis construction of [1] and has a particularly close precedent in [6] for spectral gradient descent at arbitrary depth. In fact, [6] initialize all modes at the same amplitude, while our formulation allows the amplitude to vary by mode as long as it agrees across layers. [7] also assume exact internal and target alignment for a much simpler bilinear Muon without momentum flow equations (same as L=2 in our setting). Their separate analysis of how alignment emerges from small random initialization addresses a complementary question. Our purpose in fixing the directions is to compare growth, saturation, and decay across optimizers and depths under common starting conditions.

  • The condition in Assumption 3 holds for the considered optimizers due to Assumptions 1 and 2

The original Assumption 3 combined the conditions on the optimizer with the preservation property proved in Appendix D.4. Exposing it as an assumption was incorrect, and we have separated the two initialization assumptions from the preservation result in the revision.
Balance gives each mode the same starting amplitude across layers, and target alignment fixes the directions along which we compare optimizers. Appendix D.4 verifies that each specified update preserves the mode bases and, with shared hyperparameters and matching mode states, produces equal updates across layers. This preserves balance and alignment along the resulting flow.

Therefore, our analysis builds on the prior works under similar considerations, while answering a complementary question: how do depth and optimizer state determine the growth, saturation, and decay of those modes? We use the same balanced and aligned initialization across optimizers to isolate these effects. The GD and ideal Muon growth laws provide reference cases. The main results concern the change in growth and saturation caused by the cumulative Shampoo state and the return of GD-like slow growth and decay under positive additive curvature damping in K-FAC and the stabilization parameter in Adam. The exact phase exponents apply to the specified flows and initialization conditions.

[1] Saxe et al. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. ICLR 2014

[2] Gunasekar et al. Implicit regularization in matrix factorization. NeurIPS 2017

[3] Arora et al. Implicit regularization in deep matrix factorization. NeurIPS 2019

[4] Arora et al. On the optimization of deep networks: Implicit acceleration by overparameterization. ICML 2018

[5] Li et al. Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning. ICLR 2021

[6] Vasudeva et al. How Muon’s Spectral Design Benefits Generalization: A Study on Imbalanced Data. ICLR 2026

[7] Rhee et al. Muon Learns Balanced Solutions in Matrix Factorization without Slow Saddle-to-Saddle Dynamics. arXiv:2606.30509

Reviewer_6bynOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Slowdown in gradient descent usually coincides with the appearance of singular modes. The authors study the evolution of this spectral bias when the slowdown is removed (for example through curvature normalization). Under some balanced and alignment assumptions, the authors derive precise singular model dynamics (including birth saturation and decay laws) for classical gradient descent, signed gradient descent and Muon, Adam, Shampoo, as well as K-FAC. The phase laws are verified through numerical simulations.

Critical strengths

The contributions and the analysis are interesting

Critical weaknesses

Abstract

  • The abstract is not clear. As an example the sentence “the resulting picture is not a simple ordering from stronger to weaker low rank bias” seems unnecessary to me. What are your main contributions ? Those should be clearly listed. You mention that you derive birth saturation and decay laws, where are they ? and what do you learn from them? What do they represent?
  • Same idea for the sentence “These results give normalization, accumulated state, damping, and depth a direct interpretation as controls determining…” the sentence is not even clear

Introduction

  • Lines 44-45, you may want to add a refefence to the following paper: “Why Deep Jacobian Spectra Separate: Depth-Induced Scaling and Singular-Vector Alignment”
  • Figure 1 is not very clear. The connection between the caption and the figure is not clear. In particular, what do \sigma, \mu represent ? What is the zero target mode ?
  • lines 101-103, I would recommend using bullets to clarify the contributions. E.g. “Our analysis separates three aspect : - the birth ….”

A common framework for optimizer geometry

  • lines 247: “preserves the aligned form” —> Does the “aligned form” refer to the factorization for W_\ell given in Assumption 2?
  • line 249 : “When the factor amplitudes and mode states agree across layers” —> When does that happen ?
  • line 255 - 257 “These choices retain te moment and preconditionner states responsible for the phase behavior studied below and shown in […] Under assumption 3, equal amplitude have equal update across…” Does that mean that Assumption 3 implies Assumption 2?
  • In subsection “Comparison across depth” —> The sentence “Unless stated otherwise, we compare by fixing the target matrix..” —> The sentence does not make sense to me. Can you rephrase?

Optimizer specific results

  • line 284 - 285: I feel that something is missing here: “From Theorem 1, for s<b, time is T_opt …” what time is T_opt? Also, how do you go from T_opt to T(a, b)? I understand that the z and q that appear in T_opt are the same as those that appear in the dynamics on singular amplitudes. But I don’t understand why. This should be clarified.
  • Equation (4): Shouldn’t there be a condition on a, b to ensure that T is positive? I understand that
  • The connection between the singular mode dynamics and the birth decay and saturation rates is not very clear.
  • In subsection “Birth and decay exponent” as well as “Saturation exponent”, I would better introduce the notions of birth dynamics and saturation exponent.

A couple of typos:

  • line 201: “focus on how depth an optmizer” —> “optimizer”
Questions for authors

see weaknesses

1 comment

Thank you for the positive review and for suggesting changes to improve the paper. We have revised the draft, and all the updates are marked in magenta. The abstract now defines birth, saturation, and decay and states the main findings. The introduction now lists the three phases and cites Haas et al. (2026). We also clarified the notation in Figure 1 and corrected the typo at line 201.

On the comments on the framework

  • Lines 247-257: Assumptions 1 and 2 actually result in the condition ZℓZ_\ell we originally had in Assumption 3 for each optimizer update (as derived in D.4). We have updated this part to address the comments from reviewers 5nPF and s4LR.
  • Line 273: We rephrased the comparison across depth paragraph for better clarity.

On the comments on optimizer-specific results

We have revised lines 284-285 to avoid the new notation ToptT_\mathrm{opt} and clarified it through Toptbirth(s,b)T_{\mathrm{opt}}^{\mathrm{birth}}(s,b) and Toptdecay(s,δ)T_{\mathrm{opt}}^{\mathrm{decay}}(s,\delta), and clarified the connection between singular value dynamics and birth and decay dynamics. Regarding equation (4), the conditions required are 0<a<b0<a<b and c>0c>0 to ensure the time is positive, which are already mentioned in lines 288-289.

Reviewer_s4LROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the training dynamics of different optimizers (GD, SignGD, Adam, Muon, Shampoo, K-FAC) on deep matrix factorization (under some strong balance and alignment assumptions). Based on the ODE approximations of these optimizers, this paper highlights several distinct modes, including different birth, saturation, and decay behaviors.

Critical strengths

Elegant analysis in a simple setup. The authors successfully identified a simple theoretical setup in which the dynamics of various optimizers can be mathematically analyzed and understood.

Critical weaknesses
  • The assumptions are very strong. While I view deep matrix factorization as a natural testbed for understanding deep learning, the authors further made many strong assumptions, especially "Alignment with the target" and "Compatible optimizer state and updates". This indeed matters because for some optimizers such as Adam, their coordinatewise update does not necessarily align with the eigenbasis of the matrices. This would break down the whole analysis. Meanwhile, some recent works such as Rhee et al. (2026) have already provided a more careful analysis of a simpler version of the problem.
  • The takeaway of this paper is not very clear to me. Yes, the optimizers can exhibit different behaviors in this deep matrix factorization problem under some assumptions, but how is it related to neural network training in the real world? I don't think that they are unrelated, but the authors did not provide any empirical evidence to give some clue as to how they could be related together.
Questions for authors

/

2 comments
AuthorsOpenReview ↗

Response clarifying the assumptions

Thank you for recognizing the value of a setting in which the optimizer dynamics can be derived exactly. We revised Section 4 to clarify the scope of the analysis and added a new Section 7 with experiments using nonlinear classifiers that connect the phase laws to practical neural network training. All updates are marked in magenta.

W1: Clarification on the assumptions

We understand the current exposition seems to suggest that our assumptions are strong. However, we clarify the following:

  • Assumption 1: Balanced initialization

It follows the exact mode analysis of deep linear gradient flow [1], and subsequent work on linear networks [2,3,4,5]. Its role here is to give each mode the same initial amplitude in every layer so that the comparison starts from the same factorization geometry. Under gradient flow, the difference between adjacent Gram matrices is conserved [1]. For the other optimizers, Appendix D.4 verifies preservation directly from the specified updates.

  • Assumption 2: Alignment with the target

It follows the fixed basis construction of [1] and has a particularly close precedent in [6] for spectral gradient descent at arbitrary depth. In fact, [6] initialize all modes at the same amplitude, while our formulation allows the amplitude to vary by mode as long as it agrees across layers. Rhee et al. [7, Section 3] also assume exact internal and target alignment for a much simpler bilinear Muon without momentum flow equations (same as L=2 in our setting). Their separate analysis of how alignment emerges from small random initialization addresses a complementary question. Our purpose in fixing the directions is to compare growth, saturation, and decay across optimizers and depths under common starting conditions.

  • The condition in Assumption 3 holds for the considered optimizers due to Assumptions 1 and 2

The original Assumption 3 combined the conditions on the optimizer with the preservation property proved in Appendix D.4. Exposing it as an assumption was incorrect and we have separated the two initialization assumptions from the preservation result in the revision.
Balance gives each mode the same starting amplitude across layers, and target alignment fixes the directions along which we compare optimizers. Appendix D.4 verifies that each specified update preserves the mode bases and, with shared hyperparameters and matching mode states, produces equal updates across layers. This preserves balance and alignment along the resulting flow. The reviewer’s point about Adam is important. An orthogonal rotation can change an entrywise update, so the Adam and SignGD results require diagonality in the actual parameter coordinates. The Adam phase laws additionally use the instantaneous moment approximation (as stated in lines 244-247).

Therefore, our analysis builds on the prior works under similar considerations, while answering a complementary question: how do depth and optimizer state determine the growth, saturation, and decay of those modes? We use the same balanced and aligned initialization across optimizers to isolate these effects. The GD and ideal Muon growth laws provide reference cases. The main results concern the change in growth and saturation caused by the cumulative Shampoo state and the return of GD-like slow growth and decay under positive additive curvature damping in K-FAC and the stabilization parameter in Adam. The exact phase exponents apply to the specified flows and initialization conditions.

[1] Saxe et al. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. ICLR 2014

[2] Gunasekar et al. Implicit regularization in matrix factorization. NeurIPS 2017

[3] Arora et al. Implicit regularization in deep matrix factorization. NeurIPS 2019

[4] Arora et al. On the optimization of deep networks: Implicit acceleration by overparameterization. ICML 2018

[5] Li et al. Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning. ICLR 2021

[6] Vasudeva et al. How Muon’s Spectral Design Benefits Generalization: A Study on Imbalanced Data. ICLR 2026

[7] Rhee et al. Muon Learns Balanced Solutions in Matrix Factorization without Slow Saddle-to-Saddle Dynamics. arXiv:2606.30509

AuthorsOpenReview ↗

Response to how our results relate to neural network training

W2: Clarification on the takeaway: how our results translate to neural network training in the real world

The reviewer is absolutely correct in saying that our theoretical results are related to neural network training in the real world. Our results show that optimizer settings can change which components a model learns or removes during training. This matters when training stops early or the objective changes during fine tuning. Based on this we have now added a new section on "Practical Implications of the Phase Laws" in page 9-10 with new experiments in Appendix G.3.

  1. Retaining Shampoo state slows removal of previously learned features. Our cumulative Shampoo analysis shows that accumulated gradient history slows down the removal of learned directions (we have now added a numerical validation of this result in a controlled setting in Section 7, shown in Figure 4). What this means in practice is that retaining optimizer state can therefore affect how quickly a previously useful component is removed, even when training resumes from identical weights. This has direct implications in sequential learning. We have now evaluated three sequential tasks on permuted MNIST using a nonlinear MLP, and observe that Shampoo with retained state has the smallest accuracy drop on Task 1 after three tasks among SGD with momentum, Adam with momentum, and Muon using the Newton-Schulz update (please refer to G.3.1). This also implies that persistence of learned features can prolong reliance on an obsolete feature. We verify this by training a nonlinear MLP on fixed predictive patches (shortcuts) for MNIST classes for 2 epochs, followed by randomizing the patches. We measure how long the patch effect lasts and observe that retaining Shampoo state has the highest mean time to halve the patch effect (please refer to G.3.2). This effect of Shampoo is also observed in other works in catastrophic forgetting and stable adaptation under visual distribution shifts [1,2,3].

  2. Stabilization parameters change the spectral bias. In underdetermined regression, we now validate

    • increasing Adam’s ϵ\epsilon shifts the spectral trajectories from SignGD toward GD (see G.2.1)
    • increasing additive K-FAC damping λKF\lambda_\mathrm{KF} moves them toward GD (see G.2.2). These constants control how strongly the optimizer rescales updates along weak directions within the available training budget and can shift the bias.

We believe that this new analysis clarifies how our results relate to neural network training in the real world.

[1] Pascanu et al. Optimizers qualitatively alter solutions and we should leverage this. arXiv:2507.12224 2025

[2] Mei et al. Kradagrad: kronecker approximation-domination gradient preconditioned stochastic optimization. UAI 2023

[3] Sarkar et al. Minimax Optimality of Shampoo for Nonstationary Stochastic Optimization. arXiv:2610.16923 2026

Reviewer_pvH9OpenReview ↗
Rating
2
Confidence
3(+1)
Informativeness
1
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper studies continuous-time training of deep linear networks under several geometries induced by the choice of the optimizer. Authors consider GD, SignGD, ideal Muon, cumulative Shampoo, and a block-diagonal Gauss–Newton model of K-FAC and derive the learning dynamics for singular modes in dependence to the optimizer and its induced geometry, focusing on birth (escape from a small value regime), saturation (approach to a nonzero target), and decay (removal of zero-target modes).

Interestingly, authors identify that non-adaptive sign-based methods (SignGD and Muon) saturate linearly, while GD, Adam, Shampoo and K-FAC saturate exponentially.

Experiments for the idealized optimizers in deep linear networks match the predictions closely and one regression experiment shows qualitatively similar transient differences.

Critical strengths

The birth/saturation/decay decomposition and a unified framework are useful conceptual contributions. Prior work mostly focused on the way small modes emerge. Showing that birth, saturation, and decay respond differently to the same optimizer is a real conceptual contribution.

Analysis of stabilizers as spectral thresholds is interesting. The point that ε,λShε, λ_{Sh}, and λKFλ_{KF} do not only serve as numerical safeguards but also set where depth’s suppression reappears is worth making. Fig. 3 communicates it well.

Presentation and reproducibility. Assumptions are stated explicitly, the optimizer conventions in Table 1 are clear, and code is promised.

Critical weaknesses
  1. Full observation has a unique minimizer, so the results describe transient dynamics, not low-rank solution selection, while the abstract and conclusion are framed as if hinted towards the low-rank bias selection. This is misleading.
  2. The aligned diagonal setting seems to make SignGD and Muon trivially identical.
  3. Most experiments just integrate the derived ODEs without expanding anything further, which makes the experiments and the arguments circular, so most experiments don't hint towards more realistic scenarios nor provide any intuitions.
  4. The idealized variants differ from practice: exact polar instead of Newton–Schulz, “Adam” with β₁ = β₂ = 0, full-block rather than factored K-FAC damping, and flow-only finite-time results.
  5. Absolute times are compared across optimizers under inconsistent normalizations.
  6. As far as I know, the novelty is modest for the GD, Muon and natural-gradient results.
Questions for authors
  1. How is the Shampoo accumulator scaled in discrete time? Are the reported λ_Sh in flow units?
  2. Do the measured exponents hold in the unaligned setting and with real Adam (β₁, β₂ > 0)?
  3. How much does practical Newton–Schulz Muon change weak-mode birth? For example, if we considered the spectrum after the update to be bounded in (1−ϵ,1+ϵ)(1-\epsilon, 1+\epsilon), how do predictions change?
  4. Why does Muon’s L = 2 decay exponent in Fig. 8 deviate from q = 0.5?
4 comments
AuthorsOpenReview ↗

Response to Shampoo accumulator scaling, unaligned setting, effect of Newton Schulz Muon updates

We thank the reviewer for the constructive feedback and for highlighting the conceptual contribution of our unified framework and analysis of the stabilizers. We address the questions raised below. All updates in the draft are marked in magenta.

Q1: Shampoo accumulator scaling

The reported λSh\lambda_{\mathrm{Sh}} values are in flow units. With learning rate η\eta, the flow time is tk=kηt_k=k\eta at step kk and the cumulative Shampoo implementation uses

D_{\ell,0} = \lambda_{\mathrm{Sh}}I\quad \text{ at initialization (step 0)}$$ $$C_{\ell,k+1} = C_{\ell,k}+\eta H_{\ell,k}H_{\ell,k}^{\top}, \qquad D_{\ell,k+1} = D_{\ell,k}+\eta H_{\ell,k}^{\top}H_{\ell,k}, \qquad W_{\ell,k+1} = W_{\ell,k} -\eta C_{\ell,k}^{-1/4}H_{\ell,k}D_{\ell,k}^{-1/4}.$$ The phase law experiments refresh the preconditioner every step. Both the parameters and accumulators therefore follow an explicit Euler discretization with step size $\eta$, using the current accumulator for the parameter update. The reported $\lambda_{\mathrm{Sh}}$ is the configured initial accumulator level, giving $d_0=\sqrt{\lambda_{\mathrm{Sh}}}$ in the aligned scalar model. ### Q2 and W4: Exponents in the unaligned setting and Adam with β₁, β₂ > 0 Unaligned setting can change the exponent by coupling directions, even when the factors remain balanced. This can be seen exactly for depth two GD under full observation. Take the target to be $M=\operatorname{diag}(\mu_i)$ and balanced factors $A$ and $A^\top$, so that $W=AA^\top$. Gradient flow preserves the transpose relation as shown in D.1, and gives $\dot W=MW+WM-2W^2$ (from eq. 6). Thus the amplitude $a_i=W_{ii}$ along the target direction $i$ satisfies $$\dot a_i=2a_i(\mu_i-a_i)-2\sum_{j\ne i}W_{ij}^2.$$ Alignment makes the second term vanish. Without alignment, that term evolves during training, so two predictors with the same $a_i$ can have different growth speeds. Let $\rho_i=\sum_{j\ne i} \dfrac{W_{ij}^2}{a_i(\mu_i-a_i)}$. On a growing interval with $0<a_i<\mu_i$, the log slope of $\dfrac{\dot a_i}{\mu_i-a_i}$, that is, $$\frac{d}{d\log a_i}\log \left(\frac{\dot a_i}{\mu_i-a_i}\right) =\frac{d}{d\log a_i}\bigl[\log 2+\log a_i+\log(1-\rho_i)\bigr]\\ =1+\frac{d \log(1-\rho_i)}{d\log a_i}.$$ This identifies precisely where the fitted exponent can depart from the aligned setting. Under alignment, $\rho_i=0$ and the slope is exactly $1$. With misalignment, even nonzero coupling leaves the slope at $1$ if $\rho_i$ stays constant. The slope is $<1$ when $\rho_i$ increases with $a_i$ and $>1$ when it decreases. Therefore, **the exponents can change in unaligned case even under balanced setting**. **Adam with momentum β₁, β₂ > 0** Even in the aligned setting, Adam with momentum can deviate from the instantaneous Adam phase laws. We have now added the experiment in G.1.6 of the updated draft. Both birth and decay exponents exhibit different amplitude dependence. The estimated saturation rates also differ substantially (Figure 20). These discrepancies call for a separate theoretical analysis that explicitly retains the evolving first and second moment states and their coupling to the spectral dynamics. ### Q3 and W4: Effect of Newton Schulz (NS) on weak mode birth We have now added the comparison between exact polar Muon and NS Muon for depth L=4 in G.1.5 of the revised version (Figures 17, 18). The smoothed NS birth exponent ranges approximately from 0.59 to 0.85. This is broadly consistent with the exact polar reference $q=1-1/L=0.75$, with visible deviations across the measured range. **Theoretical analysis:** Under our framework, we can derive when these two formulations can be different. The dynamics of Muon without momentum follows $\dot{\sigma}=L\sigma^q u$, where $u$ is the update singular value in the growing direction. In the case of exact polar update, $u=1$ (Theorem 2). Under the suggested bound $u\in(1-\epsilon,1+\epsilon)$, with $0<\epsilon<1$, the birth time from $s$ to $r$ satisfies $$\frac{r^{1/L}-s^{1/L}}{1+\epsilon} \le T_{\mathrm{NS}}\le \frac{r^{1/L}-s^{1/L}}{1-\epsilon},$$ whereas the time for exact polar update is $T_{\mathrm{polar}}=r^{1/L}-s^{1/L}$. This $T_{\mathrm{NS}}$ bounds birth time. The local exponent also depends on how $u$ changes along the trajectory. A fixed number of NS steps can amplify a very small direction. NS acts on the normalized gradient singular value $|g_i|/\sqrt{\sum_j |g_j|^2}$. For a weak growing mode, $|g_i|=(\mu_i-\sigma_i)\sigma_i^q \approx \mu_i\sigma_i^q$. If stronger directions keep $\sqrt{\sum_j |g_j|^2}$ near a positive constant, then $u_i$ scales as $\sigma_i^q$. Therefore, the dynamics become roughly proportional to $\dot{\sigma} \propto L\sigma^{2q}$, *similar to GD*, while exact polar normalization removes that dependence by setting $u_i=1$, giving birth exponent $q$. **Hence, a finite NS update can show GD-like birth exponent $2q$ in that regime.**
AuthorsOpenReview ↗

Response to Muon figure and other comments

Q4: Muon L=2 decay exponent

Thanks for pointing this out. It was an error in Muon analysis where the numerically computed value was for L=1. We have updated it (Figure 9 in the revised version).

W3: On intuitions and results in realistic scenarios

We have now included a new section on the 'Practical Implications of Phase Laws' (pages 9-10) going beyond the singular value trajectories. To briefly elaborate, we identify two main practical consequences:

  1. Shampoo state controls the removal of previously learned directions. The gradient history accumulated by Shampoo slows the removal of previously learned directions, helping preserve useful features as well as prolonging the obsolete features. This has immediate practical implications in finetuning a pretrained model or when there is a task switch. Specifically, in sequential learning of three tasks on permuted MNIST using nonlinear MLP, we observe that Shampoo with retained state has the smallest accuracy drop on Task 1 after three tasks among SGD with momentum, Adam with momentum, and Muon using the Newton-Schulz update (please refer to G.3.1). Similarly, after learning the fixed predictive patches (shortcuts) for MNIST classes, when the predictive patches are randomized, retaining Shampoo state has the highest mean time to halve the patch effect (please refer to G.3.2). This effect of Shampoo is also observed in other works in catastrophic forgetting and stable adaptation under visual distribution shifts [1,2,3].

  2. Stabilization changes the spectral bias. In underdetermined regression, we validate Adam stabilizer ϵ\epsilon can shift the bias toward GD when ϵ\epsilon increases (see G.2.1), and as λKF\lambda_\mathrm{KF} damping in K-FAC increases, it changes the bias toward GD (see G.2.2). These constants control how strongly the optimizer rescales updates along weak directions within the available training budget and can shift the bias.

W1: Correcting low-rank solution selection in abstract and conclusion

We have updated the abstract and conclusion to be clear in our framing by explicitly mentioning the transient behavior and removing any suggestion of low rank solution under full observation (lines 17, 531).

W4 and W5: On flow-only finite time results and absolute time comparisons

For an update Wk+1=Wk−ηZkW_{k+1}=W_k-\eta Z_k, the convention tk=kηt_k=k\eta is the standard Euler time scaling: the change per unit of flow time is −Zk-Z_k. This lets us compare each discrete run with its corresponding flow under the stated normalization. An absolute time ranking across optimizers still depends on their update scales. Additionally, although our analysis corresponds to flow, we observe the derived theoretical results align well with the discrete time experiments.

W6: On the novelty for the GD, Muon and natural-gradient results

We build on prior analyses of GD, polar Muon, and natural gradient. While the prior works derive the dynamics and mostly focus on the emergence of weak modes, we characterize birth, saturation, and decay in a common framework. This shows how an optimizer can accelerate weak mode growth and still remove unsupported modes. Our more novel technical contributions are the analysis of Shampoo, where we derived the evolving preconditioner denominator exactly in terms of the current mode amplitude (eq. 8), which resulted in Proposition 8 in page 32, and the analysis of stabilizers in Adam and K-FAC, which can change the bias as derived in Propositions 7 and 9, respectively. Please refer to the response to Q1 to Reviewer 5nPF for a detailed technical contribution discussion.

[1] Pascanu et al. Optimizers qualitatively alter solutions and we should leverage this. arXiv:2507.12224 2025

[2] Mei et al. Kradagrad: kronecker approximation-domination gradient preconditioned stochastic optimization. UAI 2023

[3] Sarkar et al. Minimax Optimality of Shampoo for Nonstationary Stochastic Optimization. arXiv:2610.16923 2026

Thank you for the revision. My questions Q1, Q4 and W1 are resolved.

I have raised soundness but keep my overall rating, as the significance of the analytical and practical advance remain my main concerns.

Thank you for your response and for raising the soundness score. For Q2 and Q3, we have added new experiments in addition to the theoretical analysis provided in our response (sections G.1.5 and G.1.6 in the revised version). We are happy to address any specific unresolved concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of updates to the draft

We thank the reviewers for their constructive feedback. The revised manuscript includes the following updates. All updates are marked in magenta.

Updates to the draft

  1. Original assumption on optimizer state and updates (Reviewers 5nPF and s4LR). Original Assumption 3 was unnecessary, as Assumptions 1 and 2, with the considered optimizer states, preserve them. Updated Section 4, lines 246 to 256.
  2. Practical implications of the phase laws (Reviewers pvH9, 5nPF and s4LR). Added a new section (Section 7 on pages 9-10) on how our theoretical results translate to practical settings using the optimizer state retention and the role of stabilization parameters.
  3. Optimizer-specific conditions (Reviewer 5nPF). Clarified the exact polar Muon without momentum, instantaneous Adam, and additive block damping conditions in Table 1 and Theorem 2.
  4. Other minor edits were made to the abstract and conclusion, clarifying the transient behavior (Reviewer 6byn), to Section 5, clarifying the passage time and exponents (Reviewer 6byn), and to Figure 9, fixing the Muon depth 2 plot (Reviewer pvH9).

New experiments

  1. Shampoo state after a target switch. Added a controlled regression comparison of retaining and resetting the accumulators (Section 7, Figure 4).
  2. Sequential learning using nonlinear ReLU MLP. Added sequential learning on permuted MNIST and split MNIST, comparing SGD with momentum, Adam with momentum, Muon, and cumulative Shampoo (Appendix G.3.1, Figures 27-30).
  3. Evaluating persistence of learned shortcuts using nonlinear ReLU MLP. Added experiments on MNIST by adding patches comparing SGD, Adam with momentum, Muon, and cumulative Shampoo (Appendix G.3.2, Figure 31).
  4. Added exact polar Muon vs Newton Schulz Muon comparison (Appendix G.1.5, Figures 17-18)
  5. Added instantaneous Adam vs Adam with momentum comparison (Appendix G.1.6, Figures 19-20)
  6. Adam’s stabilizer ϵ\epsilon interpolates between SignGD and GD. Added sweeps of Adam’s ϵ\epsilon in underdetermined regression (Appendix G.2.1, Figures 23-24)
  7. K-FAC approaches GD as damping increases. Added sweeps of additive block damping in K-FAC in underdetermined regression (Appendix G.2.2, Figures 25-26).

EDIT: (16.11.) Added sequential learning on split MNIST using nonlinear ReLU MLP experiment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.