- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how optimizer geometry and network depth shape spectral learning dynamics in deep matrix factorization. Under balance and alignment assumptions, it develops a unified framework for comparing multiple optimizers by separating the birth, decay, and saturation of singular modes. The analysis characterizes how normalization, accumulated optimizer state, and damping affect the emergence, removal, and convergence of these modes. The authors also conduct toy experiments in deep matrix factorization to numerically verify the predicted phase laws and transitions.
Critical strengths
The paper analyzes multiple optimizers within a unified deep matrix factorization framework, enabling a systematic comparison of their spectral learning dynamics. Separating these dynamics into birth, decay, and saturation provides a useful perspective on how different optimizers govern the emergence, removal, and convergence of singular modes. This distinction highlights that an optimizer can affect different stages of learning in different ways.
Critical weaknesses
-
The theoretical analysis relies on strong assumptions that substantially simplify the dynamics. In particular, Assumption 2 initializes the factors in singular bases aligned with the target, while Assumption 3 requires the joint evolution of the factors and optimizer states to preserve this aligned structure. These assumptions eliminate singular-vector dynamics in every weight matrix, leaving only scalar mode amplitudes to evolve. Consequently, Theorem 1 follows directly from substituting the assumed factorization into the gradients and differentiating the product amplitudes, which limits its substantive theoretical contribution. Several additional conditions are also insufficiently specified in the main text or theorem statements. The Muon analysis excludes momentum and replaces the Newton–Schulz approximation with exact polar normalization, with the treatment of rank-deficient gradients specified only in the appendix. Theorem 2’s Adam results require the instantaneous approximation , rather than the finite-memory dynamics defined in Table 1, but this restriction is absent from the theorem statement. Likewise, the undamped K-FAC update requires invertibility or a specified pseudo-inverse convention. These conditions should be stated explicitly alongside the results they support.
-
The paper’s main contribution remains unclear to me. Much of the core GD and Muon analysis recovers phenomena already established in prior work, including depth-dependent spectral learning and low-rank bias under GD [1,2], and uniform factor growth and the disappearance of slow sequential learning under simplified Muon [3–5]. More broadly, describing how singular values emerge, saturate, and decay under different optimizers leaves the consequences of those differences insufficiently resolved. The discussion and nonaligned regression experiment provide some evidence of relevance, but do not establish when these dynamics improve or harm generalization, affect solution selection, or inform optimizer and hyperparameter choices.
References
[1] Arora et al., Implicit Regularization in Deep Matrix Factorization. NeurIPS 2019.
[2] Li et al., Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learning. ICLR 2021.
[3] Kang et al., Uniform Spectral Growth and Convergence of Muon in LoRA-Style Matrix Factorization. arXiv:2602.06385.
[4] Rhee et al., Muon Learns Balanced Solutions in Matrix Factorization without Slow Saddle-to-Saddle Dynamics. arXiv:2606.30509.
[5] Vasudeva et al., How Muon’s Spectral Design Benefits Generalization: A Study on Imbalanced Data, ICLR 2026.
Questions for authors
- What are the main technical advances over prior analyses? In particular, what new insights does the unified framework provide?
- What consequences follow from the proposed phase laws beyond characterizing singular-value trajectories? Can the authors connect these dynamics to concrete outcomes, such as solution selection and optimizer and hyperparameter choices, and clarify which implications are supported by the experiments?
- Could the authors explicitly state the optimizer-specific conditions required by the theorems, including momentum-free exact-polar Muon, instantaneous Adam, and the inverse or pseudoinverse convention for undamped K-FAC?
3 comments
Response to Q2 consequences of derived phase laws and Q3 optimizer-specific conditions
Q2 and W2: Consequences of the phase laws in solution selection, optimizer and hyperparameter choices
Our results show that optimizer settings can change which components a model learns or removes during training. This matters when training stops early or the objective changes during fine tuning. Based on this, we have now included a new section on the 'Practical Implications of Phase Laws' (pages 9-10) going beyond the singular value trajectories and new experiments using nonlinear classifiers in G.3.
-
Optimizer state determines how much of the previously learned features survives adaptation. The Shampoo phase law predicts that retaining accumulated state slows removal of a previously learned mode. We have now added a numerical validation of this result in a controlled setting in Section 7, shown in Figure 4. What this means in practice is that retaining optimizer state can therefore affect how quickly a previously useful component is removed, even when training resumes from identical weights. This has direct implications in sequential learning. We have now evaluated three sequential tasks on permuted MNIST using a nonlinear MLP, and observe that Shampoo with retained state has the smallest accuracy drop on Task 1 after three tasks among SGD with momentum, Adam with momentum, and Muon using the Newton-Schulz update (please refer to G.3.1, Figure 27). This also implies that persistence of learned features can prolong reliance on an obsolete feature. We verify this by training a nonlinear MLP on fixed predictive patches (shortcuts) for MNIST classes for 2 epochs, followed by randomizing the patches. We measure how long the patch effect lasts and observe that retaining Shampoo state has the highest mean time to halve the patch effect (please refer to G.3.2, Figure 29). This effect of Shampoo is also observed in other works in catastrophic forgetting and stable adaptation under visual distribution shifts [1,2,3].
-
Stabilization parameters change the spectral bias. In underdetermined regression, we now validate
- increasing Adam’s shifts the spectral trajectories from SignGD toward GD (see G.2.1)
- increasing additive K-FAC damping moves them toward GD (see G.2.2). These constants control how strongly the optimizer rescales updates along weak directions within the available training budget and can shift the bias.
-
Solution selection. While our theoretical result corresponds to the full observation setting where the solution is unique, we observe that the results explain our observation in underdetermined regression results (G.2). Although all methods approach similar final singular spectra, their transient spectrum differ substantially. In addition to the known sequential mode learning of GD and more uniform growth in ideal Muon, we also observe that cumulative Shampoo accelerates weaker modes and K-FAC produces comparatively smooth growth across modes. These effects remain visible even though the singular vectors are not fixed, and the data covariance affects the modes. Selection among converged interpolants remains open.
Q3 and W1: Optimizer-specific conditions in Theorem 2
We have updated Table 1 and Theorem 2 with the optimizer-specific conditions: undamped K-FAC uses pseudoinverse, Muon applies exact polarization without momentum, Adam is with instantaneous state updates.
[1] Pascanu et al. Optimizers qualitatively alter solutions and we should leverage this. arXiv:2507.12224 2025
[2] Mei et al. Kradagrad: kronecker approximation-domination gradient preconditioned stochastic optimization. UAI 2023
[3] Sarkar et al. Minimax Optimality of Shampoo for Nonstationary Stochastic Optimization. arXiv:2610.16923 2026
Response clarifying the assumptions
W1: Clarification on the assumptions
We understand the current exposition seems to suggest that our assumptions are strong. However, we clarify the following:
- Assumption 1: Balanced initialization
It follows the exact mode analysis of deep linear gradient flow [1], and subsequent work on linear networks [2,3,4,5]. Its role here is to give each mode the same initial amplitude in every layer so that the comparison starts from the same factorization geometry. Under gradient flow, the difference between adjacent Gram matrices is conserved [1]. For the other optimizers, Appendix D.4 verifies preservation directly from the specified updates.
- Assumption 2: Alignment with the target
It follows the fixed basis construction of [1] and has a particularly close precedent in [6] for spectral gradient descent at arbitrary depth. In fact, [6] initialize all modes at the same amplitude, while our formulation allows the amplitude to vary by mode as long as it agrees across layers. [7] also assume exact internal and target alignment for a much simpler bilinear Muon without momentum flow equations (same as L=2 in our setting). Their separate analysis of how alignment emerges from small random initialization addresses a complementary question. Our purpose in fixing the directions is to compare growth, saturation, and decay across optimizers and depths under common starting conditions.
- The condition in Assumption 3 holds for the considered optimizers due to Assumptions 1 and 2
The original Assumption 3 combined the conditions on the optimizer with the preservation property proved in Appendix D.4. Exposing it as an assumption was incorrect, and we have separated the two initialization assumptions from the preservation result in the revision.
Balance gives each mode the same starting amplitude across layers, and target alignment fixes the directions along which we compare optimizers. Appendix D.4 verifies that each specified update preserves the mode bases and, with shared hyperparameters and matching mode states, produces equal updates across layers. This preserves balance and alignment along the resulting flow.
Therefore, our analysis builds on the prior works under similar considerations, while answering a complementary question: how do depth and optimizer state determine the growth, saturation, and decay of those modes? We use the same balanced and aligned initialization across optimizers to isolate these effects. The GD and ideal Muon growth laws provide reference cases. The main results concern the change in growth and saturation caused by the cumulative Shampoo state and the return of GD-like slow growth and decay under positive additive curvature damping in K-FAC and the stabilization parameter in Adam. The exact phase exponents apply to the specified flows and initialization conditions.
[1] Saxe et al. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. ICLR 2014
[2] Gunasekar et al. Implicit regularization in matrix factorization. NeurIPS 2017
[3] Arora et al. Implicit regularization in deep matrix factorization. NeurIPS 2019
[4] Arora et al. On the optimization of deep networks: Implicit acceleration by overparameterization. ICML 2018
[5] Li et al. Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning. ICLR 2021
[6] Vasudeva et al. How Muon’s Spectral Design Benefits Generalization: A Study on Imbalanced Data. ICLR 2026
[7] Rhee et al. Muon Learns Balanced Solutions in Matrix Factorization without Slow Saddle-to-Saddle Dynamics. arXiv:2606.30509
Response to Q1 main technical advances
We thank the reviewer for the detailed feedback. We have addressed the comments and questions in the revised version. All updates are marked in magenta. Below we clarify the changes made.
Q1: Main technical advances over prior analyses and new insights
We agree that sequential mode learning under GD and uniform growth under ideal exact Muon without momentum are established results. We use these as reference cases as stated in line 113. The main technical advances are the exact laws for how accumulated state of Shampoo, additive damping of K-FAC and stabilization parameter of Adam change spectral learning. Furthermore, while the dynamics of GD and ideal Muon are established, their decay law is also a unique contribution of our work. Most work focus only on the emergence of small modes.
Technical advances over prior analyses and insights
-
-
\qquad T_{\mathrm{KF}}=T_{\mathrm{KF},\mathrm{undamped}}+\lambda_{\mathrm{KF}}T_{\mathrm{GD}}.$$ At any fixed positive stabilizer, the added term makes birth and decay times diverge similar to GD at depth two or greater. The crossover explains where the two methods differ. Instantaneous Adam compares $\epsilon$ with the layer gradient magnitude $|\mu-\sigma|\sigma^q$, where $q=1-1/L$ (line 367), so even a strong mode returns to scaled GD near its target as the residual vanishes. Additive K-FAC damping competes with the block curvature $\sigma^{2q}$ (line 406), so curvature cancellation can remain effective near a sufficiently large target. At same initialization and a common threshold, Adam’s preference for larger targets is exactly $\epsilon$ times the corresponding GD birth time gap. Undamped KFAC already favors larger targets. 3. **What Muon and undamped K-FAC comparison reveals.** Ideal Muon and undamped K-FAC both give *bounded birth times* as the initialization tends to zero. From Theorem 2, at same initialization, ideal Muon reaches a common absolute threshold at the same time across supported modes, while undamped block curvature still favors larger targets. Ideal Muon also *removes unsupported modes in finite time*, while undamped block curvature removes them *exponentially*. The common analysis identifies which of these properties follows from each update rule under the same objective and initialization.Cumulative Shampoo links optimizer history to all three phases. Proposition 8 derives the evolving preconditioner denominator exactly in terms of the current mode amplitude (eq. 8, page 34). This gives three predictions from the same identity. Growth initially follows GD scaling and then becomes independent of target strength. Near a positive target, the local saturation rate can decrease with target strength, reversing the GD ordering. For a zero target, the accumulated state increases the time required to remove a mode, even when its starting amplitude and removal tolerance are held fixed. This quantifies how optimizer history can change the growth, convergence, and persistence of individual modes.
Adam’s stabilizer and K-FAC damping add the same GD time penalty through different mechanisms. Propositions 7 on page 31 and 9 on page 38 give the following identities for the same passage endpoints in each of the three phases.