- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies rotation-based matrix optimizers, which rotate the momentum into another basis, take a coordinate-wise normalized step there, and rotate the step back. It first brings these methods into one form. In Algorithm 1 the update is with , where is the momentum, and are orthogonal matrices that may depend on the current stochastic gradient, and is a positive normalizer applied entry by entry, which the analysis requires to be at least . SOAP, Conda and SPlus differ only in the choice of , and , and Adam is the case , .
For this common form the paper derives a bound of one general shape. The assumptions are -smoothness, an unbiased gradient oracle and noise variance at most , all in the Frobenius norm. Lemma 1 builds on the momentum argument of the AdamW analysis in [5] and bounds the normalized rotated gradient for any rotations and any normalizer with , using that orthogonal rotations do not change the Frobenius and nuclear norms. Lemma 2, the row-column trace control, bounds by row maxima and column maxima with and, through a Schatten-Hölder step, reduces the average nuclear norm of the gradient to the quantity of Lemma 1 and the traces of and . Once these traces are bounded for a given normalizer, the result takes the form , where and .
The special cases are then worked out. For a generalized SOAP with and , Theorem 1 gives . The same bound covers SOAP, Conda and, with and , Adam. The authors present it as the first rate for SOAP and the first nuclear-norm rate for Adam, of the same order as the rate for AdamW-style Shampoo in [4]. For a truncated SPlus, in which the sign of each entry is replaced by with , Theorem 2 gives . Appendix D replaces the row-column control by a one-sided one and obtains in place of at (Theorem 3). There are no experiments.
Critical strengths
The paper gives a convergence analysis for four methods: SOAP, Conda, a truncated version of SPlus and, as a special case, Adam with matrix-valued parameters. I went through the proofs of Lemmas 1, 2, 3, 7, 8, 9 and 10 and of Theorems 1 to 3 and could not find an error. The paper is short and easy to follow, and Algorithm 1 puts these methods into one notation. Rotations that depend on the current stochastic gradient, which SPlus and Conda need, are handled correctly. The Limitations paragraph is open about the truncation of SPlus, the lower bound on and the problem-dependent parameters, and Appendix D states the one-sided variant.
Critical weaknesses
W1. The claimed contributions are overstated and do not match what the paper shows.
The paper presents its analysis as unified, but it is not unified in the usual sense of the word. A unified analysis proves one theorem under a parametric assumption, and the rate of each method in the framework follows as a corollary that recovers or improves the known rate for that method [1], [2]. Here only Lemma 1 and Lemma 2 are shared. The bound on the traces, which sets the rate, is proved separately for each normalizer (Lemma 3 and Lemma 9), and Theorems 1, 2 and 3 are assembled one by one. The bounds of the framework do not recover the known rate of the simplest method that Algorithm 1 contains. With , and the constant normalizer , Algorithm 1 is plain SGD with momentum and step . For this algorithm the paper's own Lemma 1 gives with , so , and with the nuclear-norm bound . The route of the framework, Lemma 1 with Lemma 2 and , gives for the same algorithm. Read in the Frobenius norm through , this is times worse than the standard rate, and in the nuclear norm it is worse for every non-square matrix.
The final bounds of Theorems 1 to 3 contain only , , , , , and a numerical factor , with set to on page 8. The rotations and , the form of and the parameter do not enter, and SOAP and the truncated SPlus differ only in the numerical constant, against . It seems to me that the reason is visible in vectorized form. The update is momentum SGD with the preconditioner , the rotation fixes only its eigenvectors, and the proofs use only bounds on its eigenvalues. The theorems therefore hold for every rotation, including a random one or one chosen to make the method as slow as possible. A bound that holds for the worst rotation cannot be better than the rate of that rotation, so it cannot show any gain from a good rotation. But the choice of basis is the core of SOAP, Conda and SPlus, and it matters in practice [3]. Since the bound is the same for all rotations, it does not show which rotation is better or why SOAP outperforms Adam
Contribution 3, "the first nuclear-norm convergence rate for Adam with matrix-valued parameters", is overstated as well. Writing as the sum of the rank-one matrices gives , so the bound for AdamW of [5], quoted on page 8, is already a nuclear-norm bound of order . At the value used in [5], which Theorem 1 allows with , Theorem 1 gives , again of order . The factor therefore comes only from taking larger by .
[1] Eduard Gorbunov, Filip Hanzely, Peter Richtárik. A Unified Theory of SGD: Variance Reduction, Sampling, Quantization and Coordinate Descent. AISTATS 2020.
[2] Filip Hanzely, Peter Richtárik. One Method to Rule Them All: Variance Reduction for Data, Parameters and Many New Methods. arXiv:1905.11266, 2019.
[3] Tianyue H. Zhang et al. Understanding Adam Requires Better Rotation Dependent Assumptions. NeurIPS 2025.
[4] Huan Li, Yiming Dong, Zhouchen Lin. Convergence Rate Analysis of the AdamW-style Shampoo: Unifying One-Sided and Two-Sided Preconditioning. ICML 2026.
[5] Huan Li, Yiming Dong, Zhouchen Lin. On the Convergence Rate of AdamW Measured by Norm. NeurIPS 2025.
Questions for authors
Q1 Is there a choice of and for which the framework gives a strictly better bound than , ? If SOAP works because of the eigenbasis, which assumption, for example a smoothness or noise constant measured in the rotated basis, would let the analysis show it?
Q2 On page 8 the paper says that can be "chosen much smaller than , consistent with practical settings". Lemma 1 uses along the whole path and sets the step, so this seems to need , for some constant , for all entries and at all iterations. How is this compatible with the gradient going to zero, and under which conditions does the claim hold?
3 comments
Part 2: Response to the questions
Question 1: Is there a choice of and for which the framework gives a strictly better bound than , ? If SOAP works because of the eigenbasis, which assumption, for example a smoothness or noise constant measured in the rotated basis, would let the analysis show it?
Reply: See the common response. As pointed out by the reviewer, making stronger assumptions may allow one to prove the superiority of SOAP’s rotation matrix, such as the problem-dependent assumptions or an instance-dependent convergence analysis. We leave this question for future work.
Question 2: When is chosen much smaller than , how is the condition compatible with the gradient going to zero, and under which conditions does the claim hold?
Reply: (1) Within the theorem: with and , the condition holds deterministically at every iteration, so the guarantee needs no assumption on the trajectory.
(2) The practical remark that may be chosen much smaller than is a statement beyond the theorem, and its correct form is conditional: it holds whenever for all . Because aggregates the stochastic gradient , this condition does not require the true gradient to stay away from zero, it requires only that the noise remains non-negligible. However, we admit that for isotropic noise the typical entry of is of order , smaller than , and in the deterministic case the condition can fail, which is exactly why the theorem uses .
[1]. On the Convergence Rate of AdamW Measured by Norm. NeurIPS 2025.
[2]. On the Optimality of Adam under Heavy-Tailed Noise. https://openreview.net/forum?id=OjDxyU2Td2.
Thank you authors for the detailed response. Most of my concerns have been addressed. I still believe that the work would benefit from a more detailed examination of the convergence rates of the individual methods. Therefore, I am raising my rating only to weak accept.
Part 1: Response to the weakness
We sincerely thank the reviewer for their valuable suggestions. Below are our responses to the major concerns.
Weakness 1: Overstate the contribution and the analysis is not unified in the usual sense of the word.
Reply: Following the reviewer's suggestion, we can present our analysis in a unified framework with the following single meta-theorem in the parametric form of Gorbunov et al. (2020) and Hanzely & Richtarik (2019):
Meta-theorem. Suppose that Assumptions 1-3 and condition Dk,i,j≥δ hold, and the control matrices Bk,Ck satisfy the parametric trace-control condition
∑k=1KE[trBk]≤K(c1mσ^+c2mh)+c3SK and ∑k=1KE[trCk]≤K(c1nσ^+c2nh)+c3SK,
where h≤m+nσ^ is the regularization scale. Let σ^2=max(σ2,Kγ2L(f(X1)−f∗)) with any γ∈(0,1], 1−θ=Kσ^2L(f(X1)−f∗), and η=4LKσ^2δ2(f(X1)−f∗). Then for the general Algorithm 1, we have
K1∑k=1KE[∥∇f(Xk)∥∗]≤C(m+n+δσ^)4Kσ^2L(f(X1)−f∗),
where c1,c2,c3 are positive constants and C=217(2c1+2c2)+7c3.
Lemma 3 (generalized SOAP, Conda, Adam) verifies the condition with (c1,c2,c3,h)=(2,1,4,ε); Lemma 9 (truncated SPlus) verifies it with (c1,c2,c3,h)=(213,2,1,δ); Theorems 2-3 then follow as corollaries. This is exactly ''one theorem under a parametric assumption, with each method's rate as a corollary.''
Weakness 2: The bounds of the framework do not recover the known rate of the simplest SGD with momentum.
Reply: We agree with the reviewer's computation for the two-sided control. However, the gap is an artifact of the two-sided route, and our one-sided control (Remark 1, Appendix D of our original submitted manuscript) removes it exactly, which replaces m+n by min(m,n) in the theorems. For the special case of X∈Rm×1, where min(m,n)=1, under the setting of Theorem 4 in Appendix D, we have ε=min(m,n)τσ^2=σ^2 with τ=1, δ=σ^, η=4LKf(X1)−f∗, and the convergence rate
K1∑k=1KE[∥∇f(Xk)∥∗]≤(3min(m,n)+17δσ^)4Kσ^2L(f(X1)−f∗)=204Kσ^2L(f(X1)−f∗).
Since ∥X∥∗=∥X∥F for X∈Rm×1, our framework recovers the convergence rate of SGD measured by the vector Frobenius norm. This applies to SGD with momentum (setting Dk,i,j=δ, Pk=I, and Qk=1), vector Adam (setting Pk=I and Qk=1), and one-sided vector SOAP (setting Qk=1). However, a larger ε makes Adam in the rotated space of SOAP behave more like SGD. Accordingly, we did not regard the min(m,n) dependence as a primary contribution of this work.
Weakness 3: The theory cannot show any gain from a good rotation. Since the bound is the same for all rotations, it does not show which rotation is better or why SOAP outperforms Adam.
Reply: See the common response.
Weakness 4: ''the first nuclear-norm convergence rate for Adam with matrix-valued parameter'', is overstated as well.
Reply: From ∥A∥∗≤∥A∥1 and the convergence rate of Adam in [1], we directly obtain the nuclear-norm convergence rate
K1∑k=1KE[∥∇f(Xk)∥∗]≤K1∑k=1KE[∥∇f(Xk)∥1]≤[1]O(mnmax(4Kσ2L△,KL△)).
Furthermore, [2] proved that the above rate is optimal when measured by the ℓ1 norm, particularly with respect to the dimension. Therefore, although a nuclear-norm rate for Adam can be obtained directly from [1], it is slower by a factor of m+nmn than both our rate and the existing AdamW-style Shampoo rate. Moreover, the ℓ1 route cannot reach the m+n dependence because the ℓ1 rate is tight in the dimension regardless of the choice of ε [2]. Our analysis shows that Adam can attain the m+n nuclear-norm rate of AdamW-style Shampoo through the proposed row-column trace control argument, at the cost of requiring a larger value of ε. In the contribution section, we stated that we provide the first nuclear-norm convergence rate for Adam with the emphasis that it is of the same order as that of matrix optimizers such as AdamW-style Shampoo.