- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies high-probability convergence of momentum Muon under conditional -moment noise, . For exact-polar Muon with horizon-tuned parameters and fixed batch size, Theorem 1 establishes an average nuclear-gradient rate of , where . Theorem 2 extends this guarantee to NS5 under a pathwise inexact-LMO assumption, up to an additive approximation-error floor. For Adam with fixed , Theorem 3 constructs a uniformly normalized class of instances with bounded conditional -th noise moments, on which average gradient stationarity has a polynomial dependence on , uniformly over and positive constant learning rates. The paper compares the resulting worst-case confidence bounds and presents heavy-tailed noise experiments on Iris and a matched Qwen3-0.6B training comparison.
Critical strengths
-
S1. High-probability convergence without clipping. The main technical contribution is the control of heavy-tailed momentum errors. By capping the alignment deficit and tracking how past shocks decay, the authors obtain logarithmic dependence on without clipping or increasing the batch size. The block-supermartingale argument is interesting independently of the Adam comparison.
-
S2. Adam lower bound covering long-memory momentum. Theorem 3 covers all , including values close to one, with uniform constants across parameter-dependent instances. The rare-shock construction exploits the forgetting of second-moment information while signed first-moment information persists. This is a technically interesting extension of classical Adam counterexamples.
Critical weaknesses
-
W1. The scope and novelty of the separation need clarification. Theorem 3 assumes fixed , whereas Section 3.1 also discusses . The lower-bound mechanism does not directly cover this horizon-dependent regime. Moreover, Equation (C.4) already establishes a constant floor for a broad range of , and the proof of Lemma C.9 appears to imply a stronger piecewise lower bound than the stated common-exponent result. The choice to emphasize the weaker formulation is therefore unclear. Finally, polar normalization on the witnesses reduces to Euclidean normalization, so the comparison does not isolate a specifically matrix-geometric advantage of Muon.
-
W2. The NS5 guarantee relies on an unverified approximation condition. Assumption 4 requires sufficient alignment whenever , but finite-step NS5 alignment depends on the input singular-value distribution and normalization. The paper does not establish nonvacuous values of and for its implementation. As a result, Theorem 2 remains conditional, and its approximation floor has no quantified practical interpretation. The layerwise update scaling used in the Qwen experiment also needs to be reconciled with the theorem.
-
W3. The experiments do not directly validate the confidence separation. The radial Iris experiment uses , so positive scalar shocks cancel exactly. The six-learning-rate Adam sweep strengthens this scale-robustness comparison but does not test the proposed momentum mechanism. The directional-noise experiment omits Adam. The Qwen comparison uses one run per optimizer with gradient clipping and does not report upper-tail stationarity. The finite-pool GPD fits characterize the observed tail range but do not establish infinite variance.
Questions for authors
-
Q1 (W1). Do the cases in Lemma C.9, together with Section C.4, yield a lower bound uniform over all of when and otherwise, for sufficiently small fixed and sufficiently small ? If so, please state this stronger result and explain the role of the common-exponent envelope. Can the separation cover with constants uniform in ?
-
Q2 (W2). What spectral conditions guarantee the required NS5 alignment, and how do and depend on the input normalization and ? Can the authors justify nonvacuous values for their implementation and explain how Assumption 4 relates to existing NS approximation-error analyses?
-
Q3 (W3). Can the authors compare upper-tail stationarity for Adam and nonzero-momentum Muon under the directional noise oracle? Comparing fixed with on the rare-shock construction would also test the proposed mechanism directly.
-
Q4. What is the main technical novelty of Theorem 1 relative to Cutkosky and Mehta [1], Hübler et al. [2,4], and Choudhury et al. [3]? For Theorem 3, please clarify the distinction from Heilman and Mohanty [5], who establish nonvanishing online regret for Adam and a projected scalar Muon variant. In particular, how does their regret-based construction differ from the present stochastic-stationarity lower bound?
Required Corrections
- Incomplete NS5 specification (p. 3, Lines 129–141; p. 6, Lines 287–290; p. 16, Lines 859–863). Specify the exact five-step Newton–Schulz polynomial, input normalization, and numerical value of used in the experiments. Clarify whether the implemented update satisfies Assumption 4 for any justified and . The matrix-dependent update scaling is already reported in Appendix A.3.1.
References
- [1] Cutkosky, A., and Mehta, H. (2021). High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tails. NeurIPS 2021.
- [2] Hübler, F., Fatkhullin, I., and He, N. (2025). From Gradient Clipping to Normalization for Heavy Tailed SGD. AISTATS 2025
- [3] Choudhury, S., Cheng, X., Takáč, M., Na, S., and Kolar, M. (2026). Muon with Nesterov Momentum: Heavy-Tailed Noise and (Randomized) Inexact Polar Decomposition. arXiv:2605.06884.
- [4] Hübler, F., Pethick, T., and Sra, S. (2026). Free Heavy-Tailed Lunch for Muon: A Theoretical Justification of Empirical Success. arXiv:2606.14560.
- [5] Heilman, S., and Mohanty, S. (2026). On the Convergence of Adam, Revisited. arXiv:2607.03519.
6 comments
(continued)
Polar normalization reduces to Euclidean normalization on the instances. Yes, and this is exactly why we say that the confidence separation comes from normalization in time: the proof of Theorem 1 uses only two properties of the polar map, bounded updates and the dual descent certificate, so the same argument applies to momentum normalized SGD. The gain from matrix geometry appears elsewhere. The revision introduces the spectral smoothness constant (, ) and replaces by in Theorem 1 (Section 3.2, proof in Appendix B.2.6). In nuclear-norm stationarity at the same and satisfying Muon's noise condition, with tuned for each method, every term of Muon's guarantee except the common noise term is smaller than for Frobenius normalization by the factor , between and for the smallest . On the instances and this gain disappears, leaving only the confidence separation. See also our responses to Reviewers 4mz1 and ZbGo.
W2 / Q2 and required correction: NS5
Implementation details. The NS5 used in the experiments (Appendix A.3):
- runs in bf16, and first transposes the matrix if it has more rows than columns;
- normalizes with ;
- then performs five steps with .
We will add this information at lines 129 to 141, 287 to 290, and 859 to 863.
Values of and . NS5 applies the same scalar map to every singular value: it maps to , the fivefold iterate of . In exact arithmetic, for , and the output is smaller for smaller . This gives a sufficient spectral condition:
- if every nonzero singular value of satisfies , then and , so Assumption 4 holds with and : the factor in Theorem 2 is about , and there is no additional error floor;
- more generally, if the singular values below this threshold carry at most a fraction of the nuclear norm, Assumption 4 holds with .
These constants are for exact arithmetic; bf16 rounding perturbs them slightly. The condition can also fail, for example for matrices whose nuclear norm is concentrated on very small singular values, so for general inputs Theorem 2 remains conditional. We did not measure these quantities in the Qwen training; in the revision we will report the values of and during training. The pathwise form of Assumption 4 is adapted from Qian et al. (2026); the scalar analysis above connects it directly to how the NS iteration acts on singular values.
(continued in the next comment)
(continued)
Layerwise scaling in the Qwen experiment. In the Qwen training, the update of each matrix is multiplied by , with decoupled weight decay and gradient clipping; non-matrix parameters use AdamW. The layerwise scaling amounts to a deterministic step-size constant per matrix, but multiple matrix blocks, weight decay and clipping are outside the setting of Theorem 2. The Qwen experiment is therefore a comparison in real training, not an instance of Theorem 2. We will state this in the paper.
W3 / Q3: experiments
The reviewer's summary of the roles of the experiments is accurate:
- the radial Iris experiment uses , so positive scalar shocks cancel exactly; the six-learning-rate Adam sweep tests scale robustness, not the momentum mechanism;
- the directional-noise experiment does not include Adam;
- the Qwen comparison uses one run per optimizer with gradient clipping and does not report upper-tail stationarity;
- the GPD fits characterize the tails only within the observed range and do not establish infinite variance; we will revise the corresponding statements.
We will add the two experiments suggested by the reviewer:
- under the directional-noise oracle, a comparison of the upper-tail stationarity (quantiles for each ) of Adam and nonzero-momentum Muon;
- on the rare-shock construction, a comparison of Adam with fixed and with .
Q4: Differences from prior work
Theorem 1.
- Cutkosky and Mehta (2021): high-probability bounds for normalized SGD with clipping and momentum; our bound needs no clipping.
- Hübler et al. (2025): high-probability bounds for normalized SGD with increasing batch sizes, pointing out momentum as the difficulty; we handle momentum with a fixed batch size.
- Hübler et al. (2026) and Choudhury et al. (2026): expectation bounds for Muon under heavy-tailed noise. We use the same parameter schedule and strengthen the conclusion to a high-probability bound with dependence.
- Technical novelty: we control the error on blocks of the momentum length through conditional -th moments, and combine the blocks into a single confidence event through a log-shock supermartingale, which avoids per-step high-probability control. The noise may also grow with the gradient ().
Theorem 3 versus Heilman and Mohanty (2026). The two works answer different questions:
- they study regret in an online-learning framework, and show that Adam and a projected scalar Muon variant have non-vanishing regret;
- we study stochastic nonconvex optimization: a smooth objective and a stochastic oracle with bounded conditional -th moment are fixed before sampling, and the lower bound states that the average gradient norm exceeds a threshold with probability larger than , uniformly over and constant learning rates;
- in our setting, the Muon side is a positive high-probability guarantee rather than a failure example.
We thank the reviewer again.
We thank the reviewer for the very detailed review, including the careful reading of the proofs in Appendix C, and for recognizing the high-probability convergence without clipping (the block-supermartingale argument) and the Adam lower bound covering long-memory momentum.
W1 / Q1: scope of the separation, the stronger lower bound, and the 1×2 instances
The stronger piecewise lower bound. The reviewer's reading is correct. The case analysis of Lemma C.9 together with Section C.4 indeed gives a piecewise lower bound uniform over all γ>0: for every sufficiently small fixed ϑ>0 and sufficiently small δ, with probability larger than δ,
The cases come from:
Role of the common-exponent formulation. We used the form cmin{1,T−aδ−q} to write all cases with the same time exponent a=(p−1)/(3p−2) as Muon's upper bound, which reduces the comparison to the single ratio δ−q/(1+log(1/δ)). The price is that merging the c/log(eT) branch into this form requires log(eT)≤LζTζ with ζ=a−2(p+ϑ)q>0, which forces q<a/(2p). The piecewise form is stronger: at p=2 the confidence exponent is 1/4 rather than 1/32, and for p<2 we have (p−1)/(2p)<a, so even for fixed δ Adam's lower bound decays more slowly in T than Muon's upper bound. We will state the piecewise bound as the main form of Theorem 3 in the revision and keep the unified form for the direct comparison.
β2=1−1/T. With ϵ=0, yes: the new Theorem 4 (Section 3.3, proof in Appendix D) covers β2=1−1/T, and in fact any β2 that depends on T and δ (together with β1, γ and v0), with constants c and κ that depend only on p and are uniform in T; its conclusion is that no tuning attains Muon's guarantee CT−apolylog(1/δ) at high confidence. For ϵ>0, the constant of Theorem 3 depends on β2, and we do not claim uniformity as β2→1.
(continued in the next comment)