- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a new framework for normalized optimizers defined by rank-one decomposition, and distinguishes between two-sided and one-sided orthogonality among its members. The framework unifies normalized optimizers such as balanced SGD, Muon, and Lion and states criteria for defining new optimizers. Two such alternative optimizers tested are ROBN and SPU which provide evidence that the two-sided orthogonality structure of Muon is not crucial to its performance.
Critical strengths
The new framework introduced is an interesting and novel perspective through which to analyze the optimizers. The experiments consistently support the conclusion that one-sided orthogonal constructions can perform similarly or better to Muon's two sided orthogonality, opening a new family of potential optimizers. The paper additionally provides nice derivations and proofs in the Appendix.
Critical weaknesses
Although the framework is interesting, the paper shies away from answering the more useful questions about what the explanatory/predictive power of the framework is. The experiments support the limited conclusion that two-sided orthogonality is not necessary for Muon. However, they only weakly support the broader interpretation that this framework is the appropriate one through which to understand Muon. In fact, the proposed framework of one-sided constructions is quite broad, allowing for several significant design differences within it. As a result the paper thus presents a classification system without properly isolating the effect of its defining feature, one-sided orthogonality.
The ROBN experiments exemplify this concern. ROBN conflates multiple factors (time variance, random sampling, one-sided orthogonality, and basis selection), making its results difficult to attribute to orthogonality which is the primary focus of the paper. This is evidenced by the performance gap between the two variants which share the same one-sided condition but differ in basis selection. In addition, there are no other examples of time-varying optimizers provided or benchmarked other than ROBN which makes its results difficult to frame in context. A more direct experiment to isolate the effect of orthogonality would be to simply normalize the rows/columns (i.e. similar to balanced SGD). To that point, the paper doesn't benchmark balanced SGD or Lion despite significant discussion of them.
Questions for authors
- What makes one-sided orthogonality a useful organizational distinction? Both SGD (in the form of balanced SGD) and AdamW (holding second-moment fixed) can be adapted to the proposed framework, so does the framework pose any nontrivial restrictions to exclude optimizers?
- The induced quantity is assumed convex and its rank-one signed sum assumed to lie in . Is there a way to characterize which selection rules can guarantee this (e.g. fixed basis)?
- Is it possible to separate the impact of basis selection from orthogonality? Although different from Muon, SPU has a similar basis selection (especially compared to ROBN) which may be the better theoretical basis.
Minor feedback: Would recommend the authors to tighten the mathematical exposition
- Some quantities (e.g. ) are mentioned before being defined
- Inconsistency in variable usage (between and )
- is not explicitly given for Muon
6 comments
(2/2)
Q1. Does the framework exclude any optimizer? (Appendix I.2)
The first part of Q1, why one-sided orthogonality is a useful distinction, is answered under W1 (a)–(c) in the previous comment. In addition, the distinction makes our question well-posed: two-sided orthogonality singles out Muon exactly (Appendix A.2), and one-sided orthogonality drops one of its two constraints while keeping the system orthonormal, so it is the natural class in which to ask whether Muon's extra structure is necessary. We now address the second part.
Correcting the premise. Fixing the second moment gives u = m/(√ν̄ + ε): momentum SGD with a fixed per-coordinate scale, without Adam's adaptivity. Fixing its norm as in balanced SGD would, in turn, make it no longer AdamW with a fixed second moment. The framework does exclude optimizers, as follows.
Step 1 (size of framework updates). A framework update is ξₜ(m) = Σᵢ sign(aᵢ)Mᵢ over an orthonormal rank-one system. Only terms with aᵢ ≠ 0 survive, and they are orthonormal, so ‖ξₜ(m)‖²_F = #{i : aᵢ ≠ 0} ∈ {1, …, d₁d₂} for every m ≠ 0. However small m is, the update has norm at least one.
Step 2 (Adam-type updates vanish with the momentum). Holding past state fixed, an optimizer's update is a function Φ(m) of the current momentum.
- Adam, RMSProp, AdaGrad: the denominator √ν + ε is bounded away from zero by ε, or by an accumulated second moment that is positive in every coordinate, while the numerator is m (or g), so Φ(m) → 0 as m → 0.
- Shampoo and SOAP with accumulation: with positive-definite accumulated statistics (or damping), the preconditioners have operator norm bounded independently of m, so ‖Φ(m)‖_F ≤ C‖m‖_F.
- Plain SGD and AdamW with a fixed second moment: Φ is linear in m.
Step 3 (contradiction). If Φ(m) = γ·ξₜ(m) for a constant γ > 0, then along λm₀ the norm ‖Φ(λm₀)‖_F would stay at least γ, yet it tends to zero as λ → 0. Hence none of these optimizers is a framework update.
Remarks. The proposition concerns the size of the update, i.e., whether it is normalized; SGD likewise enters only in normalized form, as balanced SGD (§3.2). The criterion is not that m can be expanded in some orthonormal rank-one system (every matrix can) but that the update has the signed-sum form. The sign limit of Adam (β₁ = β₂ = 0, ε → 0) is the sign update of Lion, which lies inside the framework but outside the one-sided family. Conversely, idealized Shampoo and SOAP without accumulation reduce to Muon as ε → 0; e.g., Shampoo gives UΣ(Σ² + εI)^(−1/2)Vᵀ → UVᵀ (derivation and the conditions for SOAP in Appendix I.1).
Q2. Which selection rules guarantee the assumptions? (Appendix I.3)
Proposition. In the one-sided orthogonal family, if the orthogonal factors Pₜ = [p₁, …, p_{d₁}] ∈ ℝ^{d₁×d₁} with PₜᵀPₜ = I are chosen independently of m, then the assumptions of §3 hold automatically (analogously for the column side, with Qₜ ∈ ℝ^{d₂×d₂}, QₜᵀQₜ = I, chosen independently of m).
The selected basis nevertheless varies with the momentum. Only the orthogonal side is independent of m; the other factors qᵢ = mᵀpᵢ/‖mᵀpᵢ‖, and hence the rank-one basis {pᵢqᵢᵀ} of the whole matrix, adapt to m. Unlike a basis fixed on both sides, such as Lion's coordinate basis, this rule thus yields a momentum-dependent basis while guaranteeing the assumptions automatically.
Proof.
- (1) The other factors are determined by m: since Pₜ is square, PₜPₜᵀ = I, so m = Σᵢ pᵢpᵢᵀm = Σᵢ ‖mᵀpᵢ‖ pᵢqᵢᵀ with qᵢ = mᵀpᵢ/‖mᵀpᵢ‖.
- (2) 𝒦ₜ(m) = Σᵢ ‖mᵀpᵢ‖₂ is a sum of norms of the linear maps m ↦ mᵀpᵢ, hence a norm: convex and positively homogeneous.
- (3) If mᵀpᵢ ≠ 0, the gradient of the i-th term is pᵢqᵢᵀ; if mᵀpᵢ = 0, zero is a subgradient. Since the subdifferential of a sum contains the sum of the subdifferentials, Σᵢ sign(aᵢ) pᵢqᵢᵀ ∈ ∂𝒦ₜ(m).
Remarks. The simplest case fixes the orthogonal side throughout training: Pₜ = I in RMNP and in ROBN with τ = 0 when d₁ ≥ d₂, and Qₜ = I in ROBN with τ = 0 when d₁ < d₂; the orthogonal factors may also be resampled at each step, as in ROBN-Haar and ROBN-τ. A basis fixed on both sides, such as Lion's coordinate basis, is not one-sided orthogonal, but satisfies the assumptions trivially because 𝒦ₜ is then an ℓ₁ norm in a fixed basis. The condition is sufficient, not necessary: Muon and SPU depend on m on both sides and still satisfy the assumptions. Since ⟨ξₜ(m), m⟩ = 𝒦ₜ(m) holds by construction, for convex and positively homogeneous 𝒦ₜ the subgradient condition is equivalent to ξₜ(m) ∈ 𝒞ₜ (Appendix I.3); this holds for Muon because UVᵀ has operator norm one (Appendix A.2), and for SPU because each blockwise polar factor has operator norm one (Appendix C.1).
Minor. All three are addressed:
- Aₜ is now stated to be "defined below" where it first appears (§4.1);
- §4 defines Δₜ := ∂ₘ𝒦ₜ(mₜ), unifying the notation for the ROBN and SPU updates and the subgradient;
- Muon's selected subgradient ∂ₘ𝒦ₜ(mₜ) = UVᵀ is stated in §3.
Follow-up on Q3: the ~0.8B-parameter result and blockwise orthogonalization
Our larger-scale run (Appendix K) bears on Q3. There, SPU is comparable to or slightly better than Muon (final validation loss: SPU 2.9987, Muon 3.0003; perplexity: SPU 20.06, Muon 20.09; single seed). SPU reuses Muon's tuned hyperparameters without separate tuning.
For context, MuonBP (Khaled et al., ICLR 2026) reports that BlockMuon, which orthogonalizes each tensor-parallel shard independently, reaches a validation perplexity of 20.29 against Muon's 15.33 on a 960M-parameter model with four blocks. The two setups differ substantially (architecture, training and validation data, token budget, number of blocks, and learning rate), so we do not compare them directly. The point is only that blockwise orthogonalization by itself does not ensure Muon-level performance at this scale, and therefore does not by itself account for SPU's results.
This is consistent with our Discussion that performance also depends on how a realization within the one-sided orthogonal family is constructed. We do not claim, however, that one-sided orthogonality alone accounts for SPU's performance.
(1/2)
We thank the reviewer for finding the framework novel and the experiments consistent. Revisions:
Q1, Q2, and minor points follow in the next comment.
W1. The limited conclusion the reviewer finds supported, that two-sided orthogonality is not necessary for Muon, is exactly our claim. We revised sentences that read as the broader interpretation. E.g., the Discussion now ends:
We do not claim that family membership predicts performance; to our knowledge, no geometric view of normalized optimizers (e.g., Lion-𝒦, LMO-based) predicts which method performs best. Balanced SGD is a weaker member, and the ROBN variants differ. The distinction is nevertheless useful.
W2. ROBN was not designed to explain why the one-sided family performs well. As stated in §4.1:
The reviewer notes that the two variants "share the same one-sided condition but differ in basis selection." This is exactly our point: performance depends not only on the one-sided condition but also on which orthogonal matrix is selected. The Discussion draws this conclusion:
It underlies W1: (a) the controlled study holds one-sidedness fixed and varies only this selection, and (b) the design principle makes the selection rule the design variable.
ROBN is indeed the only time-varying method, but the main evidence for our claim is SPU (§5.1), a static realization, so our conclusion does not rely on time variation. We ran the suggested baselines against a rerun Muon (Appendix J): plain row/column normalization (ROBN with τ = 0) is +0.0018 (FineWeb-Edu) and +0.0151 (OpenWebText); balanced SGD is +0.085 and +0.055. Balanced SGD carries an extra equal-coefficient constraint, and its lower performance does not refute our claim. Lion lies outside the one-sided family (Figure 1).
Q3. We understand the question as asking whether SPU's gains are, in the end, those of orthogonalizing each block as Muon does the whole matrix. Our claim attributes SPU's performance neither to one-sided orthogonality nor to this orthogonalization.
Good performance thus cannot be attributed specifically to Muon-style orthogonalization, consistent with Muon being neither unique nor best in the family.