acceptodds
Under review as a conference paper at ICLR 2027

Is Muon Geometrically Unique? Orthonormal Rank-One Systems for Normalized Optimizers

Abstract

Recent normalized optimizers such as Lion and Muon highlight the importance of geometry in modern optimizer design. Yet it remains unclear whether Muon's two-sided singular-vector orthogonality is essential to its effectiveness or instead represents one realization of a broader geometric structure. We extend the Lion- perspective by representing the momentum variable in a possibly time-varying orthonormal rank-one system and defining as the corresponding coefficient-wise -type potential. Different selection rules over orthonormal rank-one systems recover SGD, Lion, and Muon within a single geometric framework. Within this framework, one-sided orthogonality defines a broader family that contains Muon's two-sided orthogonality as a structured special case. To probe whether this additional two-sided structure is necessary, we introduce Random Orthogonal Basis Normalization (ROBN) and Split Polar Update (SPU) as realizations of one-sided orthogonality. ROBN shows that one-sided orthogonality alone can retain a meaningful portion of Muon's empirical performance, while SPU shows that a realization based on one-sided orthogonality can match Muon across both vision and language modeling. These results suggest that Muon is best understood as a distinguished realization of two-sided orthogonality within the broader geometric principle of one-sided orthogonality, rather than as a uniquely privileged geometric structure.

Primary area:
optimization
Keywords:
deep learning, optimization, first-order methods, optimization geometry, Muon, orthogonality

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DgoKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a new framework for normalized optimizers defined by rank-one decomposition, and distinguishes between two-sided and one-sided orthogonality among its members. The framework unifies normalized optimizers such as balanced SGD, Muon, and Lion and states criteria for defining new optimizers. Two such alternative optimizers tested are ROBN and SPU which provide evidence that the two-sided orthogonality structure of Muon is not crucial to its performance.

Critical strengths

The new framework introduced is an interesting and novel perspective through which to analyze the optimizers. The experiments consistently support the conclusion that one-sided orthogonal constructions can perform similarly or better to Muon's two sided orthogonality, opening a new family of potential optimizers. The paper additionally provides nice derivations and proofs in the Appendix.

Critical weaknesses

Although the framework is interesting, the paper shies away from answering the more useful questions about what the explanatory/predictive power of the framework is. The experiments support the limited conclusion that two-sided orthogonality is not necessary for Muon. However, they only weakly support the broader interpretation that this framework is the appropriate one through which to understand Muon. In fact, the proposed framework of one-sided constructions is quite broad, allowing for several significant design differences within it. As a result the paper thus presents a classification system without properly isolating the effect of its defining feature, one-sided orthogonality.

The ROBN experiments exemplify this concern. ROBN conflates multiple factors (time variance, random sampling, one-sided orthogonality, and basis selection), making its results difficult to attribute to orthogonality which is the primary focus of the paper. This is evidenced by the performance gap between the two variants which share the same one-sided condition but differ in basis selection. In addition, there are no other examples of time-varying optimizers provided or benchmarked other than ROBN which makes its results difficult to frame in context. A more direct experiment to isolate the effect of orthogonality would be to simply normalize the rows/columns (i.e. similar to balanced SGD). To that point, the paper doesn't benchmark balanced SGD or Lion despite significant discussion of them.

Questions for authors
  1. What makes one-sided orthogonality a useful organizational distinction? Both SGD (in the form of balanced SGD) and AdamW (holding second-moment fixed) can be adapted to the proposed framework, so does the framework pose any nontrivial restrictions to exclude optimizers?
  2. The induced quantity Kt\mathcal{K}_t is assumed convex and its rank-one signed sum assumed to lie in ∂Kt\partial \mathcal{K}_t. Is there a way to characterize which selection rules can guarantee this (e.g. fixed basis)?
  3. Is it possible to separate the impact of basis selection from orthogonality? Although different from Muon, SPU has a similar basis selection (especially compared to ROBN) which may be the better theoretical basis.

Minor feedback: Would recommend the authors to tighten the mathematical exposition

  • Some quantities (e.g. AtA_t) are mentioned before being defined
  • Inconsistency in variable usage (between Δt\Delta_t and ∂Kt\partial \mathcal{K}_t)
  • ∂Kt\partial \mathcal{K}_t is not explicitly given for Muon
6 comments

(1/2)

We thank the reviewer for finding the framework novel and the experiments consistent. Revisions:

  • the Abstract, Introduction, and Discussion state the comparative claim of Section 5 directly;
  • new Appendix I (relation to existing optimizers; Q1, Q2);
  • new Appendix J (controlled experiments; W2, Q3).

Q1, Q2, and minor points follow in the next comment.

W1. The limited conclusion the reviewer finds supported, that two-sided orthogonality is not necessary for Muon, is exactly our claim. We revised sentences that read as the broader interpretation. E.g., the Discussion now ends:

"Muon should be understood as one well-performing realization within the broader one-sided orthogonal family, rather than as a uniquely privileged geometric structure."

We do not claim that family membership predicts performance; to our knowledge, no geometric view of normalized optimizers (e.g., Lion-𝒦, LMO-based) predicts which method performs best. Balanced SGD is a weaker member, and the ROBN variants differ. The distinction is nevertheless useful.

  • (a) Controlled study. Holding one-sidedness fixed and varying only ROBN's rotation, the gap to Muon shrinks as the rotation approaches the identity (FineWeb-Edu: Haar +0.035, τ = 0.5 +0.017, τ = 0 +0.002; OpenWebText: +0.047, +0.032, +0.015). The comparison is approximate, as τ = 0 ran on a different GPU setup (Appendix J).
  • (b) Design principle. The selection rule itself becomes a design variable, from which competitive methods arise naturally: SPU and, with the orthogonal side fixed to the identity (ROBN with τ = 0), an update of the same type as the recent RMNP (Q3), "a new family of potential optimizers," as the reviewer notes.
  • (c) Explanatory content. Two-sided orthogonality holds exactly for the SVD (Appendix A.2), and the framework has a provable boundary (Q1).

W2. ROBN was not designed to explain why the one-sided family performs well. As stated in §4.1:

"ROBN is designed to test whether one-sided orthogonality can retain a meaningful portion of Muon's empirical performance when an orthogonal matrix Aₜ … is randomly resampled independently of the current momentum at every step."

The reviewer notes that the two variants "share the same one-sided condition but differ in basis selection." This is exactly our point: performance depends not only on the one-sided condition but also on which orthogonal matrix is selected. The Discussion draws this conclusion:

"Thus, performance also depends on how a realization within the one-sided orthogonal family is constructed."

It underlies W1: (a) the controlled study holds one-sidedness fixed and varies only this selection, and (b) the design principle makes the selection rule the design variable.

ROBN is indeed the only time-varying method, but the main evidence for our claim is SPU (§5.1), a static realization, so our conclusion does not rely on time variation. We ran the suggested baselines against a rerun Muon (Appendix J): plain row/column normalization (ROBN with τ = 0) is +0.0018 (FineWeb-Edu) and +0.0151 (OpenWebText); balanced SGD is +0.085 and +0.055. Balanced SGD carries an extra equal-coefficient constraint, and its lower performance does not refute our claim. Lion lies outside the one-sided family (Figure 1).

Q3. We understand the question as asking whether SPU's gains are, in the end, those of orthogonalizing each block as Muon does the whole matrix. Our claim attributes SPU's performance neither to one-sided orthogonality nor to this orthogonalization.

  • (i) SPU's direction is not Muon's. Updates from the same momentum differ by a stable angle from about step 2,000 to the end of training: 18–19° for MLP (about 10% of the SPU update orthogonal to Muon's direction), 32–33° for QKV (about 29%), and 33–34° for the attention output (about 31%). With norms controlled, the SPU direction is ahead on FineWeb-Edu at both norms (−0.0044, 25/25 evaluation points; −0.0034, 24/25) and on OpenWebText at learning rate 2×10⁻² (−0.0013, 18/20; −0.0021, 15/20), but small and of mixed sign at the tuned one (+0.0028, −0.0001; Appendix J.4, J.7).
  • (ii) ROBN with τ = 0 keeps one-sided orthogonality but drops polar-based basis selection, yet is +0.0018 (FineWeb-Edu) and +0.0151 (OpenWebText) from Muon.
  • (iii) RMNP (Deng et al., ICML 2026; arXiv:2603.20527), an identity-basis method of the same type as ROBN with τ = 0, reports results comparable to or slightly better than Muon in GPT-2 (125M–770M) and LLaMA (60M–1B) pretraining. Implementation details differ: RMNP always normalizes the rows of the output-by-input matrix with Muon's shape scale, whereas ROBN with τ = 0 normalizes along the larger dimension without it. Nevertheless, RMNP likewise belongs to the one-sided orthogonal family.

Good performance thus cannot be attributed specifically to Muon-style orthogonalization, consistent with Muon being neither unique nor best in the family.

(2/2)

Q1. Does the framework exclude any optimizer? (Appendix I.2)

The first part of Q1, why one-sided orthogonality is a useful distinction, is answered under W1 (a)–(c) in the previous comment. In addition, the distinction makes our question well-posed: two-sided orthogonality singles out Muon exactly (Appendix A.2), and one-sided orthogonality drops one of its two constraints while keeping the system orthonormal, so it is the natural class in which to ask whether Muon's extra structure is necessary. We now address the second part.

Correcting the premise. Fixing the second moment gives u = m/(√ν̄ + ε): momentum SGD with a fixed per-coordinate scale, without Adam's adaptivity. Fixing its norm as in balanced SGD would, in turn, make it no longer AdamW with a fixed second moment. The framework does exclude optimizers, as follows.

Step 1 (size of framework updates). A framework update is ξₜ(m) = Σᵢ sign(aᵢ)Mᵢ over an orthonormal rank-one system. Only terms with aᵢ ≠ 0 survive, and they are orthonormal, so ‖ξₜ(m)‖²_F = #{i : aᵢ ≠ 0} ∈ {1, …, d₁d₂} for every m ≠ 0. However small m is, the update has norm at least one.

Step 2 (Adam-type updates vanish with the momentum). Holding past state fixed, an optimizer's update is a function Φ(m) of the current momentum.

  • Adam, RMSProp, AdaGrad: the denominator √ν + ε is bounded away from zero by ε, or by an accumulated second moment that is positive in every coordinate, while the numerator is m (or g), so Φ(m) → 0 as m → 0.
  • Shampoo and SOAP with accumulation: with positive-definite accumulated statistics (or damping), the preconditioners have operator norm bounded independently of m, so ‖Φ(m)‖_F ≤ C‖m‖_F.
  • Plain SGD and AdamW with a fixed second moment: Φ is linear in m.

Step 3 (contradiction). If Φ(m) = γ·ξₜ(m) for a constant γ > 0, then along λm₀ the norm ‖Φ(λm₀)‖_F would stay at least γ, yet it tends to zero as λ → 0. Hence none of these optimizers is a framework update.

Remarks. The proposition concerns the size of the update, i.e., whether it is normalized; SGD likewise enters only in normalized form, as balanced SGD (§3.2). The criterion is not that m can be expanded in some orthonormal rank-one system (every matrix can) but that the update has the signed-sum form. The sign limit of Adam (β₁ = β₂ = 0, ε → 0) is the sign update of Lion, which lies inside the framework but outside the one-sided family. Conversely, idealized Shampoo and SOAP without accumulation reduce to Muon as ε → 0; e.g., Shampoo gives UΣ(Σ² + εI)^(−1/2)Vᵀ → UVᵀ (derivation and the conditions for SOAP in Appendix I.1).

Q2. Which selection rules guarantee the assumptions? (Appendix I.3)

Proposition. In the one-sided orthogonal family, if the orthogonal factors Pₜ = [p₁, …, p_{d₁}] ∈ ℝ^{d₁×d₁} with PₜᵀPₜ = I are chosen independently of m, then the assumptions of §3 hold automatically (analogously for the column side, with Qₜ ∈ ℝ^{d₂×d₂}, QₜᵀQₜ = I, chosen independently of m).

The selected basis nevertheless varies with the momentum. Only the orthogonal side is independent of m; the other factors qᵢ = mᵀpᵢ/‖mᵀpᵢ‖, and hence the rank-one basis {pᵢqᵢᵀ} of the whole matrix, adapt to m. Unlike a basis fixed on both sides, such as Lion's coordinate basis, this rule thus yields a momentum-dependent basis while guaranteeing the assumptions automatically.

Proof.

  • (1) The other factors are determined by m: since Pₜ is square, PₜPₜᵀ = I, so m = Σᵢ pᵢpᵢᵀm = Σᵢ ‖mᵀpᵢ‖ pᵢqᵢᵀ with qᵢ = mᵀpᵢ/‖mᵀpᵢ‖.
  • (2) 𝒦ₜ(m) = Σᵢ ‖mᵀpᵢ‖₂ is a sum of norms of the linear maps m ↦ mᵀpᵢ, hence a norm: convex and positively homogeneous.
  • (3) If mᵀpᵢ ≠ 0, the gradient of the i-th term is pᵢqᵢᵀ; if mᵀpᵢ = 0, zero is a subgradient. Since the subdifferential of a sum contains the sum of the subdifferentials, Σᵢ sign(aᵢ) pᵢqᵢᵀ ∈ ∂𝒦ₜ(m).

Remarks. The simplest case fixes the orthogonal side throughout training: Pₜ = I in RMNP and in ROBN with τ = 0 when d₁ ≥ d₂, and Qₜ = I in ROBN with τ = 0 when d₁ < d₂; the orthogonal factors may also be resampled at each step, as in ROBN-Haar and ROBN-τ. A basis fixed on both sides, such as Lion's coordinate basis, is not one-sided orthogonal, but satisfies the assumptions trivially because 𝒦ₜ is then an ℓ₁ norm in a fixed basis. The condition is sufficient, not necessary: Muon and SPU depend on m on both sides and still satisfy the assumptions. Since ⟨ξₜ(m), m⟩ = 𝒦ₜ(m) holds by construction, for convex and positively homogeneous 𝒦ₜ the subgradient condition is equivalent to ξₜ(m) ∈ 𝒞ₜ (Appendix I.3); this holds for Muon because UVᵀ has operator norm one (Appendix A.2), and for SPU because each blockwise polar factor has operator norm one (Appendix C.1).

Minor. All three are addressed:

  • Aₜ is now stated to be "defined below" where it first appears (§4.1);
  • §4 defines Δₜ := ∂ₘ𝒦ₜ(mₜ), unifying the notation for the ROBN and SPU updates and the subgradient;
  • Muon's selected subgradient ∂ₘ𝒦ₜ(mₜ) = UVᵀ is stated in §3.

Follow-up on Q3: the ~0.8B-parameter result and blockwise orthogonalization

Our larger-scale run (Appendix K) bears on Q3. There, SPU is comparable to or slightly better than Muon (final validation loss: SPU 2.9987, Muon 3.0003; perplexity: SPU 20.06, Muon 20.09; single seed). SPU reuses Muon's tuned hyperparameters without separate tuning.

For context, MuonBP (Khaled et al., ICLR 2026) reports that BlockMuon, which orthogonalizes each tensor-parallel shard independently, reaches a validation perplexity of 20.29 against Muon's 15.33 on a 960M-parameter model with four blocks. The two setups differ substantially (architecture, training and validation data, token budget, number of blocks, and learning rate), so we do not compare them directly. The point is only that blockwise orthogonalization by itself does not ensure Muon-level performance at this scale, and therefore does not by itself account for SPU's results.

This is consistent with our Discussion that performance also depends on how a realization within the one-sided orthogonal family is constructed. We do not claim, however, that one-sided orthogonality alone accounts for SPU's performance.

Reviewer_a2cGOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper examines whether Muon's two-sided singular-vector geometry is necessary for strong optimization performance. It introduces an orthonormal rank-one framework encompassing normalized SGD, Lion, and Muon, and investigates two alternatives: random orthogonal basis normalization (ROBN) and split polar update (SPU). Ideal SPU is acknowledged as a two-block special case of BlockMuon [1]. The theory extends the Lion-K Lyapunov analysis [2] to deterministic momentum dynamics with time-varying potentials under additional assumptions. Three-seed experiments on ImageNet-1K, FineWeb-Edu, and OpenWebText show modest but consistent mean improvements for SPU over Muon, while ROBN produces mixed results. The authors argue that one-sided orthogonality provides a broader explanation of Muon's effectiveness.

Critical strengths
  • S1. A useful geometric framework. The balanced normalized-SGD representation and the characterization of two-sided representations as SVDs provide a clear connection among normalized optimizers. The distinction between orthogonality of rank-one components and orthogonality of both factor systems is conceptually useful.

  • S2. Consistent empirical results for SPU. SPU improves every reported mean final metric over Muon across three benchmarks without separate tuning. Matched training protocols, parameter groups, and language-modeling batches strengthen the comparison, although they do not establish the cause of the gains.

Critical weaknesses
  • W1. SPU does not isolate the geometric mechanism behind its gains. For an even row split with r≥2cr\ge 2c and full-column-rank blocks, the ideal unscaled SPU update satisfies D⊤D=2IcD^\top D=2I_c, with a symmetric result for column splitting. Thus, SPU can change global singular-vector alignment while retaining a flat update spectrum. Its success does not establish one-sided orthogonality, rather than spectral flattening or blockwise alignment, as the key mechanism. The actual NS5 update spectra are not examined.

  • W2. Missing normalization controls confound the empirical conclusions. In W1's regime, ideal SPU has a 2\sqrt{2} larger Frobenius update norm than Muon before layer scaling. Using the same learning rate and weight decay therefore does not ensure matched effective update scales. ROBN offers limited independent support: ROBN-Haar underperforms AdamW on both language tasks, while ROBN-τ\tau recovers only about 4.8% of Muon's mean loss improvement over AdamW on OpenWebText. Norm-matched SPU, identity-basis ROBN (τ=0\tau=0), and normalized-momentum baselines are needed to separate these effects.

  • W3. The convergence analysis has limited demonstrated applicability to resampled ROBN. The time-varying theorem assumes common-domain feasibility, no-spike regularity, and a single limiting-gap potential. Since ηt→0\eta_t\to0, no-spike requires consecutive gap differences to vanish, whereas independent rotations can sustain fluctuations near nonzero momentum. The paper does not establish these conditions for the proposed resampling scheme. This is a limitation of the theorem's applicability, not necessarily of its conditional proof.

Questions for authors
  • Q1 (W1). Is the key claim that global singular-vector alignment is unnecessary, that spectral flattening is unnecessary, or both? What fraction of optimized parameters satisfies W1's dimensional conditions, and what do the actual NS5 update spectra reveal?

  • Q2 (W2). Does SPU still outperform Muon after per-layer update-norm matching, with weight decay and auxiliary updates unchanged? How do ROBN with τ=0\tau=0 and Frobenius-normalized momentum perform under matched scales and comparable tuning budgets?

  • Q3 (W3). Under what verifiable assumptions does independently resampled ROBN satisfy conditions (310), (313)–(314), and (316), with a common K∞\mathcal K_\infty? In particular, how are fresh-rotation fluctuations reconciled with the no-spike condition?

Required Corrections

  • SPU scaling (Appendix C.1, lines 1569–1595; Algorithm 3, lines 3870–3873). Distinguish the unscaled exact-polar geometry from the layer-scaled idealization corresponding to Algorithm 3. For the latter, the potential is sℓKSPUs_\ell\mathcal{K}_{\mathrm{SPU}} and the dual block constraints have radius sℓs_\ell; finite-step NS remains an additional approximation.

  • Implementation details (Algorithms 1–3). Specify the Newton–Schulz polynomial, input normalization, internal precision, numerical value of the ROBN stabilization constant, and any method-specific shape scaling. Align the auxiliary-optimizer wording in Algorithms 1–3 with the excluded matrix parameters already described in Appendix H.

  • Mathematical wording and typos. Qualify the claim that increasing τ\tau increases the deviation of exp⁡(τS)\exp(\tau S) from identity (Appendix F, lines 3778–3779), since this need not hold globally. Include Eq. (362) in the derivation of Eq. (365) (line 2042), and correct “qiang liu” to “Qiang Liu” (line 529).

References

[1] Khaled et al. (2026). MuonBP: Faster Muon via Block-Periodic Orthogonalization. ICLR 2026.

[2] Chen et al. (2024). Lion Secretly Solves a Constrained Optimization: As Lyapunov Predicts. ICLR 2024.

3 comments

(1/2)

We thank the reviewer for the detailed review. We agree with S2 that our comparisons do not establish the cause of SPU's gains; our claim does not require it. Revisions:

  • the Abstract, Introduction, and Discussion now state the comparative claim of Section 5 directly;
  • new Appendix J reports the requested controls;
  • all required corrections have been made.

Our answers to W2/Q2 and W3/Q3 continue in the next comment.

Scope of the claim. The Abstract states:

"These results suggest that Muon, the realization characterized by two-sided orthogonality, is best understood as one well-performing realization within the broader one-sided orthogonal family, rather than as a uniquely privileged geometric structure."

We do not claim that one-sided orthogonality is the key mechanism behind SPU's gains.

W1/Q1.

  • (i) What we claim to be unnecessary is global singular-vector alignment, i.e., two-sided orthogonality; we make no claim that spectral flattening is unnecessary.
  • (ii) The dimensional condition (r ≥ 2c or c ≥ 2r) holds for 11/12 of the parameters updated by the matrix optimizer (QKV and both MLP matrices; only the square attention output is excluded).
  • (iii) We measured the actual NS5 spectra by the stable rank ‖D‖²_F/‖D‖²₂, which equals its maximum min(r, c) = 768 exactly when all singular values are equal and decreases as the spectrum becomes less flat (Appendix J.7, Table 15). On rectangular matrices, SPU's stable rank is similar to or higher than Muon's (FineWeb-Edu / OpenWebText: MLP up 471 vs. 439 / 473 vs. 440; MLP down 485 vs. 467 / 483 vs. 454; QKV 454 vs. 472 / 471 vs. 466). On the square matrix, SPU is not flat even ideally: 260 vs. 399, with largest singular value 1.57 vs. 1.20 (FineWeb-Edu).

Whether flattening or the change in singular-vector alignment drives SPU's gains does not bear on our claim (Appendix J).

(2/2)

W2/Q2 (Appendix J; seed 0, final validation loss, same setup and validation batches).

  • Norm-matched SPU (Appendix J.2–J.4): rectangular-matrix updates divided by √2, square matrices unchanged, weight decay and auxiliary AdamW unchanged. It beats Muon on FineWeb-Edu (2.7850 vs. 2.7894, ahead at 25/25 evaluation points); on OpenWebText it is +0.0028 at Muon's tuned learning rate and −0.0013 (18/20) at 2×10⁻².

  • 2×2 design (Appendix J.4): crossing direction (SPU/Muon) with rectangular-matrix norm (1/√2), the norm effect is large, but a direction effect clearly exists: the SPU direction is ahead at both norms on FineWeb-Edu (−0.0044, −0.0034) and on OpenWebText at 2×10⁻² (−0.0013, −0.0021); at Muon's tuned learning rate it is small and of mixed sign (+0.0028, −0.0001). The interaction is small relative to the scale effect, so SPU's advantage cannot be reduced to either the norm or the direction alone; both are geometric properties of SPU. The √2 follows from SPU's split geometry; multiplying Muon's updates by √2 is an external rescaling absent from Muon's selection rule under its standard shape scale (Jordan et al., 2024; Appendix J.5).

  • Per-layer norms (Appendix J.7): because of the Newton–Schulz approximation, per-layer norm ratios deviate from the ideal by up to about 9% (attention output); the total-norm ratio stays within about 1% after step 500.

  • ROBN with τ = 0 and normalized momentum (Appendix J.6): +0.0018 and +0.0151 (FineWeb-Edu, OpenWebText) for ROBN with τ = 0; +0.0854 and +0.0547 for balanced SGD. Both use Muon's tuned learning rate without separate tuning, as do ROBN and SPU in Table 1; the update norm of both baselines in the MLP down-projection is twice Muon's. Balanced SGD carries an extra equal-coefficient constraint, and its lower performance does not refute our claim.

  • Role of ROBN. ROBN is not offered as independent support for SPU's gains. As stated in §4.1:

    "ROBN is designed to test whether one-sided orthogonality can retain a meaningful portion of Muon's empirical performance when an orthogonal matrix Aₜ … is randomly resampled independently of the current momentum at every step."

    The comparison for ROBN is within the one-sided orthogonal family: ROBN and Muon both belong to it, and "meaningful portion" is measured against Muon's absolute performance, i.e., how much of Muon's performance another member of the family retains. The reviewer's 4.8% is a different measure, the fraction of Muon's improvement over AdamW. AdamW lies outside the framework altogether (Appendix I.2) and is therefore a reference point, not a comparator; the Table 1 caption states:

    "AdamW is included as a widely used reference point; the primary comparison for ROBN and SPU is Muon."

W3/Q3. We accept these points. Appendix E already states these conditions as explicit assumptions for resampled ROBN, and, as the reviewer notes, this limits the theorem's applicability rather than its conditional proof; the main text likewise claims convergence only under the stated assumptions. For the static realizations, balanced SGD, Lion, Muon, and SPU with a fixed partition, the three conditions are not assumed but proved, given λX₀ ∈ 𝒞 and the standing assumptions:

  • common-domain (Appendix C.1),
  • no-spike (Appendix D.2),
  • limiting-gap (Appendix D).

The theorem thus applies to idealized (exact-polar) SPU and Muon, our main evidence, without these three assumptions; only resampled ROBN remains unverified.

Thank you for your detailed response and the effort put into revising the manuscript. I accept the narrowed scope of the paper's claims, concerning in W1. I appreciate the addition of the requested control experiments. Although the results show mixed signs and rely on a single seed, they are sufficient to support the narrowed claim that SPU performs comparably to Muon. The authors have acknowledged that the convergence conditions for the resampled ROBN are not verified, which remains a theoretical limitation. Taking these updates into account, I am raising my score from 1 to 2.

Reviewer_vXZfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether Muon's two-sided orthogonality is essential for its performance. The authors propose a unified framework based on orthonormal rank-one systems and introduce two methods, ROBN and SPU. ROBN does not guarantee global orthogonality of the update, while SPU performs blockwise polar normalization. Experiments on vision and language modeling tasks show that SPU slightly outperforms Muon.

Critical strengths

The framework provides a common geometric interpretation of several normalized optimizers. SPU achieves modest, consistent improvements over Muon across vision and language-modeling experiments.

Critical weaknesses

Limited novelty and insufficient motivation. As acknowledged in §4.2, SPU’s ideal update is a two-block special case of BlockMuon. The additional insight provided by the framework remains unclear, and ROBN lacks a convincing optimization intuition. Repetitive presentation further obscures the main contribution.

Insufficient evidence for the central claim. SPU retains blockwise polar normalization, while ROBN generally underperforms Muon. These results do not clearly establish that one-sided orthogonality explains Muon’s effectiveness. The small gains also require validation under longer training and at larger scales.

Questions for authors

1), What new insight does the framework provide beyond existing BlockMuon and norm-based interpretations? 2), What is the intuition behind ROBN, and why should randomly resampling the basis improve optimization? 3), How can the benefits of orthogonality be separated from those of block partitioning and update scaling? 4), How do Shampoo and SOAP fit into the proposed framework? 5), Do SPU’s improvements persist with longer training, such as 50B tokens, and larger models?

5 comments

(1/2)

We thank the reviewer for recognizing the common geometric interpretation and SPU's modest but consistent improvements. Revisions:

  • the Abstract, Introduction, and Discussion now state the comparative claim of Section 5 directly;
  • new Appendix I (relation to existing optimizers; Q4);
  • new Appendix J (controlled experiments; Q3).

Our answers to Q3–Q5 continue in the next comment.

Weakness A and Q1 (novelty, BlockMuon, new insight). §4.2 states:

"the SPU construction arises from the one-sided orthogonality framework and is used to investigate the role of Muon's two-sided singular-vector structure."

SPU and BlockMuon, as analyzed in MuonBP (Khaled et al., ICLR 2026; arXiv:2510.16981), come from different design principles.

BlockMuon (as analyzed in MuonBP)SPU
Design principleDistributed trainingOne-sided orthogonality framework (§4.2)
PurposeEfficient orthogonalization under model parallelism: orthogonalization is applied "independently to matrix shards on each device," and in these block steps MuonBP "does not require any additional communication" (MuonBP)Investigate the role of Muon's two-sided singular-vector structure (§4.2)
How blocks are determinedMatrix shards on each device, set by the parallelism layout (column, row, or 2D grid)Explicit rule: two halves along the larger dimension
One-sided orthogonalityOnly when split along a single direction; not with a 2D gridAlways (the factors on the split side are mutually orthogonal)
RelationshipCoincides with SPU only when split into two halves along the larger dimension—

With a 2D grid, neither the left factors (within a block row) nor the right factors (within a block column) are mutually orthogonal.

The insight beyond norm-based views is already stated in the Discussion:

"This perspective differs from norm- or LMO-based viewpoints, which often characterize an update after a feasible geometry or norm has been specified. In our framework, the selection rule itself is a design variable."

From this principle, competitive methods arise naturally: SPU and, with the orthogonal side fixed to the identity (ROBN with τ = 0), an update of the same type as the recent RMNP (Deng et al., ICML 2026; arXiv:2603.20527).

Q2 (ROBN). ROBN is not proposed as a method that improves optimization. As stated in §4.1:

"ROBN is designed to test whether one-sided orthogonality can retain a meaningful portion of Muon's empirical performance when an orthogonal matrix Aₜ … is randomly resampled independently of the current momentum at every step."

Random resampling is thus a deliberately adverse test condition, not a mechanism for improvement: if even momentum-independent random bases retain a meaningful portion of Muon's performance, the one-sided family is a meaningful family rather than a mere relaxation of Muon. More broadly, SPU and ROBN serve as evidence for our claim that Muon is a strong member of the one-sided orthogonal family but neither its unique nor its best realization. Whether each method carries an optimization intuition of its own lies outside our scope.

Weakness B (evidence). Accordingly, we do not claim through SPU or ROBN that one-sided orthogonality explains Muon's effectiveness. We revised the Discussion to state the comparative claim directly:

"Muon should be understood as one well-performing realization within the broader one-sided orthogonal family, rather than as a uniquely privileged geometric structure."

SPU does retain blockwise polar normalization, but it is a different optimizer from Muon, which orthogonalizes the whole matrix; see Q3. For scale, see Q5.

(2/2)

Q3 (separating orthogonality from partitioning and scaling; Appendix J).

  • SPU's update is not Muon's. Computed from the same momentum, the two updates differ by a stable angle from about step 2,000 to the end of training: 18–19° (MLP), 32–33° (QKV), and 33–34° (attention output).
  • A direction effect clearly exists. A 2×2 design crosses the direction (SPU or Muon) with the rectangular-matrix norm (1 or √2). The norm effect is large, but at matched norm, the SPU direction is ahead on FineWeb-Edu at both norms (−0.0044, −0.0034) and on OpenWebText at learning rate 2×10⁻², while at Muon's tuned learning rate it is small and of mixed sign (+0.0028, −0.0001). The interaction is small relative to the scale effect, so SPU's advantage cannot be reduced to either alone; both are geometric properties of SPU.
  • The √2 comes from SPU's geometry, not from Muon. The √2 factor follows from SPU's split geometry, whereas multiplying Muon's rectangular-matrix updates by √2 is an external rescaling absent from Muon's selection rule under its standard shape scale (Jordan et al., 2024; Appendix J.5). Thus √2-scaled Muon borrows the norm that SPU provides by construction; its improvement does not indicate under-tuning of Muon.

Q4 (Shampoo and SOAP; Appendix I).

  • Without accumulation, idealized Shampoo and SOAP reduce to Muon as ε → 0. Both rotate into the singular vectors of the momentum and differ only in how they map the singular values to one: an inverse square root for Shampoo and a sign for SOAP; e.g., Shampoo gives UΣ(Σ² + εI)^(−1/2)Vᵀ → UVᵀ (details and conditions in Appendix I.1).
  • With accumulation, they lie outside the framework: their updates shrink to zero with the momentum, whereas every framework update has norm at least one (proof in Appendix I.2).

Q5 (scale). We are running a larger-scale comparison and will post the results by November 16. The setting was chosen as follows:

  • Established recipe: the Llama-520M recipe of Wen et al. (ICLR 2026; arXiv:2509.02046) (0.54B non-embedding, 0.80B total parameters), with Muon's hyperparameters taken from their tuned sweep and SPU using the same values without separate tuning.
  • Tokens: 21.5B tokens, twice the Chinchilla-optimal budget for this size.
  • Compute: about 7×10¹⁹ FLOPs (6ND, non-embedding), more than a 210M model trained on 50B tokens (about 5×10¹⁹, also non-embedding).

Follow-up on Q5 (scale): results of the ~0.8B-parameter run

We completed the larger-scale comparison announced in our response: the Llama-520M recipe of Wen et al. (ICLR 2026; arXiv:2509.02046) (0.54B non-embedding, 0.80B total parameters), 21.5B tokens, Muon with the tuned hyperparameters of that recipe, and SPU reusing them without separate tuning. This is a single seed.

  • Final validation loss (Paloma C4 English): Muon 3.0003, SPU 2.9987 (−0.0016).
  • Throughout training: SPU is below Muon at all 41 evaluation points (every 1,000 steps and at the end of training).
  • Margin: smaller than in Table 1 (FineWeb-Edu −0.0108, OpenWebText −0.0077).

As this is a single seed, we do not claim that SPU is better at this scale. SPU remains comparable to or slightly better than Muon: the gain shrinks, but the trend of Table 1 is maintained. This is what our claim requires: SPU, which lacks Muon's two-sided structure, still matches Muon at this scale. We have added this result to Appendix K.

Comments on the paper

2 comments

General response: revised manuscript uploaded

We thank all reviewers for their careful and constructive feedback. We have uploaded a revised manuscript. All changes from the submitted version are marked in red: changed sentences in the main text and in Appendices D, F, G, and H are shown in red, and the three new appendices are marked by red titles (their content is kept in black for readability).

Main changes

  • Statement of the claim. The Abstract, Introduction, and Discussion now state the comparative claim of Section 5 directly: Muon, the realization characterized by two-sided orthogonality, is one well-performing realization within the broader one-sided orthogonal family, rather than a uniquely privileged geometric structure. The claim itself is unchanged.
  • New Appendix I (Relation to Existing Optimizers and Scope of the Framework). I.1 shows that idealized Shampoo and SOAP without accumulation reduce to Muon; I.2 proves that optimizers whose updates vanish with the momentum (e.g., Adam, RMSProp, AdaGrad, accumulated Shampoo/SOAP, plain SGD) lie outside the framework; I.3 shows that choosing the orthogonal side independently of the momentum guarantees the assumptions of Section 3.
  • New Appendix J (Controlled Experiments). Norm- and direction-controlled variants of SPU and Muon in a 2×2 design, identity-basis ROBN (τ = 0) and normalized-momentum baselines, and update diagnostics (angles, per-layer norms, and spectra of the actual updates).
  • New Appendix K (Larger-Scale Experiment). SPU and Muon on a ~0.8B-parameter Llama model trained on 21.5B tokens, following the Llama-520M recipe of Wen et al. (2026); SPU reuses Muon's tuned hyperparameters without separate tuning.
  • Corrections. The layer-wise scale of SPU (Appendix G), implementation details (Appendix H), the description of τ in ROBN-τ (Appendix F), a missing equation reference in the convergence proof (Appendix D), notation (Sections 3–4), and a reference entry.

Detailed answers are given in our response to each reviewer.

AuthorsOpenReview ↗

Response to the AC and all reviewers: the claim, why one-sided orthogonality, and the controls

We thank the AC for the comment and summarize our answer here, in one place, for all reviewers.

Our claim. Muon's orthogonality on both sides of the update is not necessary for strong optimization performance: a method with orthogonality on only one side can retain a meaningful portion of Muon's performance (ROBN), and can match or exceed it (SPU).

1. The claim has not changed. The submission asks this question throughout:

  • Introduction: "Is Muon's orthogonality on both sides of the matrix update necessary for strong optimization performance, or can a method that enforces orthogonality on only one side achieve comparable performance?"
  • Abstract: "To probe whether this additional two-sided structure is necessary, we introduce…"
  • Related work: "…whether Muon's two-sided singular-vector orthogonality itself is necessary, which is the geometric question studied in this work."
  • §4.1 and §4.2: ROBN and SPU are each "designed to test" whether one-sided orthogonality can retain (ROBN) or match (SPU) Muon's performance without its two-sided structure.

These sentences remain in the revised version. Some of our phrasing could be read as claiming that one-sided orthogonality explains Muon's effectiveness. The revision therefore states the comparative claim directly; the claim itself is unchanged.

2. Why one-sided orthogonality. SGD, Lion, and Muon arise from different selection rules over orthonormal rank-one systems. Within this framework:

  • 𝒪₁, 𝒪₂: one-/two-sided orthogonal optimizers.
  • 𝒪₂ = {Muon} ⊂ 𝒪₁: any two-sided orthogonal system is a singular-vector decomposition (Appendix A.2).
  • SPU, ROBN ∈ 𝒪₁∖𝒪₂.

𝒪₁ keeps one of Muon's two orthogonality constraints, which makes it the natural class in which to ask whether the other is necessary.

  • #1 ROBN (minimal realization): one-sided orthogonality alone can retain a meaningful portion of Muon's performance. ROBN imposes nothing beyond one-sided orthogonality: its orthogonal side is resampled at random at every step, independently of the momentum. It still retains a meaningful portion of Muon's performance (Table 1).
  • #2 SPU (counterexample): two-sided orthogonality is not necessary. SPU takes its orthogonal side from the momentum through blockwise polar factors but is not orthogonal on the other side, yet matches or exceeds Muon.
    • 𝒢: optimizers that perform comparably to or better than Muon.
    • SPU ∈ 𝒢∖𝒪₂, hence 𝒢 ⊄ 𝒪₂.
Orthogonal sideRoleFinding
ROBNone side, random (independent of the momentum)minimal realization: no constraint beyond one-sided orthogonalityretains a meaningful portion of Muon's performance
Muonboth sides, from the momentumthe unique two-sided memberreference
SPUone side, from the momentum (blockwise polar)counterexample: two-sided orthogonality is neither necessary nor uniquely bestmatches or exceeds Muon

Muon is thus a well-performing member of the one-sided orthogonal family rather than its unique or best realization (§5). Explaining why SPU performs well is not our goal, and the submission nowhere offers or relies on such an explanation: §4.2 states SPU's role, and the Discussion reports its results.

3. Controls for SPU vs. Muon. Since #2 is an existence claim, the controls it needs are those showing that SPU genuinely matches Muon under conditions that favor Muon, not controls that isolate why SPU works.

  • Consistency. Three settings (ImageNet-1K, FineWeb-Edu, OpenWebText), three seeds each, with the same initialization and data order for every optimizer at a given seed. SPU's mean is better than Muon's on all six reported metrics.
  • Tuning favors Muon. SPU was not tuned; it reuses Muon's tuned hyperparameters (Appendix H).
  • Gaps exceed seed variation. FineWeb-Edu loss 2.7823 ± 0.0014 vs. 2.7931 ± 0.0009; OpenWebText 2.9813 ± 0.0033 vs. 2.9890 ± 0.0015; ImageNet-1K accuracy 80.561 ± 0.084 vs. 80.408 ± 0.057.
  • Update norm (Appendix J, 2×2 design, seed 0). SPU's updates on rectangular matrices are √2 larger than Muon's. Norm-matched SPU still matches Muon (FineWeb-Edu −0.0044; OpenWebText +0.0028 and −0.0013 at two learning rates), and SPU is equal to or better than Muon scaled by √2 (−0.0034; −0.0001, −0.0021). The norm accounts for a substantial part of SPU's margin, but a direction effect remains: at matched norm, the SPU direction is ahead on FineWeb-Edu and on OpenWebText at 2×10⁻² (Appendix J.4), so SPU is not a rescaled Muon. Under every control, SPU remains at least comparable to Muon, which is what the claim requires.
  • Scale. On a ~0.8B-parameter Llama trained on 21.5B tokens, SPU remains comparable to or slightly better than Muon (final validation loss 2.9987 vs. 3.0003; single seed; Appendix K).

Detailed answers to each point are in our responses to each reviewer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.