acceptodds
Under review as a conference paper at ICLR 2027

Muon is theoretically wrong for convolutions, but empirically effective.

Abstract

Muon, an optimizer known for its efficiency, has a clear interpretation for matrix-valued updates, but convolutional kernels are stored as four-dimensional tensors. Standard implementations reshape these tensors into matrices, a shortcut which breaks the theoretical understanding behind Muon. To investigate this, we formalize the corresponding optimization objective directly in convolutional operator geometry and introduce Convolutional Newton-Schulz (Conv-NS), which approximates the polar factor in this geometry while preserving kernel support. When applied in fast training experiments, Conv-NS and reshape-based Muon are both computationally efficient and achieve comparable accuracy on CIFAR-10 and ImageNet classification tasks. However, as one could expect a theoretically aligned Conv-NS to outperform reshape-based Muon, we investigate this mismatch between practice and theoretical understanding, with the hypothesis that exact convolutional orthogonalization may overconstrain updates. These findings highlight Muon's strong practical performance while opening directions for its further development on convolutions.

Primary area:
optimization
Keywords:
Muon, Orthogonal convolution, optimization, LMO

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_S9ZjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the geometric mismatch problem of Muon in convolutional layers. Standard Muon transforms the convolutional kernel into a matrix and orthogonals it through reshaping, but this does not guarantee the orthogonality of the actual convolution operator. Based on this, the authors define the Convolutional Polar Factor (ConvPF) and propose Conv-NS, which approximates convolutional orthogonality through a Newton-Schulz iteration in the form of convolution and support set projection. Experiments show that Conv-NS performs close to standard Muon on CIFAR-10 and ImageNet, while more exact orthogonalization may even degrade training performance.

Critical strengths
  1. This paper clearly reveals the geometric difference between kernel reshaping orthogonalization and the actual orthogonalization of the convolution operator, and verifies this phenomenon through singular value experiments.

  2. The experiments reveal a phenomenon worthy of further investigation. Smaller orthogonality errors do not necessarily lead to better optimization performance, and this phenomenon persists under different iteration counts and hyperparameter searches. This provides some insights into understanding the actual mechanism of Muon's operation.

Critical weaknesses
  1. The rationale for using ConvPF as a standard for theoretical correctness is insufficiently demonstrated. ConvPF is optimal only under fixed support and Stiefel constraints, and not necessarily the optimal solution for LMO under the spectral norm constraint of the convolution operator (Equations 15–17). Although the authors have acknowledged the difference between the two in Section 6, using ConvPF as a standard for measuring the theoretical correctness of convolutional Muon lacks sufficient basis and weakens the core argument of the paper regarding the "theoretical error" of standard Muon. Furthermore, I do not believe that such theoretical extrapolation is necessarily reasonable. LMO is only one theoretical perspective for understanding Muon, not a complete explanation of its optimization mechanism. Existing research has pointed out that more exact spectral descent cannot explain the performance advantage of the matrix whitening optimizer alone [1], and the learning dynamics and generalization behavior of Muon can also be explained from other perspectives [2]. Therefore, I think it is reasonable to use ConvPF as a reference target for convolutional orthogonalization, but the authors claim that it represents the theoretically correct update form of convolutional Muon, and thus believe that standard Muon is theoretically wrong. I think this conclusion still lacks sufficient evidence.

[1] Frans et al., What Really Matters in Matrix-Whitening Optimizers?, OPT 2025.

[2] Vasudeva et al., How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Data, ICLR 2026.

  1. The core convergence conclusion is not proven. Section 4 claims that Conv-NS converges to ConvPF, but Appendix Theorem C.4 only proves the stationary convergence of the orthogonal error target. This does not guarantee strict orthogonality, nor does it guarantee minimizing the Frobenius distance to the original gradient in the orthogonal solution. In addition, the proof does not fully cover the damping operation in the actual algorithm. Therefore, current theoretical results are insufficient to support the claimed convergence of ConvPF.

  2. Experiments have not yet verified the true optimization mechanism. The Pareto front in Figure 4 is obtained by numerical estimation using non-convex optimization, without any guarantee of global optimality. More importantly, the paper mainly compares orthogonal errors and gradient modifications, but lacks direct analysis of the actual convolutional LMO optimality gap and first-order prediction descent. Therefore, it is currently impossible to fully explain why more exact orthogonalization actually harms training.

Minor point: The writing and presentation of results in the paper still need improvement. First, Figure 1 directly uses the undefined abbreviation PF, requiring readers to refer to Section 2.1 to understand its meaning. Furthermore, there are obvious grammatical and spelling errors in the figure captions, such as "distibution" in line 65 and "a an orthogonal matrix" in line 66.

Second, the use of some terms and abbreviations is unclear. For example, what is TTA accuracy (in Table 8)? While Appendix E mentions test-time-augmented validation accuracy, it doesn't explicitly introduce the abbreviation TTA until Appendix I provides an explanation. This confused me when reading about the relevant experiments.

More importantly, the evaluation metrics in Table 1 and Table 8 appear inconsistent. Table 1 labels the approximately 94% accuracy of CIFAR-10 as "Val. Acc.", but Table 8 clearly distinguishes between ordinary Val. Acc. (approximately 93.1%) and TTA Acc. (approximately 94%). So, what kind of accuracy is reported in Table 1? If it's TTA accuracy, why isn't it explicitly stated?

Furthermore, there's a difference in the return values ​​of Algorithm 1 and Algorithm 2; the latter performs an additional division by α\alpha, and this difference isn't fully explained. Finally, the authors refer to the approximate Pareto frontier obtained from the non-convex numerical search as the ground truth, but these numerical results don't have a strict guarantee of global optimality. I believe this description is inaccurate and easily exaggerates the theoretical significance of the experimental results.

In short, these issues not only affect the readability of the paper, but some also involve the interpretation of experimental results and the reproducibility of the algorithm.

Questions for authors
  1. Why does reshape-based Muon lie near the numerically estimated Pareto frontier on real training gradients (Figure 4 and Figure 6), but deviate substantially from it on synthetic Gaussian kernels (Figure 8)?
3 comments
AuthorsOpenReview ↗

Answer to reviewer S9Zj

We thank the reviewer for their attention to this work.

W1: “The rationale for using ConvPF as a standard for theoretical correctness is insufficiently demonstrated.”

In their original work, the authors of Muon themselves admit not knowing “why it is good to orthogonalize the update” [3]. This led to two distinct lines of work: one motivated by modular duality [4], and one motivated by the linear minimization oracle (LMO).

Our paper investigates Muon’s initial thesis: updating the weights with the orthogonal matrix closest to the initial gradient (Eq. 2). For convolutions, this corresponds to ConvPF (Eq. 6), which is compatible with the dense formulation. ConvPF is thus the theoretical equivalent of PF for convolutions, which is our main claim. This aligns with [4], as ConvPF tightly estimates WRMS→RMSW_{\mathrm{RMS}\rightarrow\mathrm{RMS}}. We will clarify this claim to avoid any misunderstanding.

In Section 6, devoted to investigating why Conv-NS does not improve training, we discuss the relation to the spectral norm associated with the LMO, which is, as you indicate, “only one theoretical perspective for understanding Muon, not a complete explanation of its optimization mechanism.”

Convolutions are a specific case where modular duality and LMO theories are incompatible. We hope that highlighting this discrepancy will open directions for future research that can explain both the dense and convolutional cases.

W2 and W3: “The core convergence to ConvPF conclusion is not proven” and “Experiments have not yet verified the true optimization mechanism.”

As stated in line 267, Appendix B proves convergence to a stationary point for LorthL_{orth}. Absolute convergence cannot be proven without restrictive hypotheses: as discussed in Corollary C.5, the existence of an orthogonal kernel with fixed support is not guaranteed [5].

Knowing this limitation, we conducted the experiment shown in Figure 4, which empirically checks whether Conv-NS converges to ConvPF in practice, validating that cropping and damping do not affect the minimization of LorthL_{\mathrm{orth}}. We used simulated annealing with multiple restarts to mitigate the issue of local minima as much as possible. There were no tractable solutions for ConvPF prior to this work, so having two distinct methods converge to the same value is already valuable.

W3b: “Lacks direct analysis of the actual convolutional LMO optimality gap.”

Without a closed-form solution to the LMO, we measured ⟨G,D⟩F\langle G,D\rangle_F for both reshape-based Muon and Conv-NS on gradients sampled during CIFAR-10 training:

DirectionMean ⟨G,D⟩F\langle G,D\rangle_F
Reshape polar (exact SVD)345.41
Conv-NS (2 steps)161.56
Conv-NS (5 steps)327.85
Conv-NS (10 steps)492.74
Conv-NS (25 steps)481.83
Conv-NS (100 steps)450.09

These results align with Table 2: they explain why performance degrades beyond 10 Conv-NS iterations, but do not explain the parity between reshape-based Muon and Conv-NS.

Other remarks

“Table 1 and Table 8 appear inconsistent. TTA val accuracy.”

All CIFAR-10 results in the main paper report accuracy with test-time augmentation (TTA), consistent with the CIFAR AirBench94 evaluation protocol. We acknowledge that this was not explicitly stated in the main paper and will clarify it in the experimental setup description in Section 5.

In Appendix Table 8, we additionally report results without TTA as a sanity check, confirming that our observations remain valid without test-time augmentation.

“Difference in the return values of Algorithm 1 and Algorithm 2.”

The additional division by α\alpha does not fundamentally affect optimization, as it only rescales the update by a constant factor. In particular, dividing the output by α=0.9\alpha=0.9 is equivalent to multiplying the learning rate by 1/0.9≈1.111/0.9\approx 1.11.

We thank the reviewer for the additional comments and will revise the manuscript accordingly to address the minor points raised.

Q1: “Why does reshape-based Muon lie near the numerically estimated Pareto frontier on real training gradients but deviate substantially from it on synthetic Gaussian kernels?”

We were also surprised by this observation. However, we emphasize that reshape-based Muon does not always lie near the Pareto frontier on real training gradients either (see Figure 8).

While we cannot provide a rigorous explanation at this stage, we hypothesize that this behavior may be related to the learning dynamics and the structure of gradients induced by training. We believe this observation offers a promising direction for future work toward a deeper understanding of Muon’s optimization mechanisms.

References

[3] Jordan. Muon: An optimizer for hidden layers in neural networks. 2024.

[4] Bernstein et al. Modular Duality in Deep Learning. ICML, 2025.

[5] Achour et al. Existence, stability and scalability of orthogonal convolutional neural networks. JMLR, 2022.

The authors’ response does not resolve the main concern in my original review. ConvPF is a reasonable way to extend the nearest semi-orthogonal projection to convolutional layers. My concern is why exact semi-orthogonality should be the standard for judging whether a convolutional Muon update is theoretically correct. For dense layers, the polar factor both gives the nearest semi-orthogonal matrix and solves the spectral-norm LMO. As Section 6 shows, these two criteria can lead to different updates once the kernel support is fixed. The paper shows that reshape-based Muon generally fails to orthogonalize the full convolution operator. Calling it theoretically wrong requires a further argument for why exact semi-orthogonality is the right criterion. The rebuttal does not provide that argument. My overall rating remains unchanged.

As your review said, "LMO is only one theoretical perspective". Convolutions are a specific case where the view of [3] (i.e. "orthogonalize the operator") gives a different solution from the LMO view. This paper aim to illustrate this discrepancy, and that the actual CNNs implementation of muon respect neither of the theories. Conv-NS is a tool to achieve this rather than a solution. We believe that these findings could be of interest to the community and valuable for future research.

Reviewer_3rVYOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates Muon for convolutional layers in neural networks. The authors find that standard implementations of Muon do not orthogonalize the convolution operator, making recent arguments for Muon's effectiveness due to its gradient orthogonalization less compelling in this setting. The authors furthermore show that true (approximate) orthogonalization of convolution operators is computationally feasible but achieves similar performance to Muon in practice. They investigate this discrepancy, positing that orthogonalizing convolution gradients may actually be detrimental for training.

Critical strengths
  • The paper shows that Muon does not orthogonalize convolution operators, which is supported by experimental evidence. Furthermore, they give a practical algorithm for approximate orthogonalization of convolution operators, which is also supported by the experiments (e.g. Figure 3).
  • The authors observe that despite being theoretically unprincipled, Muon achieves similar performance to Conv-NS, giving an extended discussion in Section 6.
Critical weaknesses
  • I don't find the explanation for why Conv-NS performs similarly or worse than Muon convincing. Why does better orthogonalization hinder early optimization? Why are some matrices outside ConvPF's feasible set but in the convolutional LMO's feasible set "better" for training? It would be good to incorporate additional insights from e.g. [1, 2, 3] with supporting experiments.
  • Overall, the potential impact of the work may be limited due to the negative empirical conclusion. However, I do not consider this to be a critical weakness.
  • (minor typos)
    • Line 112: should be "orthogonalizing" instead of "orthogonalize"
    • Line 161: footnote 2 should have "are" instead of "is"
    • βt\beta_t instead of β\beta in (10)

References

[1] Lizhang Chen, Jonathan Li, Qiang Liu. Muon Optimizes Under Spectral Norm Constraints. TMLR 2026.

[2] Weijie Su. Isotropic Curvature Model for Understanding Deep Learning Optimization: Is Gradient Orthogonalization Optimal? abs/2511.00674.

[3] Damek Davis, Dmitriy Drusvyatskiy. When do spectral gradient updates help in deep learning? abs/2512.04299.

Questions for authors
  1. Address the main weakness above.
  2. What is β\beta in (3)? Define it.
  3. How does Conv-NS only incur marginal overhead on the vision experiments if it can be seen to take significantly more time than RKO in Figure 3?
  4. Is there a more principled way of choosing λ\lambda in (13)?
2 comments
AuthorsOpenReview ↗

Answer to reviewer 3rVY

We thank reviewer 3rVY for their positive comments on the paper.

W1 and Q1: “I don't find the explanation for why Conv-NS performs similarly or worse than Muon convincing.”

Our paper questions the initial hypothesis underlying Muon: that the optimal update is the orthogonal matrix closest to the gradient (Eq. 2). For convolutions, this corresponds to ConvPF (Eq. 6), which Conv-NS approximates, just as Newton–Schulz in Muon approximates the polar factor.

Section 6 discusses three possible explanations for why Conv-NS fails to improve training:

  1. Insufficient orthogonality: Increasing the number of Conv-NS iterations improves orthogonality but degrades training performance.
  2. A better objective: We investigate whether minimizing LflatL_{\mathrm{flat}}, as in Muon-Reshape, explains the difference. However, with enough iterations, Conv-NS achieves a lower LflatL_{\mathrm{flat}} than Muon-Reshape with five iterations.
  3. Overly restrictive orthogonality: The LMO perspective minimizes ⟨G,Δ⟩\langle G,\Delta\rangle under an operator-norm constraint. While this coincides with the polar factor for dense matrices, we show that Muon-Reshape does not satisfy this constraint, whereas ConvPF imposes stronger constraints than the convolutional LMO.

We thank the reviewer for suggesting references [1, 2, 3], which we will add to the paper. However, none of these works directly addresses the discrepancy observed for convolutions: they assume the reshaping operation for convolutions and focus on the Frobenius inner product between the gradient and the update, following the LMO perspective.

We believe this discussion provides essential insights for future work toward a unified theoretical understanding of Muon’s effectiveness for both dense and convolutional layers. In particular, these observations suggest that improved orthogonality alone does not explain better optimization performance.

Q2: “What is β\beta in (3)? Define it.”

β\beta is the Björck coefficient, which is usually set to 0.5. We will define it alongside Eq. 3.

Q3: “How does Conv-NS only incur marginal overhead on the vision experiments if it can be seen to take significantly more time than RKO in Figure 3?”

We acknowledge that this discrepancy may seem surprising, but several factors explain it:

  1. Different evaluation settings: The timings in Figure 3 were measured on CPU, whereas the end-to-end training experiments were performed on GPU. These timings should not have been included in the figure.
  2. Relative computational cost: In end-to-end training, computing the Muon updates, including Conv-NS, accounts for only a small fraction of the total training time, which is dominated by forward and backward passes.
  3. Different scaling behavior: RKO involves three matrix multiplications on matrices of shape (co,cik2)(c_o,c_i k^2), whereas Conv-NS performs two convolutions on tensors of shape (co,ci,k,k)(c_o,c_i,k,k). As architectures grow larger, Conv-NS appears to become relatively more efficient. However, substantial runtime variability in our infrastructure prevents us from drawing a definitive conclusion about this scaling behavior.

As the CPU timings can be misleading, we will remove them from Figure 3.

Q4: Choice of λ\lambda for Figure 4.

λ\lambda is not a hyperparameter of Conv-NS and is therefore not selected when computing its updates. It is used only to construct the Pareto frontier in Figure 4 and illustrate the positions of Conv-NS and Muon-Reshape relative to this frontier. To this end, we perform a greedy exploration of λ\lambda values in [0,1][0,1].

I thank the authors for their response. I have raised my rating to 3.

Reviewer_rJi6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies proper implementation of Muon on convolution operators. The current literature uses reshaped based Muon while this paper proposes whole-image transform based Muon and its efficient implementation Conv-NS. Experiments show that Conv-NS is not necessarily better than reshaped based Muon in image classification tasks.

Critical strengths
  1. The motivation of this paper is solid in that it measures the convolution kernel from the whole-image perspective instead of a per-patch perspective. This is inherently more sound.
  2. It provides numerical experiments to compare the effectiveness of reshaped Muon and Conv-NS.
Critical weaknesses
  1. I do not think Conv-NS is a strong approximation to whole-image based Muon for convolution kernel especially when cropping is applied. The provided theorems do not directly support this.
  2. There should be an expensive whole-image based Muon reference to prove that the exact version is better, then we do fast approximation.
  3. SGD is a weak baseline, maybe AdamW/Lion can be a stronger baseline.
Questions for authors

n.a.

3 comments
AuthorsOpenReview ↗

Answer to reviewer rji6

We thank the reviewer for their interest in our work. We hope our answers will address the remaining concerns.

W1–2: “Conv-NS is not a strong approximation to whole-image based Muon for convolution kernel” and “There should be an expensive whole-image based Muon reference.”

The whole-image polar factor is an interesting object to study. However, as shown in Figure 1, it would not yield a feasible weight update because of the finite kernel support: we require ΔW∈Rco×ci×k×k\Delta W \in \mathbb{R}^{c_o \times c_i \times k \times k}. This explains the difference between Eq. 2 and Eq. 6, and why we cannot directly compare against the whole-image polar factor as requested.

W3: “SGD is a weak baseline; maybe AdamW/Lion can be a stronger baseline.”

We include SGD in Table 1 to illustrate that Muon achieves competitive performance with both reshape-based orthogonalization and Conv-NS. This paper does not aim to introduce a novel optimizer, but to explore the limits of the theory behind Muon.

We agree that a comparison with AdamW is valuable. We therefore conducted a hyperparameter search over learning rate, weight decay, β1\beta_1, and β2\beta_2, using the same settings as Muon and SGD. We obtained the following results on CIFAR-10:

OptimizerSettingVal. accuracy (%)Train accuracy (%)Runtime (s)
Muon – reshape1 epoch87.18 ± 0.00286.46 ± 0.010.509 ± 0.003
Muon – Conv-NS1 epoch87.75 ± 0.00285.44 ± 0.020.559 ± 0.003
SGD1 epoch84.99 ± 0.00283.52 ± 0.020.496 ± 0.004
AdamW1 epoch85.06 ± 0.002683.96 ± 0.0220.563 ± 0.0003
Muon – reshape8 epochs94.02 ± 0.1497.66 ± 0.313.074 ± 0.005
Muon – Conv-NS8 epochs93.98 ± 0.1497.17 ± 0.383.608 ± 0.018
SGD8 epochs91.25 ± 0.0993.00 ± 0.352.975 ± 0.009
AdamW8 epochs91.35 ± 0.000794.92 ± 0.00303.127 ± 0.003

While AdamW performs slightly better than SGD, a performance gap remains relative to both Muon variants.

i thank the authors for the replies. 1) i got the clarification on the infeasibility of the whole-image polar factor when transferring the after-polar results back to the kernel's constraint make, but maybe doing a projection helps. i do not want to bring new question here, but i think then maybe a fundamental new design is needed to address this problem for example a constrained update, and the infeasibility of the whole-image approach does not establish that fixed-support orthogonalization (in this paper) is the appropriate solution. therefore, i am kind of concerned with the central claim of this work on muon is theoretically wrong. 2) i thank the authors for the additional experiments on adamw and multiple epoch, they are helpful. i would encourage to improve the design of Muon – Conv-NS so it can consistently outperforms muon-reshape, that will give the community a new efficient optimizer to replace muon when muon is not directly inapplicable in the cnn case. i would like to maintain my current score 2.

We thank the reviewer for his interest in this discussion. We understand that the negative results published in this work may be frustrating, as we show the issue in muon's actual theory without proposing a new solution. Muon - convNS aims to show the limits of the operator polar factorization rather than claiming a replacement for convolutions. We believe that these findings could be of interest to the community and valuable for future research.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.