- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the geometric mismatch problem of Muon in convolutional layers. Standard Muon transforms the convolutional kernel into a matrix and orthogonals it through reshaping, but this does not guarantee the orthogonality of the actual convolution operator. Based on this, the authors define the Convolutional Polar Factor (ConvPF) and propose Conv-NS, which approximates convolutional orthogonality through a Newton-Schulz iteration in the form of convolution and support set projection. Experiments show that Conv-NS performs close to standard Muon on CIFAR-10 and ImageNet, while more exact orthogonalization may even degrade training performance.
Critical strengths
-
This paper clearly reveals the geometric difference between kernel reshaping orthogonalization and the actual orthogonalization of the convolution operator, and verifies this phenomenon through singular value experiments.
-
The experiments reveal a phenomenon worthy of further investigation. Smaller orthogonality errors do not necessarily lead to better optimization performance, and this phenomenon persists under different iteration counts and hyperparameter searches. This provides some insights into understanding the actual mechanism of Muon's operation.
Critical weaknesses
- The rationale for using ConvPF as a standard for theoretical correctness is insufficiently demonstrated. ConvPF is optimal only under fixed support and Stiefel constraints, and not necessarily the optimal solution for LMO under the spectral norm constraint of the convolution operator (Equations 15–17). Although the authors have acknowledged the difference between the two in Section 6, using ConvPF as a standard for measuring the theoretical correctness of convolutional Muon lacks sufficient basis and weakens the core argument of the paper regarding the "theoretical error" of standard Muon. Furthermore, I do not believe that such theoretical extrapolation is necessarily reasonable. LMO is only one theoretical perspective for understanding Muon, not a complete explanation of its optimization mechanism. Existing research has pointed out that more exact spectral descent cannot explain the performance advantage of the matrix whitening optimizer alone [1], and the learning dynamics and generalization behavior of Muon can also be explained from other perspectives [2]. Therefore, I think it is reasonable to use ConvPF as a reference target for convolutional orthogonalization, but the authors claim that it represents the theoretically correct update form of convolutional Muon, and thus believe that standard Muon is theoretically wrong. I think this conclusion still lacks sufficient evidence.
[1] Frans et al., What Really Matters in Matrix-Whitening Optimizers?, OPT 2025.
[2] Vasudeva et al., How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Data, ICLR 2026.
-
The core convergence conclusion is not proven. Section 4 claims that Conv-NS converges to ConvPF, but Appendix Theorem C.4 only proves the stationary convergence of the orthogonal error target. This does not guarantee strict orthogonality, nor does it guarantee minimizing the Frobenius distance to the original gradient in the orthogonal solution. In addition, the proof does not fully cover the damping operation in the actual algorithm. Therefore, current theoretical results are insufficient to support the claimed convergence of ConvPF.
-
Experiments have not yet verified the true optimization mechanism. The Pareto front in Figure 4 is obtained by numerical estimation using non-convex optimization, without any guarantee of global optimality. More importantly, the paper mainly compares orthogonal errors and gradient modifications, but lacks direct analysis of the actual convolutional LMO optimality gap and first-order prediction descent. Therefore, it is currently impossible to fully explain why more exact orthogonalization actually harms training.
Minor point: The writing and presentation of results in the paper still need improvement. First, Figure 1 directly uses the undefined abbreviation PF, requiring readers to refer to Section 2.1 to understand its meaning. Furthermore, there are obvious grammatical and spelling errors in the figure captions, such as "distibution" in line 65 and "a an orthogonal matrix" in line 66.
Second, the use of some terms and abbreviations is unclear. For example, what is TTA accuracy (in Table 8)? While Appendix E mentions test-time-augmented validation accuracy, it doesn't explicitly introduce the abbreviation TTA until Appendix I provides an explanation. This confused me when reading about the relevant experiments.
More importantly, the evaluation metrics in Table 1 and Table 8 appear inconsistent. Table 1 labels the approximately 94% accuracy of CIFAR-10 as "Val. Acc.", but Table 8 clearly distinguishes between ordinary Val. Acc. (approximately 93.1%) and TTA Acc. (approximately 94%). So, what kind of accuracy is reported in Table 1? If it's TTA accuracy, why isn't it explicitly stated?
Furthermore, there's a difference in the return values of Algorithm 1 and Algorithm 2; the latter performs an additional division by , and this difference isn't fully explained. Finally, the authors refer to the approximate Pareto frontier obtained from the non-convex numerical search as the ground truth, but these numerical results don't have a strict guarantee of global optimality. I believe this description is inaccurate and easily exaggerates the theoretical significance of the experimental results.
In short, these issues not only affect the readability of the paper, but some also involve the interpretation of experimental results and the reproducibility of the algorithm.
Questions for authors
- Why does reshape-based Muon lie near the numerically estimated Pareto frontier on real training gradients (Figure 4 and Figure 6), but deviate substantially from it on synthetic Gaussian kernels (Figure 8)?
3 comments
The authors’ response does not resolve the main concern in my original review. ConvPF is a reasonable way to extend the nearest semi-orthogonal projection to convolutional layers. My concern is why exact semi-orthogonality should be the standard for judging whether a convolutional Muon update is theoretically correct. For dense layers, the polar factor both gives the nearest semi-orthogonal matrix and solves the spectral-norm LMO. As Section 6 shows, these two criteria can lead to different updates once the kernel support is fixed. The paper shows that reshape-based Muon generally fails to orthogonalize the full convolution operator. Calling it theoretically wrong requires a further argument for why exact semi-orthogonality is the right criterion. The rebuttal does not provide that argument. My overall rating remains unchanged.
As your review said, "LMO is only one theoretical perspective". Convolutions are a specific case where the view of [3] (i.e. "orthogonalize the operator") gives a different solution from the LMO view. This paper aim to illustrate this discrepancy, and that the actual CNNs implementation of muon respect neither of the theories. Conv-NS is a tool to achieve this rather than a solution. We believe that these findings could be of interest to the community and valuable for future research.
Answer to reviewer S9Zj
We thank the reviewer for their attention to this work.
In their original work, the authors of Muon themselves admit not knowing “why it is good to orthogonalize the update” [3]. This led to two distinct lines of work: one motivated by modular duality [4], and one motivated by the linear minimization oracle (LMO).
Our paper investigates Muon’s initial thesis: updating the weights with the orthogonal matrix closest to the initial gradient (Eq. 2). For convolutions, this corresponds to ConvPF (Eq. 6), which is compatible with the dense formulation. ConvPF is thus the theoretical equivalent of PF for convolutions, which is our main claim. This aligns with [4], as ConvPF tightly estimates WRMS→RMS. We will clarify this claim to avoid any misunderstanding.
In Section 6, devoted to investigating why Conv-NS does not improve training, we discuss the relation to the spectral norm associated with the LMO, which is, as you indicate, “only one theoretical perspective for understanding Muon, not a complete explanation of its optimization mechanism.”
Convolutions are a specific case where modular duality and LMO theories are incompatible. We hope that highlighting this discrepancy will open directions for future research that can explain both the dense and convolutional cases.
As stated in line 267, Appendix B proves convergence to a stationary point for Lorth. Absolute convergence cannot be proven without restrictive hypotheses: as discussed in Corollary C.5, the existence of an orthogonal kernel with fixed support is not guaranteed [5].
Knowing this limitation, we conducted the experiment shown in Figure 4, which empirically checks whether Conv-NS converges to ConvPF in practice, validating that cropping and damping do not affect the minimization of Lorth. We used simulated annealing with multiple restarts to mitigate the issue of local minima as much as possible. There were no tractable solutions for ConvPF prior to this work, so having two distinct methods converge to the same value is already valuable.
Without a closed-form solution to the LMO, we measured ⟨G,D⟩F for both reshape-based Muon and Conv-NS on gradients sampled during CIFAR-10 training:
These results align with Table 2: they explain why performance degrades beyond 10 Conv-NS iterations, but do not explain the parity between reshape-based Muon and Conv-NS.
Other remarks
All CIFAR-10 results in the main paper report accuracy with test-time augmentation (TTA), consistent with the CIFAR AirBench94 evaluation protocol. We acknowledge that this was not explicitly stated in the main paper and will clarify it in the experimental setup description in Section 5.
In Appendix Table 8, we additionally report results without TTA as a sanity check, confirming that our observations remain valid without test-time augmentation.
The additional division by α does not fundamentally affect optimization, as it only rescales the update by a constant factor. In particular, dividing the output by α=0.9 is equivalent to multiplying the learning rate by 1/0.9≈1.11.
We thank the reviewer for the additional comments and will revise the manuscript accordingly to address the minor points raised.
We were also surprised by this observation. However, we emphasize that reshape-based Muon does not always lie near the Pareto frontier on real training gradients either (see Figure 8).
While we cannot provide a rigorous explanation at this stage, we hypothesize that this behavior may be related to the learning dynamics and the structure of gradients induced by training. We believe this observation offers a promising direction for future work toward a deeper understanding of Muon’s optimization mechanisms.
References
[3] Jordan. Muon: An optimizer for hidden layers in neural networks. 2024.
[4] Bernstein et al. Modular Duality in Deep Learning. ICML, 2025.
[5] Achour et al. Existence, stability and scalability of orthogonal convolutional neural networks. JMLR, 2022.