- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MALT, a curvature-aware Muon variant that combines lightweight diagonal preconditioning with NS orthogonalization to deal with loss curvature and matrix-gradient anisotropy. MALT (variants) use two-side diagonal preconditioners and step-size scaling. Experiments on GPT-2 show that MALT and MALTER outperform Muon in LLM pretraining.
Critical strengths
-
The method is simple and easy to integrate. MALT uses left and right diagonal preconditioners and NS orthogonalization, adding only two vector states and negligible computational overhead.
-
The paper clearly explains the connections and differences between MALT and existing matrix-based optimizers, allowing readers to clearly identify the contributions of this work.
Critical weaknesses
-
The motivation is not sufficiently clear. The paper aims to address both curvature anisotropy and matrix-gradient anisotropy. While addressing the former is understandable, as it can accelerate optimization and training, what is the purpose of addressing the latter, and how does it benefit training? Do these two types of anisotropy overlap? In other words, is addressing curvature anisotropy alone sufficient, without the need to address matrix-gradient anisotropy?
-
The experimental evidence is insufficient. The paper only evaluates a single dense architecture (GPT 2), and lacks evaluation on other architectures such as MoE. In addition, the model scale is limited, with the largest model under 1B parameters. It remains unclear whether MALT’s advantage can be maintained at larger scales. It is suggested to add experiments on models above 1B parameters and on MoE architectures, and to train on more data (e.g., 1x Chinchilla-optimal token budget) to more systematically evaluate MALT (MALTER).
Questions for authors
-
In the algorithm design, the jump from Eq. (7) to Eq. (8) is somewhat abrupt. How is this step derived?
-
How does MALT or MALTER compare with other row- and column-wise normalization optimizers in experiments, such as NorMuon [1] or Conda [2]?
[1] NorMuon: Making Muon more efficient and scalable, ICML 2026.
[2] Conda: Column-Normalized Adam for Training Large Language Models Faster, arxiv.
2 comments
Response to Reviewer ZTCo point by point (2/2)
## Response to Comment 2: Experimental scale and training budget. ##
We thank the reviewer for this valuable suggestion. We agree that experiments on larger models and additional architectures would further strengthen the empirical evaluation of MALT and MALTER.
Regarding the training token budget, we would like to clarify that our GPT-2 Small (124M) model is trained for 50K steps with 491,520 tokens per step, corresponding to approximately 24.6B processed tokens, or 9.9 times the commonly used Chinchilla-optimal reference budget of 20 tokens per parameter. Both MALT and MALTER consistently outperform Muon throughout this extended training run. For GPT-2 Medium (355M) and Large (774M), the corresponding training budgets are approximately 4.9B and 8.8B processed tokens, respectively. We note that these counts include repeated exposure to the OpenWebText training data.
Regarding larger models and MoE architectures, we acknowledge that our current experiments are limited to dense GPT-2 models with up to 774M parameters. A fair evaluation on larger models and MoE architectures would require substantial computational resources, particularly because each optimizer needs an independent learning-rate sweep under matched training protocols. Given our limited computational resources and the time available during the rebuttal period, we were unable to conduct these additional experiments. We acknowledge this limitation and leave systematic evaluations on larger-scale and MoE models for future work.
Response to Question 1: Derivation from Eq. (7) to Eq. (8).
We agree that the transition from Eq. (7) to Eq. (8) was not sufficiently clear. The key idea is to replace Euclidean steepest descent with spectral-norm steepest descent in the preconditioned space, following the principle underlying Muon. Let denote the gradient in the preconditioned space. Eq. (7) corresponds to Euclidean steepest descent with a Frobenius-norm quadratic regularizer, whose minimizer is . To retain the benefit of Muon's orthogonalization, we instead consider the following spectral-norm constrained problem:
$ \max_{||D||_2\leq1} \langle\widetilde G_t,D\rangle. $
By the duality between the spectral and nuclear norms,
$ \max_{||D||_2\leq1} \langle\widetilde G_t,D\rangle =||\widetilde G_t||_*, $
and the maximum is attained at
$ D^\star =\operatorname{Orth}(\widetilde G_t), $
which gives exactly the positive update direction in Eq. (8). Mapping this direction back to the original parameter space through then yields Eq. (9).
Therefore, Eq. (8) is a deliberate extension of Eq. (7) from Euclidean to spectral-norm steepest descent in the preconditioned space, rather than the direct minimizer of the same Frobenius-regularized objective. This construction preserves Muon's matrix-gradient orthogonalization while incorporating curvature-related preconditioning.
We will clarify this transition in Section 3.1 of the revised manuscript.
Response to Experimental issues (Compared with NorMuon)
We have added a direct comparison with NorMuon [5] in the revised manuscript (page-10, 5.5 COMPARISON WITH NORMUON , Fig. 3). Specifically, we compare NorMuon with Muon, SOAP, Mousse, and MALT on GPT-2 Medium (355M), using a global batch size of 0.983M tokens and 10K training steps under the same WSD schedule.
As shown in the updated large-batch comparison Fig. 3, learning rate tuned NorMuon outperforms Muon and SOAP but achieves a higher final validation loss than Mousse and MALT.
Unlike NorMuon, which applies row-wise adaptive normalization after orthogonalization, MALT incorporates two-sided diagonal preconditioning both before and after orthogonalization. This comparison provides additional empirical evidence of MALT's effectiveness beyond vanilla Muon. We have also included the NorMuon implementation details in the revised appendix.
References
[1] Lau et al., PolarGrad: A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspective, arXiv:2505.21799, 2025.
[2] Davis and Drusvyatskiy, When Do Spectral Gradient Updates Help in Deep Learning?, arXiv:2512.04299, 2025.
[3] Shen et al., On the Convergence Analysis of Muon, TMLR 2026. arXiv:2505.23737.
[4] Paquette et al., Phases of Muon: When Muon Eclipses SignSGD, arXiv:2605.09552, 2026.
[5] Zhang et al., Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioning, arXiv:2603.09697, 2026.
[6] Du and Su, The Newton-Muon Optimizer, arXiv:2604.01472, 2026.
[7] Vyas et al., Improving SOAP Using Iterative Whitening and Muon, Technical Report, 2025
Response to Reviewer ZTCo point by point (1/2)
Thank you for your professional and valuable comments and we will response to your pointed weakness and questions point by point.
Response to Comment 1: Motivation for addressing both types of anisotropy.
We thank the reviewer for this insightful comment. We agree that the distinction between curvature anisotropy and matrix-gradient anisotropy, as well as their complementary roles, deserves further clarification.
Why address matrix-gradient anisotropy? Matrix-gradient anisotropy refers to an uneven singular-value spectrum of the gradient or momentum matrix, which can cause updates to be dominated by a few leading singular directions. Muon's orthogonalization balances these directions and can improve training efficiency. PolarGrad [1] interprets this mechanism as gradient-anisotropy preconditioning. Davis and Drusvyatskiy [2] further establish conditions under which spectral gradient updates achieve greater one-step loss reduction than conventional gradient descent, and validate their predictions in deep neural networks and Transformers. Shen et al. [3] show that, under spectral-norm smoothness, mitigating the matrix-gradient anisotropy in Muon can achieve better convergence complexity than gradient descent when the Hessian exhibits low-rank and approximately block-diagonal structure. More recently, Paquette et al. [4] theoretically demonstrate that Muon-style spectral updates can accelerate convergence in certain regimes with anisotropic data.
Are these two types of anisotropy overlapping? They can be related, but are not completely equivalent. Curvature anisotropy concerns the Hessian eigenvalue spectrum, whereas matrix-gradient anisotropy concerns the singular-value spectrum of the current gradient or momentum matrix. Even when the Hessian is perfectly conditioned, the gradient matrix may remain highly anisotropic. Therefore, addressing curvature anisotropy alone does not necessarily eliminate matrix-gradient anisotropy. However, a precise theoretical characterization of the relationship between these two types of anisotropy remains an open question.
Why address both simultaneously? The two mechanisms can be complementary in practice demonstrates the essential difference between these two anisotropies. For example, Mousse [5] combines Shampoo-based curvature preconditioning with Muon's orthogonalization and achieves better validation loss than either Muon or SOAP in its reported 800M-model experiment. Similarly, Newton–Muon [6] combines input second-moment preconditioning with matrix-gradient orthogonalization, reducing the training steps required to reach a target loss relative to Muon. The original SOAP authors also explored combining SOAP and Muon in their technical report [7], reporting encouraging improvements over the individual optimizers in preliminary large-batch experiments.
Our results provide further empirical support for this complementary design: Table 3 shows that MALT achieves lower validation loss than both Muon and SOAP under the tested GPT-2 settings. These findings motivate MALT to combine lightweight curvature-related preconditioning with matrix-gradient orthogonalization, rather than relying on either mechanism alone.
Motivation of our proposed method. Figure 1 provides further motivation for combining these two mechanisms. As shown in Fig. 1(a), although Muon alleviates matrix-gradient anisotropy through orthogonalization, its convergence becomes substantially slower as the Hessian condition number increases. In contrast, Fig. 1(b) shows that incorporating exact Hessian whitening into Muon significantly reduces this sensitivity, highlighting the potential benefit of curvature preconditioning beyond orthogonalization alone. Since exact Hessian preconditioning is computationally prohibitive for large-scale training, we propose MALT, which incorporates lightweight two-sided diagonal preconditioners constructed from gradient second moments while retaining Muon's orthogonalization. Importantly, the new Fig. 1(c) shows that even this inexpensive diagonal preconditioning can alleviate Muon's sensitivity to curvature anisotropy in the illustrative quadratic setting, although it does not achieve condition-number-independent convergence. These observations motivate our design of MALT to combine curvature-related adaptation and matrix-gradient orthogonalization at nearly the same memory and computational cost as Muon.