acceptodds
Under review as a conference paper at ICLR 2027

MALT: CURVATURE-AIDED MUON VIA LIGHTWEIGHT DIAGONAL PRECONDITIONING

Abstract

Muon has recently emerged as a promising alternative to AdamW for pretraining language models. By orthogonalizing momentum matrices using Newton–Schulz iterations, Muon mitigates the influence of gradient anisotropy by distributing the update more evenly across all singular directions, and enjoys an improved learning efficiency empirically. However, this operation does not explicitly account for the curvature of the loss landscape, and Muon may therefore still sensitive to curvature anisotropy. We bridge this gap by proposing MALT(Muon Augmented by Lightweight Two-sided preconditioning), which uses lightweight diagonal preconditioners to reduce the influence of curvature anisotropy to Muon. Specifically, MALT constructs two-sided diagonal preconditioners with low memory and computational overhead to capture a diagonal approximation of the curvature of the loss landscape. It then orthogonalizes the diagonally preconditioned momentum using Newton–Schulz iterations and maps the result back to define the update direction, while norm grafting controls the update magnitude. Convergence guarantees are provided for MALT in the stochastic non-convex setting. To improve its robustness to stochastic gradient noise, we further proposed MALTER (MALT with noise Adaptive stEpsize Rescaling). Experiments on GPT-2 Small, Medium, and Large pretraining show that the proposed methods outperform Muon while maintaining nearly the same memory footprint and wall-clock time.

Primary area:
optimization
Keywords:
Optimization, LLM, Muon

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ZTCoOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MALT, a curvature-aware Muon variant that combines lightweight diagonal preconditioning with NS orthogonalization to deal with loss curvature and matrix-gradient anisotropy. MALT (variants) use two-side diagonal preconditioners and step-size scaling. Experiments on GPT-2 show that MALT and MALTER outperform Muon in LLM pretraining.

Critical strengths
  1. The method is simple and easy to integrate. MALT uses left and right diagonal preconditioners and NS orthogonalization, adding only two vector states and negligible computational overhead.

  2. The paper clearly explains the connections and differences between MALT and existing matrix-based optimizers, allowing readers to clearly identify the contributions of this work.

Critical weaknesses
  1. The motivation is not sufficiently clear. The paper aims to address both curvature anisotropy and matrix-gradient anisotropy. While addressing the former is understandable, as it can accelerate optimization and training, what is the purpose of addressing the latter, and how does it benefit training? Do these two types of anisotropy overlap? In other words, is addressing curvature anisotropy alone sufficient, without the need to address matrix-gradient anisotropy?

  2. The experimental evidence is insufficient. The paper only evaluates a single dense architecture (GPT 2), and lacks evaluation on other architectures such as MoE. In addition, the model scale is limited, with the largest model under 1B parameters. It remains unclear whether MALT’s advantage can be maintained at larger scales. It is suggested to add experiments on models above 1B parameters and on MoE architectures, and to train on more data (e.g., 1x Chinchilla-optimal token budget) to more systematically evaluate MALT (MALTER).

Questions for authors
  1. In the algorithm design, the jump from Eq. (7) to Eq. (8) is somewhat abrupt. How is this step derived?

  2. How does MALT or MALTER compare with other row- and column-wise normalization optimizers in experiments, such as NorMuon [1] or Conda [2]?

[1] NorMuon: Making Muon more efficient and scalable, ICML 2026.

[2] Conda: Column-Normalized Adam for Training Large Language Models Faster, arxiv.

2 comments
AuthorsOpenReview ↗

Response to Reviewer ZTCo point by point (1/2)

Thank you for your professional and valuable comments and we will response to your pointed weakness and questions point by point.

Response to Comment 1: Motivation for addressing both types of anisotropy.

We thank the reviewer for this insightful comment. We agree that the distinction between curvature anisotropy and matrix-gradient anisotropy, as well as their complementary roles, deserves further clarification.

Why address matrix-gradient anisotropy? Matrix-gradient anisotropy refers to an uneven singular-value spectrum of the gradient or momentum matrix, which can cause updates to be dominated by a few leading singular directions. Muon's orthogonalization balances these directions and can improve training efficiency. PolarGrad [1] interprets this mechanism as gradient-anisotropy preconditioning. Davis and Drusvyatskiy [2] further establish conditions under which spectral gradient updates achieve greater one-step loss reduction than conventional gradient descent, and validate their predictions in deep neural networks and Transformers. Shen et al. [3] show that, under spectral-norm smoothness, mitigating the matrix-gradient anisotropy in Muon can achieve better convergence complexity than gradient descent when the Hessian exhibits low-rank and approximately block-diagonal structure. More recently, Paquette et al. [4] theoretically demonstrate that Muon-style spectral updates can accelerate convergence in certain regimes with anisotropic data.

Are these two types of anisotropy overlapping? They can be related, but are not completely equivalent. Curvature anisotropy concerns the Hessian eigenvalue spectrum, whereas matrix-gradient anisotropy concerns the singular-value spectrum of the current gradient or momentum matrix. Even when the Hessian is perfectly conditioned, the gradient matrix may remain highly anisotropic. Therefore, addressing curvature anisotropy alone does not necessarily eliminate matrix-gradient anisotropy. However, a precise theoretical characterization of the relationship between these two types of anisotropy remains an open question.

Why address both simultaneously? The two mechanisms can be complementary in practice demonstrates the essential difference between these two anisotropies. For example, Mousse [5] combines Shampoo-based curvature preconditioning with Muon's orthogonalization and achieves better validation loss than either Muon or SOAP in its reported 800M-model experiment. Similarly, Newton–Muon [6] combines input second-moment preconditioning with matrix-gradient orthogonalization, reducing the training steps required to reach a target loss relative to Muon. The original SOAP authors also explored combining SOAP and Muon in their technical report [7], reporting encouraging improvements over the individual optimizers in preliminary large-batch experiments.

Our results provide further empirical support for this complementary design: Table 3 shows that MALT achieves lower validation loss than both Muon and SOAP under the tested GPT-2 settings. These findings motivate MALT to combine lightweight curvature-related preconditioning with matrix-gradient orthogonalization, rather than relying on either mechanism alone.

Motivation of our proposed method. Figure 1 provides further motivation for combining these two mechanisms. As shown in Fig. 1(a), although Muon alleviates matrix-gradient anisotropy through orthogonalization, its convergence becomes substantially slower as the Hessian condition number increases. In contrast, Fig. 1(b) shows that incorporating exact Hessian whitening into Muon significantly reduces this sensitivity, highlighting the potential benefit of curvature preconditioning beyond orthogonalization alone. Since exact Hessian preconditioning is computationally prohibitive for large-scale training, we propose MALT, which incorporates lightweight two-sided diagonal preconditioners constructed from gradient second moments while retaining Muon's orthogonalization. Importantly, the new Fig. 1(c) shows that even this inexpensive diagonal preconditioning can alleviate Muon's sensitivity to curvature anisotropy in the illustrative quadratic setting, although it does not achieve condition-number-independent convergence. These observations motivate our design of MALT to combine curvature-related adaptation and matrix-gradient orthogonalization at nearly the same memory and computational cost as Muon.

AuthorsOpenReview ↗

Response to Reviewer ZTCo point by point (2/2)

## Response to Comment 2: Experimental scale and training budget. ##

We thank the reviewer for this valuable suggestion. We agree that experiments on larger models and additional architectures would further strengthen the empirical evaluation of MALT and MALTER.

Regarding the training token budget, we would like to clarify that our GPT-2 Small (124M) model is trained for 50K steps with 491,520 tokens per step, corresponding to approximately 24.6B processed tokens, or 9.9 times the commonly used Chinchilla-optimal reference budget of 20 tokens per parameter. Both MALT and MALTER consistently outperform Muon throughout this extended training run. For GPT-2 Medium (355M) and Large (774M), the corresponding training budgets are approximately 4.9B and 8.8B processed tokens, respectively. We note that these counts include repeated exposure to the OpenWebText training data.

Regarding larger models and MoE architectures, we acknowledge that our current experiments are limited to dense GPT-2 models with up to 774M parameters. A fair evaluation on larger models and MoE architectures would require substantial computational resources, particularly because each optimizer needs an independent learning-rate sweep under matched training protocols. Given our limited computational resources and the time available during the rebuttal period, we were unable to conduct these additional experiments. We acknowledge this limitation and leave systematic evaluations on larger-scale and MoE models for future work.

Response to Question 1: Derivation from Eq. (7) to Eq. (8).

We agree that the transition from Eq. (7) to Eq. (8) was not sufficiently clear. The key idea is to replace Euclidean steepest descent with spectral-norm steepest descent in the preconditioned space, following the principle underlying Muon. Let G~t=Bt−1/2(∇f(Xt))\widetilde G_t=\mathcal B_t^{-1/2}(\nabla f(X_t)) denote the gradient in the preconditioned space. Eq. (7) corresponds to Euclidean steepest descent with a Frobenius-norm quadratic regularizer, whose minimizer is −G~t-\widetilde G_t. To retain the benefit of Muon's orthogonalization, we instead consider the following spectral-norm constrained problem:

$ \max_{||D||_2\leq1} \langle\widetilde G_t,D\rangle. $

By the duality between the spectral and nuclear norms,

$ \max_{||D||_2\leq1} \langle\widetilde G_t,D\rangle =||\widetilde G_t||_*, $

and the maximum is attained at

$ D^\star =\operatorname{Orth}(\widetilde G_t), $

which gives exactly the positive update direction in Eq. (8). Mapping this direction back to the original parameter space through Bt−1/2\mathcal B_t^{-1/2} then yields Eq. (9).

Therefore, Eq. (8) is a deliberate extension of Eq. (7) from Euclidean to spectral-norm steepest descent in the preconditioned space, rather than the direct minimizer of the same Frobenius-regularized objective. This construction preserves Muon's matrix-gradient orthogonalization while incorporating curvature-related preconditioning.

We will clarify this transition in Section 3.1 of the revised manuscript.

Response to Experimental issues (Compared with NorMuon)

We have added a direct comparison with NorMuon [5] in the revised manuscript (page-10, 5.5 COMPARISON WITH NORMUON , Fig. 3). Specifically, we compare NorMuon with Muon, SOAP, Mousse, and MALT on GPT-2 Medium (355M), using a global batch size of 0.983M tokens and 10K training steps under the same WSD schedule.

As shown in the updated large-batch comparison Fig. 3, learning rate tuned NorMuon outperforms Muon and SOAP but achieves a higher final validation loss than Mousse and MALT.

Unlike NorMuon, which applies row-wise adaptive normalization after orthogonalization, MALT incorporates two-sided diagonal preconditioning both before and after orthogonalization. This comparison provides additional empirical evidence of MALT's effectiveness beyond vanilla Muon. We have also included the NorMuon implementation details in the revised appendix.

References

[1] Lau et al., PolarGrad: A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspective, arXiv:2505.21799, 2025.

[2] Davis and Drusvyatskiy, When Do Spectral Gradient Updates Help in Deep Learning?, arXiv:2512.04299, 2025.

[3] Shen et al., On the Convergence Analysis of Muon, TMLR 2026. arXiv:2505.23737.

[4] Paquette et al., Phases of Muon: When Muon Eclipses SignSGD, arXiv:2605.09552, 2026.

[5] Zhang et al., Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioning, arXiv:2603.09697, 2026.

[6] Du and Su, The Newton-Muon Optimizer, arXiv:2604.01472, 2026.

[7] Vyas et al., Improving SOAP Using Iterative Whitening and Muon, Technical Report, 2025

Reviewer_GadJOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper is motivated by Muon's potential sensitivity to curvature anisotropy of the loss. The authors distinguish two kinds of anisotropy. One is an unbalanced singular-value spectrum of the gradient (momentum) matrix, which lets a few leading directions dominate the update. The other is curvature anisotropy, an uneven spread of Hessian eigenvalues, which makes the convergence of first-order methods depend on the condition number. Muon's orthogonalization addresses the first but does not explicitly account for the second, so Muon may be sensitive to ill-conditioned curvature. The authors illustrate this on a matrix least-squares problem: under a WSD schedule, Muon slows down markedly as the Hessian condition number grows, whereas Muon with exact Hessian-whitening preconditioning is largely unaffected.

The core idea is to run Muon in a preconditioned space so that curvature and gradient anisotropy are handled together. Interpreting preconditioning as steepest descent in a transformed space, the authors propose MALT, which maps the momentum into the preconditioned space, orthogonalizes it with Newton--Schulz, and maps it back, i.e., Lt Orth(LtMtRt) RtL_t\,\mathrm{Orth}(L_tM_tR_t)\,R_t. Here LtL_t and RtR_t are lightweight diagonal preconditioners built from EMAs of row-wise and column-wise squared gradient norms raised to the −1/8-1/8 power, requiring only O(m+n)O(m+n) extra state per matrix, and norm grafting controls the update magnitude. MALTER further adds a NAMO-style scalar noise-adaptive stepsize. The authors provide a convergence guarantee for MALT under lower-boundedness, LL-smoothness, and bounded-variance assumptions, and show in GPT-2 (124M--774M) pretraining that both methods achieve lower validation loss than Muon at a cost comparable to Muon's.

Critical strengths
  1. Clear motivation. The paper points out that Muon's orthogonalization only balances the singular spectrum of the momentum matrix and does not use any information about the Hessian, so its behavior may still depend on how ill-conditioned the Hessian is. The toy least-squares example in Fig. 1 makes this concrete: Muon slows down noticeably as the Hessian condition number increases, while the Hessian-preconditioned version does not. I think this is a reasonable and relevant question to raise about Muon-type optimizers.

  2. Comprehensive literature review and detailed experimental setup. Section 2 summarizes recent Muon variants and Hessian-aware preconditioning methods, and Appendix A explains how MALT differs from Adafactor, Shampoo, SOAP, NorMuon and Mousse, which makes it easy to see where the paper stands. Appendix D reports the model configurations, optimizer hyperparameters, full learning-rate grids with the selected values, and how memory and runtime were measured, providing sufficient detail to reproduce the experiments.

Critical weaknesses

The following issues affect my assessment of whether the theory and experiments support the paper’s claims.

Technical issues

  1. The definition of the preconditioner is inconsistent with the algorithm. In (12)–(13), the preconditioner is defined as Bt(G)=PtGQt\mathcal{B}_t(G)=P_tGQ_t with Pt=Diag(lt+ϵ)−1/8P_t=\mathrm{Diag}(l_t+\epsilon)^{-1/8} and Qt=Diag(rt+ϵ)−1/8Q_t=\mathrm{Diag}(r_t+\epsilon)^{-1/8}. Its inverse square root is then

    Bt−1/2(G)=Diag(lt+ϵ)1/16GDiag(rt+ϵ)1/16,\mathcal{B}_t^{-1/2}(G)=\mathrm{Diag}(l_t+\epsilon)^{1/16} G \mathrm{Diag}(r_t+\epsilon)^{1/16},

    so plugging it into (9) gives an update with positive powers:

    Δt=Diag(lt+ϵ)1/16Orth(Diag(lt+ϵ)1/16∇fDiag(rt+ϵ)1/16)Diag(rt+ϵ)1/16.\Delta_t=\mathrm{Diag}(l_t+\epsilon)^{1/16} \mathrm{Orth}\big(\mathrm{Diag}(l_t+\epsilon)^{1/16} \nabla f \mathrm{Diag}(r_t+\epsilon)^{1/16}\big) \mathrm{Diag}(r_t+\epsilon)^{1/16}.

    However, Algorithm 1 uses Lt=Diag(lt+ϵ)−1/8L_t=\mathrm{Diag}(l_t+\epsilon)^{-1/8}. To match the algorithm, the preconditioner should instead be

    Bt(G)=Diag(lt+ϵ)1/4GDiag(rt+ϵ)1/4.\mathcal{B}_t(G)=\mathrm{Diag}(l_t+\epsilon)^{1/4} G \mathrm{Diag}(r_t+\epsilon)^{1/4}.

    This is more than a typo. Under the paper's definition,

    ⟨Δ,Bt(Δ)⟩=∑i,j(lt,i+ϵ)−1/8(rt,j+ϵ)−1/8Δij2,\langle\Delta,\mathcal{B}_t(\Delta)\rangle=\sum_{i,j}(l_{t,i}+\epsilon)^{-1/8}(r_{t,j}+\epsilon)^{-1/8}\Delta_{ij}^2,

    so rows with larger gradient norms are treated as having smaller curvature and receive larger updates, which is the opposite of the intuition stated in Section 3.2.

  2. The learning rate in MALTER appears to be applied twice. In Algorithm 2, αt\alpha_t in line 11 already contains η\eta, and line 12 multiplies by η\eta again, so the effective stepsize scales as η2\eta^2. In contrast, αt\alpha_t in (15) does not contain η\eta. The authors should clarify which version is implemented, since this may also explain why MALTER uses a learning-rate grid about an order of magnitude larger than the other methods (Table 5).

  3. Theorem 1 does not reflect any benefit of preconditioning. In the proof, LtL_t and RtR_t enter only through ∥Lt∥,∥Rt∥≤ϵ−1/8\Vert L_t\Vert,\Vert R_t\Vert\le\epsilon^{-1/8} and a lower bound on their smallest singular values, i.e., they are treated as bounded perturbations of Muon. As a result, the bound is worse than existing Muon bounds, so it gives no support to the claim that MALT mitigates Hessian anisotropy. Moreover, the stationarity measure min⁡{E∥∇f∥F1/2,E2∥∇f∥F1/2}\min\lbrace\mathbb{E}\Vert\nabla f\Vert_F^{1/2}, \mathbb{E}^2\Vert\nabla f\Vert_F^{1/2}\rbrace is weaker than the standard E∥∇f∥F\mathbb{E}\Vert\nabla f\Vert_F used in prior Muon analyses, as the authors acknowledge.

  4. The choice of curvature proxy is not justified. The vectors ltl_t and rtr_t are the row and column sums of the diagonal of the empirical Fisher F^\hat F. Under the corrected Bt\mathcal{B}_t, the implied curvature model is roughly diag(F^)1/4\mathrm{diag}(\hat F)^{1/4}. The paper does not explain why this power is used. The exponent −1/8-1/8 is borrowed from Shampoo, but Shampoo's −1/4-1/4 comes from a Kronecker upper bound on the full gradient accumulator in a regret analysis, and that argument does not carry over to diagonal row/column statistics. No ablation on the exponent is provided either.

Experimental issues

  1. Missing comparison with NorMuon. NorMuon (Li et al., 2025) is an earlier method that is closely related to MALT. After orthogonalization, it maintains a row-wise second-moment estimate of the update and normalizes each row by it, which is similar to the post-NS scaling in MALT, and its extra memory and compute cost is comparable. The ablation in Fig. 4 shows that applying the preconditioner only after NS already improves over Muon, so it is unclear how much MALT gains beyond NorMuon. Since MALT is positioned as a lightweight Muon variant, the fair comparison is against NorMuon under the same tuning protocol, rather than only against vanilla Muon.

Minor: The notation is inconsistent: the preconditioners are Pt/QtP_t/Q_t in the text but Lt/RtL_t/R_t in the algorithms, and MALTER uses both νt\nu_t and vtv_t for the same quantity.

Reference: (Li et al., 2025) NorMuon: Making Muon more efficient and scalable

Questions for authors

Questions for Authors:

The paper claims that MALT addresses the ill-conditioning of the Hessian, but the preconditioner it uses is not the Hessian itself; it is an approximation built from accumulated row/column gradient second moments. Can the authors show theoretically that preconditioning with this gradient-based approximation makes the convergence of the algorithm robust to the Hessian condition number? For example, in a simple setting such as the matrix quadratic problem in Fig. 1, can one show that MALT’s rate depends more mildly on the condition number than Muon’s? Currently, Theorem 1 does not involve the Hessian condition number at all, and the experiment in Fig. 1(b) uses exact Hessian whitening rather than MALT’s preconditioner. As a minimum, an empirical check running MALT itself on the toy problem across condition numbers would help. Without such evidence, the improvements of MALT cannot be attributed to its handling of ill-conditioning, which is the main claim of the paper.

If the authors can provide a convincing answer to this question and address the technical issues raised above, I would be happy to raise my score.

6 comments

Dear authors,

I would appreciate a response to my review. If my main concerns are addressed, I will consider raising my score; if they remain unaddressed at the end of the discussion period, I will have to maintain my current score.

AuthorsOpenReview ↗

Response to Reviewer GadJ point by point (1/3)

Thank you for your professional and valuable comments and we will response to your pointed weakness and questions point by point.

Response to Technical Issue 1 (Consistency of the Preconditioner Definition).

We thank the reviewer for identifying this inconsistency. We agree that the operator Bt\mathcal B_t in Eq. (13) was incorrectly defined. To be consistent with Eq. (9) and Algorithm 1, the correct definition is

$ \mathcal B_t(G)=L_t^{-2}GR_t^{-2}, $

where

$ L_t=\operatorname{Diag}(\ell_t+\epsilon)^{-1/8},\qquad R_t=\operatorname{Diag}(r_t+\epsilon)^{-1/8}. $

Equivalently,

$ \mathcal B_t(G)= \operatorname{Diag}(\ell_t+\epsilon)^{1/4} G \operatorname{Diag}(r_t+\epsilon)^{1/4}. $

Consequently, its inverse square root satisfies

$ \mathcal B_t^{-1/2}(G)=L_tGR_t, $

and substituting this into Eq. (9) yields exactly the update direction used in Algorithm 1:

$ D_t=L_t\operatorname{Orth}(L_tM_tR_t)R_t, $ which we actually have implemented in experiments. The corrected definition is also consistent with our motivation: larger row or column gradient second moments correspond to larger weights in the quadratic curvature surrogate and hence stronger preconditioning along the corresponding coordinates.

We have corrected Eq. (13) and unify the notation in the revised manuscript. This correction does not change Algorithm 1, the convergence proof, or the reported experiments, since they are based on the update using LtL_t and RtR_t.

Response to Technical Issue 2 (Learning Rate in MALTER).

We confirm that the extra factor of η\eta in Line 11 of Algorithm 2 was mistakenly included. The correct adaptive scaling factor is given by Eq. (15):

$ \alpha_t = \frac{\sqrt{1-\mu_2^t}}{1-\mu_1^t+\epsilon} \frac{||\widetilde M_t||_F}{\sqrt{\nu_t+\epsilon}}, $

and the update is

$ W_t=W_{t-1} -\eta_t\alpha_t \frac{||O_t||_F}{||D_t||_F+\epsilon}D_t. $

We have verified that our implementation follows Eq. (15) and applies the learning rate only once. Therefore, this typo does not affect any reported experimental results.

Regarding the larger learning rates used by MALTER, this is due to its noise-adaptive scaling factor αt\alpha_t, which adjusts the update magnitude according to the ratio between momentum magnitude and second-moment magnitude. In noise-dominated regimes, this factor can be substantially smaller than one, so a larger nominal learning rate may be needed to achieve a comparable effective update magnitude. The learning rates were selected through the grid searches reported in Appendix D.

We have removed the extra η\eta from Line 11 of Algorithm 2 and correct the notation in the revised manuscript.

Response to Technical Issue 3 (Convergence Analysis and the Role of Preconditioning).

We thank the reviewer for this important observation. Our convergence analysis is proved under the same standard assumptions commonly used in existing stochastic non-convex analyses of Muon [1, 2], namely lower boundedness, LL-smoothness, and unbiased stochastic gradients with bounded variance.

Compared with Muon, MALT involves a more complicated update, as it introduces time-varying diagonal preconditioners both before and after orthogonalization, making its convergence analysis more challenging. Under general assumptions, Theorem 1 achieves the same O(δ−4)O(\delta^{-4}) dependence on the target accuracy as existing stochastic Muon analyses, although under a weaker stationarity measure and with worse dimension dependent factors, as explicitly acknowledged in our paper. We do not claim that our convergence guarantee is as strong as the existing Muon bounds. Rather, the value of Theorem 1 is to establish that MALT retains a stochastic nonconvex convergence guarantee despite the additional time-varying two-sided preconditioning.

We agree that Theorem 1 does not establish improved dependence on Hessian conditioning. Under the general assumptions without any other structure assumptions, even for structured preconditioning methods such as Shampoo [3], the established theoretical guarantees focus on regret bounds in stochastic convex optimization, rather than explicitly demonstrating improved dependence on the Hessian condition number. We conjecture that establishing such curvature-dependent guarantees generally requires additional assumptions. For example, Sophia [4] establishes a condition number independent convergence bound for a simplified deterministic method using the full Hessian, under convexity and additional Hessian regularity assumptions.

Therefore, establishing a theoretically improved dependence on Hessian conditioning for MALT under current general assumptions remains challenging and would likely require additional structural assumptions. We will clarify the scope and limitations of Theorem 1 in the revised manuscript and distinguish this general convergence guarantee from a theoretical guarantee of improved Hessian conditioning.

AuthorsOpenReview ↗

Response to Reviewer GadJ point by point (2/3)

Response to Technical Issue 4 (Choice of Curvature Proxy and Preconditioning Exponent).

We thank the reviewer for this insightful comment. Our choice of the inverse eighth-root exponent is motivated by two considerations.

1. Motivated by Shampoo. MALT adopts a lightweight diagonal approximation of Shampoo-style preconditioning. Specifically, Shampoo [3] maintains two gradient accumulators,

$ A_t=A_{t-1}+G_tG_t^\top,\quad C_t=C_{t-1}+G_t^\top G_t, $

and uses the preconditioned direction in Euclidean space as

$ D_t^{\mathrm{Sh}} =(A_t+\epsilon I)^{-1/4} G_t(C_t+\epsilon I)^{-1/4}. $

MALT instead maintains exponential moving averages of their diagonal statistics, denoted by ℓt\ell_t and rtr_t, and applies

$ D_t=L_t\operatorname{Orth}(L_tM_tR_t)R_t, $

where Lt=Diag⁡(ℓt+ϵ)−1/8L_t=\operatorname{Diag}(\ell_t+\epsilon)^{-1/8} and Rt=Diag⁡(rt+ϵ)−1/8R_t=\operatorname{Diag}(r_t+\epsilon)^{-1/8}. Without orthogonalization, this reduces to

$ D_t=L_t^2M_tR_t^2, $

recovering Shampoo-style inverse fourth-root scaling with diagonal statistics. Thus, the inverse eighth-root exponent distributes this scaling across the pre- and post-orthogonalization operations. We agree that Shampoo's regret analysis does not directly extend to MALT.

2. Motivated by a diagonal quadratic model.

Consider

f(W)=12∑i=1dhi(wi−wi⋆)2,f(W)=\frac12\sum_{i=1}^d h_i(w_i-w_i^\star)^2,

where W=Diag⁡(w1,…,wd)W=\operatorname{Diag}(w_1,\ldots,w_d) and hi>0h_i>0. For deterministic gradients, the row and column squared-gradient EMAs coincide and satisfy

$ \ell_{i,t}=r_{i,t}=v_{i,t}=h_i^2q_{i,t}, $

where qi,tq_{i,t} is the EMA of squared coordinate errors.

To examine the exponent, consider general diagonal preconditioners with exponent −p-p. Ignoring damping, the eigenvalues of the Hessian in the preconditioned space satisfy

$ { \widetilde h_{i,t} =h_i v_{i,t}^{-4p} =h_i^{1-8p}q_{i,t}^{-4p}. } $

In particular, choosing p=1/8p=1/8 eliminates the explicit dependence on the Hessian eigenvalues hih_i, yielding $ \widetilde h_{i,t}=q_{i,t}^{-1/2}.$

Furthermore, when the squared error EMA moments are equal across coordinates, we obtain κ(H~t)=κ(H)∣1−8p∣.\kappa(\widetilde H_t)=\kappa(H)^{|1-8p|}. Hence, p=1/8p=1/8 gives κ(H~t)=1.{\kappa(\widetilde H_t)=1.}

This provides an additional curvature-based motivation for the inverse eighth-root exponent. We emphasize that this result applies to a simplified diagonal quadratic model under the stated conditions, and does not establish condition-number-independent convergence or the universal optimality of this exponent for general MALT. We will clarify these motivations and limitations in the revised manuscript.

3. Motivated by Exponent ablation. We have added an exponent ablation in Fig. 5 (Appendix C), comparing p={1/8,1/4,1/2}p=\{1/8,1/4,1/2\} on the same matrix quadratic problem as Fig. 1(c), with all other settings fixed. The results show that p=1/8p=1/8 achieves more stable convergence and lower optimality gaps across the tested Hessian condition numbers, while larger exponents degrade optimization performance. These findings provide empirical support for our choice of the inverse eighth-root exponent, although they do not imply its universal optimality.

Response to Experimental issues (Compared with NorMuon)

We have added a direct comparison with NorMuon [5] in the revised manuscript (page-10, 5.5 COMPARISON WITH NORMUON , Fig. 3). Specifically, we compare NorMuon with Muon, SOAP, Mousse, and MALT on GPT-2 Medium (355M), using a global batch size of 0.983M tokens and 10K training steps under the same WSD schedule.

As shown in the updated large-batch comparison Fig. 3, learning rate tuned NorMuon outperforms Muon and SOAP but achieves a higher final validation loss than Mousse and MALT.

Unlike NorMuon, which applies row-wise adaptive normalization after orthogonalization, MALT incorporates two-sided diagonal preconditioning both before and after orthogonalization. This comparison provides additional empirical evidence of MALT's effectiveness beyond vanilla Muon. We have also included the NorMuon implementation details in the revised appendix.

Reviewer_4ZvSOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to exploit curvature-related information for improving Muon. To this end, the authors suggest to incorporate preconditioners before the orthogonalization step in Muon. As preconditioners, the authors use EMAs of row/column sums of squared gradients for the sake of memory efficiency. The proposed optimizer MALT and MALTER is tested on GPT-2 style model upto 774M parameter.

Critical strengths

The paper is well-organized and easy to follow.

Critical weaknesses

In general, it seems that the idea is to combine Adafactor (Shazeer & Stern 2018) with Muon. Specifically, maintaining EMAs of row/column sums of squared gradients as an efficient form of the second moment estimate (one ideal object for Bt\mathcal{B}_t in the paper) is the exactly same as Adafactor. The main disctintion seems to be the stepsize scaling in Eq 15, but it is heuristics that doesn't have strong theoretical or empirical motivation.

Relatedly, the derivation and discussion in the main section (L179-L209) are about generic, textbook-level results on Taylor expansion and a related steepest descent discussion.

Finally, the paper lacks empirical rigors in terms of the number of models, token sizes, baselines, and model sizes. I suggest authors to consider the settings in previous works such as Liu et al 2026.

Reference

  • Shazeer, N., & Stern, M. (2018). Adafactor: Adaptive learning rates with sublinear memory cost. In ICML.

  • Liu, Y., Jiang, B., Hou, L., Tao, X., Wan, P., Lin, Z., & Fang, C. (2026). ORCA: The Annealed Spectral Conditioning Optimizer for Faster, Better LLM Training. arXiv preprint arXiv:2610.06116.

Questions for authors

The idea of combining preconditioning with Muon has been explored in Liu 2026, Chang 2026, Li 2026. Can you compare MALT with the related ideas and compare against with them?

Reference

  • Liu, Z., Zhang, R., Wang, Z., Zhao, Y., Su, Y., Yang, Z., & Zhang, Z. (2026). Muon 2^ 2: Boosting Muon via Adaptive Second-Moment Preconditioning. arXiv preprint arXiv:2604.09967.
  • Chang, D., Shi, Q., Zhang, L., Li, Y., Zhang, R., Lu, Y., ... & Yuan, G. (2026). Muoneq: Balancing before orthogonalization with lightweight equilibration. arXiv preprint arXiv:2603.28254.
  • Li, Z., Liu, L., Liang, C., Chen, W., & Zhao, T. (2026). Normuon: Making muon more efficient and scalable. In ICML.
3 comments

Reply regarding Weaknesses

1. Relationship to Adafactor [1]. We respectfully disagree with the characterization of MALT and MALTER as a combination of Adafactor and Muon. We agree that both MALT/MALTER and Adafactor leverage EMA of row/column-wise norm squares, denoted by lt\boldsymbol{l}_t and rt\boldsymbol{r}_t, and they look similar at first glance. However, our design and motivation here are fundamentally different from Adafactor. Adafactor uses lt\boldsymbol{l}_t and rt\boldsymbol{r}_t to construct a factored estimate of Adam's element-wise second moment. Up to a scalar normalization and omitting damping, its scaled gradient can be roughly written as

Diag(lt+ϵ)−12GtDiag(rt+ϵ)−12.\mathrm{Diag}(\boldsymbol{l}_t+\epsilon)^{-\frac{1}{2}}\boldsymbol{G}_t\mathrm{Diag}(\boldsymbol{r}_t+\epsilon)^{-\frac{1}{2}}.

MALT/MALTER instead uses them as curvature-aware preconditioners and performs Muon in the preconditioned coordinates. In particular, MALT/MALTER first transforms the first moment Mt\boldsymbol{M}_t into the preconditioned space before Muon orthogonalization, and then maps the orthogonalized result back to the original space:

Diag(lt+ϵ)−18Orth(Diag(lt+ϵ)−18MtDiag(rt+ϵ)−18)Diag(rt+ϵ)−18.\mathrm{Diag}(\boldsymbol{l}_t+\epsilon)^{-\frac{1}{8}}\mathrm{Orth}(\mathrm{Diag}(\boldsymbol{l}_t+\epsilon)^{-\frac{1}{8}}\boldsymbol{M}_t\mathrm{Diag}(\boldsymbol{r}_t+\epsilon)^{-\frac{1}{8}})\mathrm{Diag}(\boldsymbol{r}_t+\epsilon)^{-\frac{1}{8}}.

Our motivation here is to leverage lt\boldsymbol{l}_t and rt\boldsymbol{r}_t to address the curvature anisotropy [2]. In the updated manuscript, we provide an empirical justification of this on the matrix quadratic problem (p. 2, Sec. 1, Fig. 1(c)), showing that preconditioning in MALT reduces the sensitivity of Muon to increasing condition number.

Moreover, MALT enjoys resource efficiency compared with other curvature preconditioned Muon methods. In our benchmark, MALT uses 1.63 GB of optimizer-state memory and takes 2.65 s per training step, compared with 10.89 GB and 3.43 s for Mousse (p. 9, Sec. 5.2, Table 2). MALT also achieves lower final validation loss than Mousse in the large-batch GPT-2 Medium experiment (p. 10, Sec. 5.4, Fig. 3).

Therefore, MALT/MALTER are not trivial combinations of Adafactor and Muon, and the distinction between Adafactor and MALT/MALTER is not an adaptive stepsize in Eq. (15) (p. 7, Sec. 3.3).

2. About the role of derivation in Section 3.1. Section 3.1 (pp. 4–5) explains why MALT and MALTER apply the preconditioners both before and after orthogonalization. We agree that the quadratic surrogate and its interpretation as Euclidean steepest descent are classical. They establish the change of coordinates used in Eq. (7) (p. 5, Sec. 3.1). The design choice in Eq. (8) (p. 5, Sec. 3.1) is to take a Muon step in those coordinates. This distinguishes MALT from Adafactor, as well as other Muon variants such as NorMuon [3].

3. Motivation and evidence for the adaptive stepsize in Eq. (15) (p. 7, Sec. 3.3). As stated in Section 3.3 (pp. 6–7), MALT uses norm grafting to preserve the direction of the preconditioned update while matching its Frobenius norm to that of the orthogonalized momentum. This is motivated by the grafting method studied by Agarwal et al. [4] and used in scalable Shampoo [5]. A closely related example is NorMuon [3], whose implementation rescales the row-normalized update to recover its Frobenius norm before row normalization.

Furthermore, we empirically verified the separate benefits of the adaptive stepsize, as well as preconditioning. Please see p. 10, Sec. 5.3, Fig. 4 in the manuscript. We compare MALT, MALTER, NAMO [6] (Muon with adaptive stepsize), and two precondition variants of MALT on GPT-2 Small to see the difference. The result indicates that both diagonal preconditioning and adaptive stepsize contribute to the performance gain of MALT/MALTER. This provides empirical evidence that both the adaptive stepsize and preconditioning improve the performance of Muon.

Reply regarding Weaknesses cont'd

4. Experimental scope, training budgets, and baselines. We thank the reviewer for your valuable suggestion.

Regarding the model sizes and the training token budget, we would like to clarify that our GPT-2 Small (124M) model is trained for 50K steps with 491,520 tokens per step, corresponding to approximately 24.6B processed tokens, or 9.9 times the commonly used Chinchilla-optimal reference budget of 20 tokens per parameter. Both MALT and MALTER consistently outperform Muon throughout this extended training run (p. 9, Sec. 5.1, Fig. 2). For GPT-2 Medium (355M) and Large (774M), the corresponding training budgets are approximately 4.9B and 8.8B processed tokens, respectively. We note that these counts include repeated exposure to the OpenWebText training data.

Regarding the baselines, we compare MALT and MALTER with AdamW and Muon across all three GPT-2 model sizes (p. 8, Sec. 5.1, Table 1), with a separate learning-rate sweep for each optimizer at each scale (p. 26, App. D.1, Tables 5–6). Section 5.4 compares MALT with SOAP and Mousse on GPT-2 Small and Medium at 0.492M tokens per step (p. 9, Sec. 5.4, Table 3). We also compare these methods on GPT-2 Medium at 0.983M tokens per step under a warmup–stable–decay schedule (p. 10, Sec. 5.4, Fig. 3). In response to the reviewer's suggestion, we have added NorMuon to this large-batch comparison (p. 10, Sec. 5.5, Fig. 3). We discuss the results in our response to the reviewer's question below.

Regarding the number of models, we acknowledge that our current experiments are limited to dense GPT-2 models with up to 774M parameters. A fair evaluation on larger models and MoE architectures would require substantial computational resources, particularly because each optimizer needs an independent learning-rate sweep under matched training protocols. Given our limited computational resources and the time available during the rebuttal period, we were unable to conduct these additional experiments. We acknowledge this limitation and leave systematic evaluations on larger-scale and MoE models for future work.

Reply to the Questions

  1. Comparison with related ideas. We claim that the idea of MALT/MALTER is different from the following existing works.

Muon$^2$[7]. Muon$^2$ divides the momentum element-wise by the square root of a gradient second-moment estimate before Newton–Schulz orthogonalization. Its factored variant, Muon$^2$-F, is particularly relevant to MALT because it uses the same row and column norm information as MALT. Muon$^2$-F can be seen as a trivial combination of Adafactor and Muon. It performs Adafactor-style elementwise scaling to the momentum before orthogonalization.
As we discussed in the reply to weaknesses, MALT transforms the momentum into the preconditioned space before orthogonalization and maps the resulting direction back.

MuonEq [8]. MuonEq computes row norms, column norms, or both from the current momentum matrix and uses them to balance the momentum before orthogonalization. The motivation of MuonEq is different from ours. They claim that this balancing improves the conditioning of the momentum matrix and then reduces the orthogonalization error of finite Newton–Schulz iterations.

NorMuon [3]. NorMuon orthogonalizes the momentum first, then applies a row-wise adaptive learning rate using a moving average of the squared row norms of the orthogonalized momentum. In MALT, both row and column statistics come from the stochastic gradient.

  1. Empirical comparison. We added a direct comparison with NorMuon on GPT-2 Medium (355M), using 10K steps, 0.983M tokens per step, and the same warmup–stable–decay schedule, with the learning rate tuned separately for each optimizer. NorMuon achieves lower final validation loss than Muon and SOAP, while MALT achieves lower final validation loss than NorMuon (p. 10, Sec. 5.5, Fig. 3).

Our GPT-2 Small ablation (p. 10, Sec. 5.3, Fig. 4) examines where to apply MALT’s diagonal preconditioners relative to Newton–Schulz (NS) orthogonalization. We compare full MALT with two variants that apply the same preconditioners only before NS or only after NS. Each variant achieves a slightly lower validation loss than Muon. Applying the preconditioners both before and after NS yields a further reduction over either variant, indicating that both steps contribute to MALT’s improvement in this setting.

The pre-NS variant is most closely related to Muon²-F [7]. Both scale the momentum before orthogonalization using moving averages of row and column sums of squared gradients. Muon²-F uses an exponent of −1/2-1/2 for each factor, while our variant uses −1/8-1/8. MuonEq [8] also scales the momentum before orthogonalization, but computes its factors from the row or column norms of the current momentum matrix. Our ablation therefore isolates the effect of preconditioner placement within MALT. Direct empirical comparisons with Muon²-F and MuonEq remain open.

References

[1] N. Shazeer and M. Stern. Adafactor: Adaptive Learning Rates with Sublinear Memory Cost. ICML, 2018.

[2] T. T.-K. Lau, Q. Long, and W. J. Su. PolarGrad: A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspective. arXiv:2505.21799, 2025.

[3] Z. Li, L. Liu, C. Liang, W. Chen, and T. Zhao. NorMuon: Making Muon More Efficient and Scalable. ICML, 2026.

[4] N. Agarwal, R. Anil, E. Hazan, T. Koren, and C. Zhang. Disentangling Adaptive Gradient Methods from Learning Rates. arXiv:2002.11803, 2020.

[5] R. Anil, V. Gupta, T. Koren, K. Regan, and Y. Singer. Scalable Second Order Optimization for Deep Learning. arXiv:2002.09018, 2020.

[6] M. Zhang, Y. Liu, and H. Schaeffer. Adam Improves Muon: Adaptive Moment Estimation with Orthogonalized Momentum. arXiv:2602.17080, 2026.

[7] Z. Liu et al. Muon²: Boosting Muon via Adaptive Second-Moment Preconditioning. arXiv:2604.09967, 2026.

[8] D. Chang et al. MuonEq: Balancing Before Orthogonalization with Lightweight Equilibration. arXiv:2603.28254, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.