acceptodds
Under review as a conference paper at ICLR 2027

Accelerating Muon with Spectral Residual Correction

Abstract

Muon constructs matrix updates by applying a matrix-sign transformation to momentum. A common route to accelerating Muon is to modify the gradient or momentum before this transformation. We explore a complementary direction: refining the parameter step while retaining Muon's momentum recursion. Derived from the parameter subproblem of the ADMM-Inspired Momentum framework, AIMuon uses the gradient–momentum residual to correct the tentative spectral step. The residual undergoes a separate matrix-sign transformation, while spectral–nuclear duality determines the relative weighting of the momentum and residual directions. In practice, both directions are approximated with Newton–Schulz iterations and their combination is rescaled to match Muon's update magnitude. We establish a stochastic stationarity bound for the corresponding exact iteration. On GPT-2-small and GPT-2-medium pretraining, AIMuon reaches Muon's final validation loss with 12.8% and 7.7% fewer optimization steps, respectively, while achieving lower final validation loss than both Muon and MONA. Ablation studies further support the roles of the residual correction, separate transformation, adaptive nuclear-norm weighting, and final normalization. These results show that parameter-step refinement can complement momentum construction in matrix-sign optimization.

Primary area:
optimization
Keywords:
Deep learning optimization, Muon, Accelerated optimization, Residual correction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_X5jmOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AIMuon augments Muon with the gradient–momentum residual. It transforms momentum and residual separately using approximate polar maps, weights their contributions through estimated nuclear norms, and matches the combined update magnitude to Muon's scale. The paper derives the exact update from AIM's spectral parameter subproblem and proves a stochastic stationarity bound for that idealized iteration. GPT-2-small and GPT-2-medium experiments report lower validation loss and fewer steps to Muon's final loss. The medium-model run reports a 6.6% elapsed-time saving including the extra residual computation.

Critical strengths
  1. The construction is coherent and the exact theory is sound. Spectral–nuclear duality yields the two weighted polar directions. Appendix B correctly handles the dependence of the update on stochastic gradient noise. The manuscript clearly states the theoretical scope and explains the effect of practical normalization.

  2. The evaluation includes relevant controls and measured training time. Muon and MONA are appropriate comparators. Matched initialization and data order improve fairness, and the ablations examine the residual's placement, weighting, and normalization. Figure 2 reports 8.04 hours versus 8.61 hours for the medium-model target, including the second spectral computation.

Critical weaknesses
  1. The main gain needs a robustness check. Appendix C reports one seed per method. Every selected rate is the upper endpoint of a search using about 10% of the full training budget. The loss improvements over Muon are 0.0072 and 0.0029. These results leave uncertainty about training variation and whether nearby higher rates would change the comparison. This is the main reason for weak rejection.

  2. The Nesterov baseline needs clarification. The Muon implementation's Nesterov setting is unspecified. The Inside-MSign ablation uses the paper's Nesterov-style combination and reaches 3.2198, compared with AIMuon's 3.2161 and Muon's 3.2233. It receives no independent rate tuning. This closest control is important for establishing the benefit of a separate residual transformation.

Questions for authors
  1. Can you report existing higher-rate results and additional paired seeds? A modest rate extension and a few paired repeats on GPT-2-small could establish whether the gain is stable. Consistent loss and target-crossing improvements would address the main obstacle to acceptance.

  2. What matrix enters Muon's polar approximation, and is Nesterov enabled? Please identify the implementation and report any independently tuned result for the existing Inside-MSign control. A stable advantage over that comparator, together with the first answer, could move my rating from 2 to 3.

6 comments
AuthorsOpenReview ↗

Robustness of AIMuon’s Gains and Comparisons with Tuned Baselines

Thank you for the constructive assessment and for recognizing the coherence of the construction and the soundness of the exact analysis. We focused the additional experiments on your two main concerns: the robustness of the gains after broader tuning and repeated runs, and the comparison with Nesterov and the closest inside-transform control.

After extending the learning-rate searches and independently tuning momentum, AIMuon achieves lower final validation loss than Muon and Muon-Nesterov in all three paired GPT-2-small runs. It also reaches their final mean validation losses with 8.98% and 10.89% fewer steps, respectively, as measured from the mean validation-loss curves. The independently tuned Inside-MSign control reaches 3.1949, compared with 3.1939 for AIMuon in the seed-42 comparison.

AuthorsOpenReview ↗

W1 & Q1. Broader hyperparameter searches and additional paired seeds

We extended the learning-rate searches and independently repeated hyperparameter selection for each optimizer. We use a sequential two-stage procedure: first select the peak learning rate, then compare momentum coefficients β∈0.9,0.95\beta\in\\{0.9,0.95\\} at each optimizer's selected learning rate.

SettingPeak learning-rate candidates, in units of 10−310^{-3}Tokens per candidate in each search stageSelected peak learning rate
GPT-2-small2, 2.5, 3, 3.5, 4, 4.5, 52.48B0.0045
GPT-2-medium1, 2, 3, 4, 4.5, 50.69B0.0045
Qwen3-0.6B1, 2, 3, 3.5, 4, 4.51.2B0.004

In both the learning-rate and momentum searches for the main optimizers, each candidate is trained for 2.48B tokens on GPT-2-small, 0.69B tokens on GPT-2-medium, and 1.2B tokens on Qwen3-0.6B. These are per-candidate budgets in each search stage.

The learning-rate searches initially use β=0.95\beta=0.95 for the GPT-2 experiments and β=0.9\beta=0.9 for Qwen3-0.6B. The latter choice follows the GPT-2 momentum-search results. Each optimizer is tuned independently, although all methods select the same learning rate within each model setting. The selected learning rates are now interior points of their respective search grids.

Fixing these selected learning rates, we independently search β∈0.9,0.95\beta\in\\{0.9,0.95\\} for each optimizer. The selected momentum coefficients are:

OptimizerGPT-2-smallGPT-2-mediumQwen3-0.6B
AIMuon0.90.90.95
Muon0.90.90.95
Muon-Nesterov0.90.90.95
MONA0.90.90.9

AIMuon uses c=0.045c=0.045, 0.04750.0475, and 0.050.05 for GPT-2-small, GPT-2-medium, and Qwen3-0.6B, respectively.

Every search candidate starts from scratch using seed 42, with matched initialization, training-data order, and validation data. Each run uses a complete warmup and cosine-decay schedule for its allotted budget. Selection is based on terminal validation loss, and the AdamW fallback momentum coefficients remain unchanged. All updated experiments run on eight NVIDIA H100 GPUs.

Three-seed GPT-2-small results. With the selected hyperparameters fixed, we evaluate each optimizer across seeds 42, 43, and 44:

OptimizerValidation loss, seed 42Seed 43Seed 44Mean ± standard deviation
Muon3.19803.19743.19843.1980 ± 0.0005
Muon-Nesterov3.20003.19973.19903.1996 ± 0.0005
MONA3.21983.22153.22173.2210 ± 0.0010
AIMuon3.19393.19503.19463.1945 ± 0.0005

All runs use 2.48B tokens. Within each seed, initialization and training-data order are matched across methods. Seed 42 is also used for hyperparameter selection; seeds 43 and 44 are additional evaluations without further tuning.

AIMuon achieves the lowest validation loss in all three paired runs. Its loss improvement over Muon ranges from approximately 0.0024 to 0.0041. The advantage over the independently tuned Muon-Nesterov baseline also persists in all three seeds.

Larger-model results. We retrained GPT-2-medium for 6.9B tokens and added Qwen3-0.6B pretraining for 12B tokens:

OptimizerGPT-2-medium validation lossQwen3-0.6B validation loss
Muon3.02552.7859
Muon-Nesterov3.01952.7830
MONA3.03162.7824
AIMuon3.01872.7819

Both settings use seed 42. AIMuon achieves the lowest terminal loss in both comparisons after independent baseline tuning. These results extend the evaluation beyond GPT-2-small.

Steps to baseline performance. We additionally measure the optimization steps AIMuon requires to reach the final validation losses of the independently tuned Muon-Polyak and Muon-Nesterov baselines. AIMuon reaches both targets in fewer steps across all three models:

ModelStep reduction vs. Muon-PolyakStep reduction vs. Muon-Nesterov
GPT-2-small8.98%10.89%
GPT-2-medium10.05%4.13%
Qwen3-0.6B6.49%3.78%

For GPT-2-small, we first average validation losses across seeds 42, 43, and 44 at each evaluation step, then interpolate the mean curve to locate AIMuon's first crossing of each baseline's final mean validation loss. GPT-2-medium and Qwen3-0.6B use the seed-42 curves with the same interpolation procedure. The reduction is computed as (Tb−TAIMuon→b)/Tb×100(T_b-T_{\mathrm{AIMuon}\to b})/T_b\times100\\%, where TbT_b is the baseline's full training length and TAIMuon→bT_{\mathrm{AIMuon}\to b} is AIMuon's interpolated target-reaching step. All crossings use the original full-budget learning-rate schedules.

AuthorsOpenReview ↗

W2 & Q2. Muon implementation, Nesterov setting, and the Inside-MSign control

Thank you for pointing out the ambiguity in the baseline description. The original Muon baseline uses ordinary EMA momentum, without the Nesterov combination:

Mk=βMk−1+(1−β)Gk,DkMuon=NS⁡5(Mk).M_k=\beta M_{k-1}+(1-\beta)G_k,\qquad D_k^{\mathrm{Muon}}=\operatorname{NS}_5(M_k).

The added Muon-Nesterov baseline retains the same momentum state but changes the input to the polar approximation:

Nk=βMk+(1−β)Gk,DkNesterov=NS⁡5(Nk).N_k=\beta M_k+(1-\beta)G_k,\qquad D_k^{\mathrm{Nesterov}}=\operatorname{NS}_5(N_k).

Both use the Keller Jordan / modded-nanoGPT-style five-step Newton–Schulz construction, with polynomial coefficients (3.4445,−4.7750,2.0315)(3.4445,-4.7750,2.0315), together with the LLM-style matrix scaling convention. Parameter grouping, the AdamW fallback, data processing, and training schedules are shared. We specify these implementation details explicitly in the revised experimental description.

Inside-MSign definition. This control uses a tunable mixture:

Ek=Gk−Mk,Dkinside=NS⁡5(Mk+cEk).E_k=G_k-M_k,\qquad D_k^{\mathrm{inside}}=\operatorname{NS}_5(M_k+cE_k).

Since

Mk+cEk=(1−c)Mk+cGk,M_k+cE_k=(1-c)M_k+cG_k,

its input equals the Nesterov-style combination when c=1−βc=1-\beta. For general cc, it represents a more flexible gradient–momentum mixture. Its final magnitude is matched to the momentum-based Muon reference, following the ablation protocol. Thus, equality of the transformation inputs alone does not imply that its complete update is identical to the separate Muon-Nesterov baseline.

Tuning and updated ablations. We independently tune the correction coefficients of Inside-MSign and Linear residual over

c∈0.01,0.025,0.05,0.1,0.2.c\in\\{0.01,0.025,0.05,0.1,0.2\\}.

Each candidate uses 0.248B tokens and seed 42. Inside-MSign initially selects 0.01, the lower endpoint, so we additionally evaluate 0.005 under the same protocol. The value 0.01 remains better. Linear residual selects 0.025. Each selected configuration is then restarted from scratch and trained for 2.48B tokens.

Muon-Nesterov receives independent learning-rate and momentum tuning. Both residual controls receive independent tuning of cc and the learning rate, with the momentum coefficient fixed at 0.9. Both Inside-MSign and Linear residual use the learning-rate candidates 4.0,4.5,5.0×10−3\\{4.0,4.5,5.0\\}\times10^{-3}, with 0.248B tokens per candidate and seed 42. Both select 4.5×10−34.5\times10^{-3} based on terminal validation loss. Together with the coefficient searches above, these searches independently tune both controls under the same candidate ranges and search budgets. The no-residual variant is AIMuon with c=0c=0, using the independently selected Muon learning rate and momentum coefficient. Its retained Frobenius-norm matching introduces numerical differences from the standalone Muon baseline.

ConfigurationCoefficientFinal validation loss
Full AIMuonc=0.045c=0.0453.1939
Fixed residual weightλfix=0.19\lambda_{\mathrm{fix}}=0.193.1944
Inside-MSignc=0.01c=0.013.1949
No residualc=0c=03.1964
Linear residualc=0.025c=0.0253.1984

Removing the residual gives a final validation loss of 3.1964, compared with 3.1939 for full AIMuon. Inside-MSign improves upon the no-residual variant, while Linear residual does not. These comparisons support the benefit of independently transforming the residual.

Purpose of magnitude matching. We follow Muon's RMS-matching convention, which aligns its update scale with the typical AdamW update scale (Liu et al., 2025). We then rescale the combined direction to match the RMS magnitude of the corresponding Muon reference at the same momentum state. With the learning rate held fixed, this aligns the update magnitude and helps distinguish the benefit of the update direction from that of a larger step. AIMuon, Inside-MSign, and Linear residual all retain this matching, so their direction constructions are compared under the same magnitude-matching rule.

AIMuon retains a lower validation loss than Inside-MSign after tuning both its correction coefficient and learning rate, supporting the benefit of independently transforming the residual in the evaluated configuration. The fixed-weight variant also captures most of the gain, with adaptive nuclear-norm weighting providing a smaller additional improvement.

The expanded evaluation shows that the final-loss improvement persists after broader learning-rate searches and across three matched GPT-2-small seeds, including against an independently tuned Muon-Nesterov baseline. The independently tuned Inside-MSign comparison provides additional evidence for the separate residual transformation.

Reviewer_4UHiOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose a new Muon variant based on ADMM-Inspired Momentum. They provide experiments demonstrating that the proposed optimizer outperforms Muon on GPT-2-small and GPT-2-medium pretraining tasks.

Critical strengths
  • I could not identify any particular strengths of this paper. It is possible that the ideas offer genuine contributions, but the quality of the presentation (see below) makes it difficult to position the current manuscript as a potentially impactful work.
Critical weaknesses
  • The writing is very hard to read in many places. Repeated usage of vague phrases such as "parameter-subproblem refinement" make it difficult to understand the motivation and contribution of the proposed optimizer. Furthermore, many quantities are not defined when they first appear, such as ρ1,k\rho_{1,k} and YY in (6).
  • No motivation is provided for considering the spectral norm regularized problems in e.g. (4), (6), and (17). Indeed, the updates of Muon and related algorithms can be interpreted as such, but what is the purpose of this perspective? There is a similar problem with the presentation of the main algorithm in (11).
  • The experiments are significantly lacking, being extremely small scale and featuring hyperparameters that do not appear to be tuned well.
  • I have not checked the proofs, but the above issues are significant enough that I find the current manuscript to be below the bar for acceptance.
Questions for authors
  1. Why do you refer to (10) as Nesterov-style momentum when it is in fact Polyak or classical momentum?
  2. Can you provide motivation for what this algorithm is doing instead of merely presenting a Muon variant? Beyond the ablation studies in Section 4.4, why should one expect the design choices of AIMuon to lead to a better optimizer? Also, what is the conceptual contribution besides applying the AIM framework to Muon?
  3. Address the other weaknesses above.
8 comments
AuthorsOpenReview ↗

W1 & Q1. Presentation, notation, and the distinction between EMA momentum and the Nesterov-style update

Thank you for identifying the presentation issues and for asking us to clarify the motivation and conceptual contribution. We have reorganized the method description and expanded the experiments. Below, we first clarify the AIM construction and the Nesterov terminology, then explain the choice of spectral penalties and the resulting algorithmic insight, and finally report the updated experimental evidence.

We agree that the original presentation made the method unnecessarily difficult to follow. The AIM framework was introduced fully only in the appendix, while the main text repeatedly used “parameter-subproblem refinement” without explaining the underlying operation. Some symbols were also introduced too late, and the momentum-state recursion was not clearly distinguished from the direction used for the parameter update.

We have moved the AIM subproblems and the relationships needed to understand our construction to Section 2.2, followed by the spectral-geometry construction in Section 2.3. The auxiliary variable YY, multiplier MM, convex penalty ψ\psi, and positive penalty parameters ρ1,k\rho_{1,k} and ρ2,k\rho_{2,k} are now defined when first introduced. The revised method section explicitly describes the operation as solving a local surrogate obtained by linearizing the objective. The Euclidean example below illustrates the meaning of “parameter-subproblem refinement” in the original presentation.

1. What does “parameter-subproblem refinement” mean?

AIM consists of an auxiliary-variable subproblem, a parameter subproblem, and a multiplier update. By “parameter-subproblem refinement,” we mean taking an additional approximation step with the aim of solving the parameter subproblem more accurately. The auxiliary-variable subproblem first produces a tentative momentum step Yk+1Y_{k+1}. Instead of directly accepting Θk+1=Yk+1\Theta_{k+1}=Y_{k+1}, we further use the optimality condition of the parameter subproblem to construct an approximate solution. In the Euclidean example below, this takes the form of one fixed-point step from Θk\Theta_k using the available stochastic gradient, producing a residual correction to the tentative momentum step.

Under exact parameter-subproblem optimality, the multiplier update admits an EMA interpretation. For our explicit optimizer construction, we retain the EMA recursion and solve a locally linearized parameter subproblem, using the current stochastic gradient.

The Euclidean case provides a concrete illustration. We retain

Gk=∇f(Θk;ξk),Mk=βMk−1+(1−β)Gk.G_k=\nabla f(\Theta_k;\xi_k),\qquad M_k=\beta M_{k-1}+(1-\beta)G_k.

With ψ(X)=∥X∥F2\psi(X)=\lVert X\rVert_F^2 and

ρ1=1η,ρ2=1η(1−β),\rho_1=\frac{1}{\eta},\qquad\rho_2=\frac{1}{\eta(1-\beta)},

the auxiliary-variable subproblem gives

Yk+1=Θk−ηMk.Y_{k+1}=\Theta_k-\eta M_k.

Directly accepting this tentative point gives SGD with momentum in the EMA parameterization:

Θk+1=Yk+1=Θk−ηMk.\Theta_{k+1}=Y_{k+1}=\Theta_k-\eta M_k.

Alternatively, the first-order condition of the parameter subproblem yields the implicit relation

Θ=Yk+1−η(1−β)(∇F(Θ)−Mk).\Theta=Y_{k+1}-\eta(1-\beta)\bigl(\nabla F(\Theta)-M_k\bigr).

Taking one fixed-point step from Θk\Theta_k using the available stochastic gradient gives

Θk+1=Yk+1−η(1−β)(Gk−Mk)=Θk−η[βMk+(1−β)Gk].\Theta_{k+1}=Y_{k+1}-\eta(1-\beta)(G_k-M_k)=\Theta_k-\eta\bigl[\beta M_k+(1-\beta)G_k\bigr].

2. What is Nesterov-style, and what remains ordinary momentum?

You are correct that

Mk=βMk−1+(1−β)GkM_k=\beta M_{k-1}+(1-\beta)G_k

is ordinary Polyak/EMA momentum. The term “Nesterov-style” refers to the combination used for the parameter update,

M‾k=βMk+(1−β)Gk,\overline{M}_k=\beta M_k+(1-\beta)G_k,

not to the recursion defining MkM_k. The state retained for the next iteration is MkM_k; forming M‾k\overline{M}_k does not overwrite it.

To clarify the parameterization, consider an unnormalized momentum buffer

Bk=βBk−1+Gk.B_k=\beta B_{k-1}+G_k.

For fixed β\beta and consistent zero initialization,

Mk=(1−β)Bk,M_k=(1-\beta)B_k,

and therefore

M‾k=(1−β)(Gk+βBk).\overline{M}_k=(1-\beta)(G_k+\beta B_k).

Here, Gk+βBkG_k+\beta B_k is the usual Nesterov update combination under this implementation convention. The factor 1−β1-\beta comes from the EMA parameterization.

For the general residual mixture,

Vk(c)=Mk+cEk=(1−c)Mk+cGk,Ek=Gk−Mk,V_k(c)=M_k+cE_k=(1-c)M_k+cG_k,\qquad E_k=G_k-M_k,

the Nesterov-style correspondence holds specifically when c=1−βc=1-\beta. For other values of cc, we describe it as a tunable gradient–momentum mixture. We have corrected the discussion of the original Eq. (10) accordingly.

The Euclidean correspondence illustrates how changing the treatment of the parameter subproblem changes the update from ordinary momentum to a Nesterov-style combination. This motivates exploring an analogous correction in spectral geometry.

AuthorsOpenReview ↗

W2 & Q2. Why spectral penalties, and what conceptual contribution do they provide?

Our motivation is to use the objective-containing parameter subproblem to improve the tentative momentum step. Directly accepting the tentative point uses only the accumulated momentum. Linearizing the parameter subproblem at the current iterate introduces the gradient–momentum residual, allowing current-gradient information to contribute directly to the final update. The Euclidean correspondence with a Nesterov-style update motivates this construction. In spectral geometry, solving the resulting local subproblem jointly determines the residual's independent matrix-sign transformation and its nuclear-norm weight.

1. Why use spectral geometry?

Yang, Simon, and Bernstein (2023) connect spectral scaling to feature learning and maximal update parameterization. This connection motivates our use of spectral geometry. The effect of a weight update on layer features can be illustrated using a linear layer

h=Wx,W∈Rdout×din.h=Wx,\qquad W\in\mathbb{R}^{d_{\mathrm{out}}\times d_{\mathrm{in}}}.

At fixed input, the direct effect of a parameter update is Δh=ΔWx\Delta h=\Delta W x. Hence,

RMS⁡(Δh)≤dindout∥ΔW∥2RMS⁡(x).\operatorname{RMS}(\Delta h)\leq\sqrt{\frac{d_{\mathrm{in}}}{d_{\mathrm{out}}}}\lVert\Delta W\rVert_2\operatorname{RMS}(x).

The dimension-adjusted spectral norm therefore controls the largest amplification from input features to output changes, providing a feature-level interpretation of the spectral displacement penalty.

Spectral geometry also preserves Muon's underlying direction construction. By spectral–nuclear duality,

∥A∥∗=max⁡∥Z∥2≤1⟨A,Z⟩,\lVert A\rVert_{\ast}=\max_{\lVert Z\rVert_2\leq1}\langle A,Z\rangle,

with MSign⁡(A)\operatorname{MSign}(A) as a canonical maximizer. Thus, the constrained linear problem

min⁡Δ⟨Mk,Δ⟩subject to∥Δ∥2≤ηk\min_{\Delta}\langle M_k,\Delta\rangle\quad\text{subject to}\quad\lVert\Delta\rVert_2\leq\eta_k

admits the canonical solution

Δk=−ηkMSign⁡(Mk).\Delta_k=-\eta_k\operatorname{MSign}(M_k).

This is Muon's basic spectral direction with momentum input MkM_k.

AuthorsOpenReview ↗

W2 & Q2. Why spectral penalties, and what conceptual contribution do they provide?

2. Why place the squared spectral norm in a penalty term? Inspiration from the local optimization formulation of Adam/AdamW

Ignoring AdamW's decoupled weight decay, its preconditioned update with a given positive-definite preconditioning matrix VkV_k can be expressed as

Δk=argmin⁡Δ(⟨mk,Δ⟩+12ηk∥Δ∥Vk2),\Delta_k=\underset{\Delta}{\operatorname{argmin}}\left(\langle m_k,\Delta\rangle+\frac{1}{2\eta_k}\lVert\Delta\rVert_{V_k}^{2}\right),

whose solution is Δk=−ηkVk−1mk\Delta_k=-\eta_kV_k^{-1}m_k. Here, the update geometry is expressed through a displacement penalty in the objective, whereas the standard Muon formulation above places the spectral norm in a constraint.

This contrast motivates the question: can we likewise express Muon's spectral geometry through a displacement penalty and use it in AIM's auxiliary-variable and parameter subproblems? To this end, we consider

min⁡Δ(⟨Mk,Δ⟩+12ηk∥Δ∥22).\min_{\Delta}\left(\langle M_k,\Delta\rangle+\frac{1}{2\eta_k}\lVert\Delta\rVert_2^2\right).

By spectral–nuclear duality, choosing its canonical polar solution gives

Δk=−ηk∥Mk∥∗MSign⁡(Mk).\Delta_k=-\eta_k\lVert M_k\rVert_{\ast}\operatorname{MSign}(M_k).

This solution has the same matrix-sign direction as Muon, with an additional nuclear-norm radial factor determined by the penalized subproblem. For nonzero inputs, this common scalar factor cancels under the same per-matrix RMS matching, ignoring numerical stabilizers. Following Muon's convention for aligning the update RMS with the typical AdamW scale (Liu et al., 2025), the normalized momentum-only update therefore coincides with the corresponding Muon update. The same scalar cancellation holds for corresponding implementations using the same NS5 direction. This establishes the connection between AIM's penalized spectral subproblem and the standard Muon update.

Building on this connection, we consider the parameter subproblem containing the objective function:

min⁡Θ(F(Θ)+⟨Mk,Yk+1−Θ⟩+ρ2,k2∥Θ−Yk+1∥22).\min_{\Theta}\left(F(\Theta)+\langle M_k,Y_{k+1}-\Theta\rangle+\frac{\rho_{2,k}}{2}\lVert\Theta-Y_{k+1}\rVert_2^2\right).

Locally linearizing FF at Θk\Theta_k using the current stochastic gradient GkG_k, and setting Δ=Θ−Yk+1\Delta=\Theta-Y_{k+1}, the linearized objective and multiplier term together give

⟨Gk,Δ⟩−⟨Mk,Δ⟩=⟨Gk−Mk,Δ⟩.\langle G_k,\Delta\rangle-\langle M_k,\Delta\rangle=\langle G_k-M_k,\Delta\rangle.

Dropping terms independent of Δ\Delta therefore yields

min⁡Δ(⟨Ek,Δ⟩+ρ2,k2∥Δ∥22),Ek=Gk−Mk.\min_{\Delta}\left(\langle E_k,\Delta\rangle+\frac{\rho_{2,k}}{2}\lVert\Delta\rVert_2^2\right),\qquad E_k=G_k-M_k.

Choosing its canonical polar solution gives

Θk+1=Yk+1−∥Ek∥∗ρ2,kMSign⁡(Ek).\Theta_{k+1}=Y_{k+1}-\frac{\lVert E_k\rVert_{\ast}}{\rho_{2,k}}\operatorname{MSign}(E_k).

Thus, under the chosen spectral penalty and local linearization, the residual signal, its independent matrix-sign transformation, and its nuclear-norm weight all follow from the same local subproblem, rather than being separately added heuristic components. Our design intuition is that, compared with directly accepting the tentative momentum step, further using the parameter subproblem containing the objective function allows current objective information to contribute directly to the final update through the residual correction. The correspondence with a Nesterov-style update in the Euclidean case provides concrete inspiration for this approach.

Reviewer_JQFROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AIMuon, a residual-corrected variant of Muon. It retains the standard momentum recursion and separately applies a matrix-sign transformation to the momentum and the gradient–momentum residual. The two directions are combined using nuclear-norm-based relative weights, followed by normalization to match Muon’s update magnitude. The construction is motivated by parameter-subproblem refinement in the AIM framework. The paper establishes a stochastic stationarity bound for the exact, unnormalized iteration and reports improved validation loss and time-to-target performance on GPT-2-small and GPT-2-medium.

Critical strengths

1)A clear and motivated update-level modification. Separately transforming the momentum and residual provides a distinct way to incorporate current-gradient information because the matrix-sign transformation is nonlinear. The AIM derivation offers a coherent motivation, and final magnitude matching reduces the confounding effect of simply increasing the update size.

2)Promising empirical results with relevant component comparisons. AIMuon improves the reported final validation loss in both pretraining settings. The evaluation includes residual-construction ablations, coefficient sensitivity, and a wall-clock comparison that accounts for the additional Newton–Schulz computation. On GPT-2-medium, the reported time to reach Muon’s terminal loss decreases from 8.61 to 8.04 hours.

Critical weaknesses
  1. The benefit of separately transforming the residual remains insufficiently explained and isolated. The residual Et=Gt−MtE_t=G_t-M_t can contain both useful information about changes in the gradient and stochastic fluctuations that momentum is intended to suppress. The convergence analysis accounts for stochastic noise, but does not establish when residual correction improves upon Muon or whether the nuclear-norm weighting distinguishes useful corrections from noise. It also excludes the practical NS5 approximation and final magnitude matching. Empirically, the paper does not vary gradient-noise levels or measure update alignment against an independent reference gradient. Moreover, Muon’s momentum coefficient is fixed at β=0.95\beta=0.95, leaving open whether some benefits could be obtained by retuning momentum. Controlled noise or batch-size experiments, together with tuned momentum and inside-transform baselines, would strengthen the justification.

  2. The empirical evidence is not yet sufficient to establish a reliable advantage over well-tuned baselines. Each main comparison uses only one random seed, so the variability of the relatively small final-loss improvements and reported time savings is unknown. All methods select the largest tested learning rate in both settings, with no larger values evaluated; consequently, the search does not bracket their optima. The ablations also reuse AIMuon’s learning rate without independent retuning, which particularly limits conclusions about variants that change update magnitude. Broader learning-rate searches and repeated runs are needed to establish the robustness of the reported improvements.

Questions for authors

Could you clarify whether the improvements over Muon and MONA persist after extending the learning-rate search beyond the current upper endpoints and repeating the comparisons across multiple seeds? Appendix C states that every compared method selected the largest tested learning rate and that each main comparison uses only seed 42, leaving uncertainty about both baseline tuning and the reliability of the gains. Also, does the current analysis or experimental evidence distinguish useful gradient–momentum corrections from stochastic noise, and establish a benefit beyond retuning Muon’s momentum coefficient?

4 comments
AuthorsOpenReview ↗

W2&Q1 Extended learning-rate searches, momentum tuning, and repeated runs

Thank you for the constructive review and for recognizing the motivation behind the separately transformed residual correction. Your comments prompted us to substantially expand the experimental evaluation. We extended the learning-rate searches, independently tuned momentum coefficients, added a Muon-Nesterov baseline, repeated the GPT-2-small comparisons across three matched seeds, added Qwen3-0.6B pretraining, independently tuned the closest residual controls, and conducted batch-size and correction-strength studies. All updated experiments were run on eight NVIDIA H100 GPUs.

We independently repeated hyperparameter selection for each optimizer using a sequential procedure: first select the peak learning rate, then compare momentum coefficients β∈0.9,0.95\beta\in\\{0.9,0.95\\} at the selected learning rate.

SettingPeak learning-rate candidates, in units of 10−310^{-3}Tokens per candidate in each search stageSelected peak learning rate
GPT-2-small2, 2.5, 3, 3.5, 4, 4.5, 52.48B0.0045
GPT-2-medium1, 2, 3, 4, 4.5, 50.69B0.0045
Qwen3-0.6B1, 2, 3, 3.5, 4, 4.51.2B0.004

These are per-candidate budgets in both the learning-rate and momentum search stages.

The learning-rate searches initially use β=0.95\beta=0.95 for the GPT-2 experiments and β=0.9\beta=0.9 for Qwen3-0.6B. The latter choice follows the GPT-2 momentum-search results. The learning rate is selected independently for each optimizer; all methods select the same value within each model setting. The selected learning rates are now interior points of the tested grids.

The subsequent momentum searches select β=0.9\beta=0.9 for all four optimizers in both GPT-2 settings. For Qwen3-0.6B, AIMuon, Muon, and Muon-Nesterov select β=0.95\beta=0.95, whereas MONA selects β=0.9\beta=0.9. The AdamW fallback coefficients remain unchanged.

AIMuon uses c=0.045c=0.045, 0.04750.0475, and 0.050.05 for GPT-2-small, GPT-2-medium, and Qwen3-0.6B, respectively.

Every candidate starts from scratch with seed 42 and uses a complete warmup and cosine-decay schedule for its allotted budget. Selection is based on terminal validation loss, with matched initialization, data order, and validation data across methods.

With the selected hyperparameters fixed, we then evaluate GPT-2-small using seeds 42, 43, and 44:

OptimizerValidation loss, seed 42Seed 43Seed 44Mean ± standard deviation
Muon3.19803.19743.19843.1980 ± 0.0005
Muon-Nesterov3.20003.19973.19903.1996 ± 0.0005
MONA3.21983.22153.22173.2210 ± 0.0010
AIMuon3.19393.19503.19463.1945 ± 0.0005

AIMuon achieves the lowest validation loss in all three matched seeds. Its reduction in validation loss relative to Muon ranges from approximately 0.0024 to 0.0041 across the paired runs. Seed 42 is also used for hyperparameter selection; seeds 43 and 44 are additional evaluations without further tuning.

We also retrained GPT-2-medium for 6.9B tokens and added Qwen3-0.6B pretraining for 12B tokens:

OptimizerGPT-2-medium validation lossQwen3-0.6B validation loss
Muon3.02552.7859
Muon-Nesterov3.01952.7830
MONA3.03162.7824
AIMuon3.01872.7819

Both settings use seed 42. These results extend the evaluation to larger models and show lower terminal losses after baseline tuning.

Steps to baseline performance. We additionally measure the optimization steps AIMuon requires to reach the final validation losses of the independently tuned Muon-Polyak and Muon-Nesterov baselines. AIMuon reaches both targets in fewer steps across all three models:

ModelStep reduction vs. Muon-PolyakStep reduction vs. Muon-Nesterov
GPT-2-small8.98%10.89%
GPT-2-medium10.05%4.13%
Qwen3-0.6B6.49%3.78%

For GPT-2-small, we first average validation losses across seeds 42, 43, and 44 at each evaluation step, then interpolate the mean curve to locate AIMuon's first crossing of each baseline's final mean validation loss. GPT-2-medium and Qwen3-0.6B use the seed-42 curves with the same interpolation procedure. The reduction is computed as (Tb−TAIMuon→b)/Tb×100(T_b-T_{\mathrm{AIMuon}\to b})/T_b\times100\\%, where TbT_b is the baseline's full training length and TAIMuon→bT_{\mathrm{AIMuon}\to b} is AIMuon's interpolated target-reaching step. All crossings use the original full-budget learning-rate schedules.

AuthorsOpenReview ↗

W1&Q1 Benefit beyond momentum retuning and mixing before transformation

The updated comparisons explicitly address whether the improvement can be recovered by retuning Muon's momentum coefficient. Both ordinary Muon and Muon-Nesterov receive independent learning-rate and momentum searches. For Muon-Nesterov, the input to the spectral transformation is

Nk=βMk+(1−β)Gk,N_k=\beta M_k+(1-\beta)G_k,

while MkM_k remains the ordinary EMA momentum state. AIMuon retains a lower terminal validation loss than both tuned baselines in the three main settings.

We additionally tune the closest residual-construction controls on GPT-2-small:

  • Inside-MSign: apply the spectral transformation to Mk+cEkM_k+cE_k.
  • Linear residual: add the raw residual cEkcE_k to the transformed momentum.

Here, Ek=Gk−MkE_k=G_k-M_k, and both controls retain final magnitude matching.

For each control, we independently search

c∈0.01,0.025,0.05,0.1,0.2c\in\\{0.01,0.025,0.05,0.1,0.2\\}

using 0.248B tokens per candidate. Inside-MSign initially selects the lower endpoint, 0.01, so we additionally evaluate 0.005 under the same protocol; 0.01 remains better. Linear residual selects 0.025.

Both controls independently search learning rates over 4.0,4.5,5.0×10−3\\{4.0,4.5,5.0\\}\times10^{-3}, using 0.248B tokens per candidate, seed 42, and β=0.9\beta=0.9. Both select 4.5×10−34.5\times10^{-3} based on terminal validation loss. Each selected configuration is then restarted and trained for the full 2.48B-token budget using seed 42.

The no-residual variant is AIMuon with c=0c=0, using the independently selected Muon learning rate and momentum coefficient. Its retained Frobenius-norm matching introduces numerical differences from the standalone Muon baseline.

ConfigurationCoefficientFinal validation loss
Full AIMuonc=0.045c=0.0453.1939
Fixed residual weightλfix=0.19\lambda_{\mathrm{fix}}=0.193.1944
Inside-MSignc=0.01c=0.013.1949
No residualc=0c=03.1964
Linear residualc=0.025c=0.0253.1984

Removing the residual gives a final validation loss of 3.1964, compared with 3.1939 for full AIMuon. Inside-MSign improves upon the no-residual variant, while Linear residual does not. AIMuon achieves lower loss than both independently tuned residual controls, supporting the effectiveness of the additional residual orthogonalization.

Purpose of magnitude matching. We follow Muon's RMS-matching convention, which aligns its update scale with the typical AdamW update scale (Liu et al., 2025). We then rescale the combined direction to match the RMS magnitude of the corresponding Muon reference at the same momentum state. With the learning rate held fixed, this aligns the update magnitude and helps distinguish the benefit of the update direction from that of a larger step. AIMuon, Inside-MSign, and Linear residual all retain this matching, so their direction constructions are compared under the same magnitude-matching rule.

Contribution of nuclear-norm weighting. The fixed-weight variant uses 0.19, chosen to approximately match the effective weight around the midpoint of the reference AIMuon run. It achieves a validation loss of 3.1944, close to AIMuon's 3.1939 and below the no-residual variant's 3.1964. Thus, most of the observed improvement is retained with a separately transformed residual at a fixed weight, while adaptive nuclear-norm weighting provides a smaller additional gain.

AuthorsOpenReview ↗

Batch-size(Gradient-noise levels) dependence and sensitivity

The gradient–momentum residual contains both changes in the underlying gradient and stochastic fluctuations. Following your suggestion, we compare AIMuon with Muon and Muon-Nesterov on GPT-2-small using global batch sizes from 128 to 2048.

All configurations use approximately 2.48B tokens, seed 42, learning rate 0.0045, and β=0.9\beta=0.9. For AIMuon, we adjust the correction coefficient at the larger batch sizes, as reported below.

Global batch sizeAIMuon ccAIMuon lossMuon lossMuon-Nesterov loss
1280.053.17693.18313.1828
2560.053.17283.18013.1806
5120.0453.19393.19803.2000
10240.0253.23683.23763.2389
20480.013.31553.31593.3141

AIMuon performs best at batch sizes 128–1024, while its advantage over Muon decreases at larger batch sizes. At batch size 2048, Muon-Nesterov performs best. These observations characterize how the benefit of residual correction depends on the training regime.

We also conducted a separate correction-strength sweep:

Correction coefficient ccFinal validation loss
03.1964
0.0253.1943
0.0453.1939
0.0753.1949
0.13.1962

The c=0c=0 entry uses AIMuon without the residual correction, retaining its Frobenius-norm matching, with a final validation loss of 3.1964. All tested nonzero correction coefficients improve upon this reference, with the lowest validation loss achieved at c=0.045c=0.045. Neighboring coefficients yield similar losses, while increasing correction strength does not monotonically improve performance.

Together, these experiments show how the benefit of residual correction varies with batch size and correction strength under a fixed token budget. Regarding the distinction between useful gradient information and noise, the nuclear-norm ratio determines the relative spectral weight of the two components; it does not explicitly separate signal from noise. The results establish the empirical benefit of the separately transformed residual, while identifying how its interaction with stochastic fluctuations affects optimization remains a question for further study.

References

Jingyuan Liu et al. Muon is Scalable for LLM Training. 2025. https://arxiv.org/abs/2502.16982

AuthorsOpenReview ↗

Follow-up on Additional Experiments and Revisions

Thank you again for your constructive review. We would appreciate your thoughts on whether the additional experiments in our rebuttal address your concerns. To facilitate your assessment, we briefly summarize the main updates.

  1. Extended tuning and multi-seed evaluation. We expanded the learning-rate grids so that the selected rates are interior points, independently tuned momentum coefficients, and added Muon-Nesterov as a baseline. On GPT-2-small, AIMuon achieves the lowest validation loss across all three matched seeds, with a mean of 3.1945 ± 0.0005, compared with 3.1980 ± 0.0005 for Muon. We also extended GPT-2-medium training to 6.9B tokens and added Qwen3-0.6B pretraining for 12B tokens. AIMuon achieves lower final validation losses than the tuned baselines in both settings.

  2. Independently tuned residual controls. We separately tuned the learning rates and correction coefficients for Inside-MSign and Linear residual. AIMuon achieves lower validation loss than both controls under the same final magnitude-matching rule. The fixed-weight ablation retains most of the improvement, suggesting that the separately transformed residual contributes the larger share of the observed gain, with nuclear-norm weighting providing a smaller additional benefit.

  3. Batch-size and correction-strength studies. We evaluated global batch sizes from 128 to 2048. With the reported correction coefficients, AIMuon performs best at batch sizes 128–1024, while Muon-Nesterov performs best at 2048. We also added a correction-strength sweep and clarified that nuclear-norm weighting determines the relative spectral contribution of the residual without explicitly separating useful gradient information from stochastic noise.

We would be grateful for any further feedback on these updates. If any concerns remain or any part of our response needs clarification, we would be happy to address them in further detail.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Responses to Reviewers

We thank all three reviewers for their constructive feedback, which has helped us substantially improve our manuscript. Below we summarize the main clarifications and revisions.

1. Clarifying methodological and conceptual contributions (4UHi, JQFR, and X5jm).

We have clarified AIMuon’s central contribution as implementing AIM’s parameter-update design perspective in spectral geometry. While retaining ordinary EMA momentum, local linearization of the objective-containing parameter subproblem under a squared spectral-norm penalty jointly yields the gradient–momentum residual, its independent matrix-sign transformation, and its nuclear-norm weight. These components arise from the same local optimization problem, providing a unified rationale for the correction. We further clarified the resulting conceptual insights: because the matrix-sign transformation is nonlinear, separately transforming momentum and residual generally differs from mixing them before transformation, distinguishing AIMuon from the corresponding gradient–momentum mixtures. Moreover, although RMS matching removes a common scale, the relative nuclear-norm weights of the two components remain and influence the final direction. AIMuon also distinguishes orthogonalizing individual components from preserving the orthogonal structure of the final update: its combined direction generally does not remain orthogonal, providing a concrete setting for investigating the role of component-wise spectral transformations.

2. Expanded evaluation and targeted ablations (JQFR, X5jm, and 4UHi).

We expanded independent learning-rate and momentum searches, added Muon-Nesterov, repeated GPT-2-small comparisons across three matched seeds, extended GPT-2-medium training, and added Qwen3-0.6B pretraining. AIMuon retains lower final validation losses in these comparisons and reaches the tuned Muon baselines’ final losses in fewer optimization steps. Independently tuned Inside-MSign and Linear residual controls support the benefit of separate residual transformation. The fixed-weight ablation retains most of the improvement, with adaptive nuclear-norm weighting providing a smaller additional gain. Batch-size and correction-strength experiments further characterize the correction’s dependence on the training regime, including its reduced advantage at larger batches. We also clarify that nuclear-norm weighting does not explicitly separate signal from noise.

3. Improved presentation and organization (4UHi).

To improve readability, we reorganized the main text to introduce AIM and the spectral motivation before the derivation and implementation, clarified notation and Nesterov terminology, and added an explicit discussion of the conceptual contributions. These changes are incorporated directly into the revised manuscript.

We thank Reviewer X5jm for their positive reassessment and increased score. We welcome Reviewer JQFR’s feedback on the additional evidence and Reviewer 4UHi’s assessment of the revised exposition, including any specific passages that remain unclear. We hope these revisions assist the reviewers and Area Chair in their final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.