acceptodds
Under review as a conference paper at ICLR 2027

MARS-M: When Variance Reduction Meets Matrices

Abstract

Matrix-based preconditioned optimizers, such as Muon, have recently been shown to be more efficient than scalar-based optimizers for training large-scale neural networks, including large language models (LLMs). Recent benchmark studies of LLM pretraining optimizers have demonstrated that variance-reduction techniques such as MARS can substantially speed up training compared with standard optimizers that do not employ variance reduction. In this paper, we introduce MARS-M, a new optimizer that integrates MARS-style variance reduction with Muon. Under standard regularity conditions, we prove that MARS-M converges to a first-order stationary point at a rate of , improving upon the rate attained by Muon. Empirical results on language modeling and computer vision tasks demonstrate that MARS-M consistently yields lower losses and improved performance across various downstream benchmarks.

Primary area:
optimization
Keywords:
variance reduction, optimizer, matrix-level optimizer

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_MNrEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MARS-M, which adds a MARS-style gradient correction to the Moonlight/Muon update. It presents an exact variant that evaluates gradients at two consecutive parameter points on the same batch, and an approximate variant that reuses the previous batch's gradient to avoid the extra backward pass. The authors analyze both variants, obtaining rates of O~(T−1/3)\widetilde{O}(T^{-1/3}) and O~(T−1/4)\widetilde{O}(T^{-1/4}) under their respective assumptions and parameter schedules. The empirical study includes GPT-2 pretraining on OpenWebText and FineWeb-Edu, downstream evaluations, CIFAR-10 experiments, sensitivity and resource analyses, and repeated small/medium runs.

Critical strengths
  1. The approximate variant is, in my view, the strongest part of the paper. Eqs. (4.4)–(4.5) show precisely how it differs from Moonlight instead of presenting the method as a black-box combination of two optimizers. This interpretation is simple and useful, and it also helps motivate the role of the correction coefficient γ\gamma.
  2. The validation-loss improvements appear fairly consistent across model sizes and both language-model datasets, even though the margins are sometimes small. Appendix H is a meaningful addition: the four-seed OpenWebText results for small and medium models provide more useful evidence than the original single-seed results alone. The γ\gamma-sensitivity and resource analyses also help round out the practical evaluation.
Critical weaknesses

Two details in the main theoretical result need to be reconciled. Theorem 4.3 does not state the condition on γ\gamma used in Lemma C.1, even though the proof relies on that lemma. Also, Algorithm 2 gives M1=(1−β1)g1M_1=(1-\beta_1)g_1, whereas the initial-error estimate in Appendix C (lines 1046–1050) appears to treat M1M_1 as g1g_1. This looks like a local initialization/proof issue rather than a counterexample to the final rate. A clearer distinction would also help between the exact, idealized setting of Theorem 4.3 (zero weight decay and time-varying parameters) and the approximate, fixed-parameter implementation used for LLM training.

Questions for authors
  1. Was corrected-gradient clipping enabled in the reported runs? How were the different weight-decay values for MARS-M and Moonlight selected? Existing tuning details or matched-setting results, if already available, would help clarify the comparison; I am not assuming that both optimizers must use identical weight decay.
  2. Could you provide the complete code, configurations, and commands used for the LLM and Figure 5 experiments. A runnable snapshot corresponding to the reported runs would address most of my reproducibility concern.
  3. Could you state the γ\gamma condition required by Theorem 4.3 and reconcile Algorithm 2's initialization with the initial-error bound in Appendix C?

My hesitation comes mainly from the current difficulty of verifying the experimental setup against the submitted implementation, together with the theoretical clarifications above. Clear answers and a runnable version corresponding to the reported experiments could move my assessment toward acceptance.

4 comments

We thank the reviewer for the careful and constructive assessment. Your detailed questions have been invaluable in helping us refine our work, and here we list our responses to your questions.

Q1: Was corrected-gradient clipping enabled in the reported experiments?

A1: Yes, we use gradient clipping of 1.0 as shown in Sec. 5.1.

Q2: How were different weight-decay values for MARS-M and Moonlight selected? Is the comparison fair?

A2: To make a fair comparison, we use the same λ=0.1 for Moonlight and MARS-M, as suggested in Moonlight's original paper (Liu et al., 2025). And we have added it in Sec. 5.1. Moreover, we also corrected the weight decay value in the sample code for reproducibility.

Q3: Could you provide complete code, configurations, and commands for the LLM and Figure 5 experiments?

A3: We agree that a runnable implementation snapshot is the most direct way to address reproducibility concerns. The current manuscript already specifies the main experimental settings. For LLM training, these include GPT-2 model sizes, context length 1024, total batch size 480, 100,000 training steps, 2,000 warm-up steps, cosine learning-rate scheduling, and β=0.95. For Figure 5, the reported CIFAR-10/ResNet-18 experiments use 200 epochs, batch size 128, β=0.95, and learning-rate reductions at epochs 100 and 150. And more configurations can refer to Sec. 5 and Appendix A. We also updated the code for both LLM and CV experiments.

Q4: What condition on γt\gamma_t is required by Theorem 4.3?

A4: Thank you for pointing out that the current theorem statement does not explicitly specify the coefficient used in Lemma C.1. More importantly, the improved convergence rate does not require the expectation-dependent coefficient appearing in that lemma. We can establish the same O~(T−1/3)\tilde{O}(T^{-1/3}) rate for exact MARS-M using the explicit constant choice$\gamma_t = 1$. To see this, define the estimator error as et=Mt−∇F(Xt).e_t = M_t - \nabla F(X_t). For exact MARS-M with γt+1=1\gamma_{t+1}=1, the momentum recursion becomes Mt+1=βt+1Mt+∇f(Xt+1,ξt+1)−βt+1∇f(Xt,ξt+1)M_{t+1}= \beta_{t+1}M_t+ \nabla f(X_{t+1},\xi_{t+1})- \beta_{t+1}\nabla f(X_t,\xi_{t+1}). Let g(X,ξ)=∇f(X,ξ)g(X,\xi)=\nabla f(X,\xi) and define the centered stochastic term$z_{t+1} = (1-\beta_{t+1})[g(X_{t+1},\xi_{t+1})-\nabla F(X_{t+1})] + \beta_{t+1}[ g(X_{t+1},\xi_{t+1})-g(X_t,\xi_{t+1})-\nabla F(X_{t+1})+\nabla F(X_t)].ThenThene_{t+1}=\beta_{t+1}e_t+z_{t+1}$. Conditioning on the history Ft\mathcal{F}_t before sampling ξt+1\xi_{t+1}, the same-sample gradient correction implies E[zt+1∣Ft]=0.\mathbb{E}[z_{t+1}\mid\mathcal{F}_t]=0. Consequently, the cross term vanishes. Using bounded variance, smoothness, and ∣A+B∣F2≤2∣A∣F2+2∣B∣F2\\|A+B\\|_F^2\leq 2\\|A\\|_F^2+2\\|B\\|_F^2, we obtain

E∣et+1∣F2≤βt+12 E∣et∣F2+2βt+12L2,E∣Xt+1−Xt∣F2+2(1−βt+1)2σ2.\mathbb{E}\\|e_{t+1}\\|_F^2\le \beta_{t+1}^2\,\mathbb{E}\\|e_t\\|_F^2 +2\beta_{t+1}^2L^2,\mathbb{E}\\|X_{t+1}-X_t\\|_F^2 +2(1-\beta_{t+1})^2\sigma^2.

This is the estimator-error bound needed for the subsequent Lyapunov argument in Appendix C, without relying on the expectation-dependent γt+1\gamma_{t+1} or the nonnegative correction term in Lemma C.1. Using the theoretical parameter schedules ηt=(s+t)−2/3,βt+1=1−2ηt,\eta_t=(s+t)^{-2/3},\beta_{t+1}=1-2\eta_t, the same asymptotic convergence rate follows (with the initialization and constants handled consistently):$\frac{1}{T}\sum_{t=1}^{T} \mathbb{E}|\nabla F(X_t)|_F =\tilde{O}(T^{-1/3}).$ We incorporated this direct estimator-error proof into Appendix C and explicitly state γt=1\gamma_t=1 as a sufficient theoretical choice in Theorem 4.3. Thus, the issue concerns the presentation of the coefficient condition rather than the underlying variance-reduction mechanism or the achievable asymptotic convergence rate.

Q5: How can the initialization inconsistency between Algorithm 2 and Appendix C be resolved?

A5: The reviewer is correct that the initial-error estimate in Appendix C does not precisely match the initialization in Algorithm 2. Importantly, this can be corrected without modifying Algorithm 2 or changing the optimizer implementation. Algorithm 2 initializes M0=0M_0=0 and X1=X0X_1=X_0. Therefore, the first gradient correction vanishes, giving$M_1=(1-\beta_1)g_1,g_1=\nabla f(X_1,\xi_1).$ Let ϵ1=g1−∇F(X1)\epsilon_1=g_1-\nabla F(X_1). Then$\nabla F(X_1)-M_1=\beta_1\nabla F(X_1)-(1-\beta_1)\epsilon_1.$ By the unbiasedness of the stochastic gradient, E[ϵ1]=0\mathbb{E}[\epsilon_1]=0. Hence, $\mathbb{E}\|\nabla F(X_1)-M_1\|_F^2 =\beta_1^2\|\nabla F(X_1)\|_F^2+(1-\beta_1)^2\mathbb{E}\|\epsilon_1\|_F^2 \leq\beta_1^2\|\nabla F(X_1)\|_F^2+(1-\beta_1)^2\sigma^2$. Thus, the original initial-error bound σ2\sigma^2 should be replaced by$\beta_1^2\|\nabla F(X_1)\|_F^2 +(1-\beta_1)^2\sigma^2.$ This correction affects the initial term in the Lyapunov analysis. Since X1X_1 is fixed, the additional quantity ∣∇F(X1)∣F2\\|\nabla F(X_1)\\|_F^2 is independent of the training horizon T. Consequently, it changes the constant in the final bound without changing the O~(T−1/3)\tilde{O}(T^{-1/3}) asymptotic rate, once the corresponding Lyapunov constants are updated consistently. We will update the initial Lyapunov estimate and the corresponding constants in Appendix C. This is a local correction to the initialization bound, not a counterexample to the convergence rate and not a reason to modify the proposed optimizer.

Q6: How should the exact theoretical result be distinguished from the approximate LLM implementation?

A6: The exact convergence analysis and the practical approximation address two distinct but closely related aspects of MARS-M. The exact formulation is designed to exploit same-sample stochastic gradient correction. This mechanism permits the estimator-error recursion to benefit from the correlation between gradients evaluated at consecutive iterates on the same sample. The theoretical guarantee is established under the specified learning-rate and momentum schedules, exact polar updates, and zero weight decay. The practical approximation replaces the same-sample gradient difference with gradients from consecutive minibatches, avoiding the additional gradient evaluation. Appendix D provides a separate convergence analysis for this approximation under explicit decaying parameter schedules. The LLM experiments use fixed momentum parameters, finite Newton–Schulz iterations, and nonzero weight decay. We do not conflate the exact theorem with a guarantee covering every practical implementation detail. However, the exact and approximate methods share a common gradient-correction construction. The exact estimator establishes the theoretical value of variance-reduced matrix optimization, while the approximation provides a computationally efficient practical variant. The empirical effectiveness of the approximation is evaluated independently through training loss, validation loss, and downstream benchmarks.

Reviewer_MNrEOpenReview ↗

Rebuttal assessment and updated rating

Thank you for the clarifications. The explicit γ=1\gamma=1 argument and corrected initialization bound address my main theoretical concerns, and the weight-decay explanation is helpful. Please clarify whether clipping applies to raw gradients, corrected gradients, or both, and reconcile this with Section 4.1 and the updated code. Overall, the response sufficiently addresses my main concerns, and I am raising my rating from 2 to 3.

Thank you for your acknowledgement of our work!

For the clipping, we clip the raw gradients. And we remove the clipping on ctc_t in the main Algorithm in Section 4.1 since our log shows that clipping on ctc_t never takes effect during all our experiments, partially because we have already applied clipping to raw gradients. We have modified the code to align with Section 4.1.

Thank you again for your careful inspection.

Reviewer_qUdqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MARS-M, which adds MARS-style variance reduction (a STORM gradient correction scaled by a small γ) to the Moonlight variant of Muon, i.e. Muon with the 0.2·√max(m,n) update scaling and decoupled weight decay. The authors also give an approximate version that reuses the previous step's gradient, which avoids a second gradient computation. They show this version is equivalent to Moonlight with re-weighted momentum coefficients. Under bounded variance and per-sample L-smoothness, they prove the exact MARS-M reaches a first-order stationary point at Õ(T^{-1/3}), compared with the O(T^{-1/4}) known for Muon; the approximate version only gets Õ(T^{-1/4}) (Appendix D). Experiments train GPT-2 models from 125M to 1.5B parameters on OpenWebText and FineWeb-Edu 100B for about 50B tokens, all using the approximate version. MARS-M gives slightly lower training and validation loss than Moonlight and AdamW, and small gains on zero- and two-shot downstream benchmarks. Additional results cover ResNet-18 on CIFAR-10, an ablation on γ, a toy convergence experiment, compute and memory overhead, and 3-seed repeats for the small and medium models.

Critical strengths

1. Small but consistent gains over a tuned Moonlight baseline at negligible cost. Across GPT-2 models from 125M to 1.5B on two datasets, MARS-M ends with slightly lower training and validation loss than Moonlight, whose learning rate was tuned by grid search. The 3-seed repeats in Appendix H show the gap is larger than the seed variation, e.g. validation loss 2.644 ± 0.004 vs. 2.631 ± 0.004 for the medium model. The method uses about the same memory and wall-clock time as Moonlight, and results are flat across γ ∈ [0.005, 0.025], so it is easy to try. Whatever the underlying mechanism turns out to be, a cheap change that reliably lowers loss for a widely used optimizer is useful to practitioners, and this is the main point in the paper's favor.

2. A convergence analysis for variance-reduced Muon-type updates. The paper extends the STORM/MARS analysis to orthogonalized updates and shows an Õ(T^{-1/3}) rate for the exact version, along with an Õ(T^{-1/4}) rate for the approximate version. Section 4.2 also writes the approximate version explicitly as Moonlight with modified momentum coefficients, which makes the relationship between the two methods transparent. This adds to the still-thin theory of the Muon family. It carries less weight in my assessment than it might, because the proof largely reuses the MARS lemmas and covers a variant that the LLM experiments do not use.

Critical weaknesses

1. The LLM gains cannot be attributed to variance reduction. All LLM experiments use approximate MARS-M, whose update is M_t = βM_{t−1} + (1−β)g_t + γβ(g_t − g_{t−1}) (Eq. 4.2). By Eq. 4.6, Moonlight is M_t = βM_{t−1} + g_t + β(g_t − g_{t−1}). Newton–Schulz normalizes its input, so the overall scale of M_t does not matter. It follows that Moonlight is exactly approximate MARS-M with γ = 1−β = 0.05. The recommended γ = 0.01 therefore just lowers the Nesterov look-ahead weight from 0.95 to about 0.19, and the γ sweeps in Fig. 6 and Fig. 15 are sweeps over this momentum coefficient, with Moonlight sitting between γ = 0.025 and γ = 0.1. Fig. 14 even includes γ = 0, which is plain EMA momentum Muon with no correction term at all. Its curve is hard to distinguish from γ = 0.01–0.025. The comparison that would isolate variance reduction is missing: Moonlight with a tuned momentum or look-ahead coefficient, or Muon without Nesterov. Since the paper's central claim is that variance reduction helps matrix optimizers, this is the main reason I lean toward rejection.

2. The theoretical improvement does not apply to the method that is evaluated. The Õ(T^{-1/3}) rate holds only for exact MARS-M. That version needs two gradient evaluations per step and is used only in a single CIFAR-10 run and a GPT-2 small run, with no compute-matched comparison. The approximate version used for the LLM experiments gets Õ(T^{-1/4}) (Appendix D), the same as Muon. Even for the exact version, Theorem 4.3 depends on the data-dependent γ from Lemma C.1, which involves unknown expectations and is required for the M̃ term to be non-negative. The constant γ used in practice is not covered. The faster rate also relies on per-sample smoothness, a stronger assumption than the cited Muon analyses use. The toy experiment does not show a rate difference either: in Fig. 13, Muon's empirical slope (−0.409) matches MARS-M's (−0.408). The rate improvement over Muon is advertised in the abstract and contributions, so this gap undermines a second main claim of the paper.

Questions for authors

Clarifications and additional results that could change my assessment

  1. Relation to Moonlight. Newton–Schulz normalizes its input, so the overall scale of M_t has no effect on the update. Do you agree that, by Eq. 4.2 and Eq. 4.6, Moonlight is exactly approximate MARS-M with γ = 1−β (0.05 for β = 0.95)? If so, how do you separate the effect of variance reduction from that of a smaller Nesterov look-ahead coefficient? Two baselines would answer this:

    • Moonlight with its look-ahead coefficient tuned.
    • Muon with plain EMA momentum and no Nesterov, i.e. γ = 0.

    Please run both under the LLM setup with multiple seeds. Fig. 14 already contains γ = 0 for the medium model; could you report its final losses next to γ = 0.01 and Moonlight?

  2. Exact vs. approximate at LLM scale. Fig. 7 includes MARS-M-exact on GPT-2 small. Can you report its final losses, and compare it to approximate MARS-M and Moonlight at matched compute? The exact version needs a second gradient evaluation per step. If exact MARS-M does not beat a tuned approximate version at equal FLOPs, the case for variance reduction as the source of the gains becomes much weaker.

  3. The γ in Theorem 4.3. Which γ_t does the theorem assume? As written, M̃ ≥ 0 seems to require the data-dependent γ from Lemma C.1, which involves unknown expectations. Can the Õ(T^{-1/3}) rate be shown for a constant γ, or for an explicit schedule like the one in Appendix D? Is per-sample smoothness necessary for the result, or only convenient?

  4. Table 11. The PIQA score for MARS-M (γ = 0.01) is 74.80, while every other method is around 63, and this entry largely drives the higher average. Is this a typo? Please also give per-task numbers for the 3-seed averages in Appendix H, so it is clear whether the ~1 point gain over Moonlight there holds across tasks.

3 comments

We thank the reviewer for the detailed technical analysis. Here are our responses to your questions.

Q1: Is Moonlight exactly approximate MARS-M with γ=1-β? Does this imply that the observed gains cannot be attributed to variance reduction?

A1: We agree with the algebraic equivalence, but it is important to distinguish this identity from the broader conclusion drawn from it. Approximate MARS-M satisfies$M_t=\beta M_{t-1}+(1-\beta)g_t+\gamma\beta(g_t-g_{t-1}),whileMoonlightcanbeexpressedaswhile Moonlight can be expressed asM_t^{\mathrm{Moonlight}} =\beta M_{t-1}^{\mathrm{Moonlight}} +g_t+\beta(g_t-g_{t-1}).Consequently,whenγ=1−β,Consequently, when γ=1-β,M_t^{\mathrm{MARS-M}}=(1-\beta)M_t^{\mathrm{Moonlight}},underconsistentinitialization.Exactpolarorthogonalizationisinvariantunderpositivescalarmultiplication,sotheresultingdirectionscoincide.However,thisidentityhastwoimportantimplications.First,approximateMARS−MdecouplestheEMAmemorycoefficientfromthegradient−differencecoefficient.Considerthegeneralizedrecursionunder consistent initialization. Exact polar orthogonalization is invariant under positive scalar multiplication, so the resulting directions coincide. However, this identity has two important implications. First, approximate MARS-M decouples the EMA memory coefficient from the gradient-difference coefficient. Consider the generalized recursion\widehat M_t=\beta\widehat M_{t-1} +g_t+\kappa(g_t-g_{t-1}).$ Standard Moonlight restricts κ=β, whereas approximate MARS-M permits κ=γβ/(1-β). Thus, varying γ while keeping β fixed explores momentum mixtures that cannot be obtained simply by changing the standard Moonlight momentum coefficient without also changing the EMA timescale. For β=0.95, γ=0.01 gives κ=0.19, compared with κ=0.95 for standard Moonlight. If Moonlight is extended to tune κ independently of β, it becomes precisely the generalized family studied through our γ ablations. Therefore, the requested generalized Moonlight mixture is not a fundamentally separate algorithm, but an equivalent parameterization of the practical update family considered in this work. Second, the equivalence concerns approximate MARS-M, not the exact variance-reduced estimator analyzed in Theorem 4.3. Exact MARS-M uses the same-sample gradient difference g(Xt,ξt)−g(Xt−1,ξt),g(X_t,\xi_t)-g(X_{t-1},\xi_t), whereas the practical approximation uses g(Xt,ξt)−g(Xt−1,ξt−1).g(X_t,\xi_t)-g(X_{t-1},\xi_{t-1}).These estimators have different stochastic properties. In particular, under mean-squared smoothness,Eξ∥g(Xt,ξ)−g(Xt−1,ξ)∥F2≤L2∥Xt−Xt−1∥F2.\mathbb E_\xi\|g(X_t,\xi)-g(X_{t-1},\xi)\|_F^2\leq L^2\|X_t-X_{t-1}\|_F^2. This bound is central to the exact variance-reduction analysis. It does not generally hold for differences involving independent samples. Therefore, the algebraic equivalence of the approximation does not invalidate the variance-reduction mechanism of the exact method. We updated the paper and stated this equivalence explicitly in Appendix I.2 and clarify the respective roles of the exact estimator and its computationally efficient approximation.

Q2: What about Muon without Nesterov correction (γ=0) and Moonlight with a tuned mixture? Could you report their final losses and provide multiple-seed results?

A2: Both comparisons are already represented, at least in part, by the existing ablation experiments. Figure 6 includes the ablation on γ on GPT-2 small models. And Figure 14 explicitly includes γ=0 for GPT-2 medium models, corresponding to EMA momentum without the gradient-difference term. Figure 15 explores a wider range of coefficients, including γ=0.05, the Moonlight-equivalent choice for β=0.95. The closeness of the curves for γ=0 and small positive values is not inconsistent with our findings. It indicates that optimization performance is relatively insensitive to small changes in the mixture coefficient within this region, and that a range of small correction strengths can achieve competitive performance. And we have tried to alternate the look-ahead coefficient and the default βlook−ahead=0.95\beta_{look-ahead}=0.95 is optimal. Moreover, for repeated multi-seed experiment results, the final validation losses are: Moonlight 2.849±0.0032.849\pm0.003, MARS-M 2.840±0.0052.840\pm0.005 and MARS-M (γ=0) 2.841±0.0052.841\pm0.005 for GPT-2 small experiments. Therefore, the empirical results are not supported solely by a single γ-sweep curve. For exact value for Figure 14, the training losses for γ=0, γ=0.01 and Moonlight are 2.6116, 2.6119 and 2.6284 (smoothed). And the validation losses are 2.6292, 2.6291 and 2.6435.

Q3: The O~(T−1/3)\tilde{O}(T^{-1/3}) convergence rate depends on an unknown, data-dependent γt\gamma_t. Can the same rate be established for a constant γ?

A3: Yes. We thank the reviewer for raising this important theoretical question. We have strengthened our convergence analysis by establishing the same O~(T−1/3)\tilde{O}(T^{-1/3}) rate for exact MARS-M with the explicit constant choice γt≡1\gamma_t\equiv 1, without relying on the expectation-dependent coefficient in the original Lemma C.1. A complete, self-contained proof has been added in Appendix J. The choice γ=1 is not an artificial parameter introduced solely for the analysis. It is the full-correction choice associated with the STORM-style same-sample gradient estimator. Its significance is that the gradient-difference correction cancels a conditional drift term in the estimator-error recursion. The principal rate improvement over the previously established O~(T−1/4)\tilde{O}(T^{-1/4}) Muon bound is preserved for an explicit, time-independent correction coefficient. In particular, the improved rate does not rely on unavailable expectations or on a data-dependent selection of γt\gamma_t. The revised argument also permits a weaker smoothness assumption. Rather than requiring the stochastic objective to be L-smooth for every individual sample, it is sufficient to assume mean-squared smoothness: $\mathbb{E}_{\xi} \|\nabla f(X,\xi)-\nabla f(Y,\xi)\|_F^2 \le L^2\|X-Y\|_F^2,\forall X,Y.$ By Jensen's inequality, this assumption also implies that the population objective has an L-Lipschitz gradient:$\|\nabla F(X)-\nabla F(Y)\|_F \leq L\|X-Y\|_F.$ Therefore the improved convergence bound holds under a weaker sufficient smoothness condition than the one originally stated.

For completeness, Appendix J also proves a convergence guarantee for any fixed finite γ under an alternative explicit momentum and learning-rate schedule. This general result yields a sufficient O~(T−1/4)\tilde{O}(T^{-1/4}) bound for that broader parameter family; it is not a lower bound and does not establish that the convergence rate for other fixed coefficients cannot be faster. The main O~(T−1/3)\tilde{O}(T^{-1/3}) theoretical guarantee is fully retained through the concrete choice γ=1. We emphasize that the result concerns the exact, same-sample MARS-M estimator under the theorem's parameter schedules. As with other stochastic-optimization convergence theorems, the guarantee is asserted for a specified admissible parameter choice rather than for every possible hyperparameter value. The practical approximate LLM implementation has different settings. In summary, Appendix J removes the expectation-dependent coefficient from the main convergence claim, supplies an explicit constant-γ proof of the improved rate, and relaxes the smoothness requirement. We believe this directly addresses the reviewer's theoretical concern while strengthening the original result.

Q4: Is the per-sample smoothness assumption necessary, or can the convergence analysis use a weaker condition?

A4: The per-sample smoothness assumption can indeed be relaxed. In fact, our new convergence analysis in Appendix J establishes the same O~(T−1/3)\tilde O(T^{-1/3}) rate for exact MARS-M with the constant choice γ=1 under a weaker mean-squared smoothness assumption: $\mathbb E_\xi \|\nabla f(X,\xi)-\nabla f(Y,\xi)\|_F^2\le L^2|X-Y|_F^2$. This condition is strictly weaker than the per-sample smoothness assumption (Assumption 4.2) used in the original manuscript. The key observation is that the estimator-error analysis only requires control of the expected squared difference between stochastic gradients evaluated on the same sample. Mean-squared smoothness directly provides this control, without requiring every individual sample function to be L-smooth. Furthermore, Jensen's inequality implies ∣∇F(X)−∇F(Y)∣F2=∣Eξ[∇f(X,ξ)−∇f(Y,ξ)]∣F2≤Eξ∣∇f(X,ξ)−∇f(Y,ξ)∣F2≤L2∣X−Y∣F2\\|\nabla F(X)-\nabla F(Y)\\|_F^2=\\|\mathbb E_\xi[\nabla f(X,\xi)-\nabla f(Y,\xi)]\\|_F^2\le\mathbb E_\xi\\|\nabla f(X,\xi)-\nabla f(Y,\xi)\\|_F^2\le L^2\\|X-Y\\|_F^2, which also guarantees the population-objective smoothness required by the descent analysis. Therefore, as established in Appendix J, the improved O~(T−1/3)\tilde O(T^{-1/3}) convergence guarantee can be obtained under bounded variance and mean-squared smoothness, without requiring the stronger per-sample smoothness assumption. This further strengthens our theoretical result: the faster convergence rate is preserved with an explicit constant correction coefficient γ=1 and under a weaker smoothness condition than originally stated.

Q5: Exact MARS-M requires an additional gradient evaluation. Does this eliminate its convergence-rate advantage under equal computational cost?

A5: The additional gradient evaluation changes computational constants, but it does not eliminate the improved asymptotic stochastic first-order oracle complexity. Let N denote the total number of stochastic gradient evaluations. Exact MARS-M requires two gradient evaluations per iteration, so N≈2T. Therefore, O~((N/2){−1/3})=O~(N{−1/3})\tilde O((N/2)^\{-1/3\})=\tilde O(N^\{-1/3\}). Thus, even after accounting for the additional gradient evaluation, the exact method retains its improved convergence exponent in terms of stochastic first-order oracle calls. This is a fundamental theoretical distinction from the previously established O~(N−1/4)\tilde O(N^{-1/4}) Muon bounds. Moreover, such improvement on convergence rate implies enhancement in data efficiency, which is crucial in some specific scenarios, such as tasks with rare training samples and industrial LLM training with abundant compute but limited high-quality data. We agree that stochastic first-order oracle complexity and actual GPU wall-clock efficiency are different. So we also investigate approximate MARS-M. It avoids the second gradient evaluation while maintaining the practical correction-based momentum structure. Figure 7 already compares exact MARS-M, approximate MARS-M, and Moonlight on GPT-2 small. Figure 5 provides an additional exact-versus-approximate comparison on CIFAR-10/ResNet-18, where the exact formulation achieves favorable test-loss and accuracy results under the reported protocol. For practical training efficiency, Appendix G reports 411 minutes for Moonlight and 423 minutes for approximate MARS-M on GPT-2 small, corresponding to approximately 2.9% additional wall-clock time.

Q6: Figure 13 shows nearly identical empirical slopes for Muon and MARS-M. Does this contradict the theoretical convergence improvement?

A6: No. Theorem 4.3 establishes a worst-case convergence upper bound, not a prediction of the exact empirical slope on every optimization problem. An optimizer with a proven O~(T−1/4)\tilde O(T^{-1/4}) upper bound may converge faster than T−1/4T^{-1/4} on a particular problem instance. Therefore, observing similar empirical slopes of approximately -0.409 for Muon and -0.408 for MARS-M does not contradict either theoretical guarantee. Moreover, Figure 13 describes finite-horizon behavior on a synthetic matrix-valued optimization problem. Such an experiment cannot determine the worst-case asymptotic convergence exponent of an optimizer. The theoretical distinction follows from the estimator-error recursion and Lyapunov analysis, rather than requiring empirical rate separation on every benchmark.

Q7: The PIQA score for MARS-M (γ = 0.01) is 74.80, while every other method is around 63, and this entry largely drives the higher average. Is this a typo? Please also give per-task numbers for the 3-seed averages in Appendix H, so it is clear whether the ~1 point gain over Moonlight there holds across tasks.?

A: Thank you for noting this issue. We have double-checked the evaluation results of lm-evaluation-harness. And the 0-shot PIQA values for GPT-2 small models on the OpenWebText for all the 4 seeds are mistaken by the PICA values of different seeds by another project in progress since they have the leading values ending with similar suffices (74.80 v.s. 65.80). We are so grateful for the reviewer to notify this mistake, and we have changed the wrong value in the paper. Here we report the original values and the corrected values. The new average results are 41.40±0.13, which is slightly smaller than Moonlight. And we have changed the statement in Appendix H.

GPT-2 smallARC-CARC-EHellaswagMMLUOpenBookQAPIQASciQWGAvg.
AdamW22.8941.2132.5722.8927.5062.8966.6551.7941.05
Moonlight23.4441.1833.3323.0530.0064.2064.5051.6241.42
MARS-M23.2741.0133.2722.9628.9064.6065.7551.4441.40
GPT-2 MediumARC-CARC-EHellaswagMMLUOpenBookQAPIQASciQWGAvg.
AdamW23.9843.4537.5722.5627.1065.9868.1052.3242.63
Moonlight25.1945.1940.0823.2331.3066.2370.9052.9444.38
MARS-M24.9544.9840.1823.3132.8066.5169.0053.7544.44
Reviewer_JnVaOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors incorporate MARS-style gradient correction into Moonlight Muon, exploring both an exact two-gradient scheme and a cheaper approximation that caches previous gradients. They establish theoretical stationarity guarantees for both variants, demonstrating a faster theoretical convergence rate for the exact method and a relaxed bound for the approximation. Experiments confirm its performance on ResNet-18 on CIFAR-10, as well as GPT-2 pretraining (125M to 1.5B parameters) across OpenWebText, FineWeb-Edu, and downstream language tasks.

Critical strengths
  1. The practical gains look fairly consistent. The language-model experiments span several sizes and two datasets, rather than one favorable training run.

  2. The approximate connection is useful, and the added cost is measured.

Critical weaknesses
  1. Clarify the novelty and source of gains. The proposed update largely overlaps with MARS-Shampoo using Moonlight-style scaling and weight decay. To decouple whether gains stem from the new formulation rather than existing heuristics, please provide controlled comparisons against: (1) MARS-Shampoo with matched scaling/decay, and (2) Moonlight with a tuned momentum mixture.

  2. Bridge the theory-to-practice gap. The headline convergence rate applies only to the exact method, not the evaluated approximation. Moreover, the theory assumes ideal conditions (exact polar factors, time-varying expectation-dependent gamma, no weight decay) that diverge from the empirical setup (finite Newton-Schulz, fixed gamma, weight decay). Please explicitly state which theoretical guarantees transfer to the practical optimizer.

Questions for authors

Could you compare the practical method with a Moonlight baseline that tunes the direct-gradient/momentum mixture, and with MARS-Shampoo using the same update scaling, weight decay, and tuning budget?

4 comments

We thank the reviewer for the constructive assessment. We provide detailed responses to your questions below.

Q1: What is the novelty beyond MARS-Shampoo with Moonlight-style scaling and weight decay?

A1: The central contribution of MARS-M is not the introduction of a new polar decomposition operator or weight-decay mechanism. Actually, it is the integration of MARS-style gradient correction with practical Muon/Moonlight optimization, together with a convergence analysis establishing an improved rate for the exact matrix-based method. There are two distinct contributions. First, exact MARS-M integrates same-sample stochastic gradient correction into the Moonlight framework. Therefore, the estimator-error recursion can be controlled to establish an O~(T−1/3)\tilde O(T^{-1/3}) convergence rate, improving upon the previously established O~(T−1/4)\tilde O(T^{-1/4}) bound for Muon under the corresponding analytical assumptions. Second, the practical approximation provides independent control over the historical momentum and gradient-difference contributions. As explained below, this introduces a useful degree of freedom that cannot be achieved by tuning only the standard Moonlight momentum coefficient without simultaneously changing the EMA timescale. The update scaling and weight decay are inherited from Moonlight, as already acknowledged in Section 4.1, where MARS-M is explicitly introduced as an instantiation of MARS tailored to Moonlight. These components ensure compatibility with practical LLM training configurations; they are not claimed as new preconditioning mechanisms. Thus, the novelty lies in the gradient-estimation framework, its convergence analysis, and the independently tunable practical parameterization.

Q2: Why not compare against MARS-Shampoo with exactly matched scaling, weight decay, and tuning budget?

A2: Such a comparison can be characterized mathematically, which explains why an additional large-scale training run is not necessary to establish the relationship. Consider MARS-Shampoo and exact MARS-M using the same corrected-gradient estimator MtM_t. MARS-Shampoo employs the SVD-based direction Ot=UtVt⊤,Mt=UtΣtVt⊤.O_t=U_tV_t^\top,M_t=U_t\Sigma_tV_t^\top. MARS-M uses the corresponding polar factor, approximated through Newton–Schulz iterations in practice. If the corrected-gradient estimator, momentum recursion, clipping convention, polar computation, scaling, weight decay, and learning-rate schedule are matched, the two formulations produce identical update directions and, with exact polar factors, identical parameter updates. Therefore, a fully matched MARS-Shampoo baseline is not a fundamentally different optimization algorithm under these conditions. Any remaining difference would arise from numerical approximation or implementation details. The original MARS-Shampoo configuration and practical MARS-M nevertheless differ in their integration with Moonlight-style scaling and weight decay. These choices are important for practical LLM training, as discussed in Section 4.1. We will make this equivalence explicit in the revised manuscript. This provides a more informative comparison than treating mathematically equivalent implementations as independent optimizers.

Q3: Could you compare against Moonlight with a tuned direct-gradient/momentum mixture?

A3: Our existing γ ablations already explore precisely this generalized momentum-mixture family. Approximate MARS-M satisfies Mt=βMt−1+(1−β)gt+γβ(gt−gt−1)M_t=\beta M_{t-1}+(1-\beta)g_t+\gamma\beta(g_t-g_{t-1}). Defining M^t=Mt/(1−β)\widehat M_t=M_t/(1-\beta), we obtain M^t=βM^t−1+gt+κ(gt−gt−1)\widehat M_t=\beta\widehat M_{t-1}+g_t+\kappa(g_t-g_{t-1}), κ=γβ/(1-β). Standard Moonlight corresponds to κ=β, equivalently γ=1-β. The important distinction is that standard Moonlight couples the EMA memory coefficient and look-ahead coefficient, whereas approximate MARS-M allows these two contributions to be adjusted independently. For instance, with β=0.95, our setting γ=0.01 corresponds to κ=0.19, while standard Moonlight corresponds to κ=0.95. Simply reducing Moonlight's momentum coefficient to 0.19 would also shorten its EMA memory timescale, and therefore would not reproduce the MARS-M configuration with β=0.95 and κ=0.19. If Moonlight is generalized to include an independently tunable look-ahead coefficient, it becomes mathematically equivalent to the approximate MARS-M parameterization. Thus, our γ-sweep experiments directly investigate the requested generalized momentum-mixture family. In particular, Figure 14 already includes γ=0, corresponding to plain EMA momentum without look-ahead correction, while Figure 15 includes γ=0.05, which is Moonlight-equivalent for β=0.95. These ablations show that smaller gradient-difference coefficients can be competitive or preferable to the standard Moonlight choice. Furthermore, Appendix H reports repeated-run results on GPT-2 small and medium. The final validation losses are: Moonlight 2.849±0.0032.849\pm0.003, MARS-M 2.840±0.0052.840\pm0.005 for GPT-2 small and Moonlight 2.644±0.0042.644\pm0.004, MARS-M 2.631±0.0042.631\pm0.004 for GPT-2 medium. These results support the practical effectiveness of the selected momentum mixture beyond a single run.

Q4: Which theoretical guarantees transfer to the practical MARS-M optimizer?

A4: The exact and approximate formulations address complementary theoretical and practical objectives, and their guarantees should be distinguished.

The O~(T−1/3)\tilde O(T^{-1/3}) rate concerns exact MARS-M with same-sample stochastic gradient correction. The practical approximation eliminates the additional gradient evaluation and therefore follows a different estimator-error recursion. Importantly, the improved rate for exact MARS-M does not fundamentally require the expectation-dependent γt\gamma_t appearing in Lemma C.1. A direct estimator-error argument works with the explicit constant choice γt=1\gamma_t=1. Let et=Mt−∇F(Xt)e_t=M_t-\nabla F(X_t). Under bounded variance and smoothness, the exact estimator satisfies E∣et+1∣F2≤βt+12E∣et∣F2+2(1−βt+1)2σ2+2βt+12L2E∣Xt+1−Xt∣F2\mathbb E\\|e_{t+1}\\|_F^2\leq\beta_{t+1}^2\mathbb E\\|e_t\\|_F^2+2(1-\beta_{t+1})^2\sigma^2+2\beta_{t+1}^2L^2\mathbb E\\|X_{t+1}-X_t\\|_F^2. This provides the estimator-error control required by the Lyapunov argument in Appendix C, without dependence on unknown expectations. With the theoretical schedules ηt=(s+t)−2/3\eta_t=(s+t)^{-2/3} and βt+1=1−2ηt\beta_{t+1}=1-2\eta_t, the analysis yields the same O~(T−1/3)\tilde O(T^{-1/3}) asymptotic rate. We will present this simpler derivation in the revised proof, with the initialization and constants handled consistently. For approximate MARS-M, Appendix D separately establishes an O~(T−1/4)\tilde O(T^{-1/4}) guarantee under its theoretical parameter schedules. The practical fixed-parameter implementation additionally employs finite Newton–Schulz iterations and weight decay. We do not claim that the exact theorem automatically applies to every such implementation detail. However, this distinction does not diminish the theoretical contribution: exact MARS-M establishes a faster convergence-rate upper bound through same-sample variance reduction, while approximate MARS-M provides a computationally efficient practical variant with independently tunable momentum contributions. We thank the reviewer again. We believe these mathematical clarifications make the contributions and their relationship to existing optimizers more precise.

Thank you for the response and revisions. Appendix J, which proves the Õ(T^{-1/3}) rate with an explicit constant γ = 1 under mean-squared smoothness, addresses my question about the data-dependent γ, and I appreciate the transparent PIQA correction. However, my main concern is confirmed rather than resolved: the new results show that γ = 0 (no gradient-difference term at all) matches γ = 0.01 (2.841 vs. 2.840 on GPT-2 small, 2.6292 vs. 2.6291 on medium), both better than Moonlight. This suggests that the gain over Moonlight comes from a smaller Nesterov look-ahead rather than from variance reduction, and Theorem J.4 likewise gives the recommended γ the same Õ(T^{-1/4}) rate as γ = 0. I would ask that the abstract, Section 5, the conclusion and the downstream claims be revised to match this evidence. Given the strengthened theory and the useful practical observation, I keep my score of 3.

We thank the reviewer for carefully examining our revised manuscript and the additional experiments. We appreciate your acknowledgment that Appendix J addresses the concern regarding the data-dependent correction coefficient, as well as your recognition of the corrected PIQA results.

We agree that the new γ=0\gamma=0 ablation provides an important clarification regarding the practical behavior of approximate MARS-M. The nearly identical losses for γ=0\gamma=0 and γ=0.01\gamma=0.01 indicate that reducing the Nesterov-style look-ahead contribution in MARS-M framework is an important factor in the practical performance gains. At the same time, we would like to clarify that this observation does not invalidate the variance-reduction motivation or theoretical contribution of MARS-M. Our framework contains two closely related but distinct formulations. The exact MARS-M formulation is designed around same-sample variance-reduced gradient estimation. As established in Appendix J, it admits an O~(T−1/3)\tilde O(T^{-1/3}) convergence guarantee with the explicit constant choice γ=1\gamma=1 under the stated assumptions. This remains a central theoretical contribution of our work. The approximate formulation is introduced to eliminate the additional gradient evaluation required by the exact estimator. Through this approximation, the update acquires an equivalent generalized Moonlight representation, as formally established in Appendix I. Consequently, the practical approximation provides an additional degree of freedom for adjusting momentum memory and Nesterov look-ahead contributions. The new γ=0\gamma=0 results help clarify this second aspect of our framework. They suggest that the attenuation of the look-ahead contribution is sufficient to explain the observed improvements over default Moonlight in the tested LLM settings. We agree that these improvements should not be attributed uniquely to variance reduction, and we will revise the empirical interpretation accordingly. Importantly, we view this as a refinement of the relationship between the exact and approximate formulations, rather than a reversal of the original contribution. The central objective of MARS-M remains to investigate variance reduction in matrix-based optimization. Our theoretical analysis establishes the convergence benefits of the exact variance-reduced estimator, while the practical experiments additionally identify an effective momentum-mixture design arising from its computationally efficient approximation. Following your suggestion, we will clarify this distinction in the abstract, Section 5, conclusion and downstream claims in Appendix F.

We appreciate that the additional experiments have helped identify the respective roles of variance-reduced gradient estimation and Nesterov look-ahead adjustment. We believe this provides a more complete understanding of the proposed framework while preserving its main theoretical contribution.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of changes in the revised PDF

We have uploaded a revised PDF. New and changed text is in red. The title, the section structure and the theorem numbering are unchanged, and the main text stays within 10 pages.

Main text

  • Abstract: we add the description on approximated MARS-M and added contribution of Nesterov look-ahead reparametrization
  • Sec. 1: we separate the properties of exact MARS-M and approximated version; and added the contribution of Nesterov look-ahead reparametrization
  • Thm. 4.3: we fix the description of Thm. 4.3
  • Sec. 5.1: we add the weight decay value
  • Sec. 5.4: we add the analysis of synergy of 2 parts
  • Sec. 6: we add the conclusion of synergy

Appendix

  • Table 11&Sec. H: we correct the misleading results
  • Sec. C: we correct the proof
  • Sec. F: we rephrase the claim on the results of robustness of γ
  • Sec. I: we add discussions on relation to MARS-Shampoo and derivation of equivalence to generalized Moonlight
  • Sec. J: we add proof of the convergence under constant γ
  • we updated AI use statement

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.