- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MARS-M, which adds a MARS-style gradient correction to the Moonlight/Muon update. It presents an exact variant that evaluates gradients at two consecutive parameter points on the same batch, and an approximate variant that reuses the previous batch's gradient to avoid the extra backward pass. The authors analyze both variants, obtaining rates of and under their respective assumptions and parameter schedules. The empirical study includes GPT-2 pretraining on OpenWebText and FineWeb-Edu, downstream evaluations, CIFAR-10 experiments, sensitivity and resource analyses, and repeated small/medium runs.
Critical strengths
- The approximate variant is, in my view, the strongest part of the paper. Eqs. (4.4)–(4.5) show precisely how it differs from Moonlight instead of presenting the method as a black-box combination of two optimizers. This interpretation is simple and useful, and it also helps motivate the role of the correction coefficient .
- The validation-loss improvements appear fairly consistent across model sizes and both language-model datasets, even though the margins are sometimes small. Appendix H is a meaningful addition: the four-seed OpenWebText results for small and medium models provide more useful evidence than the original single-seed results alone. The -sensitivity and resource analyses also help round out the practical evaluation.
Critical weaknesses
Two details in the main theoretical result need to be reconciled. Theorem 4.3 does not state the condition on used in Lemma C.1, even though the proof relies on that lemma. Also, Algorithm 2 gives , whereas the initial-error estimate in Appendix C (lines 1046–1050) appears to treat as . This looks like a local initialization/proof issue rather than a counterexample to the final rate. A clearer distinction would also help between the exact, idealized setting of Theorem 4.3 (zero weight decay and time-varying parameters) and the approximate, fixed-parameter implementation used for LLM training.
Questions for authors
- Was corrected-gradient clipping enabled in the reported runs? How were the different weight-decay values for MARS-M and Moonlight selected? Existing tuning details or matched-setting results, if already available, would help clarify the comparison; I am not assuming that both optimizers must use identical weight decay.
- Could you provide the complete code, configurations, and commands used for the LLM and Figure 5 experiments. A runnable snapshot corresponding to the reported runs would address most of my reproducibility concern.
- Could you state the condition required by Theorem 4.3 and reconcile Algorithm 2's initialization with the initial-error bound in Appendix C?
My hesitation comes mainly from the current difficulty of verifying the experimental setup against the submitted implementation, together with the theoretical clarifications above. Clear answers and a runnable version corresponding to the reported experiments could move my assessment toward acceptance.
4 comments
Q5: How can the initialization inconsistency between Algorithm 2 and Appendix C be resolved?
A5: The reviewer is correct that the initial-error estimate in Appendix C does not precisely match the initialization in Algorithm 2. Importantly, this can be corrected without modifying Algorithm 2 or changing the optimizer implementation. Algorithm 2 initializes and . Therefore, the first gradient correction vanishes, giving$M_1=(1-\beta_1)g_1,g_1=\nabla f(X_1,\xi_1).$ Let . Then$\nabla F(X_1)-M_1=\beta_1\nabla F(X_1)-(1-\beta_1)\epsilon_1.$ By the unbiasedness of the stochastic gradient, . Hence, $\mathbb{E}\|\nabla F(X_1)-M_1\|_F^2 =\beta_1^2\|\nabla F(X_1)\|_F^2+(1-\beta_1)^2\mathbb{E}\|\epsilon_1\|_F^2 \leq\beta_1^2\|\nabla F(X_1)\|_F^2+(1-\beta_1)^2\sigma^2$. Thus, the original initial-error bound should be replaced by$\beta_1^2\|\nabla F(X_1)\|_F^2 +(1-\beta_1)^2\sigma^2.$ This correction affects the initial term in the Lyapunov analysis. Since is fixed, the additional quantity is independent of the training horizon T. Consequently, it changes the constant in the final bound without changing the asymptotic rate, once the corresponding Lyapunov constants are updated consistently. We will update the initial Lyapunov estimate and the corresponding constants in Appendix C. This is a local correction to the initialization bound, not a counterexample to the convergence rate and not a reason to modify the proposed optimizer.
Q6: How should the exact theoretical result be distinguished from the approximate LLM implementation?
A6: The exact convergence analysis and the practical approximation address two distinct but closely related aspects of MARS-M. The exact formulation is designed to exploit same-sample stochastic gradient correction. This mechanism permits the estimator-error recursion to benefit from the correlation between gradients evaluated at consecutive iterates on the same sample. The theoretical guarantee is established under the specified learning-rate and momentum schedules, exact polar updates, and zero weight decay. The practical approximation replaces the same-sample gradient difference with gradients from consecutive minibatches, avoiding the additional gradient evaluation. Appendix D provides a separate convergence analysis for this approximation under explicit decaying parameter schedules. The LLM experiments use fixed momentum parameters, finite Newton–Schulz iterations, and nonzero weight decay. We do not conflate the exact theorem with a guarantee covering every practical implementation detail. However, the exact and approximate methods share a common gradient-correction construction. The exact estimator establishes the theoretical value of variance-reduced matrix optimization, while the approximation provides a computationally efficient practical variant. The empirical effectiveness of the approximation is evaluated independently through training loss, validation loss, and downstream benchmarks.
Rebuttal assessment and updated rating
Thank you for the clarifications. The explicit argument and corrected initialization bound address my main theoretical concerns, and the weight-decay explanation is helpful. Please clarify whether clipping applies to raw gradients, corrected gradients, or both, and reconcile this with Section 4.1 and the updated code. Overall, the response sufficiently addresses my main concerns, and I am raising my rating from 2 to 3.
Thank you for your acknowledgement of our work!
For the clipping, we clip the raw gradients. And we remove the clipping on in the main Algorithm in Section 4.1 since our log shows that clipping on never takes effect during all our experiments, partially because we have already applied clipping to raw gradients. We have modified the code to align with Section 4.1.
Thank you again for your careful inspection.
We thank the reviewer for the careful and constructive assessment. Your detailed questions have been invaluable in helping us refine our work, and here we list our responses to your questions.
A1: Yes, we use gradient clipping of 1.0 as shown in Sec. 5.1.
A2: To make a fair comparison, we use the same λ=0.1 for Moonlight and MARS-M, as suggested in Moonlight's original paper (Liu et al., 2025). And we have added it in Sec. 5.1. Moreover, we also corrected the weight decay value in the sample code for reproducibility.
A3: We agree that a runnable implementation snapshot is the most direct way to address reproducibility concerns. The current manuscript already specifies the main experimental settings. For LLM training, these include GPT-2 model sizes, context length 1024, total batch size 480, 100,000 training steps, 2,000 warm-up steps, cosine learning-rate scheduling, and β=0.95. For Figure 5, the reported CIFAR-10/ResNet-18 experiments use 200 epochs, batch size 128, β=0.95, and learning-rate reductions at epochs 100 and 150. And more configurations can refer to Sec. 5 and Appendix A. We also updated the code for both LLM and CV experiments.
A4: Thank you for pointing out that the current theorem statement does not explicitly specify the coefficient used in Lemma C.1. More importantly, the improved convergence rate does not require the expectation-dependent coefficient appearing in that lemma. We can establish the same O~(T−1/3) rate for exact MARS-M using the explicit constant choice$\gamma_t = 1$. To see this, define the estimator error as et=Mt−∇F(Xt). For exact MARS-M with γt+1=1, the momentum recursion becomes Mt+1=βt+1Mt+∇f(Xt+1,ξt+1)−βt+1∇f(Xt,ξt+1). Let g(X,ξ)=∇f(X,ξ) and define the centered stochastic term$z_{t+1} = (1-\beta_{t+1})[g(X_{t+1},\xi_{t+1})-\nabla F(X_{t+1})] + \beta_{t+1}[ g(X_{t+1},\xi_{t+1})-g(X_t,\xi_{t+1})-\nabla F(X_{t+1})+\nabla F(X_t)].Thene_{t+1}=\beta_{t+1}e_t+z_{t+1}$. Conditioning on the history Ft before sampling ξt+1, the same-sample gradient correction implies E[zt+1∣Ft]=0. Consequently, the cross term vanishes. Using bounded variance, smoothness, and ∣A+B∣F2≤2∣A∣F2+2∣B∣F2, we obtain
E∣et+1∣F2≤βt+12E∣et∣F2+2βt+12L2,E∣Xt+1−Xt∣F2+2(1−βt+1)2σ2.
This is the estimator-error bound needed for the subsequent Lyapunov argument in Appendix C, without relying on the expectation-dependent γt+1 or the nonnegative correction term in Lemma C.1. Using the theoretical parameter schedules ηt=(s+t)−2/3,βt+1=1−2ηt, the same asymptotic convergence rate follows (with the initialization and constants handled consistently):$\frac{1}{T}\sum_{t=1}^{T} \mathbb{E}|\nabla F(X_t)|_F =\tilde{O}(T^{-1/3}).$ We incorporated this direct estimator-error proof into Appendix C and explicitly state γt=1 as a sufficient theoretical choice in Theorem 4.3. Thus, the issue concerns the presentation of the coefficient condition rather than the underlying variance-reduction mechanism or the achievable asymptotic convergence rate.