acceptodds
Under review as a conference paper at ICLR 2027

Curvature-Conditioned Multiscale Momentum with Sphere Constraints for LLM Pretraining

Abstract

Pretraining accounts for a large fraction of the total computational cost in LLM training. However, noise-dominant gradients and the highly ill-conditioned loss landscape bring severe challenges. Although modern adaptive optimizers such as AdamW and Muon have achieved great success in large-scale pretraining, their reliance on gradient normalization offers limited mitigation of the ill-conditioned curvature. The progress along flat directions (eigen-directions of small eigenvalues), which dominates the final loss reduction, remains relatively slow. To enhance training dynamics along flat directions, we propose a curvature-conditioned multiscale momentum method with sphere constraints, delivering steady acceleration in LLM pretraining. This multiscale momentum, applied only along flat directions, pairs a slow-decay component for noise reduction with a fast-decay component for rapid curvature adaptation, harnessing their complementary strengths. Crucially, we employ a sphere-constraint technique to prevent parameter inflation and effective learning rates collapse that would otherwise arise from a naive combination. Extensive experiments show that our proposed method significantly accelerates Muon across diverse architectures (dense, MoE) and model sizes (0.12B–5.1B parameters). Theoretically, we verify the acceleration effect and provide insight into the design principles underlying the flat-direction multiscale momentum.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Optimization, LLM pre-training

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_JwCqOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MuonM for language model pretraining. It combines fast and slow momentum along estimated flat directions with sphere constraints, and tests the method on dense and mixture-of-experts models. The experiments compare several optimizers and individual components across model sizes and training lengths, and report measured step time.

Critical strengths
  • Gains across model sizes and training lengths support value beyond one small setting.
  • The sphere-only baseline helps separate the effects of sphere constraints and the added momentum design.
  • Measured step time shows about 0.9% overhead for the 5.1B mixture-of-experts model, supporting practical use.
Critical weaknesses
  • Appendix B.2 supports the direction proxy with a small example. The author can check curvature directly on a trained model and strengthen the evidence for the mechanism.
  • Appendix B.3 reports step time, but total time to reach a target loss would make the speed gains easier to judge in practice.
  • The paper notes that the best learning-rate schedule under sphere constraints remains an open question. The authors should provide brief practical guidance based on the tested schedules.
Questions for authors

Please respond to the concerns raised in Critical Weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer JwCq

We appreciate the reviewer for the valuable comments and recognition to our work. Our response are as follows. Should there be any additional questions, we are more than willing to provide further clarification.

W1. Appendix B.2 supports the direction proxy with a small example. The author can check curvature directly on a trained model and strengthen the evidence for the mechanism.

Re1: Thanks for the constructive comment. We evaluate how well our sharp-direction projection estimate covers the top subspace of the Hessian (i.e., the sharp directions) in the 0.12B dense model (dmodel=768d_{\text{model}}=768) trained with MuonM in our experiments. Equivalently, this assesses the extent to which the orthogonal complement (i.e., the flat-direction projection produced by MuonM) lies within the Hessian flat directions. Details are provided in Appendix B.2, and the main results are summarized here for brevity.

For $W \in \{ W_{\text{up}}, W_{\text{gate}}, W_{\text{down}}, W_{\text{q}}, W_{\text{k}}, W_{\text{v}}, W_{\text{o}}\} $ in the second transformer layer, using a batch size of 32 and a sequence length of 1024, we estimate the top subspace of the Hessian using restarted Lanczos methods, an efficient randomized linear algebra technique that does not require computing the full Hessian matrix. Figure 11 (a) in Appendix B.2 (Pages 16-17) shows that throughout training, for each transformer block, the top-2000 subspace (sharp directions) of the Hessian exhibits rapid eigenvalue decay, concentrating sharp curvature into a low-dimensional subspace. Figure 11 (b) demonstrates that the top-2000 sharp subspace is well covered by our sharp-direction estimation (i.e., the top 50-200 left or right eigenspaces of the gradient), with most blocks exhibiting a coverage score of at least 0.8 on at least one side. Note that our sharp-direction subspace is formed by the product projection of both the left and right top eigenspaces. Thus, substantial coverage by the top eigenspace on either side alone is sufficient to demonstrate the reliability of our sharp-direction projection.

W2: Appendix B.3 reports step time, but total time to reach a target loss would make the speed gains easier to judge in practice.

Re2: Thanks for the suggestion. We have added the loss-time plots in Figure 12 (Appendix B.3, page 18). Given the nearly negligible additional time overhead of MuonM, its loss advantage is also maintained with respect to training time.

W3: The paper notes that the best learning-rate schedule under sphere constraints remains an open question. The authors should provide brief practical guidance based on the tested schedules.

Re3: Thanks for the comment. In this study, we primarily focus on cosine decay for spherical-constraint methods and find that it performs better than the WSD schedule in our experimental settings (e.g., the 0.6B MoE experiments in the left panel of Fig. 7 and in Fig. 14), which is also consistent with [1]. Therefore, cosine decay or a similar-shaped decay is an appropriate choice for spherical-constraint methods.

References:

[1] Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors, arXiv preprint arXiv:2606.25971, 2026.

Reviewer_7nWROpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MuonM adds slow momentum in estimated flat directions to Muon, using sphere-constrained weight directions, learnable scales and momentum transport. The paper evaluates dense and MoE pretraining on FineWeb-Edu and analyzes selective multiscale momentum in linear regression with a power-law spectrum.

Critical strengths
  • MuonM improves terminal loss over Muon and MuonS across the tested dense and MoE configurations, including a 0.019 gain over MuonS on Dense-0.12B. Shared momentum settings and the sensitivity study in Fig. 14 suggest modest additional tuning. The measured step-time overhead is 0.9% on the 5.1B MoE setup.

  • The linear-model analysis separates signal learning from noise accumulation in discrete time and characterizes how selective slow momentum changes the optimally tuned loss estimate.

Critical weaknesses
  • The contribution of curvature filtering is not fully isolated. Figure 4 compares flat-only with sharp-only slow momentum; the unrestricted comparison in Fig. 2(d) is limited to a toy problem. A matched LLM comparison with slow momentum throughout the tangent space, retaining the same sphere parameterization, normalization and parallel transport, would clarify whether filtering adds value beyond MuonS plus a slow EMA.

  • A comparison with the closely related method of Zhu et al. (2026) is missing from Fig. 5 and Appendix D. Its observations motivate the subspace estimator (App. B.2, Fig. 11), making it a relevant baseline for assessing the benefit over existing flat-direction methods.

Questions for authors
  • How does MuonM perform when the flat-subspace filter is replaced by tangent-space projection alone, with comparable tuning of the learning rate and slow-momentum strength?

  • Does Appendix C.2 report 1−β31-\beta_3 rather than β3\beta_3 for AdEMAMix? What retention coefficient was used under the convention in Eq. (22)?

  • How does MuonM compare with Muon-LITE (Zhu et al., 2026), the closest cited flat-direction method, under a matched training budget?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 7nWR

We appreciate the reviewers' valuable comments and recognition of our work. Our point‑by‑point response is as follows. Should there be any additional questions, we're more than willing to provide further clarification.

  • W1/Q1. Thanks for the constructive comment. We compare MuonM with a variant that applies multiscale momentum in the full space (i.e., Qk=I\mathcal{Q}_k=I in MuonM). For this variant, we tune the learning rate lr∈0.0035,0.0025\mathrm{lr}\in\\{0.0035,0.0025\\} and the slow momentum strength χ∈0.2,0.1\chi\in\\{0.2,0.1\\}, and the best result is obtained with (lr,χ)=(0.0035,0.1)(\mathrm{lr},\chi)=(0.0035,0.1). This variant exhibits a higher final validation loss of 2.046 (versus 2.042 of MuonM) and a marked degradation in downstream task performance relative to MuonM (the following table), demonstrating the benefit of applying multiscale momentum only along the flat directions.

    Downstream evaluation of MoE-5.1B (A-0.47B). All metrics are accuracy, and higher is better.

    Nameboolqhellaswagwinograndearc_challengeopenbookqasocial_iqa`
    Shot01052500
    MuonM65.5768.3463.3852.0544.4042.99
    MuonM (full space)64.89 (-0.68)68.17 (-0.17)62.51 (-0.87)51.11 (-0.94)43.80 (-0.60)42.99 (±0.00)
  • Q2. We thank the reviewer for the careful reading. The notation β3\beta_3 for AdEMAMix in the [Comparing Different Optimizers on the 0.12B Dense Model] part of Appendix C.2 is a typo. To align with the convention in Appendix~E, we have corrected it as follows:

    For AdEMAMix, we tuned β3∈0.99,0.999,0.9999\beta_3 \in \\{0.99, 0.999, 0.9999\\} and α∈1,3,5,8\alpha \in \\{1, 3, 5, 8\\}, obtaining the best configuration (lr,β3,α)=(3×10−3,0.999,5.0)(\mathrm{lr}, \beta_3, \alpha) = (3\times10^{-3}, 0.999, 5.0).

  • W2/Q3. Thanks for the constructive suggestion. We have compared Muon-LITE (applying LITE to standard Muon with weight decay) and MuonS-LITE (applying LITE to MuonS, i.e., the sphere-constrained Muon), tuning both the learning rates and LITE's flat-direction acceleration hyperparameters (χ∈2,4,β2∈0.5,1\chi\in\\{2,4\\}, \beta_2\in\\{0.5,1\\}) . We report the best-performing configurations: (lr,χ,β2)=(0.005,2,0.5)(\text{lr}, \chi, \beta_2) = (0.005, 2, 0.5) for Muon-LITE and (lr,χ,β2)=(0.01,2,0.5)(\text{lr}, \chi, \beta_2) = (0.01, 2, 0.5) for MuonS-LITE.

    Using terminal validation loss as the metric, Muon-LITE lowers the loss from 3.041 (Muon) to 3.033, a reduction of 0.008, while MuonS-LITE lowers it from 3.033 (MuonS) to 3.028, a reduction of 0.005. In contrast, MuonM lowers the terminal loss from 3.033 (MuonS) to 3.013, a substantially larger reduction of about 0.02. Thus, in both the standard weight-decay and sphere-constrained settings, the loss reduction achieved by LITE is smaller than that achieved by MuonM relative to MuonS. We attribute LITE's relatively smaller gain to its simultaneous amplification of gradient noise along flat directions (as discussed in Lines 44–46 of the Introduction), whereas MuonM suppresses this noise.

    In addition, the magnitude of LITE's loss reduction in our experiments is smaller than that reported in the original work [1]. We attribute this discrepancy primarily to the numerical precision setup: we adopt a more standard mixed-precision training recipe, maintaining FP32 master weights while running forward and backward passes in BF16 (except for numerically sensitive operations such as softmax), whereas the settings in [1] follow GaLore [2] in using pure BF16 training (with BF16 master weights as well). Storing master weights in BF16 leads to severe low-precision update swamping when the learning rate decays to small values late in training, which can alter the empirical acceleration effect of LITE.

References:

[1] Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancement, arXiv preprint arXiv:2602.22681, 2026.

[2] GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection, ICML 2024.

Reviewer_bkdLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MuonM, a curvature-conditioned multiscale momentum optimizer with sphere constraints for large language model (LLM) pretraining. The method aims to accelerate optimization along flat directions, where slow progress and stochastic gradient noise can limit training efficiency. It maintains fast- and slow-decay momentum components and selectively incorporates slow momentum into estimated flat directions, while using sphere constraints, tangent-space projection, and parallel transport to prevent parameter norm inflation and effective learning-rate collapse. The proposed method estimates the sharp subspaces online using the leading singular subspaces of the fast momentum matrix, treating their orthogonal complement as the flat-direction subspace. The algorithm combines this projection mechanism with a learnable-radius sphere parameterization and manifold-aware momentum updates. The authors evaluate MuonM on the FineWeb-Edu dataset using dense models ranging from 0.12B to 1.4B parameters and mixture-of-experts (MoE) models ranging from 0.6B to 5.1B parameters. The experiments include comparisons with Muon, the sphere-constrained MuonS baseline, several alternative optimizers, component ablations, different learning-rate schedules, and extended training budgets. The authors report consistently lower terminal validation loss than Muon and MuonS, with approximately 0.9% additional per-step runtime in the 5.1B MoE experiment. A theoretical analysis based on a linear-regression proxy and the Functional Scaling Law framework is also provided to explain the acceleration mechanism. Overall, the paper presents a technically plausible and empirically promising optimizer with encouraging scaling results. However, the evidence for the accuracy of the curvature-related subspace estimation, the statistical robustness of the reported improvements, and the applicability of the theoretical analysis to the complete MuonM algorithm could be strengthened.

Critical strengths

1.A coherent combination of multiscale momentum and sphere-constrained optimization. The method addresses two related problems: reducing gradient noise through slow momentum in estimated flat directions and preventing parameter norm inflation through sphere constraints. The use of tangent-space projection and parallel transport provides a principled mechanism for maintaining momentum across changing tangent spaces. This combination is the central methodological strength of the paper. 2.Encouraging empirical results across multiple model scales and architectures. The evaluation includes dense and MoE models up to 5.1B parameters, comparisons with Muon and MuonS, component ablations, extended training budgets, and alternative learning-rate schedules. The reported improvements in validation loss, together with the approximately 0.9% per-step runtime overhead at the largest MoE scale, suggest that the proposed method may improve optimization efficiency without introducing substantial computational cost.

Critical weaknesses

1.Insufficient direct validation of the curvature-conditioned subspace estimation and incomplete theoretical coverage. The proposed method relies on the assumption that the leading singular subspaces of the fast momentum matrix provide a useful approximation to sharp directions of the loss landscape. Since this approximation determines where the slow momentum is applied, its accuracy is central to the algorithm's rationale. Direct validation of the estimated subspaces, sensitivity analyses for the sharp-subspace ratio, or stronger empirical evidence linking the projection to actual curvature would make the central mechanism more convincing. In addition, the theoretical results rely on a simplified linear-regression model with a prescribed projection and do not directly analyze the complete sphere-constrained MuonM algorithm. This limits the extent to which the theoretical analysis can substantiate the method's overall claims. 2.Limited evidence regarding experimental robustness and comparative advantage. The main results are encouraging, but the manuscript does not clearly establish variability across independent training runs or provide uncertainty estimates for the reported loss differences. Moreover, comparisons with several alternative optimizers are conducted primarily at the 0.12B scale, leaving their relative performance at larger scales less clear. Since pretraining outcomes can be sensitive to initialization and optimization hyperparameters, repeated evaluations on manageable model sizes and more comprehensive comparisons under controlled tuning budgets would strengthen the evidence that the improvements are robust rather than configuration-dependent.

Questions for authors

1.[Additional validation request] Reliability of the curvature-conditioned projection: Can the authors provide quantitative evidence that the estimated leading singular subspaces of the fast momentum matrix correspond to sharp directions of the loss landscape? In particular, how sensitive are the results to the selected sharp-subspace ratio, and does the method retain its advantage when this ratio is varied? This evidence would help establish whether the proposed curvature-conditioned mechanism is essential to the observed improvements. 2.[Clarification and additional validation request] Statistical robustness and theoretical scope: Were the principal comparisons evaluated across multiple independent random seeds, and can the authors report the corresponding variability in terminal validation loss? In addition, could the authors clarify which properties of the complete MuonM algorithm are theoretically supported by the linear-regression analysis, given that the theoretical projection is prescribed while the practical algorithm estimates it online and incorporates sphere constraints and parallel transport? These clarifications would materially affect my assessment of the robustness and theoretical support of the proposed method.

5 comments
AuthorsOpenReview ↗

Response to Reviewer bkdL (part 1)

We greatly appreciate the reviewers' valuable comments and recognition of our work. Our point‑by‑point response is as follows. Should there be any additional questions, we're more than willing to provide further clarification.

  • W1/Q1. [Reliability of the flat-direction projection] Thanks for the constructive suggestion. We evaluate how well our sharp-direction projection estimate covers the top subspace of the Hessian (i.e., the sharp directions) in the 0.12B dense model (dmodel=768d_{\text{model}}=768) trained with MuonM in our experiments. Equivalently, this assesses the extent to which the orthogonal complement (i.e., the flat-direction projection produced by MuonM) lies within the Hessian flat directions. Details are provided in Appendix B.2, and the main results are summarized here for brevity.

    For $W \in \{ W_{\text{up}}, W_{\text{gate}}, W_{\text{down}}, W_{\text{q}}, W_{\text{k}}, W_{\text{v}}, W_{\text{o}}\} $ in the second transformer layer, using a batch size of 32 and a sequence length of 1024, we estimate the top subspace of the Hessian using restarted Lanczos methods, an efficient randomized linear algebra technique that does not require computing the full Hessian matrix. Figure 11 (a) in Appendix B.2 (Pages 16-17) shows that throughout training, for each transformer block, the top-2000 subspace (sharp directions) of the Hessian exhibits rapid eigenvalue decay, concentrating sharp curvature into a low-dimensional subspace. Figure 11 (b) demonstrates that the top-2000 sharp subspace is well covered by our sharp-direction estimation (i.e., the top 50-200 left or right eigenspaces of the gradient), with most blocks exhibiting a coverage score of at least 0.8 on at least one side. Note that our sharp-direction subspace is formed by the product projection of both the left and right top eigenspaces. Thus, substantial coverage by the top eigenspace on either side alone is sufficient to demonstrate the reliability of our sharp-direction projection.

  • W2: [Comparison with other baselines at larger scale] We thank the reviewer for the comment. We add experiments comparing AdeMaMix, EMA-Nesterov, MuonS and MuonM (ours) on the 0.6B dense model, with their learning rates tuned (see details in Fig 17 of Appendix D). The results in the following table show that MuonM also achieves the lowest loss and significantly outperforms the other methods, consistent with the 0.12B model experiments.

    Final validation losses on Dense-0.6B. Numbers in parentheses denote the absolute loss gap relative to MuonM.

    OptimizerMuonMuonSMuonM (ours)AdEMAMixEMA-Nesterov
    Dense-0.6B2.508 (+0.031)2.495 (+0.018)2.4772.514 (+0.037)2.500 (+0.023)
  • W2/Q2: [Statistical robustness] Thanks for the suggestion. The loss variation across seeds is negligible relative to the reduction achieved by MuonM. The table below reports MuonM's results on the 120M dense model across four random seeds, showing that seed-induced loss differences (consistently below 0.001) remain well below the 0.028 loss reduction achieved by MuonM over Muon and are therefore negligible. In addition, we evaluate the influence of the critical hyperparameters for flat-direction acceleration in MuonM in the [Hyper-parameters Sensitivity] section of Appendix D. The results show that MuonM is relatively robust to these hyperparameters within a reasonable range.

    Terminal validation losses of MuonM across 5 random seeds on the 120M dense model.

    Seed12345mean ± std
    MuonM3.0127573.0126723.0123623.0127373.0127893.0126634 ± 0.0001738
AuthorsOpenReview ↗

Response to Reviewer bkdL (part 2)

  • W1/Q2: [Theoretical scope] We thank the reviewer for the insightful comment. Given the extreme complexity of LLMs, an absolutely precise analysis of training dynamics is unrealistic. Recent studies [1,2] show that linear regression models with certain structures that capture the core characteristics of LLMs can approximate ground-truth training curves surprisingly well. Motivated by this, we adopt this simple yet useful linear regression model with a power-law Hessian spectrum (Assumption 7.1) and Hessian-aligned gradient noise (Lemma I.2), which can also be observed in real LLMs, as a proxy model to approximate LLM training dynamics.

    Given that the linear model has a constant Hessian, the corresponding flat directions are fixed. Regarding the missing sphere constraint in the theoretical model, we clarify that the sphere constraint and the associated parallel transport are technical operations used to avoid norm inflation and the resulting effective learning rate collapse. These issues arise from the scale invariance and scale redundancy specific to LLM architectures, which do not appear in our simplified theoretical model. Moreover, the goal of multiscale momentum is to reduce noise in updates along flat directions, so our theoretical analysis omits the sphere-constraint technical operations and focuses mainly on this noise-reduction role. Finally, we note that our theoretical prediction of the loss trajectory closely matches the empirical loss curves (Figs. 9 and 18), indicating that our theoretical analysis captures the dominant factors underlying LLM training dynamics and that the omission of the sphere constraint does not materially degrade the fidelity of the theoretical approximation.

References:

[1] A Defense of the Quadratic Model. arXiv preprint arXiv:2607.21716, 2026.

[2] Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedules, Neurips 2025.

I thank the authors for their detailed response and additional experiments. The new results on the 0.6B model are helpful. However, I believe further comparisons across different model scales and clarification of the hyperparameter tuning budgets for each baseline would strengthen the evidence for MuonM's advantages over other optimizers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.