acceptodds
Under review as a conference paper at ICLR 2027

Hybrid Momentum Methods for Optimizing Deep Learning Models

Abstract

Momentum is central to stochastic optimization in deep learning, including the training of large language models. Classical momentum methods typically rely on a fast exponential moving average (EMA) scheme, which discounts past gradients on a fixed timescale and may underweight older gradients that remain informative. This creates a trade-off between retaining long-term history and adapting to recent changes. We introduce Hybrid Momentum (HyM), which combines a linearly weighted average of past gradients with a geometrically discounted fast momentum. The former assigns weights proportional to the gradients' iteration indices and retains a growing span of history, while the latter emphasizes recent gradients. This hybrid construction is modular and can be combined with adaptive preconditioning and other update transformations. In particular, we propose HyMuon, which applies Newton–Schulz orthogonalization to the hybrid direction. Illustrative examples and controlled experiments demonstrate the benefits of combining gradient information across these two timescales. Across image classification, machine translation, and language modeling, HyM and HyMuon match or improve upon strong baselines. HyMuon improves over Muon on IWSLT14, and in both the 100k- and 600k-step GPT-2 124M OpenWebText regimes. For LLaMA-1.3B pretraining on C4, HyMuon achieves a final validation loss of , compared with for Muon.

Primary area:
optimization
Keywords:
Deep learning optimization, Momentum methods, Large language model pretraining, Multi-timescale optimization, Gradient averaging

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_812oOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The proposed

Critical strengths

The paper is well structured, begining with method, experiment and theory.

Critical weaknesses
  1. Limited methodological novelty. Under reparameterization, HyM is merely just schedule-free method. The only difference is replacing the direct current-gradient contribution with fast momentum. However, the manuscript does not sufficiently establish the significance of this incremental modification.

  2. The motivation does not isolate the proposed contribution. Section 4 primarily investigates the usefulness of retaining historical gradients, a property already present in Schedule-Free, which is totally not novel.

  3. The most relevant component ablation is missing. Appendix F.3 compares the combined method against fast-only and slow-only variants, but omits the same slow-history component combined with a raw current gradient. Consequently, it does not isolate the principal modification relative to Schedule-Free for most experimental setting. In Figure 4, it performs worse than Schedule-Free.

  4. Most of ablations are problematic. In LLM abaltion, HyMuon changes matrix momentum, auxiliary optimization, and parameter averaging simultaneously, without sufficient controls separating their effects. The headline C4 comparison additionally tunes HyMuon while retaining Muon’s imported configuration, leaving the advantage over a comparably tuned baseline unresolved. On FineWeb, the reported results are close to AdEMAMix, providing limited evidence of improvement over this relevant baseline.

  5. The convergence analysis has limited explanatory relevance. The theorem establishes an O(T−1/2)O(T^{-1/2}) guarantee by treating HyM as a perturbation of Schedule-Free. Anway, this is just the same as other optimizing method, yielding no benefit at all.

The paper also seems to be heavily AI written, many of places lose essential logic. There are also quite a lot of references that do not appear in the paper: NorMuon, Dion, Mano, etc.

Questions for authors

Please answer my questions in Critical Weakness.

7 comments
AuthorsOpenReview ↗

Author Response to Reviewer 812o (1/4: [W1, W2])

Author Response to Reviewer 812o (1/4: [W1, W2])

Part 1 covers W1–W2. Part 2 covers W3 and W4(a); Part 3, W4(b–c); Part 4, W5 and writing/references.

We thank the reviewer for recognizing the overall organization of the paper and for the comments on its contribution and evidence. Below, we clarify the relation to Schedule-Free, explain the roles of the mechanism studies and theory, and present the additional component controls and baseline tuning. We also address the presentation and citation concerns. We have highlighted the key changes in blue in the revised manuscript.

1. “Limited methodological novelty. Under reparameterization, HyM is merely just schedule-free method. The only difference is replacing the direct current-gradient contribution with fast momentum. However, the manuscript does not sufficiently establish the significance of this incremental modification.”

Ans: Thank you for asking us to clarify the methodological contribution. We respectfully disagree with the characterization of HyM as merely a reparameterization of Schedule-Free. The distinction is in the weights assigned to past gradients and in how the resulting momentum is used in the matrix update.

First, HyM changes both the recent-gradient contribution and the scaling of the slow history. For Schedule-Free SGD with a constant step size and ct=1/tc_t=1/t, define nt=t−1tnt−1+gtn_t=\frac{t-1}{t}n_{t-1}+g_t. Its update direction and that of HyM are

\qquad d_t^{\mathrm{HyM}}=\frac{\beta q_t}{t}n_t+(1-\beta)b_t,$$ where $q_t=(1-\mu^t)/(1-\mu)$ and $b_t=\mu\frac{t-1}{t}b_{t-1}+g_t$. Thus, the two methods share a slow history state, but generally assign different weights to past gradients. Second, HyMuon combines the slow and fast gradient momenta before applying Newton–Schulz (NS) orthogonalization. Applying the parameter averaging and interpolation of Schedule-Free around a Muon update would instead act on a direction that has already been orthogonalized. Because NS is nonlinear, these constructions generally produce different updates. The momentum formulation therefore provides a direct way to combine the two histories within matrix orthogonalization. Third, the raw gradient and tuned EMA controls test the choices of fast momentum and growing history, respectively. HyM and HyMuon perform better in these comparisons; see Answers 3 and 4. Together with the gradient weight analysis and convergence guarantee, these results support the proposed momentum construction. **2. “The motivation does not isolate the proposed contribution. Section 4 primarily investigates the usefulness of retaining historical gradients, a property already present in Schedule-Free, which is totally not novel.”** **Ans:** Thank you for asking us to clarify the relation between the mechanism studies and the methodological contribution. Retaining gradient information over a long history is also a feature of Schedule-Free. HyM's methodological contribution lies in the specific weighting and combination of the slow history and fast momentum. HyMuon further applies matrix orthogonalization to the combined direction. Section 4 tests whether retained gradient information remains useful during subsequent training, providing a mechanistic explanation for this design. The purpose of Keep/Delete/Refresh is to examine this mechanism, not to demonstrate better overall training performance. The methodological differences are described in Answer 1. The raw gradient and tuned EMA controls separately test the choices of fast and slow components; see Answers 3 and 4. To further test whether retained gradient information remains useful, we repeated Keep/Delete/Refresh on two additional target batches and included Muon as a comparison. Each target is injected at update 5,000; within each optimizer, the three branches share the checkpoint at update 7,500 and continue to update 10,000. Their final target losses are: | Method | Additional target batch | Delete | Keep | Refresh | |---|---|---:|---:|---:| | Muon | 1 | 3.0692 | 3.0692 | 3.0692 | | Muon | 2 | 3.0728 | 3.0728 | 3.0727 | | HyMuon | 1 | 3.0540 | 3.0446 | 3.0336 | | HyMuon | 2 | 3.0596 | 3.0497 | 3.0397 | Muon has nearly identical final target losses across the three interventions in both repetitions. HyMuon consistently gives Refresh < Keep < Delete: keeping the stored contribution remains useful, and updating that contribution with a fresh gradient further lowers the target loss. On five other fixed validation batches, the mean final losses are essentially unchanged across interventions for both methods: 3.0466 and 3.0470 for the two HyMuon repetitions, and 3.0853 for both Muon repetitions, rounded to four decimal places. This contrast supports the design motivation: HyMuon can retain and use gradient information after further training. We have clarified this purpose in Section 4.
AuthorsOpenReview ↗

Author Response to Reviewer 812o (2/4: [W3, W4(a)])

Author Response to Reviewer 812o (2/4: [W3, W4(a)])

Part 2 covers W3 and W4(a). W4(b–c) follows in Part 3.

3. “The most relevant component ablation is missing. Appendix F.3 compares the combined method against fast-only and slow-only variants, but omits the same slow-history component combined with a raw current gradient. Consequently, it does not isolate the principal modification relative to Schedule-Free for most experimental setting. In Figure 4, it performs worse than Schedule-Free.”

Ans: Thank you for suggesting the slow history plus raw gradient control. We added it on CIFAR-10, and HyM achieves higher final validation accuracy. The control retains the slow history and replaces the fast momentum contribution with the current gradient.

Both methods use WRN-16-8, 300 epochs, batch size 128, and the same split of 45,000 training and 5,000 validation images. We tuned both methods separately and selected their configurations by validation accuracy during the hyperparameter sweeps. The selected configurations and final results are:

MethodLearning rateμ\muMixture and averaging β\betaFinal validation accuracy
Slow history plus raw gradient0.050.990.9596.46%
HyM0.20.950.996.74%

The higher accuracy of HyM supports using fast momentum alongside the same slow history.

Regarding Figure 4, the final mean CIFAR-10 test accuracy is 96.25% for HyM and 96.11% for Schedule-Free. These are the original test results; the table above reports the new validation ablation. We have made the final values clearer in the figure caption and discussion.

4. “Most of ablations are problematic. In LLM abaltion, HyMuon changes matrix momentum, auxiliary optimization, and parameter averaging simultaneously, without sufficient controls separating their effects. The headline C4 comparison additionally tunes HyMuon while retaining Muon’s imported configuration, leaving the advantage over a comparably tuned baseline unresolved. On FineWeb, the reported results are close to AdEMAMix, providing limited evidence of improvement over this relevant baseline.”

Ans: Thank you for raising concerns about component contributions and baseline comparisons. We present the controls here; Part 3 covers local tuning and FineWeb.

(a) Component controls. All controls use LLaMA-134M on C4, 2,000 updates, global batch size 512, sequence length 1,024, and the same validation data. Each has a separate HyMuon reference run with the settings below. Here, yy denotes training parameters and xx their average. The matrix learning rate applies to parameters updated with NS; the auxiliary rate applies to the rest.

Parameter averaging. We added averaging to Muon without changing its gradient evaluations or updates. Both methods use the same averaging rule with β=0.9\beta=0.9:

MethodValidation loss at yyValidation loss at xx
Muon with parameter averaging3.28603.2847
HyMuon3.27173.2760

HyMuon retains its advantage under either evaluation choice.

Auxiliary optimizer. We retained the hybrid matrix update and parameter averaging, with β=0.8\beta=0.8, matrix learning rate 0.0450.045, fast momentum μ=0.85\mu=0.85, and the same schedule. Each auxiliary optimizer uses its own learning rate:

MethodAuxiliary learning rateFinal validation loss at xx
HyMuon0.033.2617
HyMuon with auxiliary AdamW0.013.2624

HyMuon maintains nearly the same performance with auxiliary AdamW.

Slow matrix component. Our simplified Muon version of AdEMAMix replaces HyMuon's growing history with a slow EMA, combined with fast momentum before the same NS transformation. The auxiliary optimizer and parameter averaging are unchanged.

We screened all 12 combinations of matrix learning rates {0.03,0.045,0.06}\{0.03,0.045,0.06\}, fast momentum {0.85,0.9}\{0.85,0.9\}, and slow EMA decay {0.999,0.9999}\{0.999,0.9999\} for 350 updates using the 2,000 update schedule. The two candidates with the lowest validation losses during this sweep were trained from initialization for 2,000 updates.

Final comparisons share initialization, validation data, 500 warmup updates, and auxiliary learning rate 0.030.03. EMA coefficients remain constant; losses are evaluated at xx:

MethodConfigurationMatrix learning rateFast momentum μ\muSlow EMA decay ρ\rhoFinal validation loss at xx
HyMuonReference0.0450.85—3.2570
Muon version of AdEMAMixOriginal settings0.0450.850.9993.2836
Muon version of AdEMAMixSweep selection 10.060.850.9993.2878
Muon version of AdEMAMixSweep selection 20.060.850.99993.3059

HyMuon retains a lower final loss after tuning the learning rate and both momentum coefficients of the EMA control.

AuthorsOpenReview ↗

Author Response to Reviewer 812o (3/4: [W4(b–c)])

Author Response to Reviewer 812o (3/4: [W4(b–c)])

Continuing W4 from Part 2, we address local tuning on C4 and the FineWeb comparison.

(b) Local tuning on C4. HyMuon retains a lower final validation loss after local tuning of Muon. For the main LLaMA-1.3B C4 comparison, both final runs use 10,000 updates and 5.2B input tokens:

OptimizerMatrix learning rateAuxiliary learning rateFinal validation loss
Muon0.0010.0010.00030.00032.7800
HyMuon0.030.030.0030.0032.7632

The Muon baseline retains the matrix update and normalization rules of the original Muon implementation.

Search settings for 1.3B. We first conducted a coarse search. For Muon, we screened five matrix learning rates, {10−4,10−3,3×10−3,10−2,5×10−2}\{10^{-4},10^{-3},3\times10^{-3},10^{-2},5\times10^{-2}\}. For HyMuon, the coarse search considered five matrix learning rates, {0.01,0.03,0.06,0.1,0.3}\{0.01,0.03,0.06,0.1,0.3\}.

The subsequent local grids each contain 12 candidates, covering all combinations of the listed matrix learning rates, auxiliary learning rates, and weight decay values:

OptimizerMatrix learning ratesAuxiliary learning ratesWeight decayCandidate configurations
Muon{0.001,0.003,0.01}\{0.001,0.003,0.01\}{0.0003,0.001}\{0.0003,0.001\}{0.1,0.2}\{0.1,0.2\}12
HyMuon{0.01,0.03,0.06}\{0.01,0.03,0.06\}{0.003,0.01}\{0.003,0.01\}{0.1,0.2}\{0.1,0.2\}12

During the hyperparameter sweeps, we selected candidate configurations by validation loss. Each candidate in the reported local grids has a sweep budget of 350 updates; the final performance comparison uses 10,000 updates per optimizer.

OptimizerLocal grid candidatesSweep updates per candidateUpdates in the final comparison
Muon1235010,000
HyMuon1235010,000

Results at 134M and 348M. HyMuon also has lower final validation loss at these two sizes after local tuning. The table gives the 348M results; each configuration lists the matrix learning rate, auxiliary learning rate, and weight decay, in that order.

ModelTraining updatesInput tokensSelected Muon configurationSelected HyMuon configurationMuon validation lossHyMuon validation loss
348M2,6001.4B(0.003,0.001,0.2)(0.003,0.001,0.2)(0.045,0.03,0.1)(0.045,0.03,0.1)3.06403.0544

At each model size, we first conducted a small preliminary sweep, examining matrix and auxiliary learning rates separately over {0.0003,0.001,0.003,0.01,0.03,0.045,0.06}\{0.0003,0.001,0.003,0.01,0.03,0.045,0.06\} around reference configurations. The weight decay values considered were {0.1,0.2}\{0.1,0.2\}. Based on validation loss in this preliminary sweep, we selected four configurations for each optimizer at each model size for further evaluation. All experiments use a global batch of 512 sequences of length 1,024.

For 134M, the four Muon configurations were (0.001,0.0003,0.1)(0.001,0.0003,0.1), (0.001,0.0003,0.2)(0.001,0.0003,0.2), (0.003,0.001,0.1)(0.003,0.001,0.1), and (0.003,0.001,0.2)(0.003,0.001,0.2). The four HyMuon configurations were (0.03,0.045,0.1)(0.03,0.045,0.1), (0.06,0.045,0.1)(0.06,0.045,0.1), (0.045,0.03,0.1)(0.045,0.03,0.1), and (0.045,0.06,0.1)(0.045,0.06,0.1).

For 348M, the four Muon configurations were (0.001,0.0003,0.1)(0.001,0.0003,0.1), (0.001,0.0003,0.2)(0.001,0.0003,0.2), (0.003,0.001,0.1)(0.003,0.001,0.1), and (0.003,0.001,0.2)(0.003,0.001,0.2). The four HyMuon configurations were (0.03,0.045,0.1)(0.03,0.045,0.1), (0.06,0.045,0.1)(0.06,0.045,0.1), (0.045,0.03,0.1)(0.045,0.03,0.1), and (0.045,0.06,0.1)(0.045,0.06,0.1).

For 134M at approximately 3.9 input tokens per parameter, Muon reaches 3.4886 after 1,001 updates, and HyMuon reaches 3.4473 after 1,000 updates. Both process approximately 0.5B input tokens. The HyMuon run was trained from initialization with matrix/auxiliary learning rates 0.045/0.030.045/0.03.

These comparisons show that HyMuon's advantage on C4 persists after local tuning of Muon. We have included the search settings and results in the revised experimental description.

(c) Comparison with AdEMAMix on FineWeb. We thoroughly tuned HyMuon and the baselines for the FineWeb experiments before submission. The final losses are close at the longer budget, while HyMuon reduces loss faster during training. The endpoint results are:

FineWeb training budgetHyMuonAdEMAMix
1.0B tokens3.47003.4790
2.1B tokens3.35003.3510

At update 8,050 of the 2.1B run, HyMuon reaches a validation loss of 3.5000, compared with 3.5380 for AdEMAMix. Thus, the training trajectory shows a difference even though the final losses become close. We have reported the intermediate comparison together with the endpoints to make the observed behavior clear.

Reviewer_uNgtOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Hybrid Momentum (HyM), a stochastic optimizer whose update mixes, with weight β\beta, two momenta: a slow one, which is a linearly weighted average of all past gradients as in the single-sequence form of Schedule-Free SGD, and a fast heavy-ball one. HyM returns a weighted average of the training iterates. HyMuon applies Newton–Schulz orthogonalization to the mixed direction for matrix parameters, and an Adam-like hybrid update for the other parameters. The paper characterizes the weights HyM assigns to past gradients, and uses GPT-2 interventions to show that a gradient stored in HyMuon's slow momentum still lowers the loss on its batch thousands of steps later. It proves an O(T−1/2)O(T^{-1/2}) bound on the expected suboptimality of the averaged iterate for smooth convex stochastic problems, through a general analysis of time-varying cumulative gradient weights. In the experiments:

  • HyM matches or slightly improves on SGD and Schedule-Free on CIFAR-10/100, SVHN, and ImageNet.
  • HyMuon improves on Muon on IWSLT14, GPT-2 124M, FineWeb (where it matches AdEMAMix), and LLaMA-1.3B on C4.
Critical strengths

The idea is sound: hybrid momentum have been shown to improve convergence independently from Muon, combining the two is a natural idea that could be powerful.

Theory is correct: I checked Lemma C.1, Theorem C.1, and Lemma C.2 step by step and tested the bounds numerically; they hold.

Critical weaknesses

The main issue I find in the paper is the experimental validation. No evidence that the gains survive scaling or per-scale tuning. The paper targets large-model training, but its language-model evidence is two 124M models and one 1.3B run at about 4 tokens per parameter (C4). No experiment varies model size, so one cannot tell whether the gap over Muon grows, persists, or vanishes with scale. Wen et al. (2025, arXiv:2509.02046) find that speedups over well-tuned baselines shrink with model size. Tuning is not scale-consistent either:

  • HyMuon's settings change between tasks, and even between the 1B- and 2.1B-token FineWeb budgets (learning rate 0.045 vs. 0.015, App. F.7).
  • There is no rule for choosing μ\mu (0.85–0.99), β\beta (0.9–0.99), or the learning rate, and especially how to scale them with the model size.

A scaling study would address this: 3–4 model sizes at a fixed number of tokens per parameter, with every method tuned at each size (or tuned at small scale and transferred by a stated rule), reporting the gap or compute multiplier against model size. This is my main concern.

The experiments do not isolate the contribution of the hybrid momentum. The main empirical claim is that HyMuon improves on Muon but other factors change along with the momentum.

  • (i) Averaging. I read in App. F.7, p. 39 that baselines use last iterate while HyMuon uses the its averaged iterate, if this is the case, this is a strong confounding factor.
  • (ii) Weight decay and step size. On GPT-2, Muon uses no matrix weight decay at 100k steps and 0.01 at 600k ( AppF.5, p. 36), against 0.2 for HyMuon (App. F.5, Table 7). It ends behind AdamW in both regimes (2.915 vs. 2.905; 2.886 vs. 2.866). Yet Liu et al. (2025, arXiv:2502.16982, Section 2.2) find that "Muon with weight decay outperforms both vanilla Muon and AdamW" in long runs.
  • (iii) Base update vs. momentum. No experiment crosses the base update with the momentum type. On FineWeb, HyMuon and AdEMAMix differ in both: Newton–Schulz vs. Adam, and hybrid vs. two-EMA momentum. So their tie says nothing about which momentum is better. It would be beneficial to run a muon version of ademamix as well.
  • (iv) Multi-epoch training. Validation and test losses here reflect generalization as much as optimization.
    • GPT-2 makes about 5 (100k steps) and 33 (600k steps) passes over OpenWebText's roughly 9B training tokens. IWSLT14 runs for 55 epochs, and the vision tasks for 100–300.
    • The methods also differ in regularization. HyMuon uses 0.2 matrix weight decay against 0 or 0.01 for Muon on GPT-2, and HyM's weight decay is 1.5–5 times the baselines' in vision (Table 3).
    • No training losses are reported, so optimization and generalization cannot be separated, while the paper sells HyMuon as a method improving optimization, and different optimizers have different implicit biases, which is confounding here. It would be great to see experiments on datasets which are effectively unlimited for the current training recipes.
Questions for authors

See above:

  • Could you provide a recipe to scale hyper parameters with model scale?
  • Could you provide evidence that the method's gain do not vanish at scale?
  • Could you show training losses as well, in order to check that the method is a better optimizer in itself?
3 comments
AuthorsOpenReview ↗

Author Response to Reviewer uNgt (1/2: [W1])

Author Response to Reviewer uNgt (1/2: [W1])

This is Part 1 of 2 (W1, Answers 1–2); Part 2 addresses W2 (Answers 3–6).

We thank the reviewer for recognizing the potential of combining hybrid momentum with Muon and for checking the theoretical results in detail. We understand that the main concerns are experimental validation across model sizes and isolation of the method's components, and address them below. We have highlighted the key changes in blue in the revised manuscript.

W1. Gains across model sizes and hyperparameter tuning

1. “No evidence that the gains survive scaling or per-scale tuning. ... A scaling study would address this: 3–4 model sizes at a fixed number of tokens per parameter, with every method tuned at each size (or tuned at small scale and transferred by a stated rule), reporting the gap or compute multiplier against model size.”

Ans: We thank the reviewer for emphasizing comparisons across model sizes. HyMuon retains a lower validation loss at 134M, 348M, and 1.3B when each model is trained on approximately 3.9 input tokens per parameter and Muon is locally tuned at each size:

ModelTraining updatesInput tokensTokens per parameterMuon lossHyMuon loss
134M1,001 / 1,0000.5B3.93.48863.4473
348M2,6001.4B3.93.06403.0544
1.3B10,0005.2B3.92.78002.7632

The 134M update counts are listed as Muon / HyMuon: 1,001 and 1,000. Both process approximately 0.5B input tokens. At 348M and 1.3B, both methods use the listed number of updates and input tokens. The new 134M and 348M experiments use the same model implementation, C4 stream, and validation protocol.

Tuning. The matrix learning rate applies to the parameter group updated with Newton–Schulz (NS); the auxiliary learning rate applies to the remaining parameters. At 134M and 348M, we first conducted a small preliminary sweep, examining matrix and auxiliary learning rates separately over {0.0003,0.001,0.003,0.01,0.03,0.045,0.06}\{0.0003,0.001,0.003,0.01,0.03,0.045,0.06\} around reference configurations. The weight decay values considered were {0.1,0.2}\{0.1,0.2\}. Based on validation loss in this preliminary sweep, we selected four configurations for each optimizer at each model size for further evaluation. The HyMuon runs at 134M and 348M both use matrix/auxiliary learning rates 0.045/0.030.045/0.03; the 1,000 update run at 134M was trained from initialization.

For 134M, the four Muon configurations were (0.001,0.0003,0.1)(0.001,0.0003,0.1), (0.001,0.0003,0.2)(0.001,0.0003,0.2), (0.003,0.001,0.1)(0.003,0.001,0.1), and (0.003,0.001,0.2)(0.003,0.001,0.2). The four HyMuon configurations were (0.03,0.045,0.1)(0.03,0.045,0.1), (0.06,0.045,0.1)(0.06,0.045,0.1), (0.045,0.03,0.1)(0.045,0.03,0.1), and (0.045,0.06,0.1)(0.045,0.06,0.1).

For 348M, the four Muon configurations were (0.001,0.0003,0.1)(0.001,0.0003,0.1), (0.001,0.0003,0.2)(0.001,0.0003,0.2), (0.003,0.001,0.1)(0.003,0.001,0.1), and (0.003,0.001,0.2)(0.003,0.001,0.2). The four HyMuon configurations were (0.03,0.045,0.1)(0.03,0.045,0.1), (0.06,0.045,0.1)(0.06,0.045,0.1), (0.045,0.03,0.1)(0.045,0.03,0.1), and (0.045,0.06,0.1)(0.045,0.06,0.1).

The gain therefore persists across the tested model sizes under a common budget of tokens per parameter. We have included these settings and results in the revised experimental description.

2. “HyMuon's settings change between tasks, and even between the 1.0B- and 2.1B-token FineWeb budgets ... Could you provide a recipe to scale hyper parameters with model scale?”

Ans: Thank you for asking for a practical rule for selecting hyperparameters across model sizes. For models in the same family, we recommend starting with fast momentum μ=0.85\mu=0.85 and mixing coefficient β=0.9\beta=0.9, then selecting the matrix and auxiliary learning rates on the smaller model.

On the larger model, first try the configuration from the smaller model. Keep the momentum coefficients fixed and tune the two learning rates separately around their transferred values. For example, around a learning rate of 0.0450.045, test {0.03,0.045,0.06}\{0.03,0.045,0.06\}. Select the configuration based on its validation loss during the hyperparameter sweep.

The C4 runs in Answer 1 illustrate using the same matrix and auxiliary learning rates at two model sizes.

We thoroughly tuned HyMuon and the baselines for the FineWeb experiments before submission. The two FineWeb runs change the training budget while keeping the same model. Their learning rates and auxiliary moment coefficients were selected separately for those budgets:

FineWeb training budgetMatrix learning rateAuxiliary learning rateAuxiliary moment coefficients
1.0B tokens0.0450.045(0.97,0.999)(0.97,0.999)
2.1B tokens0.0150.015(0.99,0.999)(0.99,0.999)

Both use mixing coefficient β=0.9\beta=0.9 and fast momentum μ=0.85\mu=0.85. Thus, the FineWeb parameter changes concern training duration, while the C4 procedure above concerns transfer across model sizes. We have made this distinction explicit in the experimental settings.

AuthorsOpenReview ↗

Author Response to Reviewer uNgt (2/2: [W2])

Author Response to Reviewer uNgt (2/2: [W2])

This is Part 2 of 2 (W2, Answers 3–6); Part 1 addresses W1 (Answers 1–2).

W2. Isolating the contribution of hybrid momentum

3. “baselines use last iterate while HyMuon uses the its averaged iterate, if this is the case, this is a strong confounding factor.”

Ans: Thank you for raising the evaluation issue. We added parameter averaging to Muon on 134M C4 without changing its gradient evaluations or updates. Both methods run for 2,000 updates with the same averaging rule, β=0.9\beta=0.9. Here, yy denotes training parameters and xx their average:

MethodValidation loss at yyValidation loss at xx
Muon with parameter averaging3.28603.2847
HyMuon3.27173.2760

HyMuon retains its advantage under either evaluation choice. We have identified the evaluation parameters in the captions and experimental settings.

4. “Weight decay and step size. On GPT-2, Muon uses no matrix weight decay at 100k steps and 0.01 at 600k ( AppF.5, p. 36), against 0.2 for HyMuon (App. F.5, Table 7). It ends behind AdamW in both regimes (2.9150 vs. 2.9050; 2.8860 vs. 2.8660). Yet Liu et al. (2025, Section 2.2) find that "Muon with weight decay outperforms both vanilla Muon and AdamW" in long runs.”

Ans: Thank you for asking why the GPT-2 results differ. Liu et al. (2025, Section 2.2) train an 800M model on 100B tokens; we use GPT-2 124M on OpenWebText. Learning rates and weight decay need not transfer directly across these models and protocols. Our settings were selected within a limited computational budget and are reported in Appendix F.5. The 600k Muon run includes matrix weight decay. A wider search could further improve Muon here.

In our 1.0B token FineWeb experiment, Muon does outperform AdamW: final validation losses are 3.5000 and 3.5300, respectively. Their relative performance thus varies across models and training settings.

5. “No experiment crosses the base update with the momentum type. ... It would be beneficial to run a muon version of ademamix as well.”

Ans: Thank you for suggesting a comparison that separates momentum from the base update. Our simplified Muon version of AdEMAMix replaces the growing history with a slow EMA, combined with fast momentum before the same Newton–Schulz (NS) transformation. The auxiliary optimizer and parameter averaging are unchanged.

We screened all 12 combinations of matrix learning rates {0.03,0.045,0.06}\{0.03,0.045,0.06\}, fast momentum {0.85,0.9}\{0.85,0.9\}, and slow EMA decay {0.999,0.9999}\{0.999,0.9999\} for 350 updates using the 2,000 update schedule. The two candidates with the lowest validation losses during this sweep were trained from initialization for 2,000 updates.

Final comparisons use LLaMA-134M on C4, the same initialization and validation data, 500 warmup updates, and auxiliary learning rate 0.030.03. EMA coefficients remain constant; losses are evaluated at xx.

MethodConfigurationMatrix learning rateFast momentum μ\muSlow EMA decay ρ\rhoFinal validation loss at xx
HyMuonReference0.0450.85—3.2570
Muon version of AdEMAMixOriginal settings0.0450.850.9993.2836
Muon version of AdEMAMixSweep selection 10.060.850.9993.2878
Muon version of AdEMAMixSweep selection 20.060.850.99993.3059

HyMuon retains a lower final loss after tuning the learning rate and both momentum coefficients of the EMA control.

6. “No training losses are reported, so optimization and generalization cannot be separated ... It would be great to see experiments on datasets which are effectively unlimited for the current training recipes.”

Ans: Thank you for suggesting training losses and a sufficiently large dataset. Our C4 comparison addresses both: HyMuon has lower training and validation losses, using a stream of C4 batches rather than repeated passes over a small dataset.

For LLaMA-134M, we average training cross-entropy at yy over the last 100 updates of a 2,000 update run. Both methods use the same sequence of C4 batches. HyMuon uses the configuration in Answer 5; Answer 3 uses a separate configuration for the averaging control. Validation losses are reported at both yy and xx:

MethodMean training loss over the last 100 updatesValidation loss at yyValidation loss at xx
Muon with parameter averaging3.30453.28603.2847
HyMuon3.26473.24403.2570

The gain is visible in optimization of the training objective as well as validation performance.

FineWeb likewise uses approximately 1.0B or 2.1B tokens from an 8.4B token training split, without requiring repeated passes over a small fixed set. The recorded validation results are included in the revision.

Dear authors,

Thank you for your reply and for the additional experiments.

Unfortunately, the added experiments with carefully tuned baselines show barely any gain of Hymuon vs Muon (cf. Table 15), and this is at a very small scale. Since there are no scaling laws from which we can extrapolate, and the gains at small scales are almost null, I maintain my score.

Reviewer_2kc4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Hybrid Momentum (HyM), which combines a linearly weighted average of historical gradients with a fast geometrically discounted momentum. It separately maintains an averaged parameter sequence for evaluation. HyMuon applies Newton-Schulz orthogonalization to the combined matrix momentum; the practical full-model implementation also uses hybrid adaptive moments for auxiliary parameters. The paper characterizes the historical-gradient weights and establishes an O(T−1/2)O(T^{-1/2}) convergence guarantee for the averaged iterate of the base HyM method under smooth convex objectives and bounded stochastic-gradient variance. Experiments cover image classification, machine translation, and language modeling. Additional diagnostics investigate the continued usefulness of stored gradient contributions and the relationship between semantic relevance and target-specific loss improvement.

Critical strengths
  1. The paper provides a concrete and analytically characterized alternative to fixed-horizon EMA mixtures. Its slow component retains a growing history with explicit linear weights, while the fast component maintains responsiveness to recent gradients. The cumulative-gradient-weight analysis accommodates this growing memory and yields a convergence guarantee for the averaged iterate without assuming uniformly bounded gradients. Although the guarantee concerns the base convex method rather than practical HyMuon, it provides a substantive theoretical foundation for the proposed construction.

  2. The empirical study includes useful attempts to separate the method's components. Appendix F.3 compares fast-only, slow-only, full HyM, and slow-EMA replacements with matched total gradient weight, evaluating both training and averaged iterates along each trajectory. The common-checkpoint Keep/Delete/Refresh intervention also directly tests whether a stored historical contribution remains useful while initially holding parameters and other states fixed. These experiments provide more informative evidence than endpoint benchmark comparisons alone.

Critical weaknesses
  1. The fairness and robustness of the main language-model comparisons are not sufficiently established. In the highlighted 1.3B C4 experiment, Muon uses a configuration reproduced from an external work, whereas HyMuon is selected through a search over matrix and auxiliary learning rates, momentum, mixing, auxiliary second-moment coefficients, and weight decay (Appendix F.6). The paper does not document comparable search budgets or a corresponding local search for Muon. Different update normalizations mean that the nominal learning-rate difference alone does not demonstrate an unfair comparison, but they make clarification of the implementation and effective update scaling especially important. The language-model results also lack reported across-seed uncertainty. On FineWeb, the closest explicit slow-memory baseline, AdEMAMix, is close to HyMuon at 1B tokens and matches it to the reported precision at 2.1B tokens. Consequently, the evidence for a robust advantage over well-tuned alternatives remains limited.

  2. The language-model experiments do not adequately isolate the benefit of the growing-history momentum from the other changes in practical HyMuon. Relative to Muon, the implementation changes the matrix direction, replaces auxiliary AdamW with hybrid first- and second-moment recurrences, and evaluates averaged parameters. The CIFAR-10 component study is valuable, and the IWSLT ablation addresses evaluation at the training iterate, but neither isolates these factors in the language-model settings supporting the main scaling claims. Furthermore, the state intervention demonstrates usefulness for a particular target batch, rather than an overall validation benefit or superiority to a suitably tuned slow EMA. Since fast-slow momentum and the long-term usefulness of gradients are already established motivations in related work, stronger evidence for the specific growing-history construction is consequential to the paper's incremental contribution.

Questions for authors
  1. For the 1.3B C4 comparison, what were the search ranges, trial counts, and selection criteria for each optimizer? Was Muon locally tuned under the same data and training implementation? Please also clarify whether the reproduced baseline preserves the source implementation's matrix-update normalization, and whether the improvement persists after an appropriate Muon learning-rate and weight-decay search.

  2. Can a controlled language-model ablation separate (a) hybrid matrix momentum, (b) the auxiliary hybrid adaptive optimizer, and (c) parameter averaging? Particularly informative controls would be Muon with the same passive parameter averaging, hybrid matrix momentum with unchanged auxiliary AdamW, and replacement of the slow component by a tuned EMA before the same NS transformation.

  3. How reproducible are the language-model loss differences across training seeds? For the FineWeb comparison, please provide unrounded endpoint losses and uncertainty sufficient to interpret the small difference from AdEMAMix.

  4. In the Keep/Delete/Refresh intervention, what happens to validation loss on data outside the injected target batch? Does retaining the historical contribution improve broader validation performance, or primarily preserve performance on that target? Repeating the intervention across multiple target batches or checkpoints would help assess how general the observed mechanism is.

5 comments
AuthorsOpenReview ↗

Author Response to Reviewer 2kc4 (1/2: [W1])

Response to Reviewer 2kc4 (1/2): W1

We thank the reviewer for recognizing our gradient weight analysis and component studies. We address your concerns in two comments: Part 1 covers comparison fairness and robustness (W1, Q1/Q3); Part 2 covers the contribution of gradient history (W2, Q2/Q4). Key revisions are highlighted in blue in the manuscript.

W1. Fairness and robustness of the comparisons

(a) C4 tuning and update normalization

“The paper does not document comparable search budgets or a corresponding local search for Muon. ... Was Muon locally tuned under the same data and training implementation?”

Ans: Thank you for asking for comparable local tuning. HyMuon retains a lower final validation loss after local tuning of Muon. For LLaMA-1.3B on C4, both final runs use 10,000 updates and 5.2B input tokens:

OptimizerMatrix learning rateAuxiliary learning rateFinal validation loss
Muon0.0010.0010.00030.00032.7800
HyMuon0.030.030.0030.0032.7632

The matrix learning rate applies to parameters updated with Newton–Schulz (NS); the auxiliary rate applies to the remaining parameters. Muon retains the matrix update and normalization rules of its original implementation.

1.3B search. The coarse search screened five matrix learning rates per method: {10−4,10−3,3×10−3,10−2,5×10−2}\{10^{-4},10^{-3},3\times10^{-3},10^{-2},5\times10^{-2}\} for Muon and {0.01,0.03,0.06,0.1,0.3}\{0.01,0.03,0.06,0.1,0.3\} for HyMuon. The subsequent local grids cover all combinations below:

OptimizerMatrix learning ratesAuxiliary learning ratesWeight decayCandidates
Muon{0.001,0.003,0.01}\{0.001,0.003,0.01\}{0.0003,0.001}\{0.0003,0.001\}{0.1,0.2}\{0.1,0.2\}12
HyMuon{0.01,0.03,0.06}\{0.01,0.03,0.06\}{0.003,0.01}\{0.003,0.01\}{0.1,0.2}\{0.1,0.2\}12

We selected configurations by validation loss during the sweeps. The local search and final training budgets are:

OptimizerLocal grid candidatesSweep updates per candidateUpdates in the final comparison
Muon1235010,000
HyMuon1235010,000

134M and 348M. HyMuon also has lower final validation loss at these sizes after local tuning. Configurations below list matrix learning rate, auxiliary learning rate, and weight decay, in that order.

ModelTraining updatesInput tokensSelected Muon configurationSelected HyMuon configurationMuon validation lossHyMuon validation loss
348M2,6001.4B(0.003,0.001,0.2)(0.003,0.001,0.2)(0.045,0.03,0.1)(0.045,0.03,0.1)3.06403.0544

At each size, a small preliminary sweep examined matrix and auxiliary learning rates separately over {0.0003,0.001,0.003,0.01,0.03,0.045,0.06}\{0.0003,0.001,0.003,0.01,0.03,0.045,0.06\} around reference configurations, considering weight decay {0.1,0.2}\{0.1,0.2\}. Validation loss in this sweep determined four candidates per optimizer for further evaluation. All runs use a global batch of 512 sequences of length 1,024.

At both 134M and 348M, the four Muon candidates were (0.001,0.0003,0.1)(0.001,0.0003,0.1), (0.001,0.0003,0.2)(0.001,0.0003,0.2), (0.003,0.001,0.1)(0.003,0.001,0.1), and (0.003,0.001,0.2)(0.003,0.001,0.2); the four HyMuon candidates were (0.03,0.045,0.1)(0.03,0.045,0.1), (0.06,0.045,0.1)(0.06,0.045,0.1), (0.045,0.03,0.1)(0.045,0.03,0.1), and (0.045,0.06,0.1)(0.045,0.06,0.1).

For 134M at approximately 3.9 input tokens per parameter, Muon reaches 3.4886 after 1,001 updates and HyMuon reaches 3.4473 after 1,000 updates. Both process approximately 0.5B tokens. HyMuon was trained from initialization with matrix/auxiliary learning rates 0.045/0.030.045/0.03.

Thus, HyMuon's C4 advantage persists after local tuning of Muon. These settings and results are included in the revision.

(b) Reproducibility and FineWeb comparisons

“How reproducible are the language-model loss differences across training seeds? For the FineWeb comparison, please provide unrounded endpoint losses and uncertainty sufficient to interpret the small difference from AdEMAMix.”

Ans: Thank you for asking about reproducibility and endpoint losses. In our experience with this setting, the training seed has only a very small effect. These experiments used a limited computational budget.

Both FineWeb budgets use the same approximately 124M model. We thoroughly tuned HyMuon and the baselines before submission. The final logged losses are:

Training budgetHyMuonAdEMAMix
1.0B tokens3.47003.4790
2.1B tokens3.35003.3510

HyMuon has a lower endpoint loss at 1.0B tokens; the final losses at 2.1B tokens are close.

AuthorsOpenReview ↗

Author Response to Reviewer 2kc4 (2/2: [W2])

Response to Reviewer 2kc4 (2/2): W2

Part 2 addresses W2 and Q2/Q4; Part 1 covers W1 and Q1/Q3.

W2. Contribution of the growing gradient history

(a) Parameter averaging and auxiliary optimizer controls

“Can a controlled language-model ablation separate (a) hybrid matrix momentum, (b) the auxiliary hybrid adaptive optimizer, and (c) parameter averaging?”

Ans: Thank you for suggesting these controls. All three controls use LLaMA-134M on C4, 2,000 updates, global batch size 512, sequence length 1,024, and the same validation data. Each has a separate HyMuon reference run with the stated settings. Here, yy denotes training parameters and xx their average.

Parameter averaging. We added averaging to Muon without changing its gradient evaluations or updates. Both methods use the same averaging rule with β=0.9\beta=0.9:

MethodValidation loss at yyValidation loss at xx
Muon with parameter averaging3.28603.2847
HyMuon3.27173.2760

HyMuon retains its advantage under either evaluation choice.

Auxiliary optimizer. We retained the hybrid matrix update and parameter averaging, with β=0.8\beta=0.8, matrix learning rate 0.0450.045, fast momentum μ=0.85\mu=0.85, and the same schedule. Each auxiliary optimizer uses its own learning rate:

MethodAuxiliary learning rateFinal validation loss at xx
HyMuon0.033.2617
HyMuon with auxiliary AdamW0.013.2624

HyMuon maintains nearly the same performance with auxiliary AdamW.

(b) Replacing the slow component with EMA

“Particularly informative controls would be ... replacement of the slow component by a tuned EMA before the same NS transformation.”

Ans: Thank you for this suggestion. Our simplified Muon version of AdEMAMix replaces HyMuon's growing history with a slow EMA, combined with fast momentum before the same Newton–Schulz (NS) transformation. The auxiliary optimizer and parameter averaging are unchanged.

We screened all 12 combinations of matrix learning rates {0.03,0.045,0.06}\{0.03,0.045,0.06\}, fast momentum {0.85,0.9}\{0.85,0.9\}, and slow EMA decay {0.999,0.9999}\{0.999,0.9999\} for 350 updates of the 2,000 update schedule. The two lowest validation loss candidates from this sweep were trained from initialization for 2,000 updates.

The final comparisons share initialization, validation data, 500 warmup updates, and auxiliary learning rate 0.030.03. EMA coefficients remain constant. Losses are evaluated at xx:

MethodConfigurationMatrix learning rateFast momentum μ\muSlow EMA decay ρ\rhoFinal validation loss at xx
HyMuonReference0.0450.85—3.2570
Muon version of AdEMAMixOriginal settings0.0450.850.9993.2836
Muon version of AdEMAMixSweep selection 10.060.850.9993.2878
Muon version of AdEMAMixSweep selection 20.060.850.99993.3059

HyMuon retains a lower final loss after tuning the learning rate and both momentum coefficients of the EMA control.

(c) Scope of the Keep/Delete/Refresh intervention

“In the Keep/Delete/Refresh intervention, what happens to validation loss on data outside the injected target batch? ... Repeating the intervention across multiple target batches or checkpoints...”

Ans: Thank you for this suggestion. We added two target batches, five other validation batches, and a Muon comparison.

Keep/Delete/Refresh tests whether historical gradient information can be retained and remain useful during training. It is not intended to demonstrate better overall training performance.

Both optimizers use the same two targets: inject the target gradient at update 5,000, branch at 7,500, and train to 10,000. Within each optimizer, branches share model parameters and all other optimizer states at branching.

Outside the target. Five fixed validation batches were evaluated every 250 updates. Their mean final losses are:

MethodAdditional target batchDeleteKeepRefresh
Muon13.08533.08533.0853
Muon23.08533.08533.0853
HyMuon13.04663.04663.0466
HyMuon23.04703.04703.0470

The means agree to four decimal places across interventions.

On the injected targets. The final losses are:

MethodAdditional target batchDeleteKeepRefresh
Muon13.06923.06923.0692
Muon23.07283.07283.0727
HyMuon13.05403.04463.0336
HyMuon23.05963.04973.0397

Muon shows almost no difference. For HyMuon, Keep improves on Delete and Refresh improves further on both new targets, consistent with the original experiment. This supports retention and use of historical gradient information after further training, with losses outside the target essentially unchanged.

Thank you for the additional tuning and language-model controls. These experiments substantially alleviate my concerns about the original Muon baseline and about attributing the improvements solely to parameter averaging or the auxiliary optimizer. The repeated state interventions also support the narrower conclusion that historical contributions remain useful for their target batches.

Two concerns remain consequential to my assessment. First, the locally tuned C4 comparison now shows a smaller loss difference, 2.7800 versus 2.7632, while the FineWeb differences are 0.009 and 0.001. Please provide any available independent-seed comparisons or qualify claims of robust superiority where uncertainty has not been measured.

Second, please clarify how the 350-update screening schedule relates to the final run's warmup and learning-rate schedule. The EMA configurations selected by early validation perform worse at the final endpoint than the original configuration, suggesting that early rankings may not reliably predict final performance. Please also report the complete selected C4 configurations, including weight decay.

The rebuttal improves my assessment, but these points remain relevant to whether the observed advantages establish a sufficiently robust benefit of the growing-history construction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.