- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The proposed
Critical strengths
The paper is well structured, begining with method, experiment and theory.
Critical weaknesses
-
Limited methodological novelty. Under reparameterization, HyM is merely just schedule-free method. The only difference is replacing the direct current-gradient contribution with fast momentum. However, the manuscript does not sufficiently establish the significance of this incremental modification.
-
The motivation does not isolate the proposed contribution. Section 4 primarily investigates the usefulness of retaining historical gradients, a property already present in Schedule-Free, which is totally not novel.
-
The most relevant component ablation is missing. Appendix F.3 compares the combined method against fast-only and slow-only variants, but omits the same slow-history component combined with a raw current gradient. Consequently, it does not isolate the principal modification relative to Schedule-Free for most experimental setting. In Figure 4, it performs worse than Schedule-Free.
-
Most of ablations are problematic. In LLM abaltion, HyMuon changes matrix momentum, auxiliary optimization, and parameter averaging simultaneously, without sufficient controls separating their effects. The headline C4 comparison additionally tunes HyMuon while retaining Muon’s imported configuration, leaving the advantage over a comparably tuned baseline unresolved. On FineWeb, the reported results are close to AdEMAMix, providing limited evidence of improvement over this relevant baseline.
-
The convergence analysis has limited explanatory relevance. The theorem establishes an guarantee by treating HyM as a perturbation of Schedule-Free. Anway, this is just the same as other optimizing method, yielding no benefit at all.
The paper also seems to be heavily AI written, many of places lose essential logic. There are also quite a lot of references that do not appear in the paper: NorMuon, Dion, Mano, etc.
Questions for authors
Please answer my questions in Critical Weakness.
7 comments
Author Response to Reviewer 812o (2/4: [W3, W4(a)])
Author Response to Reviewer 812o (2/4: [W3, W4(a)])
Part 2 covers W3 and W4(a). W4(b–c) follows in Part 3.
3. “The most relevant component ablation is missing. Appendix F.3 compares the combined method against fast-only and slow-only variants, but omits the same slow-history component combined with a raw current gradient. Consequently, it does not isolate the principal modification relative to Schedule-Free for most experimental setting. In Figure 4, it performs worse than Schedule-Free.”
Ans: Thank you for suggesting the slow history plus raw gradient control. We added it on CIFAR-10, and HyM achieves higher final validation accuracy. The control retains the slow history and replaces the fast momentum contribution with the current gradient.
Both methods use WRN-16-8, 300 epochs, batch size 128, and the same split of 45,000 training and 5,000 validation images. We tuned both methods separately and selected their configurations by validation accuracy during the hyperparameter sweeps. The selected configurations and final results are:
| Method | Learning rate | Mixture and averaging | Final validation accuracy | |
|---|---|---|---|---|
| Slow history plus raw gradient | 0.05 | 0.99 | 0.95 | 96.46% |
| HyM | 0.2 | 0.95 | 0.9 | 96.74% |
The higher accuracy of HyM supports using fast momentum alongside the same slow history.
Regarding Figure 4, the final mean CIFAR-10 test accuracy is 96.25% for HyM and 96.11% for Schedule-Free. These are the original test results; the table above reports the new validation ablation. We have made the final values clearer in the figure caption and discussion.
4. “Most of ablations are problematic. In LLM abaltion, HyMuon changes matrix momentum, auxiliary optimization, and parameter averaging simultaneously, without sufficient controls separating their effects. The headline C4 comparison additionally tunes HyMuon while retaining Muon’s imported configuration, leaving the advantage over a comparably tuned baseline unresolved. On FineWeb, the reported results are close to AdEMAMix, providing limited evidence of improvement over this relevant baseline.”
Ans: Thank you for raising concerns about component contributions and baseline comparisons. We present the controls here; Part 3 covers local tuning and FineWeb.
(a) Component controls. All controls use LLaMA-134M on C4, 2,000 updates, global batch size 512, sequence length 1,024, and the same validation data. Each has a separate HyMuon reference run with the settings below. Here, denotes training parameters and their average. The matrix learning rate applies to parameters updated with NS; the auxiliary rate applies to the rest.
Parameter averaging. We added averaging to Muon without changing its gradient evaluations or updates. Both methods use the same averaging rule with :
| Method | Validation loss at | Validation loss at |
|---|---|---|
| Muon with parameter averaging | 3.2860 | 3.2847 |
| HyMuon | 3.2717 | 3.2760 |
HyMuon retains its advantage under either evaluation choice.
Auxiliary optimizer. We retained the hybrid matrix update and parameter averaging, with , matrix learning rate , fast momentum , and the same schedule. Each auxiliary optimizer uses its own learning rate:
| Method | Auxiliary learning rate | Final validation loss at |
|---|---|---|
| HyMuon | 0.03 | 3.2617 |
| HyMuon with auxiliary AdamW | 0.01 | 3.2624 |
HyMuon maintains nearly the same performance with auxiliary AdamW.
Slow matrix component. Our simplified Muon version of AdEMAMix replaces HyMuon's growing history with a slow EMA, combined with fast momentum before the same NS transformation. The auxiliary optimizer and parameter averaging are unchanged.
We screened all 12 combinations of matrix learning rates , fast momentum , and slow EMA decay for 350 updates using the 2,000 update schedule. The two candidates with the lowest validation losses during this sweep were trained from initialization for 2,000 updates.
Final comparisons share initialization, validation data, 500 warmup updates, and auxiliary learning rate . EMA coefficients remain constant; losses are evaluated at :
| Method | Configuration | Matrix learning rate | Fast momentum | Slow EMA decay | Final validation loss at |
|---|---|---|---|---|---|
| HyMuon | Reference | 0.045 | 0.85 | — | 3.2570 |
| Muon version of AdEMAMix | Original settings | 0.045 | 0.85 | 0.999 | 3.2836 |
| Muon version of AdEMAMix | Sweep selection 1 | 0.06 | 0.85 | 0.999 | 3.2878 |
| Muon version of AdEMAMix | Sweep selection 2 | 0.06 | 0.85 | 0.9999 | 3.3059 |
HyMuon retains a lower final loss after tuning the learning rate and both momentum coefficients of the EMA control.
Author Response to Reviewer 812o (3/4: [W4(b–c)])
Author Response to Reviewer 812o (3/4: [W4(b–c)])
Continuing W4 from Part 2, we address local tuning on C4 and the FineWeb comparison.
(b) Local tuning on C4. HyMuon retains a lower final validation loss after local tuning of Muon. For the main LLaMA-1.3B C4 comparison, both final runs use 10,000 updates and 5.2B input tokens:
| Optimizer | Matrix learning rate | Auxiliary learning rate | Final validation loss |
|---|---|---|---|
| Muon | 2.7800 | ||
| HyMuon | 2.7632 |
The Muon baseline retains the matrix update and normalization rules of the original Muon implementation.
Search settings for 1.3B. We first conducted a coarse search. For Muon, we screened five matrix learning rates, . For HyMuon, the coarse search considered five matrix learning rates, .
The subsequent local grids each contain 12 candidates, covering all combinations of the listed matrix learning rates, auxiliary learning rates, and weight decay values:
| Optimizer | Matrix learning rates | Auxiliary learning rates | Weight decay | Candidate configurations |
|---|---|---|---|---|
| Muon | 12 | |||
| HyMuon | 12 |
During the hyperparameter sweeps, we selected candidate configurations by validation loss. Each candidate in the reported local grids has a sweep budget of 350 updates; the final performance comparison uses 10,000 updates per optimizer.
| Optimizer | Local grid candidates | Sweep updates per candidate | Updates in the final comparison |
|---|---|---|---|
| Muon | 12 | 350 | 10,000 |
| HyMuon | 12 | 350 | 10,000 |
Results at 134M and 348M. HyMuon also has lower final validation loss at these two sizes after local tuning. The table gives the 348M results; each configuration lists the matrix learning rate, auxiliary learning rate, and weight decay, in that order.
| Model | Training updates | Input tokens | Selected Muon configuration | Selected HyMuon configuration | Muon validation loss | HyMuon validation loss |
|---|---|---|---|---|---|---|
| 348M | 2,600 | 1.4B | 3.0640 | 3.0544 |
At each model size, we first conducted a small preliminary sweep, examining matrix and auxiliary learning rates separately over around reference configurations. The weight decay values considered were . Based on validation loss in this preliminary sweep, we selected four configurations for each optimizer at each model size for further evaluation. All experiments use a global batch of 512 sequences of length 1,024.
For 134M, the four Muon configurations were , , , and . The four HyMuon configurations were , , , and .
For 348M, the four Muon configurations were , , , and . The four HyMuon configurations were , , , and .
For 134M at approximately 3.9 input tokens per parameter, Muon reaches 3.4886 after 1,001 updates, and HyMuon reaches 3.4473 after 1,000 updates. Both process approximately 0.5B input tokens. The HyMuon run was trained from initialization with matrix/auxiliary learning rates .
These comparisons show that HyMuon's advantage on C4 persists after local tuning of Muon. We have included the search settings and results in the revised experimental description.
(c) Comparison with AdEMAMix on FineWeb. We thoroughly tuned HyMuon and the baselines for the FineWeb experiments before submission. The final losses are close at the longer budget, while HyMuon reduces loss faster during training. The endpoint results are:
| FineWeb training budget | HyMuon | AdEMAMix |
|---|---|---|
| 1.0B tokens | 3.4700 | 3.4790 |
| 2.1B tokens | 3.3500 | 3.3510 |
At update 8,050 of the 2.1B run, HyMuon reaches a validation loss of 3.5000, compared with 3.5380 for AdEMAMix. Thus, the training trajectory shows a difference even though the final losses become close. We have reported the intermediate comparison together with the endpoints to make the observed behavior clear.
Author Response to Reviewer 812o (1/4: [W1, W2])
Author Response to Reviewer 812o (1/4: [W1, W2])
Part 1 covers W1–W2. Part 2 covers W3 and W4(a); Part 3, W4(b–c); Part 4, W5 and writing/references.
We thank the reviewer for recognizing the overall organization of the paper and for the comments on its contribution and evidence. Below, we clarify the relation to Schedule-Free, explain the roles of the mechanism studies and theory, and present the additional component controls and baseline tuning. We also address the presentation and citation concerns. We have highlighted the key changes in blue in the revised manuscript.
1. “Limited methodological novelty. Under reparameterization, HyM is merely just schedule-free method. The only difference is replacing the direct current-gradient contribution with fast momentum. However, the manuscript does not sufficiently establish the significance of this incremental modification.”
Ans: Thank you for asking us to clarify the methodological contribution. We respectfully disagree with the characterization of HyM as merely a reparameterization of Schedule-Free. The distinction is in the weights assigned to past gradients and in how the resulting momentum is used in the matrix update.
First, HyM changes both the recent-gradient contribution and the scaling of the slow history. For Schedule-Free SGD with a constant step size and ct=1/t, define nt=tt−1nt−1+gt. Its update direction and that of HyM are
\qquad d_t^{\mathrm{HyM}}=\frac{\beta q_t}{t}n_t+(1-\beta)b_t,$$ where $q_t=(1-\mu^t)/(1-\mu)$ and $b_t=\mu\frac{t-1}{t}b_{t-1}+g_t$. Thus, the two methods share a slow history state, but generally assign different weights to past gradients. Second, HyMuon combines the slow and fast gradient momenta before applying Newton–Schulz (NS) orthogonalization. Applying the parameter averaging and interpolation of Schedule-Free around a Muon update would instead act on a direction that has already been orthogonalized. Because NS is nonlinear, these constructions generally produce different updates. The momentum formulation therefore provides a direct way to combine the two histories within matrix orthogonalization. Third, the raw gradient and tuned EMA controls test the choices of fast momentum and growing history, respectively. HyM and HyMuon perform better in these comparisons; see Answers 3 and 4. Together with the gradient weight analysis and convergence guarantee, these results support the proposed momentum construction. **2. “The motivation does not isolate the proposed contribution. Section 4 primarily investigates the usefulness of retaining historical gradients, a property already present in Schedule-Free, which is totally not novel.”** **Ans:** Thank you for asking us to clarify the relation between the mechanism studies and the methodological contribution. Retaining gradient information over a long history is also a feature of Schedule-Free. HyM's methodological contribution lies in the specific weighting and combination of the slow history and fast momentum. HyMuon further applies matrix orthogonalization to the combined direction. Section 4 tests whether retained gradient information remains useful during subsequent training, providing a mechanistic explanation for this design. The purpose of Keep/Delete/Refresh is to examine this mechanism, not to demonstrate better overall training performance. The methodological differences are described in Answer 1. The raw gradient and tuned EMA controls separately test the choices of fast and slow components; see Answers 3 and 4. To further test whether retained gradient information remains useful, we repeated Keep/Delete/Refresh on two additional target batches and included Muon as a comparison. Each target is injected at update 5,000; within each optimizer, the three branches share the checkpoint at update 7,500 and continue to update 10,000. Their final target losses are: | Method | Additional target batch | Delete | Keep | Refresh | |---|---|---:|---:|---:| | Muon | 1 | 3.0692 | 3.0692 | 3.0692 | | Muon | 2 | 3.0728 | 3.0728 | 3.0727 | | HyMuon | 1 | 3.0540 | 3.0446 | 3.0336 | | HyMuon | 2 | 3.0596 | 3.0497 | 3.0397 | Muon has nearly identical final target losses across the three interventions in both repetitions. HyMuon consistently gives Refresh < Keep < Delete: keeping the stored contribution remains useful, and updating that contribution with a fresh gradient further lowers the target loss. On five other fixed validation batches, the mean final losses are essentially unchanged across interventions for both methods: 3.0466 and 3.0470 for the two HyMuon repetitions, and 3.0853 for both Muon repetitions, rounded to four decimal places. This contrast supports the design motivation: HyMuon can retain and use gradient information after further training. We have clarified this purpose in Section 4.