- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes RODE, a matrix optimizer that separates weight norm and direction. It updates the scalar norm using a radial gradient and the direction through tangent-projected momentum, projected Newton–Schulz iterations, and a spherical step. Separate learning rates control the two updates. The paper provides conditional convergence analyses and evaluates language-model pretraining up to 1.5B parameters, image classification, and downstream tasks after 9B fine-tuning. Results show improvements over the evaluated Muon configurations, while fixed-radius RODE matches or exceeds the learned-radius version in several settings.
Critical strengths
The method has a clear geometric motivation. Equation (6) preserves the directional factor's unit norm, preventing the directional step from introducing incidental norm growth. This allows the norm and direction to be controlled explicitly.
The independently tuned 1.5B experiment uses evaluation documents disjoint from training. Three-seed results and explicit search budgets strengthen this comparison.
Critical weaknesses
1.Norm-control comparisons remain incomplete. Muon uses zero matrix weight decay, and Muon + NormProj inherits its learning rates without independent tuning. These choices do not establish unfairness, but leave unclear whether RODE improves over sufficiently tuned norm-control alternatives.
2.Component attribution remains incomplete: fixed-radius and retraction variants are competitive, while most ablations retain the original rates.
Questions for authors
1.Do the gains survive a Muon matrix-decay sweep and independent tuning of Muon + NormProj?
2.Can you report alignment and kernel Frobenius-norm distributions during training, including negative-alignment frequency and the effect of safeguards?
3.Can you provide final-epoch CIFAR results, paired downstream differences, and validation-loss curves against elapsed training time?
2 comments
Thank you for the additional comparisons. I will maintain my rating.
Response on Additional Comparisons, Component Attribution, and Experimental Results
We thank the reviewer for the careful review and specific suggestions on norm-control comparisons, component attribution, and additional experiments.
1. Muon comparisons with matrix weight decay
We added matrix-WD experiments for Muon Original and Muon RMS. For each task, we jointly searched the matrix learning rate, auxiliary-parameter learning rate, and matrix WD, then completed formal runs with three seeds using the selected parameters.
The table reports final validation loss as mean ± sample SD; results appear in revised Section 5.3 and Table 2.
After adding WD and retuning, RODE still achieves lower mean validation loss than both Muon variants on both tasks.
We hypothesize that the short-run search chose too high a learning rate for Qwen114M Muon RMS + WD, leading to excessive growth in model weight norms and poorer convergence during full training.
We also added Hyperball, MuonMD, NorMuon, and MuonAll. Results appear in revised Table 1; search budgets and selected parameters are in Appendix B.
2. Component attribution and ablation learning rates
We thank the reviewer for pointing this out. The caption of Table 5 and Appendix B.8 already note that these conclusions were obtained from ablations using the main method's learning rates.
3. Alignment, kernel Frobenius norms, and safeguards
To address this question, we conducted an additional diagnostic run: RODE on Qwen114M / WikiText-103, 4,000 steps, seed 42. We recorded all 84 managed matrices at every step, yielding 336,000 matrix–step observations.
Alignment uses Frobenius cosine similarity, measured separately between tangent momentum and the kernel, and between the stochastic tangent gradient and the kernel. Kernel norms are measured after projection and before normalization to a unit direction.
The radius-floor safeguard never activated in this run, and no parameter, tangent momentum, or kernel approached zero. The maximum direction-renormalization correction was approximately 2.38×10⁻⁷, with a mean of 3.04×10⁻⁸.
Along this diagnostic trajectory, the NS kernel remained positively aligned with tangent momentum, kernel norms stayed away from zero, and renormalization corrections were small.
4. Final-epoch CIFAR-100 results
We extracted epoch-100 results from the original logs for all eight optimizers, each with three seeds. Accuracy is reported in percent as mean ± sample SD:
RODE retains the highest mean accuracy when all methods are evaluated at the final epoch.
5. Paired downstream differences
We added a same-seed paired analysis, using equal-weight mean accuracy across all four downstream tasks as a supplementary summary. For each seed, we first average the four task accuracies, then compute RODE's gain over the matched-seed baseline. Values are in percentage points; SD is the sample standard deviation of the three paired differences.
RODE ranks first among the five optimizers in four-task mean accuracy for each seed and outperforms Muon Original in all twelve task–seed combinations.
6. Validation loss versus actual training time
We added validation-loss curves against cumulative runtime in revised Appendix C.8, Figure 18.