acceptodds
Under review as a conference paper at ICLR 2027

RODE: A Radial-Orthogonal Decoupled Engine for Optimization

Abstract

Matrix-aware optimizers such as Muon exploit the structure of weight matrices to improve neural network training. However, their additive updates couple changes in weight norm and direction: a directional step can also increase the weight norm, and a larger norm reduces the angular effect of subsequent steps of the same size. This coupling makes it difficult to control weight scale and directional learning independently. We introduce RODE, a Radial-Orthogonal Decoupled Engine that separates each weight matrix into a scalar norm and a unit-norm direction. This separation is nontrivial for Muon-style optimization: as the direction changes, accumulated momentum may no longer lie in the current tangent space, while Newton–Schulz conditioning does not preserve tangency. RODE controls the norm with a scalar radial update and optimizes the direction through tangent-projected Newton–Schulz conditioning and a spherical update, using separate learning rates for the two components. This design allows directional learning without unintended norm growth from additive stepping. Compared with both evaluated Muon variants, RODE achieves lower mean validation loss in GPT-2 and Qwen2-style 114M language modeling and higher mean accuracy on CIFAR-100 and ImageNet-1K using learning rates transferred from GPT-2. RODE also achieves lower validation loss than Muon RMS in Qwen2-style 1.5B pretraining from scratch, and the highest mean accuracy on all four downstream tasks after Qwen3.5-9B fine-tuning.

Primary area:
optimization
Keywords:
optimization, matrix-aware optimization, Muon, radial-directional decoupling, Newton-Schulz iteration, deep learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_LSufOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes RODE, a matrix optimizer that separates weight norm and direction. It updates the scalar norm using a radial gradient and the direction through tangent-projected momentum, projected Newton–Schulz iterations, and a spherical step. Separate learning rates control the two updates. The paper provides conditional convergence analyses and evaluates language-model pretraining up to 1.5B parameters, image classification, and downstream tasks after 9B fine-tuning. Results show improvements over the evaluated Muon configurations, while fixed-radius RODE matches or exceeds the learned-radius version in several settings.

Critical strengths

The method has a clear geometric motivation. Equation (6) preserves the directional factor's unit norm, preventing the directional step from introducing incidental norm growth. This allows the norm and direction to be controlled explicitly.

The independently tuned 1.5B experiment uses evaluation documents disjoint from training. Three-seed results and explicit search budgets strengthen this comparison.

Critical weaknesses

1.Norm-control comparisons remain incomplete. Muon uses zero matrix weight decay, and Muon + NormProj inherits its learning rates without independent tuning. These choices do not establish unfairness, but leave unclear whether RODE improves over sufficiently tuned norm-control alternatives.

2.Component attribution remains incomplete: fixed-radius and retraction variants are competitive, while most ablations retain the original rates.

Questions for authors

1.Do the gains survive a Muon matrix-decay sweep and independent tuning of Muon + NormProj?

2.Can you report alignment and kernel Frobenius-norm distributions during training, including negative-alignment frequency and the effect of safeguards?

3.Can you provide final-epoch CIFAR results, paired downstream differences, and validation-loss curves against elapsed training time?

2 comments
AuthorsOpenReview ↗

Response on Additional Comparisons, Component Attribution, and Experimental Results

We thank the reviewer for the careful review and specific suggestions on norm-control comparisons, component attribution, and additional experiments.

1. Muon comparisons with matrix weight decay

We added matrix-WD experiments for Muon Original and Muon RMS. For each task, we jointly searched the matrix learning rate, auxiliary-parameter learning rate, and matrix WD, then completed formal runs with three seeds using the selected parameters.

The table reports final validation loss as mean ± sample SD; results appear in revised Section 5.3 and Table 2.

OptimizerGPT-2Qwen114M
Muon Original3.0295 ± 0.00012.8185 ± 0.0065
Muon Original + WD3.0203 ± 0.00132.8064 ± 0.0016
Muon RMS3.0265 ± 0.00273.0205 ± 0.0103
Muon RMS + WD3.0221 ± 0.00183.4298 ± 0.0160
RODE2.9774 ± 0.00112.7731 ± 0.0013

After adding WD and retuning, RODE still achieves lower mean validation loss than both Muon variants on both tasks.

We hypothesize that the short-run search chose too high a learning rate for Qwen114M Muon RMS + WD, leading to excessive growth in model weight norms and poorer convergence during full training.

We also added Hyperball, MuonMD, NorMuon, and MuonAll. Results appear in revised Table 1; search budgets and selected parameters are in Appendix B.

2. Component attribution and ablation learning rates

We thank the reviewer for pointing this out. The caption of Table 5 and Appendix B.8 already note that these conclusions were obtained from ablations using the main method's learning rates.

3. Alignment, kernel Frobenius norms, and safeguards

To address this question, we conducted an additional diagnostic run: RODE on Qwen114M / WikiText-103, 4,000 steps, seed 42. We recorded all 84 managed matrices at every step, yielding 336,000 matrix–step observations.

Alignment uses Frobenius cosine similarity, measured separately between tangent momentum and the kernel, and between the stochastic tangent gradient and the kernel. Kernel norms are measured after projection and before normalization to a unit direction.

MetricMinimumMedian5th–95th percentilesNegative alignment
Tangent momentum–NS kernel cosine0.1110.5950.387–0.7760 / 336,000
Stochastic tangent gradient–NS kernel cosine−0.004420.2470.133–0.3325 / 336,000 (0.00149%)
Kernel Frobenius norm8.95524.05313.821–26.299—
Kernel norm / square root of the smaller matrix dimension0.3270.9020.769–0.950—

The radius-floor safeguard never activated in this run, and no parameter, tangent momentum, or kernel approached zero. The maximum direction-renormalization correction was approximately 2.38×10⁻⁷, with a mean of 3.04×10⁻⁸.

Along this diagnostic trajectory, the NS kernel remained positively aligned with tangent momentum, kernel norms stayed away from zero, and renormalization corrections were small.

4. Final-epoch CIFAR-100 results

We extracted epoch-100 results from the original logs for all eight optimizers, each with three seeds. Accuracy is reported in percent as mean ± sample SD:

OptimizerPreviously reported best epochFinal epoch 100
RODE73.75 ± 0.2973.70 ± 0.33
RODE-SNR73.64 ± 0.5673.61 ± 0.51
SGD72.77 ± 0.2472.17 ± 0.61
SOAP72.21 ± 0.0972.08 ± 0.11
Muon Original71.49 ± 0.6771.34 ± 0.69
Muon RMS71.11 ± 0.1471.04 ± 0.17
AdamW68.66 ± 0.2968.60 ± 0.34
MARS-M66.57 ± 1.3766.47 ± 1.48

RODE retains the highest mean accuracy when all methods are evaluated at the final epoch.

5. Paired downstream differences

We added a same-seed paired analysis, using equal-weight mean accuracy across all four downstream tasks as a supplementary summary. For each seed, we first average the four task accuracies, then compute RODE's gain over the matched-seed baseline. Values are in percentage points; SD is the sample standard deviation of the three paired differences.

BaselineSeed 0 gainSeed 1 gainSeed 2 gainMean gain ± SD
Muon RMS+0.932+0.523+0.930+0.795 ± 0.235
Muon Original+2.246+3.389+1.907+2.514 ± 0.777
SOAP+3.543+3.248+1.601+2.797 ± 1.046
AdamW+4.906+5.254+3.927+4.696 ± 0.688

RODE ranks first among the five optimizers in four-task mean accuracy for each seed and outperforms Muon Original in all twelve task–seed combinations.

6. Validation loss versus actual training time

We added validation-loss curves against cumulative runtime in revised Appendix C.8, Figure 18.

Thank you for the additional comparisons. I will maintain my rating.

Reviewer_ZKuhOpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+2)
Read review
Paper summary

This paper presents RODE, a combination optimizer designed to apply a different update rule to the norm versus the direction of weight matrices. More specifically, SGD is applied to the norm, while a Muon-like Newton-Schulz orthogonalization is applied to the directional component, which then produces an update to the weights via spherical interpolation. Pretraining experiments across language and vision, scaling to 1.5B parameters, and fine-tuning experiments on 9B models show the RODE optimizer to outperform other common optimizers under matched step count, at a slight cost in terms of speed and no cost in terms of memory.

Critical strengths
  • RODE shows improved training and fine-tuning performance over some other optimizers which are commonly used.
  • RODE is able to scale up while still holding an advantage over other optimizers.
  • A strong effort has been made to tune learning rates and hyperparameters for every optimizer compared and document the process thoroughly.
Critical weaknesses

Unfortunately, there is a crucial linear algebra error introduced in the text just beneath Equation (6), and it appears that this error would undermine the entire optimizer's design, interpretation of how and why it works, as well as all the convergence Theorems given. The error is in the statement that when given ⟨Mt⊥,Ut⟩F=0\langle M_t^\perp, U_t \rangle_F = 0, and Dt⊥D_t^\perp is a multiple of the orthogonalized version of Mt⊥M_t^\perp, then subsequently we also have ⟨Dt⊥,Ut⟩F=0\langle D_t^\perp, U_t \rangle_F = 0. A quick check with the matrices Ut=(1/5002/5)U_t = \begin{pmatrix} 1/\sqrt{5} & 0 \\\\ 0 & 2/\sqrt{5} \end{pmatrix} and Mt⊥=(200−1)M_t^\perp = \begin{pmatrix} 2 & 0 \\\\ 0 & -1 \end{pmatrix} reveals the error - we get Dt⊥∝(100−1)D_t^\perp \propto \begin{pmatrix} 1 & 0 \\\\ 0 & -1 \end{pmatrix} whose Frobenius inner product against UtU_t is nonzero.

Without this orthogonality statement, the fundamental behavior of this optimization procedure is not accurately understood and interpreted by the paper, as the usage of spherical interpolation and angular learning rate are no longer well-motivated designs. And as previously mentioned, the stated Theorems follow from the error as well. The experimental success goes unexplained.

In the optimizer's design, there is also renormalization of Ut+1U_{t+1} which was introduced to mitigate norm drift, but I believe the drift had been misattributed to floating point precision, when it likely came from spherical interpolation applied to non-orthogonal components, which makes renormalization a poorly-motivated design solution.

On the experimental front, I have two concerns:

  • Comparison with the most-relevant optimizers mentioned in the related work, Hyperball and Magnitude-Direction Decoupling, is missing entirely. I might also suggest comparison with NorMuon (Li et al. 2025) as well.
  • In the language model pretraining experiments, the training tokens to parameter count ratio is quite unrealistic. The ratios I calculate from settings reported in the Appendix come out to about 4.2x (GPT-2 experiments) and 1.7x (1.5B experiments) much lower than the Chinchilla ratio of 20x (Hoffmann et al. 2022). I am also concerned about what happens when the experiment is made more realistic by extending the number of training tokens - RODE's training curve appears to experience a sudden leveling off at the end of training, which may possibly reveal worse performance with extended training.

Li, Z., Liu, L., Liang, C., Chen, W., & Zhao, T. (2025). Normuon: Making muon more efficient and scalable. arXiv preprint arXiv:2510.05491.

Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., ... & Sifre, L. (2022). Training compute-optimal large language models. arXiv preprint arXiv:2203.15556.

Questions for authors
  • Is there any way to understand what the optimizer is doing despite not retaining the errant inner product property?
  • If I have made a mistake somewhere in identifying the error and it's consequences, please let me know. The error is critical and is almost entirely the cause for my rating of this paper.
6 comments
AuthorsOpenReview ↗

Clarification and Discussion of Mathematical and Other Issues

We sincerely thank Reviewer ZKuh for the constructive feedback and the openness in inviting us to clarify the orthogonality property.

1. Clarification of orthogonality and the error

The counterexample omits the tangent projection after each NS iteration in Eq. (5). This operator was already present in the original submission.

RODE actually computes

\mathcal P_t(Z)=Z-\langle Z,U_t\rangle_FU_t.$$ Projection removes the component along the current weight direction, so every iterate is orthogonal to $U_t$ in the Frobenius inner product. Final normalization preserves this property. Your matrices are $$U_t=\frac1{\sqrt5}\begin{pmatrix}1&0\\\\0&2\end{pmatrix}, \qquad M_t^\perp=\begin{pmatrix}2&0\\\\0&-1\end{pmatrix}.$$ Here $\\|U_t\\|_F=1$, $\\|M_t^\perp\\|_F=\sqrt5$, and their inner product is zero. The algorithm initializes $$X_0=\frac{M_t^\perp}{\\|M_t^\perp\\|_F} =\frac{\operatorname{diag}(2,-1)}{\sqrt5}.$$ Substituting the NS polynomial and projection in Eq. (5), each iterate has the form $X_k=a_kX_0$, where $$a_0=1,\qquad a_{k+1}=3.4445a_k-3.247a_k^3+1.05638a_k^5.$$ After five iterations, $a_5\approx1.148087>0$, so normalization gives $$D_t^\perp=\frac{X_5}{\\|X_5\\|_F} =\frac{\operatorname{diag}(2,-1)}{\sqrt5}, \qquad \langle D_t^\perp,U_t\rangle_F=\frac{2-2}{5}=0.$$ Thus $\operatorname{diag}(1,-1)$ is not the direction produced by RODE. Since $U_t$ and $D_t^\perp$ have unit Frobenius norm and are mutually orthogonal, Eq. (6) satisfies $$\\|U_{t+1}\\|_F^2=\cos^2\theta_t+\sin^2\theta_t=1.$$ Renormalization handles floating-point error; it is unnecessary in exact arithmetic. The plotted **full-model norm** also includes parameters managed by AdamW, including one-dimensional parameters. Even with fixed RODE-managed matrix norms, these parameters can change the full-model norm. This is distinct from the unit norm of an individual $U_t$, as explained in the original Section 5.1 and Appendix B.3. Therefore, this counterexample does not refute RODE’s orthogonality and cannot, on that basis, invalidate the spherical update or convergence analysis. **2. Comparisons with related optimizers** We have added Hyperball, MuonMD, NorMuon, and MuonAll to revised Table 1. Search budgets and selected parameters are provided in Appendix B. Three-seed validation loss (mean ± sample SD): | Optimizer | GPT-2 | Qwen114M | |---|---|---| | AdamW | 3.1886 ± 0.0120 | 3.0503 ± 0.0071 | | Muon Original | 3.0295 ± 0.0001 | 2.8185 ± 0.0065 | | Muon RMS | 3.0265 ± 0.0027 | 3.0205 ± 0.0103 | | SOAP | 2.9567 ± 0.0023 | 2.7684 ± 0.0008 | | MARS-M | 2.9699 ± 0.0016 | 2.8885 ± 0.0040 | | Hyperball | 2.9752 ± 0.0029 | 2.8221 ± 0.0034 | | MuonMD | **2.9428 ± 0.0003** | 2.7279 ± 0.0017 | | NorMuon | 2.9831 ± 0.0005 | **2.7263 ± 0.0024** | | MuonAll | 3.0517 ± 0.0011 | 2.7797 ± 0.0025 | | RODE | 2.9774 ± 0.0011 | 2.7731 ± 0.0013 | | RODE-SNR | 2.9724 ± 0.0557 | 2.7493 ± 0.0011 | | RODE-Gain | **2.9370 ± 0.0017** | **2.7277 ± 0.0006** | RODE-Gain is listed separately and has the same parameter-control granularity as MuonMD, down to individual rows and columns. We also independently tuned the learning rates and matrix weight decay of Muon Original and Muon RMS; results appear in Section 5.3 and Table 2. Overall, the RODE family ranks among the leading optimizers in these comparisons. **3. Longer training and end-of-run flattening** The full validation-loss trajectories of the added long-run experiment appear in revised Figure 3 (p. 7), with final results in Table 3. The original experiments use cosine scheduling. Near the end, the learning rate approaches zero, updates shrink, and loss curves flatten. This is common under this schedule and does not by itself imply that RODE will fail with longer training. The additional experiment trains Qwen114M on FineWeb for 34,830 steps, approximately 2.283 billion input tokens or 20 tokens per parameter, using seeds 42 and 666: | Optimizer | Final validation loss: mean ± sample SD | |---|---| | Muon Original | 4.74691 ± 0.05628 | | Muon RMS | 4.90878 ± 0.01677 | | RODE | 3.59004 ± 0.00046 | We hypothesize that reusing peak learning rates selected by short-run searches, combined with cosine decay, keeps Muon Original and Muon RMS at high learning rates for an extended period, causing Q/K weight norms in the models trained with these two configurations to grow and thereby amplifying attention logits, inducing attention saturation, and increasing loss. The norm and attention diagnostics in Figure 3, Appendix C.7, and Figure 17 are consistent with this explanation, while RODE’s Q/K weight norms remain close to their initial values. Kimi K2 (Kimi K2: Open Agentic Intelligence) also discusses Muon’s Q/K weight growth, logit explosion, and training instability in Section 2.1 and Appendix E, and introduces QK-Clip to control this behavior by rescaling Q/K weights.

Dear Reviewer ZKuh,

We would like to kindly follow up on our response to your central mathematical concern.

As detailed in our rebuttal, the counterexample does not account for the tangent projection explicitly applied after every Newton–Schulz iteration in the original Eq. (5). This projection guarantees Frobenius orthogonality, and the subsequent normalization preserves it. We also evaluated the provided counterexample under the actual RODE update.

Since this concern was identified as the primary reason for your rating, we would greatly appreciate your assessment of whether our clarification resolves it, or whether any specific mathematical issue remains.

Thank you again for your time and consideration.

Thank you for your clarifications and for taking the extra experiments. I now see indeed there is a projection after the NS step as is written in the paper, which I didn't see at first, and if this is is indeed what you do, then it would rectify all of my concerns on this matter. But in the code in the supplementary materials, the way you apply the post-NS projection in the implementation is actually not matching the way you have just described, still undermining your results. The code shows you are interleaving the projection with each NS iteration, which still is not do the same thing mathematically as NS orthonormalization then followed by projection. Granted, this discrepancy is only present in your implementation of original RODE and not RODE-SNR, so it only is affecting part of your results. And also many of the interpretation/conclusions now stand their ground because the orthogonality is indeed preserved in both the RODE and RODE-SNR implementations. So I will increase my score by one because of that (and because of the good performance and comparisons, and extended training runs with their subsequent analysis as well). I would have increased my score by even one more, but I cannot do so because I am seeing your implementation does not match the algorithm that is described in the paper, in quite a material way.

Also, thank you for clarifying on the cosine learning rate decay, that explains the behavior.

As a sidenote for the future, a good convention is to color the text for all revision changes to the paper in red or blue so it's easier for the reviewers to see what exactly changed between revised versions of the paper. The color can be removed in a camera-ready version upon acceptance.

Reviewer_KgpkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents RODE, a matrix optimizer that splits each weight matrix into a scalar Frobenius norm and a unit-norm direction, which it updates with separate learning rates. The paper develops an update rule that keeps the Newton–Schulz conditioning of Muon while controlling the norm and the direction separately. A convergence analysis is given under alignment and regularity assumptions, and a linear rate is derived for a first-order model. Experiments on language-model pretraining at several scales, image classification, and fine-tuning of a large pretrained model demonstrate that the optimizer outperforms the two Muon baselines with smaller norm growth, and that a variant with a fixed norm performs at least as well.

Critical strengths
  • The proposed optimizer updates the weight norm and the direction with separate learning rates, which makes the norm an explicit choice instead of a by-product of each directional step.

  • The evaluation covers many settings.

  • The paper reports unfavorable results openly.

Critical weaknesses
  • The technical contribution is limited. Previous work already trains Muon on a fixed-norm sphere [1], and the paper’s own ablations show that its additions, a learned norm update and a rotation on the sphere, do not improve on a fixed-norm variant.

  • The claimed gain over Muon is not convincing. The Muon baselines run without weight decay, and none of the recent Muon variants [2,3,4] with weight or update normalization is compared.

  • The advantage reported for 9B fine-tuning is within seed noise. RODE has a worse validation loss than three baselines, and on three of the four downstream tasks its lead is smaller than the runner-up’s own standard deviation.

  • The computation-cost evaluation is inadequate. All throughput numbers come from a 114M model on a single GPU, and the 1.5B and 9B runs report no cost at all. Numbers on large scale is necessary to better understand the cost.

[1] Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization

[2] Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors

[3] MuonAll: Muon Variant for Efficient Finetuning of Large Language Models

[4] NorMuon: Making Muon more efficient and scalable

Questions for authors

Included in weakness. Mainly on how will the performance be on Muon with weight decay, other Muon variants, and costs on larger scale.

2 comments
AuthorsOpenReview ↗

Clarification and Supplementary Results on Baselines, Large-Model Experiments, and Method Contributions

We thank the reviewer for the careful review and specific comments on method contributions, baselines, 9B results, and large-model computational costs.

1. Muon comparisons with weight decay

We added matrix-WD experiments for Muon Original and Muon RMS. On each task, we jointly searched the matrix learning rate, auxiliary-parameter learning rate, and matrix WD, then completed three-seed formal training with the selected parameters.

After adding WD and retuning, base RODE still achieves lower mean validation loss than both Muon variants on GPT-2 and Qwen114M. Results appear in revised Section 5.3 and Table 2, and below.

2. Additional related optimizers

We added Hyperball, MuonMD, NorMuon, and MuonAll. Full results appear in revised Table 1; search budgets and selected parameters are in Appendix B.

The table combines Tables 1 and 2. Values are final validation loss (three-seed mean ± sample SD); the top two per model are bold.

OptimizerGPT-2Qwen114M
AdamW3.1886 ± 0.01203.0503 ± 0.0071
Muon Original3.0295 ± 0.00012.8185 ± 0.0065
Muon Original + WD3.0203 ± 0.00132.8064 ± 0.0016
Muon RMS3.0265 ± 0.00273.0205 ± 0.0103
Muon RMS + WD3.0221 ± 0.00183.4298 ± 0.0160
SOAP2.9567 ± 0.00232.7684 ± 0.0008
MARS-M2.9699 ± 0.00162.8885 ± 0.0040
Hyperball2.9752 ± 0.00292.8221 ± 0.0034
MuonMD2.9428 ± 0.00032.7279 ± 0.0017
NorMuon2.9831 ± 0.00052.7263 ± 0.0024
MuonAll3.0517 ± 0.00112.7797 ± 0.0025
RODE2.9774 ± 0.00112.7731 ± 0.0013
RODE-SNR2.9724 ± 0.05572.7493 ± 0.0011
RODE-Gain2.9370 ± 0.00172.7277 ± 0.0006

RODE-Gain is listed separately and has the same parameter-control granularity as MuonMD, down to individual rows and columns. Overall, the RODE family ranks among the leading optimizers in these comparisons.

3. Whether 9B gains reflect seed noise

We added a same-seed paired analysis of the existing three seeds. For each seed, we first take the equal-weight mean accuracy across all four downstream tasks, then subtract the matched-seed baseline accuracy from RODE's. Results are in percentage points:

BaselineSeed 0Seed 1Seed 2Mean gain ± sample SD
Muon RMS+0.932+0.523+0.930+0.795 ± 0.235
Muon Original+2.246+3.389+1.907+2.514 ± 0.777

RODE ranks first among the five optimizers in four-task mean accuracy for each seed; against Muon Original, it improves in all twelve task–seed combinations. Thus, its cross-task mean gains have a consistent direction across these three seeds. Comparing the mean difference with the runner-up's own SD cannot directly establish whether the difference is seed noise; same-seed paired differences address this more directly.

The rankings by validation loss and downstream accuracy differ: SOAP has the lowest validation loss, while RODE has the highest mean accuracy on each downstream task. Revised Table 7 retains both results. Our conclusions are limited to performance on the evaluated downstream tasks and consistency across seeds.

4. Computational costs at 1.5B and 9B

We additionally report actual runtimes from existing logs at both scales and calculate optimizer-state storage from persistent state tensors.

1.5B / FineWeb: one RTX 4090, 20,000 steps, seed 1.

OptimizerCumulative logged runtime (hours)Optimizer state (GiB)
RODE82.943.15
RODE, fixed radius84.393.15
Muon RMS84.033.15

9B full-parameter fine-tuning: one A100 80GB, 3,000 steps; three-seed mean ± sample SD.

OptimizerTime through final validation (hours)Optimizer state (GiB)
RODE7.43 ± 0.0516.68
Muon Original6.31 ± 0.2216.68
Muon RMS6.24 ± 0.1016.68
SOAP2.48 ± 0.1333.48
AdamW2.47 ± 0.0533.36

In terms of operation counts, relative internal optimizer overhead is largely independent of total parameter count for the same matrix shapes and update configuration. These historical runtimes may reflect other processes competing for GPU, CPU, and memory resources and are for reference; Table 8 in the paper reports dedicated tests on an otherwise idle GPU and server, with matched environments and no interference from other processes.

5. Fixed-radius RODE versus Hyperball

Before adding the current gradient, RODE removes the historical momentum component along the current weight direction; Hyperball uses ordinary Muon momentum accumulation without this projection. After every NS iteration, RODE again removes the output component along the current weight direction, making every output orthogonal to the current weight; Hyperball uses ordinary Muon NS without this per-iteration projection. Fixed-radius RODE only disables radial learning and retains both projections, so its directional update still differs from Hyperball.

Thank you for the rebuttal and the additional experiments. My main concerns remain.

  • Gain is not convincing: In revised Table 1, RODE perform slightly worse than MuonMD on both models, and worse than NorMuon on one. RODE-Gain is competitive, by adopting MuonMD's gains.
  • The cost is not negligible considering that the gain is not significant. At 9B, RODE is about 20% slower than Muon and 3× slower than AdamW. (Though the author note that the measured time might not be reliable.)

I will therefore keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.