- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes the RADO optimizer, whose core idea is to first construct a matrix input from coordinate-wise adaptive statistics and then apply matrix orthogonalization. It uses dynamic damping and signed-log graded compression to control the magnitude contrasts of the adaptive direction. Experimental results show that RADO outperforms baselines such as AdamW and Muon on most metrics.
Critical strengths
The experiments cover a diverse range of tasks. The authors evaluate the advantages of RADO over other baseline optimizers across multiple tasks, including language model training and image tasks.
Critical weaknesses
-
The core research question lacks sufficient scientific significance. The paper consider “how should adaptive information enter an orthogonalized matrix update.” This formulation, however, is closer to an operator-placement or formal-combination question between Adam-style coordinate adaptation and Muon-style orthogonalization than to a deeper characterization of the underlying training mechanism. Orthogonalization already considers anisotropic structure through the singular-vector factors and performs spectral normalization through the singular values. Thus merely discussing whether adaptive statistics should be placed before or after orthogonalization risks remaining at the level of operator composition and engineering placement. It does not fully answer why such a combination is beneficial for training, or how it differs mechanistically from existing methods.
-
The experiments are insufficient in scale, and the experimental settings are not reasonable. The learning rate search grid is spaced by factors of three, which is relatively coarse and may not adequately capture the near-optimal regime of each method. In addition, Muon is mainly applied in pretraining, whereas the pretraining experiments in this paper are confined to 0.1B-scale models, with no evaluation on larger models such as those at the 1B+ scale. Moreover, the weight decay values are set too low (about 0.01 in Table 4), far below the commonly used value of 0.1 for AdamW and Muon [1]. As a result, the related experiments in this paper are not sufficiently convincing.
[1] Muon is Scalable for LLM Training. Technical Report.
Questions for authors
See Weaknesses.
2 comments
W2. Hyperparameter tuning, pretraining scale, and weight decay
Thank you for the questions about baseline tuning and pretraining evaluation. We clarify the search protocol and report a completed Qwen3-0.6B comparison with weight decay 0.1.
1. Optimizer-specific hyperparameter selection.
For ViT-CIFAR100, GPT-2/Alpaca, and GPT-2/WikiText-103, each optimizer is tuned separately using three learning rates and two weight-decay candidates. Appendix C.2 and Table 5 document the optimizer- and task-specific grid centers and selected configurations: The factor of three specifies the spacing within each grid; the center is chosen separately for each optimizer and task. For these reported grids, the selected learning rate is the central candidate, with both a smaller and a larger rate evaluated. The candidates span a ninefold range, and the selected rates are interior to that range.
Parameter-group learning-rate multipliers are held fixed within each grid, so the corresponding group rates vary proportionally. The reported configurations are selected as the best within the evaluated candidate sets. This protocol documents task-specific tuning for each baseline and the corresponding comparisons with lower and higher learning rates. The selected configurations are then evaluated with five seeds for the main ViT/GPT-2 comparisons and three seeds for LLaVA.
2. Additional Qwen3-0.6B pretraining evidence.
We have added Qwen3-0.6B training from random initialization on FineWeb, extending the pretraining evaluation beyond the original 124M-parameter setting. All four optimizers complete 8,000 updates, corresponding to 4.2 billion training token positions, with the same architecture, data, sequence length, effective batch size, update budget, and evaluation protocol.
Section 4.2 and Appendix D.4 report the following results at update 8,000 (seed 42):
| Optimizer | Validation loss ↓ | Perplexity ↓ |
|---|---|---|
| Muon | 3.080982 | 21.779784 |
| NorMuon | 3.080029 | 21.759039 |
| AdaMuon | 3.126241 | 22.788157 |
| RADO | 2.988201 | 19.849943 |
Relative to NorMuon, the lowest-perplexity baseline in this comparison, RADO reduces final validation loss by 0.091828 and perplexity by 8.77%. Figure 3 provides the full validation trajectories. The table compares all methods at the same completed training budget.
To provide a completed and controlled comparison within the available computational resources and rebuttal period, we report results after 8,000 updates (4.2B training tokens). Further scaling experiments are ongoing, and the preliminary validation trends observed thus far continue to favor RADO. We intend to incorporate the completed results, full training trajectories, and corresponding experimental settings into the final manuscript revision.
3. Weight decay 0.1 across the additional pretraining comparison.
As specified in Appendix D.4, the Qwen3-0.6B comparison uses weight decay = 0.1 for both matrix and auxiliary parameters across all four optimizers. The gains above are therefore obtained under the weight-decay setting highlighted in the review.
The smaller-task experiments use their task-specific selected configurations. The additional comparison provides direct evidence of RADO's performance in from-scratch pretraining at 0.6B parameters with weight decay 0.1.
W1. Scientific motivation and the mechanism of adaptive matrix updates
Thank you for raising the question of scientific significance. The central issue is not a rearrangement of Adam-style and Muon-style operators, but how gradient-history statistics influence the direction selected by orthogonalization, and how to control that influence.
1. The combination addresses a recognized optimization problem, not merely an operator-placement preference.
Recent methods show the value of combining adaptation and matrix geometry:
Thus, NorMuon and AdaMuon explicitly bridge Adam-style adaptation and Muon-style orthogonalization, while ROOT addresses a related input-robustness question within orthogonalized optimization. Their distinct mechanisms and reported gains show that the interaction is relevant to training performance, rather than an arbitrary composition exercise. They also motivate a further mechanistic question: how should adaptive statistics shape the singular-vector geometry supplied to orthogonalization?
2. RADO follows a concrete design rationale.
RADO develops RAD's damped adaptive normalization, motivated by relativistic dynamics and conformal-symplectic discretization (Section 3.1; Appendix A.2), into a matrix-direction construction:
U=M⊘V⊙δ2+ζ,P=Φ(U),Q=NS5(P),
where ϕ(x)=sign(x)log(1+∣x∣). Second-moment history and damping shape coordinate scales; signed-log compresses contrasts while retaining graded information; NS5 transforms the resulting matrix geometry.
For any elementwise positive denominator D, sign(M⊘D)=sign(M); hard sign therefore eliminates the denominator's influence at fixed momentum. Signed-log retains graded dependence on this scaling while compressing magnitude ratios (Proposition 1).
3. Why the input construction matters even though orthogonalization uses singular vectors.
The singular vectors considered by orthogonalization are those of its input, which coordinate scaling and compression can change. In 2D, define A(a)=I+ae1e2⊤ and J=e1e2⊤−e2e1⊤. Its exact orthogonal polar factor is
Orth(A(a))=4+a22I+aJ.
Varying one relative entry changes the direction although both output singular values equal one. Similarly, finite NS5 transforms the singular values while preserving the input's singular-vector factors in exact arithmetic (Appendix B, Lemma 1). Thus, pre-NS adaptive statistics can change the final matrix direction, not merely its step length. Subsequent positive scalar rescaling cannot generally reproduce that directional change.
4. Theory and controlled evidence support this mechanism.
Appendix B characterizes the interaction of positive coordinate and spectral gains. If κK,κH denote the maximum/minimum positive coordinate- and spectral-gain ratios defined there, one sufficient momentum/update alignment condition is
(κK−1)(κH−1)<4κKκH.
The appendix further establishes finite-NS5 energy bounds and a conditional stationarity guarantee using momentum-tracking error. These results connect the design to aligned, bounded updates.
In the matched-strength, three-seed GPT-2/WikiText-103 ablations (Table 8), Full RADO achieves mean final loss 3.129160 ± 0.003058, versus 3.133901 ± 0.004862 without compression, 3.131014 ± 0.000500 with hard sign, 3.134226 ± 0.002432 with compression after NS5, and 3.137066 ± 0.003172 with preconditioning after NS5. It has the lowest mean loss among these controls. Five-seed comparisons with NorMuon, AdaMuon, and ROOT (Table 1) provide further evidence.
Thus, RADO offers a theoretically characterized construction retaining gradient-history information in the direction selected by orthogonalization, supported by controlled experiments.