acceptodds
Under review as a conference paper at ICLR 2027

Constructing Adaptive Directions for Matrix Orthogonalization

Abstract

Coordinate-wise adaptation and matrix orthogonalization exploit complementary information—gradient history and matrix structure—but their interaction depends critically on the matrix presented to orthogonalization. We propose Relativistic Adaptive Gradient Descent with Orthogonalization (RADO), which constructs this matrix in three stages: dynamic damping forms an adaptive direction, signed-log compression transforms its coordinate magnitudes while retaining their ordering, and a finite Newton-Schulz transformation acts last. This graded compression preserves dependence on adaptive scaling that is eliminated by a hard sign map. Across independently tuned coordinate-adaptive and orthogonalized-momentum baselines, RADO achieves the highest mean peak ViT validation accuracy and the lowest mean final perplexity on both GPT-2 tasks. On WikiText-103, RADO reduces mean perplexity by approximately 7.45% relative to Muon and 6.57% relative to AdamW, with a 32.60% reduction in interpolated steps to Adam's final evaluation-loss target. In LLaVA fine-tuning, RADO attains the highest mean score on six of seven reported benchmarks. Component ablations and comparisons with alternative update constructions favor RADO in the tested setting, while resource measurements characterize its computational costs.

Primary area:
optimization
Keywords:
Optimization, Adaptive Optimization, Matrix Orthogonalization, Deep Learning Optimization, Gradient-Based Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_APtuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes the RADO optimizer, whose core idea is to first construct a matrix input from coordinate-wise adaptive statistics and then apply matrix orthogonalization. It uses dynamic damping and signed-log graded compression to control the magnitude contrasts of the adaptive direction. Experimental results show that RADO outperforms baselines such as AdamW and Muon on most metrics.

Critical strengths

The experiments cover a diverse range of tasks. The authors evaluate the advantages of RADO over other baseline optimizers across multiple tasks, including language model training and image tasks.

Critical weaknesses
  1. The core research question lacks sufficient scientific significance. The paper consider “how should adaptive information enter an orthogonalized matrix update.” This formulation, however, is closer to an operator-placement or formal-combination question between Adam-style coordinate adaptation and Muon-style orthogonalization than to a deeper characterization of the underlying training mechanism. Orthogonalization already considers anisotropic structure through the singular-vector factors and performs spectral normalization through the singular values. Thus merely discussing whether adaptive statistics should be placed before or after orthogonalization risks remaining at the level of operator composition and engineering placement. It does not fully answer why such a combination is beneficial for training, or how it differs mechanistically from existing methods.

  2. The experiments are insufficient in scale, and the experimental settings are not reasonable. The learning rate search grid is spaced by factors of three, which is relatively coarse and may not adequately capture the near-optimal regime of each method. In addition, Muon is mainly applied in pretraining, whereas the pretraining experiments in this paper are confined to 0.1B-scale models, with no evaluation on larger models such as those at the 1B+ scale. Moreover, the weight decay values are set too low (about 0.01 in Table 4), far below the commonly used value of 0.1 for AdamW and Muon [1]. As a result, the related experiments in this paper are not sufficiently convincing.

[1] Muon is Scalable for LLM Training. Technical Report.

Questions for authors

See Weaknesses.

2 comments
AuthorsOpenReview ↗

W1. Scientific motivation and the mechanism of adaptive matrix updates

Thank you for raising the question of scientific significance. The central issue is not a rearrangement of Adam-style and Muon-style operators, but how gradient-history statistics influence the direction selected by orthogonalization, and how to control that influence.

1. The combination addresses a recognized optimization problem, not merely an operator-placement preference.

Recent methods show the value of combining adaptation and matrix geometry:

  • NorMuon (Li et al., ICML 2026) combines Muon orthogonalization with neuron-wise second-moment normalization to address unbalanced row updates. Its 1.1B pretraining results demonstrate their complementarity.
  • AdaMuon (Si et al., 2025) combines sign-stabilized orthogonalization with element-wise second-moment adaptation of the orthogonalized output, for variance-dependent scaling.
  • ROOT (He et al., 2025) combines outlier suppression before orthogonalization with a shape-calibrated spectral transformation, for robustness to momentum outliers.

Thus, NorMuon and AdaMuon explicitly bridge Adam-style adaptation and Muon-style orthogonalization, while ROOT addresses a related input-robustness question within orthogonalized optimization. Their distinct mechanisms and reported gains show that the interaction is relevant to training performance, rather than an arbitrary composition exercise. They also motivate a further mechanistic question: how should adaptive statistics shape the singular-vector geometry supplied to orthogonalization?

2. RADO follows a concrete design rationale.

RADO develops RAD's damped adaptive normalization, motivated by relativistic dynamics and conformal-symplectic discretization (Section 3.1; Appendix A.2), into a matrix-direction construction:

U=M⊘V⊙δ2+ζ,P=Φ(U),Q=NS5(P),U=M\oslash\sqrt{V\odot\delta^2+\zeta},\quad P=\Phi(U),\quad Q=\mathrm{NS5}(P),

where ϕ(x)=sign⁡(x)log⁡(1+∣x∣)\phi(x)=\operatorname{sign}(x)\log(1+|x|). Second-moment history and damping shape coordinate scales; signed-log compresses contrasts while retaining graded information; NS5 transforms the resulting matrix geometry.

For any elementwise positive denominator DD, sign⁡(M⊘D)=sign⁡(M)\operatorname{sign}(M\oslash D)=\operatorname{sign}(M); hard sign therefore eliminates the denominator's influence at fixed momentum. Signed-log retains graded dependence on this scaling while compressing magnitude ratios (Proposition 1).

3. Why the input construction matters even though orthogonalization uses singular vectors.

The singular vectors considered by orthogonalization are those of its input, which coordinate scaling and compression can change. In 2D, define A(a)=I+ae1e2⊤A(a)=I+a e_1e_2^\top and J=e1e2⊤−e2e1⊤J=e_1e_2^\top-e_2e_1^\top. Its exact orthogonal polar factor is

Orth⁡(A(a))=2I+aJ4+a2.\operatorname{Orth}(A(a))=\frac{2I+aJ}{\sqrt{4+a^2}}.

Varying one relative entry changes the direction although both output singular values equal one. Similarly, finite NS5 transforms the singular values while preserving the input's singular-vector factors in exact arithmetic (Appendix B, Lemma 1). Thus, pre-NS adaptive statistics can change the final matrix direction, not merely its step length. Subsequent positive scalar rescaling cannot generally reproduce that directional change.

4. Theory and controlled evidence support this mechanism.

Appendix B characterizes the interaction of positive coordinate and spectral gains. If κK,κH\kappa_K,\kappa_H denote the maximum/minimum positive coordinate- and spectral-gain ratios defined there, one sufficient momentum/update alignment condition is

(κK−1)(κH−1)<4κKκH.(\kappa_K-1)(\kappa_H-1)<4\sqrt{\kappa_K\kappa_H}.

The appendix further establishes finite-NS5 energy bounds and a conditional stationarity guarantee using momentum-tracking error. These results connect the design to aligned, bounded updates.

In the matched-strength, three-seed GPT-2/WikiText-103 ablations (Table 8), Full RADO achieves mean final loss 3.129160 ± 0.003058, versus 3.133901 ± 0.004862 without compression, 3.131014 ± 0.000500 with hard sign, 3.134226 ± 0.002432 with compression after NS5, and 3.137066 ± 0.003172 with preconditioning after NS5. It has the lowest mean loss among these controls. Five-seed comparisons with NorMuon, AdaMuon, and ROOT (Table 1) provide further evidence.

Thus, RADO offers a theoretically characterized construction retaining gradient-history information in the direction selected by orthogonalization, supported by controlled experiments.

AuthorsOpenReview ↗

W2. Hyperparameter tuning, pretraining scale, and weight decay

Thank you for the questions about baseline tuning and pretraining evaluation. We clarify the search protocol and report a completed Qwen3-0.6B comparison with weight decay 0.1.

1. Optimizer-specific hyperparameter selection.

For ViT-CIFAR100, GPT-2/Alpaca, and GPT-2/WikiText-103, each optimizer is tuned separately using three learning rates and two weight-decay candidates. Appendix C.2 and Table 5 document the optimizer- and task-specific grid centers and selected configurations: Gη={ηc/3,ηc,3ηc}.\mathcal G_\eta=\{\eta_c/3,\eta_c,3\eta_c\}. The factor of three specifies the spacing within each grid; the center is chosen separately for each optimizer and task. For these reported grids, the selected learning rate is the central candidate, with both a smaller and a larger rate evaluated. The candidates span a ninefold range, and the selected rates are interior to that range.

Parameter-group learning-rate multipliers are held fixed within each grid, so the corresponding group rates vary proportionally. The reported configurations are selected as the best within the evaluated candidate sets. This protocol documents task-specific tuning for each baseline and the corresponding comparisons with lower and higher learning rates. The selected configurations are then evaluated with five seeds for the main ViT/GPT-2 comparisons and three seeds for LLaVA.

2. Additional Qwen3-0.6B pretraining evidence.

We have added Qwen3-0.6B training from random initialization on FineWeb, extending the pretraining evaluation beyond the original 124M-parameter setting. All four optimizers complete 8,000 updates, corresponding to 4.2 billion training token positions, with the same architecture, data, sequence length, effective batch size, update budget, and evaluation protocol.

Section 4.2 and Appendix D.4 report the following results at update 8,000 (seed 42):

OptimizerValidation loss ↓Perplexity ↓
Muon3.08098221.779784
NorMuon3.08002921.759039
AdaMuon3.12624122.788157
RADO2.98820119.849943

Relative to NorMuon, the lowest-perplexity baseline in this comparison, RADO reduces final validation loss by 0.091828 and perplexity by 8.77%. Figure 3 provides the full validation trajectories. The table compares all methods at the same completed training budget.

To provide a completed and controlled comparison within the available computational resources and rebuttal period, we report results after 8,000 updates (4.2B training tokens). Further scaling experiments are ongoing, and the preliminary validation trends observed thus far continue to favor RADO. We intend to incorporate the completed results, full training trajectories, and corresponding experimental settings into the final manuscript revision.

3. Weight decay 0.1 across the additional pretraining comparison.

As specified in Appendix D.4, the Qwen3-0.6B comparison uses weight decay = 0.1 for both matrix and auxiliary parameters across all four optimizers. The gains above are therefore obtained under the weight-decay setting highlighted in the review.

The smaller-task experiments use their task-specific selected configurations. The additional comparison provides direct evidence of RADO's performance in from-scratch pretraining at 0.6B parameters with weight decay 0.1.

Reviewer_WJYWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Relativistic Adaptive Gradient Descent with Orthogonalization (RADO), which constructs an adaptive matrix direction before applying orthogonalization. It combines RAD-inspired normalization with a coordinate-dependent damping schedule, entrywise signed-log compression, and five Newton–Schulz iterations. The paper characterizes the compression map and operation placement, and evaluates the method on vision classification, language modeling, and multimodal fine-tuning.

Critical strengths
  1. The question of “how should adaptive information enter an orthogonalized matrix update?” is an important question of broad interest, given the effectiveness of coordinate-adaptive and matrix-based optimizers and the growing interest in combining them.
  2. Signed-log compression is an interesting and intuitive design. It reduces coordinate magnitude contrasts while preserving signs and magnitude ordering. It retains dependence on the adaptive denominator, providing a natural intermediate choice between keeping magnitudes unchanged and discarding them.
  3. The empirical results are promising across several tasks, with the main comparisons reporting results over multiple seeds.
Critical weaknesses
  1. My major concern is that NorMuon and AdaMuon are discussed in the introduction as closely related approaches to integrating adaptation and orthogonalization, but neither is included in the experiments. It is unclear whether RADO improves upon existing methods that already combine adaptation and orthogonalization. Comparisons with these directly relevant methods under comparable tuning budgets are important for assessing the proposed integration strategy.
  2. While Section 3.2 states that compression “controls the matrix being transformed while preserving graded coordinate information” and Proposition 1 establishes coordinate-level properties, these do not by themselves explain how retaining magnitude ordering affects the singular directions or the quality of the final NS5 update. The no-compression ablation provides useful performance evidence, but does not isolate why retaining these distinctions helps. A targeted diagnostic, such as robustness to coordinate outliers or final-update alignment with a reference gradient, would better connect the motivation to the observed gains.
  3. The ablations do not fully isolate the contribution of each component. Figure 3 usefully compares damping and compression, but all variants use NS5, and the placement comparisons also change damping strength (Table 5). A matched variant without NS5 and placement comparisons with fixed damping would clarify the gains attributable to orthogonalization and operation order.
  4. As a minor point, I would bring to the authors’ attention a recent related work, “Can Entry-Wise Clipping Give Spectral Control of Stochastic Gradients?” (NeurIPS 2026, arXiv:2605.27733). It also studies entry-wise preprocessing before spectral normalization. Discussing the differences in motivation, transformation, and theoretical guarantees would clarify its relation to RADO.
Questions for authors
  1. What motivates signed-log over other compression maps? Could the authors compare it with hard sign and one smooth alternative, holding the remaining pipeline fixed?
2 comments
AuthorsOpenReview ↗

W1 & W3: Related Optimizer Comparisons and Controlled Ablations

Thank you for recognizing the importance of adaptive input construction and for requesting more direct comparisons and controlled ablations.

W1: Closely related baselines and tuning.

We have added NorMuon and AdaMuon, together with ROOT, to the ViT, Alpaca, and WikiText-103 benchmarks, with results reported over five seeds per optimizer. Appendix C.2 specifies the tuning protocol: each optimizer–task pair uses three learning-rate candidates, {ηc/3,ηc,3ηc}\{\eta_c/3,\eta_c,3\eta_c\}, with optimizer-specific centers. Matrix/LoRA multipliers are held fixed within each grid, native adaptive and shape scaling is retained, and weight decay is selected from the task-specific candidate set. Table 5 and Appendix C.2 report the selected configurations.

The table below highlights the directly relevant baselines from Table 1. Results are mean ± sample standard deviation across five seeds. ViT reports peak validation accuracy; Alpaca and WikiText-103 report final perplexity at a common training budget within each task.

OptimizerViT accuracy (%) ↑Alpaca PPL ↓WikiText-103 PPL ↓
Muon68.04 ± 0.436.110 ± 0.08024.593 ± 0.018
NorMuon68.01 ± 0.446.077 ± 0.00724.670 ± 0.065
ROOT67.25 ± 0.326.191 ± 0.00324.592 ± 0.045
AdaMuon67.77 ± 0.346.233 ± 0.00325.376 ± 0.038
RADO68.97 ± 0.476.017 ± 0.08122.760 ± 0.030

Across all nine optimizers reported in Table 1, RADO achieves the highest mean ViT accuracy and the lowest mean final perplexity on both language benchmarks. These comparisons directly evaluate RADO against the existing integration strategies identified in your review.

The additional Qwen3-0.6B experiment provides a further comparison after 8,000 updates and 4.19B tokens (Table 2, seed 42). Final perplexity is 19.850 for RADO, 21.780 for Muon, 21.759 for NorMuon, and 22.788 for AdaMuon.

W3: Component ablations and operation placement.

The component and placement ablations in Table 8 use three seeds per variant. All variants retaining dynamic damping use the same damping-strength coefficient, 10−410^{-4}, including both placement alternatives. Shared settings are a base learning rate of 3×10−43\times10^{-4}, weight decay of 0.01, an effective batch size of 64, and 10,000 updates. Appendix C.3 and Table 7 specify the update paths.

The ablation results, including the added asinh comparison, are mean ± sample standard deviation:

VariantFinal validation loss ↓
Full RADO3.129160 ± 0.003058
Without dynamic damping3.145858 ± 0.009848
Without compression3.133901 ± 0.004862
Without damping/compression3.132589 ± 0.004369
Signed-log replaced by sign3.131014 ± 0.000500
Signed-log replaced by asinh3.134159 ± 0.002138
Compression after NS53.134226 ± 0.002432
Preconditioning after NS53.137066 ± 0.003172

Using the same damping-strength coefficient removes the damping-strength confound in the placement comparisons. Full RADO achieves lower mean final loss than both post-NS5 alternatives.

Across the four damping/compression combinations, the full construction achieves the lowest mean final loss. The observed pattern suggests that the effect of compression depends on whether dynamic damping is present in this setting, supporting the use of the two components together.

Table 9 additionally reports a three-seed joint-removal comparison using the same full-RADO reference as Table 8. Damping and signed-log are retained, while NS5, parameter grouping, and matrix/shape scaling are jointly removed:

VariantFinal validation loss ↓Final PPL ↓
Full RADO3.129160 ± 0.00305822.854838 ± 0.069872
No NS5/grouping/scaling3.231058 ± 0.00196725.306445 ± 0.049776

Joint removal increases both mean final validation loss and mean final perplexity. This complementary experiment evaluates the combined contribution of NS5, parameter grouping, and matrix/shape scaling.

We address W2, Q1, and W4 in the accompanying response on the compression mechanism, map comparisons, and related work.

AuthorsOpenReview ↗

W2, Q1 & W4: Compression Mechanism, Alternative Maps, and Related Work

Thank you for asking us to connect coordinate-level compression to the final matrix update and for highlighting the related clipping work.

W2: How adaptive magnitudes influence the finite-NS5 update.

Entry-wise preprocessing can change the polar factor of the matrix supplied to NS5. Let P=LΣR⊤P=L\Sigma R^\top be a thin SVD of a full-rank input. By Lemma 1 and Eq. (22) in Appendix B,

=L\\,\mathrm{diag}\\!\left(F_5(\sigma_i/\nu)\right)R^\top, \qquad \nu=\lVert P\rVert_F+\epsilon_{\mathrm{NS}}.$$ where $F_5$ is the fivefold composition of the NS polynomial used in RADO. Its positive gains on nonzero singular values imply, in exact arithmetic, $$\operatorname{polar}(Q)=LR^\top=\operatorname{polar}(P).$$ Here $\operatorname{polar}$ denotes the exact polar factor. Thus preprocessing can alter an orientation preserved by finite NS5. At a fixed input, a positive scalar rescaling of the final update cannot reproduce a change in its polar factor. For the analytical example $$U(s)=\begin{pmatrix}1&s\\\\0&1\end{pmatrix},\qquad s\ge1,$$ the magnitudes of the polar rotation angles after identity, signed-log, and sign preprocessing are $$\theta_{\rm id}=\arctan(s/2),\quad \theta_{\log}=\arctan\\!\frac{\log(1+s)}{2\log2},\quad \theta_{\rm sign}=\arctan(1/2).$$ Our ratio bound gives $\theta_{\rm sign}\le\theta_{\log}\le\theta_{\rm id}$. Signed-log moderates the influence of the increasing coordinate while retaining graded sensitivity to it; sign maps all $s\ge1$ to the same matrix. These polar factors persist through finite NS5. The example isolates a directional effect of preprocessing. To relate the transformed update to the objective gradient $G$, the momentum-tracking error enters through $$\langle G,Q\rangle_F\ge \langle M,Q\rangle_F-\\|M-G\\|_F\\|Q\\|_F.$$ Propositions 3–5 in Appendix B provide sufficient conditions linking the two transformation stages to momentum alignment and then to objective-gradient alignment, with an explicit tracking-error term. **Q1: Why signed-log, and comparison with a smooth alternative.** RADO applies $\Phi(x)=\operatorname{sign}(x)\log(1+|x|)$ entry-wise to $U=M\oslash D$. For fixed $M$ and positive $D$, hard sign removes the direct dependence on $D$ at this stage: $\operatorname{sign}(M\oslash D)=\operatorname{sign}(M)$. Identity retains the full coordinate contrasts. Signed-log provides graded compression: it is approximately linear near zero and logarithmic at large magnitudes, preserving signs, the magnitude ordering of $U$, and dependence on the adaptive denominator. We have added a three-seed comparison with $\psi(x)=\operatorname{asinh}(x)$ using seeds 42, 12345, and 23456. Asinh is a relevant smooth alternative because it also has odd symmetry, strict monotonicity, unit slope at zero, and logarithmic tails. The controlled comparison keeps the remaining RADO pipeline fixed. Results are mean ± sample standard deviation over three seeds per map: | Input map | Final validation loss ↓ | Final PPL ↓ | | --- | --- | --- | | Identity | 3.133901 ± 0.004862 | 22.963569 ± 0.111569 | | Hard sign | 3.131014 ± 0.000500 | 22.897190 ± 0.011453 | | Asinh | 3.134159 ± 0.002138 | 22.969348 ± 0.049106 | | Signed-log | **3.129160 ± 0.003058** | **22.854838 ± 0.069872** | Signed-log achieves the lowest mean final loss and perplexity among all four maps in this setting. Together with the matrix-level analysis above, this controlled comparison supports its use for compressing coordinate contrasts while retaining graded adaptive information. **W4: Relation to Song et al. (2026), arXiv:2605.27733.** Both approaches preprocess entries before a spectral transformation. Song et al. motivate smooth shrinkage $S_c(x)=xe^{-|x|/c}$ through a Gaussian signal prior and heavy-tailed contamination. Its magnitude decreases for $|x|>c$, suppressing extreme entries toward zero. Signed-log is magnitude-monotone and unbounded, preserving graded adaptive information while compressing contrasts. Their pre-clipping convergence analysis applies sample-wise clipping, averaging, and exact matrix sign under a specified Gaussian–Cauchy mixture model; their Muon experiments use Newton–Schulz. RADO's analysis follows adaptive input construction and finite NS5 under explicit alignment and momentum-tracking conditions. The two analyses address different aspects of preprocessing: contamination control and the transmission of adaptive information into matrix updates.
Reviewer_fU3GOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RADO, an optimizer that combines coordinate-wise adaptation with matrix orthogonalization. The method uses dynamic damping to construct an adaptive direction, applies signed-log compression to reduce magnitude differences, and then uses Newton–Schulz iterations for orthogonalization. The authors evaluate RADO on vision classification, language modeling, instruction tuning, and multimodal fine-tuning. The results show improvements over AdamW, Muon, and other tested baselines. The paper also includes ablation studies, efficiency comparisons, and a theoretical analysis of the proposed update.

Critical strengths

The paper presents a simple and interesting approach to combining adaptive optimization with matrix orthogonalization. The motivation for using signed-log compression is clear, and the method is easy to understand. The experiments cover several tasks, including vision, language modeling, and multimodal fine-tuning. RADO achieves consistent improvements over the main tested baselines, and the multiple-seed evaluations provide useful evidence of its effectiveness.

Critical weaknesses

1. Insufficient comparison with closely related optimizers. The paper discusses methods such as NorMuon, AdaMuon, and ROOT, but does not include them in the main experiments. These methods also combine adaptation or robustness mechanisms with matrix orthogonalization. Without these comparisons, it is difficult to determine how much RADO improves over the most relevant existing approaches.

2. Limited ablation analysis. The ablation experiments use only one random seed. More importantly, the operation-placement variants use a different damping strength from the full RADO method. This makes it difficult to isolate the effect of operation ordering. Additional controlled ablations would help clarify which components are responsible for the improvements.

3. Limited theoretical justification for the proposed design. The theoretical analysis mainly establishes basic properties of signed-log compression and provides a conditional descent bound. However, the descent result assumes that the update direction satisfies an alignment condition. The paper does not show that RADO satisfies this condition. It also remains unclear why the proposed combination should outperform other adaptive orthogonalization methods.

Questions for authors
  1. Can the authors compare RADO with more closely related methods, such as NorMuon, AdaMuon, and ROOT? These comparisons would help establish the practical advantages of the proposed method.

  2. Can the authors provide additional ablation experiments with multiple seeds and matched damping strengths? In particular, it would be helpful to isolate the effects of dynamic damping, signed-log compression, and operation ordering.

  3. Can the authors provide stronger theoretical evidence for the proposed update? For example, under what conditions does RADO satisfy the descent-alignment assumption, and why should signed-log compression improve the optimization direction compared with other transformations?

4 comments
AuthorsOpenReview ↗

W1 & Q1 / W2 & Q2: Closely Related Baselines and Three-Seed Ablations

Thank you for highlighting the importance of comparisons with closely related optimizers and controlled ablations. We have expanded the experiments to address both concerns.

W1 & Q1: Comparisons with NorMuon, AdaMuon, and ROOT.

We now include NorMuon, AdaMuon, and ROOT in the main comparisons on ViT-CIFAR100, GPT-2/Alpaca, and GPT-2/WikiText-103, with five seeds per optimizer on each task. We tune the learning rate and weight decay separately for each optimizer and task. Appendix C.2 and Table 5 specify the three-candidate learning-rate grids, optimizer-specific grid centers, and selected configurations.

The table below reports RADO, AdamW, Muon, and the three requested baselines; the complete comparison with all nine optimizers appears in Table 1. Results are reported as mean ± sample standard deviation. ViT reports peak accuracy, while the language tasks report final perplexity at the common training budget within each task.

OptimizerViT accuracy (%) ↑Alpaca PPL ↓WikiText-103 PPL ↓
AdamW64.44 ± 0.486.279 ± 0.08424.361 ± 0.042
Muon68.04 ± 0.436.110 ± 0.08024.593 ± 0.018
NorMuon68.01 ± 0.446.077 ± 0.00724.670 ± 0.065
ROOT67.25 ± 0.326.191 ± 0.00324.592 ± 0.045
AdaMuon67.77 ± 0.346.233 ± 0.00325.376 ± 0.038
RADO68.97 ± 0.476.017 ± 0.08122.760 ± 0.030

RADO achieves the highest mean ViT accuracy and the lowest mean final perplexity on both language tasks among all nine evaluated optimizers, including the three closely related methods highlighted in your review.

We also compare RADO with Muon, NorMuon, and AdaMuon in an additional pretraining experiment using Qwen3-0.6B, trained from scratch for 8,000 updates (4.19B training tokens; Table 2, seed 42):

OptimizerValidation loss ↓PPL ↓
Muon3.08098221.779784
NorMuon3.08002921.759039
AdaMuon3.12624122.788157
RADO2.98820119.849943

RADO's final PPL is 8.77% lower than that of NorMuon, the lowest-PPL baseline in this experiment.

W2 & Q2: Three-seed ablations with matched damping strengths.

All ablation variants now use three seeds. Every variant retaining dynamic damping uses the same damping strength, 10−410^{-4}, including the operation-placement variants. The shared settings are a base learning rate of 3×10−43\times10^{-4}, weight decay of 0.01, an effective batch size of 64, and 10,000 updates.

The controlled comparisons are reported below (Appendix C.3, Table 8; mean ± sample standard deviation). All ablation comparisons use the Full RADO reference evaluated under the same three-seed ablation protocol.

VariantValidation loss ↓PPL ↓
Full RADO3.129160 ± 0.00305822.854838 ± 0.069872
Without dynamic damping3.145858 ± 0.00984823.240350 ± 0.228236
Without compression3.133901 ± 0.00486222.963569 ± 0.111569
Without both3.132589 ± 0.00436922.933431 ± 0.100135
Signed-log replaced by sign3.131014 ± 0.00050022.897190 ± 0.011453
Compression after NS53.134226 ± 0.00243222.970904 ± 0.055906

With the damping strength matched across variants, applying signed-log before NS5 yields a lower mean final PPL than applying it afterward: 22.854838 vs. 22.970904. Relative to Full RADO, removing dynamic damping increases mean PPL to 23.240350, while removing compression increases it to 22.963569. Replacing signed-log with hard sign yields a mean PPL of 22.897190, compared with 22.854838 for signed-log.

Among the four damping–compression combinations, Full RADO achieves the lowest mean validation loss and perplexity. These comparisons support the combined construction used in RADO, in which signed-log compression is applied to the dynamically damped adaptive direction before NS5. We discuss the design rationale and strengthened theoretical analysis in our separate response to W3/Q3.

AuthorsOpenReview ↗

W3 & Q3: Constructive Alignment Analysis and the Mechanism of RADO

Thank you for asking about alignment and the rationale for RADO's construction. We have expanded Appendix B and clarified Sections 2.2-3.4.

1. Constructive alignment conditions and proof strategy.

Lemma 1 analyzes the finite NS5 map, including its normalization stabilizer, in exact arithmetic. It establishes positive spectral gains and ∥NS5(P)∥2≤5/4\|\mathrm{NS5}(P)\|_2\le5/4. Lemma 2 bounds update energy across the matrix and scalar/vector branches.

For nonzero MM and an elementwise positive denominator DD, let P=Φ(M⊘D)P=\Phi(M\oslash D), Q=NS5(P)Q=\mathrm{NS5}(P), a=cos⁡F(M,P)a=\cos_F(M,P), and b=cos⁡F(P,Q)b=\cos_F(P,Q). Proposition 3 proves

\gamma=ab-\sqrt{(1-a^2)(1-b^2)}.$$ Sign preservation and positive NS5 spectral gains give $a,b>0$, so $a^2+b^2>1$ ensures positive alignment. Appendix B.3 gives the sufficient condition $$(\kappa_K-1)(\kappa_H-1)<4\sqrt{\kappa_K\kappa_H},$$ where $\kappa_K,\kappa_H$ are the maximum/minimum positive gain ratios for the coordinate map $M\mapsto P$ and spectral map $P\mapsto Q$. Appendix B.4 gives parameter conditions ensuring a uniform positive margin. For the joint matrix/scalar/vector direction $d_t$, Propositions 4-5 establish $$\langle G_t,d_t\rangle\ge\rho\|G_t\|^2-C_E\|\widehat M_t-G_t\|^2,$$ where $\rho>0$, $C_E\ge0$, $G_t=\nabla J(\theta_{t-1})$, and $\widehat M_t=M_t/(1-\beta_1^t)$. Thus gradient alignment is positive when the tracking-error term is smaller than $\rho\|G_t\|^2$. Lemma 3 controls this error through gradient noise and parameter drift. Under Appendix B's assumptions, smoothness and telescoping yield Theorem 1/Corollary 1: with an admissible $\eta\propto N^{-1/2}$, the average squared-gradient bound is $O(N^{-1/2})$ for full gradients, and the expected bound is $O(N^{-1/2})+O(\sigma^2/B_{\rm mb})$ for fixed minibatches of size $B_{\rm mb}$. The analysis connects descent to explicit gain conditions and quantified momentum-tracking error. **2. Distinction from related optimizers.** - **Muon (Jordan et al., 2024)** orthogonalizes momentum without a coordinate-wise second-moment denominator. - **NorMuon (Li et al., 2026)** uses row-wise second moments of the orthogonalized output, then global rescaling; its Front variant studies adaptation before orthogonalization. - **AdaMuon (Si et al., 2025)** orthogonalizes $\mathrm{sign}(M)$, then applies element-wise adaptation based on output second moments and RMS rescaling. - **ROOT (He et al., 2025)** suppresses momentum outliers before shape-calibrated AdaNewton; $M-\mathrm{softthreshold}(M)$ is entrywise clipped momentum. RADO jointly uses **coordinate-wise gradient-history adaptation and graded compression to construct the input to the final NS5 transformation**. **3. Why signed-log fits this construction.** RADO forms $$U=M\oslash\sqrt{V\odot\delta^2+\zeta},\quad P=\mathrm{sign}(U)\odot\log(1+|U|),\quad Q=\mathrm{NS5}(P).$$ $V$ carries coordinate-wise gradient history, while dynamic damping moderates amplification from small second moments. Together they shape the relative entries submitted to NS5. Identity retains full magnitude contrasts. Hard sign satisfies $\mathrm{sign}(M\oslash D)=\mathrm{sign}(M)$ for $D>0$, removing the denominator's influence at fixed momentum. Clipping saturates beyond its threshold. Signed-log remains strictly increasing and retains graded dependence on adaptive scaling while compressing positive magnitude ratios: $$1<\frac{\log(1+x)}{\log(1+y)}\le\frac{x}{y},\qquad x>y>0.$$ This information can affect the final direction. Coordinate adaptation and compression can change the input's singular-vector factors; the NS5 polynomial transforms singular values while preserving these factors. Within RADO's matrix-direction construction, subsequent positive scalar rescaling preserves each block's direction and singular-value ratios. Unequal coordinate scaling afterward generally changes this geometry; a final scalar learning-rate adjustment cannot, in general, reproduce the directional changes introduced before NS5. Three-seed controls (Appendix C.3, Table 8; mean ± sample SD) give final PPL **22.8548 ± 0.0699** for signed-log, **22.9636 ± 0.1116** for identity, and **22.8972 ± 0.0115** for sign. Signed-log has the lowest mean PPL among these alternatives. Together with the matched placement comparison, these results support retaining moderated adaptive magnitudes before NS5.
Reviewer_fU3GOpenReview ↗

Follow-up after author response

Thank you for the substantial additions. The closely related baselines and matched-damping, multi-seed ablations address much of my original empirical concern. The expanded analysis also replaces the earlier alignment assumption with more explicit sufficient conditions. My remaining question is how restrictive the required uniform angle margin is for the practical configurations evaluated. I recognize that the revision materially strengthens the submission and will take these additions into account in my final assessment.

AuthorsOpenReview ↗

Follow-up on the practical scope of the angle certificate

Thank you for this important follow-up. We examined the practical restrictiveness of the alignment condition using the Full RADO configuration on GPT-2/WikiText-103, across three seeds and 29,088 block-update observations.

The key distinction is between the sufficient angle certificate and realized momentum alignment. Proposition 3 constructs a lower bound by taking the worst-case relative orientation of the components orthogonal to the intermediate direction. Consequently, failure of this certificate does not imply negative momentum alignment.

Our diagnostics confirm that this distinction matters in practice. The certificate is nonpositive for approximately 62% of the inspected block updates, whereas all 29,088 directly measured momentum-update cosines remain positive, with a minimum observed value of 0.021334.

These observations establish that the two-stage sufficient certificate is conservative for the evaluated configuration. They also show that the implemented updates maintain positive momentum alignment at every inspected state, even where the worst-case certificate cannot certify it.

The theoretical analysis provides explicit sufficient conditions for obtaining a uniform alignment margin, while the empirical diagnostics characterize the realized alignment along the sampled training trajectories. We have clarified this distinction in the revised appendix. The stationarity result remains conditional on its stated assumptions, with the connection to the population gradient handled through the separate momentum-tracking analysis.

This comparison helps distinguish conservativeness of the analytical certificate from the alignment behavior of the optimizer itself.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for the Area Chair

Dear Area Chair,

Thank you for overseeing the review. We summarize the contribution of RADO, the evidence added during rebuttal, and how the revisions address the central review concerns.

1. A concrete methodological contribution with empirical support. RADO addresses how coordinate-wise adaptation should shape the matrix supplied to orthogonalization. Dynamic damping and signed-log compression construct an input that retains graded adaptive magnitudes before a finite Newton–Schulz transformation. This preserves dependence on the adaptive denominator that an elementwise sign map would remove. Across the reported five-seed comparisons, RADO achieves the highest mean peak ViT validation accuracy and the lowest mean final perplexity on both GPT-2 tasks. On WikiText-103, mean perplexity is 7.45% lower than Muon and 6.57% lower than AdamW (Table 1).

2. New evidence addressing baseline coverage and component attribution. We added closely related baselines, including NorMuon, ROOT, and AdaMuon, together with matched-damping, multi-seed ablations and alternative compression functions. These additions directly address concerns about comparisons with related optimizers and help assess the compression design under controlled damping settings. The revised manuscript also explains the roles and ordering of the transformations more explicitly.

3. Targeted responses to APtu’s follow-up questions. We expanded the evaluation to Qwen3-0.6B and clarified baseline configurations, optimizer-specific learning-rate conventions, and the reported training budget. In particular, nominal learning-rate coefficients must be interpreted together with each optimizer’s native normalization; their numerical differences alone do not establish corresponding differences in update magnitude. The additional experiment extends the evaluation beyond the original GPT-2 setting, with conclusions restricted to the reported model, single-seed protocol, and evaluation budget.

4. Explicit theoretical conditions and their practical scope. The revised analysis states explicit sufficient alignment conditions. We additionally conducted three-seed diagnostics to investigate the remaining question about their restrictiveness. These show that the sufficient certificate is restrictive in the tested configuration, while directly measured momentum–direction alignment remains positive at every inspected state. We distinguish this empirical observation from the uniform premise required by the conditional theorem.

The discussion reflects these substantive improvements. WJYW increased their rating following the rebuttal. fU3G explicitly acknowledged that the new baselines and matched-damping, multi-seed ablations address much of their original empirical concern, and that the revision “materially strengthens the submission.”

We believe the methodological contribution, empirical results, and targeted additions provide substantive grounds for a positive assessment. We respectfully ask that the revised manuscript and follow-up responses be considered together in the final decision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.