- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ORCA adds soft orthogonality regularization early in Muon training and removes it afterward. Matrix-factorization analysis motivates a trade-off between spectral support during optimization and restrictions on the final solution. Experiments on LLaMA, Qwen3, and MoE models report lower validation loss, downstream improvements, and token-efficiency gains.
Critical strengths
The experiments cover diverse architectures, datasets, and model scales from 130M to 8B parameters. Beyond final losses, effective-rank trajectories, gradient statistics, and controlled spectral interventions provide evidence consistent with the theoretical motivation. These auxiliary measurements help connect the observed gains to the proposed mechanism.
Critical weaknesses
-
The theory provides limited support for ORCA's schedule in practice. In the simplified matrix-factorization setting, Theorem 2.1 bounds descent speed from above; increasing effective rank relaxes this bound but does not guarantee acceleration. Theorem 2.2 gives a loss lower bound under bounded spectral norm, positive only when effective rank exceeds the number of distinct target classes. These results motivate temporary regularization but do not establish its advantage over Muon or determine when to remove it.
-
Evidence for release-time transferability remains limited. The 25% default is reused across dense models, and all tested release times improve over Muon on LLaMA 130M, but sensitivity is not studied at larger scales. Both MoE scales use a different, shared layerwise schedule whose selection procedure and sensitivity are unreported. Given the modest validation-loss improvements, it remains unclear whether the gains justify potential schedule searches on new tasks or model scales; this tuning cost is not quantified.
-
The discussion of relevant prior work is way too brief. Appendix A summarizes orthogonal and spectral training methods, but mainly distinguishes ORCA through its removal of regularization. The limited comparison of theoretical insights and optimization mechanisms makes the novelty relative to these methods difficult to assess.
Questions for authors
How were the MoE release endpoints of 20% and 60% selected, and how sensitive are the reported gains to these choices? What additional compute was spent selecting the regularization strength and release schedule, and how does accounting for it affect the reported training-efficiency gains?
4 comments
Q2. Does the theory establish ORCA’s advantage over Muon and support its release strategy?
Thank you for the careful analysis of the theoretical results. The original theorems separately establish a spectral bound on descent speed and a loss lower bound under persistent constraints, corresponding to the two bottlenecks that ORCA addresses in sequence through shaping and release. This stepwise argument can also be presented more directly through a unified comparison on the same problem.
To this end, we further prove in the revised Appendix F a direct guarantee with regularization explicitly removed at , a constant fraction of the comparison time. In the same construction, ORCA reaches zero loss at the earlier time , Muon still has positive loss at a later time , and retaining the regularizer throughout optimization leaves a positive loss floor for all time. This theorem directly establishes an advantage in the actual time to zero loss and effective release fractions, providing a strict comparison against both Muon and persistent regularization.
Informal theorem: Strict advantage of early orthogonal shaping. Consider the exact-polar matrix-factorization flow for , with unit task-update rates, no momentum, and no weight decay. Orthogonal shaping adds
\qquad G_{\mathrm{orth}}(Z)=(ZZ^\top-I)Z$$ to each factor’s update. There exist dimension-independent constants $\gamma,c>0$ and $0<t_1<t_2$ such that, for every even $k\ge2$, there exist a nonsymmetric target $A$ and a symmetric initial factor $B_0$. Define the normalized task loss as $$\bar L_t=\frac{1}{2k}\|X_tY_t-A\|_F^2.$$ Starting from $X_0=Y_0=B_0$, ORCA applies shaping with strength $\gamma$ only for $t<S=t_1/4$ and satisfies $$\bar L_{t_1}^{\mathrm{ORCA}}=0.$$ In contrast, unregularized Muon, initialized at any $X_0=B,\ Y_0=B^\top$, satisfies at the later time $$\bar L_{t_2}^{\mathrm{Muon},B}\ge c.$$ Starting from the same $X_0=Y_0=B_0$ and retaining the same shaping term throughout optimization instead gives $$\bar L_t^{\mathrm{pers}}\ge c \qquad\text{for all }t\ge0.$$ All times refer to accumulated optimization time. The formal theorem further establishes the same guarantees for every fixed release fraction $\tau\in[1/16,\,3/8]$, with $S=\tau t_1$. Intuitively, early shaping helps the factors develop directions and spectral structure favorable to subsequent optimization; release then allows that state to be used to complete the fit to the target. **The result carries the original two-bottleneck analysis forward into a strict guarantee on the actual optimization trajectories.** It complements the controlled spectral interventions, persistent-versus-transient comparisons, and shaping-window ablations already in the paper. The complete theorem and proof appear in the revised **Appendix F**. ---Q3. What distinguishes ORCA from prior orthogonal and spectral methods, in mechanism and contribution?
Thank you for suggesting a deeper comparison of theoretical insights and optimization mechanisms. Following your suggestion, we have added further relevant references to the revised Appendix A and expanded the comparisons of the methods’ control targets, optimization mechanisms, and theoretical insights. These additions make ORCA’s connections to and distinctions from prior work more explicit, emphasizing the lasting effects of early spectral shaping after release and the training gains achieved by the two-stage strategy. The main comparisons are summarized below.
| Direction | Representative approaches in prior work | ORCA’s focus |
|---|---|---|
| Early weight-spectrum studies and interventions | Study early spectral evolution or intervene to restore spectral structure | Actively shape the early spectrum and test its continuing effects on optimization after the intervention is removed |
| Update spectra: Muon and its variants | Orthogonalize or otherwise modify the spectrum of update directions | Retain the base Muon update and additionally apply early spectral shaping to the weights |
| Weight spectra: POET, Pion, and related methods | Preserve weight spectra through reparameterization or spectrum-preserving updates | Shape the spectrum early, then release it to continue adapting to the task |
Our contribution is to show that early weight spectra can be effectively shaped, that the influence can persist into training after release, and that a two-stage design translates this effect into better final results. The original persistent-versus-transient comparisons test the role of release, while Pion-start and DiagSVD examine this behavior through different interventions.
The revised Appendix A also expands the comparison of theoretical perspectives, including existing analyses of update geometry and spectral-norm control, alongside our analyses of coupled gradients and fitting restrictions under persistent constraints. These comparisons explain how the different mechanisms relate to the observed training behavior.
Thank you again for these focused and constructive questions. We have addressed each concern through the MoE schedule comparisons, sensitivity results across scales, clarification of hyperparameter transfer, a direct comparison theorem, and expanded discussion of related mechanisms. We hope these point-by-point responses address your questions and that the additions and clarifications will be helpful in your further assessment of the paper.
Thank you for the detailed response. The abundant additional experiments and analysis address most of my concerns, and I am raising my score to 3.
We thank the reviewer for the careful reading and constructive comments, and for recognizing the breadth of our experiments and the mechanistic evidence from spectral interventions and gradient statistics. We group the concerns into three main questions and address each below using the original results, additional experiments, and manuscript revisions.
Q1. How were the release schedules chosen? Do they transfer across model scales, and what additional tuning was required?
Thank you for raising the questions of schedule selection and transfer cost. Our release settings can be directly reused across model scales, and the reported gains do not depend on precisely tuned release endpoints. The original 20%–60% schedule reflected two straightforward considerations: preliminary experiments showed improvements with release at both 20% and 60%, and we wanted to distribute releases across layers over a longer interval to make the transition more gradual. In fact, the simple default of global release at 25% already provides a clear improvement on MoE.
(1) Direct comparisons on MoE
The original layerwise schedule primarily aimed to make the release transition smoother. Both global release at 25% and an additional, randomly chosen pair of layerwise endpoints, 15%–40%, also improve the final loss:
Relative to the original schedule, 15%–40% lowers the final loss by 0.0006, while global 25% raises it by 0.0021; all three schedules outperform Muon. These results show that the gains do not depend on the precise 20%–60% endpoints: the simple global 25% setting is already effective.
(2) Additional sensitivity experiments at 35M, 60M, and 350M
To further address sensitivity across scales, we conducted the following experiments. These additional runs use shorter sequences and fewer training steps so that the complete comparisons could be performed during rebuttal. The magnitude of the improvement over Muon may therefore differ from the longer training runs in the paper. Within each scale, comparisons use the corresponding Muon baseline under the same configuration.
Every release fraction in the table improves over its corresponding Muon baseline. The shared 25% default achieves the best tested result at 35M and 60M. At 350M, its loss is only 0.0012 above the best tested setting, while remaining 0.0163 below Muon.
(3) Original 130M results under the standard training configuration
Section 5 of the original paper already includes release-time ablations on LLaMA 130M using the standard configuration, with sequence length 1024 and 50k training steps:
The results at 25% and 40% differ by only 0.0001, and the other tested fractions also improve over Muon. The original standard-configuration results and the additional experiments across scales show a consistent pattern: the effective release range is broad, 25% serves as a reusable default, and obtaining improvements does not require identifying a precise optimal release point.
(4) Hyperparameter transfer and search cost
Regarding selection cost, we directly transferred the regularization strength and default settings validated on small models, without additional searches for the regularization strength or release schedule at each target scale. The two MoE scales also shared the same layerwise schedule. The reported training gains were obtained by reusing these settings. The additional sensitivity experiments test their robustness, and we will make this transfer procedure explicit in the revised manuscript.