acceptodds
Under review as a conference paper at ICLR 2027

ORCA: The Annealed Spectral Conditioning Optimizer for Faster, Better LLM Training

Abstract

Modern LLM optimizers such as Muon often produce weight matrices with higher effective rank than Adam, yet further spectral control has delivered only modest gains. We identify a tension behind this result: concentrated spectra can suppress gradient directions in coupled weight matrices and slow optimization, while constraints maintained throughout training can limit task-specific adaptation and raise the attainable loss floor. We introduce ORCA (Orthogonal Regularization, Cooled After), a minimal optimizer intervention that applies strong but temporary soft orthogonality regularization early in training, then removes it. This allows the weights to benefit from a broader spectrum early on and adapt freely afterward. Across LLaMA, Qwen3, and fine-grained mixture-of-experts models ranging from 130M to 8B parameters, ORCA achieves lower final validation loss than Muon. Its loss reduction relative to Muon matches or exceeds Muon's reduction relative to Adam. Ablations support the early-shaping, later-release design. Further, ORCA requires no architectural changes and adds minimal overhead.

Primary area:
optimization
Keywords:
Optimization, LLM, Training Method, Optimizer

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_zaMFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ORCA adds soft orthogonality regularization early in Muon training and removes it afterward. Matrix-factorization analysis motivates a trade-off between spectral support during optimization and restrictions on the final solution. Experiments on LLaMA, Qwen3, and MoE models report lower validation loss, downstream improvements, and token-efficiency gains.

Critical strengths

The experiments cover diverse architectures, datasets, and model scales from 130M to 8B parameters. Beyond final losses, effective-rank trajectories, gradient statistics, and controlled spectral interventions provide evidence consistent with the theoretical motivation. These auxiliary measurements help connect the observed gains to the proposed mechanism.

Critical weaknesses
  1. The theory provides limited support for ORCA's schedule in practice. In the simplified matrix-factorization setting, Theorem 2.1 bounds descent speed from above; increasing effective rank relaxes this bound but does not guarantee acceleration. Theorem 2.2 gives a loss lower bound under bounded spectral norm, positive only when effective rank exceeds the number of distinct target classes. These results motivate temporary regularization but do not establish its advantage over Muon or determine when to remove it.

  2. Evidence for release-time transferability remains limited. The 25% default is reused across dense models, and all tested release times improve over Muon on LLaMA 130M, but sensitivity is not studied at larger scales. Both MoE scales use a different, shared layerwise schedule whose selection procedure and sensitivity are unreported. Given the modest validation-loss improvements, it remains unclear whether the gains justify potential schedule searches on new tasks or model scales; this tuning cost is not quantified.

  3. The discussion of relevant prior work is way too brief. Appendix A summarizes orthogonal and spectral training methods, but mainly distinguishes ORCA through its removal of regularization. The limited comparison of theoretical insights and optimization mechanisms makes the novelty relative to these methods difficult to assess.

Questions for authors

How were the MoE release endpoints of 20% and 60% selected, and how sensitive are the reported gains to these choices? What additional compute was spent selecting the regularization strength and release schedule, and how does accounting for it affect the reported training-efficiency gains?

4 comments

We thank the reviewer for the careful reading and constructive comments, and for recognizing the breadth of our experiments and the mechanistic evidence from spectral interventions and gradient statistics. We group the concerns into three main questions and address each below using the original results, additional experiments, and manuscript revisions.


Q1. How were the release schedules chosen? Do they transfer across model scales, and what additional tuning was required?

Thank you for raising the questions of schedule selection and transfer cost. Our release settings can be directly reused across model scales, and the reported gains do not depend on precisely tuned release endpoints. The original 20%–60% schedule reflected two straightforward considerations: preliminary experiments showed improvements with release at both 20% and 60%, and we wanted to distribute releases across layers over a longer interval to make the transition more gradual. In fact, the simple default of global release at 25% already provides a clear improvement on MoE.

(1) Direct comparisons on MoE

The original layerwise schedule primarily aimed to make the release transition smoother. Both global release at 25% and an additional, randomly chosen pair of layerwise endpoints, 15%–40%, also improve the final loss:

Method / release scheduleFinal loss ↓Loss reduction over Muon ↑
Muon2.8006—
ORCA: layerwise 20%–60%2.77690.0237
ORCA: global 25%2.77900.0216
ORCA: layerwise 15%–40%2.77630.0243

Relative to the original schedule, 15%–40% lowers the final loss by 0.0006, while global 25% raises it by 0.0021; all three schedules outperform Muon. These results show that the gains do not depend on the precise 20%–60% endpoints: the simple global 25% setting is already effective.

(2) Additional sensitivity experiments at 35M, 60M, and 350M

To further address sensitivity across scales, we conducted the following experiments. These additional runs use shorter sequences and fewer training steps so that the complete comparisons could be performed during rebuttal. The magnitude of the improvement over Muon may therefore differ from the longer training runs in the paper. Within each scale, comparisons use the corresponding Muon baseline under the same configuration.

Release fraction35M loss ↓60M loss ↓350M loss ↓
Muon baseline3.25363.13332.8772
10%3.24223.12562.8624
15%3.23893.12302.8601
20%3.23753.12142.8597
25% (default)3.23653.12072.8609
35%3.23663.12142.8662

Every release fraction in the table improves over its corresponding Muon baseline. The shared 25% default achieves the best tested result at 35M and 60M. At 350M, its loss is only 0.0012 above the best tested setting, while remaining 0.0163 below Muon.

(3) Original 130M results under the standard training configuration

Section 5 of the original paper already includes release-time ablations on LLaMA 130M using the standard configuration, with sequence length 1024 and 50k training steps:

Method / release fractionFinal validation loss ↓Loss reduction over Muon ↑
Muon baseline2.8512—
10%2.82100.0302
25% (default)2.81060.0406
40%2.81070.0405
55%2.81660.0346

The results at 25% and 40% differ by only 0.0001, and the other tested fractions also improve over Muon. The original standard-configuration results and the additional experiments across scales show a consistent pattern: the effective release range is broad, 25% serves as a reusable default, and obtaining improvements does not require identifying a precise optimal release point.

(4) Hyperparameter transfer and search cost

Regarding selection cost, we directly transferred the regularization strength and default settings validated on small models, without additional searches for the regularization strength or release schedule at each target scale. The two MoE scales also shared the same layerwise schedule. The reported training gains were obtained by reusing these settings. The additional sensitivity experiments test their robustness, and we will make this transfer procedure explicit in the revised manuscript.


Q2. Does the theory establish ORCA’s advantage over Muon and support its release strategy?

Thank you for the careful analysis of the theoretical results. The original theorems separately establish a spectral bound on descent speed and a loss lower bound under persistent constraints, corresponding to the two bottlenecks that ORCA addresses in sequence through shaping and release. This stepwise argument can also be presented more directly through a unified comparison on the same problem.

To this end, we further prove in the revised Appendix F a direct guarantee with regularization explicitly removed at S=t1/4S=t_1/4, a constant fraction of the comparison time. In the same construction, ORCA reaches zero loss at the earlier time t1t_1, Muon still has positive loss at a later time t2t_2, and retaining the regularizer throughout optimization leaves a positive loss floor for all time. This theorem directly establishes an advantage in the actual time to zero loss and effective release fractions, providing a strict comparison against both Muon and persistent regularization.

Informal theorem: Strict advantage of early orthogonal shaping. Consider the exact-polar matrix-factorization flow for W=XYW=XY, with unit task-update rates, no momentum, and no weight decay. Orthogonal shaping adds

\qquad G_{\mathrm{orth}}(Z)=(ZZ^\top-I)Z$$ to each factor’s update. There exist dimension-independent constants $\gamma,c>0$ and $0<t_1<t_2$ such that, for every even $k\ge2$, there exist a nonsymmetric target $A$ and a symmetric initial factor $B_0$. Define the normalized task loss as $$\bar L_t=\frac{1}{2k}\|X_tY_t-A\|_F^2.$$ Starting from $X_0=Y_0=B_0$, ORCA applies shaping with strength $\gamma$ only for $t<S=t_1/4$ and satisfies $$\bar L_{t_1}^{\mathrm{ORCA}}=0.$$ In contrast, unregularized Muon, initialized at any $X_0=B,\ Y_0=B^\top$, satisfies at the later time $$\bar L_{t_2}^{\mathrm{Muon},B}\ge c.$$ Starting from the same $X_0=Y_0=B_0$ and retaining the same shaping term throughout optimization instead gives $$\bar L_t^{\mathrm{pers}}\ge c \qquad\text{for all }t\ge0.$$ All times refer to accumulated optimization time. The formal theorem further establishes the same guarantees for every fixed release fraction $\tau\in[1/16,\,3/8]$, with $S=\tau t_1$. Intuitively, early shaping helps the factors develop directions and spectral structure favorable to subsequent optimization; release then allows that state to be used to complete the fit to the target. **The result carries the original two-bottleneck analysis forward into a strict guarantee on the actual optimization trajectories.** It complements the controlled spectral interventions, persistent-versus-transient comparisons, and shaping-window ablations already in the paper. The complete theorem and proof appear in the revised **Appendix F**. ---

Q3. What distinguishes ORCA from prior orthogonal and spectral methods, in mechanism and contribution?

Thank you for suggesting a deeper comparison of theoretical insights and optimization mechanisms. Following your suggestion, we have added further relevant references to the revised Appendix A and expanded the comparisons of the methods’ control targets, optimization mechanisms, and theoretical insights. These additions make ORCA’s connections to and distinctions from prior work more explicit, emphasizing the lasting effects of early spectral shaping after release and the training gains achieved by the two-stage strategy. The main comparisons are summarized below.

DirectionRepresentative approaches in prior workORCA’s focus
Early weight-spectrum studies and interventionsStudy early spectral evolution or intervene to restore spectral structureActively shape the early spectrum and test its continuing effects on optimization after the intervention is removed
Update spectra: Muon and its variantsOrthogonalize or otherwise modify the spectrum of update directionsRetain the base Muon update and additionally apply early spectral shaping to the weights
Weight spectra: POET, Pion, and related methodsPreserve weight spectra through reparameterization or spectrum-preserving updatesShape the spectrum early, then release it to continue adapting to the task

Our contribution is to show that early weight spectra can be effectively shaped, that the influence can persist into training after release, and that a two-stage design translates this effect into better final results. The original persistent-versus-transient comparisons test the role of release, while Pion-start and DiagSVD examine this behavior through different interventions.

The revised Appendix A also expands the comparison of theoretical perspectives, including existing analyses of update geometry and spectral-norm control, alongside our analyses of coupled gradients and fitting restrictions under persistent constraints. These comparisons explain how the different mechanisms relate to the observed training behavior.


Thank you again for these focused and constructive questions. We have addressed each concern through the MoE schedule comparisons, sensitivity results across scales, clarification of hyperparameter transfer, a direct comparison theorem, and expanded discussion of related mechanisms. We hope these point-by-point responses address your questions and that the additions and clarifications will be helpful in your further assessment of the paper.

Thank you for the detailed response. The abundant additional experiments and analysis address most of my concerns, and I am raising my score to 3.

Reviewer_hmnoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ORCA, an optimizer that applies soft orthogonality regularization during early training and removes it before continuing with Muon. The motivation is that a broader weight spectrum may help optimization early, while persistent constraints can restrict later adaptation. A matrix-factorization analysis examines these two effects. Experiments cover LLaMA, Qwen3, and fine-grained MoE models from 130M to 8B parameters, with learning-rate searches, multi-seed results, downstream evaluation, and ablations on regularization strength and timing. The reported results show lower final validation loss than Muon across the evaluated settings.

Critical strengths

The method is simple and requires no architectural changes. It adds two matrix multiplications per weight matrix during regularization and returns to Muon after release.

The gains are consistent across several architectures. The 130M learning-rate sweep gives each optimizer an independent search under the same budget, and ORCA improves over the best Muon result.

The timing ablations support removing the regularizer after early training. The gain survives optimizer-state reset and holds under both WSD and cosine schedules. The Adam and SOAP results suggest that the intervention is useful beyond Muon.

Critical weaknesses

1.The claim that “early effective-rank shaping drives the observed improvements” is not isolated by the experiments. Regularization also changes singular-value magnitudes and weight norms. The V–O intervention fixes the spectral norm but changes the Frobenius norm and generally the represented function. Without a norm-matched control, the contribution of effective rank remains unclear.

2.Several gains are small, so their reliability is unclear. Multi-seed pretraining results are limited to 130M, and larger-model comparisons do not document separate learning-rate searches for each optimizer.

3.The theory does not fully establish the proposed explanation. Theorem 2.1 bounds descent speed but does not guarantee faster convergence from higher rank. Theorem 2.2 states that high rank can be “incompatible with arbitrarily small loss”, yet its bounded-scale assumption already implies a positive loss floor at every rank. The additional restriction attributable to rank needs clarification.

Questions for authors

1.Can you add a norm-matched control to separate the effect of effective rank from changes in weight scale?

2.What additional loss restriction does Theorem 2.2 establish beyond the positive floor implied by bounded scale?

3.How were larger-model learning rates selected? Can you report paired uncertainty for the downstream differences?

2 comments

Response to Reviewer hmno

Thank you for the careful review and for recognizing ORCA’s simplicity, its consistent gains across architectures, and the evidence from the release-timing ablations. We address all of your questions below with a new norm-matched control, two theoretical comparisons, and additional training and evaluation results.

Q1. Can you add a Frobenius-norm-matched control?

Thank you for suggesting this comparison. We added a Frobenius-norm-matched control, and the results support early effective-rank shaping as a substantial contributor to ORCA’s gain. To complete the comparison during the rebuttal period, we used a smaller batch size than in the main experiments and ran for 40k optimizer steps. During the first 10k steps (25% of training), we progressively rescaled the Q, K, V, and O weight matrices in each attention layer of the Muon run. Each matrix was matched separately to the Frobenius norm of its counterpart in the ORCA run.

MethodEffective rank at 10kEffective rank at 40kFinal validation loss ↓
Muon3403323.123
Muon + early Frobenius-norm matching4023463.104
ORCA5363543.091

Frobenius-norm matching raises effective rank and recovers part of ORCA’s loss reduction. ORCA’s more direct spectral intervention yields a larger gain. At 40k steps, effective rank rises from 332 to 346 with the control and reaches 354 with ORCA. The increase in effective rank helps explain the gain in the norm-matched control. Uniform rescaling preserves relative singular values when applied and affects effective rank through subsequent updates. ORCA reshapes relative singular values directly and lowers final loss by a further 0.013. This result complements the V–O intervention in Fig. 2.1, which reshapes the spectrum at fixed spectral norm.

Q2. What extra loss does high effective rank impose, and why remove the regularizer?

Thank you for asking us to make the comparison explicit. Sufficiently high effective rank raises the best achievable loss under the same norm budget. In Theorem 2.2’s softmax setting, an optimum under a norm constraint alone can align columns that share a label. A high effective-rank requirement limits that concentration. Let KK be the number of distinct labels.

  • Frobenius budget FF. For K<r≤kK < r \le k, requiring effective rank at least rr gives a loss lower bound of kh(tFK/r)kh(t_F\sqrt{K/r}), strictly above the exact optimum under the norm budget alone, kh(tF)kh(t_F). Here h(t)=log⁡(1+(k−1)e−t)h(t)=\log(1+(k-1)e^{-t}) and tF=F/k−1t_F=F/\sqrt{k-1}.
  • Original spectral-norm budget. Under the same bound ∣W∣2≤B\\|W\\|_2 \le B, requiring effective rank at least r>K+1r > K+1 strictly raises the optimal loss. At an optimum under the norm budget alone, centered logits coincide within each label group, and column offsets add at most one to the rank.

Theorem F.1 in the revised Appendix F extends the instantaneous-descent analysis to a strict comparison of time to zero loss. On the same exact-polar matrix-factorization instance, for every release fraction τ∈[1/16,3/8]\tau\in[1/16,3/8] with release time S=τt1S=\tau t_1 and comparison times t1<t2t_1 < t_2, we obtain the following guarantees.

MethodGuarantee
ORCA with release at SSReaches zero loss after release and before t1t_1
Muon, from any X0=B,Y0=B⊤X_0=B,\\ Y_0=B^\topStill has positive loss at t2t_2
Persistent shaping, from ORCA’s initializationHas a positive loss floor for all time

Q3. How consistent are the gains, and how were learning rates chosen?

Thank you for asking about the reliability of the gains and the choice of learning rates. Our experiments show a consistent advantage for ORCA across the tested settings. The experiments provide more evidence of this consistency than we highlighted in the manuscript.

In the larger-model runs, ORCA overtakes Muon after release and maintains lower validation loss through the end of training. This sustained separation in the validation curves provides stronger evidence of a persistent gain than a single final-loss value.

These comparisons span 25k–100k optimizer updates and up to 105B tokens, making them representative of long pretraining runs. Late in training, additional updates yield diminishing reductions in loss, so a small absolute gap can represent a substantial difference in training cost (Kaplan et al., Eq. 5.6). At 1.3B, ORCA reaches the same validation-loss target as standard Muon with 40% fewer tokens (Fig. 4.2a).

For each larger-model setting, we used a shared peak learning rate from the conventional range where Muon and Adam perform well at that model scale. These rates are comparable in magnitude to those reported for similarly sized models in Pythia, TinyLlama, and OLMo, among other pretraining studies.

The 130M sweep also shows robustness to learning-rate choice. With 50k updates and 26.21B tokens per trial, ORCA lowers loss by 0.0354 relative to the best independently tuned Muon run and outperforms Muon at every shared learning rate (Fig. 4.1a). The gain also holds across four seeds and under both WSD and cosine decay.

Downstream evaluation. We average accuracy equally across the five tasks. The paired standard error bound uses the reported per-task standard errors and assumes independent evaluation samples across tasks.

ComparisonMean accuracy gain (pp)Paired standard error upper bound (pp)
ORCA vs. Muon, five tasks+1.141.05

We will incorporate these additions in the revised manuscript. We will put our learning-rate choices in context in Related Work, report the norm-matched control and downstream results in the experimental section, and clarify the comparison under the same norm budget in the discussion of Theorem 2.2, with supporting results in the appendix. These focused additions strengthen the experimental evidence and clarify the theoretical argument. Thank you for pointing out where our presentation could be clearer and for helping us improve the paper. We hope these additions address all of your questions.

Reviewer_5cT4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work observes two competing effects of spectral control (inducing higher effective rank but lower expressiveness). The main motivation of the paper is to leverage the positive effect during early stages of training, and then remove the spectral regularization to remove expressiveness constraint. The motivation leads to a simple scheduled regularizer ORCA. The authors verify the effectiveness of ORCA on extensive experimental settings.

Critical strengths

The observation related to the mixed effects of spectral regularization is interesting and practical. In this work, the observation motivated to derive ORCA that aims to leverage benefits of having higher rank in early stage and then release the regularization to achieve a lower noise in the later stage. The motivation has also been verified in experiments (e.g., Figure 1).

ORCA is tested on extensive empirical settings (three different settings). The experimental setups are also proper (LR sweep, the number of tokens, and downstream evaluation).

Critical weaknesses

The idea of orthogonal regularization and/or spectral control is not novel, which has been extensively and widely studied in optimization. Even for Muon specifically, [Boreiko 2025, Lion 2026, Li 2026, Ma 2026] seem related. ORCA is not compared against the different realizations of the same idea.

  • Boreiko, V., Bu, Z., & Zha, S. (2025). Towards understanding of orthogonalization in muon. In ICML HiLD Workshop.
  • Lion, K., Hübler, F., Li, B., Orvieto, A., & He, N. (2026). Muown: Row-norm control for muon optimization. arXiv preprint arXiv:2605.10797.
  • Li, Z., Liu, L., Liang, C., Chen, W., & Zhao, T. (2026). Normuon: Making muon more efficient and scalable. In ICML.
  • Ma, J., Huang, Y., Chi, Y., & Chen, Y. (2026). Preconditioning benefits of spectral orthogonalization in muon. arXiv preprint arXiv:2601.13474.
Questions for authors

I am wondering if the effectiveness of ORCA on erank and final loss are influenced by a choice of a learning rate schedule.

2 comments

We thank the reviewer for recognizing the practical value of our findings on the mixed effects of spectral regularization, ORCA’s motivation, and the extensive and carefully designed evaluation. We also appreciate your careful reading and constructive questions. Below, we address the contribution and related methods, the connection to existing theory, and dependence on the learning-rate schedule, through discussion and experimental evidence. We hope these responses resolve all of your concerns.

Q1. Clarifying the contribution and comparisons with related methods

Comment. Orthogonal regularization and spectral control are widely studied, and comparisons with related Muon methods would help establish ORCA’s contribution.

Thank you for pointing out that the discussion of our contribution can be clearer. Our core contribution is strong spectral control of the weights early in training, followed by releasing their spectral degrees of freedom later. To our knowledge, the cited methods do not study this strong early weight-spectrum shaping regime.

The paper supports strong early shaping followed by release through extensive ablations and controlled experiments in Section 5, together with the mathematical analysis and intuition in Section 2. We distinguish two groups of comparisons below.

(a) Related Muon optimizer variants: additional comparisons.

The cited works advance important aspects of Muon, including adaptivity, stability, and parallel efficiency:

  • NorMuon integrates update orthogonalization with neuron-wise adaptive scaling, providing a simple and effective way to account for both matrix geometry and balanced update magnitudes.
  • Muown separates weight-row magnitudes from directions, providing a targeted explanation and control mechanism for spectral-norm drift, with important benefits for training stability and hyperparameter robustness.
  • Boreiko et al.’s blockwise Muon (MuonSBW) improves the parallelizability of orthogonalization and studies spectral regularization, learning-rate transfer, and weight-decay scheduling, providing important insights for scalable Muon training.

These works identify different factors underlying Muon’s effectiveness. ORCA focuses on strong early shaping of the weights’ singular-value spectrum and the subsequent release of spectral freedom.

We have completed the following direct comparisons, with ORCA achieving the lowest loss at both model scales:

ModelORCA, 25% active window ↓NorMuon ↓Muown ↓Standard Muon ↓
35M3.23653.25773.25313.2587
60M3.12013.13813.12423.1389

These small-model runs use a learning rate of 10−310^{-3} and 30,000 optimizer steps. The absolute improvements are modest in this setting, but the extended runs help assess whether optimizer benefits persist beyond the initial convergence phase.

(b) Direct weight-spectrum control: comparison already included in the paper.

A defining aspect of ORCA is direct control of the weight spectrum. Among methods sharing this focus, the paper already compares ORCA with Pion, which preserves the weight spectrum throughout training. At their respective best learning rates, the results are:

MethodFinal validation loss ↓
Pion, best LR2.98
ORCA, best LR2.81

This existing result further supports ORCA’s performance advantage over persistent weight-spectrum control, consistent with the spectral-intervention and release experiments in the paper.

These two groups of comparisons separately evaluate ORCA against adaptive Muon variants and persistent weight-spectrum control. We have clarified this distinction in the presentation and discussed the connections to the cited methods in the related-work section.

Q2. Relationship to the theory of Ma et al. (2026)

Comment. Ma et al.’s analysis of spectral orthogonalization appears closely related.

Thank you for highlighting this important connection. The key distinction is the object being controlled: Ma et al. study spectral orthogonalization of gradients/update directions, whereas ORCA directly controls the spectrum of the weight matrices themselves.

Ma et al. provide an important theoretical perspective on the preconditioning effects of spectral orthogonalization, with broad implications for understanding and designing matrix-structured optimizers. The distinction is summarized below:

AspectMa et al. (2026)ORCA
ObjectGradients/update directionsWeight matrices themselves
Theoretical questionExplains the preconditioning benefits of update orthogonalization and establishes condition-number-independent convergence.Analyzes how weight effective rank affects descent speed and endpoint loss, motivating early weight-spectrum control followed by release.

ORCA’s additional focus is therefore weight-spectrum shaping and the timing of its release. We have included Ma et al.’s work in the references of the revised version.

Q3. Dependence on the learning-rate schedule

Question. Could ORCA’s effectiveness depend on the choice of learning-rate schedule?

Thank you for emphasizing the importance of this control. The paper already includes this ablation in Appendix C.2, Table C.2, although this existing result was not clearly communicated in the main text. ORCA improves final validation loss under both WSD and cosine schedules.

Learning-rate scheduleMuon ↓ORCA ↓ORCA − Muon
WSD2.85122.8106−0.0406
Cosine2.85952.8139−0.0456

The comparable improvements show that ORCA’s final-loss advantage is not specific to WSD. Figure 1.1 and the controlled spectral interventions in Section 5 support the effective-rank mechanism, while Appendix C.2 shows that the loss improvement persists when the learning-rate schedule changes. We have added a clearer explanation and cross-reference in the revised main text.


We again thank the reviewer for the recognition and constructive suggestions. The comparisons, theoretical discussion, and existing ablations above respond to each point raised. We hope these clarifications address all of your questions and make the contribution and supporting evidence clearer.

Comments on the paper

2 comments

Dear Reviewers,

As the rebuttal deadline is approaching, please review the authors' responses and participate in the discussion. Kindly indicate whether your concerns have been addressed and update your scores if appropriate.

Thank you for your time and effort!

Best regards,
--AC

We thank the reviewers and the area chair for their careful consideration. We have addressed every concern raised in the three reviews and revised the manuscript throughout to clarify ORCA’s main contributions and the evidence supporting them.

The original submission includes extensive ablations and controlled spectral interventions (Section 5). We supplemented these with release-timing experiments showing that the default 25% release also improves on Muon in MoE, along with a Frobenius-norm-matched control. The revised manuscript covers a broader range of related work and provides a direct theoretical comparison with Muon and persistent regularization (Appendix F). In the rebuttal, we also compared ORCA with other optimization approaches and answered the reviewers’ other technical and experimental questions in detail.

We appreciate the reviewers’ recognition of ORCA and their help in improving the paper. We see strong potential for combining ORCA with some of these approaches and look forward to exploring those combinations in future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.