acceptodds
Under review as a conference paper at ICLR 2027

OrScale: Orthogonalized Optimization with Layer-Wise Trust-Ratio Scaling

Abstract

Muon fixes the direction of every matrix-valued update at the polar factor of its momentum, while each layer's step magnitude is addressed only by a static shape correction. We derive a dynamic per-layer scalar by adapting the LARS/LAMB trust-ratio principle to the orthogonalized setting, where the standard denominator candidates—the raw momentum norm or the polar-factor norm—either live in the wrong unit space or carry no update-scale information. The resulting method, OrScale, uses the norm of the parameter-space direction actually applied and anchors each layer's ratio at one via a per-layer calibration, so that the Moonlight recipe (tuned for AdamW, shared with Muon via RMS matching) transfers with no additional sweep; a component ablation confirms each design choice is individually load-bearing. Theoretically, OrScale retains a nuclear-norm convergence rate for any clipped multiplier and achieves a strict layer-adaptive descent gain under two conditions estimable from standard training diagnostics—a bound that predicts the gain should grow with architectural heterogeneity. Experiments confirm the prediction: with every hyperparameter inherited verbatim from the Moonlight recipe, OrScale matches or beats Muon+Moonlight across dense 125M–1.1B FineWeb-Edu pre-training, and on a 16B-A3B mixture-of-experts model—where the logged trust ratios separate cleanly by layer class—the gap widens by an order of magnitude to nats ( relative) at parity wall-clock cost.

Primary area:
optimization
Keywords:
Muon, orthogonalized optimization, trust ratio, layer-wise adaptive learning rates, LLM pre-training, mixture-of-experts

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_J79dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OrScale, a layer-wise trust-ratio extension of the Muon optimizer. Muon determines update directions by orthogonalizing matrix-valued momentum, but it applies similar global step magnitudes across layers. OrScale addresses this limitation by dynamically adjusting the magnitude of each layer’s update according to the parameter norm and the norm of the actual update direction. A per-layer calibration step preserves the initial Muon/Moonlight update scale, while clipping and coupled weight decay control potentially unstable growth. The paper provides convergence results for bounded layer-wise scaling and argues that OrScale can improve optimization when update statistics differ substantially across layers. Experiments on CIFAR-10, dense language models ranging from 125M to 1.1B parameters, and a 16B-A3B mixture-of-experts model show that OrScale generally matches or improves upon Muon with Moonlight. The largest reported improvement is a 0.1305-nat reduction in validation cross-entropy on the MoE model at comparable wall-clock cost, although this result is based on a single random seed.

Critical strengths

1.A simple and implementable modification of Muon. OrScale preserves Muon’s orthogonalized direction and adds only a scalar magnitude adjustment for each layer. The applied-direction denominator is better motivated than using either the raw momentum norm or the post-orthogonalization norm, and the additional computational and memory cost appears small.

2.Broad empirical coverage and useful diagnostics. The paper evaluates the method on vision, dense language models, and a MoE model with large amount of parameters. The reported trust-ratio trajectories and design ablations provide useful evidence that the proposed scaling behaves differently across layer classes, especially in the MoE setting.

Critical weaknesses

1.The motivation and novelty are not sufficiently established. OrScale’s main idea is to apply a LARS/LAMB-style trust ratio to Muon, adding dynamic layer-wise scaling to Muon’s orthogonalized updates. This is a plausible engineering adaptation, but the conceptual advance appears limited. The paper does not convincingly establish that layer-wise relative-step imbalance is a major limitation of Muon in the evaluated settings, or that OrScale addresses this limitation in a way not already captured by simpler layer-wise scaling schemes.

2.The experiments do not cleanly isolate the source or size of the reported gains. OrScale uses dynamic layer-wise scaling, while the main Muon+Moonlight baseline uses a global learning rate and a static shape correction. The zero-sweep protocol demonstrates transfer of a learning-rate recipe, but it does not establish that OrScale outperforms Muon under an equal tuning budget or isolate the benefit of dynamic scaling. Moreover, OrScale couples weight decay into the scaled update, and the paper lacks a matched Muon+Moonlight control with the same weight-decay implementation. This makes it difficult to attribute the gains specifically to the trust ratio.

Questions for authors

1.Can the authors provide a matched control experiment in which Muon+Moonlight uses the same coupled weight-decay implementation as OrScale, while keeping the trust ratio fixed to one? This is necessary to determine whether the reported improvements come from dynamic layer-wise scaling or from the change in how weight decay is applied.

2.How does OrScale compare with Muon+Moonlight under an equal hyperparameter-tuning budget? The zero-sweep protocol demonstrates recipe transfer, but it does not establish that OrScale is superior when both methods are tuned comparably. A comparison with a simple fixed layer-wise scaling baseline would also help assess the contribution of the dynamic trust ratio.

3.What evidence shows that layer-wise relative-step imbalance is a primary limitation of Muon? In particular, can the authors quantify the relative update magnitudes of Muon+Moonlight across layer classes and show that OrScale corrects this imbalance rather than merely benefiting from additional adaptive degrees of freedom?

3 comments

We thank Reviewer J79d for the sharp experimental questions, which have led us to collect detailed per-layer diagnostics that significantly strengthen the empirical picture.

W1: Motivation and novelty. We respectfully push back on the characterization of OrScale as simply "LARS/LAMB applied to Muon." The contribution includes the failure-mode diagnosis and the design of each component to address a specific failure:

  • The standard LARS denominator ∥Wℓ∥F\Vert W _ \ell \Vert _ F fails after orthogonalization because ∥Qℓ∥F\Vert Q _ \ell \Vert _ F carries no information about the parameter's magnitude — it depends only on the shape (Section 3.1, lines 202–215).
  • The correct denominator is the applied-direction norm sℓ∥Qℓ∥Fs _ \ell \Vert Q _ \ell \Vert _ F, which restores the LARS invariance in the orthogonalized setting.
  • The calibration anchor cℓc _ \ell is necessary for zero-sweep recipe transfer. Without it, a 2.5–3$\times$ LR retune is needed and the result still trails Muon+Moonlight (Table 3).
  • Coupled weight decay is necessary for bounded weight norms. Decoupling it causes 10–50$\times$ norm growth (Section 5.5).
  • The component ablation (Table 4) confirms each choice is independently load-bearing: deleting any one causes a distinct failure mode.

A "plausible engineering adaptation" that gets every design choice right on the first try would not produce this pattern of failures.

W2: Experiments do not isolate gain source. We address the three bundled changes:

(a) Dynamic layer-wise scaling. The completed fixed-multiplier baseline (General Response, Section 3) shows that Muon+Moonlight with fixed per-class LR multipliers from OrScale's logged ratios captures 75% of OrScale's gain (0.103 of 0.137 nats), confirming that the per-class allocation pattern is the primary mechanism. Dynamic adaptation adds the remaining 25% (0.034 nats).

(b) Coupled weight decay. With the trust ratio fixed to one (r^ℓ≡1\hat{r} _ \ell \equiv 1), OrScale's coupled update Wℓ←Wℓ−ηr^ℓ(λWℓ+sℓQℓ)W _ \ell \leftarrow W _ \ell - \eta \hat{r} _ \ell (\lambda W _ \ell + s _ \ell Q _ \ell) reduces to Wℓ←(1−ηλ)Wℓ−ηsℓQℓW _ \ell \leftarrow (1 - \eta\lambda) W _ \ell - \eta s _ \ell Q _ \ell, which is Muon+Moonlight's update. The coupling only changes the decay coefficient from ηλ\eta\lambda to ηλr^ℓ\eta\lambda\hat{r} _ \ell; at r^=1\hat{r} = 1 they are identical. We verified this numerically: OrScale with rmin⁡=rmax⁡=1r _ {\min} = r _ {\max} = 1 produces updates that differ from Muon+Moonlight by at most 1.2×10−71.2 \times 10^{-7} (relative) over 20 steps. Thus the existing Muon+Moonlight baseline already is the matched coupled-WD control.

Moreover, the cumulative weight-decay contribution over 596 steps is small: 1.2% for Muon+Moonlight and at most 6.1% for OrScale at r^=5\hat{r} = 5, while the update-induced weight-norm differences between the two runs are 4–5$\times$ for the affected layers (down/output projections). The WD coupling is not the source of the gap.

(c) Calibration anchor. Table 3 already tests this: uncalibrated OrScale requires 2.5–3$\times$ LR retune and still finishes behind Muon+Moonlight.

Q1: Matched WD control. Answered above: the existing Muon+Moonlight baseline is this control (r^≡1\hat{r} \equiv 1 makes the coupled and decoupled updates identical).

Q2: Equal tuning budget. At 125M, Table 7 (Appendix F) shows a matched LR sweep where OrScale is at or below Muon+Moonlight at every stable cell. At 16B, additional ablations are in progress for the revision. We also note that the zero-sweep protocol is the practical comparison: OrScale's value proposition is that gains come without additional sweeps, directly from the Moonlight recipe. The fixed-multiplier result (General Response Section 3) already shows that OrScale's per-class allocation pattern — not merely a global LR shift — accounts for 75% of the gain.

Q3: Evidence for layer-wise step imbalance. We have logged the relative update magnitude ∥ΔWℓ∥F/(ηt∥Wℓ∥F)\Vert \Delta W _ \ell \Vert _ F / (\eta _ t \Vert W _ \ell \Vert _ F) per layer class on both the OrScale (R1) and Muon+Moonlight (R2) seed-43 runs:

StepR2 (Muon): max/minR2: CVR1 (OrScale): max/minR1: CV
1011.6$\times$1.009.9$\times$0.98
606.4$\times$0.699.8$\times$0.97
2003.2$\times$0.252.4$\times$0.24
3002.2$\times$0.211.9$\times$0.18
5002.2$\times$0.232.8$\times$0.28

The initial imbalance is real and substantial: MLP/expert down projections (initialized with small norms) have 5–12$\times$ larger relative updates than gate/up projections. In Muon+Moonlight, this imbalance naturally decreases as weight norms grow, reaching ${\sim}22\times$ by step 300. OrScale does not reduce the imbalance faster; instead, it explicitly amplifies the down/output projections by ${\sim}55\times$ (clipped trust ratio) while scaling other layers by 0.8–2.2$\times$.

The fixed-multiplier ablation (General Response, Section 3) now provides the precise attribution: 75% of OrScale's 0.137-nat gain comes from the per-class allocation (R3 captures 0.103 nats with fixed multipliers), and 25% from dynamic adaptation. The per-class pattern — concentrating large multipliers on down/output projections while leaving gate/up projections near 1$\times$ — is the primary mechanism. The dense results (Table 1) are consistent with this picture: on architecturally homogeneous models, where the headroom dispersion τg\tau _ g is small, OrScale's gain is modest (${\sim}$0.01 nats). The gain grows substantially on the MoE model, where layer classes have markedly different update statistics — exactly the regime where per-class scaling matters most.

AuthorsOpenReview ↗

Gentle follow-up before the discussion period closes

Dear Reviewer J79d,

We wanted to gently follow up before the discussion period closes. We have posted detailed replies to each of your concerns and uploaded a revised paper with all changes incorporated. A quick summary:

  1. Gain-source isolation (W2, Q1, Q2): A fixed-multiplier baseline at 16B captures 75% of OrScale's gain (0.103 of 0.137 nats) using per-class LR multipliers from OrScale's logged trust ratios, confirming the per-class allocation as the primary mechanism. For the coupled-WD concern (Q1): at r^ℓ≡1\hat{r} _ \ell \equiv 1 the coupled update reduces exactly to Muon+Moonlight's, so the existing baseline is the matched control.
  2. Layer-wise diagnostics (Q3): Per-layer-class relative update magnitudes are now logged and included in the revision, showing the initial 5--12$\times$ imbalance and how OrScale redistributes step sizes across layer classes.
  3. Second MoE seed: Seed 43 confirms the gap (0.137 nats, consistent with seed 42's 0.131).
  4. Novelty (W1): The revision more clearly presents the failure-mode diagnosis and why each design choice is independently necessary (Table 4).

We would be grateful for any further feedback before the discussion concludes.

Reviewer_upcVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes OrScale, a layerwise trust-ratio extension of Muon that adapts the magnitude of orthogonalized updates. It uses the norm of the applied update direction, coupled weight decay, and clipping, with calibration to align the initial update with Muon+Moonlight. The paper analyzes failure modes of naive hybrids, presents convergence analysis, and claims a gain from layerwise adaptation. Experiments report improved CIFAR-10 accuracy, modest gains at most dense language model scales, and a larger validation-loss reduction on an MoE model.

Critical strengths
  1. Diagnosis of why naive Muon + trust-ratio hybrids fail. The unit analysis and Theorem 4 explain why momentum-based ratios saturate at the clip bound and collapse to naïve Muon.

  2. Maybe cheap drop-in method. If the method consistently delivers substantial gains across diverse settings, including MoE architectures, rather than only under limited experimental conditions, it could offer significant practical value.

Critical weaknesses
  1. Theorem 3 needs to be corrected. Consider the noiseless case with λ=μ=0\lambda=\mu=0 and two layers satisfying b1=b2=1b_1=b_2=1, a=x=(2,3)a=x=(2,3), and r^=(0.5,1.5)\hat r=(0.5,1.5). These multipliers satisfy the default clipping bounds in both the LM and vision settings. We have ρ=1\rho=1, νr^=1/4\nu_{\hat r}=1/4, and νb=0\nu_{\sqrt b}=0, so conditions (S1)–(S2) hold and, with δ(0)=0\delta(0)=0, Theorem 3 gives κeff≥5/4\kappa_{\mathrm{eff}}\ge 5/4. However, the descent functional yields Φ(r^)=121/10\Phi(\hat r)=121/10 and Φ(1)=25/2\Phi(\mathbf{1})=25/2, hence κeff=121/125=0.968<1\kappa_{\mathrm{eff}}=121/125=0.968<1. Positive correlation therefore does not guarantee better alignment with the optimum than uniform scaling: the multipliers can overemphasize one layer relative to the optimal weighting. This contradicts Theorem 3 as stated: conditions (S1)–(S2) are insufficient for the claimed gain bound.

  2. The MoE gain is a single-seed run and has only 596 steps. In addition, to test whether changing layerwise multipliers during training provides an additional benefit, OrScale should be compared with two Muon+Moonlight baselines: one with a tuned learning rate for matrix parameters, and another with fixed multipliers for expert, attention, and router layers.

Questions for authors

Questions

  1. Does the counterexample to Theorem 3 presented above hold? If so, what additional conditions would be needed to establish a valid gain bound? Alternatively, should Theorem 3 be removed? If removal is necessary, the theoretical contribution of the paper would be significantly weakened.

  2. At 16B, can you compare OrScale with two Muon+Moonlight baselines: one with a tuned matrix LR, and another with fixed multipliers for expert, attention, and router layers, while keeping the AdamW fallback LR, token budget, and schedule unchanged?

  3. Is there multi-seed or other-scale MoE evidence with more optimizer steps?

Suggestions

  1. Cite and compare related parameter-relative step-size methods: Adafactor, Fromage, Madam, and Nero.

  2. Add downstream evaluations.

  3. Move the MoE batch and step details into the main text.

2 comments

We thank Reviewer upcV for the most technically detailed review.

W1: Theorem 3 counterexample. We agree that the counterexample is correct: (a,r^)=((2,3),(0.5,1.5))(a, \hat{r}) = ((2,3), (0.5, 1.5)) gives κeff=121/125=0.968<1\kappa _ {\mathrm{eff}} = 121/125 = 0.968 < 1 despite satisfying (S1)–(S2). The corrected theorem (General Response, Section 1) replaces the bound with an exact identity. In the reviewer's example, τg=1/5\tau _ g = 1/5, τr=1/2\tau _ r = 1/2, ρb=1\rho _ b = 1, and τmax⁡=5/12\tau _ {\max} = 5/12; the multiplier ratio of 3 exceeds the admissible range (1,17/7)(1, 17/7). A gain requires r^2/r^1<17/7≈2.43\hat{r} _ 2/\hat{r} _ 1 < 17/7 \approx 2.43 — that is, the multipliers must not be more dispersed than the headroom allows.

W2: Single seed + baselines needed. See General Response, Sections 2–3. The second seed confirms the gap (0.137 nats), and we have completed a fixed-multiplier baseline at 16B: Muon+Moonlight with fixed per-class LR multipliers from OrScale's logged ratios (R3) captures 75% of OrScale's gain (0.103 of 0.137 nats).

Q1: Should Theorem 3 be removed? We have corrected it rather than removed it. The corrected statement is strictly stronger: an exact identity (no approximation error δ(λ)\delta(\lambda)) with necessary and sufficient conditions (no room for further tightening). The qualitative prediction that the gain ceiling and admissible range grow with architectural heterogeneity is preserved in a sharper form (monotonicity in τg\tau _ g, Part 3 of the corrected theorem). This prediction remains falsifiable from per-tensor diagnostics.

Q2: OrScale vs. tuned-LR and fixed-multiplier baselines at 16B. The fixed-multiplier baseline is now complete (General Response, Section 3):

  • *Fixed per-class multipliers *: routed expert gate/up 1.1, routed expert down 4.5, attention o 3.7, router 1.7, kv_a 0.9, kv_b 0.8, etc. (LR-weighted means from OrScale's logged trust ratios). Result: 3.3659, capturing 75% of OrScale's 0.137-nat gain. The per-class allocation pattern is the primary mechanism; dynamic adaptation contributes the remaining 25% (R1 vs. R3: 0.034 nats).

Q3: Multi-seed, more steps. See General Response, Section 2. We have a second seed; additional seeds are in progress. On "more steps": the 596-step / 10B-token schedule is the standard Moonlight recipe scale and matches our baseline exactly. Running longer would change the comparison (both methods would need re-optimization for the longer schedule).

S1: Citations. We will add Adafactor (Shazeer & Stern, 2018), Fromage (Bernstein et al., 2023), Madam (Modoranu et al., 2024), and Nero (Liu et al., 2021) to Section 2.

S2: Downstream evaluations. Downstream evaluations (HellaSwag, ARC, PIQA, etc.) on MoE checkpoints are in progress and will be included in the revision.

S3: MoE training details. We will move the MoE batch size, step count, and hardware details from Appendix B into the main text (Section 5.3).

AuthorsOpenReview ↗

Gentle follow-up before the discussion period closes

Dear Reviewer upcV,

We wanted to gently follow up before the discussion period closes. We have posted detailed replies to each of your concerns and uploaded a revised paper with all changes incorporated. A quick summary:

  1. Theorem 3 (W1, Q1): Your counterexample was correct and instrumental. We replaced the bound with an exact identity giving necessary-and-sufficient conditions for κeff>1\kappa _ {\mathrm{eff}} > 1, now integrated into the revision.
  2. Second MoE seed (W2, Q3): Seed 43 confirms the gap (0.137 nats, consistent with seed 42's 0.131).
  3. Fixed-multiplier baseline at 16B (Q2): Muon+Moonlight with fixed per-class LR multipliers from OrScale's logged trust ratios captures 75% of OrScale's gain, confirming the per-class allocation as the primary mechanism.
  4. Writing fixes (S1--S3): All addressed --- added citations, moved MoE details to main text, corrected clip-bound characterization.

We would be grateful for any further feedback before the discussion concludes.

Reviewer_FX81OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

OrScale adds layer-wise scaling to Muon, using a calibrated trust ratio to adjust update magnitudes while retaining the orthogonalized direction. The calibration allows reuse of the Moonlight training recipe. Experiments show modest gains on dense language models and CIFAR-10, and a larger improvement on a 16B MoE model with little additional runtime cost.

Critical strengths
  1. The method is simple and inexpensive. Matching the initial Muon+Moonlight update makes it convenient to try with an existing training recipe.
  2. The experiments cover several model sizes. The MoE improvement of 0.1305 nats at comparable training time is promising.
Critical weaknesses
  1. The strict-gain guarantee appears incorrect under the stated conditions. A two-layer example with equal curvature, gradient magnitudes of 1 and 2, and multipliers of 0.5 and 5 satisfies both conditions but gives a worse descent bound than uniform scaling. This undermines the paper’s main theoretical explanation.
  2. The MoE experiments show that layer types receive different multipliers, but do not verify the alignment condition required by the theorem. Since these multipliers also increase effective learning rates and weight decay, the current comparisons do not establish how much of the gain comes from layer-specific adaptation. The headline result is also limited to one seed and 596 steps.
Questions for authors
  1. Is there an additional assumption that rules out the counterexample above?
  2. Was the theorem’s alignment condition actually measured on the MoE run? If so, how?
  3. Does the gain persist against a Muon baseline with a shared multiplier matched to OrScale’s overall update scale?
3 comments

We thank Reviewer FX81 for the clear summary and for identifying the counterexample that, together with Reviewer upcV's, led us to a substantially stronger theorem.

W1: Theorem 3 counterexample. We agree that the counterexample is correct. Please see the General Response (Section 1) for the corrected theorem, which replaces the loose lower bound with an exact identity and provides a necessary-and-sufficient condition for κeff>1\kappa _ {\mathrm{eff}} > 1. In the reviewer's example (a=(1,2)a = (1,2), r^=(0.5,5)\hat{r} = (0.5, 5)), the multipliers point in the right direction (ρb=1\rho _ b = 1) but overshoot: τr=9/11\tau _ r = 9/11 exceeds τmax⁡=3/4\tau _ {\max} = 3/4, and the identity gives κeff=98/101≈0.970\kappa _ {\mathrm{eff}} = 98/101 \approx 0.970.

W2: MoE experiments. We address the three sub-concerns:

(a) Alignment condition not verified. The reviewer is correct that the submission showed the trust ratios separating by layer class but did not directly measure the alignment condition. The corrected theorem requires curvature-weighted statistics (ρb\rho _ b, τr\tau _ r, τg\tau _ g) rather than the original (S1)–(S2). We have collected the per-tensor diagnostics needed to compute these on the seed-43 replicate and will include the full measurement in the revision (General Response, Section 1).

(b) LR/WD confound. OrScale's aggregate update norm is approximately 3$\times$ Muon+Moonlight's (measured by the update-norm-matched shared multiplier m∗≈3.0m^\ast \approx 3.0). We have now completed a fixed-multiplier baseline (General Response, Section 3): Muon+Moonlight with fixed per-class LR multipliers taken from OrScale's logged trust ratios achieves 3.3659, capturing 75% of OrScale's gain (0.103 of 0.137 nats). This confirms that the per-class scaling pattern — not solely the overall step-size increase — is the primary mechanism. The remaining 25% comes from OrScale's dynamic per-step adaptation. We note that the existing ∥Qℓ∥F\Vert Q _ \ell \Vert _ F-denominator row of Table 4 already tests uniform scaling at 125M and shows a meaningful loss relative to OrScale.

(c) Single seed, 596 steps. See General Response, Section 2. The second seed confirms the gap (0.137 nats, consistent with the submission's 0.1305). The 596-step / 10B-token budget matches the Moonlight recipe's standard scale; extending it would require re-optimizing the baseline.

Q1: Additional assumption. Yes: the dispersion condition (S3), τr<τmax⁡\tau _ r < \tau _ {\max} (General Response, Section 1). Since the corrected statement is an equivalence, no weaker condition exists.

Q2: Alignment condition measured on MoE. We have collected per-tensor diagnostics (r^ℓ\hat{r} _ \ell, ⟨Gℓ,Qℓ⟩\langle G _ \ell, Q _ \ell \rangle, ∥Qℓ∥F\Vert Q _ \ell \Vert _ F) on the seed-43 replicate for all 5,207 matrices at every 10th step. Computing the corrected statistics requires a proxy for the unobservable per-layer smoothness LℓL _ \ell; we will include the analysis under multiple proxies in the revised Section 5.3.

Q3: Muon with shared multiplier matched to OrScale's overall scale. At 125M, the ∥Qℓ∥F\Vert Q _ \ell \Vert _ F-denominator row of Table 4 is this experiment: it sets κeff=1\kappa _ {\mathrm{eff}} = 1 (a uniform multiplier equivalent to Muon with a relabeled LR) and loses meaningfully to OrScale. At 16B, the fixed-multiplier baseline (General Response Section 3) already shows that OrScale's per-class allocation pattern — not merely the overall step-size increase — drives 75% of the gain.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' careful correction of Theorem 3, as well as the second MoE seed confirming the reported improvement. These substantially address my original concerns about theoretical soundness and experimental reproducibility. The new fixed per-class multiplier ablation is particularly informative. It strengthens the empirical evidence, but also shows that static per-class scaling accounts for approximately 75% of OrScale's gain. This raises some uncertainty about how much of the remaining improvement can be attributed specifically to dynamic adaptation rather than a simpler scaling schedule.

Overall, the rebuttal has significantly improved my assessment of the paper. While my original soundness concerns have been largely addressed, I still have some reservations about the incremental contribution of the proposed dynamic mechanism over simpler scaling alternatives. I therefore rise the soundness score, but maintain the overall rating of 2, although I now consider the paper much closer to the acceptance threshold.

We thank the reviewer for the careful re-assessment and for raising the soundness score. We address your remaining concern below.

The fixed multipliers are OrScale's output, not an independent method. The 14 per-class multipliers in R4 (ranging from 0.84 for attention kv_b to 4.92 for expert-down) were extracted from OrScale's own logged trust ratios. Without running OrScale first, it would be difficult to determine that expert-down projections need ${\sim}4.54.5\times$ amplification while attention kv_a and kv_b should be scaled down to 0.90 and 0.84. We therefore view the 75% result as evidence that OrScale correctly discovers the right per-class allocation — the fixed-multiplier baseline validates this discovery rather than offering an independent alternative, since it relies on OrScale's output to set its multipliers.

The discovered allocation is non-trivial. The multipliers span a 6$\times$ range (0.84 to 4.92) across 14 layer classes, with some classes receiving multipliers below 1. This is not "uniformly scale the learning rate": it is a specific redistribution of learning-rate budget that depends on the architecture. A uniform 3$\times$ LR increase (matching OrScale's aggregate update scale m∗≈3m^\ast \approx 3) would over-scale the kv layers and under-scale the down projections. For reference, the Muon+Moonlight base LR (4.2×10−44.2 \times 10^{-4}) is taken directly from the Moonlight paper's production pretrain recipe and is presumably already well-tuned; a simple uniform multiplier applied on top of it would need its own sweep.

The 25% likely underestimates the dynamic contribution. R4 uses time-averaged multipliers, which are suboptimal during the critical early phase: the down projections ramp from r^≈2\hat{r} \approx 2 at step 60 to the clip bound r^=5\hat{r} = 5 by step 100--200, while R4 uses the time-average ${\sim}$4.5 throughout. More broadly, OrScale's practical value lies in zero-sweep recipe transfer: it derives these multipliers automatically for any architecture, without manual per-class tuning. The dense-to-MoE transfer in the paper (Sections 5.1--5.3) demonstrates this directly — the same OrScale hyperparameters produce a 0.01-nat gain on dense models and a 0.134-nat gain on MoE, with no architecture-specific adjustment.

We hope this clarifies the relationship between the fixed-multiplier ablation and OrScale's dynamic mechanism, and we welcome any further questions.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response to All Reviewers (1/3)

We thank all three reviewers for their detailed and constructive feedback. We address the main themes below: (1) the corrected Theorem 3, (2) a second MoE seed, and (3) a fixed-multiplier ablation at 16B.

1. Corrected Theorem 3 (FX81-W1, FX81-Q1, upcV-W1, upcV-Q1)

Both counterexamples are correct, and we thank Reviewers FX81 and upcV for them. Theorem 3 as stated is false; we replace it with an exact identity strictly stronger than the original bound.

What was wrong. We found three independent errors in App. A.5:

  1. (E1, l.742–747) The proof bounded the numerator of κeff\kappa _ {\mathrm{eff}} from below but not the denominator from above. The resulting expression grows without limit in νr^\nu _ {\hat{r}} and can exceed κlayer\kappa _ {\mathrm{layer}} of Theorem 2 — as it does in both examples: 1.25 against κlayer=26/25\kappa _ {\mathrm{layer}} = 26/25 (upcV) and 1.67 against 10/910/9 (FX81).

  2. (E2, l.740–741) The τr2\tau _ r^2–νr^\nu _ {\hat{r}} inequality fails when bℓb _ \ell varies: a uniform r^≡1\hat{r} \equiv 1 satisfies (S1)–(S2) with heterogeneous bb, yet κeff=1\kappa _ {\mathrm{eff}} = 1. This error (not identified by the reviewers) means that simply adding a third condition to (S1)–(S2) would still be incorrect.

  3. (E3, l.738–739) The unweighted statistics in (S1)–(S2) do not match the b\sqrt{b}-projection when bℓb _ \ell varies. The corrected statement uses curvature-weighted statistics, coinciding with the originals when bℓb _ \ell is constant (as in both counterexamples).

Corrected Theorem 3 (Exact gain of a layer-wise multiplier). Let gℓ=aℓ/bℓg _ \ell = a _ \ell / b _ \ell be the signal-to-smoothness ratio. Let Eb\mathbb{E} _ b, Varb\mathrm{Var} _ b, corrb\mathrm{corr} _ b denote statistics under curvature weights bℓ/∑kbkb _ \ell / \sum _ k b _ k, and Ea\mathbb{E} _ a the mean under gradient weights aℓ/∑kaka _ \ell / \sum _ k a _ k. Define:

  • multiplier dispersion: τr=Varb[r^] / Eb[r^]\tau _ r = \sqrt{\mathrm{Var} _ b[\hat{r}]} \thinspace/\thinspace \mathbb{E} _ b[\hat{r}]
  • headroom dispersion: τg=Varb[g] / Eb[g]\tau _ g = \sqrt{\mathrm{Var} _ b[g]} \thinspace/\thinspace \mathbb{E} _ b[g], so κlayer=1+τg2\kappa _ {\mathrm{layer}} = 1 + \tau _ g^2
  • alignment: ρb=corrb(r^,g)\rho _ b = \mathrm{corr} _ b(\hat{r}, g)

Then for any multipliers r^ℓ>0\hat{r} _ \ell > 0:

κeff(r^) = (Ea[r^])2Eb[r^2] = (1+ρb τr τg)21+τr2.\kappa _ {\mathrm{eff}}(\hat{r}) \ =\ \frac{(\mathbb{E} _ a[\hat{r}])^2}{\mathbb{E} _ b[\hat{r}^2]} \ =\ \frac{(1 + \rho _ b \thinspace \tau _ r \thinspace \tau _ g)^2}{1 + \tau _ r^2}.

Consequently:

  1. Gain condition (equivalence). For τr>0\tau _ r > 0: κeff>1\kappa _ {\mathrm{eff}} > 1 if and only if (S2$'$) ρb>0\rho _ b > 0 and (S3) τr<τmax⁡:=2ρbτg/(1−ρb2τg2)\tau _ r < \tau _ {\max} := 2\rho _ b \tau _ g / (1 - \rho _ b^2 \tau _ g^2) (no upper limit when ρbτg≥1\rho _ b \tau _ g \ge 1). Equivalently, Ea[r^]>RMSb[r^]\mathbb{E} _ a[\hat{r}] > \mathrm{RMS} _ b[\hat{r}].
  2. Ceiling. κeff≤1+ρb2τg2≤1+τg2=κlayer\kappa _ {\mathrm{eff}} \le 1 + \rho _ b^2 \tau _ g^2 \le 1 + \tau _ g^2 = \kappa _ {\mathrm{layer}}, with equality in the first step at τr=ρbτg\tau _ r = \rho _ b \tau _ g.
  3. Monotonicity. For fixed ρb,τr>0\rho _ b, \tau _ r > 0, both κeff\kappa _ {\mathrm{eff}} and τmax⁡\tau _ {\max} are strictly increasing in τg\tau _ g.

Proof sketch. Writing aℓ=bℓgℓa _ \ell = b _ \ell g _ \ell gives κeff=Eb[r^g]2/(Eb[g]2 Eb[r^2])\kappa _ {\mathrm{eff}} = \mathbb{E} _ b[\hat{r} g]^2 / (\mathbb{E} _ b[g]^2 \thinspace \mathbb{E} _ b[\hat{r}^2]). Expanding Eb[r^g]=Eb[r^]Eb[g](1+ρbτrτg)\mathbb{E} _ b[\hat{r} g] = \mathbb{E} _ b[\hat{r}] \mathbb{E} _ b[g] (1 + \rho _ b \tau _ r \tau _ g) and Eb[r^2]=Eb[r^]2(1+τr2)\mathbb{E} _ b[\hat{r}^2] = \mathbb{E} _ b[\hat{r}]^2 (1 + \tau _ r^2) yields the identity. There is no δ(λ)\delta(\lambda) remainder: the identity holds exactly for any logged, clipped r^\hat{r}.

AuthorsOpenReview ↗

General Response to All Reviewers (2/3)

Verification on the reviewers' examples. In both examples b=(1,1)b = (1,1) (curvature-weighted = unweighted) and ρb=1\rho _ b = 1, but the multipliers overshoot:

upcVFX81
(a,r^)(a, \hat{r})(2,3), (0.5,1.5)(2,3),\ (0.5,1.5)(1,2), (0.5,5)(1,2),\ (0.5,5)
τg, κlayer\tau _ g,\ \kappa _ {\mathrm{layer}}1/5, 26/251/5,\ 26/251/3, 10/91/3,\ 10/9
τr\tau _ r vs. τmax⁡\tau _ {\max}1/2>5/121/2 > 5/129/11>3/49/11 > 3/4
Identity gives(1.1)2/1.25=0.968(1.1)^2/1.25 = 0.968(14/11)2/(202/121)=0.970(14/11)^2/(202/121) = 0.970
Gain requires r^2/r^1∈\hat{r} _ 2/\hat{r} _ 1 \in(1, 17/7≈2.43)(1,\ 17/7 \approx 2.43)(1, 7)(1,\ 7)
Example uses310

Is there an additional assumption? (FX81-Q1, upcV-Q1) Yes: condition (S3), τr<τmax⁡\tau _ r < \tau _ {\max}. Since the corrected statement is an equivalence, no weaker condition suffices. Intuitively, the gain is linear in τr\tau _ r while the cost is quadratic; beyond τmax⁡\tau _ {\max}, multiplier dispersion exceeds what the aligned headroom can absorb.

What changes in the paper. Theorems 1, 2, 4 and Corollary 1 are unchanged. The abstract and introduction state the exact equivalence rather than claiming a gain "under two conditions." The qualitative prediction that gain grows with heterogeneity is preserved: both the ceiling 1+ρb2τg21 + \rho _ b^2 \tau _ g^2 and τmax⁡\tau _ {\max} increase with τg2=κlayer−1\tau _ g^2 = \kappa _ {\mathrm{layer}} - 1. We have collected per-tensor diagnostics on the seed-43 replicate (Section 2) to compute the corrected statistics and will include the full verification — along with a sensitivity analysis over smoothness proxies for the unobservable LℓL _ \ell — in the revised Section 5.3.

2. Second MoE Seed (FX81-W2, upcV-W2, upcV-Q3, J79d)

We have replicated the 16B-A3B MoE experiment with seed 43. The validation cross-entropy gap is consistent:

seed 42 (submission)seed 43 (new)two-seed mean
OrScale3.34103.33203.3365
Muon+Moonlight3.47153.46903.4703
Gap (nats)0.1305 (3.76%)0.1370 (3.95%)0.134

The gap of 0.134 nats is 15$\times$ larger than the single-arm seed variation (0.009 for OrScale, 0.003 for Muon+Moonlight). The validation curves have identical shape across seeds: OrScale trails slightly at step 100, leads from step 150, peaks at ${\sim}$0.31 nats around step 200–250, and converges to ${\sim}$0.14 nats. OrScale at step 400 (6.7B tokens) already reaches Muon+Moonlight's final loss.

We also collected per-tensor diagnostics (trust ratios, gradient–update inner products, update norms) at every 10th step for all 5,207 matrices. The qualitative structure of the submission's Figure 4 is confirmed: trust ratios separate by layer class, with down/output projections reaching the clip bound rmax⁡=5r _ {\max} = 5 by step 100–200 and remaining there. We note that the 33% of matrices at the clip bound is caused by the hard clip (rmax⁡=5r _ {\max} = 5), not the coupled-weight-decay ceiling (the ceiling 1/(cℓλ)1/(c _ \ell \lambda) ranges from 157 to 1867 across tensors) — we will correct this characterization in the revision.

AuthorsOpenReview ↗

General Response to All Reviewers (3/3)

3. Fixed-Multiplier Ablation at 16B (upcV-Q2, J79d-Q2, J79d-W1)

To test whether OrScale's gain comes from the specific per-class allocation or solely from dynamic adaptation, we ran Muon+Moonlight with fixed per-class LR multipliers taken from OrScale's logged trust ratios (R3). The setup is identical to Section 2 (seed 43, 596 steps); the only change is that each projection type receives a constant LR multiplier equal to the LR-weighted time mean of R1's clipped trust ratio for that class:

Projection classMultiplierProjection classMultiplier
routed expert gate1.12attention q1.34
routed expert up1.11attention kv_a0.90
routed expert down4.52attention kv_b0.84
shared expert gate1.05attention o3.73
shared expert up1.05router gate1.65
shared expert down4.38layer-0 MLP gate1.39
layer-0 MLP up1.40
layer-0 MLP down4.50

The multipliers are applied as per-group LR factors in Muon+Moonlight, which also scales weight decay, exactly as OrScale's coupled WD does.

RunVal CEGain over Muon+Moonlight
R2: Muon+Moonlight (baseline)3.4690—
R3: Muon + fixed per-class multipliers3.3659+0.103 nats
R1: OrScale-LM (dynamic)3.3320+0.137 nats

R3 captures 75% of OrScale's gain (0.103 of 0.137 nats) using only fixed, time-averaged multipliers with no per-step adaptation. This confirms that the per-class scaling pattern — amplifying down/output projections by 3.7–4.5$\times$ while leaving gate/up projections near 1$\times$ — is the primary mechanism behind OrScale's gain over the baseline.

The remaining 25% (0.034 nats, R1 vs. R3) comes from OrScale's dynamic per-step trust-ratio updates, which adjust multipliers as training progresses. This is consistent with OrScale's ramp-up behavior: down/output projections reach the clip bound rmax⁡=5r _ {\max} = 5 by step 100–200 (Section 2), while R3's fixed values are time-averaged and therefore slightly lower during the critical early-training phase.

Validation curves. R3 separates from R2 at step 150–200, closely tracking R1 at every subsequent checkpoint. By step 400 (6.7B tokens), R3 already reaches Muon+Moonlight's final loss, just as R1 does. The curve shapes confirm that the per-class allocation pattern, not a transient artifact, drives the gain.

4. Writing Fixes

  • Add citations for Adafactor, Fromage, Madam, and Nero in Section 2 (upcV-S1).
  • Move MoE training details (batch size, step count, hardware) into the main text (upcV-S3).
  • Correct the characterization of the clip bound as "coupled-WD ceiling engaging" (l.419, l.894).
  • Add explicit corrected-Theorem-3 statistics from the MoE diagnostics to Section 5.3 (FX81-Q2).

Dear Reviewers,

As the discussion period concludes, we wanted to politely note that a revised paper has been uploaded incorporating all changes from the rebuttal: the corrected Theorem 3 (exact identity), second MoE seed, per-tensor diagnostics, and the writing fixes requested. We remain available for any final questions.

Thank you for your time and feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.