- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces OrScale, a layer-wise trust-ratio extension of the Muon optimizer. Muon determines update directions by orthogonalizing matrix-valued momentum, but it applies similar global step magnitudes across layers. OrScale addresses this limitation by dynamically adjusting the magnitude of each layer’s update according to the parameter norm and the norm of the actual update direction. A per-layer calibration step preserves the initial Muon/Moonlight update scale, while clipping and coupled weight decay control potentially unstable growth. The paper provides convergence results for bounded layer-wise scaling and argues that OrScale can improve optimization when update statistics differ substantially across layers. Experiments on CIFAR-10, dense language models ranging from 125M to 1.1B parameters, and a 16B-A3B mixture-of-experts model show that OrScale generally matches or improves upon Muon with Moonlight. The largest reported improvement is a 0.1305-nat reduction in validation cross-entropy on the MoE model at comparable wall-clock cost, although this result is based on a single random seed.
Critical strengths
1.A simple and implementable modification of Muon. OrScale preserves Muon’s orthogonalized direction and adds only a scalar magnitude adjustment for each layer. The applied-direction denominator is better motivated than using either the raw momentum norm or the post-orthogonalization norm, and the additional computational and memory cost appears small.
2.Broad empirical coverage and useful diagnostics. The paper evaluates the method on vision, dense language models, and a MoE model with large amount of parameters. The reported trust-ratio trajectories and design ablations provide useful evidence that the proposed scaling behaves differently across layer classes, especially in the MoE setting.
Critical weaknesses
1.The motivation and novelty are not sufficiently established. OrScale’s main idea is to apply a LARS/LAMB-style trust ratio to Muon, adding dynamic layer-wise scaling to Muon’s orthogonalized updates. This is a plausible engineering adaptation, but the conceptual advance appears limited. The paper does not convincingly establish that layer-wise relative-step imbalance is a major limitation of Muon in the evaluated settings, or that OrScale addresses this limitation in a way not already captured by simpler layer-wise scaling schemes.
2.The experiments do not cleanly isolate the source or size of the reported gains. OrScale uses dynamic layer-wise scaling, while the main Muon+Moonlight baseline uses a global learning rate and a static shape correction. The zero-sweep protocol demonstrates transfer of a learning-rate recipe, but it does not establish that OrScale outperforms Muon under an equal tuning budget or isolate the benefit of dynamic scaling. Moreover, OrScale couples weight decay into the scaled update, and the paper lacks a matched Muon+Moonlight control with the same weight-decay implementation. This makes it difficult to attribute the gains specifically to the trust ratio.
Questions for authors
1.Can the authors provide a matched control experiment in which Muon+Moonlight uses the same coupled weight-decay implementation as OrScale, while keeping the trust ratio fixed to one? This is necessary to determine whether the reported improvements come from dynamic layer-wise scaling or from the change in how weight decay is applied.
2.How does OrScale compare with Muon+Moonlight under an equal hyperparameter-tuning budget? The zero-sweep protocol demonstrates recipe transfer, but it does not establish that OrScale is superior when both methods are tuned comparably. A comparison with a simple fixed layer-wise scaling baseline would also help assess the contribution of the dynamic trust ratio.
3.What evidence shows that layer-wise relative-step imbalance is a primary limitation of Muon? In particular, can the authors quantify the relative update magnitudes of Muon+Moonlight across layer classes and show that OrScale corrects this imbalance rather than merely benefiting from additional adaptive degrees of freedom?
3 comments
Q3: Evidence for layer-wise step imbalance. We have logged the relative update magnitude per layer class on both the OrScale (R1) and Muon+Moonlight (R2) seed-43 runs:
| Step | R2 (Muon): max/min | R2: CV | R1 (OrScale): max/min | R1: CV |
|---|---|---|---|---|
| 10 | 11.6$\times$ | 1.00 | 9.9$\times$ | 0.98 |
| 60 | 6.4$\times$ | 0.69 | 9.8$\times$ | 0.97 |
| 200 | 3.2$\times$ | 0.25 | 2.4$\times$ | 0.24 |
| 300 | 2.2$\times$ | 0.21 | 1.9$\times$ | 0.18 |
| 500 | 2.2$\times$ | 0.23 | 2.8$\times$ | 0.28 |
The initial imbalance is real and substantial: MLP/expert down projections (initialized with small norms) have 5–12$\times$ larger relative updates than gate/up projections. In Muon+Moonlight, this imbalance naturally decreases as weight norms grow, reaching ${\sim}\times$ by step 300. OrScale does not reduce the imbalance faster; instead, it explicitly amplifies the down/output projections by ${\sim}\times$ (clipped trust ratio) while scaling other layers by 0.8–2.2$\times$.
The fixed-multiplier ablation (General Response, Section 3) now provides the precise attribution: 75% of OrScale's 0.137-nat gain comes from the per-class allocation (R3 captures 0.103 nats with fixed multipliers), and 25% from dynamic adaptation. The per-class pattern — concentrating large multipliers on down/output projections while leaving gate/up projections near 1$\times$ — is the primary mechanism. The dense results (Table 1) are consistent with this picture: on architecturally homogeneous models, where the headroom dispersion is small, OrScale's gain is modest (${\sim}$0.01 nats). The gain grows substantially on the MoE model, where layer classes have markedly different update statistics — exactly the regime where per-class scaling matters most.
Gentle follow-up before the discussion period closes
Dear Reviewer J79d,
We wanted to gently follow up before the discussion period closes. We have posted detailed replies to each of your concerns and uploaded a revised paper with all changes incorporated. A quick summary:
- Gain-source isolation (W2, Q1, Q2): A fixed-multiplier baseline at 16B captures 75% of OrScale's gain (0.103 of 0.137 nats) using per-class LR multipliers from OrScale's logged trust ratios, confirming the per-class allocation as the primary mechanism. For the coupled-WD concern (Q1): at the coupled update reduces exactly to Muon+Moonlight's, so the existing baseline is the matched control.
- Layer-wise diagnostics (Q3): Per-layer-class relative update magnitudes are now logged and included in the revision, showing the initial 5--12$\times$ imbalance and how OrScale redistributes step sizes across layer classes.
- Second MoE seed: Seed 43 confirms the gap (0.137 nats, consistent with seed 42's 0.131).
- Novelty (W1): The revision more clearly presents the failure-mode diagnosis and why each design choice is independently necessary (Table 4).
We would be grateful for any further feedback before the discussion concludes.
We thank Reviewer J79d for the sharp experimental questions, which have led us to collect detailed per-layer diagnostics that significantly strengthen the empirical picture.
W1: Motivation and novelty. We respectfully push back on the characterization of OrScale as simply "LARS/LAMB applied to Muon." The contribution includes the failure-mode diagnosis and the design of each component to address a specific failure:
A "plausible engineering adaptation" that gets every design choice right on the first try would not produce this pattern of failures.
W2: Experiments do not isolate gain source. We address the three bundled changes:
(a) Dynamic layer-wise scaling. The completed fixed-multiplier baseline (General Response, Section 3) shows that Muon+Moonlight with fixed per-class LR multipliers from OrScale's logged ratios captures 75% of OrScale's gain (0.103 of 0.137 nats), confirming that the per-class allocation pattern is the primary mechanism. Dynamic adaptation adds the remaining 25% (0.034 nats).
(b) Coupled weight decay. With the trust ratio fixed to one (r^ℓ≡1), OrScale's coupled update Wℓ←Wℓ−ηr^ℓ(λWℓ+sℓQℓ) reduces to Wℓ←(1−ηλ)Wℓ−ηsℓQℓ, which is Muon+Moonlight's update. The coupling only changes the decay coefficient from ηλ to ηλr^ℓ; at r^=1 they are identical. We verified this numerically: OrScale with rmin=rmax=1 produces updates that differ from Muon+Moonlight by at most 1.2×10−7 (relative) over 20 steps. Thus the existing Muon+Moonlight baseline already is the matched coupled-WD control.
Moreover, the cumulative weight-decay contribution over 596 steps is small: 1.2% for Muon+Moonlight and at most 6.1% for OrScale at r^=5, while the update-induced weight-norm differences between the two runs are 4–5$\times$ for the affected layers (down/output projections). The WD coupling is not the source of the gap.
(c) Calibration anchor. Table 3 already tests this: uncalibrated OrScale requires 2.5–3$\times$ LR retune and still finishes behind Muon+Moonlight.
Q1: Matched WD control. Answered above: the existing Muon+Moonlight baseline is this control (r^≡1 makes the coupled and decoupled updates identical).
Q2: Equal tuning budget. At 125M, Table 7 (Appendix F) shows a matched LR sweep where OrScale is at or below Muon+Moonlight at every stable cell. At 16B, additional ablations are in progress for the revision. We also note that the zero-sweep protocol is the practical comparison: OrScale's value proposition is that gains come without additional sweeps, directly from the Moonlight recipe. The fixed-multiplier result (General Response Section 3) already shows that OrScale's per-class allocation pattern — not merely a global LR shift — accounts for 75% of the gain.