acceptodds
Under review as a conference paper at ICLR 2027

ExpertMuon-Compass: Alignment-Guided Step Sizes for Mixture-of-Experts Training

Abstract

Mixture-of-experts (MoE) language models send each token to a few experts, so each expert is trained on a different part of the data, and this part changes during training. With a shared learning rate, Muon applies updates of roughly the same size to expert matrices of the same shape, even when an expert's update is poorly aligned with its current gradient. We here propose ExpertMuon-Compass (Compass), which multiplies the Muon step of each expert by two factors. A family factor compares the cosine between the expert's orthogonalized update and its gradient with the same cosine for the other experts in its layer. A scalar radius aggregates the alignment between corresponding rows of the update and gradient into one step-length multiplier. Compass keeps the update direction and the momentum buffer of Muon. In pretraining on FineWeb-Edu, Compass with Nesterov momentum on all matrices performs as well as or better than Muon, NorMuon, and other optimizers. Adding its factors to NorMuon gives the same or a lower loss than NorMuon. Compass is the most effective when the data seen by each expert varies during training, for example, as when the languages of a multilingual corpus arrive in separate blocks or in continual-learning settings. With Compass, the expert load stays balanced, and the router assigns tokens to experts more decisively. We also prove a perturbation bound for the two factors.

Primary area:
optimization
Keywords:
Muon, mixture of experts, matrix optimizers, orthogonalized momentum, adaptive step size, gradient alignment, language model pretraining

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_Cw8sOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3(+1)
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ExpertMuon-Compass (Compass), an alignment-guided step-size rule for training Mixture-of-Experts (MoE) transformers with Muon or NorMuon. Because Muon orthogonalizes each weight matrix's momentum buffer via Newton-Schulz iterations (N5\mathcal{N}_5), every expert receives an update of roughly equal Frobenius norm under a shared learning rate, even when shifting router assignments cause an expert's momentum to misalign with its current token gradients. Compass retains Muon's orthogonalized direction Oe=N5(Ge+βMe)O_e = \mathcal{N}_5(G_e + \beta M_e) and single momentum buffer, scaling each expert matrix step by two scalar factors ceαec_e \alpha_e:

  • Family factor cec_e: Computes the clamped Frobenius cosine ae=[cos⁡(Oe,Ge)]+a_e = [\cos(O_e, G_e)]_+ , caps its ratio to the layer-family mean at one (ve=min⁡(1,ae/aˉ)v_e = \min(1, a_e/\bar{a}) ), and re-normalizes across active experts in the layer (ce=ve/vˉc_e = v_e/\bar{v}) so the factors average to one.
  • Scalar radius αe\alpha_e: Computes row-wise cosines ber=[cos⁡(Oer,Ger)]+b_{er} = [\cos(O_{er}, G_{er})]_+ , caps them at the row mean (qer=min⁡(1,ber/bˉe)q_{er} = \min(1, b_{er}/\bar{b}_e) ), interpolates with the cosine schedule multiplier λt=ηt/ηmax⁡\lambda_t = \eta_t/\eta_{\max} via her=1+λt(qer−1)h_{er} = 1 + \lambda_t(q_{er}-1), and defines αe=∥diag(he)Oe∥F/∥Oe∥F∈[0,1]\alpha_e = \|\text{diag}(h_e)O_e\|_F / \|O_e\|_F \in [0, 1] , matching the norm of the row-weighted matrix while preserving the singular vectors of OeO_e.

The authors prove a single-step perturbation bound for cec_e and αe\alpha_e (Theorem 1) and evaluate Compass in FP32 pretraining on 6-layer MoE models on FineWeb-Edu (330M and 1.1B parameters, from 8.2M to 1.05B tokens) and multilingual FineWeb2 (64 experts top-2, 461M parameters, 16.4M tokens in shuffled vs. 100-step language-blocked orders):

  • On FineWeb-Edu, Compass (N/N) matches or slightly outperforms Nesterov Muon and NorMuon across token budgets while consistently reducing router entropy (e.g., 0.957 vs. 1.020 nats on Muon, p<10−15p<10^{-15}) at equal load balance.
  • On FineWeb2, while all methods achieve similar loss on shuffled batches, under the non-stationary language-blocked order Compass significantly outperforms Nesterov Muon (−0.020-0.020 nats across 16 seeds, p=0.0012p=0.0012) and NorMuon when added to the NorMuon update (−0.025-0.025 nats across 16 seeds, p=0.0005p=0.0005).
Critical strengths
  • Clean, state-free geometric design that preserves Muon's orthogonalized direction: Measuring alignment after Newton-Schulz orthogonalization and collapsing row-level alignment into the scalar radius αe=∥diag(he)Oe∥F/∥Oe∥F\alpha_e = \Vert{}\text{diag}(h_e)O_e\Vert{}_F / \Vert{}O_e\Vert{}_F{}{} is well-motivated: it leaves the singular vectors of OeO_e intact, requires zero additional optimizer state memory, composes directly with NorMuon, is backed by a clean perturbation bound (Theorem 1), and adds only 3.8–6.6% wall-clock overhead.
  • High experimental rigor and insightful analysis of non-stationary routing: The experimental protocol separates tuning seeds from evaluation seeds, pairs random seeds across initializations and data orders, and carefully controls for NorMuon's implicit per-step weight decay ηtω\eta_t \omega. Contrasting shuffled vs. language-blocked FineWeb2 (Section 4.4) clearly demonstrates when per-expert step sizes matter: ceαec_e \alpha_e spreads out most sharply 25 steps after a language switch (t=12.3t=12.3) and consistently lowers router entropy across all budgets without hurting load balance.
Critical weaknesses
  • Negligible loss improvements under standard i.i.d. shuffled pretraining at longer horizons: When training data are randomly shuffled, Compass yields very small or null loss gains once momentum conventions (N/N) and weight decay are matched. At 8.2M tokens (width 2,048), Compass (N/N) and Nesterov Muon are tied (5.323405.32340 vs. 5.322965.32296, p=0.85p=0.85); at 65.5M tokens with 1x NorMuon decay, Compass (N/N) and Nesterov Muon differ by −0.00032-0.00032 nats (p=0.93p=0.93); at 262M and 1.05B tokens (Table 3 and Appendix A.2), Compass on the NorMuon update and NorMuon are statistically indistinguishable (−0.00019-0.00019 nats at 262M, 3/8 wins; −0.0058-0.0058 nats over 5 seeds at 1.05B, 95% CI [−0.0247,0.0130][-0.0247, 0.0130] ); and on shuffled FineWeb2, all methods finish within 0.0010.001–0.0050.005 nats.
  • The "Constant expert scale" control captures almost the entire gain over Nesterov Muon in the blocked-order setting: Because αe≤1\alpha_e \le 1, the mean expert multiplier cα‾\overline{c\alpha} is slightly below 1 (0.9481490.948149 in blocked FineWeb2). In Table 4, the "Constant expert scale" ablation—which simply multiplies every expert step by the fixed scalar 0.9481490.948149 without computing any alignment cosines—achieves 5.46575±0.043415.46575 \pm 0.04341 under blocked order, within 0.00180.0018 nats of Compass (5.46396±0.047415.46396 \pm 0.04741, vs. 5.486675.48667 for Nesterov Muon). In the 8-seed replication (Table 12), the gap between Compass (N/H) and Constant expert scale is −0.00223-0.00223 nats (95%95\% CI [−0.01341,+0.00894][-0.01341, +0.00894] , p=0.648p=0.648), and even against Nesterov Muon at matched LR 0.001250.00125, the gap shrinks to −0.00886-0.00886 (p=0.133p=0.133) for N/H and −0.00449-0.00449 (p=0.500p=0.500) for N/N. This suggests that in the Muon blocked comparison, a slightly smaller fixed expert learning rate explains most of the improvement over Nesterov Muon (though Compass on the NorMuon update remains a clear win). Finally, the continued pretraining run on JetMoE-8B (Appendix E) lacks a baseline Muon run for comparison.
Questions for authors
  • Clarification on "Constant expert scale" vs. dynamic alignment in the blocked-order experiment: In Table 4 and Table 12, multiplying all expert steps by a constant 0.9481490.948149 captures over 90% of Compass (N/H)'s improvement over Nesterov Muon (p=0.648p=0.648 between Compass and Constant expert scale), and Table 12 shows that part of Nesterov Muon's worse loss came from selecting LR 0.0006250.000625 instead of 0.001250.00125 (by a margin of 7×10−67 \times 10^{-6} nats on the two tuning seeds, Appendix A.4). Did you test a Constant expert scale control on top of the NorMuon update in the blocked-order experiment to verify how much of the −0.02525-0.02525 nat gain on NorMuon comes from per-step alignment variation versus average expert step damping?
  • Clarification on JetMoE-8B baseline (Appendix E): Did you run standard Nesterov or Heavy-ball Muon without Compass factors on JetMoE-8B under the same 1,000-step setting in Table 18, and if so, how did its validation loss compare to Compass's 2.11482.1148 (at LR 10−410^{-4}) and 2.07532.0753 (at LR 5×10−55\times 10^{-5})?
  • Suggestion for future work: Since αe\alpha_e anneals toward 1 as λt→0\lambda_t \to 0 (Equation 4), keeping λt\lambda_t fixed at 1 inside herh_{er} could be beneficial in continual learning or late-stage domain shifts where the learning rate is already small.
5 comments

Thanks for your review.

In our view, the core question is: “Is dynamic alignment behind the gains, or is it just step-size damping?” Our results show that both play a role. The constant ablation you asked for sets every expert's multiplier ceαec_e\alpha_e to one value, the mean from another Compass run. It reproduces a considerable part of the final loss improvement and finishes close to Compass. We see that setting the multipliers from alignment also helps. During language blocks, when a language returns, Compass ends the block with a lower loss on that language than the constant approach does. In the default Compass approach, the radius correction weakens as the learning rate decays. As you recommended, keeping λt\lambda_t = 1 inside the row factors keeps the radius on and ends with a lower test loss than NorMuon and default Compass. On a returning language, λt\lambda_t = 1 also ends the block with a lower loss than three controls with the same mean multiplier (Part 2). We see that the new JetMoE baselines favor Compass over expert-only Nesterov and heavy-ball Muon. Please see more details below.


Part 3 answers your point on the Muon comparison under language blocks (W2).

In all tables, runs are paired by seed, and each difference is the subtraction named in its label. A negative value favors the first term (usually Compass). Brackets give 95% confidence intervals from paired tt-tests, and pp-values come from a paired sign-flip test.

Q1. We multiplied the step of every expert that receives a gradient by 0.949. This is the mean ceαec_e\alpha_e of Compass on NorMuon in a separate run (for Muon, the paper used 0.948). Table R1 compares the methods at the same learning rate (0.00566). Hence, only the expert factors are different. (By contrast, the paper's drop in test loss of 0.0253 compares the methods at their own best learning rates from a grid search, 0.00566 for Compass and 0.00283 for NorMuon.) To test robustness and estimate the differences more precisely, Table R1 adds new seeds to the paper's 16.

Table R1. Test loss on the NorMuon update under language blocks at 0.00566, and for NorMuon also at 0.004, its best rate in Table R9. "Paper" is the paper's 16 seeds, and "new" seeds are new seeds from this rebuttal period.

SeedsBaselineBaseline lossCompass lossCompass − Baselinep
Paper + 32 newNorMuon5.48375.4775−0.0062 [−0.0103, −0.0021]0.004
Paper + 32 newNorMuon at 0.0045.48295.4775−0.0054 [−0.0099, −0.0009]0.020
Paper + 16 newNorMuon5.48675.4803−0.0063 [−0.0113, −0.0014]0.014
Paper + 16 newConstant 0.9495.48315.4803−0.0028 [−0.0084, +0.0029]0.33
255 newConstant 0.9495.48035.4787−0.0017 [−0.0037, +0.0004]0.11

Here, Compass has a lower loss than NorMuon at the same rate and even at NorMuon's best rate. On the paper's seeds and the first 16 new ones, the constant control reproduces about half of Compass’s loss improvement over NorMuon. On 255 new seeds, Compass and the constant approach end with nearly the same loss. Part 2 tests your λt\lambda_t = 1 recommendation vs. the controls having the same mean multiplier.

Table R2. Mean multiplier and differences in validation loss averaged over each cycle of eight language blocks (800 updates), on the NorMuon update. The constant approach keeps the scaling factor at 0.949.

SeedsCycle 1Cycle 2Cycle 3Cycle 4Cycle 5
λt\lambda_t0.980.790.500.210.02
Mean cα, CompassPaper + 16 new0.9110.9180.9440.9760.996
Mean cα, Compass (λ\lambda = 1)Paper + 32 new0.9090.9000.8970.8990.905
Loss, Compass − NorMuonPaper + 16 new−0.0107−0.0156−0.0098+0.0007−0.0033
Loss, Compass − constantPaper + 16 new−0.0046−0.0049+0.0011+0.0071−0.0004
Loss, Compass − constant255 new−0.0042−0.0058+0.0017+0.0069+0.0023
Loss, Compass (λ\lambda = 1) − CompassPaper + 32 new−0.0013−0.0060−0.0139−0.0190−0.0106

In cycles 1 and 2, Compass has lower validation loss than both NorMuon and the constant approach. This suggests faster early learning. In those cycles, the mean multiplier of Compass is also below 0.949. As λt→0\lambda_t \rightarrow 0, the row factors and the radius →\rightarrow 1 (Eqs. 4 and 5). Compass's mean multiplier then nearly reaches 0.949 in cycle 3 and goes over it in cycle 4. The loss difference also changes sign. Under the original schedule, the early advantage does not lead to a clear final loss advantage over the constant approach (Table R1).

At the end of a language's block. Tables R1 and R2 average the loss over all eight languages. Thus, they mostly measure languages that are not being trained. Table R3 looks only at the language that is just trained, at the end of the block. Each language returns after 700 updates on the other seven. Hence, Table R3 shows the quality of relearning by each method. The last three rows test your λt\lambda_t = 1 recommendation vs. the three controls with its mean multiplier (0.902).

The first control (Constant 0.902) scales every routed expert step by a fixed 0.902 across all layers and updates.

At each update, the second control (Shuffled multipliers) computes the per-expert multipliers ceαec_e \alpha_e online from its own state using the Compass (λt=1\lambda_t = 1) rule, then randomly permutes them among the routed experts within each family (one projection in one layer).

The third control (Family mean) gives every routed expert in a family that family's mean multiplier under λt\lambda_t = 1. And, this happens at each update. Note that the average stays the same and all experts in the family receive the same value.

Table R3. Validation loss of the language just trained, at the end of its block, averaged over cycles 2 to 5 (cycle 1 is each language's first visit). The seven rates run from 0.0014 to 0.0113.

Update and rateSeedsBaselineBaseline lossCompass lossCompass − baseline
NorMuon, 0.00566Paper + 16 newNorMuon4.82334.8163−0.0070 [−0.0083, −0.0056]
NorMuon, 0.00566Paper + 16 newConstant 0.9494.82084.8163−0.0044 [−0.0062, −0.0027]
NorMuon, 0.00566255 newConstant 0.9494.82064.8168−0.0038 [−0.0043, −0.0032]
NorMuon, mean of seven rates4 tuningNorMuon4.84524.8403−0.0049 [−0.0061, −0.0036]
NorMuon, mean of seven rates4 tuningConstant 0.9494.84474.8403−0.0044 [−0.0056, −0.0032]
Muon (N/H), 0.00125PaperConstant 0.9484.83494.8309−0.0039 [−0.0062, −0.0016]
NorMuon, 0.00566, Compass (λ = 1)Paper + 95 newConstant 0.9024.81814.8160−0.0021 [−0.0028, −0.0013]
NorMuon, 0.00566, Compass (λ = 1)Paper + 32 newShuffled multipliers4.81724.8152−0.0020 [−0.0030, −0.0009]
NorMuon, 0.00566, Compass (λ = 1)Paper + 32 newFamily mean4.81784.8152−0.0026 [−0.0036, −0.0016]

Note that each difference in Table R3 is also below zero in every cycle from 2 to 5. After the same 100 updates on a returning language, Compass reaches a lower mean loss than NorMuon or a constant expert scale. In our view, this supports more effective adaptation within a fixed training budget beyond what the constant-scale control achieves.

Your suggestion of λt\lambda_t = 1. Thank you for this suggestion. We tried this approach in several experiments. Of the two factors, only the radius depends on λt\lambda_t, through the row factors of Eq. 4. In Compass, the radius →\rightarrow 1 as the learning rate decays. Fixing λt\lambda_t = 1 in the row factors leaves the learning rate schedule (Eq. 9) unchanged. The radius then stays on, and the mean multiplier stays near 0.90 (Table R2). Note that the mean multiplier (the Compass scale placed on each expert's update) is below the 0.949 constant. Hence, Tables R3 and R4 compare λt\lambda_t = 1 with the three controls above. These controls have the same mean multiplier.

Table R4. Test loss on the NorMuon update under language blocks at 0.00566.

SeedsBaselineBaseline lossCompass (λ = 1) lossCompass (λ = 1) − baselinep
Paper + 32 newNorMuon5.48375.4678−0.0159 [−0.0208, −0.0109]< 0.001
63 newNorMuon5.48585.4728−0.0129 [−0.0170, −0.0089]< 0.001
Paper + 32 newCompass5.47755.4678−0.0097 [−0.0135, −0.0059]< 0.001
Paper + 95 newConstant 0.9025.47265.4707−0.0020 [−0.0050, +0.0010]0.20
Paper + 32 newShuffled multipliers5.46835.4678−0.0005 [−0.0052, +0.0041]0.82
Paper + 32 newFamily mean5.47005.4678−0.0022 [−0.0064, +0.0021]0.31
6 new (A100)NorMuon5.49665.4712−0.0254 [−0.0402, −0.0105]0.031

At the last update, λt\lambda_t = 1 and the three controls end with nearly the same test loss (Table R4). At the end of each language's block, λt\lambda_t = 1 has lower loss on that language than all three controls (Table R3). None of the controls keeps the match of the expert and the corresponding multiplier. Thus, at the block end, it helps to give each expert the multiplier its own alignment produced. In JetMoE-8B, λt\lambda_t = 1 again gives the lowest final loss (Table R5 in Part 3). Yes, we will add λt\lambda_t = 1 to Section 3 as an option for data that keep shifting late in training.

W3 and Q2. Yes, we have now run expert-only Nesterov and heavy-ball Muon in the setting of Table 18 (released code and settings, 1000 updates of continued pretraining, both learning rates, 3 seeds). Compass reruns with the same seeds on the same machines as the baselines match the final losses of Table 18 within 0.0003 nats.

Table R5. JetMoE-8B, final validation loss, mean of three seeds. Expert-only Nesterov is Compass with both factors (the family factor cec_e and the scalar radius αe\alpha_e) fixed at one.

MethodLoss at 1e-4Compass − methodLoss at 5e-5Compass − method
Compass2.11472.0748
Compass (λ = 1)2.11392.0746
Expert-only Nesterov2.1184−0.00372.0763−0.0015
Heavy-ball Muon2.1218−0.00712.0777−0.0029

Compass is lower than the two baselines at every evaluation of every seed and rate. This includes update 100, where each run reaches its best loss, as in Table 18. Fixing λt=1\lambda_t = 1 lowers Compass's final (step-1000) validation loss on every seed at both learning rates, while matching default Compass on the lowest validation loss along the run (reached at update 100, where λt≈0.98\lambda_t \approx 0.98 is still close to 1).

W1. The abstract and the conclusion will say that the gain is the largest when the data each expert sees keep shifting, for instance under the language blocks. They will also say that with the long shuffled runs (262M and 1.05B tokens), Compass and NorMuon end close, and that in the shuffled multilingual runs, Compass is ahead early in training.

The contrast with the language-blocked training is consistent with the response of Compass to changes in an expert’s data. Compass compares the proposed update with the gradient of the current tokens and uses their alignment to adjust the step. Language blocks introduce abrupt changes that persist for 100 updates. With shuffled data, each batch still holds one language. The language changes at almost every update. Hence, no shift lasts long. Consistent with this distinction, we see that the spread of expert multipliers is greater 25 updates after a language switch than later in the block. However, it does not vary much across the corresponding checkpoints with shuffled data (Section 5).

Table R6. Data order in the multilingual setting, NorMuon update at 0.00566, test loss on the paper's seeds and the first 16 new ones (Table R1).

Data orderNorMuonCompassCompass − NorMuonp
Shuffled4.40434.4053+0.0011 [+0.0003, +0.0018]0.007
Language blocks5.48675.4803−0.0063 [−0.0113, −0.0014]0.014
Blocks minus shuffled−0.0074 [−0.0124, −0.0024]0.005

On the shuffled multilingual runs, Compass on the NorMuon update also leads NorMuon early in training (by 0.016 nats in validation loss at update 400 across the eight seeds of Table 4), though NorMuon finishes slightly lower (0.0011 nats) in final test loss across the 32 seeds of Table R6.

At 65.5M tokens of FineWeb-Edu, Compass (N/N) and Nesterov Muon reach the same loss at NorMuon's per-step decay, as you note. Twice that decay lowers the loss of every method, and there, Compass (N/N) is lower than NorMuon and Nesterov Muon (Table 11). With the NorMuon update, where only the factors differ, Compass is lower at 65.5M tokens (Tables 3 and 11) and close to NorMuon at 262M and 1.05B (Table 3). At 262M, it still lowers the router entropy (Section 5).

W2. Table R7 completes the comparison under language blocks at one rate (0.00125) on the 16 seeds of Tables 4 and 12, with the missing runs added.

Table R7. Muon update under language blocks at 0.00125, test loss over the paper's 16 seeds. The first 8 are the seeds of Table 4, and every method has higher losses on the other 8.

MethodTest lossCompass − methodp
Compass (N/H)5.4939
Constant 0.9485.4959−0.0020 [−0.0082, +0.0042]0.50
Nesterov Muon5.5036−0.0097 [−0.0180, −0.0013]0.027
Expert-only Nesterov5.5041−0.0102 [−0.0191, −0.0013]0.016

At the matched learning rate of 0.001250.00125, Compass (N/H) still achieves a lower test loss than Nesterov Muon (−0.0097-0.0097 nats, p=0.027p = 0.027), though (as you noted) the fixed 0.9480.948 constant scale accounts for most of this difference (−0.0020-0.0020 nats between Compass and the constant approach, p=0.50p = 0.50).

Section 4.4 will specify this directly. And we will ensure that Figure 1 and Table 4 show Nesterov Muon at both learning rates. At the end of a language's block, Compass is still lower than the constant scale (Table R3). In the shorter runs at 8.2M tokens and width 1024, each Compass factor alone has a lower loss than the constant expert scale on all eight seeds (Table 1).

The revision will add Tables R1 to R7 to Section 4.4 and the appendix. We hope these results answer your questions.

Reviewer_dfGZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper improves optimization efficiency in MoE training by allocating step sizes across experts according to their alignment, measured as the agreement between each expert's update and its current gradient. The allocated step is realized on top of Muon by rescaling each expert's update, with the direction and momentum buffer unchanged.

Critical strengths
  1. The core idea is well motivated: alignment directly reflects the quality of an expert's update, and using it to allocate step sizes across experts improves optimization efficiency.
  2. The mechanism is lightweight: only the step length is rescaled, leaving the update direction and momentum buffer of Muon unchanged.
Critical weaknesses
  1. The row-level cosine has no evident connection to MoE. Unlike the family factor, which compares an expert with the other experts in its layer and is therefore MoE-specific, the row-wise cosine within a single matrix is a generic quantity that is not specific to MoE.
  2. Allocating step sizes by alignment may prevent long-tail knowledge from being learned. Low alignment may reflect a rare or novel signal rather than a poor update, and shrinking the step in that case suppresses exactly the knowledge that should be learned.
  3. Gradient noise is only safeguarded, not reduced. The paper handles noise with safeguards against instability (clipping, capping, and perturbation bounds), but performs no denoising for the case where the effective sample size per expert is small and uneven.
Questions for authors
  1. What is the connection between the row-wise cosine and MoE? Is the row-level factor also useful outside MoE, for example in dense models?
  2. Will allocating step sizes by alignment prevent long-tail knowledge from being learned? Have you evaluated settings with imbalanced mixtures or rare data groups?
  3. For the small and uneven effective sample sizes per expert, have you considered denoising the alignment scores, for example by averaging over steps or weighting by routed token count?
3 comments

Thanks for your review.

The main themes in this review concern MoE specificity, dense models, long-tail knowledge, and score noise.

In short, yes, row-cosine scaling is architecture-agnostic and also benefits a tested dense architecture. However, the family factor is specific to MoE, as it compares experts with each other. For rare or long-tail inputs, gradients still accumulate into the momentum buffer Me,tM_{e,t} unscaled (Compass scales only the applied parameter step), so recurring rare signals steer the momentum toward them and can recover larger step sizes over subsequent updates. In a new rare-language experiment (where one language receives only 2% of training steps across five 14-update blocks), Compass achieves lower validation loss than NorMuon on that language immediately after each 14-update exposure (averaged over the five blocks). The final rare-language loss shows no clear difference. Split-batch checks suggest that most of the spread in the per-expert step multipliers is not batch noise, and averaging the underlying alignment scores over updates provides no clear benefit. More details below.


Q1. The row cosine works for any matrix. However, in our case, Compass applies it to each expert separately. Each expert only processes the tokens that are routed to it. When that routing shifts, it starts to see different tokens. The rows that respond to the new tokens can then get gradients that disagree with the momentum. The radius αe\alpha_e (the row-level factor you asked about) then shortens that expert's step. Across a layer, the family factor compares whole experts, and the two factors together set the step length of each expert. For the MoE runs at 8.2M tokens and width 1024, each factor alone has a lower loss than a constant expert scale (Table 1).

We also ran the radius alone on a dense model. Each layer had one feed-forward block of hidden size 2048, the active width of our MoE runs. With one block per layer, the family factor was always one, and only the radius was active. Under the same language blocks, the radius still lowered NorMuon's loss (Table R11). It helped the dense layers too when their data shifted.

Table R11. Dense model on the NorMuon update under the language blocks of Table R1, both at 0.00566, test loss over 16 new seeds on an A100.

NorMuonCompass (radius only)Compass (radius only) − NorMuonp
5.67065.6606−0.0100 [−0.0159, −0.0040]0.003

Q2. A rare signal can get a shorter step at first, as its gradient may point away from the momentum. Compass scales only the update VeV_e (Eq. 5). The gradient Ge,tG_{e,t} still enters the momentum Me,tM_{e,t} without Compass's scaling (Eq. 1). Hence, a signal that stays for several updates turns the momentum toward it, and its step grows back as the agreement rises. If a signal appears only once and leads to low alignment between update and gradient, Compass may give it a shorter step than the base optimizer would.

We did two different tests to respond to this. Under language blocks, each language is absent for 700 of every 800 updates and then returns. At the end of its block, a returning language has a lower loss with Compass than with NorMuon or a constant scale (Table R3 in our reply to Reviewer Cw8s). In the second test, we made one language rare. It gets 2% of the updates, which is one short block of 14 updates per cycle. Each block comes after about 700 updates on the other languages, when the momentum points elsewhere. Each language is rare on two of 16 new seeds.

Table R12. One rare language, 16 new seeds, all at 0.00566. The first row averages the validation loss right after each of its five blocks, and the other rows give the test loss. λ\lambda = 1 keeps the radius on (reply to Reviewer Cw8s, Part 2).

MeasureNorMuonCompassCompass (λ\lambda = 1)Compass − NorMuonCompass (λ\lambda = 1) − NorMuon
Rare language, right after its blocks6.34646.33316.3289−0.0134 [−0.0195, −0.0072]−0.0175 [−0.0274, −0.0076]
Rare language, end of training6.36466.36526.3611+0.0006 [−0.0129, +0.0141]−0.0035 [−0.0170, +0.0100]
Other seven languages, end of training5.39225.38795.3799−0.0043 [−0.0125, +0.0039]−0.0123 [−0.0177, −0.0069]

After the same 14-update exposures, we see that Compass achieves an immediate validation loss 0.0134 nats lower than NorMuon's, averaged over the five blocks. This supports more effective adaptation during brief exposures to a rare language. The final loss on the rare language shows no clear difference. In comparison, utilization-based step scaling (Excitation; Section 4.1) shrinks updates for infrequently routed experts regardless of gradient agreement. We observe that it yields higher validation loss than Compass on all eight seeds at both model widths, 1024 and 2048 (Tables 1 and 9).

Q3. As you mentioned, Compass handles noisy scores with safeguards. The caps limit the impact of a noisy high score on lengthening a step. Theorem 1 bounds the effect of score noise on the update. To measure this noise empirically, we computed the expert multipliers ceαec_e\alpha_e from two halves of each batch (Table R13). We also tested your recommendation of averaging the scores over updates (an exponential moving average with decay 0.9, stored for each expert and its rows) on the paper's 16 seeds (Table R14).

Table R13. Score noise over 200 updates of the model of Table R1. Spearman-Brown turns the half-batch correlation r into the correlation expected between two independent full batches, 2r/(1 + r), computed before rounding.

CheckValue
Multipliers from two half batches, correlation across experts (median)0.62
Reliability of the full-batch multipliers (Spearman-Brown)0.76
Routed tokens per expert, coefficient of variation (median)0.21
Least-routed expert, tokens as a share of the median (median)0.56

Table R14. Scores averaged over updates, NorMuon update under language blocks at 0.00566, test loss over the paper's 16 seeds (Table 4 shows the first 8).

MethodTest lossCompass − method
Compass5.4728
Averaged scores5.4760−0.0033 [−0.0137, +0.0072]
NorMuon5.4805−0.0078 [−0.0156, +0.0001]

Now, the two halves of a batch give similar multipliers. This suggests that most of their spread across the experts is not batch noise. The caps limit the effect of the remaining noise on the step length. We observe that the averaged scores and Compass reach the same loss within the noise of 16 seeds. Since averaging adds optimizer state, we keep the state-free score. Under language blocks, loss-free balancing keeps the routed counts close (Table R13). Thus, we kept the family mean unweighted in our approach. Section 6 will include weighting by routed tokens as a potential option for models with less even routing.

The revision will add Tables R11 to R14 to the appendix. Thanks again.

AuthorsOpenReview ↗

Reminder to reviewer dfGZ

Dear reviewer dfGZ, As the discussion period closes in a little over 24 hours, we wanted to check in case you may have any concerns that we haven't yet answered. Or, any new questions that may have come up. We would be happy to clarify any further details.

Thank you for your time.

Reviewer_89a7OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper proposes ExpertMuon-Compass, which multiplies the Muon step of each expert by two factors, namely the family factor and the scalar radius. The family factor compares the cosine between the expert’s orthogonalized update and its gradient with the same cosine for the other experts in its layer, while the scalar radius aggregates the alignment between corresponding rows of the update and gradient into one step-length multiplier. The experiments on FineWeb-Edu show that ExpertMuon-Compass generally matches other baselines, while its gains are more consistent under language-blocked multilingual training. The paper also provides an analysis of training, showing that ExpertMuon-Compass keeps the expert load balanced and makes the router assign tokens to experts more decisively.

Critical strengths

The method addresses the interaction between changing expert data and Muon's nearly fixed update norms. Scalar corrections can preserve the base direction without adding optimizer state.

The paper conducts extensive experiments to validate the effectiveness of ExpertMuon-Compass and provides an insightful analysis of the numerical specification and perturbation.

Critical weaknesses
  1. I have concerns about the gains of ExpertMuon-Compass (Compass). For example, in Table 3, Compass-on-NorMuon does not exhibit a clear advantage at either 262M or 1.05B tokens. In addition, in Table 6, it addes 5.2--6.6 % runtime.

  2. It is unclear how robust the blocked-order improvements are to a denser learning-rate sweep with momentum and per-step decay matched.

  3. The distinctive scalar-radius construction seems to be empirical. Radius-only matches the full method at width 2,048. A matched comparison isolating scalar aggregation from row weighting is missing.

Questions for authors

Please see the comments in Critical Weaknesses.

5 comments

Thanks for your review.

W1 and W2 overlap with questions from Reviewer Cw8s. For those, we give a short answer here. Brackets in our tables give 95% confidence intervals of differences paired by seed.

In our view, the major points this review focuses upon are the size of the gains, the runtime cost, the learning rate grid confounders, and the usage of a scalar radius in comparison to row-wise weighting.

To answer, in summary: Under language blocks, Compass ends with a lower test loss than NorMuon. At the end of each block, the language just trained also has a lower loss with Compass. We note that w.r.t. long shuffled runs, the two methods end close (at 262M and 1.05B tokens). When the factors are computed for all experts together, Compass's extra work is about 1% of a training step (Table R8).

Under language-blocked training, this test-loss gain for Compass on the NorMuon update holds across 48 evaluation seeds both at a shared learning rate (0.00566) and against NorMuon at its best rate (0.004) from a seven-rate sweep (Table R9).

Finally, we utilize the scalar radius to preserve the base update direction, and it yields a loss comparable to matched row-wise weighting. More details below.


W1. At 262M and 1.05B tokens of shuffled data, the two methods reach nearly the same final loss, and the abstract and conclusion will state this point. Compass corrects the step of an expert whose data shift. Under language blocks, the data keep shifting, and Compass ends with a lower test loss than NorMuon at the same rate (Table R6 and W1 in our reply to Reviewer Cw8s). At the end of each block, the language just trained also has a lower loss with Compass than with NorMuon or a constant scale (Table R3 in that reply).

Runtime. The 3.8 to 6.6% in Table 6 was measured with our reference code. The code loops over experts and synchronizes with the CPU once per family and step. The factors need a few dot products per expert, about 10$mn$ operations for an m×nm \times n matrix. Five Newton-Schulz steps take about 5(4m2n+2m3)5(4m^{2}n + 2m^{3}) operations. Hence, the factors add less than 0.1% of the total arithmetic. In Table R8, we compute the same factors for all experts of a family at once.

Table R8. One training step of the model of Table R1 on one GH200, median of 200 steps. The shares exclude the AdamW step and parameter writes. Therefore, the factors' share is an upper bound.

Part of the stepTime (ms)Share
Forward and backward pass62177%
NorMuon update of all matrices, experts batched17822%
Compass's factors, experts batched81.0%

W2. At one learning rate, Compass on NorMuon and NorMuon share the momentum and the per-step decay. The only difference is in the expert factors. Table 3 already has such a pair with shuffled data at 65.5M tokens. We note that under language blocks, the paper's selected rates are different (0.00283 for NorMuon, 0.00566 for Compass). Here, their per-step decays are different by a factor of two. Hence, we ran both the checks you had asked for under 100-step language blocks. On the paper's 16 seeds and 32 new seeds (Table R1 in our reply to Reviewer Cw8s), Compass at 0.00566 achieves a lower test loss than NorMuon both at the matched learning rate of 0.00566 (−0.0062 nats, pp = 0.004) and against NorMuon at its best sweep rate of 0.004 (−0.0054 nats, pp = 0.020). Table R9 sweeps seven rates on four tuning seeds, none of them among these 48 seeds.

Table R9. Final validation loss on the NorMuon update under language blocks, mean of the four tuning seeds. Bold marks each method's best rate (lowest mean validation loss). λ\lambda = 1 fixes λt\lambda_t at 1 in Eq. 4 and keeps the learning rate schedule. The radius then stays on (as suggested by Reviewer Cw8s).

RateNorMuonCompassCompass (λ\lambda = 1)Compass − NorMuon
0.001415.55825.55295.5568−0.0053
0.002005.50495.48945.5028−0.0155
0.002835.46825.45715.4584−0.0111
0.004005.45025.44545.4494−0.0049
0.005665.45095.44175.4338−0.0092
0.008005.45955.46065.4451+0.0011
0.011315.50045.49755.4897−0.0029
Best against best5.45025.44175.4338−0.0085

Here, we observe that Compass has the lower mean at six of the seven rates, and its best is lower than NorMuon's best. λ\lambda = 1 has the same best rate as Compass, and its best is the lowest of the three. Averaged across all seven sweep rates (as well as at 0.005660.00566), Compass also achieves lower validation loss than NorMuon on the just-trained language at the end of the returning 100-step blocks in cycles 2 to 5 (see Table R3 in our reply to Reviewer Cw8s).

W3. Row weighting changes where the step points. The radius changes only the length of that step. Muon's step is an orthogonalized matrix, as Newton-Schulz pushes its singular values close to one. Hence, we keep that direction and let the row agreement set the length. The radius is the length of the row-weighted update divided by the length of Muon's update: αe=∥diag(he)Oe∥F∥Oe∥F\alpha_e = \frac{\lVert \mathrm{diag}(h_e) O_e \rVert_F} {\lVert O_e \rVert_F} (Eqs. 5 and 6). Keeping the direction also leaves NorMuon's own row normalization intact. In the multilingual model, row weighting turns NorMuon's step by about 7.67.6 degrees and makes its singular values less equal. For instance, the ratio of the top singular value to the one at 90% of the rank rises from 1.77 to 1.95. These calculations are medians over the 768 expert updates we sampled.

For Table R10, note that the first setting (width 1024, rates tuned separately, both 0.00125) reports the final validation loss on 8 fresh evaluation seeds disjoint from Table 1. Meanwhile, the second setting reports held-out test loss across all 16 evaluation seeds from the paper's language-blocked experiment at matched learning rate 0.00566.

Table R10. Row weighting with the same norm as Compass's step at every update.

SettingSeedsRow weightingCompassCompass − row weighting
FineWeb-Edu, 8.2M tokens, Muon update (N/N)8 fresh5.44545.4481+0.0026 [−0.0009, +0.0062]
Language blocks, NorMuon update at 0.00566Paper5.47445.4728−0.0016 [−0.0125, +0.0092]

Neither method has a clearly lower loss in either setting.

We retain both factors because they play different roles. At width 1024, combining cec_e and αe\alpha_e gives a slightly lower mean validation loss than the radius-only ablation (5.4540 vs. 5.4553) and the family-factor-only ablation (5.4570, lower on all eight seeds), while tying radius-only at width 2048 (Table 1).

We note that the family factor alone significantly improves over no scaling at widths 1024 and 2560 (Appendix F). Only the family factor can lengthen a step (cec_e can exceed one, and αe≤1\alpha_e \leq 1 can only shorten it), and with both factors, Compass has the best or tied-for-best loss values of the ablations at widths 1024 and 2048 (Table 1).

Summary of planned manuscript revisions.

We'll revise the manuscript to add Tables R8, R9, and R10 (comparing computational burden, comparing final validation losses across seven learning rates, and comparing Compass with norm-matched row weighting, respectively). We'll also add the batched timing to Table 6, the comparisons at one rate to Section 4.4, and an analysis of radius geometry to the appendix. The abstract and conclusion will clearly state our contribution scope (W1). We hope this addresses your questions and will allow you to revisit your score.

AuthorsOpenReview ↗

Reminder to Reviewer 89a7

Dear reviewer 89a7, As the discussion period closes in a little over 24 hours, we wanted to check in case you may have any concerns that we haven't yet answered. Or, any new questions that may have come up. We would be happy to clarify any further details.

Thank you for your time.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

Thank you for the reviews and engaged discussion, which we think will meaningfully improve the contribution of this paper. We very much appreciate your time and efforts to provide the helpful suggestions!

We have now reuploaded the paper on OpenReview, with changes in blue and all newly added tables in Appendix H and Table 6.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.