- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 3(+1)
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes ExpertMuon-Compass (Compass), an alignment-guided step-size rule for training Mixture-of-Experts (MoE) transformers with Muon or NorMuon. Because Muon orthogonalizes each weight matrix's momentum buffer via Newton-Schulz iterations (), every expert receives an update of roughly equal Frobenius norm under a shared learning rate, even when shifting router assignments cause an expert's momentum to misalign with its current token gradients. Compass retains Muon's orthogonalized direction and single momentum buffer, scaling each expert matrix step by two scalar factors :
- Family factor : Computes the clamped Frobenius cosine , caps its ratio to the layer-family mean at one ( ), and re-normalizes across active experts in the layer () so the factors average to one.
- Scalar radius : Computes row-wise cosines , caps them at the row mean ( ), interpolates with the cosine schedule multiplier via , and defines , matching the norm of the row-weighted matrix while preserving the singular vectors of .
The authors prove a single-step perturbation bound for and (Theorem 1) and evaluate Compass in FP32 pretraining on 6-layer MoE models on FineWeb-Edu (330M and 1.1B parameters, from 8.2M to 1.05B tokens) and multilingual FineWeb2 (64 experts top-2, 461M parameters, 16.4M tokens in shuffled vs. 100-step language-blocked orders):
- On FineWeb-Edu, Compass (N/N) matches or slightly outperforms Nesterov Muon and NorMuon across token budgets while consistently reducing router entropy (e.g., 0.957 vs. 1.020 nats on Muon, ) at equal load balance.
- On FineWeb2, while all methods achieve similar loss on shuffled batches, under the non-stationary language-blocked order Compass significantly outperforms Nesterov Muon ( nats across 16 seeds, ) and NorMuon when added to the NorMuon update ( nats across 16 seeds, ).
Critical strengths
- Clean, state-free geometric design that preserves Muon's orthogonalized direction: Measuring alignment after Newton-Schulz orthogonalization and collapsing row-level alignment into the scalar radius {}{} is well-motivated: it leaves the singular vectors of intact, requires zero additional optimizer state memory, composes directly with NorMuon, is backed by a clean perturbation bound (Theorem 1), and adds only 3.8–6.6% wall-clock overhead.
- High experimental rigor and insightful analysis of non-stationary routing: The experimental protocol separates tuning seeds from evaluation seeds, pairs random seeds across initializations and data orders, and carefully controls for NorMuon's implicit per-step weight decay . Contrasting shuffled vs. language-blocked FineWeb2 (Section 4.4) clearly demonstrates when per-expert step sizes matter: spreads out most sharply 25 steps after a language switch () and consistently lowers router entropy across all budgets without hurting load balance.
Critical weaknesses
- Negligible loss improvements under standard i.i.d. shuffled pretraining at longer horizons: When training data are randomly shuffled, Compass yields very small or null loss gains once momentum conventions (N/N) and weight decay are matched. At 8.2M tokens (width 2,048), Compass (N/N) and Nesterov Muon are tied ( vs. , ); at 65.5M tokens with 1x NorMuon decay, Compass (N/N) and Nesterov Muon differ by nats (); at 262M and 1.05B tokens (Table 3 and Appendix A.2), Compass on the NorMuon update and NorMuon are statistically indistinguishable ( nats at 262M, 3/8 wins; nats over 5 seeds at 1.05B, 95% CI ); and on shuffled FineWeb2, all methods finish within – nats.
- The "Constant expert scale" control captures almost the entire gain over Nesterov Muon in the blocked-order setting: Because , the mean expert multiplier is slightly below 1 ( in blocked FineWeb2). In Table 4, the "Constant expert scale" ablation—which simply multiplies every expert step by the fixed scalar without computing any alignment cosines—achieves under blocked order, within nats of Compass (, vs. for Nesterov Muon). In the 8-seed replication (Table 12), the gap between Compass (N/H) and Constant expert scale is nats ( CI , ), and even against Nesterov Muon at matched LR , the gap shrinks to () for N/H and () for N/N. This suggests that in the Muon blocked comparison, a slightly smaller fixed expert learning rate explains most of the improvement over Nesterov Muon (though Compass on the NorMuon update remains a clear win). Finally, the continued pretraining run on JetMoE-8B (Appendix E) lacks a baseline Muon run for comparison.
Questions for authors
- Clarification on "Constant expert scale" vs. dynamic alignment in the blocked-order experiment: In Table 4 and Table 12, multiplying all expert steps by a constant captures over 90% of Compass (N/H)'s improvement over Nesterov Muon ( between Compass and Constant expert scale), and Table 12 shows that part of Nesterov Muon's worse loss came from selecting LR instead of (by a margin of nats on the two tuning seeds, Appendix A.4). Did you test a Constant expert scale control on top of the NorMuon update in the blocked-order experiment to verify how much of the nat gain on NorMuon comes from per-step alignment variation versus average expert step damping?
- Clarification on JetMoE-8B baseline (Appendix E): Did you run standard Nesterov or Heavy-ball Muon without Compass factors on JetMoE-8B under the same 1,000-step setting in Table 18, and if so, how did its validation loss compare to Compass's (at LR ) and (at LR )?
- Suggestion for future work: Since anneals toward 1 as (Equation 4), keeping fixed at 1 inside could be beneficial in continual learning or late-stage domain shifts where the learning rate is already small.
5 comments
(2 of 3)
At the end of a language's block. Tables R1 and R2 average the loss over all eight languages. Thus, they mostly measure languages that are not being trained. Table R3 looks only at the language that is just trained, at the end of the block. Each language returns after 700 updates on the other seven. Hence, Table R3 shows the quality of relearning by each method. The last three rows test your = 1 recommendation vs. the three controls with its mean multiplier (0.902).
The first control (Constant 0.902) scales every routed expert step by a fixed 0.902 across all layers and updates.
At each update, the second control (Shuffled multipliers) computes the per-expert multipliers online from its own state using the Compass () rule, then randomly permutes them among the routed experts within each family (one projection in one layer).
The third control (Family mean) gives every routed expert in a family that family's mean multiplier under = 1. And, this happens at each update. Note that the average stays the same and all experts in the family receive the same value.
Table R3. Validation loss of the language just trained, at the end of its block, averaged over cycles 2 to 5 (cycle 1 is each language's first visit). The seven rates run from 0.0014 to 0.0113.
| Update and rate | Seeds | Baseline | Baseline loss | Compass loss | Compass − baseline |
|---|---|---|---|---|---|
| NorMuon, 0.00566 | Paper + 16 new | NorMuon | 4.8233 | 4.8163 | −0.0070 [−0.0083, −0.0056] |
| NorMuon, 0.00566 | Paper + 16 new | Constant 0.949 | 4.8208 | 4.8163 | −0.0044 [−0.0062, −0.0027] |
| NorMuon, 0.00566 | 255 new | Constant 0.949 | 4.8206 | 4.8168 | −0.0038 [−0.0043, −0.0032] |
| NorMuon, mean of seven rates | 4 tuning | NorMuon | 4.8452 | 4.8403 | −0.0049 [−0.0061, −0.0036] |
| NorMuon, mean of seven rates | 4 tuning | Constant 0.949 | 4.8447 | 4.8403 | −0.0044 [−0.0056, −0.0032] |
| Muon (N/H), 0.00125 | Paper | Constant 0.948 | 4.8349 | 4.8309 | −0.0039 [−0.0062, −0.0016] |
| NorMuon, 0.00566, Compass (λ = 1) | Paper + 95 new | Constant 0.902 | 4.8181 | 4.8160 | −0.0021 [−0.0028, −0.0013] |
| NorMuon, 0.00566, Compass (λ = 1) | Paper + 32 new | Shuffled multipliers | 4.8172 | 4.8152 | −0.0020 [−0.0030, −0.0009] |
| NorMuon, 0.00566, Compass (λ = 1) | Paper + 32 new | Family mean | 4.8178 | 4.8152 | −0.0026 [−0.0036, −0.0016] |
Note that each difference in Table R3 is also below zero in every cycle from 2 to 5. After the same 100 updates on a returning language, Compass reaches a lower mean loss than NorMuon or a constant expert scale. In our view, this supports more effective adaptation within a fixed training budget beyond what the constant-scale control achieves.
Your suggestion of = 1. Thank you for this suggestion. We tried this approach in several experiments. Of the two factors, only the radius depends on , through the row factors of Eq. 4. In Compass, the radius 1 as the learning rate decays. Fixing = 1 in the row factors leaves the learning rate schedule (Eq. 9) unchanged. The radius then stays on, and the mean multiplier stays near 0.90 (Table R2). Note that the mean multiplier (the Compass scale placed on each expert's update) is below the 0.949 constant. Hence, Tables R3 and R4 compare = 1 with the three controls above. These controls have the same mean multiplier.
Table R4. Test loss on the NorMuon update under language blocks at 0.00566.
| Seeds | Baseline | Baseline loss | Compass (λ = 1) loss | Compass (λ = 1) − baseline | p |
|---|---|---|---|---|---|
| Paper + 32 new | NorMuon | 5.4837 | 5.4678 | −0.0159 [−0.0208, −0.0109] | < 0.001 |
| 63 new | NorMuon | 5.4858 | 5.4728 | −0.0129 [−0.0170, −0.0089] | < 0.001 |
| Paper + 32 new | Compass | 5.4775 | 5.4678 | −0.0097 [−0.0135, −0.0059] | < 0.001 |
| Paper + 95 new | Constant 0.902 | 5.4726 | 5.4707 | −0.0020 [−0.0050, +0.0010] | 0.20 |
| Paper + 32 new | Shuffled multipliers | 5.4683 | 5.4678 | −0.0005 [−0.0052, +0.0041] | 0.82 |
| Paper + 32 new | Family mean | 5.4700 | 5.4678 | −0.0022 [−0.0064, +0.0021] | 0.31 |
| 6 new (A100) | NorMuon | 5.4966 | 5.4712 | −0.0254 [−0.0402, −0.0105] | 0.031 |
At the last update, = 1 and the three controls end with nearly the same test loss (Table R4). At the end of each language's block, = 1 has lower loss on that language than all three controls (Table R3). None of the controls keeps the match of the expert and the corresponding multiplier. Thus, at the block end, it helps to give each expert the multiplier its own alignment produced. In JetMoE-8B, = 1 again gives the lowest final loss (Table R5 in Part 3). Yes, we will add = 1 to Section 3 as an option for data that keep shifting late in training.
(3 of 3)
W3 and Q2. Yes, we have now run expert-only Nesterov and heavy-ball Muon in the setting of Table 18 (released code and settings, 1000 updates of continued pretraining, both learning rates, 3 seeds). Compass reruns with the same seeds on the same machines as the baselines match the final losses of Table 18 within 0.0003 nats.
Table R5. JetMoE-8B, final validation loss, mean of three seeds. Expert-only Nesterov is Compass with both factors (the family factor and the scalar radius ) fixed at one.
| Method | Loss at 1e-4 | Compass − method | Loss at 5e-5 | Compass − method |
|---|---|---|---|---|
| Compass | 2.1147 | 2.0748 | ||
| Compass (λ = 1) | 2.1139 | 2.0746 | ||
| Expert-only Nesterov | 2.1184 | −0.0037 | 2.0763 | −0.0015 |
| Heavy-ball Muon | 2.1218 | −0.0071 | 2.0777 | −0.0029 |
Compass is lower than the two baselines at every evaluation of every seed and rate. This includes update 100, where each run reaches its best loss, as in Table 18. Fixing lowers Compass's final (step-1000) validation loss on every seed at both learning rates, while matching default Compass on the lowest validation loss along the run (reached at update 100, where is still close to 1).
W1. The abstract and the conclusion will say that the gain is the largest when the data each expert sees keep shifting, for instance under the language blocks. They will also say that with the long shuffled runs (262M and 1.05B tokens), Compass and NorMuon end close, and that in the shuffled multilingual runs, Compass is ahead early in training.
The contrast with the language-blocked training is consistent with the response of Compass to changes in an expert’s data. Compass compares the proposed update with the gradient of the current tokens and uses their alignment to adjust the step. Language blocks introduce abrupt changes that persist for 100 updates. With shuffled data, each batch still holds one language. The language changes at almost every update. Hence, no shift lasts long. Consistent with this distinction, we see that the spread of expert multipliers is greater 25 updates after a language switch than later in the block. However, it does not vary much across the corresponding checkpoints with shuffled data (Section 5).
Table R6. Data order in the multilingual setting, NorMuon update at 0.00566, test loss on the paper's seeds and the first 16 new ones (Table R1).
| Data order | NorMuon | Compass | Compass − NorMuon | p |
|---|---|---|---|---|
| Shuffled | 4.4043 | 4.4053 | +0.0011 [+0.0003, +0.0018] | 0.007 |
| Language blocks | 5.4867 | 5.4803 | −0.0063 [−0.0113, −0.0014] | 0.014 |
| Blocks minus shuffled | −0.0074 [−0.0124, −0.0024] | 0.005 |
On the shuffled multilingual runs, Compass on the NorMuon update also leads NorMuon early in training (by 0.016 nats in validation loss at update 400 across the eight seeds of Table 4), though NorMuon finishes slightly lower (0.0011 nats) in final test loss across the 32 seeds of Table R6.
At 65.5M tokens of FineWeb-Edu, Compass (N/N) and Nesterov Muon reach the same loss at NorMuon's per-step decay, as you note. Twice that decay lowers the loss of every method, and there, Compass (N/N) is lower than NorMuon and Nesterov Muon (Table 11). With the NorMuon update, where only the factors differ, Compass is lower at 65.5M tokens (Tables 3 and 11) and close to NorMuon at 262M and 1.05B (Table 3). At 262M, it still lowers the router entropy (Section 5).
W2. Table R7 completes the comparison under language blocks at one rate (0.00125) on the 16 seeds of Tables 4 and 12, with the missing runs added.
Table R7. Muon update under language blocks at 0.00125, test loss over the paper's 16 seeds. The first 8 are the seeds of Table 4, and every method has higher losses on the other 8.
| Method | Test loss | Compass − method | p |
|---|---|---|---|
| Compass (N/H) | 5.4939 | ||
| Constant 0.948 | 5.4959 | −0.0020 [−0.0082, +0.0042] | 0.50 |
| Nesterov Muon | 5.5036 | −0.0097 [−0.0180, −0.0013] | 0.027 |
| Expert-only Nesterov | 5.5041 | −0.0102 [−0.0191, −0.0013] | 0.016 |
At the matched learning rate of , Compass (N/H) still achieves a lower test loss than Nesterov Muon ( nats, ), though (as you noted) the fixed constant scale accounts for most of this difference ( nats between Compass and the constant approach, ).
Section 4.4 will specify this directly. And we will ensure that Figure 1 and Table 4 show Nesterov Muon at both learning rates. At the end of a language's block, Compass is still lower than the constant scale (Table R3). In the shorter runs at 8.2M tokens and width 1024, each Compass factor alone has a lower loss than the constant expert scale on all eight seeds (Table 1).
The revision will add Tables R1 to R7 to Section 4.4 and the appendix. We hope these results answer your questions.
(1 of 3)
Thanks for your review.
In our view, the core question is: “Is dynamic alignment behind the gains, or is it just step-size damping?” Our results show that both play a role. The constant ablation you asked for sets every expert's multiplier ceαe to one value, the mean from another Compass run. It reproduces a considerable part of the final loss improvement and finishes close to Compass. We see that setting the multipliers from alignment also helps. During language blocks, when a language returns, Compass ends the block with a lower loss on that language than the constant approach does. In the default Compass approach, the radius correction weakens as the learning rate decays. As you recommended, keeping λt = 1 inside the row factors keeps the radius on and ends with a lower test loss than NorMuon and default Compass. On a returning language, λt = 1 also ends the block with a lower loss than three controls with the same mean multiplier (Part 2). We see that the new JetMoE baselines favor Compass over expert-only Nesterov and heavy-ball Muon. Please see more details below.
Part 3 answers your point on the Muon comparison under language blocks (W2).
In all tables, runs are paired by seed, and each difference is the subtraction named in its label. A negative value favors the first term (usually Compass). Brackets give 95% confidence intervals from paired t-tests, and p-values come from a paired sign-flip test.
Q1. We multiplied the step of every expert that receives a gradient by 0.949. This is the mean ceαe of Compass on NorMuon in a separate run (for Muon, the paper used 0.948). Table R1 compares the methods at the same learning rate (0.00566). Hence, only the expert factors are different. (By contrast, the paper's drop in test loss of 0.0253 compares the methods at their own best learning rates from a grid search, 0.00566 for Compass and 0.00283 for NorMuon.) To test robustness and estimate the differences more precisely, Table R1 adds new seeds to the paper's 16.
Table R1. Test loss on the NorMuon update under language blocks at 0.00566, and for NorMuon also at 0.004, its best rate in Table R9. "Paper" is the paper's 16 seeds, and "new" seeds are new seeds from this rebuttal period.
Here, Compass has a lower loss than NorMuon at the same rate and even at NorMuon's best rate. On the paper's seeds and the first 16 new ones, the constant control reproduces about half of Compass’s loss improvement over NorMuon. On 255 new seeds, Compass and the constant approach end with nearly the same loss. Part 2 tests your λt = 1 recommendation vs. the controls having the same mean multiplier.
Table R2. Mean multiplier and differences in validation loss averaged over each cycle of eight language blocks (800 updates), on the NorMuon update. The constant approach keeps the scaling factor at 0.949.
In cycles 1 and 2, Compass has lower validation loss than both NorMuon and the constant approach. This suggests faster early learning. In those cycles, the mean multiplier of Compass is also below 0.949. As λt→0, the row factors and the radius → 1 (Eqs. 4 and 5). Compass's mean multiplier then nearly reaches 0.949 in cycle 3 and goes over it in cycle 4. The loss difference also changes sign. Under the original schedule, the early advantage does not lead to a clear final loss advantage over the constant approach (Table R1).