- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies router optimization in sparse MoE models. It analyzes a toy regression problem with a hard routing boundary, affine experts, and a continuous kinked target. Experts are refitted after each boundary move.
The loss grows as near the optimum. The gradient therefore vanishes quadratically. Gradient descent and Adam with fixed epsilon converge slowly. The authors propose SqrtGD. It applies a signed square root to the gradient and gives geometric convergence in the toy setting.
The practical method is SR. It keeps the base optimizer for most parameters. For the router, it takes momentum and applies a signed square root elementwise. The update is about SR helps with top 2 routing in 1.3B MoE LLM experiments. It gives little gain with top 4 or top 8. It fails with AdamW when router gates are renormalized.
Critical strengths
S1. Clear theory for a simple routing problem.
The paper explains why router gradients can vanish near a boundary. Theorem 1 gives a clean cubic loss mechanism. The proofs look correct. The simulations match the predicted rates. This is the main contribution.
S2. Simple method with strong top 2 results.
SR is cheap to implement. With top 2 routing, AdamW+SR reaches validation loss 3.203. The best AdamW baseline reaches 3.320. The gain is 0.117. SR is also stable where AdamW diverges. The authors honestly report failures with renormalized gates.
Critical weaknesses
W1. The theory does not fully explain the LLM setup.
The Adam analysis depends strongly on epsilon. The regime assumes is much smaller than epsilon. Synthetic tests use epsilon 0.1. Standard AdamW uses . With , Adam can become much faster in practice.
A simple scalar simulation shows this. After 1000 steps, Adam with epsilon reaches around . Adam with epsilon 0.1 reaches around . SqrtGD is still faster, but the gap is not simply polynomial versus geometric convergence.
The toy assumptions are also strong. The target must be continuous at the boundary. Experts must be refitted to least squares optima. LLM routers use soft gates and top k selection is non differentiable. Experts are updated throughout training. The paper does not show that LLM router gradients follow Theorem 1.
W2. The LLM experiments do not isolate why SR helps.
Table 1 uses one learning rate for all runs. The AdamW top 2 baseline diverges at that rate. Muon also seems poorly tuned. With tuned learning rates, the top 2 gain is 0.117 for AdamW but only 0.029 for Muon. Gains at top 4 and top 8 are small or absent. No random seeds or variance are reported.
The key missing baseline is a separately tuned router learning rate. SR is not scale invariant. Its update size is . It may simply reduce the effective router learning rate. This could explain both lower loss and better stability.
The paper should compare SR with AdamW using a tuned router learning rate. It should also test sign updates or plain momentum for the router.
The renormalized gate failure is serious. With Mixtral style gates, AdamW+SR diverges under top 2 routing. Under top 4 it is 0.41 worse than the baseline. The paper does not analyze this enough.
Questions for authors
-
What is the effective router learning rate under SR? Please report typical values and update magnitudes. Compare SR with AdamW using a separately tuned router learning rate. Also test identity, sign, and square root transforms.
-
How many random seeds were used? Please report variance for Tables 1 and 5. Were top 1 and runs trained? Top 1 is closest to the hard routing theory.
-
Which learning rates were used for Muon? Table 1 says . The Muon numbers match Table 5 at matrix learning rate 0.02. Please clarify router and non router learning rates.
-
Is the mechanism visible in LLM training? Do router gradients shrink as routing stabilizes? What happens before divergence in renormalized gate runs?
-
Does the rate hold for Adam with epsilon ? If not, please limit the scope of Theorem 4 and revise the claims.
-
How does Theorem 1 relate to change point regression? Please clarify the novelty relative to breakpoint estimation literature.
6 comments
Rebuttal 2/3
Q4: Is the mechanism visible in LLM training?
(1) What happens on divergence
In Appendix E.5, we measure the router dynamics on the plain AdamW run with un-renormalized gating at top-2 to analyze the routing mechanism behind the model divergence. The first loss spike (around step 8500) comes with a routing collapse in layer 0 and layer 1 (Figure 10 (a) and (b)): the load entropy values fall from around 2.2 (layer 1) / 1.4 (layer 0) to close to log 2, the value when two experts receive all tokens. Around step 14500, the loss diverges with a second spike, and at the same evaluation, every layer routes all tokens to two experts (Figure 10, (b)). Therefore, the router collapse corresponds clearly with the loss divergence.
Observational mechanistic comparison between AdamW and AdamW+SR
We run a set of experiments with a larger number of experts and top-4 routing with 10B tokens. We present the training curves with AdamW and SR at two different learning rates in our paper revision (Appendix E.6, Figure 11 and Table 10), where SR demonstrates a consistent improvement over AdamW.
Besides, we also conducted an analysis of the Router’s behavior under AdamW and AdamW with SR in Appendix E.6. According to Figure 12 and Table 11, adding SR would use more experts in the top layers (ones are close to the output), while AdamW seems to activate more experts in early layers (ones are close to the embedding layer). We don’t have a theoretical explanation for that yet, but this aligns with our intuition that in top layers, the LLM needs to acquire more sophisticated semantic features and requires more capacity and specification among various experts. The number of experts in use per MoE layer and corresponding validation loss at 25k steps (3.3B training tokens) is presented below:
| MoE layer | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | all (of 1,024) | val. loss |
|---|---|---|---|---|---|---|---|---|---|---|
| AdamW, peak LR | 11 | 25 | 34 | 41 | 37 | 17 | 16 | 16 | 197 | 3.560 |
| SR, peak LR | 4 | 96 | 16 | 28 | 60 | 66 | 81 | 67 | 418 | 3.507 |
| AdamW, peak LR | 18 | 101 | 95 | 83 | 86 | 39 | 30 | 20 | 472 | 3.514 |
| SR, peak LR | 17 | 16 | 13 | 88 | 20 | 55 | 69 | 68 | 346 | 3.475 |
Router gradient steps
We measure and present the router’s gradient steps in each run in Appendix E.6.1, Figure 13. We track the router gradient norm (Equation 53) every 500 steps. The router gradient does not decay or spike extremely under any rule. Therefore, the router gradient does not vanish in these runs, while does encourage more stable gradient steps compared to plain AdamW.
Rebuttal 3/3
Q5: Does the rate hold on Adam with ?
We present the error curves with various values of in our paper revision, Appendix E.1 and Figure 8. The performance shows that SqrtGD is faster than Adam for all .
W2: Explanation on the gate renormalization
We analyze the router gradients under gate renormalization. According to Appendix E.4, with renormalization, only the router logits from selected experts have gradients. Therefore, the language-model loss gives exactly zero gradient to unselected logits, so those coordinates are driven by the auxiliary losses only; therefore, SR amplifies the wrong gradients. It falls apart from our theoretical setup, where router logits for all experts will get gradients. We also validate that AdamW with SR under non-renormalized gating outperforms AdamW with renormalization in Table 6.
| Recipe | seed 0 | seed 1 | seed 2 |
|---|---|---|---|
| AdamW | 6.550 | 5.741 | 7.416 |
| SR | 3.203 | 3.173 | 3.218 |
| AdamW, renormalized gate | 3.255 | 3.293 | 3.282 |
Q6. Relation to change-point regression.
Thanks for pointing us to the related works we missed! We have added the related literature to the related work section in the paper revision. The breakpoint literature has three relevant cases: A jump at the change point gives a criterion linear in and a super-consistent estimator ([1, 2]). A continuous model fitted with the continuity constraint gives a quadratic criterion ([3, 4]). A continuous truth fitted without the constraint, which is our setting of independently refit experts, gives a cube-root rate for the threshold estimate (Hidalgo, Lee and Seo 2019), which we consider as the statistical counterpart of our established cubic population criterion in Theorem 1. Therefore, the cubic geometry is not new in the change-point regression literature, and we will say so and cite those related works. Our novelty lies in its optimization consequence: the rates of GD and Adam on this criterion, the square-root update that restores geometric convergence, the extension to coupled boundaries, and the relevance to a hard MoE router.
—----------------
Sorry for posting the responses late, since the experiments take time. We hope our answer and additional results can resolve your concerns, and you can consider raising the score accordingly. If you have any further questions, we are happy to address them!
[1] David V. Hinkley. Inference about the change-point in a sequence of random variables. Biometrika, 57(1):1–17, 1970.
[2] Jushan Bai and Pierre Perron. Estimating and testing linear models with multiple structural changes. Econometrica, 66(1):47–78, 1998.
[3] Paul I. Feder. On asymptotic distribution theory in segmented regression problems—identified case. The Annals of Statistics, 3(1):49–83, 1975
[4] Kung-Sik Chan and Ruey S. Tsay. Limiting properties of the least squares estimator of a continuous threshold autoregressive model. Biometrika, 85(2):413–426, 1998.
[5] Javier Hidalgo, Jungyoon Lee, and Myung Hwan Seo. Robust inference for threshold regression models. Journal of Econometrics, 210(2):291–309, 2019.
Rebuttal 1/3
We sincerely thank the reviewer for the insightful feedback, and we really appreciate your close reading through our theory and carefully assessing our experiments! We respond to each of your concerns below.
Q1: Independent learning rate sweep on the router
We conducted experiments with E=32 top-2 routing setup with an independently tuned router learning rate. All non-router parameters use a peak learning rate 3×10−4, as the optimum selected for plain AdamW according to Table 5 in the paper.
We report the results in our paper revision (Appendix E.3, Figure 9, Table 8). Firstly, we do observe that the untuned learning rate on the router is not optimal for both AdamW and AdamW+SR; second, SR still outperforms AdamW at the optimum of the router's learning rate. We also provide the statistics below.
##Q2: Top-1 hard-routing results and multiple random seeds
Top-1 Results. We conducted experiments with E=16 and E=32 experts, top-1 routing. In both cases, SR outperforms plain AdamW and Muon, where SR consistently outperforming AdamW and Muon.
Random Seeds We run E=32, top-2 experiments with three random seeds and report the statistics in the paper revision (Appendix E.2, Table 6). According to the results, AdamW with SR is consistently more stable than plain AdamW and achieves lower validation losses than AdamW with renormalization. We present the statistics below.
It’s hard to get all setups (38 pretraining setups x 3 seeds) to run with multiple random seeds during the rebuttal period, but we promise to provide the variance number in our main Table in our later revision.
Q3: Learning rate on Muon
Thanks for pointing it out, and it’s our mistake that caused the confusion! 10−3 and 0.02 are relatively applied on the non-matrices parameter groups and matrix parameter groups in Muon. We have clarified it in the revision.