- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies why routing collapse emerges and persists in sparse MoE pretraining. The authors pretrain Dense, Softmax (dense routing), Softmax-TopK, TopK-Softmax and their auxiliary-loss variants at two scales on 100B tokens, measuring utilization with a probability-based effective number of experts N_eff. N_eff drops or fluctuates sharply within the first few hundred steps, then follows one of three trajectories (stuck, rebound, or slow recovery). The collapse severity, its layer patterns, and its response to the auxiliary loss depend on the routing function.
The paper show that hidden representations reorganize early in training for every variant, including Dense, and argue that this reorganization perturbs routing logits. They then derive that, for unselected experts, the logit gradient is zero under TopK-Softmax and a shared scalar times p_i under Softmax-TopK, so the router cannot apply an expert-specific correction. To test this, they replace the router gradient with one computed from counterfactual expert outputs on per-cluster mean hidden states. This intervention raises N_eff substantially without an auxiliary loss, and average zero-shot accuracy rises from 50.62 to 53.21 at 0.68B and from 54.35 to 55.45 at 1.92B.
Critical strengths
The controlled pretraining study isolates the routing operator. The authors hold data, architecture and optimizer fixed and vary only the operator and the auxiliary loss, at two scales on 100B tokens each. This gives a clean comparison where dense Softmax stays balanced while both sparse operators collapse in different ways.
The paper proposes concrete, testable mechanism with a cheap intervention. The gradient decomposition gives a precise explanation for why TopK-Softmax responds poorly to the auxiliary loss. The intervention adds negligible FLOPs and recovers N_eff without any balancing objective.
Critical weaknesses
The causal claim is understudied. The intervention does three things at once. It gives unselected experts a nonzero gradient, it makes that gradient expert-specific, and it changes the selected experts' gradients by replacing y with y_hat as the reference. The paper attributes the effect to "expert-specific information", but no control isolates that factor. For example, a surrogate with shuffled or random counterfactual outputs (nonzero but not expert-specific) would test it directly.
With one seed per variant, the three trajectory types in Sec. 4.1 could also reflect seed variance rather than routing dynamics.
The models use 8 and 16 experts with K = 2, whereas current MoEs use 64–256 fine-grained experts, often with shared experts. Whether the mechanism and intervention carry over to practical scales is unclear. Notably, the intervention is close in spirit to prior methods that give the router a dense gradient by substituting estimated outputs for unselected experts. Examples are the default-output approach of Panda et al. (cited in the paper) and SparseMixer-style gradient estimators. These are neither discussed nor compared. The auxiliary-loss-free bias method (Wang et al., 2024) is cited but not run.
The paper reports no validation loss on pretraining data. Downstream differences are also small: at 1.92B, "Ours" (55.45) is below TopK-Softmax_aux (55.71), and Ours_aux does not improve on Softmax-TopK_aux at 0.68B (52.85 versus 53.08).
Questions for authors
How much do the N_eff trajectories and final values vary across seeds? Do the three trajectory types in Sec. 4.1 persist for a given operator and layer across seeds?
Can you add a control that gives unselected experts a nonzero but non-expert-specific gradient? This would show whether expert-specific information, causes the recovery.
4 comments
Robustness Across Seeds
We repeated the experiments with a different initialization and data order (seed 5), with results provided in Appendix E.5 ([Fig 11] and [Fig 12-13]). While the exact trajectories vary across seeds, the key qualitative dynamics are reproduced: early routing perturbation followed by persistent collapse, rapid rebound, or gradual recovery. The final layer-wise values also reproduce the operator-dependent collapse patterns.
Purpose of the Gradient Intervention
Our study aims to understand the role of sparse router-gradient limitations in sustaining routing collapse. Sec. 4.3 shows that severe routing collapse is associated with lower downstream performance, while alleviating collapse is accompanied by improved performance. Our intervention further shows that modifying the router gradient alone can alleviate collapse without an explicit balancing objective, while achieving downstream performance comparable to sparse MoEs trained with auxiliary loss.
Validation Loss
We additionally evaluated the validation loss across all model variants and both scales. The results are reported below:
| Method | 0.68B Total | 0.68B LM | 0.68B Aux | 1.92B Total | 1.92B LM | 1.92B Aux |
|---|---|---|---|---|---|---|
| Dense | 3.177 | 3.177 | 0.000 | 3.026 | 3.026 | 0.000 |
| Soft-TopK | 3.012 | 3.012 | 0.000 | 2.831 | 2.831 | 0.000 |
| TopK-Soft | 3.000 | 3.000 | 0.000 | 2.822 | 2.822 | 0.000 |
| Softmax | 2.956 | 2.956 | 0.000 | 2.731 | 2.731 | 0.000 |
| Ours | 2.989 | 2.989 | 0.000 | 2.825 | 2.825 | 0.000 |
| Soft-TopK aux | 3.002 | 2.982 | 2.026 | 2.825 | 2.805 | 2.025 |
| TopK-Soft aux | 3.014 | 2.991 | 2.267 | 2.818 | 2.794 | 2.367 |
| Ours aux | 3.007 | 2.986 | 2.048 | 2.821 | 2.801 | 2.040 |
Validation at Larger MoE Scales
We agree that validation at larger MoE scales is important. Our experiments systematically compare eight model variants across two scales under a limited compute budget, following MoE pretraining settings feasible at academic scale [4,5]. We have added Sec. 8 (Limitations) to explicitly acknowledge the limited scale and the lack of evaluation with larger expert counts and shared-expert architectures.
Reference
[4] Jin et al.,Moe++: Accelerating mixture-of-experts methods with zero-computation experts, ICLR, 2025.
[5] Lv et al., Autonomy-of-experts models, ICML, 2025.
Dear Reviewer wtpD,
Thank you again for your thoughtful and constructive feedback.
As the discussion period draws to a close, we wanted to briefly follow up on our rebuttal. We believe that the additional experiments, analyses, and revisions have addressed the concerns raised in your review.
We would be grateful to hear if there is anything further we could clarify. If you find that the new evidence addresses your concerns, we would sincerely appreciate your consideration of an updated rating.
Thank you again for your time and consideration.
Best regards, The Authors
Dear Reviewer wtpD,
We understand this is a busy time. As the discussion period is closing very soon, we would like to kindly follow up one last time.
If any concern remains after our response, please let us know and we will reply right away. If your concerns have been addressed, we would be grateful if you could consider updating your rating.
Thank you again for your time.
Best regards, The Authors
We sincerely thank you for your thoughtful and constructive feedback. In the following responses, we carefully address the points you have raised and have updated the manuscript accordingly (*changes highlighted in magenta).
To make the additional experimental results easier to access, we also provide anonymized links.
Given the rebuttal timeline and available compute, we ran the additional experiments for 5K steps, with each run taking roughly 1.3–3.5 days for the 0.68B sparse MoE model, depending on the GPU.
We agree that Eq. 10 does not fully isolate the unselected-expert gradient effect. We initially modified only unselected-expert gradients to test the limitation of selection-dependent sparse gradients, but this led to unstable training with non-finite activations. We attribute this to violating the zero-sum property of Softmax logit gradients (Appendix D.3), motivating Eq. 10 to preserve this structure and training stability.
We agree that this prevents complete isolation of the unselected-expert gradient effect. We have added Sec. 8 (Limitations) and explicitly noted this limitation of our intervention.
We agree that our intervention is related to prior approaches that provide gradient pathways to unselected experts. We have expanded Appendix B Related Work to discuss DefaultMoE [1] and SparseMixer [2] and clarify their relation to our study.
Given the rebuttal timeline, we prioritized evaluating DefaultMoE, which was also requested by Reviewer Qf4K. DefaultMoE and ours differ in three aspects:
Thank you for suggesting these important controls. Following your suggestion, we implemented RandomGrad by providing nonzero Gaussian gradients to unselected experts while preserving zero-mean router gradients. We also evaluated DefaultMoE [1] and the auxiliary-loss-free bias method (Aux-Free) [3]. RandomGrad and DefaultMoE were evaluated both w/ and w/o auxiliary loss. Results are provided in Appendix E.6 ([Fig. 14]).
Reference
[1] Panda et al., Dense backpropagation improves training for sparse mixture-of-experts, NeurIPS, 2026.
[2] Liu et. al., Sparse backpropagation for moe training, arXiv, 2023.
[3] Wang et al., Auxiliary-loss-free load balancing strategy for mixture-of-experts, arXiv, 2024.