- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper analyzes the standard Switch auxiliary load-balancing loss and disproves the common claim that its minimum value is 1 under uniform routing. It derives the sharp global infimum, characterizes all extremal and near-extremal hard-load configurations, and shows that minimizing the loss can favor configurations with roughly half the experts unused. The paper also gives loss-decreasing directions that preserve routing assignments and, for top-k routing with post-selection renormalization, routed outputs. The theory is supported by measurements on public Switch, Qwen, and DeepSeek checkpoints and by controlled MoE training experiments.
Critical strengths
- The paper does more than provide a counterexample: it derives the sharp infimum and characterizes the extremal hard-load family, making the failure mode precise.
- The observations in released checkpoints and controlled training show that sub-one losses and hard-soft anti-alignment are not purely pathological constructions.
Critical weaknesses
- The sharp infimum describes what the loss permits, but not how often realistic optimization approaches the pathological regime when combined with the task loss.
- The conclusions are highly relevant to Switch-style balancing, but the extent to which they transfer to newer balancing mechanisms is less clear.
Questions for authors
Q1. Do the authors have evidence on how close large-scale training trajectories typically move toward the characterized near-extremal geometry, rather than merely crossing below 1? Q2. Can the authors clarify which commonly used modern MoE balancing losses avoid the same hard-soft mismatch mechanism? Q3. In the controlled training experiments, how sensitive are the observed negative hard-soft correlations to the task loss weight and router regularization choices?
5 comments
Response to Reviewer rfYD (2/3)
2. Which balancing mechanisms avoid the mismatch (W2, Q2)
(W2) ... the extent to which they transfer to newer balancing mechanisms is less clear.
(Q2) Can the authors clarify which commonly used modern MoE balancing losses avoid the same hard-soft mismatch mechanism?
We agree that how far the conclusions transfer depends on the mechanism. What matters is whether a mechanism multiplies each expert's share of tokens by its mean router probability, as the Switch loss does. Mechanisms without this product do not reward the mismatch: a correction bias used alone, assignment-based routing, and soft-only objectives such as the mutual-information loss of ModuleFormer (Shen et al., 2023), although soft balance alone does not ensure balanced dispatch (Sec. 7). Every loss with this product shares the mechanism, whatever set of tokens it is computed over, and this Switch form remains widely used, including in recipes built around a correction bias.
| Mechanism | Examples | Relation to the mismatch |
|---|---|---|
| Correction bias alone | Loss-Free Balancing (Wang et al., 2024); quantile balancing, as in Kimi K3 (Kimi Team, 2026) | No soft term, so nothing rewards the mismatch; in the prototype for Solar Open (Park et al., 2026), the bias alone left early layers severely imbalanced until a load-balancing loss was added |
| Assignment-based or fixed routing | Expert Choice (Zhou et al., 2022), BASE (Lewis et al., 2021), Sinkhorn routing in Megatron-Core; hash routing (Roller et al., 2021) in DeepSeek-V4's first layers | Balanced by construction, or fixed; no soft term |
| Soft-only balancing | importance loss (Shazeer et al., 2017); the mutual-information losses of ModuleFormer, used for the released MoLM models (Shen et al., 2023), and of DS-MoE (Pan et al., 2024); the uniformity loss of Monet, with released checkpoints (Park et al., 2024); φ-balancing (Chen et al., 2026); the Top-1 LBL explored for Sigma-MoE-Tiny (Hu et al., 2025) | No cross term. Soft balance alone does not ensure balanced dispatch, since nearly tied gates can send every token to one expert (Sec. 7); the mutual-information losses and the Top-1 LBL also reward confident routing |
| Switch-form loss over a micro-batch, sequence, global batch or other set of tokens | Switch, GShard, OLMoE, Marco-MoE, MiniCPM-MoE, MiniMax-Text-01, Pangu Pro and Ultra MoE, Qwen3 and Sigma-MoE-Tiny (global batch), DeepSeek-V2, V3 and Trinity Large (sequence-wise); Megatron-Core and Hugging Face transformers | Same mechanism: on any set of tokens, moving soft mass toward less-used experts at fixed hard loads lowers the loss |
| Correction bias with a balance loss | DeepSeek-V3/V4, GLM-4.5, MiMo-V2-Flash, Nemotron 3, Trinity Large, LongCat-Flash, Step 3.5 Flash, Solar Open, K-EXAONE, A.X K1, dots.llm1, MiniMax-M2 | The bias has no soft term and an approximate balance guarantee (Han & Zhong, 2025), but these recipes also keep a small Switch-form loss |
- Correction bias with a small Switch-form loss. DeepSeek-V3 renormalizes its gates over the selected experts, so the scores of unselected experts do not enter the output, and its sequence-wise Switch-form loss, with weight , is the only term of the objective with a gradient on them. With or without its bias, its router shows the geometry this loss rewards in our analysis: in layer 3, the first sparse layer, on the evaluation data of Sec. 6 is 0.977–0.985 for V3 and V3.1, below one, with negative hard–soft correlation, and the ten least-used experts receive 10–25% of their uniform share of tokens yet hold 1.05–1.12 times the uniform soft mass. Trinity Large keeps a loss of the same form at the same weight, and its first sparse layer shows the same pattern ( of 0.973 with its bias and 0.972 without, negative correlation in both). In these recipes, pairing the bias with a small Switch-form loss therefore does not avoid the mismatch.
Additional references
- Kimi Team (2026). Kimi K3: Open Frontier Intelligence. arXiv:2607.24653.
- Lewis, M., et al. (2021). BASE Layers: Simplifying Training of Large, Sparse Models. arXiv:2103.16716.
- Pan, B., et al. (2024). Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models. arXiv:2404.05567.
- Park, J., et al. (2024). Monet: Mixture of Monosemantic Experts for Transformers. arXiv:2412.04139.
- Park, S., et al. (2026). Solar Open Technical Report. arXiv:2601.07022.
- Roller, S., et al. (2021). Hash Layers For Large Sparse Models. arXiv:2106.04426.
Response to Reviewer rfYD (3/3)
3. How the correlation depends on the loss weight and router regularization (Q3)
(Q3) In the controlled training experiments, how sensitive are the observed negative hard-soft correlations to the task loss weight and router regularization choices?
Thank you for this question, which we answered with new runs at and 0.1, and at without the router z-loss. The correlation depends on how strong the Switch term is relative to the task loss, and we vary this relative strength through the Switch coefficient . As grows from zero, both layers go through the same transition. Without the Switch term, dispatch collapses, the hard–soft correlation is close to +1 and is far above one; once the term is strong enough to shape the router, at in layer 3 and 0.003 in layer 1, falls below one and the correlation turns negative in every seed. The negative correlation persists in an ablation without the router z-loss at , where is negative for every seed in both layers.
- Task-loss weight. Under AdamW, only the relative weights of the loss terms matter: multiplying the task loss by is essentially equivalent to dividing the Switch and z-loss coefficients by . A larger task-loss weight therefore corresponds to a smaller , so the sweep over below also covers the task-loss weight; the z-loss is treated separately below. Each entry gives the range over three seeds at 4,096 updates in the paper's setup; , 0.01 and 0.03 are the paper's runs.
| 0 | 0.003 | 0.01 | 0.03 | 0.1 | 0.03, no z-loss | |
|---|---|---|---|---|---|---|
| Layer 3 | +1.00 | +0.86 to +0.88 | −0.22 to −0.04 | −0.46 to −0.33 | −0.28 to −0.20 | −0.51 to −0.43 |
| Layer 3 | 120–180 | 2.19–2.33 | 0.90–0.98 | 0.84–0.89 | 0.94–0.96 | 0.84–0.87 |
| Layer 3 hard-load TV | 0.99 | 0.78–0.80 | 0.46–0.50 | 0.24–0.26 | 0.16–0.17 | 0.25 |
| Layer 3 zero-dispatch experts | 235–247 | 2–4 | 0–1 | 0 | 0 | 0 |
| Layer 1 | +0.98 to +1.00 | −0.21 to −0.16 | −0.30 to −0.03 | −0.25 to +0.07 | −0.06 to +0.10 | −0.36 to −0.17 |
| Layer 1 | 42–55 | 0.92–0.93 | 0.88–0.99 | 0.94–1.01 | 0.99–1.01 | 0.89–0.97 |
| Layer 1 hard-load TV | 0.99–1.00 | 0.51–0.59 | 0.30–0.36 | 0.21–0.23 | 0.19–0.20 | 0.22–0.25 |
| Layer 1 zero-dispatch experts | 251–252 | 8–18 | 2–5 | 0 | 0 | 0 |
| Held-out NLL | 5.424–5.433 | 5.410–5.422 | 5.418–5.432 | 5.418–5.443 | 5.427–5.431 | 5.416–5.421 |
- The transition. In both layers, raising from zero takes from far above one to below one and from close to +1 to negative in every seed, at in layer 3 and at in layer 1. Within a seed, the runs differ only in the loss coefficients, so this transition is produced by the Switch term itself, as our analysis predicts; held-out NLL stays between 5.410 and 5.443 throughout.
- After the transition. In layer 3, stays negative in every seed up to . In layer 1, larger mainly balances the hard loads, and moves to around zero.
- Why the layers differ. The two routers differ in confidence: at , the selected expert gets on average 13.7 times the uniform weight in layer 3 but only 2.9 times in layer 1.
- In layer 3, a weak Switch term cannot overcome this confident router, so at dispatch stays concentrated and the soft mass follows it. A stronger term lowers the selected expert's weight, to 7.9 times at , and the released mass goes mainly to low-load experts: the soft mass of the lowest-load quarter rises from 0.271 to 0.314 and that of the highest-load quarter falls from 0.206 to 0.171, the redistribution that lowers the loss at fixed hard loads (Theorem 3.1). The correlation therefore turns negative and falls further at .
- In layer 1, the router is relatively close to uniform, so even a weak Switch term, at , makes negative. For the same reason, a stronger term has little soft mass to shift onto low-load experts: it mainly changes which expert wins each token, and the lowest-load quarter gains only a little soft mass (0.263 to 0.277). As the hard loads become more balanced, moves to around zero; across seeds, the less confident the router, the closer is to zero.
- Router regularization. The last column reruns without the router z-loss. Without it, is negative for every seed in both layers, and in layer 1 each seed's is lower than with the z-loss.
We hope these results address your questions. If so, we would be grateful if you would consider raising your score; if any concern remains, please let us know, and we will gladly address it before the discussion closes.
Response to Reviewer rfYD (1/3)
Thank you for the careful summary and for recognizing both the precision of the characterization and the relevance of the observations. We answer each question below.
1. How close training gets to the extremal geometry (W1, Q1)
We agree that the infimum describes what the loss permits, not where training ends up. Training stays far from the infimum, because the task loss resists approaching it. Still, values below one are common, and released models show more than a crossing. In Switch Transformer, Qwen and Marco models, these values come with experts that receive no tokens yet hold above-uniform soft mass, the signature of the extremal geometry. Through further pretraining of Trinity Nano and DeepSeek, the values stay below one and the same experts stay underused. Our practical conclusion, that a low Switch loss does not certify balance, rests on these observations and does not require reaching the infimum.