acceptodds
Under review as a conference paper at ICLR 2027

When a Load-Balancing Loss Rewards Dead Experts: Sharp Global Geometry of Switch Routing

Abstract

Top-1 mixture-of-experts routers are commonly regularized by the Switch auxiliary loss , which couples hard expert frequencies with mean soft routing probabilities . It is widely believed that the loss attains its minimum value of 1 under uniform routing. We show that this claim is false. We prove the sharp infimum The bound is approached by routing all tokens evenly among roughly *half* the experts, leaving the rest unused. For each token, the routing probabilities approach a uniform distribution over its assigned expert and all unused experts, with the assigned expert remaining the unique top-1 choice. We characterize all extremal hard loads and prove that hard-load vectors must approach this half-dead family as the loss approaches its infimum. We also construct cases where the loss decreases while expert assignments and routed outputs remain unchanged under top- routing with post-selection renormalization. We explicitly observe normalized losses below 1 in layers of released Switch Transformer, Qwen, and DeepSeek models. In Switch Transformer and Qwen, such values coexist with unused experts that retain above-uniform soft mass. Controlled language-model training experiments also produce losses below 1 alongside unused experts and show that Switch regularization can induce *negative* hard–soft correlation. These results show that minimizing the auxiliary loss does not necessarily balance expert utilization.

Primary area:
optimization
Keywords:
mixture-of-experts, large language models, load balancing, expert routing, Switch loss

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_rfYDOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper analyzes the standard Switch auxiliary load-balancing loss and disproves the common claim that its minimum value is 1 under uniform routing. It derives the sharp global infimum, characterizes all extremal and near-extremal hard-load configurations, and shows that minimizing the loss can favor configurations with roughly half the experts unused. The paper also gives loss-decreasing directions that preserve routing assignments and, for top-k routing with post-selection renormalization, routed outputs. The theory is supported by measurements on public Switch, Qwen, and DeepSeek checkpoints and by controlled MoE training experiments.

Critical strengths
  1. The paper does more than provide a counterexample: it derives the sharp infimum and characterizes the extremal hard-load family, making the failure mode precise.
  2. The observations in released checkpoints and controlled training show that sub-one losses and hard-soft anti-alignment are not purely pathological constructions.
Critical weaknesses
  1. The sharp infimum describes what the loss permits, but not how often realistic optimization approaches the pathological regime when combined with the task loss.
  2. The conclusions are highly relevant to Switch-style balancing, but the extent to which they transfer to newer balancing mechanisms is less clear.
Questions for authors

Q1. Do the authors have evidence on how close large-scale training trajectories typically move toward the characterized near-extremal geometry, rather than merely crossing below 1? Q2. Can the authors clarify which commonly used modern MoE balancing losses avoid the same hard-soft mismatch mechanism? Q3. In the controlled training experiments, how sensitive are the observed negative hard-soft correlations to the task loss weight and router regularization choices?

5 comments
AuthorsOpenReview ↗

Response to Reviewer rfYD (1/3)

Thank you for the careful summary and for recognizing both the precision of the characterization and the relevance of the observations. We answer each question below.

Your questionShort answer
How close training gets to the extremal geometry (W1, Q1)Training stays far from the infimum, which the prediction loss resists, but released models show the signature of the extremal geometry; our practical conclusion does not require reaching the infimum
Newer balancing mechanisms (W2, Q2)Mechanisms without the hard–soft product, such as a correction bias used alone, assignment-based routing and soft-only objectives, do not reward the mismatch; however, the Switch form, which has the product, remains widely used, including alongside correction biases (Part 2)
Sensitivity (Q3)Without the Switch term, rr is close to +1; it turns negative when the term becomes strong enough, and is negative for every seed in both layers in an ablation without the router z-loss (Part 3)

1. How close training gets to the extremal geometry (W1, Q1)

(W1) The sharp infimum describes what the loss permits, but not how often realistic optimization approaches the pathological regime when combined with the task loss.
(Q1) Do the authors have evidence on how close large-scale training trajectories typically move toward the characterized near-extremal geometry, rather than merely crossing below 1?

We agree that the infimum describes what the loss permits, not where training ends up. Training stays far from the infimum, because the task loss resists approaching it. Still, values below one are common, and released models show more than a crossing. In Switch Transformer, Qwen and Marco models, these values come with experts that receive no tokens yet hold above-uniform soft mass, the signature of the extremal geometry. Through further pretraining of Trinity Nano and DeepSeek, the values stay below one and the same experts stay underused. Our practical conclusion, that a low Switch loss does not certify balance, rests on these observations and does not require reaching the infimum.

  • Why training stays far from the infimum. Under top-1 routing, the infimum requires the selected expert's weight to fall to about 2/N2/N; this weight scales the expert's output, so the prediction loss resists this decrease (Sec. 6, Summary).
  • How often (W1). Sub-one values occur in 11 of the 12 sparse layers of Switch-Base-256 and in 9 of the 28 layers of Qwen2-57B-A14B, and replicate on the evaluation data of Sec. 6 in newer checkpoints, including Qwen3.5, LLM-jp-4, Trinity and Marco. In our six training runs with the Switch term, counting both MoE layers and all 4,096 updates, the Switch loss computed on each training batch is below one 72% of the time; without the Switch term, no batch loss falls below one.
  • The signature of the extremal geometry. In the extremal family, unused experts hold above-uniform soft mass, and released models show this: in Qwen2-57B-A14B layer 11, expert 49 receives none of 8.4M assignments yet holds 1.07 times the uniform soft mass; in Switch-Base-256 encoder layer 11, expert 37 receives none yet holds 2.12 times; and in Marco-Nano layer 0, expert 203 receives none of 8.4M yet holds 2.14 times.
  • Persistence at pretraining scale. From Trinity Nano's pre-anneal (8.8T tokens) to its final (10T) checkpoint, the Switch statistic of layer 2 stays below one on the evaluation data of Sec. 6 (0.991 and 0.982), while experts 1, 19 and 55 receive less than 10% of their uniform share at both checkpoints. Across DeepSeek-V3 and V3.1 (839B additional tokens), the ten least-used experts of the first sparse layer are the same.
  • Direction in our training. In the paper's runs at α=0.03\alpha=0.03, the layer-3 Switch loss is already below one at 2,048 updates and falls further by 4,096 (from 0.92–0.98 to 0.84–0.89) as soft mass shifts toward the less-used experts: in every seed, the hard–soft correlation goes from between −0.27 and −0.06 to between −0.46 and −0.33, while hard-load TV stays between 0.24 and 0.30. These runs thus do more than cross below one: they keep moving toward the extremal geometry.
AuthorsOpenReview ↗

Response to Reviewer rfYD (2/3)

2. Which balancing mechanisms avoid the mismatch (W2, Q2)

(W2) ... the extent to which they transfer to newer balancing mechanisms is less clear.
(Q2) Can the authors clarify which commonly used modern MoE balancing losses avoid the same hard-soft mismatch mechanism?

We agree that how far the conclusions transfer depends on the mechanism. What matters is whether a mechanism multiplies each expert's share of tokens by its mean router probability, as the Switch loss does. Mechanisms without this product do not reward the mismatch: a correction bias used alone, assignment-based routing, and soft-only objectives such as the mutual-information loss of ModuleFormer (Shen et al., 2023), although soft balance alone does not ensure balanced dispatch (Sec. 7). Every loss with this product shares the mechanism, whatever set of tokens it is computed over, and this Switch form remains widely used, including in recipes built around a correction bias.

MechanismExamplesRelation to the mismatch
Correction bias aloneLoss-Free Balancing (Wang et al., 2024); quantile balancing, as in Kimi K3 (Kimi Team, 2026)No soft term, so nothing rewards the mismatch; in the prototype for Solar Open (Park et al., 2026), the bias alone left early layers severely imbalanced until a load-balancing loss was added
Assignment-based or fixed routingExpert Choice (Zhou et al., 2022), BASE (Lewis et al., 2021), Sinkhorn routing in Megatron-Core; hash routing (Roller et al., 2021) in DeepSeek-V4's first layersBalanced by construction, or fixed; no soft term
Soft-only balancingimportance loss (Shazeer et al., 2017); the mutual-information losses of ModuleFormer, used for the released MoLM models (Shen et al., 2023), and of DS-MoE (Pan et al., 2024); the uniformity loss of Monet, with released checkpoints (Park et al., 2024); φ-balancing (Chen et al., 2026); the Top-1 LBL explored for Sigma-MoE-Tiny (Hu et al., 2025)No cross term. Soft balance alone does not ensure balanced dispatch, since nearly tied gates can send every token to one expert (Sec. 7); the mutual-information losses and the Top-1 LBL also reward confident routing
Switch-form loss over a micro-batch, sequence, global batch or other set of tokensSwitch, GShard, OLMoE, Marco-MoE, MiniCPM-MoE, MiniMax-Text-01, Pangu Pro and Ultra MoE, Qwen3 and Sigma-MoE-Tiny (global batch), DeepSeek-V2, V3 and Trinity Large (sequence-wise); Megatron-Core and Hugging Face transformersSame mechanism: on any set of tokens, moving soft mass toward less-used experts at fixed hard loads lowers the loss
Correction bias with a balance lossDeepSeek-V3/V4, GLM-4.5, MiMo-V2-Flash, Nemotron 3, Trinity Large, LongCat-Flash, Step 3.5 Flash, Solar Open, K-EXAONE, A.X K1, dots.llm1, MiniMax-M2The bias has no soft term and an approximate balance guarantee (Han & Zhong, 2025), but these recipes also keep a small Switch-form loss
  • Correction bias with a small Switch-form loss. DeepSeek-V3 renormalizes its gates over the selected experts, so the scores of unselected experts do not enter the output, and its sequence-wise Switch-form loss, with weight 10−410^{-4}, is the only term of the objective with a gradient on them. With or without its bias, its router shows the geometry this loss rewards in our analysis: in layer 3, the first sparse layer, L8L_8 on the evaluation data of Sec. 6 is 0.977–0.985 for V3 and V3.1, below one, with negative hard–soft correlation, and the ten least-used experts receive 10–25% of their uniform share of tokens yet hold 1.05–1.12 times the uniform soft mass. Trinity Large keeps a loss of the same form at the same weight, and its first sparse layer shows the same pattern (LL of 0.973 with its bias and 0.972 without, negative correlation in both). In these recipes, pairing the bias with a small Switch-form loss therefore does not avoid the mismatch.

Additional references

  • Kimi Team (2026). Kimi K3: Open Frontier Intelligence. arXiv:2607.24653.
  • Lewis, M., et al. (2021). BASE Layers: Simplifying Training of Large, Sparse Models. arXiv:2103.16716.
  • Pan, B., et al. (2024). Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models. arXiv:2404.05567.
  • Park, J., et al. (2024). Monet: Mixture of Monosemantic Experts for Transformers. arXiv:2412.04139.
  • Park, S., et al. (2026). Solar Open Technical Report. arXiv:2601.07022.
  • Roller, S., et al. (2021). Hash Layers For Large Sparse Models. arXiv:2106.04426.
AuthorsOpenReview ↗

Response to Reviewer rfYD (3/3)

3. How the correlation depends on the loss weight and router regularization (Q3)

(Q3) In the controlled training experiments, how sensitive are the observed negative hard-soft correlations to the task loss weight and router regularization choices?

Thank you for this question, which we answered with new runs at α=0.003\alpha=0.003 and 0.1, and at α=0.03\alpha=0.03 without the router z-loss. The correlation depends on how strong the Switch term is relative to the task loss, and we vary this relative strength through the Switch coefficient α\alpha. As α\alpha grows from zero, both layers go through the same transition. Without the Switch term, dispatch collapses, the hard–soft correlation is close to +1 and LswL_{\rm sw} is far above one; once the term is strong enough to shape the router, at α=0.01\alpha=0.01 in layer 3 and 0.003 in layer 1, LswL_{\rm sw} falls below one and the correlation turns negative in every seed. The negative correlation persists in an ablation without the router z-loss at α=0.03\alpha=0.03, where rr is negative for every seed in both layers.

  • Task-loss weight. Under AdamW, only the relative weights of the loss terms matter: multiplying the task loss by cc is essentially equivalent to dividing the Switch and z-loss coefficients by cc. A larger task-loss weight therefore corresponds to a smaller α\alpha, so the sweep over α\alpha below also covers the task-loss weight; the z-loss is treated separately below. Each entry gives the range over three seeds at 4,096 updates in the paper's setup; α=0\alpha=0, 0.01 and 0.03 are the paper's runs.
α\alpha00.0030.010.030.10.03, no z-loss
Layer 3 rr+1.00+0.86 to +0.88−0.22 to −0.04−0.46 to −0.33−0.28 to −0.20−0.51 to −0.43
Layer 3 LswL_{\rm sw}120–1802.19–2.330.90–0.980.84–0.890.94–0.960.84–0.87
Layer 3 hard-load TV0.990.78–0.800.46–0.500.24–0.260.16–0.170.25
Layer 3 zero-dispatch experts235–2472–40–1000
Layer 1 rr+0.98 to +1.00−0.21 to −0.16−0.30 to −0.03−0.25 to +0.07−0.06 to +0.10−0.36 to −0.17
Layer 1 LswL_{\rm sw}42–550.92–0.930.88–0.990.94–1.010.99–1.010.89–0.97
Layer 1 hard-load TV0.99–1.000.51–0.590.30–0.360.21–0.230.19–0.200.22–0.25
Layer 1 zero-dispatch experts251–2528–182–5000
Held-out NLL5.424–5.4335.410–5.4225.418–5.4325.418–5.4435.427–5.4315.416–5.421
  • The transition. In both layers, raising α\alpha from zero takes LswL_{\rm sw} from far above one to below one and rr from close to +1 to negative in every seed, at α=0.01\alpha=0.01 in layer 3 and at α=0.003\alpha=0.003 in layer 1. Within a seed, the runs differ only in the loss coefficients, so this transition is produced by the Switch term itself, as our analysis predicts; held-out NLL stays between 5.410 and 5.443 throughout.
  • After the transition. In layer 3, rr stays negative in every seed up to α=0.1\alpha=0.1. In layer 1, larger α\alpha mainly balances the hard loads, and rr moves to around zero.
  • Why the layers differ. The two routers differ in confidence: at α=0.01\alpha=0.01, the selected expert gets on average 13.7 times the uniform weight in layer 3 but only 2.9 times in layer 1.
    • In layer 3, a weak Switch term cannot overcome this confident router, so at α=0.003\alpha=0.003 dispatch stays concentrated and the soft mass follows it. A stronger term lowers the selected expert's weight, to 7.9 times at α=0.03\alpha=0.03, and the released mass goes mainly to low-load experts: the soft mass of the lowest-load quarter rises from 0.271 to 0.314 and that of the highest-load quarter falls from 0.206 to 0.171, the redistribution that lowers the loss at fixed hard loads (Theorem 3.1). The correlation therefore turns negative and falls further at α=0.03\alpha=0.03.
    • In layer 1, the router is relatively close to uniform, so even a weak Switch term, at α=0.003\alpha=0.003, makes rr negative. For the same reason, a stronger term has little soft mass to shift onto low-load experts: it mainly changes which expert wins each token, and the lowest-load quarter gains only a little soft mass (0.263 to 0.277). As the hard loads become more balanced, rr moves to around zero; across seeds, the less confident the router, the closer rr is to zero.
  • Router regularization. The last column reruns α=0.03\alpha=0.03 without the router z-loss. Without it, rr is negative for every seed in both layers, and in layer 1 each seed's rr is lower than with the z-loss.

We hope these results address your questions. If so, we would be grateful if you would consider raising your score; if any concern remains, please let us know, and we will gladly address it before the discussion closes.

Reviewer_Br7XOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This is a pretty interesting paper that challenges a common belieft on MoE's load balancing loss: Minimizing switch does not guarantee uniform expert allocation; a lower loss can be achieved by having some experts completely unused while putting the mass of soft routing probability to the unused ones. The paper then verified on open sourced MoEs as well as a small MoE pre-training experiments that a switch loss < 1 and unused experts indeed occur. Overall I find the claim pretty interesting and the evidence sufficient. However, the paper does not seem to provide a solution, so it makes the empirical value of this work less clear.

Critical strengths
  • The paper studies an important problem: Load balancing of MoE

  • The claim is interesting and novel, as far as I am concerned.

  • The claim is well supported by theory and empirical verification.

  • The paper is well written and easy to follow: Figure 1 is illustrative!

Critical weaknesses
  • The paper does not provide a solution to the problem.

  • More recent MoEs use auxiliary loss free load banalcing mechanism, therefore the issue noted here may not very relevant.

Questions for authors
  • Why not test more models? There are lots of newer MoEs compared with deepseek v3 or qwen 2.5.

  • How does the authors consider the usefulness of the observation when more models start to use auxiliary free load banalcing approaches.

5 comments
AuthorsOpenReview ↗

Response to Reviewer Br7X (1/3)

Thank you for the positive assessment of our claim and its theoretical and empirical support, and for the questions on its practical relevance. We answer each point below.

Your questionShort answer
Solution (W1)This paper does not yet propose a new loss. Even so, the analysis identifies what a replacement loss should avoid and how to use the existing loss safely: judge balance from hard loads, not from the loss value. Designing the replacement is future work
Auxiliary-loss-free balancing (W2, Q2)A correction bias used alone avoids the mechanism, but many auxiliary-loss-free recipes also keep a small Switch-form loss alongside the bias, and several recent models use such a loss as the main balancer. Even at a weight of 10−410^{-4}, the routers of DeepSeek-V3 and Trinity Large show the geometry this loss rewards in our analysis, with the Switch statistic below one (Part 2)
More and newer models (Q1)In Qwen3.5, LLM-jp-4, Trinity Large, Trinity Nano, Marco and Switch-Large-128, every selected layer is below one on the evaluation data of Sec. 6; the revised paper adds these checkpoints to Table 1 (Part 3)

1. Toward a solution (W1)

... the paper does not seem to provide a solution, so it makes the empirical value of this work less clear.
(W1) The paper does not provide a solution to the problem.

We agree that a new loss is a natural next step; this paper does not yet propose one. Even so, we believe its exact account of what the Switch loss rewards offers guidance for designing such a loss and practical value for training with the existing one, as the three points below show.

  • A replacement must avoid two pitfalls. At fixed hard loads, the Switch loss is lowered by moving soft mass onto the least-used experts (Theorem 3.1), and its infimum is approached when about half of the experts receive no tokens (Sec. 4). A replacement should therefore not reward soft mass on experts that receive no tokens, and balancing the mean soft mass alone is not enough, since nearly tied gates can still send every token to the same expert (Sec. 7).
  • With the existing loss, balance should be judged from the hard loads. Measures such as MaxVio (Wang et al., 2024) or the total-variation distance of the hard loads from uniform, as in Table 5 (Table 8 in the revision), can be used; a Switch value below one should prompt this check instead of being read as balance. This matters in practice: a value near one is monitored as evidence of balanced routing (Qiu et al., 2025), and decreases in the loss are reported as improved balance (Yang et al., 2025; 2026). Hu et al. (2025) call one the ideal minimum of this loss; training Sigma-MoE-Tiny with it, they find the first-layer loss approaching that value while the hard loads show the least-loaded expert receiving almost no tokens.
  • Reading the loss as balance has a real cost. We timed each layer's routed experts (dispatch, expert computation and combination) on four A100-40GB GPUs with expert parallelism, once with the model's own hard loads and once with exactly uniform loads on the same tokens. In the Qwen2-57B-A14B layers we report (layers 11–16), the routed experts reach only 74–93% of their throughput under uniform loads. All six layers have a Switch value below one, so reading that value as balance would miss this cost.

The revised paper adds these recommendations to the conclusion and the utilization and throughput measurements to Appendix H.5. We hope this work helps the broader community recognize the problem, and that we and others will find a good solution in the near future.

AuthorsOpenReview ↗

Response to Reviewer Br7X (2/3)

2. Where the issue remains under auxiliary-loss-free balancing (W2, Q2)

(W2) More recent MoEs use auxiliary loss free load ... mechanism, therefore the issue noted here may not very relevant.
(Q2) How does the authors consider the usefulness of the observation when more models start to use auxiliary free load ... approaches.

We agree that auxiliary-loss-free balancing is increasingly common, and that a correction bias used alone avoids this mechanism. The observation still applies wherever a Switch-form loss is used, which remains widespread: it is the main balancer in several recent models, and many auxiliary-loss-free recipes keep a small one alongside the bias. DeepSeek-V3 and Trinity Large, which keep such a loss at a weight of 10−410^{-4}, show the geometry this loss rewards in our analysis.

  • Some mechanisms do avoid it. Balancing with no soft term has, by construction, nothing that rewards the mismatch: a correction bias used alone, as in Loss-Free Balancing (Wang et al., 2024) and the quantile balancing of Kimi K3 (Kimi Team, 2026); assignment-based routing such as Expert Choice (Zhou et al., 2022) and BASE (Lewis et al., 2021); and fixed hash routing (Roller et al., 2021). Losses that balance only the soft probabilities, such as the importance loss (Shazeer et al., 2017), the mutual-information loss of ModuleFormer (Shen et al., 2023) and φ-balancing (Chen et al., 2026), have no cross term either, but balanced soft mass does not ensure balanced dispatch, since nearly tied gates can still send every token to one expert (Sec. 7). The bias alone does not always suffice either. In the prototype for Solar Open (Park et al., 2026), for example, it left early layers severely imbalanced until a load-balancing loss was added.
  • Several recent models still use the Switch form as their main balancer. Qwen3 adopts the global-batch load-balancing loss of Qiu et al. (2025), N∑ifˉiPˉiN\sum_i \bar f_i\bar P_i over a global batch; OLMoE uses N∑ifiPiN\sum_i f_iP_i, and Marco-MoE follows OLMoE. The same product, computed over different sets of tokens, is the balancing loss in many other released MoE language models:
MechanismExamplesRelation to the mismatch
Switch-form loss over a micro-batch, sequence, global batch or other set of tokensSwitch, GShard, OLMoE, Marco-MoE, MiniCPM-MoE, MiniMax-Text-01, Pangu Pro and Ultra MoE, Qwen3 and Sigma-MoE-Tiny (global batch), DeepSeek-V2, V3 and Trinity Large (sequence-wise); Megatron-Core and Hugging Face transformersSame mechanism: on any set of tokens, moving soft mass toward less-used experts at fixed hard loads lowers the loss
Correction bias with a balance lossDeepSeek-V3/V4, GLM-4.5, MiMo-V2-Flash, Nemotron 3, Trinity Large, LongCat-Flash, Step 3.5 Flash, Solar Open, K-EXAONE, A.X K1, dots.llm1, MiniMax-M2These recipes also keep a small Switch-form loss
  • Many auxiliary-loss-free recipes also keep a small Switch-form loss. DeepSeek-V3 adds a complementary sequence-wise loss α∑ifiPi\alpha\sum_i f_iP_i with α=10−4\alpha=10^{-4} to avoid extreme imbalance within a single sequence, which its batch-level bias does not target. Trinity Large uses the same form to promote balance within a sequence; DeepSeek-V4, GLM-4.5 and MiMo-V2-Flash also keep a sequence-level balance loss, and Nemotron 3 a standard load-balancing loss, alongside the bias (weights 10−510^{-5} to 10−410^{-4}). On a single sequence as on a batch, this loss falls when soft mass moves toward less-used experts, so whatever the weight, a low value does not show that the sequence is routed evenly.
  • Even at a weight of 10−410^{-4}, this loss shapes the router. DeepSeek-V3 renormalizes its gates over the selected experts, so the scores of unselected experts do not enter the output, and its sequence-wise Switch-form loss is the only term of the objective with a gradient on them. With or without its bias, its router shows the geometry this loss rewards in our analysis: in the first sparse layer, L8L_8 is 0.977–0.985 for V3 and V3.1, below one, with negative hard–soft correlation, and the ten least-used experts receive 10–25% of their uniform share of tokens yet hold 1.05–1.12 times the uniform soft mass. Trinity Large keeps a loss of the same form at the same weight, and its first sparse layer shows the same pattern (LL of 0.973 with its bias and 0.972 without, negative correlation in both).

Additional references

  • Kimi Team (2026). Kimi K3: Open Frontier Intelligence. arXiv:2607.24653.
  • Lewis, M., et al. (2021). BASE Layers: Simplifying Training of Large, Sparse Models. arXiv:2103.16716.
  • Park, S., et al. (2026). Solar Open Technical Report. arXiv:2601.07022.
  • Roller, S., et al. (2021). Hash Layers For Large Sparse Models. arXiv:2106.04426.
AuthorsOpenReview ↗

Response to Reviewer Br7X (3/3)

3. Newer models show the same pattern (Q1)

(Q1) Why not test more models? There are lots of newer MoEs compared with deepseek v3 or qwen 2.5.

Thank you for this suggestion. We have now tested several newer checkpoints and a larger Switch checkpoint on the evaluation data of Sec. 6, with layers selected on a preliminary set. Every selected layer has a Switch statistic below one, with its 95% upper endpoint also below one, and a negative hard–soft correlation. The revised paper adds these checkpoints to Table 1 and Sec. 6.2, with details in Appendix H.6.

CheckpointSelected layer(s)LLrrNote
Qwen3.5-35B-A3B00.928−0.35
LLM-jp-4-32B-A3B0, 2, 20, 22, 30, 310.941–0.994−0.74 to −0.22
Trinity-Large60.973−0.30correction bias
Trinity-Nano, pre-anneal / final20.991 / 0.982−0.11 / −0.23correction bias; experts 1, 19 and 55 below 0.1/N0.1/N in both
Marco-Nano / Mini / Mini-Global0 / 17, 20 / 200.984–0.988−0.17 to −0.06in Marco-Nano, expert 203 receives none of the 8.4M assignments yet holds 2.14× the uniform soft mass
Switch-Large-128encoder 11, 13, 15, 21, 23; decoder 15, 17, 19, 21, 230.913–0.983−0.24 to −0.05in decoder layer 23, expert 3 receives no assignments yet holds 2.12× the uniform soft mass
  • The same experts stay underused. In Marco and Switch-Large-128, every expert below 0.1/N0.1/N on the preliminary data is again below 0.1/N0.1/N on the evaluation data, and in Trinity Nano, experts 1, 19 and 55 stay below 0.1/N0.1/N across 1.2T training tokens.

We hope these results address your concerns about the practical relevance of the analysis. If so, we would be grateful if you would consider raising your score; if any concern remains, please let us know, and we will gladly address it before the discussion closes.

Reviewer_6ckUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the relationship between hard expert utilization and mean routing probabilities in the Switch auxiliary loss. It derives a fixed-load characterization, the sharp global infimum for top-1 routing, and the associated extremal hard-load distributions, together with stability and finite-batch results. It also analyzes local logit gradients and constructs output-preserving loss reductions for top-k routing with post-selection renormalization. Measurements on released MoE checkpoints, controlled language-model training, and a frozen-model intervention illustrate discrepancies between the auxiliary objective and hard-load balance.

Critical strengths
  • It goes beyond demonstrating losses below one to characterize the sharp infimum and all extremal hard loads. The fixed-load reduction and stability analysis explain precisely how soft probability mass can favor unused experts, providing a rigorous correction to a common interpretation of the Switch objective.
  • The controlled training comparisons use matched initialization and data order, and report both MoE layers across three seeds. It shows that, in layer 3, stronger regularization can improve hard-load balance while making hard–soft correlation more negative. Besides, it demonstrates why loss values and correlations should be interpreted alongside direct utilization measurements.
Critical weaknesses
  • The DeepSeek evaluation transfers the FP32 correction bias to BF16 and then back to FP32, causing irreversible precision loss that can change expert selection. The reported routing statistics and bias-ablation comparisons therefore need verification with the original FP32 bias, together with confirmation of which implementation produced the submitted results.
  • The results on public MoE models are restricted to selected layers, leaving the prevalence and practical impact of the reported mismatch unclear. A broader layer-wise analysis, together with measurements under the actual expert placement and load-balancing setup, would clarify whether these examples correspond to substantial utilization or throughput costs.
Questions for authors
  • The provided evaluation code converts the routing correction bias to BF16 before converting it back to FP32. If the DeepSeek-V3 and V3.1 results are obtained using this code, please verify whether preserving the original FP32 bias changes the measurements in Table 4.
  • How are the reported Switch and Qwen layers selected, and how many candidate layers have been examined? Could you provide results for all sparse layers, including (LkL_k), hard-load TV, and zero-dispatch counts, to show how representative the reported examples are?
  • Appendix H reports (L8<1L_8<1) in each of 16 document groups for every reported Qwen/DeepSeek layer. Could you provide these group-level results, including updated DeepSeek measurements with the original FP32 bias preserved?
  • Appendix I.1 distinguishes the globally aggregated evaluation loss from the Switch loss computed on each training batch. Does the latter also fall below one? Please provide the corresponding training curves.
  • Table 5 shows improved hard-load balance and no zero-dispatch experts at the stronger regularization setting. Given that Eq. (15) gives unused experts a negative logit derivative within a fixed assignment region, could you analyze the assignment trajectories of the zero-dispatch experts at alpha = 0.01 to distinguish persistent underutilization from temporary zero dispatch and clarify the empirical relevance of the near-infimum geometry?
  • In Table 5, increasing the regularization coefficient makes hard–soft correlation more negative in layer 3 across all three seeds, but increases the correlation in layer 1 for two seeds, including one positive endpoint. What explains this layer dependence, and do these patterns persist later in training?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 6ckU (1/5)

We thank the reviewer for the careful reading of both the paper and the code, and for the positive assessment of the theoretical analysis. We answer every question below.

Your questionShort answer
FP32 correction bias (W1, Q1)Re-run with the FP32 bias: L8=0.977L_8=0.977 (V3) and 0.9810.981 (V3.1), r<0r<0 in both; conclusions unchanged
Document groups (Q3)L8<1L_8<1 in all 16 groups for every reported Qwen and DeepSeek layer
Selection, all layers (W2, Q2)Screened all 40 sparse layers; reported those with LkL_k significantly below one and an expert below 10% of its uniform share; 11 of 12 Switch layers have L1<1L_1<1; in Qwen, the layers with L8<1L_8<1 are mostly the least balanced (Part 2)
Utilization and throughput (W2)Expert utilization is 27–39% in the reported Qwen layers and 17–18% in DeepSeek's first sparse layer under its correction bias; the routed experts of these Qwen layers reach 74–93% of their balanced throughput (Part 3)
Training-batch loss (Q4)Yes, it falls below one in both layers of every regularized run (Part 4)
Zero-dispatch trajectories (Q5)At α=0.01, zero dispatch of individual experts is temporary, but layer 1 has unused experts at every reported checkpoint; in pretrained models the same experts stay unused or underused (Part 4)
Layer dependence, later training (Q6)The layers differ in router confidence: layer 3's router puts far more mass on the selected expert, so raising α from 0.01 to 0.03 shifts soft mass toward low-load experts mainly there. Through 8,192 steps, the layer-3 pattern persists in every seed, while in layer 1 the coefficient still has no consistent effect (Part 5)

1. DeepSeek with the original FP32 bias: conclusions unchanged (W1, Q1)

(W1) The DeepSeek evaluation transfers the FP32 correction bias to BF16 and then back to FP32 ... need verification with the original FP32 bias, together with confirmation of which implementation produced the submitted results.
(Q1) ... please verify whether preserving the original FP32 bias changes the measurements in Table 4.

Thank you for catching this. The submitted DeepSeek values were produced by the released evaluator, which rounds the bias through BF16. Re-evaluated on the same 4,096 documents with the bias kept in FP32, both checkpoints still have L8<1L_8<1 and a negative hard–soft correlation:

Layer 3L8L_8 [95% CI]rrZero-bias L8L_8 [95% CI]Zero-bias rr
DeepSeek-V30.97665 [0.97638, 0.97691]−0.1580.98127 [0.98102, 0.98155]−0.128
DeepSeek-V3.10.98082 [0.98052, 0.98113]−0.1310.98536 [0.98507, 0.98565]−0.101
  • The bias still lowers L8L_8 relative to the zero-bias control in both checkpoints, so the conclusion of Sec. 6.2 is unchanged: bias-assisted selection contributes to the separation between soft mass and expert use but does not fully explain it. Table 4's zero-bias column does not use the bias, so it is unaffected by the precision issue; its correlations, added here, are also negative.
  • In the revised paper, we have updated Table 1, Table 4, Figure 2(C) and the numbers in Sec. 6.2; we will also release the corrected evaluator.

2. Every document group stays below one (Q3)

(Q3) Appendix H reports L8<1L_8<1 in each of 16 document groups for every reported Qwen/DeepSeek layer. Could you provide these group-level results, including updated DeepSeek measurements with the original FP32 bias preserved?

All 16 groups are below one in every reported layer, including both DeepSeek checkpoints with the FP32 bias. Range of L8L_8 over the 16 disjoint groups of 256 documents:

Model, layerGroup L8L_8 (min–max)
Qwen2-57B-A14B, 110.99239–0.99331
Qwen2-57B-A14B, 120.99429–0.99502
Qwen2-57B-A14B, 130.99529–0.99664
Qwen2-57B-A14B, 140.99676–0.99811
Qwen2-57B-A14B, 150.97761–0.98029
Qwen2-57B-A14B, 160.96520–0.96670
DeepSeek-V3, 3 (FP32 bias)0.97578–0.97735
DeepSeek-V3.1, 3 (FP32 bias)0.97982–0.98156

In Qwen layer 11, expert 49 receives no assignment in any group, while its soft mass stays at 1.07× the uniform value in each. Appendix H.4 of the revised paper lists all individual group values.

AuthorsOpenReview ↗

Response to Reviewer 6ckU (2/5)

3. Layer selection and results for all 40 sparse layers (W2, Q2)

(W2) The results on public MoE models are restricted to selected layers, leaving the prevalence and practical impact of the reported mismatch unclear.
(Q2) How are the reported Switch and Qwen layers selected, and how many candidate layers have been examined? Could you provide results for all sparse layers ...?

We screened all 40 sparse layers (12 in Switch-Base-256, 28 in Qwen2-57B-A14B) and reported those whose LkL_k is significantly below one and that contain a nearly unused expert. Results for every layer follow: 11 of 12 Switch layers have L1<1L_1<1, and the Qwen layers with L8<1L_8<1 are mostly the least balanced ones.

Selection. On the preliminary sets of Appendix H, a layer was retained if its LkL_k had a 95% bootstrap upper endpoint below one and some expert received less than 10% of its uniform share (fi<0.1/Nf_i<0.1/N). This selected encoder 11 and decoder 1 in Switch and layers 11–16 in Qwen; we fixed these layers before re-measuring them on the larger evaluation sets.

All layers on the evaluation sets (bold: layers reported in the paper; TV as in Table 5, Table 8 in the revision).

Switch-Base-256:

Encoder layerL1L_1TVDecoder layerL1L_1TV
10.95210.05110.97350.110
30.87470.07031.00610.051
50.87830.09150.95630.046
70.89530.09170.94060.049
90.91410.07590.92840.047
110.86940.101110.90400.068

Qwen2-57B-A14B:

LayerL8L_8TVLayerL8L_8TV
00.99780.095140.99760.345
11.00630.370150.97900.230
21.01160.170160.96600.298
30.99590.193170.99180.124
41.01860.185181.02480.109
51.01160.184191.03720.108
61.02430.165201.01710.085
71.02120.170211.01590.083
81.03010.200221.02110.097
91.01480.138231.01940.089
101.01240.157241.01330.074
110.99300.336251.02120.080
120.99470.298261.03290.084
130.99600.300271.03620.110

Zero-dispatch experts occur only in Switch encoder 11 (expert 37) and Qwen layer 11 (expert 49); every other layer has none.

  • Prevalence. L1<1L_1<1 holds in 11 of 12 Switch layers, each with a 95% upper endpoint below one, and L8<1L_8<1 in 9 of 28 Qwen layers (the six reported ones and layers 0, 3 and 17).
  • A loss below one mostly marks the least balanced layers. In Qwen, the 9 layers with L8<1L_8<1 have mean TV 0.247, 1.76× that of the other 19 layers (0.140), and 7 of the 9 most imbalanced layers have L8<1L_8<1. Across layers, LkL_k and TV are negatively correlated: Pearson −0.60 (Qwen) and −0.43 (Switch).
  • This confirms the reviewer's point that loss values must be read together with direct utilization measurements: across layers, a lower Switch loss tends to accompany less balanced dispatch.
AuthorsOpenReview ↗

Response to Reviewer 6ckU (3/5)

4. Utilization and throughput cost (W2)

(W2) ... A broader layer-wise analysis, together with measurements under the actual expert placement and load-balancing setup, would clarify whether these examples correspond to substantial utilization or throughput costs.

These examples carry a substantial utilization cost. Expert utilization, the average expert load divided by the load of the busiest expert, is only 27–39% in the reported Qwen layers and 17–18% in the first sparse layer of DeepSeek under its own correction bias, against 100% for perfectly balanced routing. The routed experts of these Qwen layers reach only 74–93% of their throughput under balanced loads.

Why the hard loads set the cost. An overloaded expert becomes a compute bottleneck, while an unused expert stops learning (Sec. 1). Under expert parallelism, a layer finishes only when its most loaded device has processed its tokens, so the cost depends on the hard loads alone. A low Switch loss does not bound these loads: the loss also depends on the soft masses, and at fixed hard loads it can be lowered by moving soft mass alone (Theorem 3.1).

Wang et al. (2024) measure load balance by MaxVio, the busiest expert's load in excess of the average, relative to the average. We define expert utilization as the average expert load divided by the busiest expert's load, which equals 1/(1+MaxVio)1/(1+\mathrm{MaxVio}). We compute it on the whole evaluation set, the paper's 4,096 C4 validation documents, and report each layer (Qwen: the layers of the all-layer table in item 3; DeepSeek: layer 3 with the FP32 bias, item 1):

Qwen 11Qwen 12Qwen 13Qwen 14Qwen 15Qwen 16DeepSeek-V3DeepSeek-V3.1
Expert utilization29%27%34%27%33%39%17%18%

Throughput. We timed the routed experts of each Qwen layer (dispatch, expert computation and combination) on the same documents in batches of 32, on four A100-40GB GPUs with expert parallelism, once with the model's own hard loads and once with exactly uniform loads on the same tokens. Relative to uniform loads, the routed experts reach 83%, 79%, 82%, 74%, 93% and 93% of their throughput in layers 11–16.

Load-balancing setup.

  • Qwen2-57B-A14B has no balancing step at inference: each token goes to its top-8 experts, with no capacity limit or correction bias. The loads above are therefore those its router actually produces.
  • DeepSeek-V3/V3.1 balance loads mainly through a per-expert correction bias, which is added to the routing scores when the top-8 experts are selected and remains in effect at inference. The DeepSeek columns include this bias, in FP32 as released: under it, layer 3 has L8L_8 of 0.977 (V3) and 0.981 (V3.1), while expert utilization is only 17–18%. Layer 3's ten least-used experts are the same in both checkpoints (item 6).

A Switch loss below one therefore does not certify efficient dispatch; in Qwen, it mostly picks out the layers whose dispatch is least balanced (item 3).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of Rebuttal and Manuscript Revisions

We thank the reviewers for their careful reviews and positive assessments. The table below summarizes every question and our answer; each reviewer's thread has the details.

QuestionAsked byShort answer
How common are sub-one values, and what do they cost?6ckU (W2, Q2), rfYD (W1)Sub-one values occur in 11 of the 12 sparse layers of Switch-Base-256 and in 9 of the 28 layers of Qwen2-57B-A14B. In the reported Qwen layers, expert utilization is 27–39%, and the routed experts reach 74–93% of their throughput under uniform loads
FP32 correction bias6ckU (W1, Q1)Re-run with the FP32 bias: L8=0.977L_8=0.977 (V3) and 0.9810.981 (V3.1), r<0r<0 in both; conclusions unchanged
Newer modelsBr7X (Q1)In Qwen3.5, LLM-jp-4, Trinity Large, Trinity Nano, Marco and Switch-Large-128, every selected layer is below one on the evaluation data of Sec. 6
Auxiliary-loss-free balancingBr7X (W2, Q2)Many auxiliary-loss-free recipes still keep a small Switch-form loss alongside the correction bias, and several recent models use such a loss as the main balancer. Even at a weight of 10−410^{-4}, the routers of DeepSeek-V3 and Trinity Large show the geometry this loss rewards in our analysis, with the Switch statistic below one
Which balancing losses avoid the mismatch mechanismrfYD (W2, Q2)Mechanisms without the hard–soft product, such as a correction bias used alone, assignment-based routing and soft-only objectives, do not reward the mismatch; however, the Switch form, which has the product, remains widely used, including alongside correction biases
A solutionBr7X (W1)This paper does not yet propose a new loss. Even so, the analysis identifies what a replacement loss should avoid and how to use the existing loss safely: judge balance from hard loads, not from the loss value. The revised conclusion states both
How close training gets to the infimumrfYD (W1, Q1)Training stays far from the infimum, which the prediction loss resists, but released models show the signature of the extremal geometry; our practical conclusion does not require reaching the infimum
Document groups6ckU (Q3)L8<1L_8<1 in all 16 groups for every reported Qwen and DeepSeek layer
Training-batch loss6ckU (Q4)The per-batch Switch loss falls below one in both layers of every regularized run
Zero-dispatch trajectories6ckU (Q5)At α=0.01, zero dispatch of individual experts is temporary, but layer 1 has unused experts at every reported checkpoint; in pretrained models the same experts stay unused or underused
Layer dependence, later training6ckU (Q6)The layers differ in router confidence: layer 3's router puts far more mass on the selected expert, so raising α from 0.01 to 0.03 shifts soft mass toward low-load experts mainly there. Through 8,192 steps, the layer-3 pattern persists in every seed, while in layer 1 the coefficient still has no consistent effect
Sensitivity to the loss weight and router regularizationrfYD (Q3)Without the Switch term, rr is close to +1; it turns negative when the term becomes strong enough, and is negative for every seed in both layers in an ablation without the router z-loss

We have uploaded a revised version of the paper, with all changes marked in blue. Tables 5–7 of the submission are now Tables 8–10. The main changes are:

  • FP32 correction bias: updated values in Sec. 6.2, Figure 2(C), Table 1 and Table 4, which also adds the zero-bias correlation
  • Newer models: results in Sec. 6.2 and Table 1, with details in Appendix H.6 (Table 7)
  • Document groups: L8L_8 for each of the 16 groups in Appendix H.4 (Table 5)
  • Utilization and throughput: measurements in Appendix H.5 (Table 6)
  • Training-batch loss: per-update curves in Appendix I.4 (Figure 5)
  • Recommendations: how to judge balance and what a replacement loss should avoid (Sec. 8)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.