acceptodds
Under review as a conference paper at ICLR 2027

How Sparse Probability Maps Shape Mixture-of-Experts Routing

Abstract

Mixture-of-experts (MoE) routers typically apply softmax to the router scores and keep the top- experts, making every token use exactly experts. Sparsity-inducing probability maps such as sparsemax, -entmax and normmax can adaptively assign exact zeros to selected experts, and therefore appear to offer token-dependent expert participation, even when using the same top- machinery. In this work, we study whether and how this sparsity survives training. We train matched 300M and 1B top-2 MoE language models with softmax, -entmax, sparsemax and -normmax, and find that the maps behave very differently once trained: at 1B, entmax discards 30% less probability mass than softmax while almost never dropping a selected expert, sparsemax retains the most mass, and normmax routes 21% of tokens to a single expert. These outcomes are not properties of the maps alone. Each map drops a selected expert only when the gap between the two largest scores exceeds a fixed threshold, and what the trained routers differ in is the score distribution they learn: the entmax router learns scores with roughly half the spread of softmax's, which keeps its top-2 gaps below its threshold, while sparsemax and normmax, which share the same threshold, learn different gap distributions and hence different participation. Routers thus co-adapt their scores to the map, and a map's capacity to produce zeros does not by itself determine expert participation. Overall, while none of the sparse maps improves validation loss over softmax, they make trained models more robust to inference-time changes. For example, sparsemax trained with loses nats when run with , where softmax loses . Our results indicate that adaptive MoE routing has to be designed around the joint behavior of the probability map and the learned scores, rather than around the map alone.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mixture-of-experts, sparse routing, probability maps, language models, entmax, sparsemax, normmax

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_nYE7OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies whether the sparsity of sparse probability maps (1.5-entmax, sparsemax, and 2-normmax) survives training when used as routers in top-2 Mixture-of-Experts (MoE) language models. They show that the probability map alone does not determine how many experts a token uses because the router co-adapts the scores it feeds into the map. During training, the 1.5-entmax router learns much smaller score spreads than softmax, keeping its score gaps below its threshold so it almost always uses both experts while discarding less probability mass. Meanwhile, sparsemax and 2-normmax share the exact same threshold, yet 2-normmax routes three times as many tokens to a single expert because its router learns larger score gaps. While none of the sparse maps improve validation loss over softmax at the trained top-2 budget, they make the models much less sensitive to increasing the number of selected experts at inference time because they assign zero weight to most of the extra expert slots.

Critical strengths
  1. Clear separation of the probability map from the learned router scores. A common assumption in adaptive MoE routing is that swapping softmax for a sparse probability map automatically yields dynamic expert usage. By connecting each map's exact sparsification threshold to the score distributions actually learned during training, the paper proves that routers co-adapt their score scales to the map.

  2. Rigorous distinction between probability retention, expert participation, load balance, and model quality. They show that retaining more probability mass and using fewer experts are separate phenomena. Their inference-time budget tests and load-balancing ablations are well-controlled and directly test practical hypotheses about how sparse maps interact with top-k truncation and auxiliary balancing losses.

Critical weaknesses
  1. While the paper clearly shows that sparse routers learn smaller score spreads than softmax and that 2-normmax learns larger score gaps than sparsemax, it does not explain the underlying optimization or gradient dynamics during training that cause the routers to settle into these specific score distributions.

  2. The paper does not experimentally test its own design recommendations. They conclude that adaptive routing requires explicitly controlling or learning the router's score scale and moving beyond a fixed top-2 budget, yet all experiments use unconstrained router scores at top-2.

Questions for authors

Have you tested what happens if you explicitly control or normalize the router's score spread before applying the probability map? If you prevent the score spread from shrinking under 1.5-entmax, does it continue routing tokens to single experts throughout training, and how does that affect validation loss and load balance?

3 comments
AuthorsOpenReview ↗

Part 1 of 2 - Training dynamics

While the paper clearly shows that sparse routers learn smaller score spreads than softmax and that 2-normmax learns larger score gaps than sparsemax, it does not explain the underlying optimization or gradient dynamics during training that cause the routers to settle into these specific score distributions.

This is true. To investigate this, we analyzed gradients at 15 checkpoints from initialization to update 19,073 in four 300M models trained with the same seed.

The table reports the balancing gradient norm as a percentage of the task gradient norm on router weights, including the balancing coefficient of 0.01. Thus, 100% means equal size. Measurements use fixed inputs and average norms over batches and MoE layers.

Probability map Initialization (0) Update 50 Update 1,000 Final (19,073)
Softmax36.5%186%1.82%1.68%
1.5-entmax73.2%161%2.01%2.09%
Sparsemax91.6%230%1.96%1.87%
Normmax-2107.8%215%1.78%1.75%

These checkpoints show a shared transition: balancing gradients exceed task gradients early, at update 50, but fall to approximately 2% of their magnitude later.

To examine how these losses act on score spread, we also measured scale derivatives, which describe how each loss changes when score differences are slightly widened. At updates 500 and 1,000, the combined loss locally favors narrowing under sparsemax but widening under the other three maps in the pooled measurements. The top-two gap derivatives show the same contrast. By update 2,000, all three sparse maps locally favor narrowing the score spread.

Sparsemax therefore exhibits pressure against score expansion earlier in the sampled trajectory, consistent with its subsequently smaller score spread and gaps. This difference in optimization pressures may help account for the learned distributions. We will add the full gradient and derivative trajectories alongside score spreads and gaps, with layerwise results and measurement details, to the appendix.

AuthorsOpenReview ↗

Part 2 of 2 - Score normalization, loss, and load balance

The paper does not experimentally test its own design recommendations. They conclude that adaptive routing requires explicitly controlling or learning the router's score scale and moving beyond a fixed top-2 budget, yet all experiments use unconstrained router scores at top-2.

We agree that it is worth testing how routing changes when the score scale is controlled during training. We chose top-2 routing, following influential work in the MoE literature such as GShard (Lepikhin et al., 2021) and Mixtral (Jiang et al., 2024), so we could compare each probability map directly with softmax under the same routing budget.

Even with the same top-2 budget, the spread of the scores depended on which probability map we trained with. This matters because the second expert contributes only while the gap between the two highest scores stays below the map’s threshold. The scale the router learns therefore helps determine how often both experts contribute. This is why we suggested making score scale an explicit part of the router design. We left open how best to do that.

We could control the scale in several ways, for example by letting each MoE layer learn its own scale parameter or by normalizing the scores to a fixed scale throughout training. Following your suggestion, we tested the latter by training three 1.5-entmax models with different target values for the root mean square (RMS) of each token’s centered router scores. We report the results in our answer to your next question.

Have you tested what happens if you explicitly control or normalize the router's score spread before applying the probability map?

Yes. We tested this with the 1.5-entmax probability map. We trained three 300M models using the same seed and top-2 routing. For each token, we centered its 26 expert scores and rescaled them to a fixed root mean square (RMS) before applying 1.5-entmax. We used a target RMS of 0.5, 1.0 or 1.5 from the start of training. Our baseline was a 300M 1.5-entmax model trained with the same seed and an unconstrained score scale.

The table reports full-validation results after 19,073 updates. Each token makes a routing decision at each of the four MoE layers. We count a decision as having one contributor when only one of the two selected experts receives positive weight.

1.5-entmax training condition Routing decisions with one contributor Validation loss (nats)
Unconstrained score scale0.1250%3.51220
Fixed RMS 0.50.0065%3.51619
Fixed RMS 1.08.85%3.51614
Fixed RMS 1.522.45%3.52480

Raising the target RMS from 0.5 to 1.0 made single-expert routing more common with little change in validation loss. At RMS 1.5, single-expert routing became more common still, but validation loss rose. Fixing the score scale therefore gave us some control over expert participation, with little change in validation loss in this comparison.

If you prevent the score spread from shrinking under 1.5-entmax, does it continue routing tokens to single experts throughout training, and how does that affect validation loss and load balance?

Yes. At RMS 1.0 and 1.5, we observed single-expert routing at every training step we measured. For a given set of router scores, raising the RMS widens the gap between the two highest scores, making it easier to reach the threshold of 2 at which 1.5-entmax gives all the weight to one expert. As the table in our previous answer shows, final validation loss stayed close to the unconstrained baseline at RMS 1.0 and rose slightly at RMS 1.5. Expert loads were less even than in the unconstrained model at both targets, especially at RMS 1.5.

Hello! We would like to thank you for your review and let you know that if you have any further questions, we would be happy to respond during the remaining duration of the discussion period.

Reviewer_9vfHOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper compares softmax, 1.5-entmax, sparsemax, and 2-normmax as probability maps in MoE language models. The map is applied before top-2 expert selection. A sparse map could probably assign 0 to one selected expert. The experiments involve the training of 0.3B and 1B MoE models on 10B tokens. The authors claim four contributions: a controlled comparison of mapping methods; the retained probability mass and the number of contributing experts are distinct outcomes; the activation pattern is decided by the map's threshold and router scores; sparse maps make the model trained with k=2 less sensitive than softmax when increasing k to 8 at inference time.

Critical strengths

Transparent experiments and useful empirical comparisons of sparse maps. Hyperparameters and seed are shared. Figure 2 and table 2 show sparse maps produce 0 probability not necessarily reduce expert activation after training, which is a useful empirical observation.

Critical weaknesses
  1. The analysis provides limited new insight beyond empirical comparison. The distinction between the retrained probability mass and the activated experts in Equation 7 are followed by definition instead of an independent contribution. The threshold are derived from the definition of the sparse maps. Section 4.3 show how often the router scores cross the thresholds but don't explain where the different score distributions come from. The analysis only describes the observed routing behavior without explaining the underlying mechanism. The claim that explicit score-scale control can improve adaptive router is also not validated. The claim that routers adapt their scores to the map is not supported by table 5 and table 7: 1.5-entmax and sparsemax learn similar gaps despite different threshold, while sparsemax and 2-normmax share the same threshold but learn different gaps.

  2. No practical benefit for MoE is shown, and the practical claims are not supported. Table 1 shows no validation loss improvement from sparse maps. Table 3 shows increasing k make the loss worse for every map. Table 9 show spasemax increase average expert count from 1.9 to 4.3 but loss become worse. For efficiency, fewer contributing experts do not necessarily mean less compute. The experiments don't show how the sparse maps improve moe effectiveness or efficiency.

Questions for authors
  1. Equation 7 and the thresholds in Proposition 1 follow from the definitions of the sparse maps. Which result do you consider new beyond these definitions?

  2. Table 5 and Table 7 show that 1.5-entmax and sparsemax learn similar gaps despite different thresholds. What evidence supports the claim that routers adapt their scores to the map?

  3. Why do you think the lower sensitivity to larger k is useful? In what situation do you consider it useful when your results show that a larger k makes the loss worse?

5 comments
AuthorsOpenReview ↗

Part 1 of 3 - Theory, gradients, and score-scale control

“The analysis provides limited new insight beyond empirical comparison. The distinction between the retrained probability mass and the activated experts in Equation 7 are followed by definition instead of an independent contribution. The threshold are derived from the definition of the sparse maps”

Our goal was not to propose a new map, but to study, both theoretically and empirically, the implications of applying existing differentiable sparse maps to MoEs. While the theoretical derivations arise naturally from the definitions of the sparse maps we use, they were not explored in MoEs before.

Our analysis also goes beyond the singleton thresholds. For example, Appendix B.3 shows that, on the same scores, sparsemax retains at least as much top-2 probability mass as normmax-2, even though both maps use the same number of contributing experts. Section 4.3 then shows that their trained routers learn different gap distributions and participation rates, despite sharing the same singleton threshold.

Section 4.3 show how often the router scores cross the thresholds but don't explain where the different score distributions come from.

The score distributions are learned during training, under the language-model and load-balancing objectives. Our goal in Section 4.3 is to understand how the scores learned under each map affect expert participation. We show how their gap distributions interact with the maps’ singleton thresholds to determine whether one or two experts contribute. Explaining why optimization converges to each particular distribution goes beyond that analysis. Nevertheless, we have now investigated the training dynamics further through gradient measurements. At updates 500 and 1,000 in our 300M models, the combined task and load-balancing loss locally favored narrower score differences under sparsemax and wider differences under the other three maps. This is consistent with the smaller score gaps sparsemax learned. We will report the full trajectories in the appendix.

The analysis only describes the observed routing behavior without explaining the underlying mechanism.

In our paper, we take care to explain the underlying routing mechanism. For example, Section 2.1, following Equation 5, explains how the probability map affects the load-balancing gradient: an expert assigned zero probability receives no balancing gradient from that token. Figure 3 also discusses gradient saturation. Beyond the singleton threshold, the probabilities become locally constant, so small score changes no longer affect the expert mixture and gradients through the map vanish. The observed peaks near this boundary are consistent with this effect, connecting the learned score geometry to a mechanism that affects router learning.

The claim that explicit score-scale control can improve adaptive router is also not validated.

There are several ways to control score scale. For example, we could let each MoE layer learn its own scale or normalize the scores to a fixed scale throughout training. Comparing these approaches is beyond the scope of this paper, but we tested the second approach in 300M 1.5-entmax models trained with the same seed.

Fixing the root mean square of each token’s centered scores at 1.0 increased the fraction of routing decisions with one contributing expert from 0.125% to 8.85% compared with unconstrained training. The fixed-scale model used about 4% fewer expert FLOPs on both DCLM and Paloma, with a small increase in full-validation loss from 3.51220 to 3.51614 nats per token.

AuthorsOpenReview ↗

Part 2 of 3 - Learned gaps and computational efficiency

The claim that routers adapt their scores to the map is not supported by table 5 and table 7: 1.5-entmax and sparsemax learn similar gaps despite different threshold, while sparsemax and 2-normmax share the same threshold but learn different gaps.

We disagree with this interpretation of the results. Our claim in Section 4.3 is that the geometry of the learned scores depends on the map. This does not require learned gaps to be proportional to singleton thresholds.

Tables 5 and 7 should be read together: Table 7 describes the learned gap distributions, while Table 5 reports how often they cross each threshold. Consider the 1B results. Entmax and sparsemax have similar median gaps, 0.44 and 0.40, but their singleton thresholds are 2 and 1. Consequently, similar absolute gaps can produce very different participation: their measured singleton rates are 0.05% and 6.88%.

For sparsemax and normmax-2, the threshold is the same, but the learned distributions differ: their median gaps are 0.40 and 0.61, and 6.88% versus 21.26% of their gaps reach the shared threshold. Here, the participation difference comes from the learned scores. Sharing a singleton threshold does not make the maps identical or require them to learn identical distributions. These two comparisons illustrate why participation must be understood through both the map’s threshold and the scores learned under it.

No practical benefit for MoE is shown, and the practical claims are not supported. Table 1 shows no validation loss improvement from sparse maps. Table 3 shows increasing k make the loss worse for every map. Table 9 show spasemax increase average expert count from 1.9 to 4.3 but loss become worse.

That was not the goal of our work. Our goal was to better understand how sparse maps affect routing in MoEs through a controlled theoretical and empirical comparison. We leave the development of a new routing method that improves the quality–efficiency tradeoff to future work.

For efficiency, fewer contributing experts do not necessarily mean less compute. The experiments don't show how the sparse maps improve moe effectiveness or efficiency.

First, we note that to save time we would need specialized kernels for leveraging sparsity and making our methods more efficient. For example, existing MoE systems such as MegaBlocks (Gale et al., 2023) use specialized block-sparse kernels for expert computation. In our case, we use the PyTorch implementations from the entmax package (Peters et al., 2019), which use sorting to compute 1.5-entmax and sparsemax, and root-finding for normmax-2, and are slower than softmax in the forward pass. AdaSplash (Gonçalves et al., 2025) and AdaSplash-2 (Gonçalves et al., 2026) demonstrate ongoing efforts to develop faster kernels for sparse transformations, which could reduce our probability map overhead. To compete in MoE efficiency, we would need to combine faster map computation with optimized MoE kernels that leverage sparsity. Nevertheless, we benchmarked our implementations (which use a dispatcher that skips zero-weight selections), measuring both the savings in expert computation and the full-pass latency.

The table below shows the 300M results on one RTX A6000, using BF16 inference, batch size 8 and sequence length 2,048.

Router Estimated expert matrix FLOPs saved Estimated whole-model matrix FLOPs saved Map time, separate profile (ms) Median full-pass latency (ms)
Softmax0%0%0.0358.85
1.5-entmax0.06%0.01%1.4060.72
Sparsemax5.05%1.11%12.1471.14
Normmax-211.92%2.61%11.7870.41

Fewer contributing experts reduce expert computation, while the overhead of our current sparse probability map implementations outweighs these savings as expected.

AuthorsOpenReview ↗

Part 3 of 3 - Answers to Q1–Q3

Q1: Equation 7 and the thresholds in Proposition 1 follow from the definitions of the sparse maps. Which result do you consider new beyond these definitions?

Our goal was not to propose a new map, but to study, both theoretically and empirically, the implications of applying existing differentiable sparse maps to MoEs. While the theoretical derivations arise naturally from the definitions of the sparse maps we use, they were not explored in MoEs before.

Our analysis also goes beyond the singleton thresholds. For example, Appendix B.3 shows that, on the same scores, sparsemax retains at least as much top-2 probability mass as normmax-2, even though both maps use the same number of contributing experts. Section 4.3 then shows that their trained routers learn different gap distributions and participation rates, despite sharing the same singleton threshold.

Q2: Table 5 and Table 7 show that 1.5-entmax and sparsemax learn similar gaps despite different thresholds. What evidence supports the claim that routers adapt their scores to the map?

We disagree with this interpretation of the results. Our claim in Section 4.3 is that the geometry of the learned scores depends on the map. This does not require learned gaps to be proportional to singleton thresholds.

Tables 5 and 7 should be read together: Table 7 describes the learned gap distributions, while Table 5 reports how often they cross each threshold. Consider the 1B results. Entmax and sparsemax have similar median gaps, 0.44 and 0.40, but their singleton thresholds are 2 and 1. Consequently, similar absolute gaps can produce very different participation: their measured singleton rates are 0.05% and 6.88%.

For sparsemax and normmax-2, the threshold is the same, but the learned distributions differ: their median gaps are 0.40 and 0.61, and 6.88% versus 21.26% of their gaps reach the shared threshold. Here, the participation difference comes from the learned scores. Sharing a singleton threshold does not make the maps identical or require them to learn identical distributions. These two comparisons illustrate why participation must be understood through both the map’s threshold and the scores learned under it.

Q3: Why do you think the lower sensitivity to larger k is useful? In what situation do you consider it useful when your results show that a larger k makes the loss worse?

Increasing k can be useful, for example, when serving the same model under changing workloads, allowing it to use spare computation to improve accuracy when trained for such budget changes (Elastic MoE). This provides a motivation for studying sensitivity to budget changes. Our results show that sparse probability maps improve robustness to increasing k by limiting the resulting loss degradation. Whether this robustness also benefits models trained to exploit variable inference budgets remains an open question.

Reviewer_xJJCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies what happens when the softmax in a top-2 MoE router is replaced with a sparsity-inducing router (entmax, sparsemax, or normmax), which in principle lets a token give zero weight to a selected expert and so use fewer than K experts. The authors train MoEs on 10B tokens that differ only in the probability map used, and measure how much probability mass the selected experts keep, how many selected experts actually contribute, how well things are load balanced, and final model quality on language modeling tasks.

They find that none of the sparse maps improves quality over softmax, and that the maps behave very differently once trained: entmax almost never drops an expert, while normmax routes about a fifth of tokens to a single expert. They explain this through each map's fixed threshold on the gap between the top two scores, combined with the score scale the router learns. They also show that the sparse maps are much less hurt than softmax when a model trained with K=2 is run with K=8, although this robustness it achieved by assigning 0 scores to many of the added experts.

Critical strengths

This is a clean, carefully controlled study in which everything except the probability map is matched, and the paper is clearly written and clear about its limitations. The decomposition of routing behavior into retention and participation, and the explanation of participation through learned score gaps relative to each map's threshold, is a clear and simple diagnostic that could be useful to people designing sparse routers. Overall I found this to be a well designed and executed study.

Critical weaknesses

The most major issue in my opinion is that it's not clear what motivates the paper. I don't think these sorts of sparse routers are currently in use. While it is interesting to consider using them, this paper doesn't provide a clear story on why one would do so. It studies qualitative properties of the different routers in detail, but this doesn't translate into any actionable recommendation or clear advantage. None of the sparse maps improves quality over softmax, and the adaptive participation that motivates using them comes with no measured (or at least demonstrated) compute or latency savings. The paper also doesn't compare against, or cite, established ways of getting a variable number of experts per token, such as top-p or threshold routing (Huang et al. 2024) or null and zero-computation experts (AdaMoE, MoE++). So a practitioner is left without a reason to choose a sparse map, or a sense of when it would help.

One difficulty in motivating the method could be the result of choosing K=2; with such few experts, it's hard to imaging winning by dropping one (this cuts the expert compute budget dramatically), and its hard to imaging winning by turning K up at inference time when only 2 was used for training.

The one practical advantage claimed, robustness to increasing K at inference, isn't very exciting. The sparse maps are more "robust" to running K=8 on a model trained with K=2, but they achieve this by not actually using the extra experts: sparsemax gives zero weight to almost half of the slots, and for every map the loss is lowest at the trained K=2. A result where adding experts at inference actually improved the model would be compelling, as would a result where you can remove experts at inference time without damaging performance.

Questions for authors

What is the intended practical use of these findings? Is there a setting where a sparse map gives a clear advantage over softmax in quality or in compute actually saved? How do the sparse maps compare with top-p or threshold routing and with null-expert approaches?

4 comments
AuthorsOpenReview ↗

Part 1 of 2 - Practical use, latency, and matched-budget quality

Q1 and critical weakness 1. “What is the intended practical use of these findings?”

“So a practitioner is left without a reason to choose a sparse map, or a sense of when it would help.”

While differentiable sparse transformations are not widely used in practice, we believe that our work addresses a gap in the literature by studying, both empirically and theoretically, what they are capable of in MoEs.

Our motivation was to test whether a sparse router could learn how many experts each token needs. If the router assigns positive weight to only a few experts, we could use all of them without discarding probability mass through top-K truncation. We began with the conventional top-2 setting to compare the maps under the same constraint before asking whether removing the cap improves training stability or loss.

To see when a sparse map would be useful in practice, we ran two comparisons, which we report in Q2. First, we timed full-sequence forward passes. Ordinary top-2 softmax was fastest, so we would choose it when latency matters. We then compared prediction loss on held-out web articles after adjusting each router to use 1.6 experts per token on average. At that budget, we would consider normmax-2 because it achieved lower loss than capped softmax in all three 300M runs.

Q2. “Is there a setting where a sparse map gives a clear advantage over softmax in quality or in compute actually saved?”

First, we note that to save time we would need specialized kernels for leveraging sparsity and making our methods more efficient. For example, existing MoE systems such as MegaBlocks (Gale et al., 2023) use specialized block-sparse kernels for expert computation. In our case, we use the PyTorch implementations from the entmax package (Peters et al., 2019), which use sorting to compute 1.5-entmax and sparsemax and root-finding for normmax-2, and are slower than softmax in the forward pass. AdaSplash (Gonçalves et al., 2025) and AdaSplash-2 (Gonçalves et al., 2026) demonstrate ongoing efforts to develop faster kernels for sparse transformations, which could reduce our probability map overhead. To compete in MoE efficiency, we would need to combine faster map computation with optimized MoE kernels that leverage sparsity. Nevertheless, we benchmarked our implementations (which use a dispatcher that skips zero-weight selections), measuring both the savings in expert computation and the full-pass latency.

The table below shows the 300M results on one RTX A6000, using BF16 inference, batch size 8 and sequence length 2,048.

Router Estimated expert matrix FLOPs saved Estimated whole-model matrix FLOPs saved Map time, separate profile (ms) Median full-pass latency (ms)
Softmax0%0%0.0358.85
1.5-entmax0.06%0.01%1.4060.72
Sparsemax5.05%1.11%12.1471.14
Normmax-211.92%2.61%11.7870.41

Fewer contributing experts reduce expert computation, while the overhead of our current sparse probability map implementations outweighs these savings as expected.

Prediction quality at the same expert budget.

For each of the three 300M runs per router, we calibrated the sparse routers' score scales and softmax's one-expert cutoff on 32 web articles to average 1.6 experts per token. On 160 held-out articles, expert FLOPs matched within 0.5%.

Router Mean ΔNLL vs. capped softmax (nats/token) Runs with lower NLL Mean latency increase vs. capped softmax, batch 8 (ms)
1.5-entmax+0.01930/3+1.50
Sparsemax+0.00062/3+11.97
Normmax-2−0.01083/3+11.75

At 1.6 experts per token, normmax-2 achieved lower loss than capped softmax in all three runs, by 0.0108 nats per token on average. Its full forward pass took 11.75 ms longer on average at batch size 8.

The quality advantage depended on the budget. At 1.8 experts per token, normmax-2 beat capped softmax in two of three runs. Ordinary top-2 softmax used more expert FLOPs and achieved lower loss than every sparse map at the 1.6 target.

AuthorsOpenReview ↗

Part 2 of 2 - Routing baselines and varying the expert budget

Q3. “How do the sparse maps compare with top-p or threshold routing and with null-expert approaches?”

In our top-2 models, the router selects two experts for each token. A sparse probability map can give the second expert exactly zero weight, so only the first contributes. Top-p routing instead varies how many experts are selected. Huang et al. take experts in order of probability until their cumulative probability exceeds a chosen threshold. Huang et al., 2024

Null-expert approaches change which experts the router can select. AdaMoE adds null experts to the pool. When a null expert takes a top-k slot, the token uses one fewer feed-forward expert. MoE++ also offers experts that return zero, copy the input, or mix it with a learned vector. Our sparse maps keep the original expert pool. AdaMoE, MoE++

In Q2, we compared the sparse maps with a simple threshold rule applied to a frozen softmax router. At the same average budget of 1.6 experts per token, normmax-2 achieved lower prediction loss on held-out web text in all three runs, while capped softmax was faster. How sparse maps compare with models trained from the outset with top-p routing or with null or zero-computation experts remains open.

Critical weakness 2. Choice of K=2

We thank the reviewer for pointing out a limitation of training with K=2. We agree that this choice gives a sparse map little room to vary how many experts contribute to a token. Dropping from two experts to one halves the expert computation for that token in that layer. Training with a larger K would give the map more experts to choose from and could improve the tradeoff between quality and computation.

We chose top-2 deliberately for this initial study, as done by Zhou et al. (2022). Keeping the selection rule fixed let us isolate the effect of the probability map and compare each map directly with standard top-2 softmax routing within our training budget. A natural next step is to compare larger values of K at matched training compute, including an uncapped setting where the sparse map alone determines which experts contribute.

Critical weakness 3

The one practical advantage claimed, robustness to increasing K at inference, isn't very exciting. The sparse maps are more "robust" to running K=8 on a model trained with K=2, but they achieve this by not actually using the extra experts: sparsemax gives zero weight to almost half of the slots, and for every map the loss is lowest at the trained K=2.

We agree that K=2 was the better choice for our checkpoints. As the reviewer observes, sparsemax left many of the extra slots unused. Even 1.5-entmax, which used 7.98 of eight experts on average at 1B and suffered a smaller loss increase than softmax, still performed best at K=2 (submitted paper, Tables 3 and 9).

Increasing K would be useful, for example, if a service could use spare compute during periods of low demand to improve predictions with the same checkpoint. Elastic MoE demonstrated such gains after training experts to work in different combinations. Whether sparse probability maps help in that setting remains open. Gu et al., Elastic MoE

A result where adding experts at inference actually improved the model would be compelling, as would a result where you can remove experts at inference time without damaging performance.

To test the cost of using fewer experts, we used the web evaluation described in Q2. For each frozen checkpoint, the baseline is its native top-2 routing. We count selected experts with positive weight (K₊) for each input token at each MoE layer, then average over tokens and the four MoE layers. The loss difference (ΔNLL, nats/token) is the loss at the indicated target mean minus the baseline loss. All columns are averaged equally over three 300M runs per router.

Router Mean contributing experts (K₊), native top-2 ΔNLL at target mean 1.8 ΔNLL at target mean 1.6
1.5-entmax1.999+0.0194+0.0437
Sparsemax1.882+0.0039+0.0244
Normmax-21.774+0.0021+0.0122

For sparsemax, lowering the mean contributor count from 1.882 to a target of 1.8 increased loss by 0.0039 nats per token on average. At the 1.6 target, all three routers used fewer experts and had higher loss in every run. Normmax-2 had the smallest mean loss increase at that budget. Its 1.8 target was above its native top-2 mean of 1.774.

In a separate test on 1.5-entmax, removing the second expert from the 20% of two-expert tokens with the largest router score gaps gave lower loss than removing it from the same number of randomly chosen tokens in every layer tested, across all three runs on both DCLM and Paloma.

Hello! We would like to thank you for your review and let you know that if you have any further questions, we would be happy to respond during the remaining duration of the discussion period. If our response has adequately addressed your concerns, we would be grateful if you could update your assessment.

Dear Reviewer,

Thank you again for the time and effort you have put into reviewing our paper. We have addressed each of your questions and provided additional experiments to support our responses.

As the discussion period draws to a close, we would appreciate your consideration of these responses and results. If your concerns have been resolved, we would be grateful if you could reflect this in your final assessment.

Kind regards,

Reviewer_gGqxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how sparsity-inducing probability maps affect expert participation in Mixture-of-Experts (MoE) models. The authors compare softmax, entmax, sparsemax, and normmax using matched 300M and 1B language models. They find that sparse probability maps do not necessarily produce sparse expert participation after training, as router scores adapt to the chosen map.

Critical strengths
  1. The paper demonstrates that sparsity in probability maps does not directly translate into sparse expert participation. The analysis of score distributions and sparsification thresholds provides a convincing explanation for the different behaviors.

  2. The authors train matched models at two scales while varying only the probability map. The experiments distinguish retained probability mass from actual expert participation and reveal substantial differences across maps. The inference-time budget analysis also provides an interesting observation that sparse maps are less sensitive to increasing the number of selected experts.

Critical weaknesses
  1. None of the sparse probability maps improves validation loss over softmax, and the paper does not measure actual computational savings from reduced expert participation. Since the implementation does not necessarily skip zero-weight experts, lower participation may not translate into lower inference cost. The practical implications of the findings remain unclear.

  2. The experiments are restricted to two relatively small model scales, a fixed top-2 routing configuration, and one primary load-balancing objective.

Questions for authors
  1. Can the authors quantify the actual inference FLOPs or latency savings when zero-weight experts are skipped? How do these savings compare against standard top-2 softmax routing?

  2. How consistent are the learned score distributions and expert participation patterns across different training seeds or larger MoE models?

  3. Would explicitly controlling router score scales or sparsity thresholds improve the accuracy–efficiency tradeoff?

4 comments
AuthorsOpenReview ↗

Part 1 of 2 - Expert computation, scope, and replication

W1 / Q1. Expert computation and inference latency

First, we note that to save time we would need specialized kernels for leveraging sparsity and making our methods more efficient. For example, existing MoE systems such as MegaBlocks (Gale et al., 2023) use specialized block-sparse kernels for expert computation. In our case, we use the PyTorch implementations from the entmax package (Peters et al., 2019), which use sorting to compute 1.5-entmax and sparsemax and root-finding for normmax-2, and are slower than softmax in the forward pass. AdaSplash (Gonçalves et al., 2025) and AdaSplash-2 (Gonçalves et al., 2026) demonstrate ongoing efforts to develop faster kernels for sparse transformations, which could reduce our probability map overhead. To compete in MoE efficiency, we would need to combine faster map computation with optimized MoE kernels that leverage sparsity. Nevertheless, we benchmarked our implementations (which use a dispatcher that skips zero-weight selections), measuring both the savings in expert computation and the full-pass latency.

The table below shows the 300M results on one RTX A6000, using BF16 inference, batch size 8 and sequence length 2,048.

Router Estimated expert matrix FLOPs saved Estimated whole-model matrix FLOPs saved Map time, separate profile (ms) Median full-pass latency (ms)
Softmax0%0%0.0358.85
1.5-entmax0.06%0.01%1.4060.72
Sparsemax5.05%1.11%12.1471.14
Normmax-211.92%2.61%11.7870.41

Fewer contributing experts reduce expert computation, while the overhead of our current sparse probability map implementations outweighs these savings as expected.

W2. Scope of the controlled study

While we agree with the reviewer that our study has a narrowed scope, we do so on purpose. Our goal is not to evaluate big models (which require a tremendous amount of compute), but rather to study, in detail, the impact of different activation maps in the routing component of MoEs. Our baseline follows the top-2 selection and weighted combination used in Mixtral (Jiang et al., 2024). Zhou et al. (2022) also used top-2 softmax as a baseline for comparing routing methods at similar model sizes (548M and 1B parameters). At each of our two scales, we vary only the probability map, keeping the architecture, training recipe and load-balancing objective fixed.

Q2. How consistent are the learned score distributions and expert participation patterns across different training seeds or larger MoE models?

We evaluated each 300M router on the same 61 documents from the WikiText-103 test subset of Paloma. Centered score RMS measures how spread out a token’s scores are across experts. Each token makes a routing decision at each of the four MoE layers. We count a decision as having one contributor when only one of the two selected experts receives positive weight. We report the mean ± standard deviation across four training runs per router. These include the run from Table 7 and three runs under the replication protocol, one of which reused the original seed.

Router Centered score RMS Routing decisions with one contributor (%)
Softmax0.984 ± 0.0120.00 ± 0.00
1.5-entmax0.520 ± 0.0120.15 ± 0.10
Sparsemax0.419 ± 0.00910.53 ± 1.43
Normmax-20.529 ± 0.00421.62 ± 1.49

In all four 300M runs, the sparse maps had smaller score spreads than softmax. The ordering of the four routers by how often they used a single expert was also the same at 300M and 1B. We will include the four-run results in the revision.

AuthorsOpenReview ↗

Part 2 of 2 - Score-scale control

Q3. Would explicitly controlling router score scales or sparsity thresholds improve the accuracy–efficiency tradeoff?

This is an interesting question. To answer you, we ran a simple experiment:

At inference time, we rescaled each token’s centered expert scores to a root mean square (RMS) of 1.0 for all three sparse routers. The table compares each router with its native scale and reports the mean ± standard deviation across three runs per router. Positive ΔNLL means higher loss.

Router Expert matrix FLOPs saved on DCLM DCLM ΔNLL (nats/token) Paloma ΔNLL (nats/token)
1.5-entmax7.75 ± 0.32%+0.0143 ± 0.0011+0.0088 ± 0.0054
Sparsemax18.30 ± 1.75%+0.0320 ± 0.0016+0.0303 ± 0.0049
Normmax-215.21 ± 0.44%+0.0232 ± 0.0005+0.0169 ± 0.0028

Each used fewer expert FLOPs and had higher NLL than at its native scale in every run on both samples.

For training-time control, we tested only 1.5-entmax. We trained three 300M models using the same seed and top-2 routing. Each model used one of the target scales in the table from the start of training. Our baseline was a 300M 1.5-entmax model trained with the same seed and an unconstrained score scale.

The table reports full-validation results after 19,073 updates. We count a decision as having one contributor when only one of the two selected experts receives positive weight.

1.5-entmax training condition Routing decisions with one contributor Validation loss (nats)
Unconstrained score scale0.1250%3.51220
Fixed RMS 0.50.0065%3.51619
Fixed RMS 1.08.85%3.51614
Fixed RMS 1.522.45%3.52480

By fixing the score scale during training, we could control how often 1.5-entmax used a single expert. Compared with the unconstrained model, fixing RMS at 1.0 gave us a favorable tradeoff, saving about 4% of expert FLOPs with only a small increase in full-validation loss. At the highest scale, single-expert routing became more common still, but validation loss rose further. Whether tuning the sparsity threshold offers a similar benefit remains open.

Hello! We would like to thank you for your review and let you know that if you have any further questions, we would be happy to respond during the remaining duration of the discussion period. If our response has adequately addressed your concerns, we would be grateful if you could update your assessment.

Dear Reviewer,

Thank you again for the time and effort you have put into reviewing our paper. We have addressed each of your questions and provided additional experiments to support our responses.

As the discussion period draws to a close, we would appreciate your consideration of these responses and results. If your concerns have been resolved, we would be grateful if you could reflect this in your final assessment.

Kind regards,

Comments on the paper

1 comment
AuthorsOpenReview ↗

General comment to the AC and reviewers

Dear AC and reviewers,

Thank you for your feedback. We would like to clarify the contributions of our paper. We kept top-2 selection fixed during training and varied only the probability map. We highlight three findings from this comparison.

(1) The trained 1.5-entmax router retains more probability mass than softmax while still using both selected experts almost all the time (Section 4.2).

(2) Sparsemax and normmax-2 share the same threshold for using a single expert. Normmax-2 crosses it more often because its trained router learns larger gaps between the two highest scores (Section 4.3).

(3) At both model scales, increasing the expert budget at inference causes a smaller increase in loss for every sparse map than for softmax (Section 4.4).

The score-scale experiment in our replies belongs to (2). It answers the question of whether we can control how often the router crosses the threshold. Under 1.5-entmax, raising the target score scale during training makes single-expert routing more frequent. Our aim was to understand how existing sparse maps behave after training. Proposing a new router that outperforms softmax is a separate goal. We will make this framing more explicit in the revised version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.