- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies whether the sparsity of sparse probability maps (1.5-entmax, sparsemax, and 2-normmax) survives training when used as routers in top-2 Mixture-of-Experts (MoE) language models. They show that the probability map alone does not determine how many experts a token uses because the router co-adapts the scores it feeds into the map. During training, the 1.5-entmax router learns much smaller score spreads than softmax, keeping its score gaps below its threshold so it almost always uses both experts while discarding less probability mass. Meanwhile, sparsemax and 2-normmax share the exact same threshold, yet 2-normmax routes three times as many tokens to a single expert because its router learns larger score gaps. While none of the sparse maps improve validation loss over softmax at the trained top-2 budget, they make the models much less sensitive to increasing the number of selected experts at inference time because they assign zero weight to most of the extra expert slots.
Critical strengths
-
Clear separation of the probability map from the learned router scores. A common assumption in adaptive MoE routing is that swapping softmax for a sparse probability map automatically yields dynamic expert usage. By connecting each map's exact sparsification threshold to the score distributions actually learned during training, the paper proves that routers co-adapt their score scales to the map.
-
Rigorous distinction between probability retention, expert participation, load balance, and model quality. They show that retaining more probability mass and using fewer experts are separate phenomena. Their inference-time budget tests and load-balancing ablations are well-controlled and directly test practical hypotheses about how sparse maps interact with top-k truncation and auxiliary balancing losses.
Critical weaknesses
-
While the paper clearly shows that sparse routers learn smaller score spreads than softmax and that 2-normmax learns larger score gaps than sparsemax, it does not explain the underlying optimization or gradient dynamics during training that cause the routers to settle into these specific score distributions.
-
The paper does not experimentally test its own design recommendations. They conclude that adaptive routing requires explicitly controlling or learning the router's score scale and moving beyond a fixed top-2 budget, yet all experiments use unconstrained router scores at top-2.
Questions for authors
Have you tested what happens if you explicitly control or normalize the router's score spread before applying the probability map? If you prevent the score spread from shrinking under 1.5-entmax, does it continue routing tokens to single experts throughout training, and how does that affect validation loss and load balance?
3 comments
Part 2 of 2 - Score normalization, loss, and load balance
The paper does not experimentally test its own design recommendations. They conclude that adaptive routing requires explicitly controlling or learning the router's score scale and moving beyond a fixed top-2 budget, yet all experiments use unconstrained router scores at top-2.
We agree that it is worth testing how routing changes when the score scale is controlled during training. We chose top-2 routing, following influential work in the MoE literature such as GShard (Lepikhin et al., 2021) and Mixtral (Jiang et al., 2024), so we could compare each probability map directly with softmax under the same routing budget.
Even with the same top-2 budget, the spread of the scores depended on which probability map we trained with. This matters because the second expert contributes only while the gap between the two highest scores stays below the map’s threshold. The scale the router learns therefore helps determine how often both experts contribute. This is why we suggested making score scale an explicit part of the router design. We left open how best to do that.
We could control the scale in several ways, for example by letting each MoE layer learn its own scale parameter or by normalizing the scores to a fixed scale throughout training. Following your suggestion, we tested the latter by training three 1.5-entmax models with different target values for the root mean square (RMS) of each token’s centered router scores. We report the results in our answer to your next question.
Have you tested what happens if you explicitly control or normalize the router's score spread before applying the probability map?
Yes. We tested this with the 1.5-entmax probability map. We trained three 300M models using the same seed and top-2 routing. For each token, we centered its 26 expert scores and rescaled them to a fixed root mean square (RMS) before applying 1.5-entmax. We used a target RMS of 0.5, 1.0 or 1.5 from the start of training. Our baseline was a 300M 1.5-entmax model trained with the same seed and an unconstrained score scale.
The table reports full-validation results after 19,073 updates. Each token makes a routing decision at each of the four MoE layers. We count a decision as having one contributor when only one of the two selected experts receives positive weight.
| 1.5-entmax training condition | Routing decisions with one contributor | Validation loss (nats) | ||
|---|---|---|---|---|
| Unconstrained score scale | 0.1250% | 3.51220 | ||
| Fixed RMS 0.5 | 0.0065% | 3.51619 | ||
| Fixed RMS 1.0 | 8.85% | 3.51614 | ||
| Fixed RMS 1.5 | 22.45% | 3.52480 |
Raising the target RMS from 0.5 to 1.0 made single-expert routing more common with little change in validation loss. At RMS 1.5, single-expert routing became more common still, but validation loss rose. Fixing the score scale therefore gave us some control over expert participation, with little change in validation loss in this comparison.
If you prevent the score spread from shrinking under 1.5-entmax, does it continue routing tokens to single experts throughout training, and how does that affect validation loss and load balance?
Yes. At RMS 1.0 and 1.5, we observed single-expert routing at every training step we measured. For a given set of router scores, raising the RMS widens the gap between the two highest scores, making it easier to reach the threshold of 2 at which 1.5-entmax gives all the weight to one expert. As the table in our previous answer shows, final validation loss stayed close to the unconstrained baseline at RMS 1.0 and rose slightly at RMS 1.5. Expert loads were less even than in the unconstrained model at both targets, especially at RMS 1.5.
Hello! We would like to thank you for your review and let you know that if you have any further questions, we would be happy to respond during the remaining duration of the discussion period.
Part 1 of 2 - Training dynamics
This is true. To investigate this, we analyzed gradients at 15 checkpoints from initialization to update 19,073 in four 300M models trained with the same seed.
The table reports the balancing gradient norm as a percentage of the task gradient norm on router weights, including the balancing coefficient of 0.01. Thus, 100% means equal size. Measurements use fixed inputs and average norms over batches and MoE layers.
These checkpoints show a shared transition: balancing gradients exceed task gradients early, at update 50, but fall to approximately 2% of their magnitude later.
To examine how these losses act on score spread, we also measured scale derivatives, which describe how each loss changes when score differences are slightly widened. At updates 500 and 1,000, the combined loss locally favors narrowing under sparsemax but widening under the other three maps in the pooled measurements. The top-two gap derivatives show the same contrast. By update 2,000, all three sparse maps locally favor narrowing the score spread.
Sparsemax therefore exhibits pressure against score expansion earlier in the sampled trajectory, consistent with its subsequently smaller score spread and gaps. This difference in optimization pressures may help account for the learned distributions. We will add the full gradient and derivative trajectories alongside score spreads and gaps, with layerwise results and measurement details, to the appendix.