- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argues that Top-k routing in MoE models is unstable because a small change in router logits can flip the selected experts. StaVER treats expert assignment as a latent variable, samples experts at both training and test time, and derives a variational objective whose prior term replaces the load-balancing loss. On models of three sizes trained from scratch, it reports better downstream accuracy than Top-k baselines, and lower perplexity and more balanced expert use in the settings where these are measured.
Critical strengths
- An elegant algorithm. Routing becomes probabilistic inference, and the balancing term is motivated by the variational objective, so it does not need to be designed by hand. With expert use more even, fewer experts are likely left under-trained, and the lower perplexity suggests the model makes better use of its capacity.
- Thorough experiments: three model sizes, a second pretraining corpus, component ablations and hyperparameter sweeps.
Critical weaknesses
- Sampling at inference makes routing random, and the paper does not measure what this does to train-inference consistency. The same input can take different experts on each pass, so outputs vary unless the sampling noise is fixed. In RL on sampled rollouts, for example, a rollout may be generated with one set of experts and scored for the update with another. The two passes then see slightly different models. Deterministic routing at test time is not evaluated, and all downstream tasks are multiple-choice or classification, so generation is not tested.
- Results appear to come from a single training run and a single stochastic inference pass, with no variance across seeds or passes reported. Several downstream tasks are near chance and the lead over the strongest baseline is a couple of points, so it is hard to tell how much of the gain is stable.
Questions for authors
- How does StaVER perform with deterministic routing at test time, and does it stay consistent on generation tasks and under further fine-tuning or RL?
- Which balancing coefficient do the main results use, and how many MoE layers does each model have? The setup states α = 0.2 for all methods, but the sensitivity sweep only covers 0, 0.001, 0.01, 0.1, 1 and 10. The expert-load analysis also refers to all 12 layers, while the setup says every feed-forward block of the 24 layers is an MoE layer.
- Can you report variance over several seeds and repeated inference passes?
5 comments
Q1: How does StaVER perform with deterministic routing at test time, and does it remain consistent on generation tasks and under further fine-tuning or RL?
A1. Thanks for the question. As suggested, we conduct two additional experiments: (i) inference-time comparison between deterministic and stochastic routing, and (ii) few-shot evaluation on the generation task SQuAD v1.1. We leave further fine-tuning or RL to future work, for the reasons explained in (iii).
(i) Deterministic vs. stochastic routing. The results are summarized in Table II below, with details in Appendix G.9 Table 8. Stochastic routing consistently outperforms deterministic routing across all three model scales. We attribute this to a training–inference mismatch: StaVER learns a routing posterior during pretraining, whereas deterministic routing collapses this posterior into a single point estimate and this mismatch degrades performance. In contrast, the results show that the advantage of stochastic routing grows further as the number of samples K increases, since a larger K reduces the estimation error of the posterior and thus improves performance.
(ii) Performance on genration Task. Since StaVER is a pretraining framework, we evaluate it in the few-shot setting without finetuning, which most directly evaluate the pretrained model quality. On SQuAD v1.1, StaVER outperforms the best baseline G-Shard by 1.3 EM points and 3.83 F1 points (see Table III). But without post-training the base model produce limited-quality outputs for all methods, making such comparisons less informative.
(iii) Further training with fine-tuning/RL. These stages introduce additional factors, e.g., data, training recipes, and reward design, that confound the final results, so the observed differences can no longer be explicitly attributed to the pretraining method. So, we leave this setting to our future work.
Summary. These results further show that stochastic routing consistently outperforms deterministic routing at inference, with further gains as K increases. StaVER also maintains its advantage on generation tasks.
Table II. Average accuracy over 8 benchmarks with deterministic and stochastic routing with repeated inference passes. Deterministic routing selects the top-2 / 4 / 8 experts for different model scales. For stochastic routing, we evaluate three sample sizes K for each model and report repeated result over 3 random seeds. Detailed results are provided in Appendix G.9, Table 8.
| Setting | Deterministic Routing | Stochastic (Small K) | Stochastic (Medium K) | Stochastic (Large K) |
|---|---|---|---|---|
| 8 Experts, 1.8B | 47.5 (top-2) | 47.7 ± 0.16 (K = 2) | 47.9 ± 0.12 (K = 4) | 48.0 ± 0.09 (K = 8) |
| 16 Experts, 3.4B | 50.1 (top-4) | 50.3 ± 0.17 (K = 4) | 50.4 ± 0.13 (K = 8) | 50.7 ± 0.09 (K = 12) |
| 32 Experts, 6.6B | 52.9 (top-8) | 53.1 ± 0.14 (K = 8) | 53.3 ± 0.13 (K = 12) | 53.4 ± 0.10 (K = 16) |
Table III. Comparison of exact match (EM) and token-level F1 scores on SQuAD v1.1 under the 3-shot setting.
| Method | GShard | Switch Transformer | StableMoE | Auxiliary-FreeMoE | DeepseekMoE | Ours |
|---|---|---|---|---|---|---|
| EM(%) | 5.1 | 1.5 | 1.8 | 0.2 | 3.6 | 6.4 |
| Token F1(%) | 13.12 | 10.32 | 9.54 | 8.11 | 11.01 | 16.95 |
Q2: Which balancing coefficient is used in the main results, and how many MoE layers does each model contain?
A2. We thank the reviewer for checking all the details. We clarify both points below and have revised the corresponding text in the manuscript.
(i) Balancing coefficient. The main results use α = 0.2, selected through grid search. The appendix sweep was intended to show that our method is robust to the coefficient choice over a broad range. We have now added the results for α = 0.2 to Table 6 in the appendix.
(ii) Number of MoE layers. Each model has 24 MoE layers. The original expert-load analysis in Figure 4 reported only the last 12 layers (layers 13–24), where expert loads were generally more stable. We have clarified this in the title description and added the results for all 24 layers in Appendix G.10 Figure 12. The full results show that StaVER maintains the most consistently balanced expert utilization across all layers, while the baselines exhibit varying degrees of layer-dependent imbalance.
Dear Reviewer 6amR,
We have provided responses to your questions and comments. We would appreciate it if you could let us know whether you have any additional questions or concerns.
Thank you for your time and consideration.
Best regards,
The Authors
We sincerely thank you for your insightful comments, and for recognizing our algorithmic design and experimental coverage. We address each of your comments below and hope our clarifications resolve your concerns.
Concern 1: Effect of stochastic routing on train–inference consistency.
R1. Thank you for raising this interesting point. While sampling does introduce variance in the posterior estimate, we would like to clarify that this does not amount to training–inference inconsistency.
First, our method already accounts for this sampling variability during pretraining. Specifically, we learn a posterior over routing variables and sample expert subsets from this posterior to update the model. The posterior and model parameters are jointly optimized under the training objective until convergence. Inference then continues to sample from the learned posterior. Training and inference are therefore consistent at the distributional level: for a fixed input and trained model, different expert selections are samples from the same routing distribution and share the same expected prediction. In other words, the selected experts may vary across sampling passes for an individual token, but sequence-level inference still follows the probabilistic routing mechanism learned during pretraining.
Second, to further support this claim, we conducted two additional experiments.
(i) Routing divergence across training. In Figure 9(a) in Appendix G.7, we track the KL divergence between the training-time and inference-time routing distributions throughout pretraining. The divergence decreases rapidly and stabilizes at ~0.002 after 50% of training. Thus, the posterior converges early in pretraining, ****and thereafter the posterior used at inference closely resembles the one used during training.
(ii) Per-token routing consistency. As shown in Figure 9(b), for a fixed input token, we compare the router posterior with its empirical routing frequency over the final training epoch and over repeated inference passes. As both are drawn from the converged posterior, the two empirical distributions closely align with it, confirming that the sampled expert sets at training and inference are statistically consistent.
Summary. (i) shows that training and inference share a very close posterior, and (ii) shows that sampling in both phases provides an accurate estimate of it. The sampling variance therefore does not amount to a training–inference inconsistency.
Concern 2 (Q3): The reliability of the reported improvements is unclear without variance across training runs and inference passes.
R2. Thanks for the suggestion. We provide variance across multiple training random seeds and repeated inference passes. We summarize the key results in Table I and Table II below, with full details in Appendix G.9 Table 8.
(i) Variance across training runs (Table I). Due to the limited time, we compare against the strongest baseline, and train both models under 3 random seeds. Across all 8 benchmarks, our method's stds are similar to GShard's at both the task level and on average, with averages of 0.32 and 0.27, respectively. This indicates that our sampling strategy does not introduce extra variance during training. Meanwhile, the average improvement of 2.5 points remains clearly beyond the variance range. We will include multi-seed results for the other baselines in the final version.
(ii) Variance across inference passes (Table II). We evaluate our stochastic routing at three sample sizes K, with 3 random seeds per setting. The std remains below 0.17 in all settings, with variance shrinking as K grows.
Summary. In both training and inference, the variance of our method is small and comparable to that of a deterministic baseline, while the improvement is consistently and substantially larger. Hence, we believe the reported gains are reliable and our method is robust to randomness in both training and inference. Table I. Comparison with the strongest baseline under different training seeds.