acceptodds
Under review as a conference paper at ICLR 2027

StaVER: A Probabilistic Formulation for Stable Routing in Mixture-of-Experts

Abstract

Mixture-of-Experts (MoE) models scale capacity efficiently by routing each token to a small subset of experts. A key challenge in MoE models is routing instability, which can reduce training efficiency and impair model performance. Through empirical studies and theoretical analysis, we identify the discontinuity of hard Top- selection as a source of this instability: perturbations can abruptly change the selected expert set, leading to performance degradation. To address this, we propose StaVER, a unified Bayesian framework that formulates MoE routing as a probabilistic inference problem. StaVER models expert assignment as a latent variable and performs routing by sampling experts from its distribution. Under this probabilistic formulation, the expected expert-selection frequencies vary continuously to perturbations, smoothing the abrupt changes in expert selection induced by hard Top- routing. To compute this intractable posterior distribution over expert assignment, we follow variational inference to derive a unified training objective. This yields a prior-matching term that naturally promotes load balancing, thus eliminating the need for an additional heuristic auxiliary loss. Through extensive experiments on MoE models ranging from 1.8B to 6.6B parameters, we demonstrate that StaVER achieves more stable routing, more balanced expert utilization, and consistent performance gains on both upstream and downstream tasks. Across eight downstream benchmarks, StaVER improves average accuracy up to 2.6 percentage points over the strongest baseline. In the pre-training evaluation, it obtains the lowest PPL, while reducing estimated training FLOPs by approximately 18%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Expert, Foundation Models, Variational Inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6amROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that Top-k routing in MoE models is unstable because a small change in router logits can flip the selected experts. StaVER treats expert assignment as a latent variable, samples experts at both training and test time, and derives a variational objective whose prior term replaces the load-balancing loss. On models of three sizes trained from scratch, it reports better downstream accuracy than Top-k baselines, and lower perplexity and more balanced expert use in the settings where these are measured.

Critical strengths
  1. An elegant algorithm. Routing becomes probabilistic inference, and the balancing term is motivated by the variational objective, so it does not need to be designed by hand. With expert use more even, fewer experts are likely left under-trained, and the lower perplexity suggests the model makes better use of its capacity.
  2. Thorough experiments: three model sizes, a second pretraining corpus, component ablations and hyperparameter sweeps.
Critical weaknesses
  1. Sampling at inference makes routing random, and the paper does not measure what this does to train-inference consistency. The same input can take different experts on each pass, so outputs vary unless the sampling noise is fixed. In RL on sampled rollouts, for example, a rollout may be generated with one set of experts and scored for the update with another. The two passes then see slightly different models. Deterministic routing at test time is not evaluated, and all downstream tasks are multiple-choice or classification, so generation is not tested.
  2. Results appear to come from a single training run and a single stochastic inference pass, with no variance across seeds or passes reported. Several downstream tasks are near chance and the lead over the strongest baseline is a couple of points, so it is hard to tell how much of the gain is stable.
Questions for authors
  1. How does StaVER perform with deterministic routing at test time, and does it stay consistent on generation tasks and under further fine-tuning or RL?
  2. Which balancing coefficient do the main results use, and how many MoE layers does each model have? The setup states α = 0.2 for all methods, but the sensitivity sweep only covers 0, 0.001, 0.01, 0.1, 1 and 10. The expert-load analysis also refers to all 12 layers, while the setup says every feed-forward block of the 24 layers is an MoE layer.
  3. Can you report variance over several seeds and repeated inference passes?
5 comments

We sincerely thank you for your insightful comments, and for recognizing our algorithmic design and experimental coverage. We address each of your comments below and hope our clarifications resolve your concerns.

Concern 1: Effect of stochastic routing on train–inference consistency.

R1. Thank you for raising this interesting point. While sampling does introduce variance in the posterior estimate, we would like to clarify that this does not amount to training–inference inconsistency.

First, our method already accounts for this sampling variability during pretraining. Specifically, we learn a posterior over routing variables and sample expert subsets from this posterior to update the model. The posterior and model parameters are jointly optimized under the training objective until convergence. Inference then continues to sample from the learned posterior. Training and inference are therefore consistent at the distributional level: for a fixed input and trained model, different expert selections are samples from the same routing distribution and share the same expected prediction. In other words, the selected experts may vary across sampling passes for an individual token, but sequence-level inference still follows the probabilistic routing mechanism learned during pretraining.

Second, to further support this claim, we conducted two additional experiments.

(i) Routing divergence across training. In Figure 9(a) in Appendix G.7, we track the KL divergence between the training-time and inference-time routing distributions throughout pretraining. The divergence decreases rapidly and stabilizes at ~0.002 after 50% of training. Thus, the posterior converges early in pretraining, ****and thereafter the posterior used at inference closely resembles the one used during training.

(ii) Per-token routing consistency. As shown in Figure 9(b), for a fixed input token, we compare the router posterior with its empirical routing frequency over the final training epoch and over repeated inference passes. As both are drawn from the converged posterior, the two empirical distributions closely align with it, confirming that the sampled expert sets at training and inference are statistically consistent.

Summary. (i) shows that training and inference share a very close posterior, and (ii) shows that sampling in both phases provides an accurate estimate of it. The sampling variance therefore does not amount to a training–inference inconsistency.

Concern 2 (Q3): The reliability of the reported improvements is unclear without variance across training runs and inference passes.

R2. Thanks for the suggestion. We provide variance across multiple training random seeds and repeated inference passes. We summarize the key results in Table I and Table II below, with full details in Appendix G.9 Table 8.

(i) Variance across training runs (Table I). Due to the limited time, we compare against the strongest baseline, and train both models under 3 random seeds. Across all 8 benchmarks, our method's stds are similar to GShard's at both the task level and on average, with averages of 0.32 and 0.27, respectively. This indicates that our sampling strategy does not introduce extra variance during training. Meanwhile, the average improvement of 2.5 points remains clearly beyond the variance range. We will include multi-seed results for the other baselines in the final version.

(ii) Variance across inference passes (Table II). We evaluate our stochastic routing at three sample sizes K, with 3 random seeds per setting. The std remains below 0.17 in all settings, with variance shrinking as K grows.

Summary. In both training and inference, the variance of our method is small and comparable to that of a deterministic baseline, while the improvement is consistently and substantially larger. Hence, we believe the reported gains are reliable and our method is robust to randomness in both training and inference. Table I. Comparison with the strongest baseline under different training seeds.

MethodARC-EPIQASIQAWINOHELLAMNLIQNLISST-2Avg.
GShard41.4 ± 0.4759.3 ± 0.3839.2 ± 0.6350.1 ± 0.7232.2 ± 0.3634.3 ± 0.5849.7 ± 0.5355.6 ± 0.9445.2 ± 0.27
Ours44.3 ± 0.4361.4 ± 0.5139.3 ± 0.5651.1 ± 0.6133.3 ± 0.3935.6 ± 0.4952.5 ± 0.6263.8 ± 0.8347.7 ± 0.32

Q1: How does StaVER perform with deterministic routing at test time, and does it remain consistent on generation tasks and under further fine-tuning or RL?

A1. Thanks for the question. As suggested, we conduct two additional experiments: (i) inference-time comparison between deterministic and stochastic routing, and (ii) few-shot evaluation on the generation task SQuAD v1.1. We leave further fine-tuning or RL to future work, for the reasons explained in (iii).

(i) Deterministic vs. stochastic routing. The results are summarized in Table II below, with details in Appendix G.9 Table 8. Stochastic routing consistently outperforms deterministic routing across all three model scales. We attribute this to a training–inference mismatch: StaVER learns a routing posterior during pretraining, whereas deterministic routing collapses this posterior into a single point estimate and this mismatch degrades performance. In contrast, the results show that the advantage of stochastic routing grows further as the number of samples K increases, since a larger K reduces the estimation error of the posterior and thus improves performance.

(ii) Performance on genration Task. Since StaVER is a pretraining framework, we evaluate it in the few-shot setting without finetuning, which most directly evaluate the pretrained model quality. On SQuAD v1.1, StaVER outperforms the best baseline G-Shard by 1.3 EM points and 3.83 F1 points (see Table III). But without post-training the base model produce limited-quality outputs for all methods, making such comparisons less informative.

(iii) Further training with fine-tuning/RL. These stages introduce additional factors, e.g., data, training recipes, and reward design, that confound the final results, so the observed differences can no longer be explicitly attributed to the pretraining method. So, we leave this setting to our future work.

Summary. These results further show that stochastic routing consistently outperforms deterministic routing at inference, with further gains as K increases. StaVER also maintains its advantage on generation tasks.

Table II. Average accuracy over 8 benchmarks with deterministic and stochastic routing with repeated inference passes. Deterministic routing selects the top-2 / 4 / 8 experts for different model scales. For stochastic routing, we evaluate three sample sizes K for each model and report repeated result over 3 random seeds. Detailed results are provided in Appendix G.9, Table 8.

SettingDeterministic RoutingStochastic (Small K)Stochastic (Medium K)Stochastic (Large K)
8 Experts, 1.8B47.5 (top-2)47.7 ± 0.16 (K = 2)47.9 ± 0.12 (K = 4)48.0 ± 0.09 (K = 8)
16 Experts, 3.4B50.1 (top-4)50.3 ± 0.17 (K = 4)50.4 ± 0.13 (K = 8)50.7 ± 0.09 (K = 12)
32 Experts, 6.6B52.9 (top-8)53.1 ± 0.14 (K = 8)53.3 ± 0.13 (K = 12)53.4 ± 0.10 (K = 16)

Table III. Comparison of exact match (EM) and token-level F1 scores on SQuAD v1.1 under the 3-shot setting.

MethodGShardSwitch TransformerStableMoEAuxiliary-FreeMoEDeepseekMoEOurs
EM(%) ↑\uparrow5.11.51.80.23.66.4
Token F1(%) ↑\uparrow13.1210.329.548.1111.0116.95

Q2: Which balancing coefficient is used in the main results, and how many MoE layers does each model contain?

A2. We thank the reviewer for checking all the details. We clarify both points below and have revised the corresponding text in the manuscript.

(i) Balancing coefficient. The main results use α = 0.2, selected through grid search. The appendix sweep was intended to show that our method is robust to the coefficient choice over a broad range. We have now added the results for α = 0.2 to Table 6 in the appendix.

(ii) Number of MoE layers. Each model has 24 MoE layers. The original expert-load analysis in Figure 4 reported only the last 12 layers (layers 13–24), where expert loads were generally more stable. We have clarified this in the title description and added the results for all 24 layers in Appendix G.10 Figure 12. The full results show that StaVER maintains the most consistently balanced expert utilization across all layers, while the baselines exhibit varying degrees of layer-dependent imbalance.

Dear Reviewer 6amR,

We have provided responses to your questions and comments. We would appreciate it if you could let us know whether you have any additional questions or concerns.

Thank you for your time and consideration.

Best regards,

The Authors

Reviewer_Nj9mOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces StaVER, a probabilistic routing method for Mixture-of-Experts (MoE) models. The authors argue that much of the instability in standard MoE routing comes from the discontinuous nature of hard Top-k selection. StaVER instead samples expert assignments from a categorical distribution. Training is made possible through a straight-through Gumbel-Softmax estimator, and a batch-level KL-divergence term encourages the average routing distribution to remain close to a uniform prior, thereby promoting balanced expert usage.

Experiments on models ranging from 1.8B to 6.6B parameters show improvements in perplexity, downstream accuracy, expert utilization, and robustness to several perturbations. The authors also observe that repeated samples may select the same experts, reducing the number of unique expert evaluations. On this basis, they estimate an 18% reduction in FLOPs relative to comparable Top-k baselines.

Critical strengths

The paper addresses an important practical problem and reports strong empirical results. Routing instability remains a meaningful challenge in sparse MoE training. Across all three model scales, StaVER achieves the best reported average accuracy on eight downstream tasks, while also improving test perplexity and balancing expert utilization more effectively. The additional experiments on Dolma and the 6.6B model make the empirical evidence more convincing.

The link between stochastic routing and load balancing is intuitive and well-motivated. Sampling directly from routing probabilities offers a natural alternative to deterministic Top-k selection. Matching the aggregate routing distribution to a prior also provides a simple way to balance expected expert usage. The ablation studies suggest that both stochastic routing and prior matching contribute meaningfully to the final performance.

Critical weaknesses

The variational interpretation is not fully supported by the derivation. The exact negative ELBO includes a token-level KL term, which can be decomposed into an aggregate KL term and a mutual-information penalty, with coefficient K. StaVER drops the mutual-information component and replaces K with a tuned coefficient, alpha. As a result, Equation 19 is better understood as an ELBO-inspired surrogate than as the ELBO directly implied by the proposed probabilistic model. This distinction matters because it weakens the claim that load balancing follows naturally from a unified variational objective and removes the need for heuristic regularization.

The theoretical analysis does not establish the stability of the routing decisions actually used by the model. Proposition 3.1 bounds changes in the underlying categorical distribution, or equivalent in expected expert-selection frequencies. In practice, however, routing is based on a finite number of stochastic samples. The selected experts may therefore vary across runs even when the underlying distributions are close. Because stochastic sampling is retained at inference time, the evaluation should include prediction variance, routing agreement across random seeds, and the number of samples needed for reliable inference. Without these measurements, the comparison with deterministic Top-k routing risks conflating smooth changes in routing probabilities with stable computation and predictions.

Minor comments

The manuscript is generally clear, but several grammatical and presentation issues should be corrected. Examples include

“vary continuously to perturbations” -> “vary continuously with perturbations” (p. 1),

“routing changed rates” -> “routing change rates” (p. 2),

“Kullbeck-Leibler” -> “Kullback-Leibler” (p. 5),

“Conventional MoE balance” -> “Conventional MoEs balance” (p. 6),

“for each model sizes” -> “for the different model sizes” (p. 6),

Questions for authors
  1. Can Equation 19 be formulated exactly as an ELBO under a different generative model, or would it be more accurate to describe it as an ELBO-inspired surrogate?

  2. How much do sampled expert assignments and model predictions vary across repeated inference passes with different Gumbel noise?

  3. Does StaVER still outperform compute-matched Top-k baselines when the comparison is based on the actual average number of uniquely activated experts rather than the nominal sampling budget K?

  4. How do the FLOP estimates and wall-clock results account for expert-capacity constraints, duplicate samples, token dispatch, and communication overhead?

4 comments

We appreciate the reviewer’s careful reading and constructive feedback. We hope the clarifications below address your concerns and provide a clearer understanding of our method.

Concern 1: The variational interpretation is not fully supported by the derivation.

R1. We agree that Eq. (19) is not the exact ELBO, though our original wording could suggest otherwise. A more precise description would be “Guided by the ELBO, we formulate a variational objective that unifies language modeling and load balancing,” and the relevant claims have been narrowed accordingly in the revised version.

However, we’d like to clarify that our variational interpretation still holds under the derivation, as Eq. (19) is a principled adaptation of the ELBO, rather than a heuristic objective. As we’ve discussed in Appendix B.3, after decomposing the token-level KL into an aggregate KL and a mutual-information (MI) term, we make two explicit and justified changes. First, we remove the MI term, since it pushes token-wise routing toward the aggregate distribution and suppresses expert specialization, whereas the router needs uniform aggregate routing while preserving token-level specialization. Second, we replace K with a tunable α, since K imposes an overly strong regularization that overwhelms language modeling.

These adaptations are consistent with the well-established practice of tailoring variational objectives to practical needs, e.g., contrastive divergence [1]. More relevantly, β-VAE [2] reweights the KL term with a tunable coefficient, and InfoVAE [3] modifies the MI and aggregate-posterior terms of the ELBO to better suit the desired properties of the latent representation. Since the aggregate KL that yields load balancing is also adapted from ELBO, our variational interpretation is supported by the derivation.

References

[1] Geoffrey E. Hinton. Training Products of Experts by Minimizing Contrastive Divergence. Neural Computation, 14(8), 1771–1800, 2002.

[2] Irina Higgins et al. β-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework. ICLR, 2017.

[3] Shengjia Zhao et al. InfoVAE: Balancing Learning and Inference in Variational Autoencoders. AAAI, 2019.

Concern 2: Distributional stability does not directly establish stability of finite sampled routing decisions or predictions.

R2. We thank the reviewer for the constructive suggestion, which helps us further strengthen the connection between our theory and experiments.

Following the suggestion, we’ve added three experiments in Appendix G Fig. 10, Table 8, and Fig. 8 and summarize the key results in Table I, II below. Specifically, (i) we repeat the K=2 sampling across 3 random seeds to measure routing agreement; (ii) we run repeated inference passes to measure prediction variance; and (iii) we compare the empirical routing distribution with the posterior distribution across tokens of different entropy levels. Obs1: Sampling-induced variation is much smaller than perturbation-induced variation. With K=2, the routing agreement reaches 94.67%, i.e., only 5.33% of the selected experts differ, with a variance of 0.061%. In comparison, under the most severe input perturbation, the expert set changes by up to 80% in our method. Obs2: Sampling introduces negligible fluctuations in predictive performance. At K=2, the prediction std across inference passes stays below 0.17. Hence, K=2 is already sufficient for reliable performance. Obs3: The empirical frequency closely tracks the posterior across tokens of different entropy levels, with variance = 0, 0.15, 0.39, respectively.

Summary. Obs3 shows, the empirical distribution of sampled experts closely matches the posterior distribution, providing empirical support for the relevance of Proposition 3.1 to actual routing behavior. Further evidence from Obs1,2 indicates that sampling variability is is relatively small and has little effect on predictive performance. These findings support our interpretation that the gains arise from improved robustness to input perturbations.

Table I. Average accuracy over 8 benchmarks with stochastic routing at inference over 3 random seeds. Small K is our original setting, where K equals the number of experts activated by deterministic routing. Details are in Appendix G.9 Table 8.

SettingSmall K (original)Medium KLarge K
8 Experts, 1.8B47.7 ± 0.16 (K = 2)47.9 ± 0.12 (K = 4)48.0 ± 0.09 (K = 8)
16 Experts, 3.4B50.3 ± 0.17 (K = 4)50.4 ± 0.13 (K = 8)50.7 ± 0.09 (K = 12)
32 Experts, 6.6B53.1 ± 0.14 (K = 8)53.3 ± 0.13 (K = 12)53.4 ± 0.10 (K = 16)

Table II. Token-level sampling statistics across different entropy levels.

Token entropySupport sizeTop posteriorSampled frequencySampled variance
low10.991.000
medium30.844~0.83~0.15
high40.318~0.31~0.39

Q1: Can Equation 19 be formulated exactly as an ELBO under a different generative model, or should it be described as an ELBO-inspired surrogate?

A1. Thank you for the clarification. Eq. (19) is more accurately understood as an ELBO-inspired surrogate and we have narrowed the corresponding claims in the revised manuscript.

Q2: How much do sampled expert assignments and model predictions vary across repeated inference passes with different Gumbel noise?

A2. Thanks for the question. As we’ve discussed in the response to C2, with details provided in Appendix G. Our experiments show that both the sampled expert assignments and the model predictions are stable across repeated inference passes: (i) with K=2, the expert change rate is only 5.33% with var=0.061**%**; (ii) across multiple inference passes, the standard deviation of predictive performance is less than 0.17. This indicates that our sampling introduces only relatively small variance and has little effect on predictive performance.

Q3: Does StaVER still outperform compute-matched Top-k baselines when computation is based on the actual average number of uniquely activated experts rather than nominal K?

A3. Thanks for the question. Yes, StaVER still outperforms Top-k baselines when compute is measured by the actual number of uniquely activated experts.

First, this is supported by our existing results. In the main table, StaVER with K=2 already outperforms all baselines. As shown in Appendix Figure 7, StaVER with K=4 activates on average 1.91 unique experts per token, close to deterministic routing at this scale, and further improves over K=2 on all four evaluated datasets. This is because drawing more samples reduces the estimation variance of stochastic routing.

As shown in Table I, this trend holds across scales. We evaluate larger K for all model scales, using the same checkpoints as the main table. Increasing K consistently reduces the std and improves accuracy at every scale. Therefore, when StaVER’s actual activations are matched to deterministic routing, it can use a larger K and achieve better performance.

Summary. StaVER already outperforms Top-k baselines in the original setting. Compute matching based on actual activations only allows a larger K, which further strengthens its advantage.

Q4: How do the FLOP estimates and wall-clock results account for expert capacity, duplicate samples, token dispatch, and communication overhead?

A4. Thanks for the question. Our FLOP estimates account for duplicate samples, while system-level overheads such as expert capacity, token dispatch, and communication overhead are not included. We did not report wall-clock time originally, but we have added these results to Table 2 in the revised version.

For duplicate samples, each expert is evaluated only once per token, and its output is weighted by its sampling frequency to avoid redundant computation. We do not model these system-level overheads, as they depend on the implementation and hardware and are orthogonal to our routing algorithm. Following such as StableMoE [5] and X-MoE [6], and the recent ERC loss [7], our evaluation focuses on routing quality and algorithmic computational efficiency, while the wall-clock results in Table 2 further complement these FLOP estimates with a practical runtime view.

References

[5] Damai Dai et al. StableMoE: Stable Routing Strategy for Mixture of Experts. ACL, 2022.

[6] Zewen Chi et al. On the Representation Collapse of Sparse Mixture of Experts. NeurIPS, 2022.

[7] Ang Lv, Jin Ma, Yiyuan Ma, and Siyuan Qiao. Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss. ICLR, 2026.

Minor: Grammar and presentation.

A5. Thank you for pointing these out. We have corrected the issues noted above.

Thank you to the authors for their detailed rebuttal, particularly the clarification of Equation 19 and the additional experimental evidence. To improve readability, I suggest highlighting the changes made in the revised manuscript, for example, by using a different font color.

Most of my concerns have been satisfactorily addressed. I have therefore decided to raise my score to 3: Weak Accept.

Dear Reviewer,

Thank you for your recognition of our work and for raising your score. We really appreciate the time and effort you have devoted to helping us improve the paper!

We’ve uploaded the revised version with all modifications shown in blue font.

Best regards,

The Authors

Reviewer_pLQcOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose to treat the router distribution as a probability distribution where they sample the experts from. Now the expert selection is stochastic rather than decided by top-k operation. They interpret the expert selection as latent variables and derive an ELBO with KL divergence between the batch-averaged routing distribution and a uniform distribution as penalty. Unlike conventional MoE routing, instead of using the probabilities produced by the router as weights to sum the expert outputs, they use the normalized sampled counts.

Critical strengths
  • The experiments show meaningful improvement over other baselines.
  • The training cost in FLOPs shows that the method is cheaper than several baselines, and so is the inference cost.
Critical weaknesses
  • The authors do not separate the instability caused by perturbation from that introduced by sampling. The perturbation can change the expert selection, but so can the sampling they introduce. Sampling with replacement, in addition to using the normalized sampled counts as weights in summation, also introduces variance in expert selection, and therefore the output. Their chosen number of samples is small, too, which may introduce substantial variance.
  • The theory measures stability of the routing distribution, but the empirical results measure stability of sampled expert sets.

Minor: most citations are in the wrong format.

Questions for authors
  • Could authors reveal the sampled distribution over experts given data for individual tokens? If expert selection is highly concentrated to single or few experts, then the low expert-set change rate may come from the concentration of the routing distribution instead of the robustness to the perturbations. What is the variance across samples drawn from the routing distribution?
  • Could authors specify the difference from other work framing MoE routing as probability distribution, aside from using the normalized sample counts as the weights in summing the experts?
  • Could authors provide the end-to-end training cost in wall clock time?
  • Could the authors report the expert selection change rate between two independent forward passes without any perturbation under the proposed method?
5 comments

We thank the reviewer for the detailed feedback and for recognizing the empirical improvements and computational efficiency. We address each of your comments below and hope our clarifications resolve your concerns.

Concern 1: Sampling variability versus perturbation-induced instability.

R1. We thank the reviewer for pointing this out. We address it from three aspects.

First, our comparison already bounds the effect of the two sources. For the deterministic routing baselines, the reported routing change rate comes purely from perturbation. For StaVER, the reported change rate includes both perturbation-induced and sampling-induced changes. Even with both sources combined, StaVER still has a significantly lower change rate than that of deterministic routing, and outperforms the deterministic baselines with perturbation. This indicates that the effect of sampling variance is small.

Second, we would like to clarify an important distinction between the two sources of expert set variation. Perturbation-induced instability is a structural weakness of deterministic routing: a tiny shift in the routing distribution can cause an abrupt change of the selected expert set. Tokens are then routed to under-trained experts, which directly hurts training consistency and inference performance. In contrast, sampling variance is already accounted for in StaVER, as the router is optimized with it during pre-training and finally converge. Thus, the expert sets sampled from it are statistically stable.

To better address this concern, we conduct additional experiments that isolate the sampling variance.

(i) Sampling-only routing change. To isolate the sampling variance, we repeat inference 100 times on the same unperturbed input and compare the routing with the original expert set. As shown in Appendix G.8, Fig. 10, our sampling method changes only 5.33% of the expert set on average, with a variance of 0.061. This is much smaller than the perturbation-induced change rate which reaches up to 80%, indicating that even K = 2 samples introduce only small expert change rate.

(ii) Effect of sampling variance on predictive performance. We run repeated inference passes with different random seeds to measure the prediction variance across different selection of K, and summarize the result in Table I below, with details in Appendix G.9, Table 8. The result suggests that, our sampling routing introduces negligible fluctuations in predictive performance. At K=2, the prediction std across inference passes stays below 0.17, and this can be further reduced by increasing the sampling numbers.

Summary. Even with K = 2 samples, sampling variance is small and has negligible impact on performance, so our conclusions still hold. We will add this analysis to the revised version.

Table I. Average accuracy over 8 benchmarks with stochastic routing at inference over 3 random seeds. Small K is our original setting, where K equals the number of experts activated by deterministic routing. Details are in Appendix G.9.

SettingSmall K (original)Medium KLarge K
8 Experts, 1.8B47.7 ± 0.16 (K = 2)47.9 ± 0.12 (K = 4)48.0 ± 0.09 (K = 8)
16 Experts, 3.4B50.3 ± 0.17 (K = 4)50.4 ± 0.13 (K = 8)50.7 ± 0.09 (K = 12)
32 Experts, 6.6B53.1 ± 0.14 (K = 8)53.3 ± 0.13 (K = 12)53.4 ± 0.10 (K = 16)

Concern 2: The theory measures routing-distribution stability, whereas the experiments measure sampled expert-set stability.

R2. Thanks for raising this point. While the theory and experiments examine routing at different levels, both aim to understand how perturbations affect routing and performance, and the two levels are tightly connected.

(i) In the experiments, changes in sampled expert sets provide a directly observable measure of routing changes. In Fig. 1 we report the overall expert-set change rate, which captures the combined effects of perturbations and sampling. Even with sampling variability included, our method exhibits fewer expert changes and less performance degradation under perturbation, supporting its robustness to input perturbations.

(ii) The routing distribution is the cleaner object for theoretical analysis. Sampled expert set is an unbiased estimation of this distribution, but is subject to sampling variance. Analyzing the distribution therefore lets us isolate the effect of perturbations on routing from sampling variance.

(iii) The additional experiment suggested by the reviewer provides further evidence for this connection. The result shows that sampling variability is relatively small and has little effect on predictive performance (see our response to C1).

Summary. These findings suggest that the expert changes observed under perturbation mainly reflect the distribution shifts described by our theory, and that the experimental gains come from improved robustness to input perturbations.

Q1: Could the authors show token-level expert-selection distributions and sampling variance? Could routing concentration explain the low expert-set change rate?

A1. Thanks for the question. We’d like to clarify that our method samples experts directly from the router posterior, so the sampled frequencies are an unbiased estimate of the posterior. In other words, sampling doesn’t introduce any extra concentration.

Following the suggestion, we’ve added a token-level analysis in Appendix G.6 Figure 8. For each token, we repeat the K=2 sampling 32 times on 3 tokens selected from different entropy levels, and summarize the key information in Table II for the reviewer’s convenience. Obs1: Even with K=2, the empirical frequency closely tracks the posterior distribution. Tokens with higher posterior entropy are assigned to more distinct experts across repeated samples. Obs2: The sampling variance stays small at K=2, at 0, 0.15 and 0.39, respectively. While low-entropy tokens are highly concentrated, the expert support size grows as entropy increases. Hence, ****the routing is not the cause of the low change rate.

Summary. Even at K=2, the sampled distribution closely matches the posterior with low variance. Routing concentration is limited to low-entropy tokens and does not explain the low expert-set change rate.

Table II. Token-level sampling statistics across different entropy levels.

Token id (entropy)Support sizeTop posteriorSampled frequencySampled variance
4400 (low)10.9901.0000
7565 (medium)30.844~0.83~0.15
118 (high)40.318~0.31~0.39

Q2: What distinguishes StaVER from prior probabilistic MoE routing methods, beyond using normalized sample counts as aggregation weights?

A2. Thanks for the question. Beyond the aggregation weights, StaVER differs from prior methods at the level of problem formulation, as we’ve discussed in Related Work. Prior methods identify problems in deterministic routing and propose heuristic mitigations. In contrast, StaVER is grounded in Bayesian inference. Specifically, we formulate routing as a learnable posterior. From this formulation, we derive an optimization framework that unifies language modeling and load balancing within one variational objective. The detailed comparison is provided in Table III below.

Table III. Comparison of probabilistic routing methods. Stage denotes each method’s intended training stage. Rationale for Probabilistic Routing indicates whether each method provides a theoretical justification for probabilistic routing. “—” denotes no theoretical analysis.

StageMethodRationale for Probabilistic RoutingMethodological details
Post-trainingProbMoE [1]No.Samples a subset of K experts and combines their outputs using the corresponding softmax routing weights.
VMoER [2]No.Uses an additional neural network to learn a posterior over routing logits during post-training. At inference, it averages the routing probabilities obtained from multiple posterior samples, then applies Top-k selection.
Pre-trainingNoisy Top-k [3]—Adds noise to routing logits before Top-k selection.
DirMoE [4]No.Selects experts and assigns their aggregation weights in two separate steps. Specifically, for every expert, they sample from a Bernoulli distribution to decide whether this expert is activated. Then separately decide the aggregation weight of the selected expert by sampling from a Dirichlet distribution.
OursYes.Formulates the routing distribution as a learnable posterior. We sample experts with replacement from this posterior and use normalized sample counts as aggregation weights to obtain an unbiased estimate of the marginalized MoE output.

References

[1] Heng Zhao et al. ProbMoE: Differentiable Probabilistic Routing for Mixture-of-Experts. ICML, 2026.

[2] Albus Li and Matthew Robert Wicker. Variational Routing: A Scalable Bayesian Framework for Calibrated Mixture-of-Experts Transformers. ICML, 2026.

[3] Noam Shazeer et al. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. ICLR, 2017.

[4] Amirhossein Vahidi et al. DirMoE: Dirichlet-Routed Mixture of Experts. ICLR, 2026.

Q3: Could the authors report end-to-end training cost in wall-clock time?

A3. Thanks for the question. We have added end-to-end training wall-clock time to Table 2 of the revised manuscript, summarized below.

Table IV. End-to-end training wall-clock time and FLOPs.

MetricGShardSwitch TransformerStableMoEDeepSeekMoEAuxiliary-FreeMoEOurs
Wall-clock time (h) ↓92.983.388.491.790.784.0
Training FLOPs (×10¹⁹) ↓9.576.626.629.579.577.80

Summary. StaVER achieves the lowest test perplexity with the second-shortest training time. This demonstrates that combining Bayesian sampling with routing improves both training efficiency and model performance.

Q4: What is the expert-selection change rate between two independent forward passes without perturbation?

A4. Thanks for the question. We added an experiment showing that the expert-selection change rate induced by sampling is only 5.5% on average, with a variance of 0.06%. As discussed in our response to C1, this was measured by running 100 independent forward passes without any perturbation and comparing the experts selected in each pass against those of the original forward pass. The results are reported in Appendix G.8 Fig. 10.

In addition, the change rates in our original perturbation experiments discussed in Sec 4.3 already account for both data perturbations and sampling variance. Even so, they remain much lower than those of deterministic routing under the same perturbations, and our method consistently outperforms all the baseline. Therefore, the changes introduced by sampling are small and have a minor effect on performance, especially compared with the effect of perturbation.

Minor: Citation formatting.

A5. Thank you for pointing this out. We have corrected the citation format for all parenthetical citations throughout the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.