acceptodds
Under review as a conference paper at ICLR 2027

Expert-Space Exploration in MoE Reinforcement Learning

Abstract

Reinforcement learning (RL) has become central to post-training of large language models. Recent advances in RL for Mixture-of-Experts (MoE) models have primarily focused on improving optimization stability and training efficiency. Since routing determines the sparse computation paths that induce output distributions, expert selection offers an additional source of rollout diversity beyond conventional token-level sampling. Through empirical analysis, we find that routing perturbation changes model outputs and increases rollout diversity, whose effect resembles that of increasing the decoding temperature. However, direct perturbation can activate unsuitable experts and substantially degrade rollout quality. Motivated by these observations, we introduce **E**xpert-**S**pace Exploration **R**einforcement **L**earning (**ESRL**), an architecture-aware framework that explicitly explores the expert-routing space of MoE models. ESRL preserves high-confidence experts as anchors, and restricts stochastic routing to a plausible candidate pool, thereby retaining reliable computation paths. The perturbation strength is further adapted according to router entropy to avoid over-perturbation. To mitigate the routing mismatch introduced by perturbation, ESRL records the expert paths used during rollout and replays them during policy optimization. Experiments demonstrate that ESRL achieves the best performance across MoE backbones with top-, top-1, and shared-expert routing on mathematics, science, and code tasks, without increasing the number of rollout samples or activated experts per token. Specifically, ESRL on Qwen3-30B-A3B improves average Pass@1 and Pass@8 over GRPO by **3.2** and **4.5** percentage points, respectively. Further analyses of expert utilization and training dynamics provide insights into how exploiting MoE-specific routing structure benefits RL training. These results establish expert routing as an effective and complementary dimension for RL exploration beyond its architectural role.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement learning, Mixture-of-Experts, routing noise, rollout diversity

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CoCxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates expert routing as an additional source of exploration during reinforcement learning for MoE language models. ESRL preserves high-confidence experts, samples alternative experts from a restricted candidate pool using entropy-adaptive noise, and replays the sampled paths during policy updates. Expert mixture weights are computed from unperturbed router logits, and routing perturbations are disabled at evaluation. Experiments across three MoE model families and mathematical, scientific, and coding benchmarks report performance improvements, with ablations examining the main exploration mechanisms.

Critical strengths
  1. The method has a clear motivation and provides a practical way to explore alternative expert paths while controlling the loss of rollout quality.
  2. Experiments across different routing architectures and task domains, together with component ablations, provide useful evidence for the approach’s empirical benefits.
Critical weaknesses
  1. The contribution of ESRL could be further clarified through comparison with MoE-ERL [1], which is not discussed in the paper. That work already explores perturbed expert routes and uses reinforcement learning feedback to improve routing. Although ESRL differs in its rollout setting and exploration controls, explaining these distinctions would help establish its advance beyond the broader idea of expert-path exploration.

  2. The adaptation of MoE-GRPO [2] is insufficiently specified to interpret this comparison. The original method combines token and gate objectives with modality-aware routing guidance, whereas the baseline description here emphasizes categorical expert sampling. It remains unclear which components were retained or modified for language-only models. Given the reported collapse, these details matter for distinguishing limitations of the original approach from those of its adaptation.

[1] Kim and Kang. Exploration-Driven Reinforcement Learning for Expert Routing Improvement in Mixture-of-Experts Language Models. 2025.
[2] Ko et al. MoE-GRPO: Optimizing Mixture-of-Experts via Reinforcement Learning in Vision-Language Models. 2026.

Questions for authors
  1. Could you discuss the relationship to MoE-ERL [1], particularly the difference between its feedback based on correct-token probabilities and ESRL’s task-reward-driven rollouts? Explaining what the anchored sampling and entropy-adaptive perturbation contribute beyond prior route exploration would make the methodological distinction clearer. A conceptual comparison of the objectives, exploration mechanisms, and intended capabilities would suffice; an additional experimental comparison is not necessary for this clarification.

  2. Could you specify how MoE-GRPO [2] was adapted, including whether both Token-GRPO and Gate-GRPO were used, the auxiliary load-balancing configuration, and how modality-aware candidate restrictions were handled in the language-only setting? These details would help assess whether the collapse concerns the full adapted method or a sampling-only variant. If the implementation is a simplified variant, identifying it explicitly and qualifying the comparison would clarify the evidence.

2 comments
AuthorsOpenReview ↗

Response for Reviewer CoCx

W1+Q1: Relationship compared to MoE-ERL

We thank the reviewer for pointing out MoE-ERL. Both ESRL and MoE-ERL explore alternative expert routes, but they use different feedback and optimize for different goals. MoE-ERL perturbs the routes with random noise, and compares the changes in the correct token probability to train the router. Since the reference sequence and multiple additional routes are produced, it requires additional forward passes and limits the usage to existing reference. In comparison, ESRL explores alternative routes during rollouts without additional rollout budgets. The routing changes can produce different responses, and sequence-level task rewards will guide the policy update without reference sequence.

ESRL is designed to preserve rollout quality while exploring alternative routes. In a long rollout sequence, a poorly matched expert can change the next token distribution and affect the context for subsequent decisions. Therefore, we keep high-confidence experts as anchors, sample the remaining experts from a router-ranked candidate pool, and adapt perturbation strength to the router’s entropy. In our ablation, anchoring preserves higher accuracy at comparable rollout diversity, supporting its role in limiting the quality cost of exploration.

We have incorporate the discussion of MoE-ERL into the related works.

W2+Q2: Details about the implementation of MoE-GRPO

We thank the reviewer for raising this concern. MoE-GRPO explores expert assignments through stochastic routing and jointly optimizes Token-GRPO and Gate-GRPO using group-relative task rewards. Token-GRPO optimizes generated-token probabilities, while Gate-GRPO directly optimizes the probabilities of sampled expert selections. In its original vision-language setting, it also restricts candidate experts according to modality-specific activation statistics and applies an auxiliary load-balancing loss.

Since our training is text-only, we adapt MoE-GRPO to this setting and retain both of its optimization objectives, i.e., Token-GRPO and Gate-GRPO. Both objectives use the same group-relative advantage derived from the task reward. During rollout, we sample experts from the router distribution over the full expert pool, since our inputs contain only text and the original vision–language mask does not apply. The auxiliary load-balancing coefficient is set to zero and we report the effect of adding this loss separately in the Aux-Loss baseline. We have specified these details in the revised version.

The authors have posted their responses to your comments. Could you please take a look and let us know whether your concerns have been addressed? We encourage active discussion between reviewers and authors during this period.

Reviewer_JmibOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ESRL, which explores alternative expert-routing paths during MoE reinforcement learning through anchored expert sampling, entropy-adaptive noise, and routing replay. Experiments across three MoE models demonstrate improvements over GRPO, including gains of 3.2 and 4.5 percentage points in Pass@1 and Pass@8 on Qwen3-30B-A3B.

Critical strengths
  • Expert routing provides an interesting additional dimension for exploration beyond token sampling.

  • The evaluation covers multiple MoE architectures, with extensive baseline comparisons and informative ablation studies.

Critical weaknesses
  • The technical novelty appears somewhat limited, as ESRL primarily combines relatively simple routing perturbations with existing R3.

  • The mechanisms underlying the improvements could be better explained. Although the paper demonstrates increased rollout diversity and improved expert utilization, their respective contributions to the performance gains remain unclear.

Questions for authors
  • What is the main advantage of anchored sampling over simpler stochastic routing approaches?

  • Could the authors further discuss how improved expert utilization contributes to ESRL's performance gains?

4 comments
AuthorsOpenReview ↗

Response for Reviewer Jmib

W1+Q1: Contribution and advantage of anchored sampling

We thank the reviewer for asking what ESRL adds to stochastic routing and existing routing replay. Our contribution is to characterize how expert-route perturbations affect generation and use these findings to design controlled expert-space exploration for autoregressive RL. Turning alternative expert paths into useful exploration poses two challenges. First, replacing well-matched experts can degrade rollout quality. ESRL therefore retains high-confidence experts as anchors, samples the remaining experts from a router-ranked candidate pool, and adapts the noise scale to router entropy. Second, recomputing routes during optimization can bypass the alternative expert paths explored during rollout. ESRL therefore records and replays those paths so that the experts that generated each response participate in its policy update. Our contribution also includes a systematic analysis of router perturbations, with component ablations and studies of how noise-injection locations and perturbation methods affect the accuracy–diversity trade-off.

The main advantage of anchored sampling over fully stochastic routing is that it provides a trade-off between the generation quality and diversity. It preserves some of the original computation path while still allowing alternative experts to be sampled. In our accuracy–diversity analysis (Figure 7), configurations with anchors achieve higher accuracy at comparable Self-BLEU. The training ablation (Figure 6) also shows that removing anchored experts makes performance more sensitive to the noise scale. We further ablate entropy-adaptive scaling and vary the layers where noise is applied to characterize how perturbation strength and placement affect exploration. Together with the gains across MoE architectures and task domains, these results provide an empirical basis for designing controlled expert-space exploration in RL for MoE LLMs.

W2+Q2: Mechanism behind the performance gain

We thank the reviewer for asking how broader expert utilization contributes to ESRL’s gains. We believe that broader expert utilization is one step in how ESRL improves exploration. Sampling alternative expert paths can produce different responses to the same prompt. When those responses differ in rewards, the resulting reward variation gives GRPO a signal to distinguish successful from unsuccessful trajectories. Figure 5 supports this explanation. At matched rollout group sizes, ESRL maintains a higher fraction of groups containing both correct and incorrect responses than GRPO-R3, particularly later in training.

In the revised manuscript (Appendix G, Page 22), we also report raw reward, token-level policy entropy, response length, and gradient norm. ESRL shows higher entropy through much of training, although the entropy curves largely converge toward the end. ESRL achieves competitive rewards with shorter responses late in training and gradient norms of comparable magnitude. Taken together, broader expert utilization and higher entropy describe how ESRL expands exploration, while sustained within-group reward variation provides a more direct explanation for how that exploration can improve learning.

The authors have posted their responses to your comments. Could you please take a look and let us know whether your concerns have been addressed? We encourage active discussion between reviewers and authors during this period.

Thank you for the detailed rebuttal and additional analyses.

The new results clarify the benefits of anchored sampling and address some of my concerns.

However, I still find the methodological novelty limited, as the approach mainly combines existing routing perturbation techniques. While the empirical improvements are promising, it remains unclear what unique benefits expert-space exploration provides beyond increasing rollout diversity.

Therefore, I will maintain my current rating.

Thank you for your feedback and for acknowledging our additional analyses.

We respectfully believe that our contribution extends beyond combining routing perturbation techniques. Our work extends RL exploration from sampling output tokens to sampling the model’s internal expert computation paths. We demonstrate that stochastic expert selection alone can support LLM reinforcement learning under deterministic token decoding, establishing internal computation paths as an alternative exploration space. Routing replay serves a specific purpose within this framework. Once an alternative path generates a response, optimization must preserve the connection between that path and its reward feedback. Otherwise, recomputed routing may bypass the experts responsible for the response, weakening the link between exploration and learning. ESRL thus forms a coherent exploration–feedback–update process. Controlled perturbation explores alternative paths while preserving rollout quality, task rewards evaluate the resulting responses, and replay ensures that the explored paths participate in policy optimization. This connection between internal-path exploration and reward-driven learning is central to our methodological contribution.

Furthermore, our analyses show that routing configurations with comparable diversity can yield different generation quality. Anchoring, perturbation strength, candidate selection, and layer placement therefore matter beyond simply increasing diversity. ESRL translates these findings into controlled exploration that improves performance without increasing rollout samples or activated experts per token. We believe that demonstrating this capability, characterizing the diversity–quality trade-off, and providing systematic experimental guidance constitute a substantive contribution and a foundation for further research on expert-space exploration in MoE reinforcement learning.

Reviewer_keyFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ESRL, which adds noise to expert routing during MoE reinforcement-learning rollouts. It combines high-confidence expert retention, a restricted candidate pool, and adaptive noise to explore different paths. Sampled expert combinations are replayed during optimization, while standard evaluation disables routing noise. Experiments cover mathematical training across several MoE architectures, science and code training, and an instruct-model initialization, with overall task gains.

Critical strengths
  1. The task results are promising and cover more than one MoE architecture. Tables 1–3 include mathematical benchmarks, science and code training, and a strong instruct-model initialization. Standard evaluation disables routing noise, so these results reflect post-training gains rather than the immediate effect of adding noise at evaluation time.

  2. The main design choices are supported by ablations. Table 4 compares adaptive and fixed noise at matched expert-change rates. Figures 5–7 also examine temperature, sample counts, anchors, and perturbation locations. These controls provide useful evidence for the design choices.

Critical weaknesses

The paper does not check whether routing becomes worse in normal inference after training. ESRL repeatedly sends tokens through experts that the router would not normally select and updates the model along these paths, which may change the router's behavior, for example by concentrating the load on a few experts. Figure 2 measures expert loads while noise is applied, and Figure 9 records loads during training rollouts, so both include the effect of the noise itself and do not reflect routing with noise disabled. The authors should compare the layer-wise expert loads of the final ESRL and GRPO-R3 models on the same inputs with noise disabled, to show that ESRL does not noticeably degrade routing.

The paper does not provide the training curves needed to explain where the gains come from. The improvement of ESRL could come from more diverse generation, longer responses or a change in gradient magnitude, but the paper does not compare generated-token entropy, response length and gradient norm between ESRL and the main baselines during training. The existing Self-BLEU, informative-group and clipping-fraction analyses do not replace these curves, and the router entropy used to scale the noise is not the entropy of generated tokens. The authors should report these metrics together with the reward curve on the same training-step axis.

Perturbing the routing during training may damage precise internal computation, such as mathematical calculation, and the paper does not test this. Some experts in an MoE model may carry functions that must be executed precisely, such as arithmetic or symbolic manipulation, and ESRL sends tokens through experts that the router would not normally select during rollout and then updates the model along these expert paths, which could weaken such structures. The paper itself acknowledges that perturbation can activate experts poorly matched to the current hidden state, and keeps high-confidence experts as anchors to "retain reliable computation paths", but it does not test whether these abilities are preserved after training. Higher scores on complete math problems do not show that basic calculation has not degraded, since final-answer accuracy may also come from better reasoning strategies. The authors should compare the initialization, GRPO-R3 and ESRL models with noise disabled on multi-digit arithmetic and exact symbolic manipulation.

Questions for authors
  1. Please compare the layer-wise expert loads of the final GRPO-R3 and ESRL models on the same inputs with routing noise disabled, for example the load CV and the ratio of maximum to mean load, to show that ESRL does not noticeably degrade routing.

  2. Please report generated-token entropy, response length and gradient norm over training for ESRL, GRPO-R3 and GSPO on the same training-step axis, together with the reward curve.

  3. Does the routing perturbation during training damage precise internal computation, such as mathematical calculation? Please compare the initialization, GRPO-R3 and ESRL models with routing noise disabled on basic arithmetic, such as multi-digit addition, subtraction, multiplication and division, and on exact symbolic manipulation.

Update: Thank you for the author's rebuttal. I have updated my score.

2 comments
AuthorsOpenReview ↗

Response for Reviewer keyF

Thanks for the reviewer for the suggestions. We have provided a point-to-point response as follows. Some of the figures can be accessed through the link https://anonymous.4open.science/r/rebuttal_reviewer_keyF-F891/, and we also revised the manuscript the content.

W1+Q1: Final expert load statistics at evaluation stage.

We thank the reviewer for raising the concern. We evaluated ESRL and GRPO-R3 on the same prompts from four math benchmarks (i.e., which we used in Table 1) using standard Top-KK routing with routing noise disabled. The results are provided in the link, and in the revised manuscript (Appendix F, page 21). We can observe that ESRL has lower expert-load CV and a lower maximum-to-mean load ratio in all 48 MoE layers. Averaged across layers, the CV decreases from 1.11 to 1.06, and the maximum-to-mean ratio decreases from 5.35 to 4.67. These results show that ESRL does not introduce additional expert-load concentration under standard inference on the evaluated benchmarks. Expert load is less concentrated than with GRPO-R3 by both measures.

W2+Q2: Training dynamics.

We thank the reviewer for raising this concern. We report token-level policy entropy, response length, raw reward, and gradient norm for ESRL, GRPO-R3, and GSPO. These training dynamics can be found in the above link and in the revised manuscript (Appendix F, page 22). ESRL shows higher token-level entropy during training, although the curves largely converge toward the end. Figure 5 (in the manuscript) also shows a complementary pattern, where ESRL maintains a higher fraction of groups containing both correct and incorrect responses than GRPO-R3 with a clearer gap later in the runs. These mixed-outcome groups provide nonzero group-relative advantages, suggesting that expert-path exploration helps sustain a useful learning signal as the policy becomes more concentrated.

Regarding rewards, response lengths and gradnorm, ESRL achieves competitive raw rewards while generating shorter responses than both baselines late in training. Its gradient norms settle to the same order of magnitude as the baselines. Together, these curves are consistent with stable optimization and show that longer responses are not required to achieve the observed rewards.

W3+Q3: Basic capability of the trained model

We evaluated the initialization and the two math-trained models on MMLU-Pro and MMLU-Redux, which spans quantitative reasoning, domain knowledge, and logical reasoning. On MMLU-Pro, the initialization, GRPO-R3, and ESRL achieved 47.2%, 38.9%, and 40.5% Pass@1, respectively. On MMLU-Redux, the corresponding scores were 66.3%, 54.3%, and 55.2%. Both models scored lower after RL, but the drop was smaller with ESRL. This suggests that ESRL better preserves performance across a broader range of tasks than GRPO-R3 under the same math-only training setting.

ModelMMLU-Pro Pass@1 (%)MMLU-Redux Pass@1 (%)
Initialization47.266.3
GRPO-R338.954.3
ESRL40.555.2

The authors have posted their responses to your comments. Could you please take a look and let us know whether your concerns have been addressed? We encourage active discussion between reviewers and authors during this period.

Comments on the paper

2 comments

The authors have posted their rebuttal, and the discussion period will end soon. Could you please read it and reply at your earliest convenience, stating whether your concerns have been addressed and whether your rating changes? Thank you so much for your active participation.

AuthorsOpenReview ↗

Discussion Summary

Dear Area Chair and Reviewers,

We sincerely thank you for your time and constructive feedback. During the discussion, we added experimental evidence and clarified the contribution and implementation of ESRL. As the rebuttal phase is approaching the end, we would like to summarize the main revisions and response as follows.

  • Additional routing and training diagnostics (R1.1, R1.2). With routing noise disabled, ESRL shows less concentrated expert loads than GRPO-R3 across all 48 MoE layers. We also added entropy, response-length, reward, and gradient-norm curves, providing further evidence of stable training.
  • Basic and Broader capability evaluation (R1.3). We added results on MMLU-Pro and MMLU-Redux after RL training on Math task. Although math-only RL reduces performance relative to initialization, ESRL shows smaller declines than GRPO-R3 on both benchmarks, supporting better retention of general capabilities.
  • Clearer contribution and mechanism analysis (R2.1, R2.2). We explained how ESRL’s design addresses the quality and optimization challenges of routing perturbation, supported by our ablations and perturbation analyses. We also connected the training curves with existing Figure 5, highlighting sustained within-group reward variation as a mechanism through which exploration can improve learning.
  • Clarified prior-work comparisons (R3.1, R3.2). We expanded the discussion of MoE-ERL to distinguish reference-based router feedback from ESRL’s task-reward-driven rollouts. We also specified the retained Token-GRPO and Gate-GRPO objectives, modality-mask adaptation, and load-balancing configuration of our MoE-GRPO baseline.

These additions strengthen the evidence for controlled expert-space exploration as a useful complement to token sampling in MoE reinforcement learning. We appreciate the reviewers’ suggestions and welcome further feedback on the remaining concerns.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.