- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates expert routing as an additional source of exploration during reinforcement learning for MoE language models. ESRL preserves high-confidence experts, samples alternative experts from a restricted candidate pool using entropy-adaptive noise, and replays the sampled paths during policy updates. Expert mixture weights are computed from unperturbed router logits, and routing perturbations are disabled at evaluation. Experiments across three MoE model families and mathematical, scientific, and coding benchmarks report performance improvements, with ablations examining the main exploration mechanisms.
Critical strengths
- The method has a clear motivation and provides a practical way to explore alternative expert paths while controlling the loss of rollout quality.
- Experiments across different routing architectures and task domains, together with component ablations, provide useful evidence for the approach’s empirical benefits.
Critical weaknesses
-
The contribution of ESRL could be further clarified through comparison with MoE-ERL [1], which is not discussed in the paper. That work already explores perturbed expert routes and uses reinforcement learning feedback to improve routing. Although ESRL differs in its rollout setting and exploration controls, explaining these distinctions would help establish its advance beyond the broader idea of expert-path exploration.
-
The adaptation of MoE-GRPO [2] is insufficiently specified to interpret this comparison. The original method combines token and gate objectives with modality-aware routing guidance, whereas the baseline description here emphasizes categorical expert sampling. It remains unclear which components were retained or modified for language-only models. Given the reported collapse, these details matter for distinguishing limitations of the original approach from those of its adaptation.
[1] Kim and Kang. Exploration-Driven Reinforcement Learning for Expert Routing Improvement in Mixture-of-Experts Language Models. 2025.
[2] Ko et al. MoE-GRPO: Optimizing Mixture-of-Experts via Reinforcement Learning in Vision-Language Models. 2026.
Questions for authors
-
Could you discuss the relationship to MoE-ERL [1], particularly the difference between its feedback based on correct-token probabilities and ESRL’s task-reward-driven rollouts? Explaining what the anchored sampling and entropy-adaptive perturbation contribute beyond prior route exploration would make the methodological distinction clearer. A conceptual comparison of the objectives, exploration mechanisms, and intended capabilities would suffice; an additional experimental comparison is not necessary for this clarification.
-
Could you specify how MoE-GRPO [2] was adapted, including whether both Token-GRPO and Gate-GRPO were used, the auxiliary load-balancing configuration, and how modality-aware candidate restrictions were handled in the language-only setting? These details would help assess whether the collapse concerns the full adapted method or a sampling-only variant. If the implementation is a simplified variant, identifying it explicitly and qualifying the comparison would clarify the evidence.
2 comments
The authors have posted their responses to your comments. Could you please take a look and let us know whether your concerns have been addressed? We encourage active discussion between reviewers and authors during this period.
Response for Reviewer CoCx
We thank the reviewer for pointing out MoE-ERL. Both ESRL and MoE-ERL explore alternative expert routes, but they use different feedback and optimize for different goals. MoE-ERL perturbs the routes with random noise, and compares the changes in the correct token probability to train the router. Since the reference sequence and multiple additional routes are produced, it requires additional forward passes and limits the usage to existing reference. In comparison, ESRL explores alternative routes during rollouts without additional rollout budgets. The routing changes can produce different responses, and sequence-level task rewards will guide the policy update without reference sequence.
ESRL is designed to preserve rollout quality while exploring alternative routes. In a long rollout sequence, a poorly matched expert can change the next token distribution and affect the context for subsequent decisions. Therefore, we keep high-confidence experts as anchors, sample the remaining experts from a router-ranked candidate pool, and adapt perturbation strength to the router’s entropy. In our ablation, anchoring preserves higher accuracy at comparable rollout diversity, supporting its role in limiting the quality cost of exploration.
We have incorporate the discussion of MoE-ERL into the related works.
We thank the reviewer for raising this concern. MoE-GRPO explores expert assignments through stochastic routing and jointly optimizes Token-GRPO and Gate-GRPO using group-relative task rewards. Token-GRPO optimizes generated-token probabilities, while Gate-GRPO directly optimizes the probabilities of sampled expert selections. In its original vision-language setting, it also restricts candidate experts according to modality-specific activation statistics and applies an auxiliary load-balancing loss.
Since our training is text-only, we adapt MoE-GRPO to this setting and retain both of its optimization objectives, i.e., Token-GRPO and Gate-GRPO. Both objectives use the same group-relative advantage derived from the task reward. During rollout, we sample experts from the router distribution over the full expert pool, since our inputs contain only text and the original vision–language mask does not apply. The auxiliary load-balancing coefficient is set to zero and we report the effect of adding this loss separately in the Aux-Loss baseline. We have specified these details in the revised version.