acceptodds
Under review as a conference paper at ICLR 2027

Reinforcing Internal Computations for Large Language Model Reasoning

13 views

Abstract

Mixture-of-Experts (MoE) models can solve a problem through many different combinations of experts, yet standard training largely treats expert selection as fixed internal computation. We ask whether reasoning can improve by learning not only what the model outputs, but also how it routes computation internally. We introduce a reinforcement learning approach that jointly optimizes output tokens and expert selection, allowing the model to explore alternative computation paths and reinforce those that lead to better solutions. On Moonlight-16B-A3B, our method improves over token-only GRPO by 2.06 percentage points on mathematical reasoning, 7.49 points on out-of-domain academic reasoning, and 3.83 points on out-of-domain commonsense reasoning. It also reduces average response length by 8.3% under the same training budget, while preserving the standard MoE inference cost. Our results suggest that learning how a model computes can complement learning what it generates.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Reinforement Learning, Post-training, Mixture-of-Experts

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_P3oHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RLRP, which jointly optimizes the output policy and routing policy of MoE models using response-level rewards. During training, it samples experts to explore different computation paths. It adds a routing loss to GRPO to increase the probability of selecting expert paths that lead to better answers. The authors train Moonlight-16B-A3B on mathematical reasoning data and evaluate it on seven mathematical, academic, and commonsense reasoning benchmarks. RLRP improves average accuracy from 47.10% with GRPO to 51.22% and reduces the mean response length. The experiments also examine the effects of expert sampling alone, different routing clipping rules, and expert sampling at inference time.

Critical strengths
  1. The ablation supports the effectiveness of the added routing loss. With all other settings kept the same, adding the routing loss improves average accuracy from 47.11% to 51.22%.

  2. This also shows that expert exploration alone does not achieve the same gains. Training the routing policy with response-level rewards is helpful. This comparison shows the benefit of the added routing loss.

Critical weaknesses
  1. The comparison with the closest method is not convincing.MoE-GRPO already uses outcome rewards to jointly optimize token and expert selection policies. The main changes in this paper concern routing probability modeling, the loss, and clipping. However, this baseline suffers from training collapse in the experiments. Its best checkpoint achieves an average accuracy of only 10.45%. The paper provides insufficient detail about the collapse and the relevant training settings. This makes it difficult to assess how much the proposed changes improve upon the existing method.

  2. The ablation shows that RLRP improves upon GRPO with expert sampling. However, it remains unclear whether the proposed changes offer an improvement over MoE-GRPO. I therefore lean toward weak rejection.

Questions for authors
  1. How was MoE-GRPO adapted to Moonlight, and were its training settings adjusted? RLRP and MoE-GRPO are similar in several respects. Why did MoE-GRPO suffer training collapse in these experiments?

  2. What advantages does the proposed routing loss have over the MoE-GRPO routing loss? Do the authors have experiments or analyses that separate the contributions of the routing loss and token sampling?

Reviewer_HLHyOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RLRP, a method that jointly optimizes token generation and expert routing in MoE language models. RLRP explores alternative expert selections through stochastic K-expert sampling and optimizes the routing policy using advantage-weighted log-likelihood with a shared token-level clipping mask. Experiments on Moonlight-16B-A3B demonstrate improvements over standard GRPO across seven reasoning benchmarks, including better out-of-domain performance and shorter responses, while maintaining the original active-expert budget.

Critical strengths
  1. Promising Empirical Results. RLRP consistently outperforms standard GRPO across all seven evaluated benchmarks, with improvements in both in-domain and out-of-domain reasoning tasks. The gains are achieved while retaining deterministic Top-K routing at inference.

  2. Useful Ablation Studies. The paper includes controlled comparisons between GRPO, GRPO with expert sampling, and RLRP, helping distinguish the effects of stochastic exploration and explicit routing optimization. The clipping ablation also provides evidence supporting the proposed optimization strategy.

Critical weaknesses

1. Limited Novelty

The core idea of exploring and explicitly optimizing MoE routing policies through reinforcement learning has already been investigated in prior work, such as ERL [1], MoE-GRPO [2], and ESRL [3]. Several key findings, including the benefits of expert exploration and reward-guided routing optimization, have been demonstrated previously. The paper does not sufficiently establish its novel contributions beyond these existing approaches.

2. Insufficient Comparison with Prior Methods

Although MoE-GRPO is included as a baseline, its training suffers from severe collapse, resulting in an average accuracy of only 10.45%, compared to 47.10% for standard GRPO. This makes the comparison inconclusive. Furthermore, ESRL, which has already evaluated expert-space exploration on Moonlight-16B-A3B, is not included.

3. Insufficient Ablation of Key Methodological Designs

The paper introduces an exact ordered-action likelihood and a shared token-level clipping mask. However, the advantages of the ordered-action likelihood over alternative routing probability formulations are not independently evaluated. Moreover, the expert-ratio clipping baseline uses clean router probabilities rather than the actual ordered-action probability ratios, making it difficult to isolate the benefit of the proposed clipping strategy.

[1] Exploration-Driven Reinforcement Learning for Expert Routing Improvement in Mixture-of-Experts Language Models, Findings of EMNLP 2025.

[2] MoE-GRPO: Optimizing Mixture-of-Experts via Reinforcement Learning in Vision-Language Models, CVPR 2026.

[3] Expert-Space Exploration in MoE Reinforcement Learning, arXiv 2026.

Questions for authors
  1. What are the main methodological contributions of RLRP compared with ERL, MoE-GRPO, and ESRL? Which findings have not already been demonstrated by these approaches?

  2. Why does MoE-GRPO experience such severe training collapse? Could the authors provide a more stable and fairly tuned comparison, and include ESRL under comparable experimental settings?

  3. Could the authors compare the exact ordered-action likelihood with simpler routing objectives or probability approximations? Additionally, how does the shared token clipping mask compare with clipping based on the actual ordered-action probability ratio?

  4. Can the authors validate RLRP on additional MoE architectures and larger evaluation sets, and report results across multiple training seeds with uncertainty estimates?

  5. Can the authors provide more direct evidence that the learned routing decisions contribute to improved reasoning, for example through controlled expert-routing interventions or counterfactual comparisons?

Reviewer_1pp7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RLRP augments token-level GRPO with reward-weighted optimization of MoE routing actions. It samples experts without replacement, replays their ordered selections, and updates router projections using exact action likelihoods. On Moonlight-16B-A3B, it improves the seven-benchmark average from 47.10% to 51.22% while retaining deterministic top-K deployment and producing shorter responses.

Critical strengths
  • The ordered-action formulation handles sampling without replacement carefully and avoids enumerating expert-subset permutations. Appendix C clearly distinguishes its valid on-policy interpretation from the limitations introduced by clipping, replay, and shared parameters.
  • The matched sampling-only baseline isolates the additional routing objective: accuracy rises from 47.11% to 51.22% with the sampler, replay, and token objective unchanged (Table 1). Deployment preserves the expert budget, with modest measured training overhead (Table 2).
Critical weaknesses
  • Empirical reliability and generality remain uncertain. Experiments use one model, with no reported variation across training seeds. The largest gain is on MMLU-Pro's 70-question validation split, while commonsense evaluations use 100-example subsets; these sizes precede further prompt filtering (Appendix E.2). Three sampled responses per question do not establish robustness across training runs or broader test distributions.
  • Comparisons with alternative routing objectives are inconclusive. MoE-GRPO's run collapses, and its reported best checkpoint averages only 10.45% (Table 1), without a substantive diagnosis. The expert-ratio ablation instead clips untempered normalized sigmoid probabilities, rather than the actual ordered sampling-policy ratios (Eq. 11 versus Eq. 6). Thus Table 3 does not establish superiority over clipping the likelihood of the actions actually sampled.
Questions for authors
  1. Do the gains persist across independent training seeds and full held-out academic/commonsense test sets, with paired uncertainty estimates?
  2. What causes MoE-GRPO's collapse, and how does RLRP compare with a stabilized baseline that clips the exact ordered-action likelihood ratio? Reporting routing-ratio or KL drift during minibatch updates would clarify this comparison.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.