- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes RLRP, which jointly optimizes the output policy and routing policy of MoE models using response-level rewards. During training, it samples experts to explore different computation paths. It adds a routing loss to GRPO to increase the probability of selecting expert paths that lead to better answers. The authors train Moonlight-16B-A3B on mathematical reasoning data and evaluate it on seven mathematical, academic, and commonsense reasoning benchmarks. RLRP improves average accuracy from 47.10% with GRPO to 51.22% and reduces the mean response length. The experiments also examine the effects of expert sampling alone, different routing clipping rules, and expert sampling at inference time.
Critical strengths
-
The ablation supports the effectiveness of the added routing loss. With all other settings kept the same, adding the routing loss improves average accuracy from 47.11% to 51.22%.
-
This also shows that expert exploration alone does not achieve the same gains. Training the routing policy with response-level rewards is helpful. This comparison shows the benefit of the added routing loss.
Critical weaknesses
-
The comparison with the closest method is not convincing.MoE-GRPO already uses outcome rewards to jointly optimize token and expert selection policies. The main changes in this paper concern routing probability modeling, the loss, and clipping. However, this baseline suffers from training collapse in the experiments. Its best checkpoint achieves an average accuracy of only 10.45%. The paper provides insufficient detail about the collapse and the relevant training settings. This makes it difficult to assess how much the proposed changes improve upon the existing method.
-
The ablation shows that RLRP improves upon GRPO with expert sampling. However, it remains unclear whether the proposed changes offer an improvement over MoE-GRPO. I therefore lean toward weak rejection.
Questions for authors
-
How was MoE-GRPO adapted to Moonlight, and were its training settings adjusted? RLRP and MoE-GRPO are similar in several respects. Why did MoE-GRPO suffer training collapse in these experiments?
-
What advantages does the proposed routing loss have over the MoE-GRPO routing loss? Do the authors have experiments or analyses that separate the contributions of the routing loss and token sampling?