- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Predictive Routing Replay (), a method designed to improve the stability of reinforcement learning (RL) for Mixture-of-Experts (MoE) based large language models. The authors identify router staleness as a limitation of conventional routing replay, where previously recorded expert selections become outdated as the routing policy evolves during training.
To address this issue, introduces a lightweight evolution predictor that anticipates future router states and selects expert routes accordingly. These predicted routes are cached and replayed during subsequent policy updates, maintaining routing consistency while better reflecting the evolving model. The authors also provide theoretical analyses connecting router staleness, gradient deviation, and routing prediction accuracy.
Experiments across three MoE architectures and different off-policy settings demonstrate improvements in training stability and downstream mathematical reasoning performance compared with GRPO, GSPO, and conventional routing replay.
Critical strengths
S1. Clear problem formulation and a well-motivated solution to router staleness.
The paper identifies an important limitation of existing routing replay methods in MoE-based RL. While conventional routing replay addresses rollout-training mismatch by fixing expert selections, it introduces router staleness as the routing policy evolves. The proposed method addresses this limitation by predicting future expert selections rather than simply reusing previously observed routes. This provides a clear motivation and a technically meaningful extension of existing routing replay approaches.
S2. Theoretical analysis supported by empirical observations.
The paper provides theoretical analyses connecting router staleness to gradient deviation and establishing conditions for routing consistency. These analyses are complemented by empirical evaluations of top-k agreement, routing divergence, and training dynamics, providing evidence that can better track router evolution and stabilize policy optimization.
S3. Consistent improvements across different models and off-policy settings.
The experimental results demonstrate improvements over conventional routing replay across three MoE architectures and multiple off-policy settings. In particular, achieves substantial gains on Qwen3-30B-A3B-Base, while maintaining relatively stable training dynamics. These results suggest that predictive routing can be an effective strategy for improving MoE-based RL.
Critical weaknesses
W1. Insufficient isolation of the benefits of predictive routing from general routing adjustments.
The proposed method introduces a learnable bias to modify the router’s expert selection. However, it remains unclear whether the observed improvements primarily result from accurately predicting future router states or simply from introducing additional flexibility into expert selection.
Although the paper demonstrates higher top-k agreement and lower routing divergence, these observations do not fully establish that future routing prediction is the primary source of downstream performance improvements. Additional comparisons with non-predictive routing adjustments would help clarify the specific contribution of the evolution predictor.
W2. Substantial variation in performance improvements across different MoE architectures.
While achieves substantial improvements on Qwen3-30B-A3B-Base, the gains on other models are considerably smaller. In particular, the improvements on OLMoE-1B-7B are marginal across all evaluated off-policy settings.
This variation raises questions about the generalizability of the proposed method and the conditions under which predictive routing is most effective. Although consistently achieves the highest average performance, the results do not sufficiently explain why certain architectures benefit substantially more than others.
Furthermore, the relatively small improvements on some models make it difficult to determine whether the observed gains are robust to experimental variability.
W3. Limited evaluation of end-to-end computational efficiency.
The paper reports that the evolution predictor introduces relatively small additional computational overhead in terms of active-expert forward FLOPs. However, also requires caching router features and logits, maintaining additional predictor parameters, and performing predictor optimization during training.
The reported computational analysis does not sufficiently establish how these additional operations affect end-to-end training efficiency in practical MoE RL systems. In particular, wall-clock training time, peak GPU memory consumption, and performance under matched computational budgets would provide a more comprehensive assessment of the practical benefits of .
Questions for authors
Q1. How can the authors distinguish the benefits of predictive routing from those of additional routing perturbations?
The proposed method introduces a learnable bias to adjust expert selection. However, it remains unclear whether the performance improvements primarily arise from accurately anticipating future router states or simply from selecting alternative expert configurations.
Could the authors provide comparisons with non-predictive routing adjustments, such as fixed or randomly initialized routing biases, to isolate the contribution of the evolution predictor?
Q2. What determines the effectiveness of across different MoE architectures?
While achieves substantial improvements on Qwen3-30B-A3B-Base, the performance gains on OLMoE-1B-7B are relatively marginal.
Could the authors investigate whether these differences are associated with router drift magnitude, expert specialization, model scale, or architectural characteristics? Additionally, could the authors provide further evidence that the smaller improvements are statistically significant?
Q3. How does affect end-to-end training efficiency under matched computational budgets?
Although the additional computational cost of the evolution predictor is reported to be relatively small, the proposed method requires additional feature caching and predictor updates.
Could the authors provide comparisons of wall-clock training time, peak GPU memory consumption, and downstream performance under matched computational budgets? Such evaluations would help clarify whether the improvements justify the additional system complexity.
2 comments
Follow-up for Reviewer R4X6
Dear Reviewer R4X6,
Thank you again for your positive and constructive review. We have answered all three of your questions in our response under your review:
- On prediction versus flexibility: same-size perturbations without a prediction behind them, shuffled or random, fall below in both accuracy and agreement, while the learned correction gives the full gain.
- On the variation across architectures: the room for is set by how much the router drifts, and the smaller Moonlight gain repeats across matched runs with lower variance than .
- On cost: training time and peak memory are essentially unchanged, and under the same schedule finishes points higher.
We believe these results address the points you raised, and we would be grateful if you could consider updating your score in light of them. If anything is still unclear, we are glad to answer before the discussion period closes.
Best regards,
The authors
Response to Reviewer R4X6
Thank you for the careful review. We answer the points in the order you raised them. Unless stated otherwise, all new results are on Moonlight-16B-A3B in the Miles codebase, GSM8K at off-4, and top-6 agreement is the share of token-layers whose recorded top-6 expert set equals the current router's.
W1 / Q1: Does the gain come from prediction, or just from the flexibility of a learned bias? It comes from prediction. We replaced the learned correction with perturbations of the same size that carry no prediction: the learned correction with its experts shuffled, and a random correction. We also tried a single fixed bias per layer, set to the mean of the learned correction. The last column is the change in top-6 agreement after one update, relative to recording with the unperturbed router.
Perturbations of the same size without a prediction behind them make both accuracy and agreement worse than R2. The fixed bias keeps only the average direction of the prediction and gets about half of the gain. The full per-token prediction gives the whole gain.
W2 / Q2: Why do gains differ so much across architectures, and are the small gains reliable? PR2 beats R2 in all nine model and off-policy settings of Tables 1 and 2.
PR2 can only fix records that have gone stale, so its room to help depends on how much the router drifts. Moonlight drifts little: within three updates the recorded top-6 set differs from the current router's on only about 3.2% of token-layers, and the predictor recovers about a quarter of that. The gain there is smaller than on Qwen3 but holds across runs. In Table 5, PR2 has both a higher mean and a lower variance than R2, and in Table 4 the rollout-side variant PR3 improves on R3 in every setting.
W3 / Q3: Wall-clock time, memory, and accuracy at matched budget. Measured per rollout batch:
The feature cache and predictor updates add 0.04 GiB per GPU and no measurable time. Under the same schedule and the same training time, PR2 finishes 1.04 points higher.
We will add the control table to the analysis section and the cost table to Appendix E.
We believe these results address all three of your concerns: the gain comes from prediction rather than added flexibility, the size of the gain follows the amount of router drift, and the extra cost is negligible. We would be grateful if you would take them into account when updating your score, and we are happy to answer any further questions.