acceptodds
Under review as a conference paper at ICLR 2027

: Predictive Routing Replay for MoE-Based LLM Reinforcement Learning

Abstract

Mixture of Experts (MoE) Large Language Models (LLMs) achieve strong performance at scale. However, reinforcement learning (RL) on MoE-based LLMs often suffers from training instability. A root cause is router drift, i.e., expert activations can change drastically across model updates and differ between disaggregated rollout and training phases, causing large rollout–training mismatch and unstable importance sampling weights in PPO-style RL algorithms. Routing replay mitigates this issue by freezing the replay route within each reasoning trajectory, but it ignores how the router evolves under off-policy updates and thus causes router staleness. We introduce Predictive Routing Replay (PR2), which augments each router with a lightweight evolution predictor that learns to anticipate short-horizon router evolution. During route recording, the predictor selects expert sets that anticipate router evolution. During training, it replays these sets with the current model. We bound recorded-route sensitivity through predictor fit and update drift, derive an exact KL threshold for top- agreement, and show how local support certificates yield matching forward computations and fixed-batch policy gradients. Across three MoE models and reuse factors , and , PR2 achieves the highest average accuracy in all nine comparisons, improving average math accuracy over routing replay by to percentage points on Qwen3-30B-A3B-Base.

Primary area:
foundation or frontier models, including LLMs
Keywords:
mixture of experts, reinforcement learning, large language models, expert routing, training stability, off-policy reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_R4X6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Predictive Routing Replay (PR2PR^2), a method designed to improve the stability of reinforcement learning (RL) for Mixture-of-Experts (MoE) based large language models. The authors identify router staleness as a limitation of conventional routing replay, where previously recorded expert selections become outdated as the routing policy evolves during training.

To address this issue, PR2PR^2 introduces a lightweight evolution predictor that anticipates future router states and selects expert routes accordingly. These predicted routes are cached and replayed during subsequent policy updates, maintaining routing consistency while better reflecting the evolving model. The authors also provide theoretical analyses connecting router staleness, gradient deviation, and routing prediction accuracy.

Experiments across three MoE architectures and different off-policy settings demonstrate improvements in training stability and downstream mathematical reasoning performance compared with GRPO, GSPO, and conventional routing replay.

Critical strengths

S1. Clear problem formulation and a well-motivated solution to router staleness.

The paper identifies an important limitation of existing routing replay methods in MoE-based RL. While conventional routing replay addresses rollout-training mismatch by fixing expert selections, it introduces router staleness as the routing policy evolves. The proposed method addresses this limitation by predicting future expert selections rather than simply reusing previously observed routes. This provides a clear motivation and a technically meaningful extension of existing routing replay approaches.

S2. Theoretical analysis supported by empirical observations.

The paper provides theoretical analyses connecting router staleness to gradient deviation and establishing conditions for routing consistency. These analyses are complemented by empirical evaluations of top-k agreement, routing divergence, and training dynamics, providing evidence that PR2PR^2 can better track router evolution and stabilize policy optimization.

S3. Consistent improvements across different models and off-policy settings.

The experimental results demonstrate improvements over conventional routing replay across three MoE architectures and multiple off-policy settings. In particular, PR2PR^2 achieves substantial gains on Qwen3-30B-A3B-Base, while maintaining relatively stable training dynamics. These results suggest that predictive routing can be an effective strategy for improving MoE-based RL.

Critical weaknesses

W1. Insufficient isolation of the benefits of predictive routing from general routing adjustments.

The proposed method introduces a learnable bias to modify the router’s expert selection. However, it remains unclear whether the observed improvements primarily result from accurately predicting future router states or simply from introducing additional flexibility into expert selection.

Although the paper demonstrates higher top-k agreement and lower routing divergence, these observations do not fully establish that future routing prediction is the primary source of downstream performance improvements. Additional comparisons with non-predictive routing adjustments would help clarify the specific contribution of the evolution predictor.

W2. Substantial variation in performance improvements across different MoE architectures.

While PR2PR^2 achieves substantial improvements on Qwen3-30B-A3B-Base, the gains on other models are considerably smaller. In particular, the improvements on OLMoE-1B-7B are marginal across all evaluated off-policy settings.

This variation raises questions about the generalizability of the proposed method and the conditions under which predictive routing is most effective. Although PR2PR^2 consistently achieves the highest average performance, the results do not sufficiently explain why certain architectures benefit substantially more than others.

Furthermore, the relatively small improvements on some models make it difficult to determine whether the observed gains are robust to experimental variability.

W3. Limited evaluation of end-to-end computational efficiency.

The paper reports that the evolution predictor introduces relatively small additional computational overhead in terms of active-expert forward FLOPs. However, PR2PR^2 also requires caching router features and logits, maintaining additional predictor parameters, and performing predictor optimization during training.

The reported computational analysis does not sufficiently establish how these additional operations affect end-to-end training efficiency in practical MoE RL systems. In particular, wall-clock training time, peak GPU memory consumption, and performance under matched computational budgets would provide a more comprehensive assessment of the practical benefits of PR2PR^2.

Questions for authors

Q1. How can the authors distinguish the benefits of predictive routing from those of additional routing perturbations?

The proposed method introduces a learnable bias to adjust expert selection. However, it remains unclear whether the performance improvements primarily arise from accurately anticipating future router states or simply from selecting alternative expert configurations.

Could the authors provide comparisons with non-predictive routing adjustments, such as fixed or randomly initialized routing biases, to isolate the contribution of the evolution predictor?

Q2. What determines the effectiveness of PR2PR^2 across different MoE architectures?

While PR2PR^2 achieves substantial improvements on Qwen3-30B-A3B-Base, the performance gains on OLMoE-1B-7B are relatively marginal.

Could the authors investigate whether these differences are associated with router drift magnitude, expert specialization, model scale, or architectural characteristics? Additionally, could the authors provide further evidence that the smaller improvements are statistically significant?

Q3. How does PR2PR^2 affect end-to-end training efficiency under matched computational budgets?

Although the additional computational cost of the evolution predictor is reported to be relatively small, the proposed method requires additional feature caching and predictor updates.

Could the authors provide comparisons of wall-clock training time, peak GPU memory consumption, and downstream performance under matched computational budgets? Such evaluations would help clarify whether the improvements justify the additional system complexity.

2 comments
AuthorsOpenReview ↗

Response to Reviewer R4X6

Thank you for the careful review. We answer the points in the order you raised them. Unless stated otherwise, all new results are on Moonlight-16B-A3B in the Miles codebase, GSM8K at off-4, and top-6 agreement is the share of token-layers whose recorded top-6 expert set equals the current router's.

W1 / Q1: Does the gain come from prediction, or just from the flexibility of a learned bias? It comes from prediction. We replaced the learned correction with perturbations of the same size that carry no prediction: the learned correction with its experts shuffled, and a random correction. We also tried a single fixed bias per layer, set to the mean of the learned correction. The last column is the change in top-6 agreement after one update, relative to recording with the unperturbed router.

Recording ruleGSM8K Avg@1Top-6 agreement change
R2\mathrm{R}^286.73±1.1886.73 \pm 1.1800
Shuffled correction86.15±1.3086.15 \pm 1.30−1.03-1.03
Random correction, same norm86.10±1.2286.10 \pm 1.22−1.09-1.09
Fixed per-layer bias87.30±0.9087.30 \pm 0.90+0.34+0.34
Full PR2\mathrm{PR}^287.77±0.3187.77 \pm 0.31+0.76+0.76

Perturbations of the same size without a prediction behind them make both accuracy and agreement worse than R2\mathrm{R}^2. The fixed bias keeps only the average direction of the prediction and gets about half of the gain. The full per-token prediction gives the whole gain.

W2 / Q2: Why do gains differ so much across architectures, and are the small gains reliable? PR2\mathrm{PR}^2 beats R2\mathrm{R}^2 in all nine model and off-policy settings of Tables 1 and 2.

Gain over R2\mathrm{R}^2off-2off-4off-8
Qwen3-30B-A3B-Base9.259.2513.0013.008.168.16
Moonlight-16B-A3B1.191.191.191.193.443.44
OLMoE-1B-7B0.780.780.610.610.190.19

PR2\mathrm{PR}^2 can only fix records that have gone stale, so its room to help depends on how much the router drifts. Moonlight drifts little: within three updates the recorded top-6 set differs from the current router's on only about 3.2%3.2\% of token-layers, and the predictor recovers about a quarter of that. The gain there is smaller than on Qwen3 but holds across runs. In Table 5, PR2\mathrm{PR}^2 has both a higher mean and a lower variance than R2\mathrm{R}^2, and in Table 4 the rollout-side variant PR3\mathrm{PR}^3 improves on R3\mathrm{R}^3 in every setting.

W3 / Q3: Wall-clock time, memory, and accuracy at matched budget. Measured per rollout batch:

R2\mathrm{R}^2PR2\mathrm{PR}^2
Training time362.5362.5 s361.5361.5 s
Peak memory per GPU60.3960.39 GiB60.4360.43 GiB
Extra FLOPs over the active-expert forwardnone0.19%0.19\% to 0.69%0.69\%
GSM8K after the same schedule86.7386.7387.7787.77

The feature cache and predictor updates add 0.040.04 GiB per GPU and no measurable time. Under the same schedule and the same training time, PR2\mathrm{PR}^2 finishes 1.041.04 points higher.

We will add the control table to the analysis section and the cost table to Appendix E.

We believe these results address all three of your concerns: the gain comes from prediction rather than added flexibility, the size of the gain follows the amount of router drift, and the extra cost is negligible. We would be grateful if you would take them into account when updating your score, and we are happy to answer any further questions.

AuthorsOpenReview ↗

Follow-up for Reviewer R4X6

Dear Reviewer R4X6,

Thank you again for your positive and constructive review. We have answered all three of your questions in our response under your review:

  • On prediction versus flexibility: same-size perturbations without a prediction behind them, shuffled or random, fall below R2\mathrm{R}^2 in both accuracy and agreement, while the learned correction gives the full gain.
  • On the variation across architectures: the room for PR2\mathrm{PR}^2 is set by how much the router drifts, and the smaller Moonlight gain repeats across matched runs with lower variance than R2\mathrm{R}^2.
  • On cost: training time and peak memory are essentially unchanged, and under the same schedule PR2\mathrm{PR}^2 finishes 1.041.04 points higher.

We believe these results address the points you raised, and we would be grateful if you could consider updating your score in light of them. If anything is still unclear, we are glad to answer before the discussion period closes.

Best regards,

The authors

Reviewer_qmGXOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies route staleness in fixed routing replay for MoE reinforcement learning. R² records the expert sets used during the old-logprob pass and reuses them during updates. PR² adds a linear predictor per MoE layer that adjusts the router logits at recording time and selects a predicted support set; during updates the index set is held fixed while the current model and current gate are used. The predictor is trained on cached detached features with the current router distribution as teacher, and affects only the recording of the next batch. The paper derives a categorical sensitivity bound, an exact KL threshold for hard top-k agreement, and a conditional result on whole-graph consistency.

Experiments cover Qwen3-30B-A3B-Base, Moonlight-16B-A3B and OLMoE-1B-7B at three reuse strengths, with held-out free generation, a three-seed study of the generation-side variant, matched repeats in a second codebase, and a combination with GSPO. Routing perturbation is disabled during standard evaluation.

Critical strengths

The reported gains are real and span several MoE architectures. All three models improve at all three reuse strengths, the margins on Qwen are substantial, and the results are supported by held-out free generation, matched repeats in a second codebase, and a seed study of the generation-side variant. This indicates that the complete method works in actual training, rather than only improving intermediate training statistics.

The method and the theory both have concrete content. The timing of recording, replay and distillation is made explicit in the algorithms, the predictor does not enter the policy gradient, and the hard top-k analysis derives an exact KL radius at which the selected support can change, together with the conditions needed for a matching forward computation and fixed-batch gradient.

Critical weaknesses

The learned WpW_p may simply relearn the momentum that the optimizer already has. Optimizers such as AdamW keep a first-order momentum, so where the router weights will move over the next few updates on the same batch is already largely determined by the optimizer's momentum buffer and is therefore predictable. The predictor is a single linear map per layer that turns the cached hidden state h0h_0 into a logit bias h0Wph_0W_p, trained toward the router distribution after a few updates on the same batch. The change in router logits can be written as zi−z0=(hi−h0)W0+hiΔWiz_i-z_0=(h_i-h_0)W_0+h_i\Delta W_i, so as long as the hidden states drift little, the best WpW_p is approximately the within-batch increment ΔWi\Delta W_i of the router weights. Since WpW_p is trained on earlier batches and used to record the next one, it can only work if consecutive batches change the router weights in similar directions, which is precisely the effect of the optimizer's momentum. PR² therefore looks like a separately trained predictor that reproduces the optimizer's momentum, and the analysis of route evolution and staleness mainly redescribes a parameter update with momentum.

The current experiments do not distinguish between the two explanations. In Table 7 the linear predictor matches a two-layer MLP, but the target is already approximately linear in h0h_0; in Figure 7 standard KL matches delta-matching, but both fit the same logit shift and differ only in normalization. The paper neither compares the learned WpW_p with the actual ΔWi\Delta W_i or the direction of the optimizer's momentum buffer, nor uses a bias built directly from the optimizer's momentum as a control. If such a simple alternative performs as well, a separately trained predictor is unnecessary.

Questions for authors
  1. Please state the optimizer and its momentum coefficient, and report the similarity, for example per-layer cosine similarity, between the learned WpW_p and each of the following: the actual increment ΔWi\Delta W_i, the optimizer's first-order momentum for the router weights, and the update direction given by the optimizer, such as m/(v+ϵ)m/(\sqrt{v}+\epsilon) for Adam.

  2. Please replace the learned WpW_p with the router-weight increment computed directly from the optimizer's momentum buffer, for example a few Adam steps computed from the current first and second moments, and compare its top-k agreement and final accuracy with full PR² under the same setting.

  3. Please rerun the experiments with a momentum-free optimizer, for example AdamW with β1=0, and report whether the predictor's top-k agreement stays close to its level with momentum. A clear drop would indicate that the predictive ability comes from the optimizer's momentum.

Update: Thank you for the author's rebuttal. I have updated my score.

2 comments
AuthorsOpenReview ↗

Response to Reviewer qmGX

Thank you for the careful review. Unless stated otherwise, all new results are on Moonlight-16B-A3B in the Miles codebase, GSM8K at off-4, and top-6 agreement is the share of token-layers whose recorded top-6 expert set equals the current router's. You proposed three concrete tests, and we ran all three. Momentum does help PR2\mathrm{PR}^2, since it makes the router's drift smoother and easier to predict. But WpW_p does not reproduce the momentum. Its cosine with the momentum and with the actual router increment is near zero, and an increment built from the momentum buffer gives no gain at all. Without first-order momentum, PR2\mathrm{PR}^2 still keeps more than half of its accuracy gain.

Your decomposition shows why. The logit change is zi−z0=(hi−h0)W0+hiΔWiz_i - z_0 = (h_i - h_0)W_0 + h_i \Delta W_i, and the router's optimizer state only describes the second term. WpW_p has essentially zero cosine with ΔWi\Delta W_i and still improves routing agreement, which points to the first term: the drift of the hidden states that feed the router as the whole network updates. The router's optimizer state does not carry that information.

Q1: Which optimizer and momentum coefficient, and how similar is WpW_p to each update quantity? The VeRL runs behind Tables 1 and 2 use Adam with β1=0.9\beta_1 = 0.9, β2=0.999\beta_2 = 0.999, weight decay 0.010.01, gradient clipping 1.01.0 and a constant learning rate. The Miles runs below use Adam with β1=0.9\beta_1 = 0.9, β2=0.98\beta_2 = 0.98, weight decay 0.10.1 and a predictor learning-rate multiplier of 5050. Mean cosine similarity between WpW_p and each quantity:

WpW_p compared withCosine
First-order momentum−0.006-0.006
Adam update direction m/(v+ϵ)m/(\sqrt{v}+\epsilon)0.0030.003
Actual router increment ΔWi\Delta W_i0.0000.000

Q2: What happens if WpW_p is replaced by the increment computed from the momentum buffer?

Recording ruleGSM8K Avg@1Top-6 agreement after one update
R2\mathrm{R}^286.73±1.1886.73 \pm 1.1896.83%96.83\%
Momentum-derived increment86.62±1.1286.62 \pm 1.1296.83%96.83\%
Full PR2\mathrm{PR}^287.77±0.3187.77 \pm 0.3197.59%97.59\%

The momentum-derived increment moves the logits by only about 10−510^{-5} per step, so agreement and accuracy both stay at the R2\mathrm{R}^2 level, and scaling it up 16×16\times does not change this. The learned WpW_p improves both.

Q3: What happens with a momentum-free optimizer, β1=0\beta_1 = 0 for the policy?

OptimizerR2\mathrm{R}^2Full PR2\mathrm{PR}^2Gain
β1=0.9\beta_1 = 0.986.73±1.1886.73 \pm 1.1887.77±0.3187.77 \pm 0.31+1.04+1.04
β1=0\beta_1 = 086.04±1.3186.04 \pm 1.3186.60±0.9886.60 \pm 0.98+0.56+0.56

Without momentum PR2\mathrm{PR}^2 keeps more than half of its accuracy gain. Its gain in top-6 agreement falls further, to 0.250.25 to 0.300.30 points on the shared trajectory against 0.760.76 with momentum, which is what we would expect. With momentum, consecutive updates point in similar directions, so the drift they cause is regular and easy to predict. Without it the drift between batches is less regular and R2\mathrm{R}^2 itself loses 0.690.69 points, which leaves any predictor less to learn. Since the momentum buffer on its own gives nothing in Q2, what PR2\mathrm{PR}^2 uses is the regularity that momentum creates in the drift, not the momentum vector.

W2: Table 7 and Figure 7 do not separate the two explanations. We agree, because they were built to test design choices rather than the mechanism. Table 7 asks whether a linear predictor has enough capacity. It does, which is why PR2\mathrm{PR}^2 uses a single matrix per layer. Figure 7 asks whether the result depends on the loss, and it does not. As you point out, a target close to linear in h0h_0 and two losses that fit the same logit shift would give these results under either explanation, so Q1 to Q3 are the direct tests. In the revision we will present Table 7 and Figure 7 as design ablations, and make Q1 to Q3 and the decomposition above the evidence on this question in the mechanism section.

W3: No direct comparison with the actual weight increment, the momentum buffer, or a bias built from momentum. All three are now covered: the actual increment and the momentum buffer in Q1, and the momentum-built increment in Q2.

Your review pointed us to the right tests, and we believe they now settle the question: WpW_p does not reproduce the optimizer's momentum, and a predictor built from the momentum buffer gives no benefit. We hope this resolves your concern about soundness, and we would be grateful if you would reconsider your score. We are happy to run any further check you would find useful.

AuthorsOpenReview ↗

Follow-up for Reviewer qmGX

Dear Reviewer qmGX,

Thank you again for your detailed review. Your question of whether WpW_p simply relearns the optimizer momentum was the central concern, and we ran all three tests you proposed to answer it directly:

  • WpW_p has near-zero cosine with the first-order momentum, with the Adam update direction, and with the actual router increment.
  • Replacing WpW_p with the increment computed from the momentum buffer leaves both top-6 agreement and final accuracy at the R2\mathrm{R}^2 level, while the learned WpW_p improves both.
  • With β1=0\beta_1 = 0, PR2\mathrm{PR}^2 still keeps more than half of its accuracy gain.

We believe these results resolve the soundness concern, and the details are in our response under your review. We would be grateful if you could take another look and consider updating your score. If any point is still unclear, we are glad to answer before the discussion period closes.

Best regards,

The authors

Reviewer_RZfPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Predictive Routing Replay, a framework designed to improve the stability of reinforcement learning for Mixture-of-Experts (MoE) language models. The central motivation is that conventional routing replay reduces rollout-training mismatch by fixing expert assignments, but these cached routes may become stale as the router evolves during policy updates. To address this issue, PR2PR^2 introduces lightweight evolution predictors that anticipate changes in routing preferences and select expert sets for subsequent replay. The predictors are trained through a KL-based objective using cached router features and updated routing distributions. The paper also provides theoretical analyses of routing staleness, gradient deviation, and top-k support consistency. Experiments across three MoE models and different off-policy settings demonstrate improved routing agreement and training stability, with particularly substantial accuracy gains on Qwen3-30B-A3B-Base.

Critical strengths
  1. The paper addresses an important and practical challenge in MoE-based LLM reinforcement learning. Router drift and stale expert assignments can introduce optimization instability, particularly in off-policy training settings. The motivation for improving conventional routing replay is therefore well justified.
  2. The proposed framework is conceptually clear and relatively simple. By introducing lightweight evolution predictors without modifying the underlying RL optimization objective, PR2PR^2 provides a practical approach that could potentially be integrated into existing training frameworks.
  3. The paper provides extensive theoretical analysis and detailed implementation descriptions. The experiments cover three MoE architectures, multiple off-policy settings, and several reasoning benchmarks. Additional analyses of routing agreement, training dynamics, and expert utilization further support the empirical findings.
Critical weaknesses

W1. The explanation of why the predictive mechanism works remains insufficient

  • The central contribution is not simply to improve routing replay, but to do so by predicting future routing changes. It is therefore important to establish whether the gains actually come from anticipating these changes.
  • The results show that PR2PR^2 achieves better agreement between cached and subsequent routes, together with more stable training. However, different methods produce different model update trajectories. Higher agreement may result from more accurate prediction, but it may also arise because routing changes become smaller during training.
  • Comparisons with standard GRPO and routing replay support the overall effectiveness of the method, but do not sufficiently distinguish these two explanations. The theoretical analysis establishes properties when routing distributions are close, but does not by itself explain which effect drives the observed improvements.
  • A more direct analysis of which routing changes the predictor anticipates within the same training trajectory, and how these changes affect subsequent learning, would better support the proposed mechanism.

W2. The gains vary considerably across models, and the conditions under which the method is effective remain unclear

  • PR2PR^2 achieves substantial improvements on Qwen3, but its average gains on OLMoE are small, and it does not consistently outperform other methods on GSM8K. The gains on Moonlight also differ considerably from those on Qwen3.
  • The paper emphasizes that PR2PR^2 achieves the highest average score in all nine model–setting comparisons. However, these rankings do not sufficiently explain why the same routing intervention produces substantially different benefits across models.
  • If router staleness is the key bottleneck addressed by this work, the paper should further examine whether the initial degree of router drift and the extent to which it is reduced correspond to the resulting performance gains.
  • This matters for the method’s practical scope: does PR2PR^2 mainly help training configurations with severe routing instability, or can it also consistently improve otherwise stable MoE reinforcement learning? The current results demonstrate settings where the method works, but provide limited explanation of this boundary.
Questions for authors
  1. Does PR2 improve routing agreement primarily by anticipating future routing changes, or by changing the training dynamics so that the router itself becomes more stable? Can the existing experiments distinguish these two effects?
  2. Why are the gains on Qwen3 substantially larger than those on OLMoE? Can differences in router staleness and its reduction explain this variation in performance gains?
2 comments
AuthorsOpenReview ↗

Response to Reviewer RZfP

Thank you for the careful review. We answer the points in the order you raised them. Table and figure numbers refer to the submitted PDF. Unless stated otherwise, all new results are on Moonlight-16B-A3B in the Miles codebase, GSM8K at off-4, and top-6 agreement is the share of token-layers whose recorded top-6 expert set equals the current router's.

W1 / Q1: Does PR2\mathrm{PR}^2 anticipate router changes, or just make the router move less? It anticipates them. We ran the within-trajectory analysis you suggested. A single reference run decides all routing and all updates, so the router moves in exactly the same way under every proposal, and a more stable router cannot be the explanation. Each proposal is scored only on how well its recorded top-6 set matches the router hh updates later. The first three columns are the change in top-6 agreement relative to the unperturbed router, and the last is the final accuracy after full training with each rule.

Recorded proposal1 update2 updates3 updatesFinal GSM8K
Unperturbed router, R2\mathrm{R}^200000086.7386.73
Learned correction, PR2\mathrm{PR}^2+0.76+0.76+0.77+0.77+0.80+0.8087.7787.77
Shuffled correction, same norm−1.03-1.03−1.02-1.02−0.96-0.9686.1586.15
One-step optimizer extrapolation0.000.000.000.000.000.0086.6286.62

With the router's movement held fixed, the learned correction still recovers about a quarter of the drift at every horizon, and final accuracy follows the same order.

W2 / Q2: Why are gains on Qwen3 so much larger than on OLMoE, and can router staleness explain it? The gain over R2\mathrm{R}^2 is positive in all nine settings.

Gain over R2\mathrm{R}^2off-2off-4off-8
Qwen3-30B-A3B-Base9.259.2513.0013.008.168.16
Moonlight-16B-A3B1.191.191.191.193.443.44
OLMoE-1B-7B0.780.780.610.610.190.19

Staleness limits what any replay correction can recover, since only stale records can be fixed. On Moonlight the recorded top-6 set differs from the current router's on only about 3.2%3.2\% of token-layers within three updates, and the predictor recovers about a quarter of that. So PR2\mathrm{PR}^2 has the most room where routing is least stable, as on Qwen3. Where routing is already stable it still gives a smaller gain, and that gain repeats across runs in Tables 4 and 5.

We will add the within-trajectory table to the mechanism section and state this scope there.

We believe this answers both of your questions: with the router's movement held fixed, the gain still comes from anticipating it, and staleness accounts for where the method helps most. If you find these results convincing, we would be grateful if you would consider raising your score. We are happy to discuss any remaining point.

Thank you for your detailed response and the additional experiments. I will maintain my current rating.

Reviewer_kaz6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies router staleness in MoE-based LLM reinforcement learning. It proposes Predictive Routing Replay (PR²), which uses a lightweight predictor to select expert routes that anticipate future router updates while keeping routes fixed during training. The paper provides theoretical bounds on routing differences and gradient deviation. Experiments on three MoE models under different off-policy settings show improved reasoning accuracy and routing consistency, with particularly strong gains on Qwen3-30B-A3B-Base.

Critical strengths
  1. The paper offers a meaningful new perspective on routing replay. Rather than simply fixing past routes, it predicts future routes to balance training stability and effective learning.

  2. The experimental results are strong, especially on Qwen3, where PR² improves average accuracy over routing replay by 8.16–13.00 percentage points. Results across three models and multiple off-policy settings further support its effectiveness.

  3. The paper provides useful theoretical analysis of routing differences and gradient deviation, supported by routing analysis, additional experiments, and ablation studies.

Critical weaknesses
  1. PR² changes both the selected experts and the gating weights during route recording. The paper does not clearly separate their effects, making it difficult to confirm that predictive expert selection is the main source of improvement.

  2. The paper lacks a comparison with simpler ways to estimate future routes, such as extrapolating recent router updates. This makes it harder to assess the value of the learned predictor itself.

  3. The main Qwen3 results lack repeated runs, while improvements on smaller models are relatively modest. This limits confidence in the consistency of the reported gains.

Questions for authors
  1. Could the authors provide an ablation that uses predicted expert indices but keeps the original router logits for gating weights? This would help isolate the effect of predictive expert selection.

  2. How does PR² compare with a simple prediction baseline, such as extrapolating recent router updates without training an additional predictor?

  3. Could the authors provide repeated runs for the main Qwen3 experiments to confirm the reliability of the large gains? Also, how often does the exact KL condition for Top-k agreement hold in practice?

2 comments
AuthorsOpenReview ↗

Response to Reviewer kaz6

Thank you for the careful review. We answer the points in the order you raised them. Table and figure numbers refer to the submitted PDF. Unless stated otherwise, all new results are on Moonlight-16B-A3B in the Miles codebase, GSM8K at off-4, and top-6 agreement is the share of token-layers whose recorded top-6 expert set equals the current router's.

W1 / Q1: Is it the predicted expert indices or the recorded gate weights that matter? We ran the ablation you asked for: predicted indices, with gate weights taken from the original router logits.

Recording ruleGSM8K Avg@1
R2\mathrm{R}^286.73±1.1886.73 \pm 1.18
Predicted indices, original gate weights87.30±0.8287.30 \pm 0.82
Full PR2\mathrm{PR}^287.77±0.3187.77 \pm 0.31

The predicted indices alone give more than half of the gain, and taking the recording-time gate weights from the corrected logits gives the rest. In both variants the updates use the current gate weights.

W2 / Q2: How does PR2\mathrm{PR}^2 compare with simply extrapolating the router update, with no predictor? We built this baseline from the recorded optimizer state, using the Adam update direction m/(v+ϵ)m/(\sqrt{v}+\epsilon).

Recording ruleGSM8K Avg@1Top-6 sets changed
R2\mathrm{R}^286.73±1.1886.73 \pm 1.1800
Extrapolated router update86.62±1.1286.62 \pm 1.120.004%0.004\%
Full PR2\mathrm{PR}^287.77±0.3187.77 \pm 0.313.0%3.0\%

At this learning rate the extrapolation moves the logits by only about 10−510^{-5} and almost never changes which experts are picked. Scaling it up 16×16\times does not help. The learned predictor works where extrapolation does not, because part of the routing change comes from drift in the hidden states, which extrapolating the router weights cannot see.

W3 / Q3a: Are the main Qwen3 results repeated, and are the gains on smaller models smaller? The rollout-side Qwen3 variant is repeated over three seeds in Table 4.

Qwen3, Table 4off-2off-4off-8
R3\mathrm{R}^342.1442.1442.7342.7337.7537.75
PR3\mathrm{PR}^344.7044.7046.0346.0340.0040.00
Gain2.562.563.303.302.252.25

Table 5 adds matched repeats on Moonlight. The run-to-run standard deviation in these repeats is at most 1.21.2 points, while the Qwen3 PR2\mathrm{PR}^2 gains range from 8.168.16 to 13.0013.00 points. We will add repeated Qwen3 PR2\mathrm{PR}^2 runs to the final version. The gains on the smaller models are smaller because their routers leave less to correct: on Moonlight only about 3.2%3.2\% of recorded top-6 sets go stale within three updates.

Q3b: How often does the exact KL condition hold in practice? We measured it on Moonlight.

ConditionShare of token-layers where it holds
Squared-gap condition13%13\%
Exact condition61%61\%

The exact condition certifies top-6 agreement on 61%61\% of token-layers, close to five times the squared-gap bound. Wherever it holds, the recorded top-6 set equals the router's.

We believe we have now addressed each of your questions: both parts of the recording change contribute, a simple extrapolation baseline does not come close to the learned predictor, the gains repeat across runs, and the exact KL condition holds on most token-layers. We would be grateful if you would reconsider your score in light of these results, and we are happy to provide anything else that would help.

AuthorsOpenReview ↗

Follow-up for Reviewer kaz6

Dear Reviewer kaz6,

Thank you again for your focused questions. We have answered each of them in our response under your review:

  • The index-only ablation you asked for: predicted indices alone give more than half of the gain, and the corrected gate weights give the rest.
  • The router-extrapolation baseline built from the optimizer state stays at the R2\mathrm{R}^2 level in final accuracy, while PR2\mathrm{PR}^2 is clearly higher.
  • The exact KL condition holds on 61%61\% of token-layers, close to five times the squared-gap bound, and wherever it holds the recorded top-6 set equals the router's.
  • For repeated runs, we summarized the three-seed results in Table 4 and the matched repeats in Table 5, and we will add repeated Qwen3 PR2\mathrm{PR}^2 runs to the final version.

We believe these address the concerns behind your rating, and we would be grateful if you could consider updating your score. If anything is still unclear, we are glad to answer before the discussion period closes.

Best regards,

The authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Kind reminder: our responses are posted

Dear Reviewers kaz6, qmGX and R4X6,

Thank you again for your reviews. We posted our responses on November 15, each under your own review. We would be grateful if you could take a look before the discussion period closes and let us know whether they address your concerns. If anything is still unclear, we are happy to answer follow-up questions or provide more detail.

A short summary of what each response contains:

  • qmGX: we ran all three tests you proposed. WpW_p has near-zero cosine with the optimizer momentum and with the actual router increment, an increment built from the momentum buffer gives no gain in agreement or accuracy, and PR2\mathrm{PR}^2 keeps more than half of its accuracy gain with β1=0\beta_1 = 0.
  • kaz6: we added the index-only ablation and the router-extrapolation baseline, both with final accuracy, discussed the repeated runs, and measured how often the exact KL condition holds in practice.
  • R4X6: we added same-size non-predictive controls and a fixed-bias control with final accuracy, together with wall-clock time and peak memory measurements.

We also thank Reviewer RZfP for reading our response.

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.