acceptodds
Under review as a conference paper at ICLR 2027

Router Drift Control: Mitigating Off-Policy Degradation in Mixture-of-Experts Reinforcement Learning

Abstract

Reinforcement learning (RL) can become off-policy due to training-inference mismatch, mini-batch updates, and asynchronous training. The resulting mismatch is especially harmful in Mixture-of-Experts (MoE) models, where rollout–training discrepancies can change routing decisions and lead to router drift. Rollout Routing Replay (R3) mitigates this mismatch by reusing expert selections from the rollout stage during training. Our analysis shows that routing replay introduces gradient bias by losing contributions from experts selected by the training router but not the rollout router. To address this, we propose Router Drift Control (RDC), which uses routing disagreement between rollout and training as a signal of off-policy staleness and selectively rejects high-drift responses. The sequence-level filtering avoids gradient bias introduced by replay while preserving native expert selection and the associated learning signals for retained responses. Experiments show that RDC achieves larger gains as policy staleness increases and remains effective in asynchronous RL. In asynchronous training, RDC combined with Truncated Importance Sampling (TIS) improves the average peak validation accuracy across AIME 2024 and AIME 2025 by 5.5 points over TIS and 4.5 points over R3.

Primary area:
reinforcement learning
Keywords:
Mixture‑of‑Experts, Reinforcement Learning, Router Drift, Large Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Y6RfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Large language models are often trained with reinforcement learning using two separate systems: one that generates responses (rollout) and one that updates the model (training). Small numerical differences between these systems, plus training on rollout data that is a few updates old, mean the policy that generated a response is not quite the same as the policy being trained, even though the setup is meant to be on-policy. In Mixture-of-Experts models, where each token is routed to only a handful of expert sub-networks chosen by a router, this mismatch can make the router select different experts for the same token during generation versus during training, which the paper calls router drift. Since only the selected experts receive a learning signal for a given token, drift changes which parts of the model actually get trained. A prior method fixes this by forcing training to reuse the exact experts chosen during generation, but the paper shows this introduces a different problem: it discards the learning signal for experts the training model would otherwise have chosen on its own. The paper's method instead measures, for each response, how much the training-time routing differs from the generation-time routing, and filters out only the most mismatched responses, leaving the rest to train normally with their own routing. Tested on math and logic reasoning tasks, this filtering approach gives larger gains as the mismatch grows, and keeps training from collapsing in a fully asynchronous setup where generation and training run at different speeds.

Critical strengths
  • The paper is well written and easy to follow.
  • The problem statement is well motivated.
  • The proposed algorithm, Router Drift Control, is a sensible design. It filters high-drift responses and leaves the routing of the remaining responses unchanged.
  • The explanation of MoE routing, and how the method differs from R3, is clear and helps justify the approach.
  • The paper reports training dynamics such as reward, gradient norm, and response length, showing that unmodified training collapses under asynchronous conditions while the method's advantage grows as staleness increases.
Critical weaknesses
  • The results come from a single model and a single training dataset, and hyperparameters such as the filtering ratio and the TIS threshold are reported as fixed values without an explained tuning process. Even within this one setting the baselines sometimes outperform RDC, so it is unclear how much of the result would generalize to a different model, dataset, or hyperparameter search.
  • There is no ablation isolating RDC's effect. All reported RDC results are combined with TIS, so there is no result for RDC applied alone or for R3 combined with TIS, and no sensitivity analysis is reported for the method's hyperparameters.
  • The filtering signal is a coarse, response-level average of expert-usage counts, while the proof of gradient bias concerns specific tokens, so a response could score as low-drift even with substantial token-level mismatch. The paper also never reports how much of each batch is actually judged high-drift or how that amount changes with staleness, which would show directly how much drift occurs between rollout and training rather than just the effect of filtering it.
Questions for authors
  1. Why doesn't RDC perform well in some cases? What are its limitations?

  2. How was the filtering ratio of 8% and the TIS threshold of 2.0 chosen? Were they tuned on the same benchmarks being reported, or on a separate setup?

  3. Can you report RDC applied without TIS, to separate its effect from TIS's contribution?

  4. Can you report R3 combined with TIS, so it is compared on equal footing with RDC+TIS?

  5. What fraction of responses in a batch are filtered as high-drift at different points in training, and how does this fraction change with staleness?

  6. How do you results generalize across different models and datasets?

Reviewer_xhUSOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper addresses router drift in MoE reinforcement learning. In asynchronous RL, the same response activates different experts during rollout and training because of numerical differences or policy staleness. The authors show that forcing training to replay rollout routes can change gradients relative to the current policy’s native routing. To address this, they propose Router Drift Control (RDC), which measures differences in expert distributions between rollout and training, averages them across layers, and filters responses with the highest drift. RDC is combined with truncated importance sampling to also address token-probability mismatch. The method is validated empirically using Qwen3-30B model.

Critical strengths

The strengths are summarized below:

i. The paper is well written and easy to follow. ii. The idea is fairly straightforward and the efficacy is therefore expected. iii. It is sound to consider expert routing decisions/discrepancies to deal with router policy mismatch during asynchronous RL training. iv. The authors validate their method on a state-of-the-art model i.e. Qwen-3-30B model. v. The experiments use only one MoE model and one training dataset, leaving suitability to architectures, expert configurations, and non-reasoning tasks unclear.

Critical weaknesses

My main concern with the novelty of the paper. The authors introduce a metric to filter the dataset. This makes the novelty limited. My decision is primarily motivated by this concern. The other weaknesses are summarized below:

i. Although routing replay produces different gradients from native routing, the paper presents no evidence to justify that this can destabilize training. ii. It appears that the proposed metric has a failure mode. If reassignment of experts across tokens preserves per-expert usage counts it will gives zero drift. iii. The paper only considers GRPO as baseline lacking rigorous comparison to other baselines such as PR2, RSPO, and trust-region masking to establish the merit of the proposed approach. iv. The results do not include multiple training seeds or confidence intervals to support statistical significance. v. The paper lacks any ablation study for the choice of metric, filtering ratio, and response-length bucketing.

Questions for authors

Please find my questions below: i What evidence did you use to arrive that higher drift scores for responses can harm optimization or increase instability? ii. How does varying policy lag affect training performance and stability when the update batch size is held fixed? iii. Can you provide empirical comparisons with alternative regularization or filtering methods that constrain policy divergence or use token-probability mismatch? iv. Can you comment on the metric’s failure to detect token-level routing changes that preserve aggregate expert-usage counts, and its implications for filtering effectiveness?

Reviewer_ekPsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In RL training for MoE, differences in precision between inference and training, sequential mini-batch updates, and asynchronous training delays often cause the same response to activate different experts during rollout and training—a phenomenon termed "router drift". Existing methods such as TIS only apply corrections at the token-probability level and cannot perceive underlying changes in expert assignments. Meanwhile, Rollout Routing Replay (R3) forcibly reuses rollout-stage expert selections during training, which prevents experts that would have been selected by the current training model from receiving gradient updates and thus introduces gradient bias. To address this, the paper proposes Router Drift Control (RDC): instead of forcibly altering routing during training, it measures the discrepancy in expert usage frequencies between rollout and training for each response, directly filters out the top 8% of responses with the most severe drift, and updates the policy using only the retained responses under the training model's native routing.

Critical strengths
  1. The paper clearly identifies the blind spots of existing approaches: looking solely at output probabilities (e.g., TIS) fails to capture changes in expert paths, whereas locking in old routes (e.g., R3) deprives newly selected experts of gradient updates.

  2. RDC simply applies a binary filtering mask to samples prior to training without modifying the underlying MoE computation graph, making it easy to implement in existing frameworks and directly compatible with TIS.

  3. Under small update batches and asynchronous training, the baseline (GRPO) quickly collapses and R3 degrades in later stages, whereas RDC+TIS maintains stable training dynamics and achieves higher accuracy.

Critical weaknesses
  1. The research motivation overlooks the functional equivalence of different Top-KK expert combinations. The core motivation of this paper assumes that an MoE model activates specific Top-KK experts for different information, and once the Top-KK combination selected during training differs from that in rollout (router drift), it represents harmful off-policy degradation that must be filtered out. In practice, however, an MoE layer selects a subset of KK experts at a time, and many experts in large models exhibit functional overlap or redundancy. Different Top-KK combinations can produce the exact same output and response for the same input. Ironically, in Section 3.4, to prove that "token probabilities do not identify routing," the authors themselves assume two distinct expert combinations R1=S∪{e1}R_1=S\cup\{e_1\} and R2=S∪{e2}R_2=S\cup\{e_2\} that yield the exact same output El,e1(h)=El,e2(h)E_{l,e_1}(h)=E_{l,e_2}(h) and identical token probabilities. Since different Top-KK combinations can produce equivalent outputs, switching to another functionally equivalent expert combination during training is entirely reasonable. Treating all routing disagreements as "harmful drift" and discarding them creates a self-contradiction in the paper's motivation.

  2. The drift metric ignores token positions. In Sections 3.3 and 3.4, the authors analyze how selecting different experts at a single token tt alters local gradient attribution. Yet when actually computing the router drift score Droute(y)D_{\text{route}}(y) in Section 3.2, the metric sums over all time steps t=1,…,Tt=1,\ldots,T, merely counting how many times each expert is used across the entire response while completely ignoring token order. For instance, if a response uses Expert A for the first half and Expert B for the second half during rollout, and completely reverses this during training (Expert B for the first half and Expert A for the second half), the expert selection at every single token has completely changed. However, because the total usage counts of A and B remain unchanged, the drift calculated by Equations (2) and (3) is 0. Furthermore, with training responses reaching up to 20,480 tokens in Section 4.1, averaging across all positions easily smooths out sharp routing changes at critical reasoning steps. The authors do not explain why they do not directly compare expert consistency token by token.

  3. Filtering high-drift samples alters the training distribution and breaks GRPO's within-group balance. Section 3.5 criticizes R3 for introducing gradient bias by reusing old routes, while the abstract and Section 3.2 claim that RDC avoids this bias by filtering out the top q=0.08q=0.08 highest-drift responses. However, these 8% samples are not chosen at random. They are often hard samples the model is actively exploring. Excluding them via the mask mB(y)m_{\mathcal{B}}(y) similarly shifts the training data distribution. More importantly, GRPO relies on within-group balance. As noted in Section 4.1, GRPO computes group-relative advantages across n=8n=8 responses for the same prompt so that positive and negative scores cancel out (maintaining a zero mean). Because Algorithm 1 directly removes samples within batch-level length buckets, if one or two responses for a given prompt are dropped, the remaining responses for that prompt are no longer balanced, introducing a new bias into policy updates.

  4. The asynchronous implementation contradicts the algorithmic description. Section 3.2 explicitly states: "Before every update, we recompute the routing scores and filtering mask using the current training policy." However, when describing the asynchronous training details in Appendix A, the text states: "RDC computes the response-level expert-usage difference once, using the BF16 routing assignments recorded when computing old_log_prob." This indicates that during actual multi-step updates, the model does not dynamically track the latest routing changes before each update; the actual implementation in the appendix does not match the mechanism claimed in Algorithm 1.

  5. The experiments completely lack visualization analysis of individual MoE experts. Although the entire paper centers on MoE expert selection, router drift, and differences in expert usage distributions across responses, the actual experiments in Section 4 only present macro-level curves such as accuracy, reward, gradient norm, and KL divergence, without a single visualization of the internal activation behavior of individual MoE experts. Without heatmaps of layer-wise expert activation distributions or visualizations of routing shifts, readers cannot intuitively see which experts are activated by different responses, how rollout and training expert selections actually differ, or whether the high-drift responses filtered out by RDC genuinely exhibit abnormal expert activations.

Questions for authors
  1. How do you distinguish between "harmful router drift" and "equivalent outputs from different expert combinations"? Since you assume in Equations (9)–(10) that different Top-KK expert combinations can produce identical outputs and token probabilities, why not examine the actual discrepancy in expert output vectors to construct a more fine-grained score?

  2. Why not compare expert assignments token by token? If Equations (2)–(3) in Section 3.2 were modified to compare the overlap between rollout routes Rlμ(t)R_l^\mu(t) and training routes Rlπ(t)R_l^\pi(t) at each individual token tt and then averaged across the sequence, would it perform better than only comparing overall usage proportions?

  3. Does dropping samples break GRPO's score balance? Among the discarded q=0.08q=0.08 samples, what are the respective proportions of correct (reward +1) and incorrect (reward -1) responses? After Algorithm 1 removes a subset of responses, how do you handle the issue where the mean advantage of the remaining responses for the same prompt is no longer zero?

  4. Do the experiments recompute routing scores before every mini-batch update? Please clarify separately for the synchronous multi-step settings in Table 1 (e.g., (2048, 128)) and the asynchronous setting in Table 2. Is a forward pass re-executed before every mini-batch update to obtain the current routes? If so, how much training time overhead does this add? If not, why is Algorithm 1 written as recomputing before every update?

  5. Can you provide visualizations of MoE expert activations? Could you provide visual comparisons of layer-wise expert activation frequencies before and after training, as well as the expert routing trajectories of filtered high-drift responses versus retained normal responses, to intuitively demonstrate expert behavioral differences across responses?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.