- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Large language models are often trained with reinforcement learning using two separate systems: one that generates responses (rollout) and one that updates the model (training). Small numerical differences between these systems, plus training on rollout data that is a few updates old, mean the policy that generated a response is not quite the same as the policy being trained, even though the setup is meant to be on-policy. In Mixture-of-Experts models, where each token is routed to only a handful of expert sub-networks chosen by a router, this mismatch can make the router select different experts for the same token during generation versus during training, which the paper calls router drift. Since only the selected experts receive a learning signal for a given token, drift changes which parts of the model actually get trained. A prior method fixes this by forcing training to reuse the exact experts chosen during generation, but the paper shows this introduces a different problem: it discards the learning signal for experts the training model would otherwise have chosen on its own. The paper's method instead measures, for each response, how much the training-time routing differs from the generation-time routing, and filters out only the most mismatched responses, leaving the rest to train normally with their own routing. Tested on math and logic reasoning tasks, this filtering approach gives larger gains as the mismatch grows, and keeps training from collapsing in a fully asynchronous setup where generation and training run at different speeds.
Critical strengths
- The paper is well written and easy to follow.
- The problem statement is well motivated.
- The proposed algorithm, Router Drift Control, is a sensible design. It filters high-drift responses and leaves the routing of the remaining responses unchanged.
- The explanation of MoE routing, and how the method differs from R3, is clear and helps justify the approach.
- The paper reports training dynamics such as reward, gradient norm, and response length, showing that unmodified training collapses under asynchronous conditions while the method's advantage grows as staleness increases.
Critical weaknesses
- The results come from a single model and a single training dataset, and hyperparameters such as the filtering ratio and the TIS threshold are reported as fixed values without an explained tuning process. Even within this one setting the baselines sometimes outperform RDC, so it is unclear how much of the result would generalize to a different model, dataset, or hyperparameter search.
- There is no ablation isolating RDC's effect. All reported RDC results are combined with TIS, so there is no result for RDC applied alone or for R3 combined with TIS, and no sensitivity analysis is reported for the method's hyperparameters.
- The filtering signal is a coarse, response-level average of expert-usage counts, while the proof of gradient bias concerns specific tokens, so a response could score as low-drift even with substantial token-level mismatch. The paper also never reports how much of each batch is actually judged high-drift or how that amount changes with staleness, which would show directly how much drift occurs between rollout and training rather than just the effect of filtering it.
Questions for authors
-
Why doesn't RDC perform well in some cases? What are its limitations?
-
How was the filtering ratio of 8% and the TIS threshold of 2.0 chosen? Were they tuned on the same benchmarks being reported, or on a separate setup?
-
Can you report RDC applied without TIS, to separate its effect from TIS's contribution?
-
Can you report R3 combined with TIS, so it is compared on equal footing with RDC+TIS?
-
What fraction of responses in a batch are filtered as high-drift at different points in training, and how does this fraction change with staleness?
-
How do you results generalize across different models and datasets?