- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies why mixed-policy RL collapses when an external correct trace is added to the GRPO rollout group and the previous policy is used as a proxy for the unknown behavior policy. The authors argue that the cause is not importance-ratio variance but a conflict between the gradients of low-probability external tokens and those of the rest of the batch. They show that LUFFY and RL-PLUS can both be read as implicitly suppressing these gradients, and propose a simple method that masks external tokens below a probability threshold. Experiments on four backbones show results competitive with LUFFY and RL-PLUS.
Critical strengths
- Clear and useful diagnosis. The interval-swap ablation (Table 3) is a clean single-variable test: keeping Vanilla IS on low-probability tokens always collapses training, while replacing it there always stabilizes it.
- The unified view is predictive. The framework correctly anticipates the outcome of three newly designed ratio variants (Table 4), which goes beyond re-explaining existing methods.
- The method adds essentially no overhead and is evaluated on four backbones and nine benchmarks, including out-of-domain ones.
Critical weaknesses
- Relation to prior work is underdeveloped. The gradient bound in Eq. 4 and the idea of down-weighting low-probability tokens overlap considerably with Lopti (Yang et al., ICLR 2026), which is not discussed.
- The gradient-conflict evidence could be more controlled. The low-probability external tokens all have positive advantage, while the remaining tokens include many negative-advantage rollouts, so part of the negative cosine in Figure 4 may reflect this difference rather than low probability itself.
- Gains are modest on larger models. On Qwen2.5-Math-7B the method improves over RL-PLUS by only 0.2 points (Table 2), and results are from single runs, which makes the advantage hard to judge.
- Minor issues. The introduction mentions five backbones, but only four are used in the experiments.
Questions for authors
See weakness. I will raise my score if my concern is addressed.