- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes a method for handling the distribution mismatch in RL caused by using different inference and training infrastructure, resulting in numerical differences and/or different experts being used in MOE architectures. The proposed method clips the update for a token using top-k bucket KL divergence and another ratio clip. They show that the KL-clip cannot control importance-weight moments hence propose using the second ratio clip. Experiments show that the method outperforms DAPO and GSPO.
Critical strengths
- Useful observation that the KL-clip cannot fully control the importance-weight moments.
- Good performance in experiments.
Critical weaknesses
- Limited novelty. Clipping by divergence has been studied in prior works, although the observation that it does not fully control the importance-weight moment could be more novel.
- Weak baselines. Experiments do not compare with recent methods, e.g. DPPO, which according to the authors is the closest to this work.
Questions for authors
- How would the method compare to more recent methods such as DPPO?
4 comments
Thanks for your experimental comparison with DPPO. Would you be able to clarify the novelty of your second clip against DPPO's truncated importance sampling?
Dear Reviewer fSZD,
Thank you for your following-up question. We clarify the differences between the second clip and DPPO's truncated importance sampling below.
(1) The second clip treats the two advantage signs differently. TIS uses the same coefficient for positive- and negative-advantage tokens; DPPO's scaling experiments use (DPPO arXiv v2, Eq. 23 and Appendix F). The second clip uses an upper cutoff for . For , it uses a lower cutoff together with the upper dual-clip cutoff (Section 2.2).
The table compares effective importance coefficients for tokens admitted by the divergence masks.
| Advantage | DPPO's TIS | The second clip |
|---|---|---|
| for ; 0 for | ||
| 0 for ; for ; 0 for |
(2) The second clip removes the direct gradient on saturated branches. For an unmasked token with , TIS retains its direct policy-gradient contribution with coefficient . On our clipped constant branches, the objective no longer depends on , so that token's direct policy-gradient contribution is zero. Thus, the two mechanisms share a coefficient-bounding role but produce different updates.
The second clip removes direct gradient contributions on saturated branches instead of retaining TIS's nonzero weights up to . It also stops further direct suppression of negative-advantage tokens below , where TIS's upper cap is inactive. These safeguards match our design expectations; our theory motivates controlling effective update coefficients separately from distributional drift (Prop. 3.2; Thm. 3.3).
We hope our response has addressed your concerns. We know your time is valuable, and we would be very grateful for any further questions or suggestions.
Dear Reviewer fSZD,
We appreciate your comments. Below, we respond to your concerns.
W2&Q (DPPO and recent baselines)
We have added DPPO TV and DPPO KL at R0. We align the experimental settings with ours as closely as possible; DPPO KL uses top-16, the same as ours. VoKL-Clip uses different divergence thresholds for positive and negative advantages, whereas DPPO uses a shared threshold, so this aspect cannot be matched exactly. We also ran DPPO KL with R3, following the settings in the original DPPO paper. We will provide the full hyperparameter settings in the appendix. We report VoKL-Clip at its paper checkpoint. For each DPPO variant, we select the checkpoint with the highest AIME24 score, following the AIME24-based selection criterion used in our paper.
VoKL-Clip continues to lead on five of the six benchmarks. These results further demonstrate the effectiveness of our method.
W1 (novelty)
Thank you for highlighting our observation that divergence control alone cannot globally bound the second moment of importance weights. This limitation motivates our study of RL stability, particularly for MoE models. We investigate double clipping at R0 and provide a detailed theoretical analysis. We would like to reiterate our main contributions: