- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper mainly studies the off-policy errors in reinforcement learning caused by the mismatch between the behavior policy and the training policy. Similar to methods such as TRPO, this paper follows the approach of constraining the magnitude of policy updates. The main contribution of the paper is combining the TV divergence constraint of DPPO with the smooth objective function of SPO. The experimental results show that the proposed method is effective.
Critical strengths
- The off-policy error problem studied in this paper is a classical problem in reinforcement learning. It is valuable and broadly applicable.
- The paper cleverly uses the quadratic objective function of SPO to implement a smooth constraint, while adopting the TV constraint of DPPO to address the scale mismatch problem of KL.
- The experimental results are good, and the method is relatively clean and easy to reproduce.
Critical weaknesses
- The contributions of the paper are limited. The main method simply combines the smooth constraint of SPO with the TV constraint of DPPO. Essentially, it is a combination of two existing methods, making the contribution relatively limited. The authors could have strengthened the contribution by providing more theoretical analysis. However, the current theoretical section mainly analyzes the properties of DRPO and lacks a more substantial contribution that directly and systematically explains, in an end-to-end manner, why DRPO leads to improved RL performance. The current version looks more like an incremental improvement based on combining existing methods.
- The experiments could be further expanded. Although AIME24 and AIME25 are both mainstream benchmarks, they are still too similar, which may limit the evaluation perspectives. Adding other benchmarks, such as Olympiad and HMMT (just as examples), would strengthen the credibility of the experimental results.
- Some derivations are not sufficiently rigorous. For example, the probabilities given by Equation 11 may not sum to 1, meaning that they do not form a valid probability distribution. The authors should carefully check the mathematical rigor throughout the paper and make corrections. Taking this case as an example, the authors could obtain a more rigorous solution using Lagrange multipliers. They would only need to add a constant to Equation 11 (when delta is sufficiently small) and introduce clipping (when delta is sufficiently large).
Questions for authors
What do you consider to be the main contributions of this submission?