- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the unstable importance-sampling (IS) updates in existing RL methods that stem from their off-policy nature, and proposes using structured Gaussian noise to ‘fight’ the random noise introduced by off-policy mismatch.
Critical strengths
The experiments are thorough, and the method is simple to apply and gives consistent gains across a variety of settings, which supports its effectiveness. My main concerns are about the motivation.
Critical weaknesses
- The paper does not directly show how much IS instability actually hurts training. How large is the mismatch between inference and training in practice? At what level does it start to hurt performance, and below what level can the model stabilize itself?
- The single-turn RL experiments are mainly run on small models (1.5B/4B). On larger models, is this IS instability perhaps negligible for training?
- If the problem lies in IS instability, an extreme design would be to replace π_old with stop gradient(π_θ), so that the ratio is stably equal to 1. Would that also work?
- For these reasons, I don’t think “fighting noise with structured noise” is a natural choice, and the authors need to discuss the motivation further.
- Minor: I did not get the purpose of showing the rollout probability in Figures 1 and 2.
Questions for authors
See weaknesses above.
Dear all, The discussion phase will end soon. The authors have provided their responses to your comments. Some of you have started involving discussion, but all of us can take action now. If the provided responses do not address (even part of) your concerns, this is your chance to raise it. Please engage with the authors as soon as possible. We really appreciate your efforts! Thanks, AC