- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper aims to solve a key limitation in standard RL: optimizing for mean reward can cause generative policies to lose coverage over rare, high-reward rollouts. To address this, the authors propose TailRL, an objective that maximizes the expected log-probability of exceeding uniformly sampled reward thresholds. This effectively treats continuous rewards as a family of binary events, up-weighting gradients for rare, high-reward outcomes. The proposed method requires only a simple advantage function modification and outperforms baselines (GRPO, RLOO) in escaping suboptimal shortcuts, learning from low-coverage initial policies, and improving inference-time compute scaling.
Critical strengths
-
The motivation and objective are very principled and Intuitive. It shifts the focus from mean reward to the log-probability of upper-tail events, and transforms continuous rewards into a family of binary events, up-weighting gradients for rare, high-reward outcomes.
-
The proposed method achieves strong empirical validation. The experiments effectively isolate the method's strengths, particularly in the code optimization task (avoiding the "copying" shortcut) and GUI grounding (achieving massive inference compute savings).
Critical weaknesses
None.
Questions for authors
None.
Response to authors
I appreciate the work done by the authors to address my concerns. I'll admit that I'm still a bit shaken by the initial errors made in the earlier version of this work. I would generally err on the side of rejection based on that alone. In addition, I'm not really sure what you can really say statistically with three seeds at all. But I accept that the authors have put in substantial work to address my concerns and I'm not one to be the single hold-out. So I'll raise my score to a weak accept.