acceptodds
Under review as a conference paper at ICLR 2027

Tail-Likelihood Reinforcement Learning

Abstract

Reinforcement learning typically optimizes average reward. For generative policies, the average can hide an important distinction: two policies can achieve the same mean reward while having very different chances of producing a rare but high-reward rollout. This matters as sampling increases during training and inference, since its benefit depends on retaining probability mass on high-reward outcomes. We propose to optimize this coverage directly. Rather than considering only expected reward, we consider all of its upper tails: for each reward threshold, how likely is the policy to exceed it? This turns a continuous reward into a family of binary success events. We introduce Tail-Likelihood Reinforcement Learning (TailRL), which maximizes the log-probability of exceeding a randomly chosen reward threshold. Its gradient gives more weight to rare, high-reward rollouts and can be interpreted as a mixture of Best-of- gradients. TailRL requires only a simple modification to the advantage function, making it compatible with existing reinforcement learning pipelines. Across object localization, maze navigation, GUI grounding, and code optimization, TailRL leverages rare high-reward training samples to avoid suboptimal solutions and yields models that benefit more from additional samples at inference time.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Large Language Models, Tail-Likelihood

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_g1LBOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper aims to solve a key limitation in standard RL: optimizing for mean reward can cause generative policies to lose coverage over rare, high-reward rollouts. To address this, the authors propose TailRL, an objective that maximizes the expected log-probability of exceeding uniformly sampled reward thresholds. This effectively treats continuous rewards as a family of binary events, up-weighting gradients for rare, high-reward outcomes. The proposed method requires only a simple advantage function modification and outperforms baselines (GRPO, RLOO) in escaping suboptimal shortcuts, learning from low-coverage initial policies, and improving inference-time compute scaling.

Critical strengths
  1. The motivation and objective are very principled and Intuitive. It shifts the focus from mean reward to the log-probability of upper-tail events, and transforms continuous rewards into a family of binary events, up-weighting gradients for rare, high-reward outcomes.

  2. The proposed method achieves strong empirical validation. The experiments effectively isolate the method's strengths, particularly in the code optimization task (avoiding the "copying" shortcut) and GUI grounding (achieving massive inference compute savings).

Critical weaknesses

None.

Questions for authors

None.

Reviewer_BuuhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors present a method to directly optimize the upper tails of the reward distribution, theoretically weighting rare, high rewards more effectively. They show empirically that this induces a prioritization of large but rare rewards.

Critical strengths
  • The work includes quite a good array of both theoretical and empirical results. I cannot find anything obviously wrong with them.
  • There is quite a bit to gain in employing more sophisticated methods than GRPO on training large language models. Given how importance these methods have been shown to be in the era of language models, I greatly appreciate work that improves on them.
Critical weaknesses
  • There are a lot of problems in the writing here. The canary here is that the first sentence of the abstract misuses an established term. There is a quite extensive sub-domain of reinforcement learning called "average-reward reinforcement learning," and it does not mean what the authors use it to mean here (do the authors mean the expected reward of a single-step generative policy? that is the only standard case outside of average-reward RL where maximizing the mean reward is equivalent to maximizing the expected return). I would advise the authors of this work to review Sutton & Barto, as this is a very problematic oversight and the terminology needs to be improved.
  • There seems to be an insufficient treatment of prior work here more generally. Additional properties of returns have been studied quite extensively. I noted that the authors relegated distributional RL to the appendix, despite its exceptional relevance to this work. I do not think that is really sufficient here, as this work realistically needs to be carefully positioned in relation to such things and not merely mention it as a generally unrelated idea---which is what a relegation to an appendix for related work means. There have also been several other works studying specific properties of the return (work on estimating the variance (see, e.g., Comparing Direct and Indirect Temporal-Difference Methods for Estimating the Variance of the Return) or various moments (see, e.g., Incrementally Learning Functions of the Return)) which are seemingly completely absent from the text. The fact that (with exceptions of REINFORCE), all the citations in the introduction are from the past two years epitomizes this problem.
  • They also neglect to position this amongst all the work on handling sparse rewards (e.g., the exploration literature, which is likewise only discussed in the appendix), which seems to me to be the core goal of this method.
Questions for authors
  • In what situations would this fail as compared to GRPO?
  • Would this lead to situations where a low but common reward that was, in expectation, larger than a rare but high reward is not preferred over the high reward? If so, is this a likely situation we would ever encounter?
  • Are the results presented here statistically significant?

Please also respond to the points raised in the weaknesses section.

1 comment
Reviewer_BuuhOpenReview ↗

Response to authors

I appreciate the work done by the authors to address my concerns. I'll admit that I'm still a bit shaken by the initial errors made in the earlier version of this work. I would generally err on the side of rejection based on that alone. In addition, I'm not really sure what you can really say statistically with three seeds at all. But I accept that the authors have put in substantial work to address my concerns and I'm not one to be the single hold-out. So I'll raise my score to a weak accept.

Reviewer_CBM6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents TailRL, a policy gradient objective for continuous rewards. Each reward threshold as treated as a binary event: "does the reward exceed the threshold?". The approach then maximizes the average log-probabilities of these events over a uniform draw. Standard RL integrates the tail probabilities themselves. TailRL integrates their logarithms, so hard-to-reach thresholds get gradient weight 1/p.

Evaluation on a set of diverse environments (e.g., object localization, mazes, code improvement) shows strong results compared to GRPO and RLOO.

The claimed contributions are:

  1. A likelihood objective for continuous rewards
  2. A compute scaling alignment.
  3. Unbiased gradient estimator - With N rollouts, a simple sort-based advantage gives an unbiased gradient
  4. Strong empirical results
Critical strengths
  1. A simple and effective method - the approach is intuitive and easy to implement.

  2. Strong empirical results - while this varies across datasets, some results are strong (code improvement, for example)

  3. A through intialization sweep that helps in framing the approach's contribution

Critical weaknesses
  1. Incremental paper/limited novelty - the contribtuions of the paper, although accurate, are a direct continuation of MaxRL (the authors are clear regarding this fact). Many of the paper's theorem follow directly from MaxRL. Moreover, MaxRL clearly outline the directions this paper takes. This leaves me wondering whether this paper stands on its own, or is it simply an extension of an existing work.

  2. Missing a critical ablation - the authors claim they address a limitation of MaxRL (continuous rewards), yet they do not compare against MaxRL on binarized rewards. This seems to me as a logical and necessary step to show the merit (and need) of the proposed approach. Another very relevant baseline - ODRPO - is mentioned by the authors, but not compared against. ODRPO with MaxRL normalization would go a long way in showing the need for the proposed approach (if justified by the results).

  3. Lack of statistical rigor - seed counts are unclear, and I couldn't find a clear explanation on the shaded regions in some of the graphs. There are no reports on standard deviation, and statistical significance tests are not reported.

Questions for authors
  1. The authors are welcome to address my concerns above

  2. Was the learning rate tuned separately for each method? Do you normalize TailRL advantages by any group statistic?

  3. How many seeds underlie each figure, and what do the shaded regions show?

Comments on the paper

1 comment

Dear Reviewers

The author-reviewer discussion period will close soon. The authors put much efforts for their rebuttal. Please read and respond to authors if you have not done yet.

Thank you.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.