acceptodds
Under review as a conference paper at ICLR 2027

When Errors Can Be Beneficial: A Categorization of Imperfect Rewards for Policy Gradient

Abstract

Training language models via reinforcement learning often relies on imperfect proxy rewards, since ground truth rewards that precisely define the intended behavior are rarely available. Standard metrics for assessing the quality of proxy rewards, such as ranking accuracy, treat incorrect rewards as strictly harmful. In this work, however, we highlight that not all deviations from the ground truth are equal. By theoretically analyzing which outputs attract probability during policy gradient optimization, we categorize reward errors according to their effect on the increase in ground truth reward. The analysis establishes that reward errors, though conventionally viewed as harmful, can also be benign or even beneficial by preventing the policy from stalling around outputs with mediocre ground truth reward. We then present two practical implications of our theory. First, for reinforcement learning from human feedback (RLHF), we develop reward model evaluation metrics that account for the harmfulness of reward errors. Compared to standard ranking accuracy, these metrics typically correlate better with the performance of a language model after RLHF, yet gaps remain in robustly evaluating reward models. Second, we provide insights for reward design in settings with verifiable rewards. A key theme underlying our results is that the effectiveness of a proxy reward function depends heavily on its interaction with the initial policy and learning algorithm.

Primary area:
learning theory
Keywords:
Policy gradient, Proxy rewards, Language models, Theory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_dYcsOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the setting of LLM post-training with RL from proxy rewards, such as reward models, and RLVR with multiple reward terms. The authors analyze under which settings errors of the proxy reward may be harmful, beneficial, or not matter. This is followed by experiments on RLHF, which show that accounting for one of the previous insights in the evaluation of reward models simplifies the prediction of post-RL performance. Finally, there is a small experiment considering instruction following with multiple instructions, to consider how we should treat cases with multiple reward terms.

Critical strengths
  • the paper is very well written and informative
  • I appreciate the clear introduction of the theory, experiments, and takeaways
  • particularly the discussion of beneficial reward errors is interesting and not directly obvious
Critical weaknesses
  • the scale of the models evaluated in the experiments is quite small (<=3B), but I do not think that this is a major issue as this paper focuses methodological understanding
  • some insights, such as "reward errors for low reward actions do not matter" and "reward errors for high reward actions do matter", are not particularly surprising. However, overall I think the paper is still useful.
  • the introduction and motivation discuss mathematical reasoning, but it is not investigated in the actual experiments
Questions for authors
  • Considering the multi-term experiments, have the authors considered formatting rewards in RLVR? These might be the most commonly used type of multi-term rewards and have been shown to also be prone to be exploited, slowing or preventing convergence [1, Fig. 6] [2, Fig. 6]. They also seem a bit less constructed than the example of multiple instructions in IF-Bench, so their discussion would be helpful.

[1] Zeng et al. "SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild", [2] Ackermann et al, "Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards", ICML 2026

2 comments

Thank you for your thoughtful feedback and for raising helpful suggestions. We treat your comments and questions below. If any further questions arise during the discussion period, please let us know and we will gladly elaborate.

Q1 & W3. Thank you for raising this point! Format rewards in RLVR are indeed another common case in practice where partially correct outputs are rewarded. In the terminology of our theoretical analysis, outputs that satisfy the format but are incorrect play the role of “mediocre” outputs. Our analysis therefore provides a formal explanation for the empirical observations that the papers brought up by the review ([1, 2]) made in mathematical reasoning tasks. That is, it implies that when outputs that satisfy the format but are incorrect are rewarded, the policy may concentrate solely on adhering to the format, causing the increase in correctness to stall. We added a discussion of this point to Section 5 of the revised manuscript (marked in blue), citing [1, 2] and relating it to the introduction’s motivating discussion of proxy rewards in mathematical reasoning. More broadly, while the experiments of Section 5 focus on instruction following, partially correct outputs are also rewarded in other settings, for example, by rubric-based rewards that give credit for satisfying a subset of some desired criteria. We believe that further exploring the implications of our theory by considering such settings is a valuable direction for future work.

W2. We agree that some of the categorized reward errors are intuitive. Nonetheless, we believe that formalizing them is still useful for two reasons. First, alongside our main theoretical result on the existence of beneficial reward errors, it provides a more complete overview of possible reward errors. Second, their formalization enabled the design of the HAcc measure in Section 4.

I would like to thank the authors for their rebuttal.

My concerns about the scale of the experiments and specificity to instruction following / RLHF persist, but I also still believe that this paper is a useful contribution.

I thus still lean towards acceptance maintain my score of 3.

Reviewer_JW9LOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how imperfect proxy rewards affect policy gradient optimization, challenging the assumption that reward errors are always harmful. Through theoretical analysis, the authors categorize reward errors as harmful, benign, or beneficial, showing that certain errors can accelerate learning by preventing the policy from concentrating on suboptimal outputs. Building on these findings, they propose harm-aware ranking accuracy (HAcc) for reward model evaluation and explore the implications for reward design in verifiable-reward settings. Experiments across multiple LLMs provide empirical support for these findings.

Critical strengths
  1. The paper challenges the common assumption that reward errors are always harmful and offers a new perspective on evaluating imperfect rewards.

  2. The paper provides rigorous theoretical results explaining when reward errors can be harmful, benign, or even beneficial for policy gradient optimization.

Critical weaknesses

My main concerns are related to the proposed HAcc metric.

  1. The theoretical analysis suggests that the impact of reward errors depends on policy dynamics and feature interactions. However, HAcc only uses a simple expected-reward threshold, and it is unclear how well this captures the factors identified in the theory.

  2. Moreover, although HAcc generally performs better than standard ranking accuracy, its correlation with downstream RLHF performance remains weak and sometimes even negative. This makes me question its practical usefulness for reward model evaluation and selection.

Questions for authors

None

3 comments

Thank you for your thoughtful feedback and for recognizing the value of our theoretical results!

Both weaknesses mentioned in the review pertain to the harm-aware accuracy (HAcc) measure discussed in Section 4 of our work. First, regarding W2, we would like to emphasize that the main contribution of the paper is the theoretical categorization of reward errors according to their effect on policy gradient optimization. The proposed HAcc measure and experiments in verifiable domains are intended as demonstrations of the theory’s potential utility. In particular, we do not claim that HAcc is a robust measure that is ready to be used for selecting reward models in practice. Rather, we claim that HAcc improves upon standard accuracy, yet robustly evaluating reward models remains an important open problem for future work. This point is made in the abstract, introduction, and Section 4. To further clarify it, in light of your comments, we have moved the limitations and future work appendix, which includes a discussion of potential barriers toward robust reward model evaluation, into the main text as part of the conclusion section (changes marked in blue).

Regarding W1, the ability to categorize a reward error in practice heavily depends on the available information. In the context of RLHF, the ground truth reward is unavailable and access to it is limited to output rankings. Thus, as discussed in Section 4.1, it is not possible to determine for every incorrect ranking whether it is harmful or not. HAcc is based on the parts of our theory that allow identifying in practice types of incorrect rankings that are unlikely to be harmful. The threshold based on the initial expected reward follows from Proposition 1. Namely, outputs with proxy reward below the initial expected proxy reward do not attract probability (Benign Error I), and by Theorem 1 such errors may even be beneficial when the preferred output is mediocre (Beneficial Error I). The weighted variants follow from Proposition 2 (Benign Error II), which states that outputs with small probability have negligible effect on the outcome of policy gradient.

Furthermore, it is worth noting that there is a tradeoff between the cost of a reward model evaluation measure and its predictive power. With no computational cost concerns, one could train with every candidate reward model and select the one that works best. Thus, our aim was to provide a measure that can be computed efficiently at initialization, like standard ranking accuracy. The quantities that HAcc relies on are indicative of the training dynamics and can be computed at a cost comparable to ranking accuracy (see computational cost discussion in Appendix E.2). Nonetheless, we agree that further leveraging information on the training dynamics may be useful, for example, by examining the policy after a brief warmup training phase. We believe that exploring such evaluation measures, and whether they can go beyond the limitations of current ones at a reasonable cost, is an exciting direction for future work.

Conclusion:

Thank you for taking the time to read our response. If it addresses your concerns, we would greatly appreciate it if you would consider raising your score. Otherwise, please let us know if you have any further questions, and we will gladly respond.

Dear reviewer,

We are grateful for your feedback and thank you again for the time and effort in reviewing our paper. We believe that our response fully addresses the points raised in the review. Since the discussion period is nearing its end, we kindly ask that you let us know whether you have any remaining questions or comments.

Thank you!

Thank you for the response. The clarification of the theoretical contributions and the motivation behind HAcc addresses my main concerns. While the practical effectiveness of HAcc remains limited, I find the theoretical analysis of imperfect rewards valuable and relevant to reward design and evaluation. I will raise my rating to Weak Accept.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.