- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates the setting of LLM post-training with RL from proxy rewards, such as reward models, and RLVR with multiple reward terms. The authors analyze under which settings errors of the proxy reward may be harmful, beneficial, or not matter. This is followed by experiments on RLHF, which show that accounting for one of the previous insights in the evaluation of reward models simplifies the prediction of post-RL performance. Finally, there is a small experiment considering instruction following with multiple instructions, to consider how we should treat cases with multiple reward terms.
Critical strengths
- the paper is very well written and informative
- I appreciate the clear introduction of the theory, experiments, and takeaways
- particularly the discussion of beneficial reward errors is interesting and not directly obvious
Critical weaknesses
- the scale of the models evaluated in the experiments is quite small (<=3B), but I do not think that this is a major issue as this paper focuses methodological understanding
- some insights, such as "reward errors for low reward actions do not matter" and "reward errors for high reward actions do matter", are not particularly surprising. However, overall I think the paper is still useful.
- the introduction and motivation discuss mathematical reasoning, but it is not investigated in the actual experiments
Questions for authors
- Considering the multi-term experiments, have the authors considered formatting rewards in RLVR? These might be the most commonly used type of multi-term rewards and have been shown to also be prone to be exploited, slowing or preventing convergence [1, Fig. 6] [2, Fig. 6]. They also seem a bit less constructed than the example of multiple instructions in IF-Bench, so their discussion would be helpful.
[1] Zeng et al. "SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild", [2] Ackermann et al, "Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards", ICML 2026
2 comments
I would like to thank the authors for their rebuttal.
My concerns about the scale of the experiments and specificity to instruction following / RLHF persist, but I also still believe that this paper is a useful contribution.
I thus still lean towards acceptance maintain my score of 3.
Thank you for your thoughtful feedback and for raising helpful suggestions. We treat your comments and questions below. If any further questions arise during the discussion period, please let us know and we will gladly elaborate.
Q1 & W3. Thank you for raising this point! Format rewards in RLVR are indeed another common case in practice where partially correct outputs are rewarded. In the terminology of our theoretical analysis, outputs that satisfy the format but are incorrect play the role of “mediocre” outputs. Our analysis therefore provides a formal explanation for the empirical observations that the papers brought up by the review ([1, 2]) made in mathematical reasoning tasks. That is, it implies that when outputs that satisfy the format but are incorrect are rewarded, the policy may concentrate solely on adhering to the format, causing the increase in correctness to stall. We added a discussion of this point to Section 5 of the revised manuscript (marked in blue), citing [1, 2] and relating it to the introduction’s motivating discussion of proxy rewards in mathematical reasoning. More broadly, while the experiments of Section 5 focus on instruction following, partially correct outputs are also rewarded in other settings, for example, by rubric-based rewards that give credit for satisfying a subset of some desired criteria. We believe that further exploring the implications of our theory by considering such settings is a valuable direction for future work.
W2. We agree that some of the categorized reward errors are intuitive. Nonetheless, we believe that formalizing them is still useful for two reasons. First, alongside our main theoretical result on the existence of beneficial reward errors, it provides a more complete overview of possible reward errors. Second, their formalization enabled the design of the HAcc measure in Section 4.