acceptodds
Under review as a conference paper at ICLR 2027

Reward Shaping to Mitigate Reward Hacking in RLHF

Abstract

Reinforcement learning from human feedback (RLHF) is widely used to align large language models (LLMs) with human preferences. However, RLHF remains vulnerable to reward hacking, whereby a policy exploits imperfections in the reward function instead of learning the intended behavior, thereby undermining alignment. Although reward shaping can stabilize RLHF training and partially mitigate reward hacking, shaping methods and their underlying design principles have not been systematically investigated. To address this gap, we conduct a comprehensive study of prevalent reward-shaping techniques. Our analysis identifies two key design principles: (1) the reinforcement-learning reward should be bounded, and (2) it should grow rapidly at first and then gradually saturate. Motivated by these principles, we propose Preference as Reward (PAR), a novel method that uses the latent preferences encoded in the reward model as the reinforcement-learning signal. We further show that PAR possesses two variance-reduction properties that stabilize RLHF training and substantially widen the practical window for early stopping. Our evaluation consists of two parts. First, we compare PAR with several other reward-shaping strategies using Proximal Policy Optimization (PPO) as the reinforcement-learning algorithm and Gemma2-2B as the base model. Second, we compare PAR with the vanilla baseline (i.e., unshaped reward) across four base models and four reinforcement-learning algorithms. In the first set of experiments, PAR consistently outperforms other reward-shaping methods and also reflects high data efficiency and robustness. The second set of experiments shows that PAR is particularly effective for actor-critic RL algorithms when value estimates become unstable and demonstrates its effectiveness across different base models. The code will be released after the review process.

Primary area:
reinforcement learning
Keywords:
RLHF, Reward Hacking, Reward Shaping

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Ee7GOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Preference as Reward (PAR), a reward-shaping algorithm intended to mitigate reward hacking in RLHF. However, significant concerns remain regarding the strength of the theoretical guarantees, the novelty of the proposed approach, and whether the empirical results sufficiently support the authors’ claims.

Critical strengths

None

Critical weaknesses

Unclear connection between preference probability and reward hacking. The paper does not adequately explain why modeling the preference probability of a policy response relative to a reference response should directly mitigate reward hacking. While the proposed transformation may bound the reward signal, the authors do not establish a convincing between this construction and the prevention of reward hacking.

Theorem 3.1 does not establish reward-hacking mitigation. The theorem establishes a bound on the variance of returns under bounded rewards. However, bounded variance alone does not imply reduced reward hacking, improved alignment with human preferences, or more reliable policy optimization. The result is a generic consequence of bounded rewards rather than a guarantee specific to PAR. The paper does not sufficiently bridge the gap between this variance bound and its central claim of mitigating reward hacking.

Theorem 3.2 provides limited theoretical insight. The theorem appears largely tautological: a deterministic reward estimator has minimum variance compared with stochastic estimators having the same conditional expectation.

Weak and inconclusive empirical evidence. The experimental results are not sufficiently convincing to support the paper's broad claims. The evaluation does not clearly establish statistical significance or robustness across random seeds, and the reported mitigation is inconsistent across settings. Moreover, the experiments do not convincingly isolate the benefits of PAR from those of bounded reward transformations more generally.

Questions for authors

See weaknesses

Reviewer_tZYeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Preference as Reward (PAR), which converts reward-model scores into normalized Bradley-Terry preference probabilities compared to SFT reference responses. It gives arguments on variance reduction and analyzes PAR compared to baselines of reward shaping, different models, and RL algorithms. The most notable result is higher stability during extended RLHF optimization, rather than peak performance increase.

Critical strengths

Practical empirical observation. Figure 10 illustrates that PAR retains a high win rate during long training, whereas Minmax and WARM drop. The larger training window that PAR has is practically relevant. Extensive analysis with useful ablations. The paper compares multiple reward-shaping techniques, studies centering and saturation, and conducts evaluation with respect to various models and algorithms. At the same time, the paper recognizes that PAR provides only a small improvement in peak performance and is not universally useful.

Critical Weaknesses

  1. Evidence does not distinguish reward hacking mitigation from avoidance of optimization collapse. PPO almost collapses in Table 1; however, WARM and Minmax avoid this problem too. While the largest clear advantage of PAR is slower drop during long training (Figure 10), rather than peak performance increase, fixed-epoch comparisons might be misleading. The authors should compare best checkpoints according to the same validation rule and report the area under training curves (AUC). Independent evaluation of the output quality would also help to distinguish between reward hacking mitigation and training stability in general.
Critical weaknesses

According to the paper's own ablations, there is a simple alternative explanation for PAR effectiveness: centered tanh, polynomial, and sigmoid transformations behave similarly (Figure 4); one reference response is enough (Figure 9); and reward ceiling already increases stability (Figure 3). Those results suggest centering and bounded saturation as the possible factors, rather than Bradley-Terry preference probabilities. Theorem 3.2 assumes sigmoid as a conditional mean and thus cannot prove that it is superior to other transformations. The assumptions in Theorem 3.1 also do not coincide with those in the PPO setup in the experiment. Furthermore, effects of reward scaling, effective KL regularization, and sample re-weighting are still not controlled. More comparisons with matched simpler baselines and achieved/KL normalized critic diagnostics are required to justify PAR's distinctiveness.

Questions for authors

Is there any significant advantage of PAR after a fair checkpoint selection? Could the authors compare best checkpoints according to a consistent validation rule, calculate AUC on training curves, and conduct independent evaluations of output quality? Is there any evidence that separates PAR from simpler centered bounded transformations? In particular, could the authors compare hard ceiling or centered tanh variants, reward-scale schedules, or reward re-weighting based on saturation, and report achieved KL and scale-normalized critic diagnostics? Please also limit the variance theorems to their proven conclusions.

Reviewer_tuJrOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies reward transformations for mitigating reward hacking in RLHF. It proposes two design principles: bounded rewards and rapid initial growth followed by gradual saturation. Based on these principles, the authors introduce Preference As Reward (PAR), a simple and effective reward-shaping method that mitigates reward hacking.

Critical strengths

The paper addresses the important problem of reward hacking in RLHF and proposes a simple and effective reward-shaping method that is easy to implement and integrate into existing training pipelines.

Critical weaknesses

W1. Insufficient theoretical support: The two proposed design principles are primarily based on empirical observations. The theoretical analysis does not sufficiently justify their general applicability beyond the evaluated settings.

W2. Bounded rewards do not rule out reward hacking: Bounding rewards does not distinguish genuine quality gains from exploitation. Saturation may improve training stability but also weaken incentives for genuine improvement, potentially limiting peak performance.

W3. Incomplete causal evidence: The experiments support improved stability and sustained performance, but do not directly establish the proposed causal chain from variance reduction to training stability and reduced reward hacking.

Questions for authors

Q1. Comparison with centered clipping: Could the authors compare PAR with directly clipping the centered reward, (R=\operatorname{clip}(r-r_{\mathrm{ref}},0,1))? Both approaches use reference centering and bounded rewards. What additional benefit does sigmoid provide?

Q2. Relationship to the earlier workshop paper: We noticed a closely related paper with the same title at an ICML 2025 workshop. Could the authors clarify how the current submission differs from that paper and identify its new methodological, theoretical, or experimental contributions?

Comments on the paper

1 comment
Area_Chair_8cvAOpenReview ↗

Reviewers: please read and comment on author's feedback, and consider editing your rating.

Dear Reviewers,

The authors have replied to your reviews. Does the author solve your concerns? Please respond to the author and consider editing your rating. The author-reviewer discussion phase will end on November 18 at 11:59pm AoE. Thanks.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.