- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes Preference as Reward (PAR), a reward-shaping algorithm intended to mitigate reward hacking in RLHF. However, significant concerns remain regarding the strength of the theoretical guarantees, the novelty of the proposed approach, and whether the empirical results sufficiently support the authors’ claims.
Critical strengths
None
Critical weaknesses
Unclear connection between preference probability and reward hacking. The paper does not adequately explain why modeling the preference probability of a policy response relative to a reference response should directly mitigate reward hacking. While the proposed transformation may bound the reward signal, the authors do not establish a convincing between this construction and the prevention of reward hacking.
Theorem 3.1 does not establish reward-hacking mitigation. The theorem establishes a bound on the variance of returns under bounded rewards. However, bounded variance alone does not imply reduced reward hacking, improved alignment with human preferences, or more reliable policy optimization. The result is a generic consequence of bounded rewards rather than a guarantee specific to PAR. The paper does not sufficiently bridge the gap between this variance bound and its central claim of mitigating reward hacking.
Theorem 3.2 provides limited theoretical insight. The theorem appears largely tautological: a deterministic reward estimator has minimum variance compared with stochastic estimators having the same conditional expectation.
Weak and inconclusive empirical evidence. The experimental results are not sufficiently convincing to support the paper's broad claims. The evaluation does not clearly establish statistical significance or robustness across random seeds, and the reported mitigation is inconsistent across settings. Moreover, the experiments do not convincingly isolate the benefits of PAR from those of bounded reward transformations more generally.
Questions for authors
See weaknesses
Reviewers: please read and comment on author's feedback, and consider editing your rating.
Dear Reviewers,
The authors have replied to your reviews. Does the author solve your concerns? Please respond to the author and consider editing your rating. The author-reviewer discussion phase will end on November 18 at 11:59pm AoE. Thanks.
AC