- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies how to train a reward model given a dataset with selectively observed user feedback. This paper designs a heuristic method that combines propensity clipping with a conservative regularization to constrain reward extrapolation in low-support regions. Experiments in this paper demonstrate its advantage in reward estimation and downstream RLHF safety performance on three datasets that do not include selection bias, but on which selection bias was simulated.
Critical strengths
The paper studies an interesting problem in reward modeling: selection bias may exist in principle and may affect the generalizability of the learned reward models.
The proposed method is intuitive and can be integrated with IPS and DR estimators.
Critical weaknesses
Lack of soundness. The computation of reward intervals seems self-referential. The method uses the model's own reward landscape to determine whether its predictions in another region are reliable. This seems to be problematic. In addition, does being "well-supported" imply more reliable reward estimation? This question also lacks justification.
Insufficient Evidence. The paper’s motivation appears to follow: Sparse feedback → Spurious reward peaks → Policy exploitation. The experiments only show the performance, but not follow the motivation at all. In addition, assumption 1 seems overly strong, and there are no empirical support to its practical feasibility. By the way, the paper does not provide sensitivity analyses on the two key hyperparameters
Questions for authors
The paper aims to address long-tailed observational user feedback, but it only evaluates on simulated datasets instead of real-world observational feedback. This is like developing a car and testing it by simulation, without ever testing its safety under real driving conditions. Would consumers trust such a car?
It seems the paper lacks implementation and experimental details to ensure reproducibility and does not seem to release the related code. Will the related code be released? Are the results in this paper reproducible?