- Rating
- 1
- Confidence
- 2(−1)
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper combines adaptive KL control with leave-one-out group advantage estimates for RLHF. It also proposes a rule for mixing advantage estimates and reports gains on several tasks. The experimental section includes variance analysis, benchmark tests, and component comparisons, but there are problems with the research method and presentation.
Critical strengths
- Adaptive KL control addresses policy drift is a importan problem in RLHF.
- The leave-one-out design gives a concrete way to study how group baselines affect advantage estimates.
- The reported component tests help separate the effects of the controller and the advantage rule.
Critical weaknesses
- Appendix B treats exchangeable rewards as independent. This does not hold in general and affects the claim that the covariance is zero.
- Corollary 1 ignores the covariance between estimates based on the same samples. The authors call this a higher-order term, but this does not prove that it is small enough.
- The overall quality and presentation of this paper do not meet the standards required for submission. In particular, several figures and tables are unclear and make the paper hard to read.
Questions for authors
Please respond to the concerns raised in Critical Weaknesses.