- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether a policy learned through the KL-regularized RLHF objective can perform well on prompts that are not seen during training. Previous theoretical work mainly estimates the reward parameters from data and plugs them into the closed-form optimal policy, which requires the data to cover all directions of the feature space and assumes that the optimization is solved exactly. Instead, the authors directly analyze the learned policy through algorithmic stability, under a linear reward model, a Boltzmann-form policy class and the assumption that the true reward is available. They show that, under a feature coverage condition that only requires new features to be approximately covered by the training data, the suboptimality gap of empirical stationary points decreases at a rate of order one over the square root of the sample size, without explicit dependence on the feature dimension. They further extend the results to gradient ascent and stochastic gradient ascent. Simulation experiments are provided to illustrate the theoretical rates.
Critical strengths
-
In this paper, the authors study an important problem: whether a policy trained with RLHF on a limited set of prompts can perform well on unseen prompts. The authors clearly explain why this question matters in practice and why existing theoretical analyses, which rely on accurate reward estimation and exact optimization, do not fully answer it. The motivation is clear and easy to follow.
-
The authors provide intuitive explanations after the main theorems. For example, after the result for the insufficient-coverage case, the authors explain why the bound does not grow with the feature dimension, and after the result for gradient ascent, the authors explain why stability improves as the number of iterations increases and compare this with prior stability results in the nonconvex setting. These explanations help readers without a strong theoretical background understand the meaning of the results.
-
The authors are transparent about the limitations of their technical choices. In Remark 2, the authors explain that taking the iterate with the smallest gradient norm as the output requires computing the full gradient, and that a guarantee for the last iterate would need stronger conditions. The authors also note that the assumed decay rate of the coverage residual is not essential, and a slower rate only leads to a correspondingly slower bound. This helps readers understand the scope of the results accurately.
Critical weaknesses
-
The authors analyze a setting that differs from the standard RLHF pipeline suggested by the title. Standard RLHF first learns a reward model from human preference comparisons and then trains the policy with this reward model. In Section 3.1, the authors assume that the true reward is directly available, so the step of learning the reward from human preferences is not covered by the analysis. This assumption is stated only briefly and is not mentioned in the abstract or the introduction. The authors may consider stating it explicitly in the abstract and introduction, and discussing how the results would change when the reward is learned from preference data.
-
The authors use in Section 4 several results that are introduced later or only in the appendix. For example, Corollary 1 relies on the feature coverage assumption, which is only introduced in Section 5.2, and on a key lemma that appears only in the appendix. In addition, one equality is described as "discussed above", although its justification is given in the appendix. As a result, readers who read the paper in order may find Section 4 hard to follow. The authors may consider stating the assumption and the key lemma before their first use.
-
The authors' terminology, cross-references and figure descriptions contain some inconsistencies. For example, the main result is called a generalization bound in the abstract but a suboptimality gap elsewhere, and the text states that the proof of Lemma 3 is provided in Section 4, which does not match its actual location. In the experiments, the subplot titles of Figure 1 appear to report the effective dimension rather than the ambient dimension given in the appendix, and the plots against the number of iterations show the suboptimality gap while the text describes them as evidence for the decrease of the generalization error. The authors may consider proofreading the terminology and cross-references, and aligning the figure labels and descriptions with the experimental settings.
Questions for authors
-
Could the authors explain whether the analysis can be extended to the setting where the reward is learned from human preference data? If so, how would the reward estimation error enter the decomposition of the suboptimality gap?
-
In the simulations on the sample size, both the ambient dimension and the effective dimension change between the two settings. Could the authors explain how these results support the claim that the bound does not depend on the dimension? An experiment that fixes the effective dimension and varies only the ambient dimension may provide more direct support.
-
The assumption that the features of different responses to the same prompt are linearly independent implies that the number of responses cannot exceed the feature dimension. Since the response space of an LLM is much larger than its feature dimension, could the authors explain what this assumption means for the full response space?