acceptodds
Under review as a conference paper at ICLR 2027

Towards a theoretical understanding to the generalization of RLHF

Abstract

Reinforcement Learning from Human Feedback (RLHF) and its variants have emerged as the dominant approaches for aligning Large Language Models (LLMs) with human intent. While empirically effective, the theoretical generalization properties of these methods in high-dimensional settings remain to be explored. To this end, we build the generalization theory on RLHF of LLMs under the linear reward model, through the framework of algorithmic stability. In contrast to the existing works built upon the consistency of maximum likelihood estimations on reward model, our analysis is presented under an end-to-end learning framework, which is consistent with practice. Concretely, we prove that under a key feature coverage condition, the empirical optima of the policy model have a generalization bound of order . Moreover, the results can be extrapolated to parameters obtained by gradient-based algorithms, i.e., Gradient Ascent (GA) and Stochastic Gradient Ascent (SGA). Thus, we argue that our results provide new theoretical evidence for the empirically observed generalization of LLMs after RLHF.

Primary area:
learning theory
Keywords:
RLHF, generalization bound, algorithmic stability;

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5SUFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether a policy learned through the KL-regularized RLHF objective can perform well on prompts that are not seen during training. Previous theoretical work mainly estimates the reward parameters from data and plugs them into the closed-form optimal policy, which requires the data to cover all directions of the feature space and assumes that the optimization is solved exactly. Instead, the authors directly analyze the learned policy through algorithmic stability, under a linear reward model, a Boltzmann-form policy class and the assumption that the true reward is available. They show that, under a feature coverage condition that only requires new features to be approximately covered by the training data, the suboptimality gap of empirical stationary points decreases at a rate of order one over the square root of the sample size, without explicit dependence on the feature dimension. They further extend the results to gradient ascent and stochastic gradient ascent. Simulation experiments are provided to illustrate the theoretical rates.

Critical strengths
  1. In this paper, the authors study an important problem: whether a policy trained with RLHF on a limited set of prompts can perform well on unseen prompts. The authors clearly explain why this question matters in practice and why existing theoretical analyses, which rely on accurate reward estimation and exact optimization, do not fully answer it. The motivation is clear and easy to follow.

  2. The authors provide intuitive explanations after the main theorems. For example, after the result for the insufficient-coverage case, the authors explain why the bound does not grow with the feature dimension, and after the result for gradient ascent, the authors explain why stability improves as the number of iterations increases and compare this with prior stability results in the nonconvex setting. These explanations help readers without a strong theoretical background understand the meaning of the results.

  3. The authors are transparent about the limitations of their technical choices. In Remark 2, the authors explain that taking the iterate with the smallest gradient norm as the output requires computing the full gradient, and that a guarantee for the last iterate would need stronger conditions. The authors also note that the assumed decay rate of the coverage residual is not essential, and a slower rate only leads to a correspondingly slower bound. This helps readers understand the scope of the results accurately.

Critical weaknesses
  1. The authors analyze a setting that differs from the standard RLHF pipeline suggested by the title. Standard RLHF first learns a reward model from human preference comparisons and then trains the policy with this reward model. In Section 3.1, the authors assume that the true reward is directly available, so the step of learning the reward from human preferences is not covered by the analysis. This assumption is stated only briefly and is not mentioned in the abstract or the introduction. The authors may consider stating it explicitly in the abstract and introduction, and discussing how the results would change when the reward is learned from preference data.

  2. The authors use in Section 4 several results that are introduced later or only in the appendix. For example, Corollary 1 relies on the feature coverage assumption, which is only introduced in Section 5.2, and on a key lemma that appears only in the appendix. In addition, one equality is described as "discussed above", although its justification is given in the appendix. As a result, readers who read the paper in order may find Section 4 hard to follow. The authors may consider stating the assumption and the key lemma before their first use.

  3. The authors' terminology, cross-references and figure descriptions contain some inconsistencies. For example, the main result is called a generalization bound in the abstract but a suboptimality gap elsewhere, and the text states that the proof of Lemma 3 is provided in Section 4, which does not match its actual location. In the experiments, the subplot titles of Figure 1 appear to report the effective dimension rather than the ambient dimension given in the appendix, and the plots against the number of iterations show the suboptimality gap while the text describes them as evidence for the decrease of the generalization error. The authors may consider proofreading the terminology and cross-references, and aligning the figure labels and descriptions with the experimental settings.

Questions for authors
  1. Could the authors explain whether the analysis can be extended to the setting where the reward is learned from human preference data? If so, how would the reward estimation error enter the decomposition of the suboptimality gap?

  2. In the simulations on the sample size, both the ambient dimension and the effective dimension change between the two settings. Could the authors explain how these results support the claim that the bound does not depend on the dimension? An experiment that fixes the effective dimension and varies only the ambient dimension may provide more direct support.

  3. The assumption that the features of different responses to the same prompt are linearly independent implies that the number of responses cannot exceed the feature dimension. Since the response space of an LLM is much larger than its feature dimension, could the authors explain what this assumption means for the full response space?

Reviewer_EaZgOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies prompt generalization in a KL-regularized setting for linearly parametrized reward models with known rewards and for a fixed-feature Boltzmann policy. With the help of covariance-gradient identity and algorithmic stability, this paper derives the claimed suboptimality bounds of O~(n−1/2)\tilde O(n^{-1/2}) for empirical stationary points, O~(T−1/4+n−1/2)\tilde O(T^{-1/4}+n^{-1/2}) for gradient ascent (GA) and O~(T−1/8+n−1/2)\tilde O(T^{-1/8}+n^{-1/2}) for stochastic gradient ascent (SGA). Authors provide synthetic experiments and LLM post-training experiments in this paper.

Critical strengths

1. Useful geometric characterization. The covariance-gradient identity ∇JS(θ)=VS(θ)(θ∗−θ)\nabla J_S(\theta)=V_S(\theta)(\theta^*-\theta) explains empirical stationary points and why parameter non-identifiability does not preclude empirical policy optimality.

2. Potentially meaningful finite-step analysis. Studying actual GA/SGA iterates, rather than assuming exact policy improvement, is a valuable direction. LLM post-training experiments also provide useful motivation, even though they cannot directly validate the theoretical guarantees.

Critical weaknesses

My primary concern is that the central coverage assumption already entails a stronger stationary-point generalization guarantee, while the advertised finite-step rates require additional conditions to be nonvacuous or even to have the claimed probabilities.

1. Assumption 5 essentially assumes the stationary-point generalization result.

Assumption 5 (page 6) demands the uniform residual bound ϵn=O(1/n)\epsilon_n=O(1/n) for all training datasets SS and all features ϕ(x,a)\phi(x,a).

Let us take the allowed training dataset S=(x0,…,x0)S=(x_0,\ldots,x_0). The range of its covariance does not depend on nn. Also, since Boltzmann policy does not shrink the action support, this range is independent of the finite policy parameter. Taking n→∞n\to\infty makes all global features within the range of this single prompt. It contradicts the intended interpretation of the paper where obtaining progressively more diverse prompts improves the coverage.

Even more important, the paper itself derives a stronger result directly. At an empirical stationary point, Lemma 11 gives

VS(θS∗)(θ∗−θS∗)=0.V_S(\theta_S^*)(\theta^*-\theta_S^*)=0.

Combining this with Assumption 5, we have

∣⟨ϕ(x,a),θ∗−θS∗⟩∣≤4Dϵn.|\langle\phi(x,a),\theta^*-\theta_S^*\rangle| \leq 4D\epsilon_n.

Using KL identity in Eq. (39) and Hoeffding's lemma, we obtain the bound

0≤J(πθ∗)−J(πθS∗)≤8D2ϵn2.\boxed{ 0\leq J(\pi_{\theta^*})-J(\pi_{\theta_S^*}) \leq 8D^2\epsilon_n^2. }

Under the assumed ϵn=O(1/n)\epsilon_n=O(1/n), this is a deterministic O(n−2)O(n^{-2}) bound on population suboptimality, independent of the dimension and spectral condition number and not requiring neither algorithmic stability nor any concentration inequalities.

It does not entail an O(n−2)O(n^{-2}) train-population generalization gap, the two notions being different. Nonetheless, this considerably diminishes the claimed value of the stability-based O~(n−1/2)\tilde O(n^{-1/2}) empirical stationary point suboptimality bound.

The authors should explain what additional insight on stationary point performance provides their stability-based analysis.

2. The finite-step bounds are not derived to be nonvacuous under the stated spectral and probabilistic assumptions.

The constants Γ\Gamma in Theorems 2, 4, and 6 are defined through suprema of the smallest positive eigenvalues of empirical covariances. Assumptions 1, 2, and 5 do not guarantee that these suprema are finite, uniformly bounded in n,Tn,T or independent of the effective rank.

For instance, take the one-dimensional case ϕ(x,±)=±x\phi(x,\pm)=\pm x where x∈(0,1]x\in(0,1] and there is a uniform reference policy. Assumption 5 holds with ϵn=0\epsilon_n=0, but the positive eigenvalues of covariances can be arbitrarily small so that the supremum over these eigenvalues is infinite.

Take θ∗=1\theta^*=1, zero initialization, the prescribed GA step size 1/251/25, and T=n2T=n^2. Compare the dataset S=(1/T,…,1/T)S=(1/T,\ldots,1/T) with its neighbor that has one element 11. The GA outputs in the two cases will differ in their test objective at x=1x=1 by the quantity converging to

log⁡cosh⁡(1)>0,\log\cosh(1)>0,

though T−1/4+n−1/2→0T^{-1/4}+n^{-1/2}\to0. Therefore, the explicit bound with the infinite Γ\Gamma is vacuous, and the advertised vanishing uniform stability rate is not implied by the listed assumptions.

Separately, the SGA proof has a quantification error. The proof in Appendix G uses Lemma 4 for each fixed neighboring dataset pair and then takes the supremum over all pairs in Eq. (92). However, Definition 8 (Eq. (89)) requires a single high-probability event with respect to the algorithmic randomness when the stability holds simultaneously for all neighboring pairs.

Having a high-probability statement for each fixed pair does not imply having a simultaneous high-probability statement for all pairs. Thus, the provided proof does not establish the claimed high-probability results in Theorems 6–7.

Questions for authors
  1. Coverage and necessity: Is Assumption 5 supposed to hold uniformly over all datasets or only with high probability? If the latter, how should the proof of the neighboring-dataset stability be modified? What does the stability analysis tell beyond the direct O(ϵn2)O(\epsilon_n^2) bound on population suboptimality?

  2. Nonvacuous rates: Under what additional assumptions, the Γ\Gamma factors are guaranteed to be finite, uniformly bounded in nn, TT and the effective rank? Do these assumptions exclude the one-dimensional GA construction mentioned above?

  3. SGA proof: How do the authors ensure that there is a single high-probability event, where the stability holds simultaneously for all neighboring datasets, as in Definition 8? If some other stability notion is considered, what generalization theorem is used to get the claimed probability guarantee?

Reviewer_KdA8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper derives bounds on the performance gap between learned and optimal policies in KL-regularized RLHF, assuming linear rewards and a posterior Boltzmann policy model. It extends the analysis from idealized, fully optimized policies to policies obtained after finitely many gradient ascent (GA) or stochastic gradient ascent (SGA) steps.

Critical strengths
  1. Studying finite-step generalization for both GA and SGA is interesting and makes the analysis more relevant to practical RLHF training.
Critical weaknesses
  1. Limited validation in real LLM training. The finite-step bounds are evaluated mainly on idealized synthetic tasks, without direct validation using real LLM training data.

  2. Errors in the theoretical derivations.

    • Equation (92): a high-probability guarantee for each fixed dataset pair does not establish the claimed guarantee for all pairs simultaneously.

    • Equation (75): a simple one-dimensional counterexample contradicts this bound. Consider two equally likely prompts, each with two responses. Their scalar response features are (−1/2,1/2)(-1/2,1/2) and (−1,1)(-1,1), respectively. With a uniform reference policy and θ=0\theta=0, both responses are equally likely, giving conditional variances 1/41/4 and 11. Thus C=1C=1 and V=5/8V=5/8. For a single training prompt (n=1n=1), either sampling outcome gives ∣VS−V∣=3/8|V_S-V|=3/8. Setting γ=0.01\gamma=0.01, the claimed inequality becomes

      =e^{0.00375} \leq e^{8C^4\gamma^2/n} =e^{0.0008},$$ which is false.
    • Equations (27)–(28): in (27), the absolute value is incorrectly placed outside the sum, making the right-hand side zero. In (28), the signs of the two log-normalization terms are reversed.

    • Equation (70): the claimed mt≤ηt/2m_t\leq\eta_t/2 is false. At t=4t=4, the definitions give mt=3ηt/4m_t=3\eta_t/4.

These issues must be resolved before the validity of the overall theoretical results can be properly assessed.

Questions for authors

See above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.