acceptodds
Under review as a conference paper at ICLR 2027

Beyond RLHF: A Theoretical Framework of LLM Alignment

Abstract

Reinforcement learning from human feedback (RLHF) has become the dominant paradigm for aligning large language models (LLMs) to produce higher-quality responses. However, the standard training objective for RLHF lacks a learning-theoretic justification, and existing theories do not allow comparisons of the guarantees between various methods because different methods are often analyzed under different frameworks. Toward a unified framework of alignment, we ask under what assumptions we can justify existing algorithms or derive new ones. To this end, we reframe alignment as distribution learning from pairwise preferences, which makes a probabilistic assumption describing how preferences reveal information about the target LM. This leads us to derive three principled training objectives: preference maximum likelihood estimate, preference distillation, and reverse KL minimization. These can be viewed as corrections to existing objectives that, as we show, enjoy strong non-asymptotic convergence to the target LM under our framework. Furthermore, our theory explains, for the first time, the empirical finding that on-policy objectives (e.g., RLHF) typically outperform likelihood-style objectives (e.g., DPO). Our empirical results indicate that our algorithms are competitive with strong baselines across several tasks and models.

Primary area:
learning theory
Keywords:
RLHF, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_b2pQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a unified theoretical framework for language-model alignment by viewing alignment as distribution learning from pairwise preferences, rather than as reward maximization. The authors posit the existence of an ideal target language model π∗\pi^*, and assume that human preferences are generated according to a Bradley-Terry-style model whose scores are directly determined by π∗\pi^*. Based on this assumption, the paper derives three objectives: Preference MLE, Preference Distillation, and Reverse KL minimization. These objectives are connected to DPO, REBEL, and RLHF, respectively. The paper further proves non-asymptotic O(1/n)O(1/n) forward-KL convergence guarantees under finite-class, realizability, boundedness, and coverage assumptions. Empirically, the authors validate the proposed objectives on contextual bandit experiments, TL;DR summarization, and general chat benchmarks, showing competitive or improved performance over corresponding baselines.

Critical strengths

The main strength of the paper is its attempt to provide a unified probabilistic perspective on several important alignment objectives. Recasting alignment as distribution learning from pairwise preferences is conceptually appealing, and the connection among PMLE, Preference Distillation, and Reverse KL provides a useful way to compare DPO-like, REBEL-like, and RLHF-like methods under a common framework. A second strength is that the paper provides explicit statistical guarantees for the proposed objectives. The O(1/n)O(1/n) forward-KL bounds, together with the contextual bandit experiments, give a clear theoretical narrative and help explain why on-policy or reverse-KL-style objectives may behave differently from likelihood-style objectives.

Critical weaknesses

The most important weakness is that the central preference assumption is quite strong and not sufficiently validated. The paper assumes that pairwise preferences are generated directly from the target LM probabilities via a Bradley-Terry model. This makes the theory elegant, but it is unclear whether real human preferences behave this way. In practice, human preferences often depend on factuality, safety, style, instruction following, and task-specific utility, which may not correspond directly to the target model's output probability. As a result, the theoretical guarantees may rely on an idealized assumption whose relevance to real alignment settings is not fully established. A second weakness is that the empirical evidence does not fully support the paper's broader claims. The contextual bandit experiments are useful for verifying the theory under controlled assumptions, but they are relatively far from real LLM alignment. The TL;DR and chat experiments show some improvements, but the gains are mixed and sometimes modest. In particular, the results do not conclusively establish that the proposed framework explains why RLHF-style objectives generally outperform DPO-style objectives in practice. More direct experiments isolating coverage, on-policy sampling, entropy, and reward-model error would be needed to support this claim.

Questions for authors

Can the authors provide empirical evidence that the proposed preference model P(a≻b∣x)∝π∗(a∣x)γP(a \succ b|x)\propto \pi^*(a|x)^\gamma is a reasonable approximation of real human preference data?

How sensitive are the proposed methods to misspecification of the preference model? For example, if preferences depend on external task reward or factual correctness rather than the target LM probability, do the convergence guarantees or empirical advantages still hold?

The paper argues that the framework explains why RLHF-style objectives can outperform DPO-style objectives. Can the authors provide experiments that more directly isolate the effect of on-policy sampling versus the specific loss design?

In the chat experiments, some gains are relatively small or mixed across benchmarks. Are the improvements statistically significant, and how sensitive are they to reward model choice and hyperparameter tuning?

4 comments

We thank the reviewer for the thoughtful suggestions on the framework and experiments.

W1: Human preference criteria and the BT assumption

Eq. (3) does not exclude factuality or safety because the BT reward can score these properties just as it can score style or task utility. If human preferences follow such a reward r⋆r^\star through BT then π⋆(a∣x)∝exp⁡(r⋆(x,a)/γ)\pi^\star(a\mid x)\propto\exp(r^\star(x,a)/\gamma) yields Eq. (3) on a finite response space.

The BT model is a standard assumption in RLHF and DPO [1]. Recent open models also use LLMs as judges to construct preference data for DPO. For example, Tülu 3 uses GPT-4o ratings [2] and Olmo 3 uses GPT-4.1 judgments for part of its preference data [3].

References

[1] Rafailov et al. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023.

[2] Lambert et al. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. 2024. §5.2.1 and §5.3.

[3] Olmo 3. 2025. §5.3.1 and Appendix A.7.4.


W2: Empirical support for the RLHF and DPO comparison

The updated comparisons appear under DPO vs. RLHF and Entropy Regularization in our General Response.


Q1: Fit to real human preference data

Eq. (3) is a reparameterization of the BT model on a finite response space. BT is a standard preference model in RLHF and DPO [1].

Existing studies also provide empirical support for using preferences annotated by LLMs. Lee et al. [2] find that training on these preferences achieves performance comparable to RLHF in human evaluations. Recent open models use this approach to construct alignment data: Tülu 3 uses GPT-4o ratings for DPO [3] and Olmo 3 uses GPT-4.1 judgments for part of its DPO data [4].

Also, we will measure the fit of Eq. (3) on a preference set and plot it against task performance. We will share these results as they become available.

References

[1] Rafailov et al. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023.

[2] Lee et al. RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. ICML 2024.

[3] Lambert et al. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. 2024. §5.2.1 and §5.3.

[4] Olmo 3. 2025. §5.3.1 and Appendix A.7.4.


Q2: Guarantees under preference misspecification

External task reward and factual correctness do not themselves violate Eq. (3). On a finite response space any finite scalar reward r⋆r^\star yields the same BT preferences as the distribution π⋆(a∣x)∝exp⁡(r⋆(x,a)/γ)\pi^\star(a\mid x)\propto\exp(r^\star(x,a)/\gamma).

Appendix C.6 also extends our PMLE and RKL guarantees to approximate BT preferences. For a target π⋆\pi^\star fixed independently of the data, define

ε2=Ex∼D, a,b∼μ(⋅∣x)[(Ppref(a≻b∣x)−Pπ⋆(a≻b∣x))2]\varepsilon^2=\mathbb E_{x\sim D,\,a,b\sim\mu(\cdot\mid x)}[(P_{\mathsf{pref}}(a\succ b\mid x)-P_{\pi^\star}(a\succ b\mid x))^2].

Here PprefP_{\mathsf{pref}} denotes the actual preference probability and Pπ⋆P_{\pi^\star} the BT probability in Eq. (3). Thus ε2\varepsilon^2 measures their mean squared difference over the data distribution.

We use the boundedness and coverage assumptions, and use global empirical minimizers with β=0\beta=0. The target satisfies π⋆∈Π\pi^\star\in\Pi for PMLE and π⋆∈Pγ(R)⊆Π\pi^\star\in\mathcal P_\gamma(\mathcal R)\subseteq\Pi for RKL. For fixed finite classes and parameters, both estimates satisfy

Ex∼D[KL(π⋆(⋅∣x)∥π^(⋅∣x))]≤O(1/n+ε2)\mathbb E_{x\sim D}[\mathrm{KL}(\pi^\star(\cdot\mid x) \Vert \widehat\pi(\cdot\mid x))]\le O(1/n+\varepsilon^2)

with probability at least 1−δ1-\delta for any fixed δ∈(0,1)\delta\in(0,1). Departures from BT therefore contribute an additional O(ε2)O(\varepsilon^2) term. If ε=O(1/n)\varepsilon=O(1/\sqrt n) then the O(1/n)O(1/n) guarantee is preserved. The full bounds and proof appear in blue in Appendix C.6.


Q3: Separating sampling from loss design

Our General Response compares DPO with PMLE and RLHF with RKL. DPO learns from fixed preference pairs, whereas PMLE also uses samples from the current policy to evaluate its explicit KL regularizer. In the RLHF vs. RKL comparison, we add only the entropy regularization term to the objective and keep the other experimental settings the same. This isolates the effect of adding entropy regularization.


Q4: Statistical significance and sensitivity

Please see our General Response, which illustrates that RKL outperforms GRPO on all four metrics with different reward models.

Dear reviewer,

We hope our response made sense to you. Please let us know if you have further questions or concerns!

AuthorsOpenReview ↗

Follow-up respone on W1, Q1, Q2, and Q3

On a finite response space, setting π⋆(a∣x)∝exp⁡(r(x,a)/γ)\pi^\star(a\mid x)\propto\exp(r(x,a)/\gamma) in Eq. (3) gives the Bradley–Terry (BT) probability σ(r(x,a)−r(x,b))\sigma(r(x,a)-r(x,b)) that a person prefers aa to bb for prompt xx. The reward rr can depend on factuality or safety.

W1 and Q1: Does BT predict real human choices?

We consider human votes from Chatbot Arena and MT-Bench and exclude ties.

To check whether BT probabilities match human votes, we train two models that use P(a≻b∣x)=σ(u(x,a)−u(x,b))P(a\succ b\mid x)=\sigma(u(x,a)-u(x,b)). The score u(x,a)u(x,a) measures how much the model favors response aa to prompt xx.

For the first model, we estimate a rating for each LLM from the Arena votes. The rating is the same for all responses from that LLM. We predict how people will vote by comparing the ratings of the two LLMs that generated the responses.

For the second model, we use the ArmoRM reward model to score each response to its prompt on both datasets. We learn a coefficient that multiplies the difference between the two ArmoRM scores in the BT formula. We include only comparisons with an ArmoRM score for both responses.

We train both BT models by maximizing the log-likelihood of the human votes.

For validation, we randomly split the data in half by prompt and use each half to evaluate models trained on the other.

For each validation pair, let p=max⁡{P(a≻b∣x),P(b≻a∣x)}≥0.5p=\max\{P(a\succ b\mid x),P(b\succ a\mid x)\}\ge 0.5, the probability that the BT model assigns to the response it predicts people will choose. We split the pairs into quintiles of pp (the columns of Table 1). In each quintile, Predicted is the mean of pp and Observed is the fraction of pairs in which people chose that response.

Table 1. Calibration on the validation sets (%).

Dataset, modelRate (%)Lowest 20%20–40%40–60%60–80%Highest 20%
Arena, per-LLM scorePredicted5257627183
Arena, per-LLM scoreObserved5157627182
Arena, ArmoRMPredicted5257637287
Arena, ArmoRMObserved5358657387
MT-Bench, ArmoRMPredicted5667799198
MT-Bench, ArmoRMObserved5772839196

For both datasets, the predicted BT probabilities are close to the observed human choice rates.

Q2: What happens when the labels do not follow BT?

We use the contextual bandits from Figure 2(b) (Appendix D.2). In each environment, we keep the target policy π⋆∝exp⁡(r/γ)\pi^\star\propto\exp(r/\gamma) and the policy used to sample response pairs fixed. We change only the probability Plabel(a≻b∣x)P_{\mathrm{label}}(a\succ b\mid x) that the label favors aa over bb. Under BT, this probability is σ(Δr)\sigma(\Delta r) with Δr=r(x,a)−r(x,b)\Delta r=r(x,a)-r(x,b).

To simulate departures from BT, we consider two cases.

  • We first draw a label from the original BT model and flip it with probability ρ\rho. The probability that the resulting label favors aa over bb is (1−ρ)σ(Δr)+ρσ(−Δr)(1-\rho)\sigma(\Delta r)+\rho\sigma(-\Delta r).
  • We draw each label from one of two equally likely annotator groups. One group has reward r+δsr+\delta s and the other has reward r−δsr-\delta s. Here s(x,a)s(x,a) is an additional criterion and δ\delta controls how much the groups differ. The probability that a random annotator prefers aa over bb is 12σ(Δr+δΔs)+12σ(Δr−δΔs)\tfrac12\sigma(\Delta r+\delta\Delta s)+\tfrac12\sigma(\Delta r-\delta\Delta s) with Δs=s(x,a)−s(x,b)\Delta s=s(x,a)-s(x,b). We keep π⋆\pi^\star as the target defined by the average reward rr. We scale ss to match the root mean square of rr after centering rr separately for each prompt.

Given the limited time for rebuttal, we focus on PMLE and DPO in this experiment. Both methods maximize the log-likelihood of the same nn preference labels. PMLE uses γlog⁡π(a∣x)\gamma\log\pi(a\mid x) as the score for response aa. DPO uses γlog⁡[π(a∣x)/π0(a∣x)]\gamma\log[\pi(a\mid x)/\pi_0(a\mid x)] with reference policy π0\pi_0.

We measure how much the two label mechanisms change the original BT probabilities by ε2=E[(Plabel(a≻b∣x)−σ(Δr))2]\varepsilon^2=\mathbb{E}[(P_{\mathrm{label}}(a\succ b\mid x)-\sigma(\Delta r))^2]. We give each training prompt equal weight. For each prompt, we average over pairs of responses drawn independently using the same policy as for the training pairs. We evaluate the learned policy π^\widehat\pi by averaging KL(π⋆∥π^)\mathrm{KL}(\pi^\star\|\widehat\pi) over training prompts.

Corollary 17 bounds the forward KL of PMLE by O(1/n+ε2)O(1/n+\varepsilon^2) under the stated assumptions.

Table 2. Forward KL of PMLE (×10−2\times 10^{-2}) under label noise for each sample size nn.

nn2122^{12}2162^{16}2202^{20}2242^{24}
ρ=0\rho=0 (BT)17.40.990.06<0.01
ρ=0.05\rho=0.0517.61.620.730.67
ρ=0.1\rho=0.119.03.442.572.50
ρ=0.3\rho=0.334.519.618.818.7

While PMLE’s forward KL decreases roughly as 1/n1/n with BT labels, it drops at first and then changes little at larger sample sizes in both cases with modified labels. This is consistent with Corollary 17 because increasing nn reduces the statistical term while the approximation term depends on ε2\varepsilon^2.

Table 3. Forward KL at n=224n=2^{24}.

Label noise ρ\rho0 (BT)0.050.10.20.3
ε2\varepsilon^2 (×10−3\times 10^{-3})00.31.14.410
PMLE (×10−2\times 10^{-2})<0.010.72.59.018.7
DPO (×10−2\times 10^{-2})4.25.06.913.523.3
Two annotator groups δ\delta0 (BT)1248
ε2\varepsilon^2 (×10−3\times 10^{-3})00.83.37.913
PMLE (×10−2\times 10^{-2})<0.010.73.39.117.7
DPO (×10−2\times 10^{-2})4.25.07.613.622.2

As ε2\varepsilon^2 increases, both learned policies move away from π⋆\pi^\star in forward KL. PMLE has lower forward KL than DPO at every level in Table 3. Under BT, Eq. (6) gives a DPO population optimum proportional to π0exp⁡(r/γ)\pi_0\exp(r/\gamma). Because the reference policy assigns different probabilities to different responses, this optimum differs from π⋆\pi^\star.

Q3: On-policy sampling vs. the loss

Due to time constraints during the rebuttal period, we will include an ablation separating the effects of on-policy sampling and loss design in the revision.

Reviewer_7VPCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This manuscript aims to establish a unified learning-theoretic framework for language model alignment by framing the problem as distribution learning directly from pairwise preference feedback. Rather than adopting reward maximization as an axiomatic learning goal, the authors model human preference probabilities using a Bradley-Terry model directly parameterized by a target language model. From this assumption, the paper derives three training objectives: Preference Maximum Likelihood Estimation (PMLE), Preference Distillation, and Reverse KL minimization (RKL). Afterward, the paper presents non-asymptotic upper bounds on forward KL divergence for all three approaches. The paper concludes that RKL's sample complexity depends on the reward class's generalized coverage coefficient rather than the full policy class's, providing a theoretical explanation for the empirical superiority of on-policy RLHF over likelihood-based methods like DPO. Empirical experiments are conducted on synthetic contextual bandits, TL;DR summarization with Pythia-2.8B/6.9B, and general chat benchmarks with LLaMA-3-8B-Instruct.

Critical strengths

The paper provides a unified perspective on preference optimization by framing alignment as direct distribution estimation under a parameterized Bradley-Terry model. This formulation provides a principled alternative to conventional RLHF approaches that primarily justify alignment through empirical reward maximization.

Critical weaknesses

There are three weaknesses I need more clarifications from the authors:

  • First, Assumption 2 imposes strict boundedness on centered rewards, effectively requiring response probabilities to remain bounded away from zero. This assumption is highly restrictive for autoregressive language models, where sequence probabilities can decrease exponentially with generation length. Moreover, the convergence bounds in Theorems 4, 6, and 7, together with Proposition 16, appear to involve unfavorable scaling factors hidden in their constants, as detailed in Appendix C. As a result, I believe that the resulting non-asymptotic guarantees may become excessively loose or practically vacuous for realistic language models, substantially limiting the applicability of the theoretical results.
  • Second, in the proposed framework, the paper assumes that human preference probabilities can be characterized through sequence probability ratios under a target language model, potentially conflating model likelihood with human utility. In practice, highly probable sequences are not necessarily preferred by humans, as repetitive, generic, or otherwise degenerate responses may receive relatively high likelihood despite their poor quality. For this reason, this discrepancy raises concerns about the validity of the underlying Bradley-Terry parameterization and whether the resulting theoretical guarantees adequately characterize real-world human preferences and alignment objectives.
  • Third, the proposed RKL objective, as mentioned in Equation 15, appears closely related to conventional maximum-entropy reinforcement learning. It could raise some questions about its relevant novelty beyond established entropy-regularized policy optimization. Furthermore, the empirical improvements over GRPO reported in Table 3 are relatively small, including MT-Bench (8.27 vs. 8.13), Arena-Hard (32.5 vs. 32.0), and AlpacaEval 2.0 LC (36.67% vs. 35.97%). Without additional statistical validation or controlled comparisons against appropriately tuned entropy-regularized baselines, it remains unclear whether these gains reflect the proposed distribution-learning framework or conventional optimization effects.
Questions for authors

I have two questions for the authors as follows:

  • First, if Assumption 2 is relaxed to allow sequence probabilities to approach zero, can you establish non-asymptotic rates that do not scale exponentially with sequence length?
  • Second, given that RKL in Equation 15 is identical to maximum-entropy policy optimization with an entropy coefficient specified in Table 8, what empirical evidence demonstrates that the 0.5-point gain on Arena-Hard is a consequence of distribution recovery rather than standard entropy-induced exploration tuning?
4 comments

We appreciate the reviewer's detailed feedback on our theoretical results.

W1: Boundedness and the size of the bounds

Assumption 2 follows conventional boundedness assumptions in preference learning theory. Policy log-probabilities are bounded in Assumption 3.1 of [1] and candidate rewards are bounded in Assumption 3 of [2]. Our assumption bounds centered log-probabilities and implies π(a∣x)≥e−2R/∣A∣\pi(a\mid x)\ge e^{-2R}/|\mathcal A|. Since this lower bound decreases with the response space size it does not rule out exponential decay of sequence probabilities with length.

Coverage assumptions and coefficients are standard in RLHF and DPO theory [1–14]. Theorem 3.10 of [5] proves a minimax lower bound that grows with the coverage coefficient for offline RLHF with linear rewards. Theorem 2.2 of [6] further gives a lower bound that depends on coverage for estimating the target policy in forward KL. Such coverage dependence is therefore statistically unavoidable in general for offline preference learning.

Exponential constants also appear in the forward KL bound of [1, Theorem 3.6]. Our analysis uses this conventional statistical setting to compare how the learning objectives depend on data coverage. For β=0\beta=0, the PMLE bound includes CΠC_\Pi in the policy class term whereas the RKL bound still depends on data coverage through CRC_{\mathcal R} in the reward estimation term. Both bounds use the same boundedness condition. A numerically tight forward KL guarantee for the full LLM architectures and training procedures used in our experiments is a separate open problem beyond this comparison.

The additional constant in Proposition 16 is needed to convert a forward KL bound into a reverse KL bound. This constant is not used in the forward KL bounds in Theorems 4, 6 and 7.

An interesting direction for future work is to investigate whether the exploration techniques of Chen et al. [15] can avoid the exponential dependence on RR in our distribution recovery bounds.

References

[1] Agarwal et al. Design Considerations in Offline Preference-based RL. ICML 2025.

[2] Zhan et al. Provable Offline Preference-Based Reinforcement Learning. ICLR 2024.

[3] Liu et al. Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer. NeurIPS 2024.

[4] Huang et al. Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization. ICLR 2025.

[5] Zhu et al. Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons. ICML 2023.

[6] Juno Kim et al. Coverage Improvement and Fast Convergence of On-policy Preference Learning. ICML 2026.

[7] Xiong et al. Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint. ICML 2024.

[8] Jia et al. Do We Need to Verify Step by Step? Rethinking Process Supervision from a Theoretical Perspective. ICML 2025.

[9] Ye et al. Online Iterative Reinforcement Learning from Human Feedback with General Preference Model. NeurIPS 2024.

[10] Rosset et al. Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences. 2024.

[11] Gao et al. REBEL: Reinforcement Learning via Regressing Relative Rewards. NeurIPS 2024.

[12] Xie et al. Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF. ICLR 2025.

[13] Zhao et al. Sharp Analysis for KL-Regularized Contextual Bandits and RLHF. NeurIPS 2025.

[14] Li et al. Policy Evaluation for Reinforcement Learning from Human Feedback: A Sample Complexity Analysis. AISTATS 2024.

[15] Chen et al. Avoiding expRR scaling in RLHF through Preference-based Exploration. NeurIPS 2025.


W2: Target probabilities and human utility

Eq. (3) uses the conventional BT preference model rather than assuming that an arbitrary trained LLM's likelihood represents human utility. BT is widely used in RLHF and DPO as in Ouyang et al. (2022) and Rafailov et al. (2023). On a finite response space any finite BT reward r⋆r^\star can be represented by π⋆(a∣x)∝exp⁡(r⋆(x,a)/γ)\pi^\star(a\mid x)\propto\exp(r^\star(x,a)/\gamma) without changing its pairwise preference probabilities. Here, π⋆\pi^\star is a statistical oracle defined by the preference model, not the output distribution of an existing pretrained LM. A pretrained model assigning high probability to an undesirable response therefore does not contradict Eq. (3).

Recent alignment pipelines also use preferences annotated by LLMs to train their policies. Tülu 3 (§5.2.1) constructs DPO preference pairs from GPT-4o ratings and Olmo 3 (§5.3.1) uses GPT-4.1 judgments for part of its DPO data. Nemotron 3 Nano uses LLM judges to filter synthetic conversations and instruction following data. Lee et al. (2024) further show that training on preferences annotated by LLMs achieves performance comparable to RLHF in human evaluations of summarization and helpful dialogue.

W3: Contribution beyond maximum entropy RL

Our contribution is the statistical guarantee for recovering π⋆\pi^\star from preference data rather than a new maximum entropy objective. Remark 8 (p. 8) states the objective equivalence. Under BT, the entropy coefficient γ\gamma makes the population RKL objective equal to γKL(π∥π⋆)\gamma\mathrm{KL}(\pi \Vert \pi^\star) up to a constant when the true reward is used and β=0\beta=0.

Theorem 7 goes beyond this population identity by bounding Ex[KL(π⋆(⋅∣x)∥π^(⋅∣x))]\mathbb{E}_{x}[\mathrm{KL}(\pi^\star(\cdot\mid x)\Vert\widehat{\pi}(\cdot\mid x))] when the reward must be estimated from nn preference samples. For fixed RR and γ\gamma, its bound is O((log⁡(∣Π∣/δ)+γ−2CRlog⁡(∣R∣/δ))/n)O((\log(|\Pi|/\delta)+\gamma^{-2}C_{\mathcal{R}}\log(|\mathcal{R}|/\delta))/n) with probability at least 1−δ1-\delta under the assumptions of Theorem 7 and reward realizability. The PMLE bound in Theorem 4 for β=0\beta=0 is O(CΠlog⁡(∣Π∣/δ)/(γ2n))O(C_{\Pi}\log(|\Pi|/\delta)/(\gamma^2 n)).

The difference comes from where offline responses are used. PMLE fits the policy from those responses, whereas RKL uses them to estimate the reward and evaluates its policy objective under the policy being learned. The policy-class term therefore has no CΠC_\Pi factor, while the reward-class term still depends on CRC_{\mathcal R}.

Our General Response gives additional experiments. See Entropy Regularization for the updated comparison.


Q1: Rates as sequence length increases

We believe that such rates cannot be guaranteed in general even if Assumption 2 is relaxed to allow its bound on centered log-probabilities to grow with sequence length. Note that Assumption 2 already allows sequence probabilities to decrease exponentially with length.

Consider two target policies that assign different probabilities to the first token but use the same conditional distributions for all later tokens. The total variation distance between the two policies stays constant as sequence length increases. We can choose a response sampling distribution with coverage coefficient one for which the BT preference probabilities under the two target policies become exponentially close on average over the sampled pairs. Recovering the target to a fixed forward KL accuracy with high probability then requires exponentially many comparisons.

We will include the construction and detailed proof in the revision.


Q2: Evidence for the source of the Arena-Hard gain

RKL adds only an entropy regularization term to the RLHF objective. All other experimental settings are the same, so the performance gain reflects the effect of this term. Our theoretical contribution is to justify this objective through distribution learning. The additional experiments are reported in our General Response.

We will also measure the entropy of the models in Table 3 and examine whether higher entropy is associated with larger gains from power sampling. We will share these results as they become available.

Thank you for addressing all my concerns. I think the paper is much better now. I have increased my evaluation from 2 to 3. Good luck!

We are glad that our response addressed your concerns. Thank you for raising your score and positive assessment!

Reviewer_Tc3POpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the problem of aligning language models through pairwise preference-based learning objectives that model response distributions. Under the probabilistic assumption that preferences are explicitly linked to an ideal language model, it derive three training objectives: preference maximum likelihood estimation, preference distillation, and reverse KL minimization. These three objectives are closely related to DPO, reward distillation, and RLHF, respectively. The paper provides a non-asymptotic O(1/n) bound on forward KL error and analyzes how these guarantees depend on data coverage and the complexity of the reward model. Experiments on context bandits, summarization, and general-purpose chat tasks compare the proposed objectives with their corresponding baseline methods.

Critical strengths

The paper addresses a meaningful question: how to study different alignment objectives within a common theoretical framework. Using a shared distribution-learning target provides a clear basis for comparing their statistical guarantees.

The evaluation includes both controlled synthetic experiments and LLM experiments on summarization and general chat. The synthetic settings allow direct measurement of target-distribution recovery, while the LLM experiments provide evidence that the derived objectives can be competitive in practice.

Critical weaknesses

The framework assumes a specific relationship between human preference probabilities and the probability of an ideal language model generating a response. Synthetic experiments generate labels according to this relationship, thus primarily validating the method's performance under the assumption that it holds perfectly. The LLM experiments demonstrate competitive results, but it remains unclear to what extent this assumption aligns with actual preference data, and whether the paper's conclusions still hold when the assumption is only approximately valid.

Some improvements in the chat experiments are small, and their reliability is unclear. The results in Table 3 are reported without uncertainty estimates. Repeated runs or confidence intervals would help determine whether these gains are stable and provide stronger evidence that the proposed changes improve practical alignment.

Questions for authors

How do the proposed methods behave when the preference data do not exactly follow Eq. (3)? A controlled experiment that varies the departure from this assumption would help clarify the robustness of the framework.

The paper claims to explain why RLHF outperforms DPO, but the theoretical comparison concerns RKL and PMLE, and RLHF/GRPO performs worse than DPO in Table 3. Under what conditions should the proposed explanation apply to the original methods?

Do the guarantees for preference distillation and RKL require the reward-model class to represent the true preferences exactly?

5 comments

We thank the reviewer for the helpful comments and questions.

W1: Preference assumptions and fit to real data

Eq. (3) is a reparameterization of the standard Bradley–Terry (BT) model used in RLHF and DPO [1, 2]. On a finite response space, any finite BT reward r⋆r^\star induces the distribution π⋆(a∣x)∝exp⁡(r⋆(x,a)/γ)\pi^\star(a\mid x)\propto\exp(r^\star(x,a)/\gamma), which yields the same pairwise preference probabilities. We assume that the target policy π⋆\pi^\star determines the observed pairwise preference probabilities through Eq. (3).

Recent open models also use preferences annotated by LLMs for alignment. Tülu 3 constructs DPO preference pairs from GPT-4o ratings [3] and Olmo 3 uses GPT-4.1 judgments for part of its DPO data [4]. Lee et al. [5] further show that training on preferences annotated by LLMs achieves performance comparable to RLHF in human evaluations of summarization and helpful dialogue.

We also obtain guarantees when BT holds only approximately. For a target policy π⋆\pi^\star fixed independently of the data, let PprefP_{\mathsf{pref}} denote the actual preference probabilities and Pπ⋆P_{\pi^\star} the BT probabilities in Eq. (3). Define

ε2=Ex∼D, a,b∼μ(⋅∣x)[(Ppref(a≻b∣x)−Pπ⋆(a≻b∣x))2]\varepsilon^2=\mathbb E_{x\sim D,\,a,b\sim\mu(\cdot\mid x)}[(P_{\mathsf{pref}}(a\succ b\mid x)-P_{\pi^\star}(a\succ b\mid x))^2].

Here, ε2\varepsilon^2 averages the squared difference between the actual and modeled preference probabilities over the prompts and response pairs used to collect the data.

We keep Assumption 2 and the coverage condition in Definition 3 relative to π⋆\pi^\star. We consider finite classes with β=0\beta=0 and global empirical minimization. PMLE requires π⋆∈Π\pi^\star\in\Pi, while RKL requires π⋆∈Pγ(R)⊆Π\pi^\star\in\mathcal P_\gamma(\mathcal R)\subseteq\Pi.

For fixed classes and problem parameters, each estimate satisfies

Ex∼D[KL(π⋆(⋅∣x)∥π^(⋅∣x))]≤O(1/n+ε2)\mathbb E_{x\sim D}[\mathrm{KL}(\pi^\star(\cdot\mid x)\|\widehat\pi(\cdot\mid x))]\le O(1/n+\varepsilon^2)

with probability at least 1−δ1-\delta for any fixed δ∈(0,1)\delta\in(0,1). The statistical term decreases at rate O(1/n)O(1/n) and the additional O(ε2)O(\varepsilon^2) term accounts for the BT approximation error. If ε=O(1/n)\varepsilon=O(1/\sqrt n) then the total forward KL bound remains O(1/n)O(1/n).

Please see Appendix C.6 of the revised manuscript for the full bounds and proof written in blue.

We also plan to investigate Eq. (3) against human preference data and running experiments to see how deviations from the BT model affect performance. We will share these results as they become available.

References

[1] Ouyang et al. Training language models to follow instructions with human feedback. NeurIPS 2022.

[2] Rafailov et al. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023.

[3] Lambert et al. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. 2024. §5.2.1.

[4] Olmo 3. 2025. §5.3.1 and Appendix A.7.4.

[5] Lee et al. RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. ICML 2024.


W2: Reliability of the chat improvements

In response to the reviewer's feedback, we revisited the general chat experiments. The results are in our General Response.


Q1: Robustness to departures from Eq. (3)

Appendix C.6 of the revised manuscript gives our guarantees for approximate BT preferences. Our main contribution is learning theory, and we are also conducting the suggested experiments. We will include the results in the revised paper.


Q2: Applicability to RLHF and DPO

The comparison uses common assumptions. In the PMLE (≈\approx DPO) bound, the log⁡(∣Π∣)/n\log(|\Pi|)/n term is multiplied by a coverage coefficient. In the RKL (≈\approx RLHF) bound, it is not. The proof shows that this difference comes from evaluating the RKL objective under the policy being learned. To our knowledge, this is the first rigorous comparison that explains the statistical benefit of on-policy evaluation for recovery of the target distribution in this setting. The difference follows from the objectives under a common set of assumptions, rather than from comparing unrelated theoretical settings. For PMLE, the bound includes CΠC_\Pi because preference feedback is collected on responses generated by πref\pi_{\mathrm{ref}}, rather than by the policy being trained.

The updated experiments show higher mean scores for RLHF (GRPO) than DPO on all four metrics. Our General Response gives the results and experimental settings.

Q3: Reward model realizability

We are sorry for the confusion. As the reviewer noted, the guarantees require π⋆∈Pγ(R)\pi^\star\in\mathcal P_\gamma(\mathcal R) so that the reward class contains a function generating the true BT preferences. This is an assumption on the reward class, not a requirement that the learned reward model R^\widehat R equal the true reward. Theorems 6 and 7 account for the finite-sample estimation error.

The same reward realizability assumption appears in Assumption 2 of Zhan et al. (2024) and Assumption 1 of Wang et al. (2023). We will state π⋆∈Pγ(R)\pi^\star\in\mathcal P_\gamma(\mathcal R) explicitly in Theorems 6 and 7 alongside the existing inclusion Pγ(R)⊆Π\mathcal P_\gamma(\mathcal R)\subseteq\Pi in Assumption 5.

We leave extensions beyond reward realizability to future work.

Dear reviewer,

We hope our response made sense to you. Please let us know if you have further questions or concerns!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

General Response

We thank the reviewers for their careful reading and constructive feedback.

Note that our paper's primary area is learning theory. We derive PMLE, preference distillation and RKL as variants of DPO, REBEL and RLHF under a unified distribution learning framework based on pairwise preferences. We prove O(1/n)O(1/n) forward KL convergence to the same target distribution and compare how the guarantees depend on data coverage and reward-model complexity. These guarantees provide a statistical justification for the objectives. To our knowledge, this is the first such non-asymptotic comparison of all three objectives for recovery of the target distribution. We hope that our paper will be evaluated primarily on these theoretical contributions.

We compare these guarantees with the closest prior results. Dumoulin et al. [5] study density estimation from pairwise preferences in the infinite-data limit. Agarwal et al. [6] analyze offline DPO-style objectives, with a square-root dependence on excess preference risk in their forward-KL bound. Our PMLE analysis controls forward KL by a quadratic log-probability error to obtain the O(1/n)O(1/n) rate. We also analyze preference distillation and RKL, making their dependence on reward-model complexity and coverage explicit.

We provide additional experiments following the reviewers' suggestions. As described in the paper, we primarily follow the experimental protocol of REBEL [1]. We use the hyperparameters in Table 6 of the appendix as the starting point and perform additional tuning for both the baselines and our methods.

We implement RLHF with GRPO. GRPO and RKL start from the same policy and use the same reward model, batch size and group size. Both use a learning rate of 5×10−75\times10^{-7}, matching DPO, and a reference KL coefficient of 10−310^{-3} following the default GRPO configuration in verl [4]. GRPO does not include an entropy regularization term. For RKL, we tune the entropy coefficient over {10−2,10−3,10−4}\lbrace 10^{-2},10^{-3},10^{-4} \rbrace.

IMPORTANT:

We now use GPT-4.1 as the LLM judge. The GPT-4-Turbo models used in the original evaluation have been deprecated [2] and the official Arena-Hard configuration now uses GPT-4.1 [3]. All methods in the table below are evaluated with the updated judges.

We also identified and corrected a mismatch between the log-probabilities computed by vLLM and the trainer in our implementation. We reran all general chat experiments after this correction with 3 different seeds. The table below replaces the original Table 3 and reports mean ± standard deviation.

MethodAlpacaEval 2.0 LC Win-rateAlpacaEval 2.0 Win-rateArena-HardMT-Bench
DPO24.1 ± 0.924.3 ± 0.726.8 ± 0.57.17 ± 0.10
PMLE (Ours)26.3 ± 0.528.2 ± 0.231.0 ± 0.87.35 ± 0.04
RLHF (GRPO)25.4 ± 0.226.2 ± 1.427.6 ± 0.27.19 ± 0.09
RKL (Ours)26.9 ± 1.228.0 ± 0.529.2 ± 1.57.26 ± 0.05
REBEL28.0 ± 0.823.5 ± 0.725.0 ± 0.57.06 ± 0.07
Pref. Distill. (Ours)28.4 ± 0.627.0 ± 0.728.1 ± 0.87.30 ± 0.05

DPO vs. RLHF

RLHF (GRPO) and PMLE now both outperform DPO on all four metrics averaged over three random seeds.

Entropy Regularization

Across three random seeds, RKL and preference distillation achieve higher mean scores than their respective baselines, GRPO and REBEL, on all four metrics. Our analysis establishes a distribution recovery guarantee for the entropy regularized objective with an estimated reward model.

We also compare RLHF and RKL using Skywork/Skywork-Reward-Llama-3.1-8B-v0.2 as the reward model.

MethodAlpacaEval 2.0 LC Win-rateAlpacaEval 2.0 Win-rateArena-HardMT-Bench
Base12.613.119.46.93
RLHF (GRPO)14.320.126.87.28
RKL (Ours)15.121.128.47.34

With the Skywork reward model, both methods obtain lower AlpacaEval 2.0 and Arena-Hard scores but higher MT-Bench scores than in the first table. RKL (GRPO + entropy reg.) outperforms RLHF (GRPO) on all four metrics with both reward models.

References

[1] Gao et al. REBEL: Reinforcement Learning via Regressing Relative Rewards. NeurIPS 2024.

[2] OpenAI. API deprecations. (https://developers.openai.com/api/docs/deprecations)

[3] Arena-Hard. Official judge configuration. (https://github.com/lmarena/arena-hard-auto/blob/main/config/arena-hard-v2.0.yaml)

[4] verl. GRPO configuration. (https://verl.readthedocs.io/en/latest/examples/config.html)

[5] Dumoulin et al. A density estimation perspective on learning from pairwise human preferences. 2024.

[6] Agarwal et al. Design Considerations in Offline Preference-based RL. ICML 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.