- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a unified theoretical framework for language-model alignment by viewing alignment as distribution learning from pairwise preferences, rather than as reward maximization. The authors posit the existence of an ideal target language model , and assume that human preferences are generated according to a Bradley-Terry-style model whose scores are directly determined by . Based on this assumption, the paper derives three objectives: Preference MLE, Preference Distillation, and Reverse KL minimization. These objectives are connected to DPO, REBEL, and RLHF, respectively. The paper further proves non-asymptotic forward-KL convergence guarantees under finite-class, realizability, boundedness, and coverage assumptions. Empirically, the authors validate the proposed objectives on contextual bandit experiments, TL;DR summarization, and general chat benchmarks, showing competitive or improved performance over corresponding baselines.
Critical strengths
The main strength of the paper is its attempt to provide a unified probabilistic perspective on several important alignment objectives. Recasting alignment as distribution learning from pairwise preferences is conceptually appealing, and the connection among PMLE, Preference Distillation, and Reverse KL provides a useful way to compare DPO-like, REBEL-like, and RLHF-like methods under a common framework. A second strength is that the paper provides explicit statistical guarantees for the proposed objectives. The forward-KL bounds, together with the contextual bandit experiments, give a clear theoretical narrative and help explain why on-policy or reverse-KL-style objectives may behave differently from likelihood-style objectives.
Critical weaknesses
The most important weakness is that the central preference assumption is quite strong and not sufficiently validated. The paper assumes that pairwise preferences are generated directly from the target LM probabilities via a Bradley-Terry model. This makes the theory elegant, but it is unclear whether real human preferences behave this way. In practice, human preferences often depend on factuality, safety, style, instruction following, and task-specific utility, which may not correspond directly to the target model's output probability. As a result, the theoretical guarantees may rely on an idealized assumption whose relevance to real alignment settings is not fully established. A second weakness is that the empirical evidence does not fully support the paper's broader claims. The contextual bandit experiments are useful for verifying the theory under controlled assumptions, but they are relatively far from real LLM alignment. The TL;DR and chat experiments show some improvements, but the gains are mixed and sometimes modest. In particular, the results do not conclusively establish that the proposed framework explains why RLHF-style objectives generally outperform DPO-style objectives in practice. More direct experiments isolating coverage, on-policy sampling, entropy, and reward-model error would be needed to support this claim.
Questions for authors
Can the authors provide empirical evidence that the proposed preference model is a reasonable approximation of real human preference data?
How sensitive are the proposed methods to misspecification of the preference model? For example, if preferences depend on external task reward or factual correctness rather than the target LM probability, do the convergence guarantees or empirical advantages still hold?
The paper argues that the framework explains why RLHF-style objectives can outperform DPO-style objectives. Can the authors provide experiments that more directly isolate the effect of on-policy sampling versus the specific loss design?
In the chat experiments, some gains are relatively small or mixed across benchmarks. Are the improvements statistically significant, and how sensitive are they to reward model choice and hyperparameter tuning?
4 comments
Dear reviewer,
We hope our response made sense to you. Please let us know if you have further questions or concerns!
Follow-up respone on W1, Q1, Q2, and Q3
On a finite response space, setting in Eq. (3) gives the Bradley–Terry (BT) probability that a person prefers to for prompt . The reward can depend on factuality or safety.
W1 and Q1: Does BT predict real human choices?
We consider human votes from Chatbot Arena and MT-Bench and exclude ties.
To check whether BT probabilities match human votes, we train two models that use . The score measures how much the model favors response to prompt .
For the first model, we estimate a rating for each LLM from the Arena votes. The rating is the same for all responses from that LLM. We predict how people will vote by comparing the ratings of the two LLMs that generated the responses.
For the second model, we use the ArmoRM reward model to score each response to its prompt on both datasets. We learn a coefficient that multiplies the difference between the two ArmoRM scores in the BT formula. We include only comparisons with an ArmoRM score for both responses.
We train both BT models by maximizing the log-likelihood of the human votes.
For validation, we randomly split the data in half by prompt and use each half to evaluate models trained on the other.
For each validation pair, let , the probability that the BT model assigns to the response it predicts people will choose. We split the pairs into quintiles of (the columns of Table 1). In each quintile, Predicted is the mean of and Observed is the fraction of pairs in which people chose that response.
Table 1. Calibration on the validation sets (%).
| Dataset, model | Rate (%) | Lowest 20% | 20–40% | 40–60% | 60–80% | Highest 20% |
|---|---|---|---|---|---|---|
| Arena, per-LLM score | Predicted | 52 | 57 | 62 | 71 | 83 |
| Arena, per-LLM score | Observed | 51 | 57 | 62 | 71 | 82 |
| Arena, ArmoRM | Predicted | 52 | 57 | 63 | 72 | 87 |
| Arena, ArmoRM | Observed | 53 | 58 | 65 | 73 | 87 |
| MT-Bench, ArmoRM | Predicted | 56 | 67 | 79 | 91 | 98 |
| MT-Bench, ArmoRM | Observed | 57 | 72 | 83 | 91 | 96 |
For both datasets, the predicted BT probabilities are close to the observed human choice rates.
Q2: What happens when the labels do not follow BT?
We use the contextual bandits from Figure 2(b) (Appendix D.2). In each environment, we keep the target policy and the policy used to sample response pairs fixed. We change only the probability that the label favors over . Under BT, this probability is with .
To simulate departures from BT, we consider two cases.
- We first draw a label from the original BT model and flip it with probability . The probability that the resulting label favors over is .
- We draw each label from one of two equally likely annotator groups. One group has reward and the other has reward . Here is an additional criterion and controls how much the groups differ. The probability that a random annotator prefers over is with . We keep as the target defined by the average reward . We scale to match the root mean square of after centering separately for each prompt.
Given the limited time for rebuttal, we focus on PMLE and DPO in this experiment. Both methods maximize the log-likelihood of the same preference labels. PMLE uses as the score for response . DPO uses with reference policy .
We measure how much the two label mechanisms change the original BT probabilities by . We give each training prompt equal weight. For each prompt, we average over pairs of responses drawn independently using the same policy as for the training pairs. We evaluate the learned policy by averaging over training prompts.
Corollary 17 bounds the forward KL of PMLE by under the stated assumptions.
Table 2. Forward KL of PMLE () under label noise for each sample size .
| (BT) | 17.4 | 0.99 | 0.06 | <0.01 |
| 17.6 | 1.62 | 0.73 | 0.67 | |
| 19.0 | 3.44 | 2.57 | 2.50 | |
| 34.5 | 19.6 | 18.8 | 18.7 |
While PMLE’s forward KL decreases roughly as with BT labels, it drops at first and then changes little at larger sample sizes in both cases with modified labels. This is consistent with Corollary 17 because increasing reduces the statistical term while the approximation term depends on .
Table 3. Forward KL at .
| Label noise | 0 (BT) | 0.05 | 0.1 | 0.2 | 0.3 |
|---|---|---|---|---|---|
| () | 0 | 0.3 | 1.1 | 4.4 | 10 |
| PMLE () | <0.01 | 0.7 | 2.5 | 9.0 | 18.7 |
| DPO () | 4.2 | 5.0 | 6.9 | 13.5 | 23.3 |
| Two annotator groups | 0 (BT) | 1 | 2 | 4 | 8 |
|---|---|---|---|---|---|
| () | 0 | 0.8 | 3.3 | 7.9 | 13 |
| PMLE () | <0.01 | 0.7 | 3.3 | 9.1 | 17.7 |
| DPO () | 4.2 | 5.0 | 7.6 | 13.6 | 22.2 |
As increases, both learned policies move away from in forward KL. PMLE has lower forward KL than DPO at every level in Table 3. Under BT, Eq. (6) gives a DPO population optimum proportional to . Because the reference policy assigns different probabilities to different responses, this optimum differs from .
Q3: On-policy sampling vs. the loss
Due to time constraints during the rebuttal period, we will include an ablation separating the effects of on-policy sampling and loss design in the revision.
We thank the reviewer for the thoughtful suggestions on the framework and experiments.
W1: Human preference criteria and the BT assumption
Eq. (3) does not exclude factuality or safety because the BT reward can score these properties just as it can score style or task utility. If human preferences follow such a reward r⋆ through BT then π⋆(a∣x)∝exp(r⋆(x,a)/γ) yields Eq. (3) on a finite response space.
The BT model is a standard assumption in RLHF and DPO [1]. Recent open models also use LLMs as judges to construct preference data for DPO. For example, Tülu 3 uses GPT-4o ratings [2] and Olmo 3 uses GPT-4.1 judgments for part of its preference data [3].
References
[1] Rafailov et al. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023.
[2] Lambert et al. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. 2024. §5.2.1 and §5.3.
[3] Olmo 3. 2025. §5.3.1 and Appendix A.7.4.
W2: Empirical support for the RLHF and DPO comparison
The updated comparisons appear under DPO vs. RLHF and Entropy Regularization in our General Response.
Q1: Fit to real human preference data
Eq. (3) is a reparameterization of the BT model on a finite response space. BT is a standard preference model in RLHF and DPO [1].
Existing studies also provide empirical support for using preferences annotated by LLMs. Lee et al. [2] find that training on these preferences achieves performance comparable to RLHF in human evaluations. Recent open models use this approach to construct alignment data: Tülu 3 uses GPT-4o ratings for DPO [3] and Olmo 3 uses GPT-4.1 judgments for part of its DPO data [4].
Also, we will measure the fit of Eq. (3) on a preference set and plot it against task performance. We will share these results as they become available.
References
[1] Rafailov et al. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023.
[2] Lee et al. RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. ICML 2024.
[3] Lambert et al. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. 2024. §5.2.1 and §5.3.
[4] Olmo 3. 2025. §5.3.1 and Appendix A.7.4.
Q2: Guarantees under preference misspecification
External task reward and factual correctness do not themselves violate Eq. (3). On a finite response space any finite scalar reward r⋆ yields the same BT preferences as the distribution π⋆(a∣x)∝exp(r⋆(x,a)/γ).
Appendix C.6 also extends our PMLE and RKL guarantees to approximate BT preferences. For a target π⋆ fixed independently of the data, define
ε2=Ex∼D,a,b∼μ(⋅∣x)[(Ppref(a≻b∣x)−Pπ⋆(a≻b∣x))2].
Here Ppref denotes the actual preference probability and Pπ⋆ the BT probability in Eq. (3). Thus ε2 measures their mean squared difference over the data distribution.
We use the boundedness and coverage assumptions, and use global empirical minimizers with β=0. The target satisfies π⋆∈Π for PMLE and π⋆∈Pγ(R)⊆Π for RKL. For fixed finite classes and parameters, both estimates satisfy
Ex∼D[KL(π⋆(⋅∣x)∥π(⋅∣x))]≤O(1/n+ε2)
with probability at least 1−δ for any fixed δ∈(0,1). Departures from BT therefore contribute an additional O(ε2) term. If ε=O(1/n) then the O(1/n) guarantee is preserved. The full bounds and proof appear in blue in Appendix C.6.
Q3: Separating sampling from loss design
Our General Response compares DPO with PMLE and RLHF with RKL. DPO learns from fixed preference pairs, whereas PMLE also uses samples from the current policy to evaluate its explicit KL regularizer. In the RLHF vs. RKL comparison, we add only the entropy regularization term to the objective and keep the other experimental settings the same. This isolates the effect of adding entropy regularization.
Q4: Statistical significance and sensitivity
Please see our General Response, which illustrates that RKL outperforms GRPO on all four metrics with different reward models.