- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
The paper argues alignment with human preferences is different from alignment with human behaviors, and as a result, alignment with human preferences can make the model less human-like. The paper then suggests that human-likeness should be used as an explicit target and directly fitting human responses improves this.
Critical strengths
-
The experiments are well-designed to support the claims being made under the assumptions and model choices made with the use of a model tuned on the set of human responses to model the human response distribution being an example of this.
-
The experiments about different ways of trying to get the model to better model human behavior seems useful for understanding how well common interventions for trying to simulate human behavior work and how these interventions differ along axes like response length or variation.
Critical weaknesses
-
One concern with the paper is that one of the main points is that the distribution trained to model human preferences is different from the distribution where responses are uniformly sampled. However, this is exactly the intention of preference learning, so it seems the concern that preference learning does not do something that is was not intended to do seems unfair, especially when we have base models which are trained to model human text and behavior. This is not addressed in the analysis or in the experiments.
-
The Turing-test gap is exactly the KL between the model after preference learning and the model before, a metric which is well understood and taken into consideration when analyzing model behavior and post-training.
-
A concern with Section 3 and the introduction of the Turing-test gap is that the pairwise judgements themselves do not define a scalar reward and only a ranking. A reward model must be learned in order to quantify the reward assigned to each response. As a result, the framing of the evaluator reward as being built from the response distribution seems like a misleading characterization of the relationship between the reward and response distribution.
-
Another concern is that it is assumed that the set of human responses is representation of the human response distribution with each response being just as likely. However, this seems unlikely to be true or a strong model of human behavior as if we consider the StackExchange responses as an example, a highly rated response is much more likely to be written than a lower rated response or an incorrect response. In the absence of the highly rated response, it is likely someone else will write a similar response as many people agree with this response, but this is not the case for responses with lower rating. In this case, the model trained to follow human preferences can be a more accurate model of human behavior.
Questions for authors
-
Would you be able to provide analysis or experiments that demonstrate the coverage of base models?
-
Could you clarify the role of modeling human behavior in language models as successfully doing so while modeling preferences seems to be conflicting objectives?
-
Could you provide further justification around why uniformly sampling responses acts as a good model of human behavior?
7 comments
Response to Reviewer sh85 (Part2/2)
3. Base-model coverage (Q1)
Following the reviewer's suggestion, we evaluated the human-response NLL of the Base, Instruct, and Base→Human-SFT models on the same held-out response pool, using the same prompt template within each model family. In all four settings, Base assigns higher likelihood to human responses than Instruct, and Base→Human-SFT higher still:
| Model / Dataset | Instruct | Base | Human-SFT from Base |
|---|---|---|---|
| Qwen2.5-7B / SHP | 437.97 | 354.86 | 319.31 |
| Llama-3-8B / SHP | 387.68 | 335.91 | 311.70 |
| Qwen2.5-7B / StackExchange | 510.38 | 417.75 | 378.90 |
| Llama-3-8B / StackExchange | 444.66 | 390.88 | 368.97 |
For these models, post-training reduces coverage of human responses relative to the base model; at the same time, the base model is not the ceiling of what training directly on human responses achieves.
4. The Turing-test gap and the KL before and after preference learning (W2)
The Turing-test gap in Eq. 4, , is defined between the human response distribution and its preference-tilted distribution: a property of the human data itself rather than of a trained model. Figure 1 illustrates this directly: given human responses and preference rankings, the gap is computed without training any model. As shown in Section 3.2 (Eqs. 3–4), it coincides with the KL divergence between the model before and after preference learning only in the idealized case where training starts from and reaches .
In practical post-training, however, , and monitoring the KL before and after preference learning only tells us how far the model has moved from its starting point : not whether it has moved closer to or farther from human behavior . While the KL from is always non-negative, the distance to can move in either direction depending on : in Figure 4b, the exact same DPO recipe and data move generated responses away from human responses when starting from Human-SFT, but move them closer in three of four settings when starting from Instruct. We have made this distinction clearer in Section 3.2.
5. Pairwise judgments and scalar rewards (W3)
We agree, and thank the reviewer for pointing out the imprecise wording. In the revision, Section 3.1 now states that pairwise judgments are modeled through a scalar reward under the Bradley–Terry model, and in Section 3.2 we changed "the response distribution it is built from" to "the response distribution recorded in the same data".
Note that this change does not affect the conclusions: the reward is identified up to a prompt-dependent constant , and changes neither Eq. 2 nor Eq. 5; Section 4.1 and Figure 1 also use only the within-prompt standardized ranks .
We thank the reviewer again for the constructive review; the paper has genuinely improved through the corresponding revisions. We hope our responses have addressed your concerns, and we would be grateful if you could reconsider your assessment of our paper accordingly.
Thank you for thorough response, and I appreciate the clarifications and additional experiments especially using the base model. Many of my concerns have been addressed, and a few concerns remain.
One of the main remaining concerns is about the pairwise judgements and scalar rewards. I believe that it is still not true that the gap and the reward can be directly computed from the human responses and preferences over them. Even under the assumption of a BT model or the use of a log-likelihood ratio, the reward is heavily dependent on the model used to model or learn the preferences whether this is a reward model or a fine-tuned model . This is true especially in practice where data is finite and as a result, rewards may not be finite or may be undefined without additional assumptions such as regularization or having a learned reward model. Further explanation should be provided about how the reward is computed or about assumptions being made and how this influences results such as those in Figure 1.
I appreciate the additional results on reweighting, but a question that comes up from the results is that if the smallest difference is positive, then does that suggest that all human responses have decreased in likelihood, and if so, it seems like a general shift in likelihoods could be a factor in the results. One thing that would strengthen the analysis and comparisons would be using centered comparisons similar to the ones used in the initial analysis.
One remaining concern is about the existing understanding of the KL change with post-training (the derivations remain based on this) and that it is true in general that if an initial model captures a human distribution, any change not just preference training and even including additional SFT on a finite sample dataset of human responses, would shift the model away from the human distribution. However, I appreciate the paper's experiments demonstrating the implications of such changes thoroughly.
Overall, the paper has been strengthened and the clarifications and discussion have changed my rating, and I will raise my score from a 1 to a 2.
Response to Reviewer sh85 (Part1/2)
We thank the reviewer for the careful and in-depth review. We understand the reviewer to have two main concerns: (1) whether the assumption that equal weighting of observed responses represents human behavior is reasonable (W4, Q3); and (2) given that preference learning was never meant to imitate humans, why the gap between the two should be studied (W1, Q2). We first give new empirical analysis and clarification on these two points, and then report the base-model comparison (Q1), the relation between the Turing-test gap and the KL before and after training (W2), and the wording corrections in Section 3 (W3).
In keeping with the conference's request for brevity, and to reduce the reading burden on the reviewer, we keep our responses concise.
1. Equal weighting of human responses (W4, Q3)
We thank the reviewer for pointing out this important property of observational forum data. The reviewer's point separates into two independent parts: (a) because forums discourage repeating an answer once a good one exists, the equal-weighted distribution over observed responses need not equal the distribution of independent human answers; and (b) if the true human distribution leans toward highly voted responses, a model trained to follow human preferences may in fact be the more accurate model of human behavior. We respond to each in turn.
On (a). We agree with the reviewer: the observed response pool on a forum is shaped by its posting mechanism and is not an unbiased distribution of independent human answers (we have added emphasis on this limitation in Section 6). We chose SHP and StackExchange in order to control the source of training text as a confound: both datasets contain human-written responses together with human preference votes, so we can compare SFT and preference optimization on exactly the same pool of human text, and thereby separate, as far as possible, the effect of the preference objective itself from the effect of the data content (Sections 3.1 and 4).
Note that the theoretical derivation in Section 3 does not depend on equal weighting at all: for any true human distribution pH (however it leans toward highly voted responses), once the reference policy has fit pH, any non-constant preference reward pushes the optimal policy away from pH (Eqs. 4–5).
On (b). This connects to the paper's analysis of the difference between human preference and human behavior: endorsing a response is not the same as being someone who would write it. Since this may sound abstract, we test the implication of (b) directly: if the true human response distribution leans toward highly voted responses, then weighting evaluation toward highly voted responses should make the preference-trained model look better.
We tested this in a controlled setting. Two models start from the same Human-SFT checkpoint; one is trained with DPO on human preferences, and the other continues to fit the same human responses with equal weights. For each held-out prompt, we compute the difference in sequence NLL between the two (DPO minus continued SFT) on every observed human response, take the smallest difference for that prompt, and then average over prompts with equal weight. This amounts to placing all evaluation weight, for each prompt, on the single response most favorable to DPO; no other within-prompt weighting, including using only the top-voted response, can make the mean difference smaller. Even so, DPO's mean sequence NLL is still higher in all four settings (positive values mean DPO assigns lower likelihood to human responses):
2. The role of human-likeness (W1, Q2)
W1. We agree that preference learning is designed to tilt the model toward preferred responses, and we do not regard this as a failure of assistant alignment. In Section 6 ("The paradox of successful alignment") we discuss this point: the gap arises precisely because preference optimization succeeds at its objective.
Q2. Simulating human behavior with language models matters because LLMs are increasingly used as human proxies, and the models used are aligned models, not base models, for example to simulate survey respondents or to replicate behavioral experiments.
We agree that, under the current alignment stack, perfectly simulating human behavior and modeling preferences are in general conflicting goals, and we do not claim that one model can perfectly pursue both. Showing this conflict is one of the core contributions of the paper: successfully modeling preferences makes the model a better AI assistant, while implicitly making it worse at simulating human behavior.