acceptodds
Under review as a conference paper at ICLR 2027

Are Human-Aligned LLMs Models of Humans? A Turing-Test Gap in Preference Alignment

Abstract

Human-feedback alignment has made language models useful assistants and is commonly described as aligning them with humans. However, the responses people prefer from an AI need not be the responses they themselves would give. We distinguish alignment with human preferences from alignment with human behavior, and show that alignment with human preferences can make model behavior less human-like even when both preferences and responses come entirely from humans. We call this the Turing-test gap. We show that preference alignment preserves the human response distribution only under a restrictive condition, and find no consistent evidence that real human preferences satisfy it. Empirically, the loss of human-response likelihood increases with the strength of preference weighting, regardless of its direction, and the gap also appears under standard DPO. These results establish human-likeness as an explicit dimension of alignment rather than something assumed to follow from preference alignment.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
artificial intelligence, alignment, preference optimization, RLHF, DPO, human-likeness, Turing test, language models, human feedback

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_sh85OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper argues alignment with human preferences is different from alignment with human behaviors, and as a result, alignment with human preferences can make the model less human-like. The paper then suggests that human-likeness should be used as an explicit target and directly fitting human responses improves this.

Critical strengths
  • The experiments are well-designed to support the claims being made under the assumptions and model choices made with the use of a model tuned on the set of human responses to model the human response distribution being an example of this.

  • The experiments about different ways of trying to get the model to better model human behavior seems useful for understanding how well common interventions for trying to simulate human behavior work and how these interventions differ along axes like response length or variation.

Critical weaknesses
  • One concern with the paper is that one of the main points is that the distribution trained to model human preferences is different from the distribution where responses are uniformly sampled. However, this is exactly the intention of preference learning, so it seems the concern that preference learning does not do something that is was not intended to do seems unfair, especially when we have base models which are trained to model human text and behavior. This is not addressed in the analysis or in the experiments.

  • The Turing-test gap is exactly the KL between the model after preference learning and the model before, a metric which is well understood and taken into consideration when analyzing model behavior and post-training.

  • A concern with Section 3 and the introduction of the Turing-test gap is that the pairwise judgements themselves do not define a scalar reward and only a ranking. A reward model must be learned in order to quantify the reward assigned to each response. As a result, the framing of the evaluator reward as being built from the response distribution seems like a misleading characterization of the relationship between the reward and response distribution.

  • Another concern is that it is assumed that the set of human responses is representation of the human response distribution with each response being just as likely. However, this seems unlikely to be true or a strong model of human behavior as if we consider the StackExchange responses as an example, a highly rated response is much more likely to be written than a lower rated response or an incorrect response. In the absence of the highly rated response, it is likely someone else will write a similar response as many people agree with this response, but this is not the case for responses with lower rating. In this case, the model trained to follow human preferences can be a more accurate model of human behavior.

Questions for authors
  • Would you be able to provide analysis or experiments that demonstrate the coverage of base models?

  • Could you clarify the role of modeling human behavior in language models as successfully doing so while modeling preferences seems to be conflicting objectives?

  • Could you provide further justification around why uniformly sampling responses acts as a good model of human behavior?

7 comments
AuthorsOpenReview ↗

Response to Reviewer sh85 (Part1/2)

We thank the reviewer for the careful and in-depth review. We understand the reviewer to have two main concerns: (1) whether the assumption that equal weighting of observed responses represents human behavior is reasonable (W4, Q3); and (2) given that preference learning was never meant to imitate humans, why the gap between the two should be studied (W1, Q2). We first give new empirical analysis and clarification on these two points, and then report the base-model comparison (Q1), the relation between the Turing-test gap and the KL before and after training (W2), and the wording corrections in Section 3 (W3).

In keeping with the conference's request for brevity, and to reduce the reading burden on the reviewer, we keep our responses concise.


1. Equal weighting of human responses (W4, Q3)

We thank the reviewer for pointing out this important property of observational forum data. The reviewer's point separates into two independent parts: (a) because forums discourage repeating an answer once a good one exists, the equal-weighted distribution over observed responses need not equal the distribution of independent human answers; and (b) if the true human distribution leans toward highly voted responses, a model trained to follow human preferences may in fact be the more accurate model of human behavior. We respond to each in turn.

On (a). We agree with the reviewer: the observed response pool on a forum is shaped by its posting mechanism and is not an unbiased distribution of independent human answers (we have added emphasis on this limitation in Section 6). We chose SHP and StackExchange in order to control the source of training text as a confound: both datasets contain human-written responses together with human preference votes, so we can compare SFT and preference optimization on exactly the same pool of human text, and thereby separate, as far as possible, the effect of the preference objective itself from the effect of the data content (Sections 3.1 and 4).

Note that the theoretical derivation in Section 3 does not depend on equal weighting at all: for any true human distribution pHp_H (however it leans toward highly voted responses), once the reference policy has fit pHp_H, any non-constant preference reward pushes the optimal policy away from pHp_H (Eqs. 4–5).

On (b). This connects to the paper's analysis of the difference between human preference and human behavior: endorsing a response is not the same as being someone who would write it. Since this may sound abstract, we test the implication of (b) directly: if the true human response distribution leans toward highly voted responses, then weighting evaluation toward highly voted responses should make the preference-trained model look better.

We tested this in a controlled setting. Two models start from the same Human-SFT checkpoint; one is trained with DPO on human preferences, and the other continues to fit the same human responses with equal weights. For each held-out prompt, we compute the difference in sequence NLL between the two (DPO minus continued SFT) on every observed human response, take the smallest difference for that prompt, and then average over prompts with equal weight. This amounts to placing all evaluation weight, for each prompt, on the single response most favorable to DPO; no other within-prompt weighting, including using only the top-voted response, can make the mean difference smaller. Even so, DPO's mean sequence NLL is still higher in all four settings (positive values mean DPO assigns lower likelihood to human responses):

SettingΔ\Delta NLL under the most DPO-favorable weighting (DPO − continued SFT)
Qwen2.5-7B / SHP+25.83
Llama-3-8B / SHP+11.14
Qwen2.5-7B / StackExchange+28.67
Llama-3-8B / StackExchange+38.64

2. The role of human-likeness (W1, Q2)

W1. We agree that preference learning is designed to tilt the model toward preferred responses, and we do not regard this as a failure of assistant alignment. In Section 6 ("The paradox of successful alignment") we discuss this point: the gap arises precisely because preference optimization succeeds at its objective.

Q2. Simulating human behavior with language models matters because LLMs are increasingly used as human proxies, and the models used are aligned models, not base models, for example to simulate survey respondents or to replicate behavioral experiments.

We agree that, under the current alignment stack, perfectly simulating human behavior and modeling preferences are in general conflicting goals, and we do not claim that one model can perfectly pursue both. Showing this conflict is one of the core contributions of the paper: successfully modeling preferences makes the model a better AI assistant, while implicitly making it worse at simulating human behavior.

AuthorsOpenReview ↗

Response to Reviewer sh85 (Part2/2)

3. Base-model coverage (Q1)

Following the reviewer's suggestion, we evaluated the human-response NLL of the Base, Instruct, and Base→Human-SFT models on the same held-out response pool, using the same prompt template within each model family. In all four settings, Base assigns higher likelihood to human responses than Instruct, and Base→Human-SFT higher still:

Model / DatasetInstructBaseHuman-SFT from Base
Qwen2.5-7B / SHP437.97354.86319.31
Llama-3-8B / SHP387.68335.91311.70
Qwen2.5-7B / StackExchange510.38417.75378.90
Llama-3-8B / StackExchange444.66390.88368.97

For these models, post-training reduces coverage of human responses relative to the base model; at the same time, the base model is not the ceiling of what training directly on human responses achieves.

4. The Turing-test gap and the KL before and after preference learning (W2)

The Turing-test gap in Eq. 4, GT=ExDKL(pH∥pP)\mathcal{G}_T = \mathbb{E}_x D_{\mathrm{KL}}(p_H \Vert p_P), is defined between the human response distribution and its preference-tilted distribution: a property of the human data itself rather than of a trained model. Figure 1 illustrates this directly: given human responses and preference rankings, the gap is computed without training any model. As shown in Section 3.2 (Eqs. 3–4), it coincides with the KL divergence between the model before and after preference learning only in the idealized case where training starts from π0=pH\pi_0 = p_H and reaches πP∗\pi^*_P.

In practical post-training, however, π0≠pH\pi_0 \neq p_H, and monitoring the KL before and after preference learning only tells us how far the model has moved from its starting point π0\pi_0: not whether it has moved closer to or farther from human behavior pHp_H. While the KL from π0\pi_0 is always non-negative, the distance to pHp_H can move in either direction depending on π0\pi_0: in Figure 4b, the exact same DPO recipe and data move generated responses away from human responses when starting from Human-SFT, but move them closer in three of four settings when starting from Instruct. We have made this distinction clearer in Section 3.2.

5. Pairwise judgments and scalar rewards (W3)

We agree, and thank the reviewer for pointing out the imprecise wording. In the revision, Section 3.1 now states that pairwise judgments are modeled through a scalar reward under the Bradley–Terry model, and in Section 3.2 we changed "the response distribution pH(y∣x)p_H(y \mid x) it is built from" to "the response distribution pH(y∣x)p_H(y \mid x) recorded in the same data".

Note that this change does not affect the conclusions: the reward is identified up to a prompt-dependent constant c(x)c(x), and c(x)c(x) changes neither Eq. 2 nor Eq. 5; Section 4.1 and Figure 1 also use only the within-prompt standardized ranks sis_i.


We thank the reviewer again for the constructive review; the paper has genuinely improved through the corresponding revisions. We hope our responses have addressed your concerns, and we would be grateful if you could reconsider your assessment of our paper accordingly.

Thank you for thorough response, and I appreciate the clarifications and additional experiments especially using the base model. Many of my concerns have been addressed, and a few concerns remain.

One of the main remaining concerns is about the pairwise judgements and scalar rewards. I believe that it is still not true that the gap and the reward can be directly computed from the human responses and preferences over them. Even under the assumption of a BT model or the use of a log-likelihood ratio, the reward is heavily dependent on the model used to model or learn the preferences whether this is a reward model rPr_P or a fine-tuned model πP\pi_P. This is true especially in practice where data is finite and as a result, rewards may not be finite or may be undefined without additional assumptions such as regularization or having a learned reward model. Further explanation should be provided about how the reward is computed or about assumptions being made and how this influences results such as those in Figure 1.

I appreciate the additional results on reweighting, but a question that comes up from the results is that if the smallest difference is positive, then does that suggest that all human responses have decreased in likelihood, and if so, it seems like a general shift in likelihoods could be a factor in the results. One thing that would strengthen the analysis and comparisons would be using centered comparisons similar to the ones used in the initial analysis.

One remaining concern is about the existing understanding of the KL change with post-training (the derivations remain based on this) and that it is true in general that if an initial model captures a human distribution, any change not just preference training and even including additional SFT on a finite sample dataset of human responses, would shift the model away from the human distribution. However, I appreciate the paper's experiments demonstrating the implications of such changes thoroughly.

Overall, the paper has been strengthened and the clarifications and discussion have changed my rating, and I will raise my score from a 1 to a 2.

Reviewer_uixuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper distinguishes alignment with human preferences from matching the distribution of human responses. It derives conditions under which preference optimization preserves that distribution, then evaluates preference reweighting and DPO using human-written responses. Across two model families and two datasets, preference weighting reduces held-out human-response likelihood relative to equal-weight training. The paper also investigates whether prompting, human-over-model DPO, and supervised training on human responses can recover behavioral similarity. The conceptual distinction is useful, but the practical conclusions are constrained by unvalidated distributional proxies, a confounded comparison between recovery methods, and limited evidence connecting the measured effects to human judgments or interactive behavior.

Critical strengths
  • A well-controlled reweighting experiment. Using the same human-written responses and training pipeline while varying response weights helps isolate the effect of weighting from additional training or exposure to synthetic text. The reversed-preference and randomized-weight controls are informative: all 40 random reassignments increase human-response NLL relative to equal weighting.

  • Explicit examination of the reference policy. Comparing DPO from unchanged Instruct checkpoints with DPO after Human-SFT reveals an important dependence on initialization. Reporting outcomes that differ across these starting points makes the empirical picture more informative than a single comparison would.

  • Several useful evaluation controls. Prompt-level data splitting, clustered bootstrap intervals, classifier cross-fitting, and length matching address concrete sources of leakage and confounding. The sampled-response analysis also checks sensitivity to using two or four samples per prompt.

Critical weaknesses
  1. The comparison supporting Human-SFT as the strongest recovery method confounds the objective with data and training budgets. Appendix F.2 reports that Human-SFT uses 17,096 SHP responses or 24,371 StackExchange responses, whereas human-over-model DPO uses only 2,000 pairs per setting. Moreover, Llama Human-SFT trains for up to three epochs, selecting the checkpoint with the lowest validation loss, while DPO trains for one epoch. These are documented differences in the experimental design: the observed advantage cannot be attributed specifically to the SFT objective because human-data exposure, optimization budget, and checkpoint selection also differ. A matched-data and matched-budget comparison is needed to support a methodological conclusion about SFT versus DPO.

  2. The empirical test of the theoretical density-ratio condition does not adequately validate its key estimator. Equation 5 requires a particular log-density ratio involving the unknown human distribution. The paper approximates this quantity using Human-SFT checkpoints and validates the resulting scores through human-versus-model discrimination, reporting AUCs of 0.990–1.000. However, excellent source discrimination does not establish accurate density-ratio estimation, or reliable ranking among two human-written responses. Consequently, weak agreement between these scores and preference labels cannot cleanly distinguish a mismatch in human preferences from error in the estimated behavioral distribution. The length-controlled analysis further retains only 810, 370, and 400 of the original 5,000 comparisons per dataset—respectively 16.2%, 7.4%, and 8.0%, calculated from the reported counts—so those findings concern a substantially restricted subset.

  3. The theoretical result provides a narrower characterization of practical alignment than the framing suggests. When the reference already equals the human distribution, nonconstant exponential reweighting necessarily changes that distribution. For a general reference, the paper characterizes exact recovery of the human distribution; failure to satisfy that condition does not establish that optimization moves the model farther away. The empirical results illustrate this distinction: DPO from unchanged Instruct checkpoints reduces sampled-response distance in three of four settings, with reported changes of −0.028, −0.036, and −0.047, whose 95% intervals all exclude zero. After Human-SFT, the changes are positive in all four settings. This supports the paper’s qualified claim that preference optimization can reduce behavioral fidelity, but leaves unresolved the practically important question of when it improves or degrades fidelity from realistic starting policies.

  4. The evaluations establish changes in particular proxies, while the connection to successful alignment and Turing-style human-likeness remains indirect. “Preference fit” is measured through likelihood margins on existing preferred/rejected pairs, rather than fresh human preferences over the models’ generated responses. Behavioral similarity is assessed through human-response NLL, embedding-space distance, response statistics, and a source classifier. These measures can disagree: DPO increases human-response NLL from both starting points, while generated responses become closer in embedding space in three of four unchanged-checkpoint settings. In addition, the source classifier is trained only on human and unchanged Instruct responses, so reduced distinguishability after intervention may partly reflect changes outside its training distribution. Cross-fitting and length matching are useful controls, but they do not establish human indistinguishability or consistency across interactions; the authors explicitly acknowledge that interactive and individual-level behavior are not evaluated.

Questions for authors

Refer to weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer uixu (Part1/2)

We thank the reviewer for the careful reading. We respond to each point below.

In keeping with the conference's request for brevity, and to reduce the reading burden on the reviewer, we keep our responses concise.

1. Training budget in Section 5 (W1)

To test whether the differences in Figure 5 come from the amount of training data and the number of updates, on Qwen2.5-7B / SHP we extended human-over-model DPO to the same pool of 17096 human responses used by full Human-SFT (267 updates per epoch), trained for three epochs, and selected the best checkpoint. The held-out human-response sequence NLL is 360.39 for DPO and 314.63 for SFT, a paired difference (DPO − SFT) of +45.75. In the equal-length evaluation, after removing the slots with anomalous DPO responses from all methods, the source DAUCD_{\mathrm{AUC}} is 0.88 for DPO and 0.69 for SFT, a paired difference of +0.19. With the DPO data scaled up, SFT therefore still shows a clear advantage in both coverage of human responses and source distinguishability of its generations.

2. Density-ratio estimators (W2)

We agree that Figure 2a only verifies that these scores rank human responses above model responses, and does not guarantee that they correctly rank two human responses. The weak agreement in Figure 2b therefore cannot fully distinguish "preference is not aligned with the direction of human behavior" from "estimation error". We have stated the test in Section 3.2 more carefully in the revision.

And we added an analysis using all 5,000 comparisons in each dataset, with no pairs discarded. We controlled for word-count difference continuously and tested whether the frozen density-ratio scores provided additional predictive information, using prompt-grouped five-fold cross-validation. Across all nine dataset–estimator combinations, the largest out-of-fold AUC gain was only 0.001.

3. When preference optimization improves or harms behavioral fidelity from a realistic starting point (W3)

We agree that Eq. 5 characterizes when pHp_H can be recovered exactly, and that failing to satisfy it does not imply that optimization moves the model away from pHp_H. We have adjusted the wording of Section 3.2 accordingly.

You asks when preference optimization improves or degrades fidelity from a realistic starting policy. For the optimum of Eq. 1, this has an exact answer. With πλ∝π0exp⁡(λrP)\pi_\lambda \propto \pi_0 \exp(\lambda r_P) (λ=1/β\lambda = 1/\beta), and when the relevant KLs are finite:

DKL(pH∥πλ)−DKL(pH∥π0)=DKL(π0∥πλ)−λ(EpH[rP]−Eπ0[rP]).D_{\mathrm{KL}}(p_H \Vert \pi_\lambda) - D_{\mathrm{KL}}(p_H \Vert \pi_0) = D_{\mathrm{KL}}(\pi_0 \Vert \pi_\lambda) - \lambda\big(\mathbb{E}_{p_H}[r_P] - \mathbb{E}_{\pi_0}[r_P]\big).

The first term on the right is how far the update moves from its starting point, and is always non-negative. Preference optimization therefore reduces the KL to pHp_H if and only if human responses have higher expected reward than the starting model's own responses, and the preference strength λ\lambda is weak enough for this gain to exceed DKL(π0∥πλ)D_{\mathrm{KL}}(\pi_0 \| \pi_\lambda). Conversely, if π0\pi_0 already equals pHp_H, or if the starting model's own responses are already preferred over human responses, any non-trivial preference update increases the KL to pHp_H.

This identity describes the optimum of Eq. 1, not finite-step DPO training, so we use it as a condition on the direction of movement.

AuthorsOpenReview ↗

Response to Reviewer uixu (Part2/2)

4. Proxy metrics and human-likeness (W4)

We agree that these metrics cannot speak to human-likeness in interactive settings or at the level of individuals, which is a considerably more complex problem. We state this in the Limitations of Section 6.

On the disagreement between metrics: held-out NLL measures how well the model covers human responses, whereas energy distance measures how far the model's generated samples are from human responses. The two need not agree: a model's generations can be closer to human responses while it assigns lower likelihood to human responses overall. This is why we report both.

On the source classifier: we agree that post-intervention outputs lie outside the classifier's training distribution. However, the classifier does not assign lower distinguishability to just any change away from Instruct: prompting clearly shortens responses and changes their style (Figure 5b), yet distinguishability changes very little (Figure 5a).


We thank the reviewer again for the constructive review; the paper has genuinely improved through the corresponding revisions. We hope our responses have addressed your concerns, and we would be grateful if you could reconsider your assessment of our paper accordingly.

AuthorsOpenReview ↗

Gentle follow-up on our response to Reviewer uixu

Dear Reviewer uixu,

Thank you again for your detailed review. We wanted to gently follow up on our response.

In particular, we added the matched-budget comparison you asked for: with human-over-model DPO extended to the same scale as Human-SFT, SFT still shows a clear advantage (Reviewer aa6V, who raised the same question, noted that this experiment addressed the question).

We also added an analysis on all 5,000 comparisons with no pairs discarded, and an exact condition for when preference optimization moves a model toward or away from human behavior from a realistic starting point.

We are very grateful for your suggestions, which have strengthened the paper. We hope these new results have addressed your questions, and we would deeply appreciate your feedback or any remaining questions!

Best regards,

The Authors

AuthorsOpenReview ↗

Turing-Test Gap: brief check-in before the discussion closes

Dear Reviewer uixu,

As the discussion closes later AOE today, we wanted to briefly check whether you have any remaining questions on our responses, in particular the comparison of Human-SFT with human-over-model DPO trained on the same data scale, the analysis on all 5,000 comparisons with no pairs discarded, and the condition for when preference optimization moves a model toward or away from human behavior from a realistic starting point.

We would like to note that the matched-budget experiment you asked for has also been recognized by Reviewer aa6V as useful evidence that Human-SFT's advantage is not explained by the amount of training data. We are grateful for the suggestion, which has genuinely strengthened the paper.

Please let us know if anything remains unclear; we would be glad to clarify it in the remaining time.

Best regards,

The Authors

Reviewer_xwyUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper distinguishes alignment with human preferences from alignment with human behavior, and defines human-likeness as how much a model fits to the human response distribution, as well as turing-test gap as the difference. Under a shared-support assumption, it derives the exact condition for preference optimization to recover human behavior. This paper tests its claim using Qwen2.5-7B-Instruct and Llama-3-8B-Instruct on SHP and StackExchange, supplemented by a preference-direction analysis involving LFQA, Dolly, HH-RLHF, WebGPT, and SHP. The results show that positive human-preference weighting improves preference fit while reducing human-response coverage, whereas nonuniform weighting reduces coverage regardless of its direction or assignment. DPO after Human-SFT moves sampled responses away from human responses in all four settings, while DPO from the unchanged Instruct checkpoint moves them closer in three of four settings. Human-SFT provides the largest and most consistent recovery of measured human-likeness.

Critical strengths
  1. The paper identifies a consequential distinction that is routinely blurred in alignment research: responses humans prefer from an assistant need not match the responses humans themselves produce. This distinction is particularly important when models are used as proxies for people.

  2. Under the shared-support assumption, the formal analysis gives a clear compatibility condition for when preference optimization can recover the human response distribution. The result also makes the dependence on the reference policy explicit.

  3. The reweighting experiment is carefully controlled. Equal weighting, positive and negative preference weighting, and random reassignment operate on the same human-written response pools, isolating the effect of nonuniform weighting from changes in training data.

  4. The empirical analysis combines human-response NLL, preference fit, energy distance in BGE-M3 embedding space, source distinguishability, and within-question variation. Comparing DPO from the unchanged Instruct checkpoint with DPO after Human-SFT provides a particularly informative test of the reference-dependent prediction.

Critical weaknesses
  1. The formal results are mathematically straightforward consequences of the closed-form exponential tilt. In the special case π0=pH\pi_0=p_H, any reward that is nonconstant on the support of pHp_H yields a target different from pHp_H; under the shared-support assumption, the general compatibility condition follows directly by substituting pHp_H into the closed-form optimum. The novelty therefore lies mainly in the conceptual distinction and compatibility interpretation rather than in a technically substantial formal contribution, which is consistent with the paper’s empirical focus but limits the weight of its theory.

  2. The empirical pHp_H in the core experiments is constructed by weighting retained responses equally within each prompt and assigning every selected prompt equal total weight. Because the SHP and StackExchange responses are self-selected and platform-conditioned, exposure, voting, moderation, filtering, and respondent selection may make this distribution unrepresentative of a broader population response distribution. Section 6 acknowledges population-versus-individual and distributional-versus-interactive limitations, but does not clarify which population the empirical pHp_H represents or how platform selection constrains the claims. This concern does not invalidate the comparisons relative to the observed response pools, but it limits their external interpretation.

  3. Section 3.2 substitutes Human-SFT-based estimates b^j\hat b_j for the unobserved behavior density-ratio score bHb_H. Figure 2a establishes that these scores rank a human response above a Qwen2.5-Instruct response to the same prompt, but it does not validate the within-source ordering among the human–human or model–model candidates on which the Figure 2b test of Equation 7 depends. Estimation error could attenuate the preference AUC toward chance even if the true density-ratio direction were aligned with human preferences, so Figure 2b cannot cleanly distinguish genuine incompatibility from insufficient proxy fidelity. The three estimators and cross-domain source tests mitigate but do not resolve this concern. This specific issue is confined to Section 3.2 because Sections 4–5 do not use b^j\hat b_j; those sections nevertheless operationalize human-likeness through held-out NLL, energy distance in BGE-M3 embedding space, a frozen source classifier, and response-level statistics rather than human Turing-style judgments.

  4. The core reweighting and Section 4.2 DPO experiments cover Qwen2.5-7B-Instruct and Llama-3-8B-Instruct on two forum-style domains, and no repeated optimization seeds are reported; Section 5 uses only two model–domain training settings. Although the authors explicitly choose four model–dataset settings over multiple seeds in Section 4.1, prompt-level bootstrap intervals quantify uncertainty over evaluated prompts conditional on the trained checkpoints, not optimization or data-order variance. Moreover, the Section 4.2 DPO experiments use human–human preference pairs, unlike many practical preference-optimization pipelines that rank model-generated candidates. This limits external validity, although it should not be generalized to the human-over-model DPO intervention in Section 5.

Questions for authors
  1. The paper treats equal weighting over observed SHP and StackExchange responses as an empirical approximation to pHp_H. What sampling interpretation supports this choice, given respondent self-selection, platform exposure, voting, moderation, and filtering? If pHp_H should instead be understood as a platform-conditioned response distribution, which claims require corresponding qualification?

  2. Section 4.2 presents DPO after Human-SFT as the practical counterpart of the π0=pH\pi_0=p_H case. What empirical evidence establishes that Human-SFT is sufficiently close to pHp_H for this interpretation? Please identify any existing held-out NLL or other diagnostics supporting this approximation and clarify which conclusions depend on it.

  3. Were the training conditions repeated across optimization or data-order seeds? If so, how large is between-run variance relative to the reported effects? The current prompt-level bootstrap intervals quantify evaluation-sample uncertainty but not training stochasticity, which materially affects confidence in the robustness claims.

3 comments
AuthorsOpenReview ↗

Response to Reviewer xwyU (Part1/2)

We thank the reviewer for the careful reading. We respond to each point below.

In keeping with the conference's request for brevity, and to reduce the reading burden on the reviewer, we keep our responses concise.

1. Interpreting the empirical distribution p^H\hat p_H (W2, Q1)

We agree that the empirical distribution p^H\hat p_H should be understood as a specific, filtered distribution of responses on these platforms. Under this reading, equal weighting corresponds to the following sampling interpretation: the retained responses under each selected prompt are treated as samples from that prompt's platform response distribution pH(⋅∣x)p_H(\cdot \mid x), and every prompt receives the same weight in evaluation.

We chose SHP and StackExchange because both the responses and the preference votes come from humans: the same human responses can be used for SFT and, together with their votes, for preference optimization. This lets us separate the effect of the preference objective itself from the effect of model-generated data while holding the training text fixed (Sections 3.1 and 4).

Under this interpretation, our conclusions divide as follows.

First, the formal analysis in Section 3 needs no qualification; it holds for any human response distribution pHp_H.

Second, within the observed response pool, the conclusion of Section 4.2 that preference optimization moves a model that has already fit human responses away from them does not depend on within-prompt equal weighting. Two models start from the same Human-SFT checkpoint; one is trained with DPO on human preferences, and the other continues to fit the same human responses with equal weights. For each held-out prompt, we place all evaluation weight on the single human response most favorable to DPO; no other within-prompt weighting can favor DPO more, and DPO's mean sequence NLL is still higher in all four settings.

Third, the size of the gap and the degree of recovery measured in Sections 4–5 do need qualification. As the reviewer points out, this does not affect comparisons relative to the observed response pool, but within-prompt reweighting cannot account for filtered-out responses, unselected prompts, or people who do not answer on these platforms.

We have added these limitations to Sections 3 and 6.

2. Positioning of the theory (W1)

The derivation in Section 3 is indeed a consequence of the exponential-tilting closed form. However, standard KL analyses track only how far the optimized policy moves from the reference model π0\pi_0. In Section 3 we bring in the human response distribution pHp_H as a third object, which allows separate three questions that are easily conflated: how far preference optimization moves a model from its starting point, whether it moves the model closer to pHp_H, and whether it can exactly recover pHp_H.

This separation allows the experiments of Sections 4 and 5 possible: Section 4.1 tests the case π0=pH\pi_0 = p_H, where any non-constant preference reward moves the optimum away from pHp_H; Section 4.2 compares DPO from two starting points, since the direction of movement relative to pHp_H depends on where the model starts; and Section 5 asks which interventions bring an aligned model back toward pHp_H.

AuthorsOpenReview ↗

Response to Reviewer xwyU (Part2/2)

3. How close Human-SFT is to pHp_H (Q2)

We use Human-SFT in place of pHp_H because pHp_H itself is not directly available. Human-SFT is the model obtained by equal-weight maximum likelihood on the same human responses; without access to pHp_H, it is the most direct approximation to p^H\hat p_H available to us.

The diagnostics already in the paper support "Human-SFT is closer to pHp_H than the original starting point", and we do not claim "Human-SFT is a perfect approximation of pHp_H". In all four settings, Human-SFT lowers held-out human-response sequence NLL relative to the Instruct checkpoint (Figure 4a); on Qwen2.5-7B / SHP, responses generated from the same Human-SFT checkpoint are also harder to distinguish from human responses, and their length and within-prompt variation are closer to the human values (Figure 5).

Note that the measured results of Section 4.2 do not themselves depend on this approximation: relative to its own starting point, DPO after Human-SFT raises human-response NLL in all four settings (Figure 4a) and increases the energy distance between generated and human responses (Table 2). What depends on the approximation is only the reading of these results as an instance of the case predicted by Eq. 4.

We now emphasize this approximation in the revision.

4. Density-ratio estimators (W3)

We agree that Figure 2a only verifies that these scores rank human responses above model responses, and does not guarantee that they correctly rank two human responses. The weak agreement in Figure 2b therefore cannot fully distinguish "preference is not aligned with the direction of human behavior" from "estimation error". As the reviewer already notes, this issue is confined to Section 3.2; the conclusions of Sections 4–5 do not rely on these estimators. We have stated the test in Section 3.2 more carefully in the revision.

5. Robustness and external validity (W4, Q3)

With a limited, academic-scale compute budget (and having already spent nearly 10,000 GPU hours), we prioritized repeating every comparison across several model–dataset combinations, to make the conclusions as general as possible.

We agree that Section 4.2 uses human–human preference pairs, which differs from the many practical pipelines that rank model-generated candidates. We designed it this way to test the effect of the preference objective itself without introducing model-generated text. We now state this limitation on external validity in Section 6.


We thank the reviewer again for the constructive comments.

AuthorsOpenReview ↗

Gentle follow-up on our response to Reviewer xwyU

Dear Reviewer xwyU,

Thank you again for your careful review. We wanted to briefly follow up on our response, in which we clarified the platform-conditioned interpretation of pHp_H, provided empirical diagnostics for Human-SFT, and added the most DPO-favorable within-prompt weighting test. We hope these have addressed your questions.

In our discussions with the other reviewers, we have also added base-model coverage evaluations and centered NLL comparisons, which further support our findings in Section 4.

Thank you again for your thoughtful review and supportive evaluation of our work. Please let us know if there is anything else we can clarify!

Best regards,

The Authors

Reviewer_aa6VOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the distinction between alignment with human preferences and alignment with human behavior. The authors introduce the Turing-test gap to characterize the discrepancy between preferred responses and the distribution of responses humans actually produce. They derive the condition under which KL-regularized preference optimization preserves the human response distribution.

Empirically, the paper examines preference reweighting and DPO using human-written responses from SHP and StackExchange. The results show that preference optimization can improve preference fit while reducing the likelihood assigned to held-out human responses. The authors further compare prompting, human-over-model DPO, and Human-SFT, finding that direct training on human responses provides the most consistent improvements in their human-likeness metrics.

Critical strengths
  1. Important and well-motivated research question. The distinction between human preference alignment and human behavior modeling is meaningful, particularly as LLMs are increasingly used to simulate human respondents and behavior.
  2. Clear theoretical formulation. The paper derives an explicit condition under which preference optimization recovers the human response distribution (Eq. 5), providing a useful framework for understanding the relationship between preference optimization and behavioral fidelity.
  3. Well-controlled experiments. Section 4.1 compares equal weighting, preference weighting, and randomly reassigned weights while keeping the training responses and pipeline fixed. The random reassignment controls in Figure 3 help isolate the effects of nonuniform weighting from those of human preference direction.
  4. Complementary empirical evaluations. The authors evaluate both held-out human-response likelihood and sampled responses. The comparison of different DPO starting points in Figure 4 provides a more nuanced understanding of when preference optimization may increase or decrease behavioral similarity.
Critical weaknesses
  1. Approximation of the human behavior distribution. The paper uses collected human-written responses to approximate (p_H(y\mid x)). However, these datasets may not fully represent the underlying population response distribution. In particular, assigning equal weight to observed responses does not necessarily recover their actual probabilities in the broader human population. This limits the interpretation of the measured gap as a general discrepancy from human behavior.
  2. Limited theoretical novelty. The main theoretical results follow from the established KL-regularized preference optimization framework. While the interpretation is valuable, the paper's theoretical contribution appears to be primarily a reframing and application of existing results rather than a fundamentally new optimization principle.
  3. Limited evaluation of human-likeness. The experiments primarily rely on held-out response likelihood, embedding-based energy distance, source distinguishability, and response-level statistics. These metrics provide useful evidence of distributional similarity but do not establish human-like behavior in interactive settings or at the individual level.
  4. Limited experimental generalizability. The experiments mainly involve two model families at the 7B–8B scale and a small number of response domains. It remains unclear whether the findings generalize to larger models and more complex post-training pipelines.
  5. Unequal training budgets in Section 5. The comparison between Human-SFT and human-over-model DPO uses different amounts of training data. Consequently, the performance differences cannot be attributed entirely to the respective training objectives.
Questions for authors
  1. Validity of the human behavior approximation: How representative are the collected human responses of the underlying population response distribution? How sensitive are the conclusions to the assumption of equal weighting among observed responses?
  2. Human-likeness evaluation: Could the improvements reported in Figure 5 primarily reflect changes in response length, style, or dataset-specific characteristics rather than broader behavioral fidelity? Would evaluation with independent classifiers or human judgments provide additional evidence?
  3. Training budget comparison: Could the authors provide a more controlled comparison between Human-SFT and human-over-model DPO using matched training data sizes or token budgets?
  4. Robustness and generalization: How stable are the results across training seeds, larger model scales, and different response domains?
4 comments
AuthorsOpenReview ↗

Response to Reviewer aa6V (Part1/2)

We thank the reviewer for the careful reading. We respond to each point below.

In keeping with the conference's request for brevity, and to reduce the reading burden on the reviewer, we keep our responses concise.

1. Approximating the human response distribution (W1, Q1)

We agree that the data we chose need not represent the response distribution of humans in general. We chose SHP and StackExchange because both the responses and the preference votes come from humans: the same human responses can be used for SFT and, together with their votes, for preference optimization. This lets us separate the effect of the preference objective itself from the effect of model-generated data while holding the training text fixed (Sections 3 and 4).

Note that the formal analysis in Section 3 does not depend on this choice: for any human response distribution pHp_H, whenever the preference reward does not satisfy Eq. 5, the preference-optimal policy differs from pHp_H. What this limitation affects is therefore the size of the gap we measure, which applies only to responses on these platforms; the mechanism by which preference optimization moves a model away from the human response distribution does not depend on it.

On the sensitivity of our conclusions to equal weighting, we tested it directly in the controlled setting of Section 4.2. Two models start from the same Human-SFT checkpoint; one is trained with DPO on human preferences, and the other continues to fit the same human responses with equal weights. For each held-out prompt, we place all evaluation weight on the single human response most favorable to DPO; no other within-prompt weighting, including weighting by votes or using only the top-voted response, can favor DPO more. Even so, DPO's mean sequence NLL remains higher in all four settings.

We have added this limitation to Sections 6.

2. Theoretical novelty (W2)

We agree that the derivation in Section 3 uses the known exponential-tilting closed form, and our aim is not to propose a new optimization principle for achieving human-likeness. Standard KL-regularized analyses concern how far the optimized policy moves from the reference model π0\pi_0. In Section 3 we apply it to a different question: how the human response distribution pHp_H recorded in a dataset relates to the human preferences recorded in that same data.

Simple as it is, this formalization directly yields the two experimental designs of Section 4:

(1) When π0=pH\pi_0 = p_H, since Fx′′(λ)=Varpλ(rP)≥0F_x^{\prime\prime}(\lambda)=\mathrm{Var}_{p_\lambda}(r_P)\ge 0 (Eq. 9), the departure from the human distribution arises not because the preference direction opposes human behavior, but from non-uniform weighting itself: following the preferences, reversing them, or randomly reassigning the weights all push the target away from pHp_H. This leads directly to the reversed-preference control and the 40 random-reassignment controls in Section 4.1, and it is confirmed in the trained models.

(2) When π0≠pH\pi_0 \neq p_H, whether preference optimization can recover pHp_H depends on how the preference reward relates to the starting point π0\pi_0 (Eq. 5), which motivates our comparison of DPO from two starting points in Section 4.2.

AuthorsOpenReview ↗

Response to Reviewer aa6V (Part2/2)

3. Evaluating human-likeness (W3, Q2)

We agree that these metrics cannot show human-likeness in interactive settings or at the level of individuals, which is a considerably harder problem. We state this in the Limitations of Section 6.

On whether the improvement in Figure 5 comes mainly from length, style, or dataset properties: the source classifier never saw the output of any intervention during training or selection, and the compared responses are truncated to the same number of words, so length alone cannot determine distinguishability. Moreover, prompting clearly shortens responses and changes their style (Figure 5b), yet it changes distinguishability very little, which indicates that the classifier does not rely on length or style alone.

4. Training budget in Section 5 (W5, Q3)

To test whether the differences in Figure 5 come from the amount of training data and the number of updates, on Qwen2.5-7B / SHP we extended human-over-model DPO to the same pool of 17096 human responses used by full Human-SFT (267 updates per epoch), trained for three epochs, and selected the best checkpoint. The held-out human-response sequence NLL is 360.39 for DPO and 314.63 for SFT, a paired difference (DPO − SFT) of +45.76. In the equal-length evaluation, after removing the slots with anomalous DPO responses from all methods, the source DAUCD_{\mathrm{AUC}} is 0.88 for DPO and 0.69 for SFT, a paired difference of +0.19. With DPO scaled up to the same data, SFT still shows a clear advantage.

5. Robustness and generalization (W4, Q4)

With a limited, academic-scale compute budget (and having already spent nearly 10,000 GPU hours), we prioritized repeating every comparison across several model–dataset combinations, to make the conclusions as general as possible.


We thank the reviewer again for the constructive comments.

Thank you for the detailed rebuttal and additional experiments.

The responses address several of my concerns, particularly the training-budget comparison in Section 5. The additional experiment using a larger DPO training set provides useful evidence that Human-SFT's advantage is not solely explained by the amount of training data.

I also appreciate the clarifications regarding the theoretical contribution and the limitations of the human-likeness evaluation.

Some limitations remain, particularly regarding the representativeness of human response distributions, the scope of the human-likeness metrics, and generalization across model scales and domains. However, I recognize that these are partly limitations of the current study's scope.

Overall, the rebuttal strengthens my confidence in the empirical findings. I will maintain my original rating of 3 (Weak Acceptance).

Thank you for your quick and thoughtful response. We are glad that the additional matched-budget experiment and clarifications strengthened your confidence in our findings, and we truly appreciate your supportive assessment of our work. We will make sure to include all these updates in the next version of the paper.

Best regards,

The Authors

Comments on the paper

1 comment

Dear Reviewers,

We hope you all had a wonderful weekend!

We are writing to gently follow up on our rebuttal and manuscript revision. While we sincerely thank Reviewer aa6V for already engaging with our response, we wanted to check in with Reviewers xwyU, uixu, and sh85 to see if our responses and new experiments have resolved your questions:

  • Matched-budget DPO vs. Human-SFT comparison (Section 5) and unfiltered density-ratio control (Section 3.2) for Reviewer uixu;
  • Most DPO-favorable within-prompt weighting test (Section 4.2) and Base vs. Instruct coverage evaluation for Reviewer sh85;
  • Empirical approximation diagnostics for pHp_H and platform-distribution qualifications for Reviewer xwyU.

We remain fully available to answer any remaining questions and would greatly appreciate your feedback.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.