acceptodds
Under review as a conference paper at ICLR 2027

Re-Centering Humans in LLM Personalization

Abstract

Despite growing interest, most evaluations of large language models' (LLMs') personalization abilities have relied on synthetic data. It remains unclear how well current personalization systems work for real users. In this paper, we study the gap in LLM personalization performance when using synthetic versus human data. We collect human conversations (550 conversations) and judgments across three stages of personalization: extracting user attributes from conversations (5,949 judgments), pairing relevant attributes with new prompts (11,919), and incorporating relevant attributes into a personalized response (1,101). Incorporating human data reveals system limitations at each stage. Models struggle to extract attributes from human conversations, disagree with human judgments on relevant attributes, and generate personalized responses that humans judge no better than generic responses (though that LLM judges widely rate as better). We introduce several lightweight training-based interventions that shift automated personalization evaluation closer to human data in all three stages. However, our exploration in Stage 3 suggests that a one-size-fits-all model may obscure meaningful differences in personalization preferences across users. Our collected data provides a foundation for studying how models should extract, select, and incorporate user information in ways that humans find useful.

Primary area:
foundation or frontier models, including LLMs
Keywords:
personalization, human-centered evaluation, large language model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4fAkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines the gap between automated personalization evaluations and human judgments. It decomposes personalization into extracting user attributes from conversation history, selecting attributes relevant to a new prompt, and incorporating them into a response. The study uses 550 conversations from 47 WildChat user histories and reports human judgments across all three stages. It finds that attributes extracted from real conversations are harder to validate than those from synthetic benchmarks, that LLM relevance decisions diverge from human judgments despite strong agreement among models, and that LLM judges evaluate personalized responses more favorably than the human panel. The authors also investigate attribute verification, trained relevance selection, and reward modeling as ways to improve agreement with the collected annotations.

Critical strengths
  1. The staged evaluation distinguishes failures that an end-to-end personalization score would obscure. The paper separately examines whether an attribute is supported by conversation history, whether it should influence a response, and whether using human-selected relevant attributes produces a preferred answer. Attribute acceptance differs substantially between real and synthetic conversations, at approximately 58% versus 80% (Figure 2). For relevance selection, mean human–human, LLM–LLM, and human–LLM agreement are approximately 0.426, 0.597, and 0.300, respectively (§5.2). This is useful evidence that agreement among automated evaluators does not necessarily validate their underlying judgment standard.

  2. The annotation effort supports practical calibration experiments. The authors go beyond documenting failures by training models for verification, relevance selection, and preference evaluation. The paper reports verifier F1 of 0.726 (Table 3), a relevance F1 improvement from 0.417 to 0.641 for Qwen3-4B after GRPO (Figure 3), and reward-model Spearman correlation of 0.441 (Table 5). These results suggest that the annotations can support learnable, useful evaluation tasks. Their significance lies in improving agreement with explicitly defined human targets, even though improved user satisfaction has not yet been established.

Critical weaknesses
  1. The human evaluation does not directly establish the target users’ preferences. Stage 2 pairs external LIMA prompts with WildChat profiles, while Stage 3 asks three third-party annotators to evaluate responses from the perspective of those profiles. This is a reasonable annotation task, but agreement with that panel does not demonstrate agreement with the original users. The small panel and screening partly based on label distributions and mutual agreement further limit the range of preferences represented. The additional ten-participant analysis is encouraging, but its description does not clearly establish first-person evaluation. Consequently, the broader claims about user experience and a generally human-aligned personalization reward should be restricted to the evidence available.

  2. The empirical contribution needs a sharper distinction from closely related work. AlpsBench already combines real-dialogue data with staged evaluation, while BESPOKE and PRISM-X provide first-person personalization evidence. These studies limit the novelty of using real interactions, involving humans, or decomposing the pipeline. The more distinctive contribution here is the fine-grained comparison of judgment standards across stages and the associated calibration tasks. The paper should explain more precisely which findings depend on this joint design and how they complement existing evidence. This would help establish the significance of the contribution without relying on broad claims about the need for human-centered evaluation.

Questions for authors
  1. Whose preferences does the human evaluation represent? Are the three main Stage 3 annotators and the additional ten participants evaluating their own profiles and requests? If they primarily take the perspective of other users, how should the findings be scoped with respect to actual user experience?

  2. What distinctive insight does the joint annotation design provide? Compared with AlpsBench’s staged evaluation and the first-person studies in BESPOKE/PRISM-X, which central finding depends on the present joint human annotation design and is not already covered by those studies?

4 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We sincerely thank the reviewer for the thoughtful and constructive feedback. Below, we respond to each point and include several additional results that we hope will clarify your concerns.


W1. The human evaluation does not directly establish the target users’ preferences. Stage 2 pairs external LIMA prompts with WildChat profiles, while Stage 3 asks three third-party annotators to evaluate responses from the perspective of those profiles. This is a reasonable annotation task, but agreement with that panel does not demonstrate agreement with the original users. The small panel and screening partly based on label distributions and mutual agreement further limit the range of preferences represented. The additional ten-participant analysis is encouraging, but its description does not clearly establish first-person evaluation. Consequently, the broader claims about user experience and a generally human-aligned personalization reward should be restricted to the evidence available.

Q1. Whose preferences does the human evaluation represent? Are the three main Stage 3 annotators and the additional ten participants evaluating their own profiles and requests? If they primarily take the perspective of other users, how should the findings be scoped with respect to actual user experience?

We can confirm that the additional 10 participants in Section 6.3 evaluated personalized responses to their own requests, based on their own preferences and profiles, rather than judging on behalf of other users. We will make this first-person evaluation setting clearer in the revision.

We acknowledge that the three main third-party annotators cannot fully represent the preferences of the original WildChat users. However, we believe our current setting may actually underestimate the gap between LLM judgments and real user preferences. Our annotators received task-specific training to encourage consistent judgments, and we aggregated judgments from three annotators to reduce individual noise. Both choices are intended to produce more standardized evaluations, which may be closer to the consistent judgments produced by LLMs than the heterogeneous preferences of real users. In contrast, real users are unlikely to receive such training, and their judgments may reflect more diverse, subjective, and context-dependent preferences.

Our additional analysis of the 10 first-party participants provides empirical support for this hypothesis. All five LLM judges show lower agreement with first-party users than with third-party annotators. The gaps are particularly pronounced for Claude-S4.6 (ρ = 0.062 vs. 0.301), Llama-3.3-70B (0.071 vs. 0.311), and GPT-5.4 (0.099 vs. 0.253), with statistically significant differences for all three (p < 0.05). These results further suggest that third-party evaluation may underestimate LLM–human misalignment.

We view this study as a first step toward robust human evaluations of personalization. We will include these additional findings and clarify that our main evaluation measures alignment with third-party human judgments, while the first-party analysis provides complementary evidence about actual user preferences.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

W2. The empirical contribution needs a sharper distinction from closely related work. AlpsBench already combines real-dialogue data with staged evaluation, while BESPOKE and PRISM-X provide first-person personalization evidence. These studies limit the novelty of using real interactions, involving humans, or decomposing the pipeline. The more distinctive contribution here is the fine-grained comparison of judgment standards across stages and the associated calibration tasks. The paper should explain more precisely which findings depend on this joint design and how they complement existing evidence. This would help establish the significance of the contribution without relying on broad claims about the need for human-centered evaluation.

Q2. What distinctive insight does the joint annotation design provide? Compared with AlpsBench’s staged evaluation and the first-person studies in BESPOKE/PRISM-X, which central finding depends on the present joint human annotation design and is not already covered by those studies?

Thank you for suggesting these relevant works! We have also recently become aware of some of them. In particular, AlpsBench (March 2026) and PRISM-X (May 2026) were developed concurrently with our work, which was also completed in the first half of this year. We therefore did not have the opportunity to discuss them in detail in the paper. Nevertheless, we would be happy to share our thoughts here and incorporate these comparisons into our revision.

AlpsBench is closely related to our work, as both study multiple stages of personalization using real conversations. However, while AlpsBench focuses on how effectively models extract, retrieve, and utilize personalized memories, we additionally question whether using such memories is appropriate or beneficial in the first place. For example, successfully incorporating a user's interest in hiking may demonstrate memory utilization, but mentioning it in an unrelated programming answer could actually hurt the response.

BESPOKE collects first-person feedback on personalized search, providing detailed diagnoses of response quality, such as whether the content is sufficiently detailed or the tone matches user preferences. Our work instead diagnoses the underlying personalization decisions before generation. For example, if a user frequently requests French translations, should the model infer that they prefer French responses? And even if that preference is valid, should it influence an unrelated programming question? These questions cannot be answered through response-level feedback alone.

PRISM-X primarily evaluates personalization end-to-end, comparing different methods through real-user interactions and preference judgments. While this establishes which methods work better, it does not systematically isolate failures at different stages of personalization. Our work instead provides fine-grained, stage-wise diagnosis, distinguishing whether a failure arises from incorrectly inferred attributes, inappropriate attribute selection, or ineffective use of relevant information. This distinction matters because each failure requires a different intervention, which end-to-end evaluations alone cannot identify.

Overall, our joint annotation design provides actionable supervision at three distinct decision points—what to infer, when to use it, and how to evaluate its use. This allows us to distinguish upstream personalization errors from downstream generation and evaluation failures, rather than treating personalization quality as a single outcome. We will incorporate the discussion above into our revision.

AuthorsOpenReview ↗

Following up on our rebuttal

Dear Reviewer 4fAk, thank you again for your valuable feedback! We hope our responses and additional experiments have adequately addressed your questions. We would be happy to address any further questions or suggestions you may have. If you feel your concerns have been resolved, we would greatly appreciate your consideration in updating your assessment. Thank you again for your time and thoughtful review!

AuthorsOpenReview ↗

Second Follow-up on the rebuttal

Dear Reviewer 4fAk,

As the discussion period is approaching its end, we wanted to check whether you have had a chance to review our rebuttal and whether any concerns remain. We would be happy to provide further clarification if needed, and would appreciate your consideration of an updated rating if our responses have addressed your concerns. Thank you!

Reviewer_vnH1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies an interesting problem: existing benchmarks over-rely on synthetic data and therefore cannot evaluate the real-world ability of LLM personalization well. Based on real dialogues and human judgments, it reveals the failure modes of personalization models in three stages. I think the identified problem and the analysis are interesting and beneficial to LLM personalization research. However, the proposed methods for helping current models avoid these failures are not satisfying, and I think a better way to build a more realistic benchmark would be more interesting.

Critical strengths
  1. This paper identifies an interesting and important problem: existing benchmarks over-rely on synthetic data, which means they cannot evaluate real-world personalization performance well.
  2. The three-stage analysis (attribute extraction, relevance matching, and response generation) is based on real-world personalization data. The analysis method is solid, and the conclusions are interesting and convincing.
Critical weaknesses
  1. The proposed methods that tackle the three identified problems do not seem promising. Addressing these problems by adding extra plugins to existing models does not seem very meaningful. An inspiring method or guidance for synthesizing more realistic benchmarks would be more beneficial to LLM personalization research.
Questions for authors
  1. Can the authors provide some ideas on improving synthetic data generation to build higher-quality, more realistic benchmarks? I think this would be more interesting and useful.
  2. Can the authors further explain the motivation or applications of the proposed methods for the three identified problems? This would help clarify their contribution.
  3. The human-aligned reward model is only evaluated by rank correlation (Spearman's ρ). A further human evaluation of its outputs would make this evaluation more solid.
5 comments
AuthorsOpenReview ↗

Official Comment by Authors

We sincerely thank the reviewer for the thoughtful and constructive feedback. Below, we respond to each point and include several additional results that we hope will clarify your concerns.


W1. The proposed methods that tackle the three identified problems do not seem promising. Addressing these problems by adding extra plugins to existing models does not seem very meaningful. An inspiring method or guidance for synthesizing more realistic benchmarks would be more beneficial to LLM personalization research.

The primary contribution of our work is identifying and characterizing the gaps between synthetic evaluation and real human data throughout the personalization pipeline, rather than proposing fundamentally new methods. To our knowledge, these gaps have not previously been systematically examined across all three stages of personalization. Our proposed interventions are intended as initial steps toward addressing the identified challenges, demonstrating that incorporating human data can already lead to meaningful improvements. We believe understanding where and why existing evaluations fall short is a necessary foundation for developing more effective solutions, including the more realistic synthetic benchmarks suggested by the reviewer. We hope our findings will provide concrete directions for future research in this area.


Q1. Can the authors provide some ideas on improving synthetic data generation to build higher-quality, more realistic benchmarks? I think this would be more interesting and useful.

We believe our findings provide several concrete directions for developing more realistic synthetic personalization benchmarks. First, diversity alone does not guarantee realism: synthetic conversations should better reflect the incomplete, ambiguous, and sometimes misleading evidence about user attributes found in real interactions. Second, benchmarks should explicitly evaluate when not to personalize, incorporating plausible but irrelevant attributes and cases where personalization provides no additional benefit. Third, our findings suggest that synthetic evaluations should reward meaningful improvements in response quality rather than explicit attribute mentions, which LLM judges often overvalue. More broadly, we believe human data should be used to calibrate synthetic benchmarks at each stage. These insights provide practical guidance for improving synthetic benchmarks beyond simply increasing their diversity or scale. We will include these insights into the discussion.


Q2. Can the authors further explain the motivation or applications of the proposed methods for the three identified problems? This would help clarify their contribution.

Our main goal is not to propose fundamentally new methods, but to identify challenges in existing personalization pipelines that are overlooked by synthetic evaluations. The proposed interventions serve as proof-of-concept solutions, demonstrating how human data can be used to address these challenges with relatively simple modifications to existing systems. We deliberately explore lightweight approaches to show that meaningful improvements do not necessarily require complex new architectures, but instead new data. More broadly, these interventions illustrate how our collected data can support the development and evaluation of more effective personalization methods in future work.


Q3. The human-aligned reward model is only evaluated by rank correlation (Spearman's ρ). A further human evaluation of its outputs would make this evaluation more solid.

We would like to clarify that our reward model is already evaluated directly against human judgments. Specifically, both human annotators and the reward model evaluate the same generated responses by assigning scalar preference scores. Rank correlation with human preferences is a standard way to evaluate reward models, as these models assign scalar scores rather than generate responses themselves. Spearman's ρ measures how well these scores align with human preferences. Our reward model is evaluated against human ratings on held-out users, achieving ρ = 0.441 compared with 0.349 for the best zero-shot LLM judge.

AuthorsOpenReview ↗

Following up on our rebuttal

Dear Reviewer vnH1, thank you again for your valuable feedback! We hope our responses and additional experiments have adequately addressed your questions. We would be happy to address any further questions or suggestions you may have. If you feel your concerns have been resolved, we would greatly appreciate your consideration in updating your assessment. Thank you again for your time and thoughtful review!

AuthorsOpenReview ↗

Second Follow-up on the rebuttal

Dear Reviewer vnH1,

As the discussion period is approaching its end, we wanted to check whether you have had a chance to review our rebuttal and whether any concerns remain. We would be happy to provide further clarification if needed, and would appreciate your consideration of an updated rating if our responses have addressed your concerns. Thank you!

Reviewer_TEGnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Evaluations of LLM personalization mostly rely on synthetic personas, simulated conversations and LLM judges, so it is unclear how well personalization works for real users. The authors frame personalization as a three-stage pipeline: extracting user attributes from conversation history, matching attributes to a new prompt by relevance, and generating a personalized response. They build a dataset from real WildChat users and collect crowd-annotator judgments at each stage. They compare human and LLM judgments, and in the first stage compare real with synthetic conversations. They also train lightweight interventions on the human annotations: an attribute verifier, a relevance classifier and a reasoning model trained with reinforcement learning, and a personalization reward model.

They report that attribute extraction is harder on real conversations than on synthetic ones, that LLMs select many more attributes as relevant than humans do, and that humans often rate personalized responses as no better than generic ones while LLM judges rate them higher. The interventions bring automated evaluation closer to human judgments, and a small exploratory analysis suggests different user groups judge personalization differently.

Critical strengths

The paper evaluates the whole personalization pipeline with human judgments on real conversations and breaks failures down by stage. The error analysis of extracted attributes (overgeneralization 53.9%, missing evidence 20.3%, task-context confusion 16.1%, Table 2) is informative. The finding that LLMs select far more attributes as relevant than humans do (humans mark about 20% as relevant, LLMs 40–60%, with precision below 0.4 for every LLM) points to a concrete failure that synthetic evaluations would miss. The released human annotations are a useful resource.

The analysis of why LLM judges differ from humans is very insightful. It links explicit attribute mentions in generated responses to how judges reward them (Figures 4b and 5). The interventions are simple and clearly motivated, and the authors acknowledge limits such as annotator noise and individual differences. The ethics statement reports IRB approval and pay of about $18 per hour.

Critical weaknesses

The headline framing appears to be broader than the evidence. A synthetic-versus-real comparison exists only in Stage 1 (10 users from each of three synthetic datasets against 47 WildChat users, one extractor, Llama-3.3-70B, and three annotators with κ=0.314). In Stages 2 and 3 the comparison is human versus LLM judges on WildChat-derived attributes and LIMA prompts, with no synthetic counterpart. The Stage 1 gap (80% vs 58% accepted) is almost entirely the “uncertain” label (20% vs 39%, Figure 2b), the category likely least reliable at that agreement. The diversity claim uses a real subset chosen by diversity-based sampling (0.737) and measured by embedding diversity, while unfiltered real data (0.656) is below PrefEval (0.689) and PersonaLens (0.706). The annotators are crowd workers, not the WildChat users, so the “human” judgments are third-party judgments of a profile.

The Stage 2–3 evidence is thin, and some conclusions do not follow from it. The same three annotators judge all stages, and the abstract’s judgment counts triple-count the items (5,949, 11,919 and 1,101 are 3 × 1,983, 3,973 and 367). Stage 3 rests on 80 items and 367 item–attribute ratings (κ=0.31), and the mean human ratings per generator (3.65, 3.37, 3.14, 3.00, 2.71) have no intervals. The abstract says humans judge responses “no better than generic”, yet three of five generator means are above 3, and the 54.6% “no better” figure includes ties at 3.0. The text also says Gemma “degrades” responses, but its mean is 3.00. LLM judges are called unreliable at ρ≈0.35 (Claude 0.349, Llama 0.344), but the reported human–human Spearman is 0.325, and the trained reward model’s 0.44 is described as near the practical limit. These comparisons are not on the same footing: a judge’s correlation with the mean of three annotators is not the same as one annotator’s correlation with another. The Figure 5 correlation uses five models, and p=0.04 equals the one-sided exact value for ρ=0.9 with n=5 (two-sided about 0.08, my calculation). The “one-size-fits-all” claim in the abstract rests on 10 participants (5 per group, ρ 0.33 vs 0.14, no intervals).

The interventions are weakly evaluated. Stage 2 splits by prompt (test prompts 33–37, 5 of 41), while Stage 1 splits by user. Since each prompt is paired with 4 of 47 users (164 user–prompt pairs), the same users’ attributes can appear in both train and test. It is unclear whether the zero-shot LLMs are scored on the same split as RoBERTa and GRPO. With about 20% of attributes relevant, a classifier that always answers “not relevant” would reach about 80% accuracy (my calculation), against 0.859 for RoBERTa and 0.870 for GRPO, and no such baseline or threshold-calibrated LLM is reported. The statement that prompt refinement gives only marginal gains has no numbers. The verify-and-refine result (58% to above 90% acceptance on 250 attributes) can rise simply by dropping attributes, and coverage is not reported.

Questions for authors

The judgments come from three crowd annotators who are not the WildChat users. How do they reflect what the real users would judge, and how do you reconcile this with the goal of evaluating personalization for real users? The synthetic-versus-real comparison is only in Stage 1, with 10 users per synthetic dataset, one extractor and three annotators with κ=0.314. What are the per-dataset results, and does the gap hold for other extractors and for rejected labels separately from uncertain ones? Stage 3 rests on 80 items and 367 item–attribute ratings. How are the ratings split across the five generators, what is the unit behind the 54.6%, and what are the intervals on the per-generator means? Three of five means are above 3, so how does this support “no better than generic” in the abstract? LLM judges are called unreliable at ρ≈0.35, while human–human Spearman is 0.325 and the trained reward model’s 0.44 is called near the limit. What is each individual annotator’s correlation with the mean of the other two, and how does the Figure 5 result hold with five models (two-sided p)? In Stage 2 the split is by prompt (5 test prompts) and users and attributes recur across prompts. Is there train–test overlap, were zero-shot LLMs scored on the same test split, and how do the results compare with the always-“not relevant” baseline at about 80% accuracy and with a threshold-calibrated LLM? In the verify-and-refine experiment (58% to above 90% on 250 attributes), how many attributes were removed versus revised, do these attributes overlap with the verifier’s training data, and how much valid information is lost? The individual-differences claim in the abstract rests on 10 participants (5 frequent and 5 occasional AI users). What supports it beyond ρ 0.33 vs 0.14 without intervals, and were these participants rating their own preferences or those of others?

7 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/4)

We sincerely thank the reviewer for the thoughtful and constructive feedback. Below, we respond to each point and include several additional results that we hope will clarify your concerns.


W1.1. A synthetic-versus-real comparison exists only in Stage 1; In Stages 2 and 3 the comparison is human versus LLM judges on WildChat-derived attributes and LIMA prompts, with no synthetic counterpart.

We agree that the direct comparison between synthetic and real conversation datasets is conducted primarily in Stage 1. However, our broader goal is to investigate the limitations of relying on synthetic data and automated evaluation throughout the personalization pipeline, rather than to perform an identical dataset comparison at every stage.

After Stage 1, our pipeline focuses on individual user attributes and their associated judgements rather than their source conversations. Therefore, in Stages 2 and 3, we focus on comparing human and LLM judgments, as existing synthetic personalization benchmarks commonly rely on LLMs for relevance matching and response evaluation. Our findings show that these automated judgments can substantially differ from human preferences, revealing another important limitation of synthetic evaluation. We will clarify this distinction in the revised introduction and in Sections 5 and 6 where Stages 2 and 3 are discussed.


W1.2. The Stage 1 gap (80% vs 58% accepted) is almost entirely the “uncertain” label (20% vs 39%, Figure 2b), the category likely least reliable at that agreement.

We would like to clarify that the “uncertain” judgments reflect meaningful uncertainty about the extracted attribute itself rather than unreliable annotations. For example, inferring that “the user is learning French” from one or two translation requests is clearly an overgeneralization, yet the available evidence is insufficient to confidently “accept” or “reject” the attribute. **Our main finding is that such ambiguity occurs substantially more frequently in real interactions but often absent from synthetic conversations constructed around predefined user attributes. ** Since both real and synthetic data were evaluated by the same annotators under the same protocol, the significantly higher uncertainty rate for WildChat (39.1%, 95% CI [35.9, 42.3]) compared with synthetic data (19.8%, [17.5, 22.0]) reflects a meaningful difference in attribute verifiability. This suggests that synthetic benchmarks underestimate the difficulty of reliably extracting user attributes from real conversations. We will further clarify this distinction in the revision.


W1.3. The diversity claim uses a real subset chosen by diversity-based sampling (0.737) and measured by embedding diversity, while unfiltered real data (0.656) is below PrefEval (0.689) and PersonaLens (0.706).

We would like to clarify that we do not claim real conversational data is inherently more diverse than synthetic data. In fact, our results show that unfiltered WildChat exhibits lower diversity than some synthetic benchmarks, partly because real-world usage is dominated by common tasks such as coding. Nevertheless, WildChat's large user pool allows us to sample a substantially more diverse subset, whereas the diversity of synthetic datasets remains relatively stable across sampling strategies.

Prior work has emphasized diversity as an important goal in constructing synthetic personalization datasets [1][2]. However, our findings suggest that diversity alone is insufficient to explain the real–synthetic gap: despite comparable diversity, synthetic datasets substantially underestimate the difficulty of extracting reliable user attributes. We will further clarify this distinction in the revision.

[1]. Kim, T. S., Lee, Y., Park, Y., Kim, J., Kim, Y.-H., & Kim, J. (2025). CUPID: Evaluating Personalized and Contextualized Alignment of LLMs from Interactions. Conference on Language Modeling (COLM 2025).

[2]. Singh, A., Hsu, S., Hsu, K., Mitchell, E., Ermon, S., Hashimoto, T., Sharma, A., & Finn, C. (2025). FSPO: Few-Shot Preference Optimization of Synthetic Preference Data in LLMs Elicits Effective Personalization to Real Users. arXiv preprint arXiv:2502.19312.

AuthorsOpenReview ↗

Official Comment by Authors (2/4)

W1.4. The annotators are crowd workers, not the WildChat users, so the “human” judgments are third-party judgments of a profile.

Q1. The judgments come from three crowd annotators who are not the WildChat users. How do they reflect what the real users would judge, and how do you reconcile this with the goal of evaluating personalization for real users?

We agree that third-party annotators cannot fully represent the preferences of the original WildChat users. However, we believe our current setting may actually underestimate the gap between LLM judgments and real user preferences. Our annotators received task-specific training to encourage consistent judgments, and we aggregated judgments from three annotators to reduce individual noise. Both choices are intended to produce more standardized evaluations, which may be closer to the consistent judgments produced by LLMs than the heterogeneous preferences of real users. In contrast, real users are unlikely to receive such training, and their judgments may reflect more diverse, subjective, and context-dependent preferences. Thus, the disagreement we observe even under this relatively controlled setting suggests that the challenges of aligning LLM evaluations with actual user preferences could be even greater.

Our additional analysis of the 10 first-party participants in Section 6.3 provides empirical support for this hypothesis. All five LLM judges show lower agreement with first-party users than with third-party annotators. The gaps are particularly pronounced for Claude-S4.6 (ρ = 0.062 vs. 0.301), Llama-3.3-70B (0.071 vs. 0.311), and GPT-5.4 (0.099 vs. 0.253), with statistically significant differences for all three (p < 0.05). These results further suggest that third-party evaluation may underestimate LLM-human misalignment.

We view this study as a first step toward robust human evaluations of personalization and will include these additional findings in the revision.


W2.1. The Stage 2–3 evidence is thin, and some conclusions do not follow from it. The same three annotators judge all stages, and the abstract’s judgment counts triple-count the items (5,949, 11,919 and 1,101 are 3 × 1,983, 3,973 and 367). Stage 3 rests on 80 items and 367 item–attribute ratings (κ=0.31), and the mean human ratings per generator (3.65, 3.37, 3.14, 3.00, 2.71) have no intervals.

Q3. Stage 3 rests on 80 items and 367 item–attribute ratings. How are the ratings split across the five generators, what is the unit behind the 54.6%, and what are the intervals on the per-generator means?

We confirm that the reported 5,949, 11,919, and 1,101 annotations represent individual human judgments. Our experiments across all three stages demonstrate that these annotations are sufficient to train models that better align with human judgments. More importantly, our key findings are well-supported by statistical analyses. For example,in Stage 3, four of five LLM judges assign significantly higher personalization ratings than humans on the same response pairs, with mean differences ranging from +0.25 to +1.00 (95% CIs).

For Stage 3, we confirm that the 54.6% figure is calculated over individual ratings: it represents the proportion of the 1,101 ratings that are ≤3 (i.e., personalization is not preferred over the generic response), with a 95% bootstrap CI of [48.9%, 59.9%] obtained by resampling response pairs.

We also computed per-generator 95% bootstrap confidence intervals:

GeneratorResponse pairsAttribute pairsMean rating [95% CI]
Claude-S4.616603.65 [3.37, 3.93]
GPT-5.419903.37 [3.17, 3.59]
Llama-3.316813.14 [2.88, 3.42]
Gemma-414683.00 [2.84, 3.20]
Qwen3.515682.71 [2.46, 2.98]
Overall803673.18 [3.04, 3.31]

We will clarify the annotation counts and incorporate these confidence intervals into the revised paper.

AuthorsOpenReview ↗

Official Comment by Authors (3/4)

W2.2. The abstract says humans judge responses “no better than generic”, yet three of five generator means are above 3, and the 54.6% “no better” figure includes ties at 3.0. The text also says Gemma “degrades” responses, but its mean is 3.00.

Q4. Three of five means are above 3, so how does this support “no better than generic” in the abstract?

We would like to clarify that our claim is not that personalization never improves response quality, but rather that a substantial proportion of personalized responses fail to outperform their generic counterparts. Specifically, 54.6% of the 1,101 individual ratings are ≤3 (95% CI [48.9%, 59.9%]), including ties where neither response is preferred. We believe this is particularly meaningful because these responses were generated using attributes already judged relevant by humans, yet personalization still frequently fails to provide additional value.

We also agree that the benefits vary across generators. While Claude-S4.6 (3.65) and GPT-5.4 (3.37) show statistically significant improvements, the other three generators do not show significant improvements over the generic baseline (as we shown in the table above of Q3 response). We agree that Gemma-4's mean of 3.00 indicates neutral preference rather than degradation, so we will correct the statement to "Gemma-4 shows no clear improvement, while Qwen3.5 can even degrade the performance."


W2.3. LLM judges are called unreliable at ρ≈0.35 (Claude 0.349, Llama 0.344), but the reported human–human Spearman is 0.325, and the trained reward model’s 0.44 is described as near the practical limit. These comparisons are not on the same footing: a judge’s correlation with the mean of three annotators is not the same as one annotator’s correlation with another. The Figure 5 correlation uses five models, and p=0.04 equals the one-sided exact value for ρ=0.9 with n=5 (two-sided about 0.08, my calculation).

Q5. What is each individual annotator’s correlation with the mean of the other two, and how does the Figure 5 result hold with five models (two-sided p)?

We would like to clarify that these correlations serve different purposes. The zero-shot LLM judges' ~0.35 and our trained reward model's 0.44 are directly comparable, as both are evaluated against the same aggregated human judgments (i.e., a human majority vote). In contrast, the human–human correlation (0.325) is reported to measure inter-annotator agreement and annotation quality, rather than as a direct baseline for LLM evaluation.

For a more directly comparable analysis, we additionally computed each annotator's correlation with the mean of the other two, obtaining 0.340, 0.293, and 0.361 (mean 0.331). Against these same leave-one-out targets, LLM judges achieve 0.323 (Llama), 0.314 (Claude), 0.270 (GPT), 0.134 (Qwen), and 0.080 (Gemma), all below the human average. Moreover, under cross-validation with matched leave-one-out targets, our fine-tuned reward model achieves 0.386, compared with 0.351 for held-out human annotators, further supporting its effectiveness and explaining why we describe its performance as approaching the practical limit of this annotation setting.

Regarding Figure 5, the reported correlation is ρ = 0.90, with an exact permutation p-value of 0.042 (one-sided) or 0.083 (two-sided). Given the small number of models (n = 5), we will clarify the statistical test and present this finding as suggestive rather than conclusive in the revision.


W2.4. The “one-size-fits-all” claim in the abstract rests on 10 participants (5 per group, ρ 0.33 vs 0.14, no intervals)

Q8. What supports it beyond ρ 0.33 vs 0.14 without intervals, and were these participants rating their own preferences or those of others?

We would like to clarify that these participants evaluated responses based on their own preferences, rather than judging on behalf of other users. Our study reveals a difference in reward model alignment between less frequent (ρ = 0.33) and very frequent AI users (ρ = 0.14). Although the small sample size prevents a statistically conclusive finding (difference = +0.19, bootstrap 95% CI [−0.04, 0.41]; Mann–Whitney p = 0.10), the observed gap suggests potentially meaningful differences in how well aggregated reward models align with different user populations. We view this as an encouraging initial finding and plan to investigate it further with larger and more diverse user populations. We will further clarify the exploratory nature of this analysis in the revised version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.