- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper examines the gap between automated personalization evaluations and human judgments. It decomposes personalization into extracting user attributes from conversation history, selecting attributes relevant to a new prompt, and incorporating them into a response. The study uses 550 conversations from 47 WildChat user histories and reports human judgments across all three stages. It finds that attributes extracted from real conversations are harder to validate than those from synthetic benchmarks, that LLM relevance decisions diverge from human judgments despite strong agreement among models, and that LLM judges evaluate personalized responses more favorably than the human panel. The authors also investigate attribute verification, trained relevance selection, and reward modeling as ways to improve agreement with the collected annotations.
Critical strengths
-
The staged evaluation distinguishes failures that an end-to-end personalization score would obscure. The paper separately examines whether an attribute is supported by conversation history, whether it should influence a response, and whether using human-selected relevant attributes produces a preferred answer. Attribute acceptance differs substantially between real and synthetic conversations, at approximately 58% versus 80% (Figure 2). For relevance selection, mean human–human, LLM–LLM, and human–LLM agreement are approximately 0.426, 0.597, and 0.300, respectively (§5.2). This is useful evidence that agreement among automated evaluators does not necessarily validate their underlying judgment standard.
-
The annotation effort supports practical calibration experiments. The authors go beyond documenting failures by training models for verification, relevance selection, and preference evaluation. The paper reports verifier F1 of 0.726 (Table 3), a relevance F1 improvement from 0.417 to 0.641 for Qwen3-4B after GRPO (Figure 3), and reward-model Spearman correlation of 0.441 (Table 5). These results suggest that the annotations can support learnable, useful evaluation tasks. Their significance lies in improving agreement with explicitly defined human targets, even though improved user satisfaction has not yet been established.
Critical weaknesses
-
The human evaluation does not directly establish the target users’ preferences. Stage 2 pairs external LIMA prompts with WildChat profiles, while Stage 3 asks three third-party annotators to evaluate responses from the perspective of those profiles. This is a reasonable annotation task, but agreement with that panel does not demonstrate agreement with the original users. The small panel and screening partly based on label distributions and mutual agreement further limit the range of preferences represented. The additional ten-participant analysis is encouraging, but its description does not clearly establish first-person evaluation. Consequently, the broader claims about user experience and a generally human-aligned personalization reward should be restricted to the evidence available.
-
The empirical contribution needs a sharper distinction from closely related work. AlpsBench already combines real-dialogue data with staged evaluation, while BESPOKE and PRISM-X provide first-person personalization evidence. These studies limit the novelty of using real interactions, involving humans, or decomposing the pipeline. The more distinctive contribution here is the fine-grained comparison of judgment standards across stages and the associated calibration tasks. The paper should explain more precisely which findings depend on this joint design and how they complement existing evidence. This would help establish the significance of the contribution without relying on broad claims about the need for human-centered evaluation.
Questions for authors
-
Whose preferences does the human evaluation represent? Are the three main Stage 3 annotators and the additional ten participants evaluating their own profiles and requests? If they primarily take the perspective of other users, how should the findings be scoped with respect to actual user experience?
-
What distinctive insight does the joint annotation design provide? Compared with AlpsBench’s staged evaluation and the first-person studies in BESPOKE/PRISM-X, which central finding depends on the present joint human annotation design and is not already covered by those studies?
4 comments
Official Comment by Authors (2/2)
W2. The empirical contribution needs a sharper distinction from closely related work. AlpsBench already combines real-dialogue data with staged evaluation, while BESPOKE and PRISM-X provide first-person personalization evidence. These studies limit the novelty of using real interactions, involving humans, or decomposing the pipeline. The more distinctive contribution here is the fine-grained comparison of judgment standards across stages and the associated calibration tasks. The paper should explain more precisely which findings depend on this joint design and how they complement existing evidence. This would help establish the significance of the contribution without relying on broad claims about the need for human-centered evaluation.
Q2. What distinctive insight does the joint annotation design provide? Compared with AlpsBench’s staged evaluation and the first-person studies in BESPOKE/PRISM-X, which central finding depends on the present joint human annotation design and is not already covered by those studies?
Thank you for suggesting these relevant works! We have also recently become aware of some of them. In particular, AlpsBench (March 2026) and PRISM-X (May 2026) were developed concurrently with our work, which was also completed in the first half of this year. We therefore did not have the opportunity to discuss them in detail in the paper. Nevertheless, we would be happy to share our thoughts here and incorporate these comparisons into our revision.
AlpsBench is closely related to our work, as both study multiple stages of personalization using real conversations. However, while AlpsBench focuses on how effectively models extract, retrieve, and utilize personalized memories, we additionally question whether using such memories is appropriate or beneficial in the first place. For example, successfully incorporating a user's interest in hiking may demonstrate memory utilization, but mentioning it in an unrelated programming answer could actually hurt the response.
BESPOKE collects first-person feedback on personalized search, providing detailed diagnoses of response quality, such as whether the content is sufficiently detailed or the tone matches user preferences. Our work instead diagnoses the underlying personalization decisions before generation. For example, if a user frequently requests French translations, should the model infer that they prefer French responses? And even if that preference is valid, should it influence an unrelated programming question? These questions cannot be answered through response-level feedback alone.
PRISM-X primarily evaluates personalization end-to-end, comparing different methods through real-user interactions and preference judgments. While this establishes which methods work better, it does not systematically isolate failures at different stages of personalization. Our work instead provides fine-grained, stage-wise diagnosis, distinguishing whether a failure arises from incorrectly inferred attributes, inappropriate attribute selection, or ineffective use of relevant information. This distinction matters because each failure requires a different intervention, which end-to-end evaluations alone cannot identify.
Overall, our joint annotation design provides actionable supervision at three distinct decision points—what to infer, when to use it, and how to evaluate its use. This allows us to distinguish upstream personalization errors from downstream generation and evaluation failures, rather than treating personalization quality as a single outcome. We will incorporate the discussion above into our revision.
Following up on our rebuttal
Dear Reviewer 4fAk, thank you again for your valuable feedback! We hope our responses and additional experiments have adequately addressed your questions. We would be happy to address any further questions or suggestions you may have. If you feel your concerns have been resolved, we would greatly appreciate your consideration in updating your assessment. Thank you again for your time and thoughtful review!
Second Follow-up on the rebuttal
Dear Reviewer 4fAk,
As the discussion period is approaching its end, we wanted to check whether you have had a chance to review our rebuttal and whether any concerns remain. We would be happy to provide further clarification if needed, and would appreciate your consideration of an updated rating if our responses have addressed your concerns. Thank you!
Official Comment by Authors (1/2)
We sincerely thank the reviewer for the thoughtful and constructive feedback. Below, we respond to each point and include several additional results that we hope will clarify your concerns.
We can confirm that the additional 10 participants in Section 6.3 evaluated personalized responses to their own requests, based on their own preferences and profiles, rather than judging on behalf of other users. We will make this first-person evaluation setting clearer in the revision.
We acknowledge that the three main third-party annotators cannot fully represent the preferences of the original WildChat users. However, we believe our current setting may actually underestimate the gap between LLM judgments and real user preferences. Our annotators received task-specific training to encourage consistent judgments, and we aggregated judgments from three annotators to reduce individual noise. Both choices are intended to produce more standardized evaluations, which may be closer to the consistent judgments produced by LLMs than the heterogeneous preferences of real users. In contrast, real users are unlikely to receive such training, and their judgments may reflect more diverse, subjective, and context-dependent preferences.
Our additional analysis of the 10 first-party participants provides empirical support for this hypothesis. All five LLM judges show lower agreement with first-party users than with third-party annotators. The gaps are particularly pronounced for Claude-S4.6 (ρ = 0.062 vs. 0.301), Llama-3.3-70B (0.071 vs. 0.311), and GPT-5.4 (0.099 vs. 0.253), with statistically significant differences for all three (p < 0.05). These results further suggest that third-party evaluation may underestimate LLM–human misalignment.
We view this study as a first step toward robust human evaluations of personalization. We will include these additional findings and clarify that our main evaluation measures alignment with third-party human judgments, while the first-party analysis provides complementary evidence about actual user preferences.