acceptodds
Under review as a conference paper at ICLR 2027

VitaBench 2.0: Evaluating Personalized and Proactive Agents in Long-Term User Interactions

Abstract

Large language models (LLMs) have evolved into interactive agents that collaborate with users in real-world tasks. Effective collaboration in such settings increasingly depends on understanding the user beyond what is explicitly stated, as user intent is often reflected in fragmented daily interactions and requires both personalized modeling and proactive interaction. However, existing agent benchmarks primarily evaluate reasoning and tool use, largely overlooking the challenges of inferring and leveraging user preferences in realistic scenarios. To address this gap, we introduce VitaBench 2.0, a benchmark for evaluating personalized and proactive agent behavior in long-term user interactions. In VitaBench 2.0, tasks are organized as temporally ordered sequences for individual users, where preferences are embedded in fragmented and heterogeneous interactions. Successful completion of tasks requires the agent to continuously extract, utilize, and update user preferences from these interactions. We further evaluate proactiveness through tasks that require agents to recognize missing information and actively acquire it from users or environments before making decisions. To support systematic analysis, we provide an extensible memory interface that enables controlled comparison across different memory architectures. We benchmark a diverse set of frontier proprietary and open-source LLMs. Results show that real-world personalization remains highly challenging even for state-of-the-art models, revealing a substantial gap between current capabilities and practical requirements. Extensive analysis further reveals the failure modes and capability bottlenecks of current agents in real-world personalized decision-making, providing insights for future model improvements.

Primary area:
datasets and benchmarks
Keywords:
Large Language Model, Agent, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_uFQbOpenReview ↗
Rating
3
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes VitaBench 2.0 for evaluate personalized and proactive agents that must provide long-term assistance to users in realistic scenarios. The framework should extract and update implicit user preferences from fragmented histories, and proactively asks missing contexts when necessary. Experimental results across a diverse range of LLMs reveal that reliable, long-term personalized task completion remains a significant challenge, particularly in maintaining preference consistency and initiating proactive information gathering.

Critical strengths
  1. The benchmark evaluates personalization through executed service actions along a user’s timeline, bridging interactive tool-use, long-term personalization, and clarification benchmarks. Its rubrics change over time with the user’s preferences, which enables preference updating to be measured directly through task success.

  2. The benchmark is evaluated on 27 configurations spanning a diverse set of LLMs, including think-on and think-off settings. Each configuration is run under three memory settings (Full Context, Agentic Memory, and RAG Memory) through an extensible memory interface, and the LLM-based evaluator is validated against human judgments. The paper also explicitly bounds its own claims and provides full prompts and compute details, which makes the results easy to interpret and reproduce.

Critical weaknesses
  1. One of the main contribution for this benchmark is about the preference drifting, but the main results do not compare task performance on drifted versus stable preferences, and neither Updating F1 (C.3) nor the task-index trend (Figure 3) directly tests this.

  2. Section 3.3 states that agents should decide when additional information is needed and proactively ask the user for it. However, the rubrics do not penalize unnecessary questions, so an agent that always asks would not be disadvantaged. As a result, the benchmark captures whether agents ask when needed, but not whether they refrain when not.

Questions for authors
  1. Even with ground-truth preferences, the best models still fail roughly half of the tasks (Figure 4). How many of these failures come from preference-derived rubrics versus explicit-request rubrics, i.e., how much of the main score reflects personalization rather than general agent capability?

  2. How many of the 771 tasks are proactive tasks, and is the “Personalization” score in Figure 4 computed only on non-proactive tasks? Without this, the size and reliability of the reported proactiveness gap cannot be assessed.

Reviewer_Vk21OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces VitaBench 2.0, a benchmark for evaluating personalized and proactive agents over long-term user interactions. It contains 771 tasks across 56 users, with evolving preferences and interaction histories spanning multiple service domains. The benchmark evaluates whether agents can infer, update, and apply user preferences during task execution, as well as proactively seek missing information when needed. The authors evaluate a wide range of LLMs under different memory settings and analyze their performance on personalization and proactive tasks. Results show that current agents still struggle to consistently apply user preferences, and that memory mechanisms do not always improve task performance.

Critical strengths
  1. Well-designed benchmark. The benchmark covers several important aspects of long-term personalization, including preference extraction, updating, utilization, and proactive interaction. The continuous task sequences provide a realistic setting for evaluating personalized agents.

  2. High-quality dataset construction. The benchmark includes 771 tasks across 56 users, with over 2,000 human-annotated preferences. The authors put considerable effort into constructing diverse user profiles, interaction histories, and executable task environments.

  3. Comprehensive experiments. The paper evaluates a wide range of LLMs under different memory configurations, with repeated runs and detailed analyses of reasoning, memory, and proactive behavior.

  4. Useful empirical findings. The results show that memory does not always improve performance and that agents often fail to apply user preferences even when the correct information is available. These findings provide useful insights into the limitations of current personalized agents.

Critical weaknesses

Limited validation with real users. The user profiles and interaction histories are manually constructed rather than collected from real users. While this allows controlled evaluation, it remains unclear how well the benchmark captures the complexity of real-world preference changes and user behavior.

Questions for authors

See weakness

1 comment

Dear Reviewer, thank you again for your valuable feedback. With the discussion period closing in less than one day, we would greatly appreciate it if you could review our responses and let us know whether any concerns remain. We are happy to discuss any further questions. Thanks for your time and consideration.

Reviewer_oetjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VitaBench 2.0 is introduced in this paper to assess personalization and proactive interaction capabilities of agents that assist one and the same user for an extended period of time. This benchmark includes 56 users created manually and 771 tasks ordered by time including those for delivery, in-store consumption, and online travel services. The agent should extract and update the user preferences based on the incrementally observed dialogues and behavioral records and finish the corresponding service tasks using tools. Some of the tasks require proactive gathering of missing context in addition to that. The authors compare Full Context, Agentic Memory, and RAG Memory in 27 different model/mode configurations evaluated four times under each memory setting. The highest Avg@4 and Pass⁴ scores achieved under the Full Context setting are 0.503 and 0.337, respectively.

Critical strengths

S1. The research addresses an interesting real-life problem. The agent should infer user preferences in the course of interaction with the user in the past and adapt its behavior accordingly due to the changing needs of the user. Such problem is examined on the basis of executable tasks that involve ordering and booking and where the models have to make sure that user constraints are satisfied in the process of searching, comparison and executing actions. It is crucial for the agent to be able to remember the preference, since it allows it to take appropriate actions regarding it; this is the key value of the benchmark.

S2. The experiments include a broad variety of settings and offer interesting analyses of system behavior. Table 1 contains the results of the experiments performed on various model families, using different reasoning modes, and also three different memory settings, giving the average task performance and consistency of results in the repeated execution of models. Furthermore, there are the experiments with ground-truth preferences, metrics of preference extraction and updating, and even some examples of failures (for instance, ordering desserts without accounting for the merchant-rating constraint).

Critical weaknesses

W1. The evidence supporting the use of certain implicit preferences as hard evaluation constraints could be strengthened In the example with dessert ordering, the authors describe the agent’s inference of the rating constraint based on the fact that the user selected only merchants rated at least 4.3. Yet this can be seen as a general habit rather than a hard constraint. Further evidence on whether independent annotators can recover these hard constraints from the user’s history would help assess the reliability of these evaluation criteria. Moreover, the user simulation setup prevents clarifying user’s preferences explicitly, thus the errors might be due to discrepancies between the constraints and the preferences inferable from the user’s history.

W2. The relative contributions of different capabilities to task failures remain difficult to distinguish. Figure 4 indicates low scores for the proactive tasks; however, these tasks involve preference inference, acquisition of missing context, and tool execution at the same time. The reported results leave open how performance would change if the missing context were provided upfront and how failures are distributed across interaction stages. So it is unclear whether the poor scores indicate problems with clarification or execution. The human–LLM judge agreement validation lacks sample size, sampling procedure and the unit of analysis information.

Questions for authors

Q1. Is there a hard requirement in the user’s history of a minimum rating of 4.3 for merchants? In case it can only be deduced from past orders, how did the authors verify that it was a hard requirement and not a preference allowing for trade-offs? Q2. How would performance on proactive tasks change if the missing context were provided upfront rather than obtained through clarification? Would it be possible for the authors to provide the results of this comparison on the same set of tasks in order to separate clarification failures from execution failures? Q3. How many samples have been taken to reach the value of 94.5% human–LLM judge agreement, and how have these samples been selected? Did 94.5% correspond to individual rubric judgments or task success/failure? Q4. Figure 4 shows the scores of 50.7 and 43.8 for Kimi and MiniMax respectively while in the main text the scores are 43.8 and 42.6. Could the authors clarify which set of values is correct?

1 comment

Dear Reviewer, thank you again for your valuable feedback. With the discussion period closing in less than one day, we would greatly appreciate it if you could review our responses and let us know whether any concerns remain. We are happy to discuss any further questions. Thanks for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.