- Rating
- 3
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes VitaBench 2.0 for evaluate personalized and proactive agents that must provide long-term assistance to users in realistic scenarios. The framework should extract and update implicit user preferences from fragmented histories, and proactively asks missing contexts when necessary. Experimental results across a diverse range of LLMs reveal that reliable, long-term personalized task completion remains a significant challenge, particularly in maintaining preference consistency and initiating proactive information gathering.
Critical strengths
-
The benchmark evaluates personalization through executed service actions along a user’s timeline, bridging interactive tool-use, long-term personalization, and clarification benchmarks. Its rubrics change over time with the user’s preferences, which enables preference updating to be measured directly through task success.
-
The benchmark is evaluated on 27 configurations spanning a diverse set of LLMs, including think-on and think-off settings. Each configuration is run under three memory settings (Full Context, Agentic Memory, and RAG Memory) through an extensible memory interface, and the LLM-based evaluator is validated against human judgments. The paper also explicitly bounds its own claims and provides full prompts and compute details, which makes the results easy to interpret and reproduce.
Critical weaknesses
-
One of the main contribution for this benchmark is about the preference drifting, but the main results do not compare task performance on drifted versus stable preferences, and neither Updating F1 (C.3) nor the task-index trend (Figure 3) directly tests this.
-
Section 3.3 states that agents should decide when additional information is needed and proactively ask the user for it. However, the rubrics do not penalize unnecessary questions, so an agent that always asks would not be disadvantaged. As a result, the benchmark captures whether agents ask when needed, but not whether they refrain when not.
Questions for authors
-
Even with ground-truth preferences, the best models still fail roughly half of the tasks (Figure 4). How many of these failures come from preference-derived rubrics versus explicit-request rubrics, i.e., how much of the main score reflects personalization rather than general agent capability?
-
How many of the 771 tasks are proactive tasks, and is the “Personalization” score in Figure 4 computed only on non-proactive tasks? Without this, the size and reliability of the reported proactiveness gap cannot be assessed.
Dear Reviewer, thank you again for your valuable feedback. With the discussion period closing in less than one day, we would greatly appreciate it if you could review our responses and let us know whether any concerns remain. We are happy to discuss any further questions. Thanks for your time and consideration.