- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the challenge of providing personalized services when user intent is unclear and preferences vary. It introduces the AdaptArena benchmark and the retrieval based AdaptiveAgent framework to evaluate how agents infer and use implicit preferences from past interactions. The results show that current agents still have clear weaknesses in preference inference and web task execution. Even when agents correctly understand user preferences, they may still fail to complete tasks. The study identifies implicit preference inference and reliable execution as key challenges for personalized web agents.
Critical strengths
- The paper introduces the AdaptArena benchmark. It covers multiple web environments and settings with one or two preferences. Agents need to infer user preferences from past interaction trajectories when instructions lack detail. This evaluates their ability to personalize their behavior at test time.
- The paper proposes the AdaptiveAgent framework. It retrieves past trajectories, infers hidden preferences, and uses them to guide task execution. This provides a simple and reproducible standard process. The results show that its performance remains much worse than when agents directly receive the true preferences.
- The paper highlights key challenges in personalization. All models perform worse in settings with two preferences, showing the difficulty of handling combined preferences. Correct preference inference also does not guarantee task success, as action execution remains a major bottleneck.
Critical weaknesses
- All models rely on GPT-5-nano to extract preferences. If this module makes mistakes, even strong models may struggle to complete the task. The paper needs to compare more models on preference extraction.
- The paper focuses on 10 personas, and models generate historical trajectories based on clearly defined preferences. It does not verify whether these 10 personas cover the complex and changing behaviors and traits of real users.
- The history is small. Each user has only 12 historical trajectories. This setting does not test whether the system handles large amounts of irrelevant records, repeated records, and similar but unrelated behavior as the history grows.
- Does the evaluation consider the performance of models together with their agent harnesses, such as Codex and Claude Code? Including these systems can further strengthen the paper.
Questions for authors
See above.